Weak supervision image defect segmentation method and system based on text guidance

Abstract

The invention discloses a weak supervision image defect segmentation method and system based on text guidance. The method comprises the following steps: acquiring an industrial defect image data set; an image segmentation model is established and trained, wherein the preliminary prediction module is used for extracting multi-scale image block features and inputting the multi-scale image block features into a segmentation decoder to obtain a preliminary segmentation prediction result; the text guiding module is used for constructing a foreground text and a background text, inputting the foreground text and the background text into a text encoder of a CLIP model, generating a foreground text feature set and a background text feature set, splicing the foreground text feature set and the background text feature set, and inputting the spliced foreground text feature set and the spliced background text feature set and multi-scale image block features into Obtaining an image-text correlation score graph and weighting the image-text correlation score graph and the preliminary segmentation prediction result to obtain a final segmentation prediction result; and detecting a to-be-segmented industrial defect image by using the trained image segmentation model to obtain a final segmentation prediction result. The problems of sample scarcity, high labeling cost, insufficient model generalization ability and the like in industrial detection tasks are solved, and the precision and efficiency of image defect detection are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and industrial automatic detection, and particularly relates to a weakly supervised image defect segmentation method and system based on text guidance. BACKGROUND

[0002] Industrial defect detection is a key link in the modern intelligent manufacturing process, and its precision and robustness are directly related to product quality control and production line automation level. In recent years, image segmentation methods based on deep neural networks have been widely used in defect detection, especially convolutional neural networks (CNN) and other structures have shown strong ability in extracting semantic features and structural details. However, the data in the industrial scene usually has the following challenges: on the one hand, pixel-level labeling is costly, and it is extremely difficult to obtain large-scale training samples; on the other hand, the defect morphology is diverse, the background is complex, and the sample class distribution is extremely unbalanced, and traditional full-supervised methods are difficult to adapt to the needs of industrial deployment in low-labeled or unlabeled environments.

[0003] To overcome the above problems, weakly supervised semantic segmentation (WSSS) has gradually become a research hotspot. Such methods usually only use image-level labels or class information, and generate pseudo labels by combining class activation maps (CAM) to replace accurate labeling. On this basis, some research introduces a multi-modal model to enhance the semantic understanding ability through image-text alignment. However, the existing weakly supervised segmentation methods based on multi-modal models generally have the following shortcomings: 1) only rely on static global text and shallow image features for coarse-grained semantic association, lack of pixel-level alignment capability, and difficult to accurately locate the target area; 2) cannot effectively utilize the hierarchical structure features within the image, such as the contrastive information between local texture and global semantics, resulting in blurred segmentation boundaries; 3) do not explicitly model the contrast between foreground and background, leading to false activation when there are co-occurring interference backgrounds. SUMMARY

[0004] The purpose of the present application is to solve the above problems, and to propose a weakly supervised image defect segmentation method and system based on text guidance, to solve the problems of sample scarcity, high labeling cost and insufficient model generalization ability in industrial detection tasks, and to improve the precision and efficiency of image defect detection.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] The weakly supervised image defect segmentation method based on text guidance proposed by the present application comprises the following steps:

[0007] S1, obtaining an industrial defect image dataset composed of a plurality of industrial defect images and corresponding class labels;

[0008] S2, an image segmentation model is established and trained using an industrial defect image dataset, the image segmentation model comprising a preliminary prediction module and a text guiding module, wherein:

[0009] The preliminary prediction module performs the following operations:

[0010] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, the multi-scale image block features comprising N image block features and class labels;

[0011] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result, the segmentation decoder comprising a plurality of convolution modules connected in sequence;

[0012] The text guiding module performs the following operations:

[0013] For the foreground class and the background class in the industrial defect image dataset, foreground text and background text are respectively constructed;

[0014] All of the foreground text and the background text are input into a text encoder of a CLIP model to correspondingly generate a foreground text feature set and a background text feature set;

[0015] The foreground text feature set and the background text feature set are spliced and then input into a context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set;

[0016] A graph-text relevance score map is obtained according to the multi-scale image block features, the enhanced foreground text feature set, and the enhanced background text feature set;

[0017] The preliminary segmentation prediction result is weighted with the graph-text relevance score map to obtain a final segmentation prediction result;

[0018] S3, the trained image segmentation model is used to detect a to-be-segmented industrial defect image to obtain a final segmentation prediction result.

[0019] Preferably, the preliminary prediction module further performs the following operations:

[0020] An auxiliary classifier is introduced at the output end of the middle layer of the ViT encoder, and when the industrial defect image dataset is input into the ViT encoder, the output feature map of the auxiliary classifier is obtained at the same time, and the output feature map of the auxiliary classifier is used to form an auxiliary class activation map by a CAM method;

[0021] A dual-threshold partitioning method is used to construct auxiliary pseudo-labels for the auxiliary class activation map, and the image block features in the multi-scale image block features corresponding to the foreground area in the auxiliary pseudo-labels are used as the foreground set, and the image block features in the multi-scale image block features corresponding to the background area are used as the background set;

[0022] After the multi-scale image patch features are input into the first classifier, the initial activation map is formed using the CAM method;

[0023] The single threshold partitioning method is used to construct the initial pseudo label for the initial activation map.

[0024] Preferably, the ViT encoder performs the following operations:

[0025] Divide the industrial defect image into N non-overlapping image blocks and flatten them into a first vector;

[0026] Projecting the first vector through a linear mapping function to obtain a feature vector;

[0027] The feature vector is concatenated with the introduced class feature vector and then input into the feature extractor to obtain the multi-scale image block features. ,in, For the Image block features, n=1~N, is the class label, the feature extractor is a Transformer encoder composed of 12 layers of sequentially connected Transformer modules, and the class feature vector is a one-dimensional vector of 1×C, where the element corresponding to the defect category of the industrial defect image is recorded as 1, and the rest are recorded as 0. C is the total number of defect categories in all category labels.

[0028] Preferably, the auxiliary classifier is a fully connected layer and is connected to the output end of the 10th layer Transformer module of the ViT encoder.

[0029] Preferably, the first classifier is configured to perform the following operations:

[0030] Perform dot product operation on multi-scale image patch features and weights of the first classifier to obtain linear correlation map , where N is the multi-scale image block feature The number of image patch features in , is the total number of defect categories in all category labels;

[0031] Mapping Linear Dependencies After the ReLU activation function, the response map is obtained, and the response map is normalized to the maximum value to obtain the initial activation map , Linear dependency map height, For linear correlation mapping Width of the.

[0032] Preferably, the auxiliary pseudo label is constructed by using a double threshold division method on the auxiliary class activation map, specifically as follows:

[0033] When the value of the kth pixel point in the auxiliary class activation map is less than or equal to the preset lower threshold, it is considered that the corresponding pixel point is a background region and is marked as 0; when the value of the kth pixel point in the auxiliary class activation map is greater than or equal to the preset upper threshold, it is considered that the corresponding pixel point is a foreground region and is marked as 1, k = 1 ~ K, K is the total number of pixel points of the auxiliary class activation map; otherwise, it is considered that the corresponding pixel point is an uncertain region and is marked as 255, and the values of the preset number of neighbor pixel points of the corresponding pixel point in the auxiliary pseudo label are compared, when the proportion of equal to 1 is less than or equal to the preset ratio, it is considered that the corresponding pixel point is a background region, when the proportion of equal to 1 is greater than the preset ratio, it is considered that the corresponding pixel point is a foreground region.

[0034] The initial pseudo label is constructed by using a single threshold division method on the initial activation map, specifically as follows:

[0035] It is judged whether the value of each pixel point in the initial activation map is within the preset division threshold, if yes, it is considered that the corresponding pixel point is a background region and is marked as 0, if not, it is considered that the corresponding pixel point is a foreground region and is marked as 1.

[0036] Preferably, the image segmentation model further comprises a multi-level contrast optimization module and a total loss calculation module, wherein:

[0037] The multi-level contrast optimization module comprises a global inter-block feature contrast module, a local-global feature contrast module and an image-text semantic contrast module, wherein

[0038] The global inter-block feature contrast module is used to calculate the inter-block similarity of the image block features within the foreground set and the heterogeneous similarity of the image block features between the foreground set and the background set based on the cosine similarity, and to generate an inter-block contrast loss function by maximizing the inter-block similarity while minimizing the heterogeneous similarity;

[0039] The local-global feature contrast module performs the following operations:

[0040] The class label in the multi-scale image block feature and the text embedding are projected into the same space to obtain a first feature;

[0041] In the industrial defect image, Q local image blocks are formed by random cropping multiple times with size HxW, and label blocks corresponding to the Q local image blocks are cropped on the auxiliary pseudo label, wherein H is the height of the local image block, W is the width of the local image block, and Q is a positive integer;

[0042] Inputting Q local image blocks into the ViT encoder obtains first extracted features, and taking the features of foreground regions in corresponding label blocks of the first extracted features as first visual features and the features of background regions as second visual features;

[0043] According to the auxiliary pseudo label, the first visual feature, the second visual feature, and the first feature, an InfoNCE contrast loss is generated, and the text embedding is the spliced enhanced foreground text feature set and the enhanced background text feature set;

[0044] The image-text semantic contrast module is used to maximize the cosine similarity between the first feature and the enhanced foreground text feature set, and minimize the cosine similarity between the first feature and the enhanced background text feature set to generate an image-text contrast loss function;

[0045] The total loss calculation module is used to construct a total loss function, which is obtained by weighting the first classification loss, the second classification loss, the block contrast loss function, the InfoNCE contrast loss, the image-text contrast loss function, and the segmentation loss, wherein:

[0046] The first classification loss is used to calculate a multi-label cross-entropy loss according to the output feature map of the first classifier and the corresponding category label to supervise the first classifier;

[0047] The second classification loss is used to calculate a multi-label cross-entropy loss according to the output feature map of the auxiliary classifier and the corresponding category label to supervise the auxiliary classifier;

[0048] The segmentation loss is used to calculate a cross-entropy loss by pixel-by-pixel comparison between the output feature map of the segmentation decoder and the initial pseudo label to supervise the segmentation decoder.

[0049] Preferably, the segmentation decoder includes three convolution modules connected in sequence, wherein the first convolution module and the second convolution module each include a convolution layer and a ReLU activation function connected in sequence, and the third convolution module includes a convolution layer; the context decoder includes a plurality of Transformer modules connected in sequence, which is a positive integer; the foreground text and the background text are constructed using a preset natural language template.

[0050] Preferably, the image-text relevance score map is obtained according to the multi-scale image block feature, the enhanced foreground text feature set, and the enhanced background text feature set, specifically as follows:

[0051] The enhanced foreground text feature set and the enhanced background text feature set are spliced to form a text embedding;

[0052] The normalized multi-scale image block feature and the normalized text embedding are subjected to Einstein summation operation to obtain an image-text relevance score map.

[0053] A text-guided weakly supervised image defect segmentation system based on any of the above text-guided weakly supervised image defect segmentation methods, comprising a data acquisition module, a model construction module and a model prediction module, wherein:

[0054] The data acquisition module is configured to acquire an industrial defect image dataset composed of a plurality of industrial defect images and corresponding category labels.

[0055] The model construction module is configured to establish an image segmentation model and train the image segmentation model using the industrial defect image dataset, wherein the image segmentation model comprises a preliminary prediction module and a text-guided module.

[0056] The preliminary prediction module performs the following operations:

[0057] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, wherein the multi-scale image block features comprise N image block features and a class label.

[0058] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result, wherein the segmentation decoder comprises a plurality of convolution modules connected in sequence.

[0059] The text-guided module performs the following operations:

[0060] For the foreground category and the background category in the industrial defect image dataset, foreground text and background text are respectively constructed.

[0061] All of the foreground text and the background text are input into a text encoder of a CLIP model to correspondingly generate a foreground text feature set and a background text feature set.

[0062] The foreground text feature set and the background text feature set are spliced and then input into a context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set.

[0063] A graph-text relevance score map is obtained according to the multi-scale image block features, the enhanced foreground text feature set and the enhanced background text feature set.

[0064] The preliminary segmentation prediction result is weighted with the graph-text relevance score map to obtain a final segmentation prediction result.

[0065] The model prediction module is configured to use the trained image segmentation model to detect a to-be-segmented industrial defect image to obtain a final segmentation prediction result.

[0066] Compared with the prior art, the present application has the following advantages:

[0067] The application combines text semantic embedding guidance mechanism and multi-level feature comparison strategy, takes ViT encoder and CLIP model as feature extraction network of image and text, realizes preliminary positioning of target region by constructing image-text correlation score map, and introduces triple contrast mechanism to realize fine perception of target defects and effective suppression of background noise. Specifically: first, the industrial defect image dataset is obtained, and the image-level label is used as the only supervision information to avoid high-cost pixel-level labeling to realize weakly supervised semantic segmentation and improve the landing in actual industrial scene; the industrial defect image is input into the ViT encoder to extract multi-scale image block features, and the output feature map of the introduced auxiliary classifier is obtained to form initial pseudo label and auxiliary pseudo label, and foreground text and background text are constructed to input the text encoder of the CLIP model to obtain semantic embedding and introduce the context decoder for semantic enhancement, realizing the deep fusion of semantic information and image features, and significantly improving the positioning ability of the defect region; and aiming at the foreground blur or background interference problem caused by the insufficient activation accuracy of the activation map, a multi-level contrast learning mechanism is designed, that is, the global block contrast, local-global contrast and image-text contrast are combined with the pixel-level segmentation loss to optimize the image feature representation, enhance the separability of foreground and background, effectively suppress the pseudo-activated region, improve the segmentation boundary clarity and overall robustness, output more fine and robust features and activation map, realize the deep fusion of text semantic prior and multi-level visual features, and fully utilize the contrast information provided by the background semantics to guide the model to perform structure perception and noise suppression in image defect recognition. The weights of the trained image segmentation model can be used for pixel-level segmentation prediction of the industrial defect image, and the end-to-end training capability is possessed; the application breaks through the dependence of traditional methods on pixel-level labels, and through the cooperative design of semantic enhancement, spatial alignment and feature discrimination optimization, the semantic alignment relationship between image and text is fully tapped, the focusing ability and robustness of the model on the defect region are improved, and high-performance, low-labeling-cost industrial defect image segmentation is realized. Under the condition of only image classification level (image-level label) labeling sample, pixel-level segmentation of industrial defects is realized, which has high-precision segmentation capability, significantly reduces the time cost of data labeling, realizes structural modular design and strong semantic perception, the overall architecture is clear, the coupling degree between modules is low, the maintenance of the whole model can be realized by adjusting each module separately, and it is suitable for weakly supervised segmentation tasks of multiple industrial defects, especially for industrial defect detection tasks under complex background interference. The application significantly reduces the manual labeling cost, improves the generalization ability and industrial deployment adaptability in actual application. The accuracy, stability and engineering applicability of the weakly supervised industrial defect segmentation are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The flowchart of the weakly supervised image defect segmentation method based on text guidance of the application;

[0069] Figure 2 A structural schematic diagram of an image segmentation model of the present application;

[0070] Figure 3 A structural schematic diagram of a segmentation decoder of the present application;

[0071] Figure 4 A comparison schematic diagram of an industrial defect image and its segmentation results under different conditions, wherein, (A) is an industrial defect image, (B) is an artificial annotation result corresponding to the industrial defect image, and (C) is an industrial defect segmentation result corresponding to the industrial defect image of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0073] It should be noted that when a component is referred to as being "connected" with another component, it can be directly connected with the other component or there can be a component in between. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0074] The core of the method is to fuse the CLIP model and the ViT encoder. In the absence of pixel-level annotation, the positioning and segmentation accuracy of the defect area are improved by means of text prompt guidance and hierarchical feature optimization strategy.

[0075] Embodiment 1:

[0076] As shown in the figure, a weakly supervised image defect segmentation method based on text guidance includes the following steps: Figures 1-4

[0077] S1, an industrial defect image dataset composed of a plurality of industrial defect images and corresponding category labels is obtained.

[0078] In the process of constructing an industrial defect image dataset with classification information for weakly supervised segmentation, the industrial defect images are classified and organized according to the category labels. According to the defect category, the industrial defect images of the same defect category are stored in the same folder and have the corresponding category label (as an image-level label), so as to facilitate the automatic association of the defect category and the industrial defect image, which can also be adjusted according to the actual situation.​

[0079] S2, an image segmentation model is established and trained using an industrial defect image dataset, the image segmentation model comprising a preliminary prediction module and a text-guided module, wherein:

[0080] The preliminary prediction module performs the following operations:

[0081] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, the multi-scale image block features comprising N image block features and class labels;

[0082] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result, the segmentation decoder comprising a plurality of convolution modules connected in sequence;

[0083] The text-guided module performs the following operations:

[0084] For the foreground class and the background class in the industrial defect image dataset, foreground text and background text are respectively constructed;

[0085] All of the foreground text and the background text are input into a text encoder of a CLIP model to correspondingly generate a foreground text feature set and a background text feature set;

[0086] The foreground text feature set and the background text feature set are spliced and then input into a context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set;

[0087] A graph-text correlation score map is obtained according to the multi-scale image block features, the enhanced foreground text feature set, and the enhanced background text feature set;

[0088] The preliminary segmentation prediction result is weighted with the graph-text correlation score map to obtain a final segmentation prediction result.

[0089] In an embodiment, the preliminary prediction module further performs the following operations:

[0090] An auxiliary classifier is introduced at the output end of the middle layer of the ViT encoder, and when the industrial defect image dataset is input into the ViT encoder, an output feature map of the auxiliary classifier is simultaneously obtained, and the output feature map of the auxiliary classifier is used to form an auxiliary class activation map by a CAM method;

[0091] An auxiliary pseudo-label is constructed for the auxiliary class activation map by using a double-threshold division method, and image block features in the multi-scale image block features corresponding to foreground regions in the auxiliary pseudo-label are taken as a foreground set, and image block features in the multi-scale image block features corresponding to background regions in the auxiliary pseudo-label are taken as a background set;

[0092] After the multi-scale image patch features are input into the first classifier, the initial activation map is formed using the CAM method;

[0093] The single threshold partitioning method is used to construct the initial pseudo label for the initial activation map.

[0094] In one embodiment, the ViT encoder performs the following operations:

[0095] Divide the industrial defect image into N non-overlapping image blocks and flatten them into a first vector;

[0096] Projecting the first vector through a linear mapping function to obtain a feature vector;

[0097] The feature vector is concatenated with the introduced class feature vector and then input into the feature extractor to obtain the multi-scale image block features. ,in, For the Image block features, n=1~N, is the class label, the feature extractor is a Transformer encoder composed of 12 layers of sequentially connected Transformer modules, and the class feature vector is a one-dimensional vector of 1×C, where the element corresponding to the defect category of the industrial defect image is recorded as 1, and the rest are recorded as 0. C is the total number of defect categories in all category labels.

[0098] In one embodiment, the auxiliary classifier is a fully connected layer and is connected to the output end of the 10th layer Transformer module of the ViT encoder.

[0099] In one embodiment, the first classifier is configured to perform the following operations:

[0100] Perform dot product operation on multi-scale image patch features and weights of the first classifier to obtain linear correlation map , where N is the multi-scale image block feature The number of image patch features in , is the total number of defect categories in all category labels;

[0101] Mapping Linear Dependencies After the ReLU activation function, the response map is obtained, and the response map is normalized to the maximum value to obtain the initial activation map , Linear dependency map height, Linear dependency map width.

[0102] In one embodiment, a dual threshold partitioning method is used to construct auxiliary pseudo labels for the auxiliary class activation map, as follows:

[0103] When the value of the kth pixel point in the auxiliary class activation map is less than or equal to the preset lower threshold, the corresponding pixel point is considered as a background region and marked as 0; when the value of the kth pixel point in the auxiliary class activation map is greater than or equal to the preset upper threshold, the corresponding pixel point is considered as a foreground region and marked as 1, k = 1 ~ K, K is the total number of pixel points of the auxiliary class activation map; otherwise, the corresponding pixel point is considered as an uncertain region and marked as 255, and the values of the preset number of neighboring pixel points of the corresponding pixel point in the auxiliary pseudo label are compared, when the proportion of the values equal to 1 is less than or equal to the preset ratio, the corresponding pixel point is considered as a background region, and when the proportion of the values equal to 1 is greater than the preset ratio, the corresponding pixel point is considered as a foreground region;

[0104] The initial pseudo label is constructed for the initial activation map by using a single threshold division method, and the construction is specifically as follows:

[0105] It is judged whether the value of each pixel point in the initial activation map is within the preset division threshold, if yes, the corresponding pixel point is considered as a background region and marked as 0, and if no, the corresponding pixel point is considered as a foreground region and marked as 1.

[0106] In an embodiment, the image segmentation model further comprises a multi-level contrast optimization module and a total loss calculation module, wherein:

[0107] The multi-level contrast optimization module comprises a global inter-block feature contrast module, a local-global feature contrast module and an image-text semantic contrast module, wherein:

[0108] The global inter-block feature contrast module is used to calculate the inter-block similarity of the image block features in the foreground set and the heterogeneous similarity of the image block features between the foreground set and the background set based on the cosine similarity, and to generate an inter-block contrast loss function by maximizing the inter-block similarity while minimizing the heterogeneous similarity;

[0109] The local-global feature contrast module performs the following operations:

[0110] The class label in the multi-scale image block feature and the text embedding are projected into the same space to obtain a first feature;

[0111] In the industrial defect image, Q local image blocks are formed by random cropping multiple times with a size of HxW, and label blocks corresponding to the Q local image blocks are cropped on the auxiliary pseudo label, wherein H is the height of the local image block, W is the width of the local image block, and Q is a positive integer;

[0112] The Q local image blocks are input into the ViT encoder to obtain a first extraction feature, and the features of the foreground region in the corresponding label block of the first extraction feature are taken as a first visual feature, and the features of the background region are taken as a second visual feature;

[0113] InfoNCE contrast loss is generated based on the auxiliary pseudo-label, the first visual feature, the second visual feature, and the first feature. The text is embedded into the concatenated enhanced foreground text feature set and the enhanced background text feature set.

[0114] The image-text semantic comparison module is used to maximize the cosine similarity between the first feature and the foreground text feature set, while minimizing the cosine similarity between the first feature and the background text feature set to generate the image-text comparison loss function;

[0115] The total loss calculation module is used to construct the total loss function, which is obtained by weighting the first classification loss, second classification loss, inter-block contrast loss function, InfoNCE contrast loss, image-text contrast loss function, and segmentation loss.

[0116] The first classification loss is used to calculate the multi-label cross entropy loss based on the output feature map of the first classifier and the corresponding category label to supervise the first classifier;

[0117] The second classification loss is used to calculate the multi-label cross entropy loss based on the output feature map of the auxiliary classifier and the corresponding category label to supervise the auxiliary classifier;

[0118] The segmentation loss is used to compare the output feature map of the segmentation decoder with the initial pseudo label pixel by pixel to calculate the cross entropy loss to supervise the segmentation decoder.

[0119] In one embodiment, the segmentation decoder includes three sequentially connected convolution modules, wherein the first convolution module and the second convolution module each include sequentially connected convolution layers and ReLU activation functions, and the third convolution module includes a convolution layer; the context decoder includes Transformer modules connected in sequence, is a positive integer; the foreground text and background text are constructed using preset natural language templates.

[0120] In one embodiment, an image-text correlation score map is obtained based on the multi-scale image block features, the enhanced foreground text feature set, and the enhanced background text feature set, as follows:

[0121] Concatenate the enhanced foreground text feature set and the enhanced background text feature set to form a text embedding;

[0122] The normalized multi-scale image block features and the normalized text embedding are subjected to Einstein summation to obtain the image-text correlation score map.

[0123] Specifically, if Figure 2 As shown in the figure, the image segmentation model execution process is as follows:

[0124] 1) Preliminary prediction module:

[0125] An auxiliary classifier is introduced at the output end of the middle layer of the ViT encoder, and the industrial defect image dataset is input into the ViT encoder for feature extraction to obtain the multi-scale image block features output by the ViT encoder. and auxiliary class activation maps output by the auxiliary classifier ; Use double threshold division method to activate auxiliary class Constructing auxiliary pseudo labels , and the auxiliary pseudo labels Multi-scale image patch features corresponding to the middle foreground area The image block features in are used as the foreground set , auxiliary pseudo-label Multi-scale image patch features corresponding to the background area The image patch features in are used as background sets , foreground set and background collection Can be used for subsequent multi-level feature comparison optimization; multi-scale image block features Input the first classifier and segmentation decoder respectively to obtain the initial activation map and preliminary segmentation prediction results ; Use single threshold partitioning method to divide the initial activation map Constructing initial pseudo labels .

[0126] Among them, the industrial defect images in the industrial defect image dataset are passed through the ViT (Vision Transformer) encoder to extract multi-scale image block features , used for subsequent multi-level comparison and learning. Further, the ViT encoder performs the following operations: the input industrial defect image is first divided into N non-overlapping image blocks, and the image blocks are flattened into the first vector , through the linear mapping function, the first vector Projecting into the feature space to obtain the feature vector , is a learnable linear layer, which is then combined with the introduced class feature vector After splicing (such as using the Concat function to splice along the channel dimension), the image is input into the feature extractor (Transformer encoder) for context modeling and global feature extraction to obtain multi-scale image block features. ,in, represents the feature extractor (Transformer encoder), For the The image block features, i.e. The features extracted by the feature extractor for each image block are: Class label, i.e., class feature vector Feature extracted by feature extractor, class feature vector is a one-dimensional vector of 1xC, where the element corresponding to the defect category to which the industrial defect image belongs is recorded as 1, and the rest is recorded as 0, and C is the total number of defect categories in all category labels. The preliminary prediction module can extract image block features with spatial and contextual information, and introduce class feature vectors to aggregate global semantic information and provide a basis for subsequent foreground-background contrast modeling. The output end of the 10th layer of the ViT encoder (Transformer Encoder Block) is provided with an auxiliary classifier to obtain classification features, and the auxiliary classifier is implemented through a fully connected layer and generates an auxiliary class activation map based on the CAM method (class activation map generation method, CAM) on the classification features output by the auxiliary classifier , the auxiliary class activation map is divided into background and foreground regions by a double-threshold division method, and each pixel point in the auxiliary class activation map is divided into background and foreground regions, and the classification features output by the auxiliary classifier are used as intermediate feature representations for extracting local semantic information. The ViT encoder is a component in the ViT model known to those skilled in the art, i.e., the ViT model after deleting the MLP classification module (MLP Head), which will not be described here.

[0127] Initial activation map satisfies the following formula:

[0128] ,

[0129] In the formula, is the mapping map of the cth defect category in the linear correlation mapping map, is the weight of the cth defect category of the first classifier in the dth channel, , , represents the number of channels of the first classifier, represents an activation function for eliminating negative activations, is the image block feature of the dth channel in the multi-scale image block feature, and N represents the total number of image block features in the multi-scale image block feature , i.e., one-to-one correspondence with the image block, and the response map is scaled to the interval [0, 1] through maximum value normalization processing.

[0130] The auxiliary class activation map is divided into background and foreground regions by a double-threshold division method to construct auxiliary pseudo labels , and the auxiliary class activation map Middle Pseudo labels for pixels The calculation is as follows:

[0131]

[0132] in, Represents the auxiliary class activation map The value of the k-th pixel in , specifically, when Less than or equal to the preset lower threshold (such as 0.2), the corresponding pixel is considered to be the background area and marked as 0; when Greater than or equal to the preset upper threshold (such as 0.7), the corresponding pixel is considered to be the foreground area, marked as 1, k = 1~K; otherwise, the corresponding pixel is considered to be the uncertain area, marked as 255, and compared with the auxiliary pseudo label The values ​​of the preset number of neighboring pixels (such as 8) of the corresponding pixel points in the image are considered to be in the background area when the proportion of those equal to 1 is less than or equal to the preset ratio (such as 20%). When the proportion of those equal to 1 is greater than the preset ratio, the corresponding pixel point is considered to be in the foreground area. Multi-scale image patch features corresponding to the middle foreground area The image block features in are used as the foreground set , auxiliary pseudo-label Multi-scale image patch features corresponding to the background area The image patch features in are used as background sets , complete the foreground and background selection. Figure 3 As shown in the figure, it is a schematic diagram of the structure of the segmentation decoder, where Conv represents the convolution layer. express Activation function.

[0133] The initial activation map is divided into two parts using a single threshold method. Constructing initial pseudo labels , as follows:

[0134] Determine the initial activation map Whether the value of each pixel in the If yes, the corresponding pixel is considered to be the background area and marked as 0. If no, the corresponding pixel is considered to be the foreground area and marked as 1.

[0135] Among them, based on the generated initial activation map , manually set the preset division threshold , for the initial activation map the value (response intensity value for defect category) of each spatial position (i.e. pixel point) in the initial activation map

[0136]

[0137] , the corresponding pixel point is considered as a background region, and the pseudo label of the kth pixel point in the initial activation map is 0 (i.e. 0.25), the corresponding pixel point is considered as a foreground region, and the pseudo label of the kth pixel point in the initial activation map is 1 (i.e. 1.0). , where k = 1 ~ K, until the value of each pixel point in the initial activation map is traversed, and the initial pseudo label is used as a supervision signal for segmentation prediction. The initial activation map , the initial pseudo label , the auxiliary activation map and the auxiliary pseudo label have the same pixel points.

[0138] 2) Text-guided module:

[0139] For the foreground class and the background class in the industrial defect image dataset, corresponding foreground text and background text are constructed; all foreground text and background text are input into the text encoder of the CLIP model to correspondingly generate a foreground text feature set and a background text feature set ; the foreground text feature set and the background text feature set are spliced and input into the context decoder together with the multi-scale image block feature to obtain an enhanced foreground text feature set and an enhanced background text feature set ; a graph-text relevance score map is obtained according to the multi-scale image block feature , the enhanced foreground text feature set and the enhanced background text feature set ; the preliminary segmentation prediction result is weighted with the graph-text relevance score map to obtain the final segmentation prediction result ​​​​​Furthermore, foreground text and background text are constructed based on the category labels of the industrial defect images. The corresponding natural language templates are: a clean origami "class" and a clean origami "background", where "class" can be replaced by the defect category (corresponding category label) divided in step S1. The constructed foreground text and background text are input into the text encoder of the CLIP model ( Figure 2 CLIP text encoder) is used to encode the foreground text feature set and background text feature set . Set the foreground text feature set and background text feature set and scaled original image patch features The text semantic embedding representation of the aligned and enhanced foreground and background categories is obtained through a context decoder consisting of L sequentially connected Transformer modules, that is, the enhanced foreground text feature set and enhanced background text feature set , enhanced foreground text feature set Each feature in is represented as b3, and the specific content of each b3 can be different, determined by the corresponding foreground text, to enhance the background text feature set Each feature in is represented as f3, and the specific content of each f3 can be different and is determined by the corresponding background text. Figure 2 The enhanced text features (b3 or f3) in the context decoder are used to align multimodal semantic input with text features. The text prompt template is generated using a natural language template to enhance its ability to express semantic context, which is beneficial to the alignment of features between images and text. The context decoder is composed of The enhanced foreground text feature set is obtained by implementing cross-modal alignment and feature fusion in the context decoder. and enhanced background text feature set According to the multi-scale image block features , Enhanced foreground text feature set and enhanced background text feature set Get the image-text relevance score graph Specifically: enhance the foreground text feature set and enhanced background text feature set Splicing to form text embedding T; performing Einstein sum operation on the normalized multi-scale image block features and the normalized text embedding to obtain the image-text correlation score graph Among them, Einstein summation is used to calculate the cosine similarity between each image block feature in the normalized multi-scale image block feature and each category text in the normalized text embedding to obtain the image-text correlation score map The image-text relevance score graph It reflects the matching strength between each image block and the semantics of each category, which helps to locate the region guided by semantics. Image and text correlation score chart Perform pixel-by-pixel weighted fusion operations to enhance the response strength of semantic matching areas and suppress irrelevant areas, thereby obtaining the final segmentation prediction results. , where the initial segmentation prediction result Image and text correlation score chart Middle The weighted formula corresponding to each defect category is as follows:

[0140]

[0141] in, For the The final segmentation prediction results of defect categories, For the The initial segmentation prediction results of defect categories, For the Image-text correlation score graph for defect categories, , Represents the total number of defect categories in all category labels. Each defect category corresponds to a category label. is the balance coefficient, which is used to control the weight of semantic enhancement.

[0142] By combining the multi-scale image block features and the enhanced foreground text feature set after splicing and enhanced background text feature set Perform similarity matching and construct a graph of image-text relevance scores , used to enhance the semantic consistency of image segmentation prediction; at the same time, based on the preliminary prediction results and the image-text correlation score map The weighted fusion between them can obtain a more refined segmentation output.

[0143] 3) Multi-level comparison optimization module:

[0144] The multi-level contrast optimization module is used to update the weights during the training process, including the global inter-block feature contrast module, the local-global feature contrast module, the image-text semantic contrast module and the total loss calculation module. Among them, in order to improve the segmentation decoder's ability to distinguish defect areas, three parallel execution contrast optimization strategies are designed: (1) The global inter-block feature contrast module, which constructs the contrast loss of similarity and heterogeneity of image block features based on the foreground set and the background set, strengthens the consistency of similar features and the discrimination of heterogeneous features, enhances the overall semantic discrimination of the image, and improves the model's ability to model local details; (2) The local-global feature contrast module, which randomly crops Q local image regions in the auxiliary pseudo-labels and compares them with the multi-scale image block features. Class markers in The first characteristic formed Compare and guide the segmentation decoder to learn consistent representation at different scales; (3) Image-text semantic comparison module, the first feature Respectively with the foreground text feature set and background text feature set Calculate the cosine similarity and use InfoNCE loss to maximize the similarity of foreground image-text pairs and minimize the similarity of background image-text pairs, thereby achieving cross-modal semantic alignment optimization and enhancing the consistency modeling of model image-text semantics. Furthermore, the global inter-block feature comparison module is used to calculate the foreground set based on cosine similarity. The inter-block similarity of the image block features within, and the foreground set and background collection The heterogeneous similarity of the image block features between the blocks is calculated, and the inter-block similarity is maximized while the heterogeneous similarity is minimized to generate the inter-block contrast loss function. , the formula is as follows:

[0145]

[0146] in, Represents the foreground set The number of elements in , Represents a background collection The number of elements in , Indicates cosine similarity calculation, Represents the foreground set The i-th image block feature in , Represents the foreground set The j-th image block feature in Represents a background collection The qth image block feature in the foreground set Each feature in is represented as b1, and the specific content of each b1 can be different and is determined by the corresponding image block. The background set Each feature in is represented as f1. Similarly, the specific content of each f1 can be different and is determined by the corresponding image block. Figure 2 The image patch features (b1 or f1) based on the auxiliary pseudo-label in .

[0147] The global inter-block feature comparison module uses cosine similarity as the feature similarity metric and uses the inter-block comparison loss function Maximizing the similarity between similar image block features and minimizing the similarity between foreground and background image block features enhances the segmentation decoder's ability to discern the semantic distribution of the entire image. By shortening the foreground-foreground distance and increasing the foreground-background distance, the overall visual feature distribution is optimized, guiding the segmentation decoder to more accurately distinguish different semantic regions, improving the structural separation of foreground and background and the clarity of segmentation boundaries.

[0148] The local-global feature comparison module is used to perform the following operations:

[0149] Multi-scale image patch features Class markers in Projected into the same space as the text embedding T to obtain the first feature , the text embedding T is the enhanced foreground text feature set after splicing and enhanced background text feature set ; On each industrial defect image in the industrial defect image dataset, Q local image blocks are formed by random cropping with size H×W multiple times, and the auxiliary pseudo label Cut out the label blocks corresponding to Q local image blocks, where H is the height of the local image block, W is the width of the local image block, and Q is a positive integer; input the Q local image blocks into the ViT encoder to obtain the first extracted features, and use the features of the foreground area in the label block corresponding to the first extracted features as the first visual features , the features of the background area in the label block corresponding to the first extracted feature are used as the second visual features , wherein the features of the foreground area and the background area in the label block corresponding to the first extracted feature are the image block features based on the auxiliary pseudo label; according to the auxiliary pseudo label , first visual features , second visual features , the first feature Generate InfoNCE contrastive loss , the formula is as follows:

[0150]

[0151] in, For auxiliary pseudo labels The total number of pixels in the foreground area, For auxiliary pseudo labels The total number of pixels in the background area, , , is the temperature coefficient, Represents a natural exponential function, which is used to amplify the similarity difference between positive and negative samples, thereby enhancing the discrimination, the first visual feature That is, as a positive sample, the second visual feature That is, it is used as a negative sample. The first visual feature Each feature in is represented as b2. The specific content of each b2 can be different and is determined by the corresponding image block. The second visual feature Each feature in is represented as f2. Similarly, the specific content of each f2 can be different and is determined by the corresponding image block. Figure 2 The image patch features (b2 or f2) based on the auxiliary pseudo-label in .

[0152] The local-global feature comparison module introduces InfoNCE contrast loss The temperature coefficient is used to control the distance penalty between positive and negative samples, thereby enhancing the segmentation decoder's ability to focus on local structural features and improving the adaptability of global semantic aggregation features to local changes. This reduces interference in irrelevant regions of local features, enabling the segmentation decoder to maintain stable semantic consistency modeling capabilities under conditions of high local uncertainty, significantly improving segmentation robustness under complex defect morphologies.

[0153] The image-text semantic comparison module is used to maximize the first feature and enhanced foreground text feature set The cosine similarity of , while minimizing the first feature and enhanced background text feature set The cosine similarity of the image and text comparison loss function is generated , the formula is as follows:

[0154]

[0155] in, Indicates the The class labels of the defect classes, , Indicates the total number of defect categories in all category labels.

[0156] The image-text semantic comparison module uses the first feature As image semantic feature representation, and respectively with the enhanced foreground text feature set and enhanced background text feature set Perform cosine similarity calculation to generate image-text comparison loss function maximizing the similarity between image features (first features ) and corresponding foreground semantic texts (enhanced foreground text feature sets ) while minimizing the similarity between image features and background semantic texts (enhanced background text feature sets ) in the training process. This helps to improve cross-modal semantic consistency between images and texts, effectively suppresses the problem of false activation caused by background class co-occurrence, and strengthens the attention focusing effect of semantic matching, enabling the segmentation decoder to achieve stronger target positioning ability and noise robustness under the guidance of image and language dual modalities.

[0157] wherein the initial pseudo-labels and the global inter-block contrast module, the local-global feature contrast module, and the image-text semantic contrast module jointly supervise the training of the segmentation decoder. The total loss function of the training process is obtained by weighting the first classification loss , the second classification loss , the inter-block contrast loss function , the InfoNCE contrast loss , the image-text contrast loss function , and the segmentation loss , i.e., the total loss function is constructed by a total loss calculation module, and the formula is as follows:

[0158]

[0159] wherein, is the weight parameter of the first classification loss , is the weight parameter of the second classification loss , is the weight parameter of the inter-block contrast loss function , is the weight parameter of the InfoNCE contrast loss , is the weight parameter of the image-text contrast loss function , is the weight parameter of the segmentation loss . As in the present embodiment , , , , , are respectively set to 1, 1, 0.1, 0.5, 0.2, and 0.1, and can be adjusted according to actual needs. The first classification loss is used to supervise the image-level defect class prediction ability of the first classifier, and the multi-scale image block features Input the first classifier to obtain the predicted probability of each type of defect, and use the multi-label cross entropy loss function to compare it with the image-level label (category label) corresponding to the industrial defect image to calculate the defect category discrimination error. It is used to supervise the output stability of the auxiliary classifier. It also uses the multi-label cross entropy loss function and uses the image-level label as the supervision signal to provide a semantic basis for the accurate generation of the subsequent auxiliary activation map. The pixel-level prediction results of the segmentation decoder are used to supervise the segmentation decoder. The output feature map of the segmentation decoder is compared with the initial pseudo label pixel by pixel, and the prediction error is calculated through the pixel-level cross entropy loss function, thereby guiding the image segmentation model to learn the boundary distribution of the foreground and background areas. , second classification loss , segmentation loss , the formula is as follows:

[0160]

[0161]

[0162]

[0163] in, The first classifier predicts The probability of a defect class, The auxiliary classifier predicts The probability of defect categories, is the initial pseudo label Middle The value of the pixel, is the initial pseudo label The total number of pixels in Predict the first The probability that a pixel belongs to the foreground area.

[0164] In the present embodiment, the model training iteration is set to a maximum of 500 rounds, and the early termination mechanism is controlled according to the set precision improvement judgment condition: that is, when the model precision no longer appears to improve within the set 100 rounds, the training will automatically terminate; otherwise, continue to the set upper limit number of rounds to end, and obtain the trained segmentation model. The image segmentation model extracts multi-scale image block features of the industrial defect image dataset through the ViT encoder; extracts foreground text and background text according to the natural language template, inputs the text encoder of the CLIP model to generate foreground semantic embedding and background semantic embedding, aligns with the image features in the shared space, and enriches the cross-modal semantic modeling capability of the model; further combines the first classifier weight to generate an initial activation map; and forms an output pseudo label based on the single threshold division method, and generates an additional auxiliary pseudo label based on the double threshold division method to divide the foreground set and the background set. Then, a multi-level feature optimization mechanism based on global block contrast, local-global semantic contrast and image-text contrast is constructed to strengthen the model's focus on the foreground area and the background suppression capability; finally, based on the image segmentation model supervised by the fused and optimized image features and the pseudo labels, the pixel-level segmentation result of the target defect area is realized.

[0165] S3, using the trained image segmentation model to detect the industrial defect image to be segmented, and obtaining a final segmentation prediction result. Wherein, the image segmentation model after training obtains a corresponding weight file, which can directly predict the pixel level of the industrial defect image to be segmented, and output an industrial defect segmentation result with high semantic consistency and high boundary definition as the final segmentation prediction result, realizing high-precision defect segmentation under weak supervision. The industrial defect image in actual detection can be a surface defect detection image in an industrial manufacturing scene, such as metal part scratch, welding defect, composite material surface flaw, etc.

[0166] The specific experiment of the present application uses the public industrial surface defect segmentation dataset NEU-Seg for verification. The industrial surface defect segmentation dataset NEU-Seg has a total of 6 defect categories, including Rolled-in Scale, Patches, Crazing, Pitted Surface, Inclusion, and Scratches, with a total of 4470 images. In the present embodiment, 80% of the images are randomly selected as the training set, and the remaining 20% of the images are selected as the test set. The final segmentation prediction result is as shown in Figure 4 (A) is an industrial defect image, (B) is the corresponding artificial annotation result of the industrial defect image, and (C) is the industrial defect segmentation result of the present application. Compared with the artificial annotation result, the present application has extremely high segmentation accuracy, effectively saving manpower.

[0167] The method aims at the problems of high labeling cost, small sample quantity and complex defect morphology in industrial defect detection tasks, especially the limitations of existing weakly supervised methods in insufficient text-image alignment granularity and being easily disturbed by background, and realizes weakly supervised defect segmentation by fusing a text guiding mechanism and a multi-level feature comparison strategy. Specifically, by combining a ViT encoder and a CLIP model, corresponding activation maps and pseudo labels are obtained, and the model's ability to distinguish foreground-background semantics is enhanced through text semantic embedding. On this basis, three optimization mechanisms, namely global block comparison, local-global comparison and image-text semantic comparison, are designed to constrain and optimize the fine granularity of image features and activation maps. Finally, the optimized features and generated pseudo labels are used to guide the training of the image segmentation model, enabling precise segmentation of industrial defect regions under weak supervision, with strong semantic modeling and cross-modal alignment capabilities, showing good robustness, generalization and industrial deployment adaptability in real-world scenarios lacking pixel-level annotations.

[0168] Embodiment 2

[0169] A weakly supervised image defect segmentation system based on text guidance, based on embodiment 1, includes a data acquisition module, a model construction module and a model prediction module, wherein:

[0170] The data acquisition module is configured to acquire an industrial defect image dataset composed of a plurality of industrial defect images and corresponding category labels.

[0171] The model construction module is configured to establish an image segmentation model and train the image segmentation model using the industrial defect image dataset. The image segmentation model includes a preliminary prediction module and a text guidance module.

[0172] The preliminary prediction module performs the following operations:

[0173] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, including N image block features and class labels.

[0174] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result. The segmentation decoder includes a plurality of convolution modules connected in sequence.

[0175] The text guidance module performs the following operations:

[0176] For the foreground and background categories in the industrial defect image dataset, corresponding foreground and background texts are constructed.

[0177] All foreground and background texts are input into the text encoder of the CLIP model to generate a set of foreground text features and a set of background text features.

[0178] The foreground text feature set and the background text feature set are spliced and input into the context decoder together with the multi-scale image block feature to obtain an enhanced foreground text feature set and an enhanced background text feature set;

[0179] A graph-text correlation score map is obtained according to the multi-scale image block feature, the enhanced foreground text feature set and the enhanced background text feature set;

[0180] The preliminary segmentation prediction result is weighted with the graph-text correlation score map to obtain a final segmentation prediction result.

[0181] The model prediction module is configured to use the trained image segmentation model to detect the industrial defect image to be segmented to obtain the final segmentation prediction result.

[0182] The data acquisition module can acquire industrial defect images with only image-level labels, and organize them according to defect categories, with images of the same category stored in the same folder, facilitating task construction and label management, and providing standardized input for subsequent feature extraction.

[0183] The preliminary prediction module inputs the industrial defect image into a ViT encoder to extract a multi-scale image block feature; and uses a natural language template set artificially to construct foreground text and background text for the foreground category and the background category, and inputs the text into a text encoder of a CLIP model to generate corresponding semantic embedding representations (corresponding to a foreground text feature set and a background text feature set, respectively, projected into a space consistent with the image feature to establish a graph-text alignment relationship; then a graph-text correlation score map is obtained according to the multi-scale image block feature, the enhanced foreground text feature set and the enhanced background text feature set; the preliminary segmentation prediction result is weighted with the graph-text correlation score map to obtain a final segmentation prediction result.

[0184] The model prediction module uses the trained image segmentation model to perform segmentation prediction on the industrial defect image to be segmented, and outputs a pixel-level segmentation result map of the defect region.

[0185] The system aims at the problems of high labeling cost, small sample quantity and complex defect morphology in industrial defect detection tasks, especially the limitations of existing weakly supervised methods in insufficient text-image alignment granularity and being easily disturbed by background, and realizes weakly supervised defect segmentation by fusing text guidance mechanism and multi-level feature comparison strategy. Specifically, by combining the ViT encoder and the CLIP model, the corresponding activation map and pseudo label are obtained, and the discrimination ability of the foreground-background semantics is enhanced by the text semantic embedding enhancement model. On this basis, three optimization mechanisms of global block comparison, local-global comparison and image-text semantic comparison are designed to constrain and optimize the fine granularity of image features and activation maps. Finally, the optimized features and generated pseudo labels are used to guide the training of the image segmentation model, realizing accurate segmentation of industrial defect regions under weak supervision, having strong semantic modeling ability and cross-modal alignment ability, and showing good robustness, generalization and industrial deployment adaptability in real scenarios lacking pixel-level labels.

[0186] It should be noted that each module in the system can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to the above modules by the processor.

[0187] The memory and the processor are directly or indirectly electrically connected to realize the transmission or interaction of data. For example, these elements can be electrically connected through one or more communication buses or signal lines. The memory stores computer programs that can run on the processor. The processor realizes the method in the embodiment by running the computer programs stored in the memory.

[0188] The memory can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM) and the like. The memory is used to store programs. After receiving the execution instruction, the processor executes the program.

[0189] The processor can be an integrated circuit chip with data processing capability. The processor can be a general purpose processor, including a central processing unit (CPU), a network processor (NP), or the like. The processor can implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0190] Any combination of the above-described technical features of the embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present disclosure.

[0191] The above-described embodiments only express the specific and detailed embodiments described in the present application, but should not be construed as a limitation on the scope of the application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A text-guided weakly supervised image defect segmentation method, characterized by: The steps include: S1. Obtain an industrial defect image dataset consisting of several industrial defect images and corresponding category labels; S2. Establish an image segmentation model and train it using an industrial defect image dataset. The image segmentation model includes a preliminary prediction module and a text guidance module, wherein: The preliminary prediction module performs the following operations: Input the industrial defect image dataset into the ViT encoder for feature extraction to obtain multi-scale image block features, wherein the multi-scale image block features include N image block features and class labels; Inputting the multi-scale image block features into a segmentation decoder to obtain a preliminary segmentation prediction result, wherein the segmentation decoder includes a plurality of sequentially connected convolution modules; The text guidance module performs the following operations: Foreground text and background text are constructed for the foreground and background categories in the industrial defect image dataset respectively; Input all foreground text and background text into the text encoder of the CLIP model to generate a foreground text feature set and a background text feature set respectively; The foreground text feature set and the background text feature set are concatenated and input into the context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set; Obtain an image-text correlation score map based on multi-scale image block features, enhanced foreground text feature set, and enhanced background text feature set; Weight the preliminary segmentation prediction result and the image-text correlation score map to obtain the final segmentation prediction result; S3. Use the trained image segmentation model to detect the industrial defect image to be segmented and obtain the final segmentation prediction result.

2. The text-guided weakly supervised image defect segmentation method according to claim 1, wherein: The preliminary prediction module also performs the following operations: An auxiliary classifier is introduced at the output end of the middle layer of the ViT encoder. When the industrial defect image dataset is input into the ViT encoder, the output feature map of the auxiliary classifier is obtained at the same time. The output feature map of the auxiliary classifier is then used to form an auxiliary class activation map using the CAM method. A dual-threshold partitioning method is used to construct auxiliary pseudo-labels for the auxiliary class activation map, and the image block features in the multi-scale image block features corresponding to the foreground area in the auxiliary pseudo-labels are used as the foreground set, and the image block features in the multi-scale image block features corresponding to the background area are used as the background set; After the multi-scale image patch features are input into the first classifier, the initial activation map is formed using the CAM method; The single threshold partitioning method is used to construct the initial pseudo label for the initial activation map.

3. The text-guided weakly supervised image defect segmentation method according to claim 2, wherein: The ViT encoder performs the following operations: Divide the industrial defect image into N non-overlapping image blocks and flatten them into a first vector; Projecting the first vector through a linear mapping function to obtain a feature vector; The feature vector is concatenated with the introduced class feature vector and then input into the feature extractor to obtain the multi-scale image block features. ,in, For the Image block features, n=1~N, is the class label, the feature extractor is a Transformer encoder composed of 12 layers of sequentially connected Transformer modules, and the class feature vector is a one-dimensional vector of 1×C, where the elements corresponding to the defect categories to which the industrial defect image belongs are recorded as 1, and the rest are recorded as 0, and C is the total number of defect categories in all category labels.

4. The text-guided weakly supervised image defect segmentation method according to claim 3, wherein: The auxiliary classifier is a fully connected layer and is connected to the output end of the 10th layer Transformer module of the ViT encoder.

5. The text-guided weakly supervised image defect segmentation method according to claim 2, wherein: The first classifier is used to perform the following operations: Perform dot product operation on multi-scale image patch features and weights of the first classifier to obtain linear correlation map , where N is the multi-scale image block feature The number of image patch features in , is the total number of defect categories in all category labels; Mapping Linear Dependencies After the ReLU activation function, the response map is obtained, and the response map is normalized to the maximum value to obtain the initial activation map , Linear dependency map height, Linear dependency map width.

6. The text-guided weakly supervised image defect segmentation method according to claim 2, wherein: The dual threshold partitioning method is used to construct auxiliary pseudo labels for the auxiliary class activation map, specifically as follows: When the value of the k-th pixel in the auxiliary class activation map is less than or equal to the preset lower threshold, the corresponding pixel is considered to be the background area and marked as 0; when the value of the k-th pixel in the auxiliary class activation map is greater than or equal to the preset upper threshold, the corresponding pixel is considered to be the foreground area and marked as 1, k=1~K, K is the total number of pixels in the auxiliary class activation map; otherwise, the corresponding pixel is considered to be an uncertain area and marked as 255, and the values ​​of the preset number of neighboring pixels of the corresponding pixel in the auxiliary pseudo-label are compared. When the proportion equal to 1 is less than or equal to the preset ratio, the corresponding pixel is considered to be the background area, and when the proportion equal to 1 is greater than the preset ratio, the corresponding pixel is considered to be the foreground area; The single threshold partitioning method is used to construct the initial pseudo label for the initial activation map, as follows: Determine whether the value of each pixel in the initial activation map is within the preset segmentation threshold. If so, the corresponding pixel is considered to be the background area and marked as 0. If not, the corresponding pixel is considered to be the foreground area and marked as 1.

7. The text-guided weakly supervised image defect segmentation method according to claim 2, wherein: The image segmentation model also includes a multi-level contrast optimization module and a total loss calculation module, wherein: The multi-level comparison optimization module includes a global inter-block feature comparison module, a local-global feature comparison module and an image-text semantic comparison module, wherein The global inter-block feature comparison module is used to calculate the inter-block similarity of image block features in the foreground set and the heterogeneous similarity of image block features between the foreground set and the background set based on cosine similarity, and to generate an inter-block comparison loss function by maximizing the inter-block similarity while minimizing the heterogeneous similarity; The local-global feature comparison module performs the following operations: The class labels and text embeddings in the multi-scale image patch features are projected into the same space to obtain the first feature; In the industrial defect image, Q local image blocks are randomly cropped multiple times with a size of H×W, and label blocks corresponding to the Q local image blocks are cropped on the auxiliary pseudo-label, where H is the height of the local image block, W is the width of the local image block, and Q is a positive integer; Input Q local image blocks into the ViT encoder to obtain the first extracted features, and use the features of the foreground area in the label block corresponding to the first extracted features as the first visual features, and the features of the background area as the second visual features; Generate InfoNCE contrast loss according to the auxiliary pseudo-label, the first visual feature, the second visual feature, and the first feature, wherein the text is embedded into a concatenated enhanced foreground text feature set and an enhanced background text feature set; The image-text semantic comparison module is used to maximize the cosine similarity between the first feature and the enhanced foreground text feature set, and to minimize the cosine similarity between the first feature and the enhanced background text feature set to generate an image-text comparison loss function; The total loss calculation module is used to construct a total loss function, which is obtained by weighting the first classification loss, the second classification loss, the inter-block contrast loss function, the InfoNCE contrast loss, the image-text contrast loss function, and the segmentation loss, where: The first classification loss is used to calculate a multi-label cross entropy loss based on the output feature map of the first classifier and the corresponding category label to supervise the first classifier; The second classification loss is used to calculate the multi-label cross entropy loss based on the output feature map of the auxiliary classifier and the corresponding category label to supervise the auxiliary classifier; The segmentation loss is used to compare the output feature map of the segmentation decoder with the initial pseudo label pixel by pixel to calculate the cross entropy loss to supervise the segmentation decoder.

8. The text-guided weakly supervised image defect segmentation method according to claim 1, wherein: The segmentation decoder includes three sequentially connected convolution modules, wherein the first convolution module and the second convolution module each include sequentially connected convolution layers and ReLU activation functions, and the third convolution module includes a convolution layer; the context decoder includes Transformer modules connected in sequence, is a positive integer; the foreground text and background text are constructed using a preset natural language template.

9. The text-guided weakly supervised image defect segmentation method according to claim 1, wherein: The image-text correlation score map is obtained according to the multi-scale image block features, the enhanced foreground text feature set, and the enhanced background text feature set, as follows: Concatenate the enhanced foreground text feature set and the enhanced background text feature set to form a text embedding; The normalized multi-scale image block features and the normalized text embedding are subjected to Einstein summation to obtain the image-text correlation score map.

10. A text-guided weakly supervised image defect segmentation system, based on the text-guided weakly supervised image defect segmentation method according to any one of claims 1 to 9, characterized in that: It includes data acquisition module, model building module and model prediction module, among which: The data acquisition module is used to acquire an industrial defect image dataset consisting of a plurality of industrial defect images and corresponding category labels; The model building module is used to establish an image segmentation model and use the industrial defect image dataset for training. The image segmentation model includes a preliminary prediction module and a text guidance module, wherein: The preliminary prediction module performs the following operations: Input the industrial defect image dataset into the ViT encoder for feature extraction to obtain multi-scale image block features, wherein the multi-scale image block features include N image block features and class labels; Inputting the multi-scale image block features into a segmentation decoder to obtain a preliminary segmentation prediction result, wherein the segmentation decoder includes a plurality of sequentially connected convolution modules; The text guidance module performs the following operations: Foreground text and background text are constructed for the foreground and background categories in the industrial defect image dataset respectively; Input all foreground text and background text into the text encoder of the CLIP model to generate a foreground text feature set and a background text feature set respectively; The foreground text feature set and the background text feature set are concatenated and input into the context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set; Obtain an image-text correlation score map based on multi-scale image block features, enhanced foreground text feature set, and enhanced background text feature set; Weight the preliminary segmentation prediction result and the image-text correlation score map to obtain the final segmentation prediction result; The model prediction module is used to detect the industrial defect image to be segmented using the trained image segmentation model to obtain the final segmentation prediction result.

Images