Weakly supervised image defect segmentation method and system based on text guidance

Abstract

The application discloses a kind of weakly supervised image defect segmentation method and system based on text guide, comprising: obtaining industrial defect image dataset;Establish image segmentation model and train, wherein: preliminary prediction module is used to extract multi-scale image block feature input segmentation decoder to obtain preliminary segmentation prediction result;Text guide module is used to construct foreground text and background text and input the text encoder of CLIP model, generate foreground text feature set and background text feature set after splicing again with multi-scale image block feature input context decoder;Obtain the correlation score map of image and text and carry out weighting with preliminary segmentation prediction result, obtain final segmentation prediction result;Using the trained image segmentation model to the industrial defect image to be segmented is detected, obtains final segmentation prediction result.Solve the problems such as sample scarcity, high labeling cost and insufficient model generalization ability in industrial detection task, improve the precision and efficiency of image defect detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and industrial automatic detection, and particularly relates to a weakly supervised image defect segmentation method and system based on text guidance. BACKGROUND

[0002] Industrial defect detection is a key link in the modern intelligent manufacturing process, and its precision and robustness are directly related to product quality control and production line automation level. In recent years, image segmentation methods based on deep neural networks have been widely used in defect detection, especially convolutional neural networks (CNN) and other structures have shown strong ability in extracting semantic features and structural details. However, the data in the industrial scene usually has the following challenges: on the one hand, pixel-level labeling is costly, and it is extremely difficult to obtain large-scale training samples; on the other hand, the defect morphology is diverse, the background is complex, and the sample class distribution is extremely unbalanced, and traditional full-supervised methods are difficult to adapt to the needs of industrial deployment in low-labeled or unlabeled environments.

[0003] To overcome the above problems, weakly supervised semantic segmentation (WSSS) has gradually become a research hotspot. Such methods usually only use image-level labels or class information, and generate pseudo labels by combining class activation maps (CAM) to replace accurate labeling. On this basis, some research introduces a multi-modal model to enhance the semantic understanding ability through image-text alignment. However, the existing weakly supervised segmentation methods based on multi-modal models generally have the following shortcomings: 1) only rely on static global text and shallow image features for coarse-grained semantic association, lack of pixel-level alignment capability, and difficult to accurately locate the target area; 2) cannot effectively utilize the hierarchical structure features within the image, such as the contrastive information between local texture and global semantics, resulting in blurred segmentation boundaries; 3) do not explicitly model the contrast between foreground and background, leading to false activation when there are co-occurring interference backgrounds. SUMMARY

[0004] The purpose of the present application is to solve the above problems, and to propose a weakly supervised image defect segmentation method and system based on text guidance, to solve the problems of sample scarcity, high labeling cost and insufficient model generalization ability in industrial detection tasks, and to improve the precision and efficiency of image defect detection.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] The weakly supervised image defect segmentation method based on text guidance proposed by the present application comprises the following steps:

[0007] S1, obtaining an industrial defect image dataset composed of a plurality of industrial defect images and corresponding class labels;

[0008] S2, an image segmentation model is established and trained using an industrial defect image dataset, the image segmentation model comprising a preliminary prediction module and a text guiding module, wherein:

[0009] The preliminary prediction module performs the following operations:

[0010] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, the multi-scale image block features comprising N image block features and class labels;

[0011] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result, the segmentation decoder comprising a plurality of convolution modules connected in sequence;

[0012] The text guiding module performs the following operations:

[0013] For the foreground class and the background class in the industrial defect image dataset, foreground text and background text are respectively constructed;

[0014] All of the foreground text and the background text are input into a text encoder of a CLIP model to correspondingly generate a foreground text feature set and a background text feature set;

[0015] The foreground text feature set and the background text feature set are spliced and then input into a context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set;

[0016] A graph-text relevance score map is obtained according to the multi-scale image block features, the enhanced foreground text feature set, and the enhanced background text feature set;

[0017] The preliminary segmentation prediction result is weighted with the graph-text relevance score map to obtain a final segmentation prediction result;

[0018] S3, the trained image segmentation model is used to detect a to-be-segmented industrial defect image to obtain a final segmentation prediction result.

[0019] Preferably, the preliminary prediction module further performs the following operations:

[0020] An auxiliary classifier is introduced at the output end of the middle layer of the ViT encoder, and when the industrial defect image dataset is input into the ViT encoder, the output feature map of the auxiliary classifier is obtained at the same time, and the output feature map of the auxiliary classifier is used to form an auxiliary class activation map by a CAM method;

[0021] A dual threshold partitioning method is used to construct auxiliary pseudo-labels for auxiliary class activation maps. The image patch features in the multi-scale image patch features corresponding to the foreground region in the auxiliary pseudo-label are used as the foreground set, and the image patch features in the multi-scale image patch features corresponding to the background region are used as the background set.

[0022] After inputting multi-scale image patch features into the first classifier, the CAM method is used to form an initial activation map.

[0023] Initial pseudo-labels are constructed from the initial activation map using a single threshold partitioning method.

[0024] Preferably, the ViT encoder performs the following operations:

[0025] The industrial defect image is divided into N non-overlapping image blocks and flattened into a first vector;

[0026] The feature vector is obtained by projecting the first vector onto it using a linear mapping function;

[0027] The feature vector is concatenated with the introduced class feature vector and then input into the feature extractor to obtain multi-scale image patch features. ,in, For the first Image patch features, n=1~N, The class label is a Transformer encoder consisting of 12 layers of Transformer modules connected in sequence. The class feature vector is a 1×C one-dimensional vector, where the element corresponding to the defect category of the industrial defect image is marked as 1, and the rest are marked as 0. C is the total number of defect categories in all category labels.

[0028] Preferably, the auxiliary classifier is a fully connected layer and is connected to the output of the 10th layer Transformer module of the ViT encoder.

[0029] Preferably, the first classifier is used to perform the following operations:

[0030] A linear correlation mapping is obtained by performing a dot product operation on the multi-scale image patch features and the weights of the first classifier. Where N represents the multi-scale image patch features The number of image patch features, This represents the total number of defect categories across all category labels.

[0031] Mapping the linear correlation The response map is obtained after applying the ReLU activation function, and then the response map is normalized to obtain the initial activation map. , Linear correlation mapping diagram height, For linear correlation mapping Width of the.

[0032] Preferably, the auxiliary pseudo label is constructed by using a double threshold division method on the auxiliary class activation map, specifically as follows:

[0033] When the value of the kth pixel point in the auxiliary class activation map is less than or equal to the preset lower threshold, it is considered that the corresponding pixel point is a background region and is marked as 0; when the value of the kth pixel point in the auxiliary class activation map is greater than or equal to the preset upper threshold, it is considered that the corresponding pixel point is a foreground region and is marked as 1, k = 1 ~ K, K is the total number of pixel points of the auxiliary class activation map; otherwise, it is considered that the corresponding pixel point is an uncertain region and is marked as 255, and the values of the preset number of neighbor pixel points of the corresponding pixel point in the auxiliary pseudo label are compared, when the proportion of equal to 1 is less than or equal to the preset ratio, it is considered that the corresponding pixel point is a background region, when the proportion of equal to 1 is greater than the preset ratio, it is considered that the corresponding pixel point is a foreground region.

[0034] The initial pseudo label is constructed by using a single threshold division method on the initial activation map, specifically as follows:

[0035] It is judged whether the value of each pixel point in the initial activation map is within the preset division threshold, if yes, it is considered that the corresponding pixel point is a background region and is marked as 0, if not, it is considered that the corresponding pixel point is a foreground region and is marked as 1.

[0036] Preferably, the image segmentation model further comprises a multi-level contrast optimization module and a total loss calculation module, wherein:

[0037] The multi-level contrast optimization module comprises a global inter-block feature contrast module, a local-global feature contrast module and an image-text semantic contrast module, wherein

[0038] The global inter-block feature contrast module is used to calculate the inter-block similarity of the image block features within the foreground set and the heterogeneous similarity of the image block features between the foreground set and the background set based on the cosine similarity, and to generate an inter-block contrast loss function by maximizing the inter-block similarity while minimizing the heterogeneous similarity;

[0039] The local-global feature contrast module performs the following operations:

[0040] The class label in the multi-scale image block feature and the text embedding are projected into the same space to obtain a first feature;

[0041] In the industrial defect image, Q local image blocks are formed by random cropping multiple times with size HxW, and label blocks corresponding to the Q local image blocks are cropped on the auxiliary pseudo label, wherein H is the height of the local image block, W is the width of the local image block, and Q is a positive integer;

[0042] Inputting Q local image blocks into the ViT encoder obtains first extracted features, and taking the features of foreground regions in corresponding label blocks of the first extracted features as first visual features and the features of background regions as second visual features;

[0043] According to the auxiliary pseudo label, the first visual feature, the second visual feature, and the first feature, an InfoNCE contrast loss is generated, and the text embedding is the spliced enhanced foreground text feature set and the enhanced background text feature set;

[0044] The image-text semantic contrast module is used to maximize the cosine similarity between the first feature and the enhanced foreground text feature set, and minimize the cosine similarity between the first feature and the enhanced background text feature set to generate an image-text contrast loss function;

[0045] The total loss calculation module is used to construct a total loss function, which is obtained by weighting the first classification loss, the second classification loss, the block contrast loss function, the InfoNCE contrast loss, the image-text contrast loss function, and the segmentation loss, wherein:

[0046] The first classification loss is used to calculate a multi-label cross-entropy loss according to the output feature map of the first classifier and the corresponding category label to supervise the first classifier;

[0047] The second classification loss is used to calculate a multi-label cross-entropy loss according to the output feature map of the auxiliary classifier and the corresponding category label to supervise the auxiliary classifier;

[0048] The segmentation loss is used to calculate a cross-entropy loss by pixel-by-pixel comparison between the output feature map of the segmentation decoder and the initial pseudo label to supervise the segmentation decoder.

[0049] Preferably, the segmentation decoder includes three convolution modules connected in sequence, wherein the first convolution module and the second convolution module each include a convolution layer and a ReLU activation function connected in sequence, and the third convolution module includes a convolution layer; the context decoder includes a plurality of Transformer modules connected in sequence, which is a positive integer; the foreground text and the background text are constructed using a preset natural language template.

[0050] Preferably, the image-text relevance score map is obtained according to the multi-scale image block feature, the enhanced foreground text feature set, and the enhanced background text feature set, specifically as follows:

[0051] The enhanced foreground text feature set and the enhanced background text feature set are spliced to form a text embedding;

[0052] The normalized multi-scale image block feature and the normalized text embedding are subjected to Einstein summation operation to obtain an image-text relevance score map.

[0053] A text-guided weakly supervised image defect segmentation system based on any of the above text-guided weakly supervised image defect segmentation methods, comprising a data acquisition module, a model construction module and a model prediction module, wherein:

[0054] The data acquisition module is configured to acquire an industrial defect image dataset composed of a plurality of industrial defect images and corresponding category labels.

[0055] The model construction module is configured to establish an image segmentation model and train the image segmentation model using the industrial defect image dataset, wherein the image segmentation model comprises a preliminary prediction module and a text-guided module.

[0056] The preliminary prediction module performs the following operations:

[0057] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, wherein the multi-scale image block features comprise N image block features and a class label.

[0058] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result, wherein the segmentation decoder comprises a plurality of convolution modules connected in sequence.

[0059] The text-guided module performs the following operations:

[0060] For the foreground category and the background category in the industrial defect image dataset, foreground text and background text are respectively constructed.

[0061] All of the foreground text and the background text are input into a text encoder of a CLIP model to correspondingly generate a foreground text feature set and a background text feature set.

[0062] The foreground text feature set and the background text feature set are spliced and then input into a context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set.

[0063] A graph-text relevance score map is obtained according to the multi-scale image block features, the enhanced foreground text feature set and the enhanced background text feature set.

[0064] The preliminary segmentation prediction result is weighted with the graph-text relevance score map to obtain a final segmentation prediction result.

[0065] The model prediction module is configured to use the trained image segmentation model to detect a to-be-segmented industrial defect image to obtain a final segmentation prediction result.

[0066] Compared with the prior art, the present application has the following advantages:

[0067] The application combines text semantic embedding guidance mechanism and multi-level feature comparison strategy, takes ViT encoder and CLIP model as feature extraction network of image and text, realizes preliminary positioning of target region by constructing image-text correlation score map, and introduces triple contrast mechanism to realize fine perception of target defects and effective suppression of background noise. Specifically: first, the industrial defect image dataset is obtained, and the image-level label is used as the only supervision information to avoid high-cost pixel-level labeling to realize weakly supervised semantic segmentation and improve the landing in actual industrial scene; the industrial defect image is input into the ViT encoder to extract multi-scale image block features, and the output feature map of the introduced auxiliary classifier is obtained to form initial pseudo label and auxiliary pseudo label, and foreground text and background text are constructed to input the text encoder of the CLIP model to obtain semantic embedding and introduce the context decoder for semantic enhancement, realizing the deep fusion of semantic information and image features, and significantly improving the positioning ability of the defect region; and aiming at the foreground blur or background interference problem caused by the insufficient activation accuracy of the activation map, a multi-level contrast learning mechanism is designed, that is, the global block contrast, local-global contrast and image-text contrast are combined with the pixel-level segmentation loss to optimize the image feature representation, enhance the separability of foreground and background, effectively suppress the pseudo-activated region, improve the segmentation boundary clarity and overall robustness, output more fine and robust features and activation map, realize the deep fusion of text semantic prior and multi-level visual features, and fully utilize the contrast information provided by the background semantics to guide the model to perform structure perception and noise suppression in image defect recognition. The weights of the trained image segmentation model can be used for pixel-level segmentation prediction of the industrial defect image, and the end-to-end training capability is possessed; the application breaks through the dependence of traditional methods on pixel-level labels, and through the cooperative design of semantic enhancement, spatial alignment and feature discrimination optimization, the semantic alignment relationship between image and text is fully tapped, the focusing ability and robustness of the model on the defect region are improved, and high-performance, low-labeling-cost industrial defect image segmentation is realized. Under the condition of only image classification level (image-level label) labeling sample, pixel-level segmentation of industrial defects is realized, which has high-precision segmentation capability, significantly reduces the time cost of data labeling, realizes structural modular design and strong semantic perception, the overall architecture is clear, the coupling degree between modules is low, the maintenance of the whole model can be realized by adjusting each module separately, and it is suitable for weakly supervised segmentation tasks of multiple industrial defects, especially for industrial defect detection tasks under complex background interference. The application significantly reduces the manual labeling cost, improves the generalization ability and industrial deployment adaptability in actual application. The accuracy, stability and engineering applicability of weakly supervised industrial defect segmentation are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The flowchart of the weakly supervised image defect segmentation method based on text guidance of the application;

[0069] Figure 2 A structural schematic diagram of an image segmentation model of the present application;

[0070] Figure 3 A structural schematic diagram of a segmentation decoder of the present application;

[0071] Figure 4 A comparison schematic diagram of an industrial defect image and its segmentation results under different conditions, wherein, (A) is an industrial defect image, (B) is an artificial annotation result corresponding to the industrial defect image, and (C) is an industrial defect segmentation result corresponding to the industrial defect image of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0073] It should be noted that when a component is referred to as being "connected" with another component, it can be directly connected with the other component or there can be a component in between. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0074] The core of the method is to fuse the CLIP model and the ViT encoder. In the absence of pixel-level annotation, the positioning and segmentation accuracy of the defect area are improved by means of text prompt guidance and hierarchical feature optimization strategy.

[0075] Embodiment 1:

[0076] As shown in the figure, a weakly supervised image defect segmentation method based on text guidance includes the following steps: Figures 1-4

[0077] S1, an industrial defect image dataset composed of a plurality of industrial defect images and corresponding category labels is obtained.

[0078] In the process of constructing an industrial defect image dataset with classification information for weakly supervised segmentation, the industrial defect images are classified and organized according to the category labels. According to the defect category, the industrial defect images of the same defect category are stored in the same folder and have the corresponding category label (as an image-level label), so as to facilitate the automatic association of the defect category and the industrial defect image, which can also be adjusted according to the actual situation.​

[0079] S2, an image segmentation model is established and trained using an industrial defect image dataset, the image segmentation model comprising a preliminary prediction module and a text-guided module, wherein:

[0080] The preliminary prediction module performs the following operations:

[0081] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, the multi-scale image block features comprising N image block features and class labels;

[0082] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result, the segmentation decoder comprising a plurality of convolution modules connected in sequence;

[0083] The text-guided module performs the following operations:

[0084] For the foreground class and the background class in the industrial defect image dataset, foreground text and background text are respectively constructed;

[0085] All of the foreground text and the background text are input into a text encoder of a CLIP model to correspondingly generate a foreground text feature set and a background text feature set;

[0086] The foreground text feature set and the background text feature set are spliced and then input into a context decoder together with the multi-scale image block features to obtain an enhanced foreground text feature set and an enhanced background text feature set;

[0087] A graph-text correlation score map is obtained according to the multi-scale image block features, the enhanced foreground text feature set, and the enhanced background text feature set;

[0088] The preliminary segmentation prediction result is weighted with the graph-text correlation score map to obtain a final segmentation prediction result.

[0089] In an embodiment, the preliminary prediction module further performs the following operations:

[0090] An auxiliary classifier is introduced at the output end of the middle layer of the ViT encoder, and when the industrial defect image dataset is input into the ViT encoder, an output feature map of the auxiliary classifier is simultaneously obtained, and the output feature map of the auxiliary classifier is used to form an auxiliary class activation map by a CAM method;

[0091] An auxiliary pseudo-label is constructed for the auxiliary class activation map by using a double-threshold division method, and image block features in the multi-scale image block features corresponding to foreground regions in the auxiliary pseudo-label are taken as a foreground set, and image block features in the multi-scale image block features corresponding to background regions in the auxiliary pseudo-label are taken as a background set;

[0092] After inputting multi-scale image patch features into the first classifier, the CAM method is used to form an initial activation map.

[0093] Initial pseudo-labels are constructed from the initial activation map using a single threshold partitioning method.

[0094] In one embodiment, the ViT encoder performs the following operations:

[0095] The industrial defect image is divided into N non-overlapping image blocks and flattened into a first vector;

[0096] The feature vector is obtained by projecting the first vector onto it using a linear mapping function;

[0097] The feature vector is concatenated with the introduced class feature vector and then input into the feature extractor to obtain multi-scale image patch features. ,in, For the first Image patch features, n=1~N, The class label is a Transformer encoder consisting of 12 layers of Transformer modules connected in sequence. The class feature vector is a 1×C one-dimensional vector, where the element corresponding to the defect category of the industrial defect image is marked as 1, and the rest are marked as 0. C is the total number of defect categories in all category labels.

[0098] In one embodiment, the auxiliary classifier is a fully connected layer and is connected to the output of the 10th layer Transformer module of the ViT encoder.

[0099] In one embodiment, the first classifier is used to perform the following operations:

[0100] A linear correlation mapping is obtained by performing a dot product operation on the multi-scale image patch features and the weights of the first classifier. Where N represents the multi-scale image patch features The number of image patch features, This represents the total number of defect categories across all category labels.

[0101] Mapping the linear correlation The response map is obtained after applying the ReLU activation function, and then the response map is normalized to obtain the initial activation map. , Linear correlation mapping diagram height, Linear correlation mapping diagram The width.

[0102] In one embodiment, a dual-threshold partitioning method is used to construct auxiliary pseudo-labels for the auxiliary class activation graph, as detailed below:

[0103] When the value of the kth pixel point in the auxiliary class activation map is less than or equal to the preset lower threshold, the corresponding pixel point is considered as a background region and marked as 0; when the value of the kth pixel point in the auxiliary class activation map is greater than or equal to the preset upper threshold, the corresponding pixel point is considered as a foreground region and marked as 1, k = 1 ~ K, K is the total number of pixel points of the auxiliary class activation map; otherwise, the corresponding pixel point is considered as an uncertain region and marked as 255, and the values of the preset number of neighboring pixel points of the corresponding pixel point in the auxiliary pseudo label are compared, when the proportion of the values equal to 1 is less than or equal to the preset ratio, the corresponding pixel point is considered as a background region, and when the proportion of the values equal to 1 is greater than the preset ratio, the corresponding pixel point is considered as a foreground region;

[0104] The initial pseudo label is constructed for the initial activation map by using a single threshold division method, and the construction is specifically as follows:

[0105] It is judged whether the value of each pixel point in the initial activation map is within the preset division threshold, if yes, the corresponding pixel point is considered as a background region and marked as 0, and if no, the corresponding pixel point is considered as a foreground region and marked as 1.

[0106] In an embodiment, the image segmentation model further comprises a multi-level contrast optimization module and a total loss calculation module, wherein:

[0107] The multi-level contrast optimization module comprises a global inter-block feature contrast module, a local-global feature contrast module and an image-text semantic contrast module, wherein:

[0108] The global inter-block feature contrast module is used to calculate the inter-block similarity of the image block features in the foreground set and the heterogeneous similarity of the image block features between the foreground set and the background set based on the cosine similarity, and to generate an inter-block contrast loss function by maximizing the inter-block similarity while minimizing the heterogeneous similarity;

[0109] The local-global feature contrast module performs the following operations:

[0110] The class label in the multi-scale image block feature and the text embedding are projected into the same space to obtain a first feature;

[0111] In the industrial defect image, Q local image blocks are formed by random cropping multiple times with a size of HxW, and label blocks corresponding to the Q local image blocks are cropped on the auxiliary pseudo label, wherein H is the height of the local image block, W is the width of the local image block, and Q is a positive integer;

[0112] The Q local image blocks are input into the ViT encoder to obtain a first extraction feature, and the features of the foreground region in the corresponding label block of the first extraction feature are taken as a first visual feature, and the features of the background region are taken as a second visual feature;

[0113] The InfoNCE contrast loss is generated based on the auxiliary pseudo-label, first visual feature, second visual feature, and first feature. The text embedding is a concatenated set of enhanced foreground text features and a set of enhanced background text features.

[0114] The image-text semantic comparison module is used to maximize the cosine similarity between the first feature and the foreground text feature set, while minimizing the cosine similarity between the first feature and the background text feature set to generate the image-text comparison loss function.

[0115] The total loss calculation module is used to construct the total loss function, which is obtained by weighting the first classification loss, the second classification loss, the inter-block comparison loss function, the InfoNCE comparison loss, the image-text comparison loss function, and the segmentation loss, where:

[0116] The first classification loss is used to calculate the multi-label cross-entropy loss based on the output feature map of the first classifier and the corresponding category label to supervise the first classifier;

[0117] The second classification loss is used to calculate the multi-label cross-entropy loss based on the output feature map of the auxiliary classifier and the corresponding category label to supervise the auxiliary classifier;

[0118] The segmentation loss is used to calculate the cross-entropy loss by comparing the output feature map of the segmentation decoder with the initial pseudo-labels pixel by pixel to supervise the segmentation decoder.

[0119] In one embodiment, the segmentation decoder includes three sequentially connected convolutional modules, wherein the first and second convolutional modules each include sequentially connected convolutional layers and ReLU activation functions, and the third convolutional module includes a convolutional layer; the context decoder includes A series of Transformer modules connected in sequence. It is a positive integer; the foreground and background texts are constructed using preset natural language templates.

[0120] In one embodiment, an image-text relevance score map is obtained based on multi-scale image patch features, an enhanced foreground text feature set, and an enhanced background text feature set, as detailed below:

[0121] The enhanced foreground text feature set and the enhanced background text feature set are concatenated to form a text embedding;

[0122] The normalized multi-scale image patch features and the normalized text embedding are subjected to an Einstein summation operation to obtain an image-text relevance score map.

[0123] Specifically, such as Figure 2 As shown, the image segmentation model execution process is as follows:

[0124] 1) Preliminary prediction module:

[0125] An auxiliary classifier is introduced at the output of the intermediate layer of the ViT encoder, and an industrial defect image dataset is input into the ViT encoder for feature extraction to obtain multi-scale image patch features of the ViT encoder output. And the activation map of the auxiliary class output by the auxiliary classifier The activation graph of auxiliary classes is analyzed using a dual threshold partitioning method. Constructing auxiliary pseudo-tags And auxiliary pseudo-tags Multi-scale image patch features corresponding to the mid-foreground region Image patch features in the foreground set Auxiliary pseudo-tags Multi-scale image patch features corresponding to the mid-background region Image patch features in the background set Foreground set and background collection It can be used for subsequent multi-level feature comparison and optimization; it can be used to optimize multi-scale image patch features. The initial activation maps are obtained by inputting the first classifier and the segmentation decoder respectively. and preliminary segmentation prediction results The initial activation map was partitioned using a single threshold method. Construct initial pseudo-tags .

[0126] Specifically, multi-scale image patch features are extracted from industrial defect images in the industrial defect image dataset using the ViT (Vision Transformer) encoder. This is used for subsequent multi-level comparison and learning. Further, the ViT encoder performs the following operations: the input industrial defect image is first divided into N non-overlapping image blocks, and the image blocks are flattened into a first vector. The first vector is transformed through a linear mapping function. Projecting onto the feature space yields the feature vector. , It is a learnable linear layer, followed by an introduced class feature vector. After concatenation (e.g., using the Concat function to concatenate along the channel dimension), the images are input into a feature extractor (Transformer encoder) for context modeling and global feature extraction to obtain multi-scale image patch features. ,in, This refers to the feature extractor (Transformer encoder). For the first The image patch feature, i.e. the th image patch feature Features extracted from image patches by a feature extractor Class label, i.e., class feature vector Feature extracted by feature extractor, class feature vector is a one-dimensional vector of 1xC, where the element corresponding to the defect category to which the industrial defect image belongs is recorded as 1, and the rest is recorded as 0, and C is the total number of defect categories in all category labels. The preliminary prediction module can extract image block features with spatial and contextual information, and introduce class feature vectors to aggregate global semantic information and provide a basis for subsequent foreground-background contrast modeling. The output end of the 10th layer of the ViT encoder (Transformer Encoder Block) is provided with an auxiliary classifier to obtain classification features, and the auxiliary classifier is implemented through a fully connected layer and generates an auxiliary class activation map based on the CAM method (class activation map generation method, CAM) on the classification features output by the auxiliary classifier , the auxiliary class activation map is divided into background and foreground regions by a double-threshold division method, and each pixel point in the auxiliary class activation map is divided into background and foreground regions, and the classification features output by the auxiliary classifier are used as intermediate feature representations for extracting local semantic information. The ViT encoder is a component in the ViT model known to those skilled in the art, i.e., the ViT model after deleting the MLP classification module (MLP Head), which will not be described here.

[0127] Initial activation map satisfies the following formula:

[0128] ,

[0129] In the formula, is the mapping map of the cth defect category in the linear correlation mapping map, is the weight of the cth defect category of the first classifier in the dth channel, , , represents the number of channels of the first classifier, represents an activation function for eliminating negative activations, is the image block feature of the dth channel in the multi-scale image block feature, and N represents the total number of image block features in the multi-scale image block feature , i.e., one-to-one correspondence with the image block, and the response map is scaled to the interval [0, 1] through maximum value normalization processing.

[0130] The auxiliary class activation map is divided into background and foreground regions by a double-threshold division method to construct auxiliary pseudo labels , and the auxiliary class activation map The Middle A pseudo-label of one pixel The calculation is as follows:

[0131]

[0132] in, Representing the activation graph of auxiliary classes The value of the k-th pixel, specifically, when Less than or equal to the preset lower threshold When the value is 0.2, the corresponding pixel is considered to be a background area and marked as 0; when... Greater than or equal to the preset upper limit threshold If the value is 0.7, the corresponding pixel is considered to be in the foreground region and marked as 1, k = 1~K; otherwise, the corresponding pixel is considered to be in an uncertain region and marked as 255, and the auxiliary pseudo-label is compared. The values ​​of neighboring pixels corresponding to a given pixel (e.g., 8) are used. If the percentage of pixels equal to 1 is less than or equal to a preset ratio (e.g., 20%), the corresponding pixel is considered a background area; if the percentage of pixels equal to 1 is greater than the preset ratio, the corresponding pixel is considered a foreground area. Auxiliary pseudo-tags are then used. Multi-scale image patch features corresponding to the mid-foreground region Image patch features in the foreground set Auxiliary pseudo-tags Multi-scale image patch features corresponding to the mid-background region Image patch features in the background set Complete the foreground and background selection. For example... Figure 3 The diagram shown is a schematic of the segmentation decoder, where Conv represents a convolutional layer. express Activation function.

[0133] The initial activation map is divided using a single threshold partitioning method. Construct initial pseudo-tags The details are as follows:

[0134] Determine the initial activation graph Are the values ​​of each pixel within the preset threshold? If the pixel is in the background region, it is considered a background area and marked as 0; otherwise, it is considered a foreground region and marked as 1.

[0135] Among them, based on the generated initial activation graph Manually set preset threshold For the initial activation graph the value (response intensity value for defect category) of each spatial position (i.e. pixel point) in the initial activation map

[0136]

[0137] , the corresponding pixel point is considered as a background region, and the pseudo label of the kth pixel point in the initial activation map is 0 (i.e. 0.25), the corresponding pixel point is considered as a foreground region, and the pseudo label of the kth pixel point in the initial activation map is 1 (i.e. 1.0). , where k = 1 ~ K, until the value of each pixel point in the initial activation map is traversed, and the initial pseudo label is used as a supervision signal for segmentation prediction. The initial activation map , the initial pseudo label , the auxiliary activation map and the auxiliary pseudo label have the same pixel points.

[0138] 2) Text-guided module:

[0139] For the foreground class and the background class in the industrial defect image dataset, corresponding foreground text and background text are constructed; all foreground text and background text are input into the text encoder of the CLIP model to correspondingly generate a foreground text feature set and a background text feature set ; the foreground text feature set and the background text feature set are spliced and input into the context decoder together with the multi-scale image block feature to obtain an enhanced foreground text feature set and an enhanced background text feature set ; a graph-text relevance score map is obtained according to the multi-scale image block feature , the enhanced foreground text feature set and the enhanced background text feature set ; the preliminary segmentation prediction result is weighted with the graph-text relevance score map to obtain the final segmentation prediction result ​​​​​Further, foreground and background text are constructed based on the category label of the industrial defect image, with corresponding natural language templates: a clean origami “class” and a clean origami “background”, where class can be replaced with the defect category (corresponding category label) defined in step S1. The constructed foreground and background text are input into the text encoder of the CLIP model. Figure 2 The foreground text feature set is obtained by encoding with the CLIP text encoder. and background text feature set Set of foreground text features and background text feature set Compared with scale, original image patch features A context decoder consisting of L sequentially connected Transformer modules obtains aligned and enhanced text semantic embedding representations of the foreground and background categories, i.e., an enhanced foreground text feature set. and enhanced background text feature set Enhance the foreground text feature set Each feature in the text is represented as b3, and the specific content of each b3 can be different, determined by the corresponding foreground text, thus enhancing the background text feature set. Each feature in the text is represented as f3, and the specific content of each f3 can be different, determined by the corresponding background text. Figure 2 Enhanced text features (b3 or f3) are used. A context decoder aligns multimodal semantic input with text features. Text prompt templates are generated using natural language templates, enhancing their ability to express semantic context and facilitating feature alignment between images and text. The context decoder consists of... It consists of several Transformer modules, which achieve cross-modal alignment and feature fusion in the context decoder to obtain an enhanced foreground text feature set. and enhanced background text feature set Based on multi-scale image patch features Enhanced foreground text feature set and enhanced background text feature set Obtain the image-text relevance score graph Specifically, this involves enhancing the foreground text feature set. and enhanced background text feature set The text embedding T is formed by concatenating the normalized multi-scale image patch features and the normalized text embedding; then, an Einstein summation operation is performed on the normalized multi-scale image patch features and the normalized text embedding to obtain the image-text relevance score map. Specifically, the cosine similarity between each image patch feature in the normalized multi-scale image patch features and each category of text in the normalized text embedding is calculated using Einstein summation to obtain the image-text relevance score map. The image and text relevance score chart This reflects the matching strength between each image patch and its semantic category, aiding in semantically guided region localization. The preliminary segmentation prediction results... Relevance score of text and image A pixel-wise weighted fusion operation is performed to enhance the response intensity of semantically matching regions and suppress irrelevant regions, thereby obtaining the final segmentation prediction result. Among them, the preliminary segmentation prediction results Image-text relevance score chart The Middle The weighted formulas for each defect category are as follows:

[0140]

[0141] in, For the first The final segmentation prediction results for each defect category, For the first Preliminary segmentation prediction results for each defect category. For the first Image-text relevance score graph for each defect category , This represents the total number of defect categories among all category labels, with each defect category corresponding to one category label. This is a balancing coefficient used to control the weights of semantic enhancement.

[0142] By using multi-scale image patch features With the concatenated enhanced foreground text feature set and enhanced background text feature set Similarity matching was performed, and a text-image relevance score graph was constructed. This is used to enhance the semantic consistency of image segmentation prediction; simultaneously, it is based on the preliminary prediction results and the image-text relevance score. The weighted fusion between them yields a more refined segmentation output.

[0143] 3) Multi-level comparison and optimization module:

[0144] The multi-level contrast optimization module is used to update weights during training, including a global inter-block feature comparison module, a local-global feature comparison module, an image-text semantic comparison module, and a total loss calculation module. Among them, in order to improve the segmentation decoder's ability to distinguish defective regions, three parallel contrast optimization strategies are designed: (1) The global inter-block feature comparison module constructs a contrast loss based on the foreground set and the background set to distinguish between similar and dissimilar image block features, strengthens the consistency of similar features and the distinguishability of dissimilar features, enhances the overall semantic distinguishability of the image, and improves the model's ability to model local details; (2) The local-global feature comparison module randomly crops Q local image regions from the auxiliary pseudo-labels and compares them with multi-scale image block features. Class tags in The first characteristic of formation Comparison is performed to guide the segmentation decoder to learn consistent representations at different scales; (3) Image-text semantic comparison module, which compares the first feature Each with the foreground text feature set and background text feature set Cosine similarity is calculated, and InfoNCE loss is used to maximize the similarity of foreground image-text pairs and minimize the similarity of background image-text pairs, thereby achieving cross-modal semantic alignment optimization and enhancing the model's consistent modeling of image-text semantics. Furthermore, a global inter-block feature comparison module is used to calculate the foreground set based on cosine similarity. Inter-block similarity of image patch features and foreground set and background collection The method calculates the heterogeneous similarity of image patch features and generates an inter-patch contrast loss function that maximizes inter-patch similarity while minimizing heterogeneous similarity. The formula is as follows:

[0145]

[0146] in, Represents the foreground set The number of elements in Represents the background set The number of elements in This indicates the calculation of cosine similarity. Represents the foreground set The features of the i-th image patch, Represents the foreground set The features of the j-th image patch, Represents the background set Features of the q-th image patch. Foreground set. Each feature in the image is represented by b1, and the specific content of each b1 can be different, determined by the corresponding image patch, and the background set. Each feature in the image is represented as f1. Similarly, the specific content of each f1 can be different, determined by the corresponding image patch. Figure 2 Image patch features (b1 or f1) based on auxiliary pseudo-labels.

[0147] The global inter-block feature comparison module uses cosine similarity as the feature similarity metric, and employs an inter-block comparison loss function. By maximizing the similarity between similar image patch features and minimizing the similarity between image patch features in the foreground and background regions, the segmentation decoder's ability to discriminate the semantic distribution of the entire image is enhanced. By narrowing the distance between foreground regions and widening the distance between foreground and background regions, the overall visual feature distribution is optimized, guiding the segmentation decoder to more accurately distinguish different semantic regions and improving the structural separation ability of the foreground and background, as well as the clarity of segmentation boundaries.

[0148] The local-global feature comparison module is used to perform the following operations:

[0149] Multi-scale image patch features Class tags in The first feature is obtained by projecting the text embedding T into the same space. The text embedding T is the concatenated set of enhanced foreground text features. and enhanced background text feature set Q local image patches of size H×W were formed by multiple random cropping operations on each industrial defect image in the industrial defect image dataset, and auxiliary pseudo-labels were used to create these patches. Label blocks corresponding to Q local image patches are cropped out, where H is the height of the local image patch, W is the width of the local image patch, and Q is a positive integer; the Q local image patches are input into the ViT encoder to obtain the first extracted features, and the features of the foreground region in the corresponding label blocks of the first extracted features are used as the first visual features. The features of the background region in the corresponding label block of the first extracted features are used as the second visual features. In this context, the features of the foreground and background regions in the corresponding label blocks of the first extracted features are the image patch features based on the auxiliary pseudo-labels; according to the auxiliary pseudo-labels First visual features Second visual features First feature Generate InfoNCE contrast loss The formula is as follows:

[0150]

[0151] in, To assist pseudo-tags The total number of pixels in the foreground region. To assist pseudo-tags The total number of pixels in the background area. , , For temperature coefficient, The natural exponential function is used to amplify the similarity difference between positive and negative samples, thereby enhancing discrimination; it is a primary visual feature. That is, as a positive sample, the second visual feature That is, as a negative sample. First visual feature. Each feature in the image is represented as b2, and the specific content of each b2 may be different, determined by the corresponding image patch. (Second visual feature) Each feature in the image is represented as f2. Similarly, the specific content of each f2 can differ, determined by the corresponding image patch. Figure 2 Image patch features (b2 or f2) based on auxiliary pseudo-labels.

[0152] The local-global feature contrast module introduces InfoNCE contrast loss. By controlling the distance penalty between positive and negative samples using a temperature coefficient, the segmentation decoder's ability to focus on local structural features is enhanced, and the adaptability of global semantic aggregation features to local changes is improved. This reduces interference from irrelevant regions in local features, enabling the segmentation decoder to maintain stable semantic consistency modeling capabilities even under conditions of high local uncertainty, and significantly improving the segmentation robustness under complex defect morphologies.

[0153] The image-text semantic comparison module is used to maximize the first feature. With enhanced foreground text feature set The cosine similarity is calculated while minimizing the first feature. With enhanced background text feature set Cosine similarity generation image-text comparison loss function The formula is as follows:

[0154]

[0155] in, Indicates the first Category labels for each defect category, , This indicates the total number of defect categories across all category labels.

[0156] The image-text semantic comparison module utilizes the first feature As a representation of image semantic features, and respectively compared with the enhanced foreground text feature set With enhanced background text feature set Perform cosine similarity calculation to generate an image-text comparison loss function. maximizing the similarity between image features (first features ) and corresponding foreground semantic texts (enhanced foreground text feature sets ) while minimizing the similarity between image features and background semantic texts (enhanced background text feature sets ) in the training process. This helps to improve cross-modal semantic consistency between images and texts, effectively suppresses the problem of false activation caused by background class co-occurrence, and strengthens the attention focusing effect of semantic matching, enabling the segmentation decoder to achieve stronger target positioning ability and noise robustness under the guidance of image and language dual modalities.

[0157] wherein the initial pseudo-labels and the global inter-block contrast module, the local-global feature contrast module, and the image-text semantic contrast module jointly supervise the training of the segmentation decoder. The total loss function of the training process is obtained by weighting the first classification loss , the second classification loss , the inter-block contrast loss function , the InfoNCE contrast loss , the image-text contrast loss function , and the segmentation loss , i.e., the total loss function is constructed by a total loss calculation module, and the formula is as follows:

[0158]

[0159] wherein, is the weight parameter of the first classification loss , is the weight parameter of the second classification loss , is the weight parameter of the inter-block contrast loss function , is the weight parameter of the InfoNCE contrast loss , is the weight parameter of the image-text contrast loss function , is the weight parameter of the segmentation loss . As in the present embodiment , , , , , are respectively set to 1, 1, 0.1, 0.5, 0.2, and 0.1, and can be adjusted according to actual needs. The first classification loss is used to supervise the image-level defect class prediction ability of the first classifier, and the multi-scale image block features The first classifier is used to obtain the predicted probability of each type of defect. A multi-label cross-entropy loss function is then applied, and the predicted probability is compared with the image-level labels (category labels) corresponding to the industrial defect images to calculate the defect category discrimination error. The second classification loss... To supervise the output stability of the auxiliary classifier, a multi-label cross-entropy loss function is also used, with image-level labels as the supervision signal, providing a semantic basis for the accurate generation of subsequent auxiliary activation maps. Segmentation loss. The pixel-level prediction results of the segmentation decoder are used to supervise its operation. The output feature map of the segmentation decoder is compared pixel-by-pixel with the initial pseudo-labels, and the prediction error is calculated using the pixel-level cross-entropy loss function. This guides the image segmentation model to learn the boundary distributions of the foreground and background regions. First classification loss. Second category of loss Segmentation loss The formula is as follows:

[0160]

[0161]

[0162]

[0163] in, The first classifier predicts it as the first... The probability of each defect category, The auxiliary classifier predicts the first... The probability of each defect category For initial pseudo-tags The Middle The value of each pixel. For initial pseudo-tags Total number of pixels For the segment decoder to predict the first The probability that a pixel belongs to the foreground region.

[0164] In the present embodiment, the model training iteration is set to a maximum of 500 rounds, and the early termination mechanism is controlled according to the set accuracy improvement judgment condition: that is, when the model accuracy no longer improves within the set 100 rounds, the training will automatically terminate; otherwise, continue to the set upper limit number of rounds to end, and obtain the trained segmentation model. The image segmentation model extracts multi-scale image block features of the industrial defect image dataset through the ViT encoder; extracts foreground text and background text according to the natural language template, inputs the text encoder of the CLIP model to generate foreground semantic embedding and background semantic embedding, aligns with the image features in the shared space, and enriches the cross-modal semantic modeling capability of the model; further combines the first classifier weight to generate an initial activation map; and forms an output pseudo label based on the single threshold division method, and generates an additional auxiliary pseudo label based on the double threshold division method to divide the foreground set and the background set. Then, a multi-level feature optimization mechanism based on global block contrast, local-global semantic contrast and image-text contrast is constructed to strengthen the model's focus on the foreground area and the background suppression capability; finally, based on the image segmentation model supervised by the fused and optimized image features and the pseudo labels, the pixel-level segmentation result of the target defect area is realized.

[0165] S3, using the trained image segmentation model to detect the industrial defect image to be segmented, and obtaining a final segmentation prediction result. Wherein, the image segmentation model training is completed to obtain the corresponding weight file, which can directly predict the pixel level of the industrial defect image to be segmented, and output an industrial defect segmentation result with high semantic consistency and high boundary definition as the final segmentation prediction result, realizing high-precision defect segmentation under weak supervision. The industrial defect image in actual detection can be a surface defect detection image in an industrial manufacturing scene, such as metal part scratch, welding defect, composite material surface flaw, etc.

[0166] The specific experiment of the present application uses the public industrial surface defect segmentation dataset NEU-Seg for verification. The industrial surface defect segmentation dataset NEU-Seg has a total of 6 defect categories, including Rolled-in Scale, Patches, Crazing, Pitted Surface, Inclusion, and Scratches, with a total of 4470 images. In the present embodiment, 80% of the images are randomly selected as the training set, and the remaining 20% of the images are selected as the test set. The final segmentation prediction result is as shown in Figure 4 The present application has a very high segmentation accuracy compared with the manual annotation result, effectively saving manpower.

[0167] The method aims at the problems of high labeling cost, small sample quantity and complex defect morphology in industrial defect detection tasks, especially the limitations of existing weakly supervised methods in insufficient text-image alignment granularity and being easily disturbed by background, and realizes weakly supervised defect segmentation by fusing a text guiding mechanism and a multi-level feature comparison strategy. Specifically, by combining a ViT encoder and a CLIP model, corresponding activation maps and pseudo labels are obtained, and the model's ability to distinguish foreground-background semantics is enhanced through text semantic embedding. On this basis, three optimization mechanisms, namely global block comparison, local-global comparison and image-text semantic comparison, are designed to constrain and optimize the fine granularity of image features and activation maps. Finally, the optimized features and generated pseudo labels are used to guide the training of the image segmentation model, enabling precise segmentation of industrial defect regions under weak supervision, with strong semantic modeling and cross-modal alignment capabilities, showing good robustness, generalization and industrial deployment adaptability in real-world scenarios lacking pixel-level annotations.

[0168] Embodiment 2

[0169] A weakly supervised image defect segmentation system based on text guidance, based on embodiment 1, includes a data acquisition module, a model construction module and a model prediction module, wherein:

[0170] The data acquisition module is configured to acquire an industrial defect image dataset composed of a plurality of industrial defect images and corresponding category labels.

[0171] The model construction module is configured to establish an image segmentation model and train the image segmentation model using the industrial defect image dataset. The image segmentation model includes a preliminary prediction module and a text guidance module.

[0172] The preliminary prediction module performs the following operations:

[0173] The industrial defect image dataset is input into a ViT encoder for feature extraction to obtain multi-scale image block features, including N image block features and class labels.

[0174] The multi-scale image block features are input into a segmentation decoder to obtain a preliminary segmentation prediction result. The segmentation decoder includes a plurality of convolution modules connected in sequence.

[0175] The text guidance module performs the following operations:

[0176] For the foreground and background categories in the industrial defect image dataset, corresponding foreground and background texts are constructed.

[0177] All foreground and background texts are input into a text encoder of a CLIP model to generate a set of foreground text features and a set of background text features.

[0178] The foreground text feature set and the background text feature set are spliced and input into the context decoder together with the multi-scale image block feature to obtain an enhanced foreground text feature set and an enhanced background text feature set;

[0179] A graph-text correlation score map is obtained according to the multi-scale image block feature, the enhanced foreground text feature set and the enhanced background text feature set;

[0180] The preliminary segmentation prediction result is weighted with the graph-text correlation score map to obtain a final segmentation prediction result.

[0181] The model prediction module is configured to use the trained image segmentation model to detect the industrial defect image to be segmented to obtain the final segmentation prediction result.

[0182] The data acquisition module can acquire industrial defect images with only image-level labels, and organize them according to defect categories, with images of the same category stored in the same folder, facilitating task construction and label management, and providing standardized input for subsequent feature extraction.

[0183] The preliminary prediction module inputs the industrial defect image into a ViT encoder to extract a multi-scale image block feature; and uses a natural language template set artificially to construct foreground text and background text for the foreground category and the background category, and inputs the text into a text encoder of a CLIP model to generate corresponding semantic embedding representations (corresponding to a foreground text feature set and a background text feature set, respectively, projected into a space consistent with the image feature to establish a graph-text alignment relationship; then a graph-text correlation score map is obtained according to the multi-scale image block feature, the enhanced foreground text feature set and the enhanced background text feature set; the preliminary segmentation prediction result is weighted with the graph-text correlation score map to obtain a final segmentation prediction result.

[0184] The model prediction module uses the trained image segmentation model to perform segmentation prediction on the industrial defect image to be segmented, and outputs a pixel-level segmentation result map of the defect region.

[0185] The system aims at the problems of high labeling cost, small sample quantity and complex defect morphology in industrial defect detection tasks, especially the limitations of existing weakly supervised methods in insufficient text-image alignment granularity and being easily disturbed by background, and realizes weakly supervised defect segmentation by fusing text guidance mechanism and multi-level feature comparison strategy. Specifically, by combining the ViT encoder and the CLIP model, the corresponding activation map and pseudo label are obtained, and the discrimination ability of the foreground-background semantics is enhanced by the text semantic embedding enhancement model. On this basis, three optimization mechanisms of global block comparison, local-global comparison and image-text semantic comparison are designed to constrain and optimize the fine granularity of image features and activation maps. Finally, the optimized features and generated pseudo labels are used to guide the training of the image segmentation model, realizing accurate segmentation of industrial defect regions under weak supervision, having strong semantic modeling ability and cross-modal alignment ability, and showing good robustness, generalization and industrial deployment adaptability in real scenarios lacking pixel-level labels.

[0186] It should be noted that each module in the system can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to the above modules by the processor.

[0187] The memory and the processor are directly or indirectly electrically connected to realize the transmission or interaction of data. For example, these elements can be electrically connected through one or more communication buses or signal lines. The memory stores computer programs that can run on the processor. The processor realizes the method in the embodiment by running the computer programs stored in the memory.

[0188] The memory can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM) and the like. The memory is used to store programs. After receiving the execution instruction, the processor executes the program.

[0189] The processor can be an integrated circuit chip with data processing capability. The processor can be a general purpose processor, including a central processing unit (CPU), a network processor (NP), or the like. The processor can implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0190] Any combination of the above-described technical features of the embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present disclosure.

[0191] The above-described embodiments only express the specific and detailed embodiments described in the present application, but should not be construed as a limitation on the scope of the application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the present application should be subject to the appended claims.

Claims

1. A text-guided weakly supervised image defect segmentation method, characterized in that: Includes the following steps: S1. Obtain an industrial defect image dataset consisting of several industrial defect images and corresponding category labels; S2. Establish an image segmentation model and train it using an industrial defect image dataset. The image segmentation model includes a preliminary prediction module and a text-guided module, wherein: The preliminary prediction module performs the following operations: The industrial defect image dataset is input into the ViT encoder for feature extraction to obtain multi-scale image patch features, which include N image patch features and class labels. Multi-scale image patch features are input into a segmentation decoder to obtain preliminary segmentation prediction results. The segmentation decoder includes multiple convolutional modules connected in sequence. The text guidance module performs the following operations: Foreground and background categories in the industrial defect image dataset are used to construct corresponding foreground and background text, respectively. All foreground and background text are input into the text encoder of the CLIP model to generate foreground text feature sets and background text feature sets respectively; The foreground text feature set and the background text feature set are concatenated and then input into the context decoder along with the multi-scale image patch features to obtain the enhanced foreground text feature set and the enhanced background text feature set. The image-text relevance score map is obtained based on multi-scale image patch features, enhanced foreground text feature set, and enhanced background text feature set; The preliminary segmentation prediction results are weighted and combined with the image-text relevance score to obtain the final segmentation prediction results. S3. Use the trained image segmentation model to detect the industrial defect image to be segmented and obtain the final segmentation prediction result.

2. The text-guided weakly supervised image defect segmentation method as described in claim 1, characterized in that: The preliminary prediction module also performs the following operations: An auxiliary classifier is introduced at the output of the intermediate layer of the ViT encoder. When the industrial defect image dataset is input into the ViT encoder, the output feature map of the auxiliary classifier is obtained at the same time, and the output feature map of the auxiliary classifier is used to form an auxiliary class activation map using the CAM method. A dual threshold partitioning method is used to construct auxiliary pseudo-labels for auxiliary class activation maps. The image patch features in the multi-scale image patch features corresponding to the foreground region in the auxiliary pseudo-label are used as the foreground set, and the image patch features in the multi-scale image patch features corresponding to the background region are used as the background set. After inputting multi-scale image patch features into the first classifier, the CAM method is used to form an initial activation map. Initial pseudo-labels are constructed from the initial activation map using a single threshold partitioning method.

3. The text-guided weakly supervised image defect segmentation method as described in claim 2, characterized in that: The ViT encoder performs the following operations: The industrial defect image is divided into N non-overlapping image blocks and flattened into a first vector; The feature vector is obtained by projecting the first vector onto it using a linear mapping function; The feature vector is concatenated with the introduced class feature vector and then input into the feature extractor to obtain multi-scale image patch features. ,in, For the first Image patch features, n=1~N, For class labeling, the feature extractor is a Transformer encoder consisting of 12 layers of Transformer modules connected in sequence. The class feature vector is a one-dimensional vector of 1×C, where the element corresponding to the defect category to which the industrial defect image belongs is marked as 1, and the rest are marked as 0, and C is the total number of defect categories in all category labels.

4. The text-guided weakly supervised image defect segmentation method as described in claim 3, characterized in that: The auxiliary classifier is a fully connected layer and is connected to the output of the 10th layer Transformer module of the ViT encoder.

5. The text-guided weakly supervised image defect segmentation method as described in claim 2, characterized in that: The first classifier is used to perform the following operations: A linear correlation mapping is obtained by performing a dot product operation on the multi-scale image patch features and the weights of the first classifier. Where N represents the multi-scale image patch features The number of image patch features, This represents the total number of defect categories across all category labels. Mapping the linear correlation The response map is obtained after applying the ReLU activation function, and then the response map is normalized to obtain the initial activation map. , Linear correlation mapping diagram height, Linear correlation mapping diagram The width.

6. The text-guided weakly supervised image defect segmentation method as described in claim 2, characterized in that: The auxiliary pseudo-labels are constructed by using a dual-threshold partitioning method to construct the auxiliary class activation graph, as detailed below: When the value of the k-th pixel in the auxiliary activation map is less than or equal to a preset lower threshold, the corresponding pixel is considered a background region and marked as 0; when the value of the k-th pixel in the auxiliary activation map is greater than or equal to a preset upper threshold, the corresponding pixel is considered a foreground region and marked as 1, k=1~K, where K is the total number of pixels in the auxiliary activation map; otherwise, the corresponding pixel is considered an uncertain region and marked as 255. The values ​​of the corresponding pixel's neighboring pixels in the auxiliary pseudo-label are compared. When the proportion of pixels equal to 1 is less than or equal to a preset ratio, the corresponding pixel is considered a background region; when the proportion of pixels equal to 1 is greater than the preset ratio, the corresponding pixel is considered a foreground region. The initial pseudo-labels are constructed from the initial activation map using a single threshold partitioning method, as detailed below: Determine whether the value of each pixel in the initial activation image is within the preset division threshold. If yes, the corresponding pixel is considered to be a background area and marked as 0. If no, the corresponding pixel is considered to be a foreground area and marked as 1.

7. The text-guided weakly supervised image defect segmentation method as described in claim 2, characterized in that: The image segmentation model further includes a multi-level contrast optimization module and a total loss calculation module, wherein: The multi-level comparison and optimization module includes a global inter-block feature comparison module, a local-global feature comparison module, and an image-text semantic comparison module, wherein... The global inter-block feature comparison module is used to calculate the inter-block similarity of image block features within the foreground set and the heterogeneous similarity of image block features between the foreground set and the background set based on cosine similarity, and to generate an inter-block comparison loss function that maximizes inter-block similarity while minimizing heterogeneous similarity. The local-global feature comparison module performs the following operations: The class labels and text embeddings in the multi-scale image patch features are projected into the same space to obtain the first feature; In an industrial defect image, Q local image blocks are formed by multiple random croppings of size H×W. Label blocks corresponding to the Q local image blocks are cropped from an auxiliary pseudo-label. Here, H is the height of the local image block, W is the width of the local image block, and Q is a positive integer. Q local image blocks are input into the ViT encoder to obtain the first extracted features, and the features of the foreground region in the corresponding label block of the first extracted features are used as the first visual features, and the features of the background region are used as the second visual features. The InfoNCE contrast loss is generated based on the auxiliary pseudo-label, the first visual feature, the second visual feature, and the first feature. The text embedding consists of a concatenated set of enhanced foreground text features and a set of enhanced background text features. The image-text semantic comparison module is used to maximize the cosine similarity between the first feature and the enhanced foreground text feature set, while minimizing the cosine similarity between the first feature and the enhanced background text feature set to generate an image-text comparison loss function. The total loss calculation module is used to construct the total loss function, which is obtained by weighting the first classification loss, the second classification loss, the inter-block comparison loss function, the InfoNCE comparison loss, the image-text comparison loss function, and the segmentation loss, wherein: The first classification loss is used to calculate the multi-label cross-entropy loss based on the output feature map of the first classifier and the corresponding category label to supervise the first classifier; The second classification loss is used to calculate the multi-label cross-entropy loss based on the output feature map of the auxiliary classifier and the corresponding class label to supervise the auxiliary classifier; The segmentation loss is used to calculate the cross-entropy loss by comparing the output feature map of the segmentation decoder with the initial pseudo-label pixel by pixel to supervise the segmentation decoder.

8. The text-guided weakly supervised image defect segmentation method as described in claim 1, characterized in that: The segmentation decoder includes three sequentially connected convolutional modules, wherein the first and second convolutional modules each include sequentially connected convolutional layers and ReLU activation functions, and the third convolutional module includes a convolutional layer; the context decoder includes A series of Transformer modules connected in sequence. The value is a positive integer; the foreground text and background text are constructed using preset natural language templates.

9. The text-guided weakly supervised image defect segmentation method as described in claim 1, characterized in that: The image-text relevance score map is obtained based on multi-scale image patch features, enhanced foreground text feature set, and enhanced background text feature set, as detailed below: The enhanced foreground text feature set and the enhanced background text feature set are concatenated to form a text embedding; The normalized multi-scale image patch features and the normalized text embedding are subjected to an Einstein summation operation to obtain an image-text relevance score map.

10. A text-guided weakly supervised image defect segmentation system, based on the text-guided weakly supervised image defect segmentation method according to any one of claims 1-9, characterized in that: It includes a data acquisition module, a model building module, and a model prediction module, among which: The data acquisition module is used to acquire an industrial defect image dataset consisting of several industrial defect images and corresponding category labels; The model building module is used to establish an image segmentation model and train it using an industrial defect image dataset. The image segmentation model includes a preliminary prediction module and a text-guided module, wherein: The preliminary prediction module performs the following operations: The industrial defect image dataset is input into the ViT encoder for feature extraction to obtain multi-scale image patch features, which include N image patch features and class labels. Multi-scale image patch features are input into a segmentation decoder to obtain preliminary segmentation prediction results. The segmentation decoder includes multiple convolutional modules connected in sequence. The text guidance module performs the following operations: Foreground and background categories in the industrial defect image dataset are used to construct corresponding foreground and background text, respectively. All foreground and background text are input into the text encoder of the CLIP model to generate foreground text feature sets and background text feature sets respectively; The foreground text feature set and the background text feature set are concatenated and then input into the context decoder along with the multi-scale image patch features to obtain the enhanced foreground text feature set and the enhanced background text feature set. The image-text relevance score map is obtained based on multi-scale image patch features, enhanced foreground text feature set, and enhanced background text feature set; The preliminary segmentation prediction results are weighted and combined with the image-text relevance score to obtain the final segmentation prediction results. The model prediction module is used to detect industrial defect images to be segmented using a trained image segmentation model, and obtain the final segmentation prediction result.

Images