Product defect unsupervised semantic segmentation method and device based on pre-training model

Abstract

The invention discloses a product defect unsupervised semantic segmentation method and device based on a pre-training model, and is used for solving the technical problem that the defect classification precision is not ideal due to an existing industrial defect unsupervised semantic segmentation method. The method comprises the following steps: acquiring a to-be-detected industrial product image, an industrial product reference image corresponding to the to-be-detected industrial product image and a plurality of to-be-matched defect categories; a preset segmentation network of a self-learning point generation prompter based on defect potential space features is adopted to generate a target defect segmentation mask according to a to-be-detected industrial product image and an industrial product reference image, and a target cutting image is generated based on the target defect segmentation mask, so that the target cutting image with visual prompt can be generated; and processing the target cutting image and a plurality of defect categories to be matched through a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module, and combining segmentation and classification, thereby improving the defect classification precision.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a method and device for unsupervised semantic segmentation of product defects based on a pre-training model. Background Art

[0002] In recent years, with the in-depth research on artificial intelligence deep learning technology, intelligent detection technology in the industrial field has become increasingly mature and has gradually been widely used.

[0003] Surface defect classification is an important task in the industrial field. At present, the most widely used deep learning technology in this field is image classification, which is to classify the input image into defects and good products. However, the classification task provides limited information, and in the industrial field, it is more desirable to obtain detailed information on product defects. Unlike the classification task, semantic segmentation can classify each pixel in the image, thereby segmenting the defective area from the image and providing more detailed defect information.

[0004] Most of the existing unsupervised semantic segmentation methods for industrial defects extract defect features in images through algorithm models, then calculate the type and location of the defect based on the features, and finally feed back the detection results to the actuator to perform the product sorting task. The above method is too dependent on manual labor. The manually selected and adjusted features may not be able to comprehensively and accurately cover all possible defect situations, resulting in less than ideal defect classification accuracy. Summary of the invention

[0005] The present invention provides a method and device for unsupervised semantic segmentation of product defects based on a pre-trained model, which are used to solve the technical problem that the existing unsupervised semantic segmentation method for industrial defects leads to unsatisfactory defect classification accuracy.

[0006] The first aspect of the present invention provides a method for unsupervised semantic segmentation of product defects based on a pre-trained model, comprising:

[0007] Acquire an industrial product image to be detected, an industrial product reference image corresponding to the industrial product image to be detected, and multiple defect categories to be matched;

[0008] Input the industrial product image to be detected, the industrial product reference image and the plurality of defect categories to be matched into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network based on a self-learning point generation prompter of defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module;

[0009] A preset segmentation network of a self-learning point generation prompter based on defect latent space features is used to generate a target defect segmentation mask according to the industrial product image to be inspected and the industrial product reference image, and a target cropped image is generated based on the target defect segmentation mask;

[0010] Using a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module to extract image and text features according to the target cropped image and a plurality of defect categories to be matched, and outputting image features corresponding to the target cropped image and text features corresponding to each defect category to be matched;

[0011] Calculate the matching degree between the image feature and each text feature respectively, and output the matching degree corresponding to each text feature;

[0012] The defect category associated with the text feature corresponding to the maximum matching degree is selected as the defect classification result corresponding to the industrial product image to be detected.

[0013] Optionally, the preset segmentation network of the self-learning point generation prompter based on defect latent space features includes a self-learning point generation prompter based on defect latent space features, a segmentation network image decoder, a segmentation network point prompt encoder and a mask proposal generator; the preset segmentation network using the self-learning point generation prompter based on defect latent space features generates a target defect segmentation mask according to the industrial product image to be detected and the industrial product reference image, and generates a target cropped image based on the target defect segmentation mask, including:

[0014] Using a self-learning point generation prompter based on defect latent space features to determine a target background point prompt coordinate set and a target foreground point point prompt coordinate set according to the industrial product image to be detected and the industrial product reference image;

[0015] A segmentation network point prompt encoder and a segmentation network image decoder are used to generate a target defect segmentation mask according to the industrial product image to be detected, the target background point prompt coordinate set and the target foreground point prompt coordinate set, and a target cropped image is generated based on the target defect segmentation mask.

[0016] Optionally, the self-learning point generation prompter based on defect latent space features is used to determine the target background point prompt coordinate set and the target foreground point point prompt coordinate set according to the industrial product image to be detected and the industrial product reference image, including:

[0017] Generate defect features, product inspection image reconstruction features, and product reference image reconstruction features according to the industrial product image to be inspected and the industrial product reference image;

[0018] The Hungarian algorithm is used to output a plurality of positive point prompt feature coordinates and a plurality of reverse point prompt feature coordinates according to the defect feature, the product inspection image reconstruction feature and the product reference image reconstruction feature;

[0019] Based on the mean clustering algorithm, the plurality of forward point prompt feature coordinates and the plurality of reverse point prompt feature coordinates are clustered to determine a target background point prompt coordinate set and a target foreground point prompt coordinate set.

[0020] Optionally, generating defect features, product inspection image reconstruction features, and product reference image reconstruction features according to the industrial product image to be inspected and the industrial product reference image includes:

[0021] Encode the to-be-detected industrial product image and the industrial product reference image respectively, and output features of the to-be-detected industrial product image and features of the industrial product reference image;

[0022] Reshaping the image features of the industrial product to be detected and the industrial product reference image features respectively to generate product detection image reshaping features and product reference image reshaping features;

[0023] Perform spatial multiplication on the reference mask corresponding to the industrial product reference image and the feature of the industrial product reference image to output defect features.

[0024] Optionally, the Hungarian algorithm is used to output a plurality of positive point prompt feature coordinates and a plurality of reverse point prompt feature coordinates according to the defect feature, the product inspection image reconstruction feature and the product reference image reconstruction feature, including:

[0025] Performing matrix multiplication on the defect feature and the product inspection image reconstruction feature to generate a positive correlation feature matrix;

[0026] Using the Hungarian algorithm, according to the forward correlation feature matrix, output a plurality of forward point prompt feature coordinates;

[0027] Selecting positive features corresponding to the positive point prompt feature coordinates from the product inspection image reconstruction features, and performing matrix multiplication using the positive features and the product reference image reconstruction features to output a reverse correlation feature matrix;

[0028] A Hungarian algorithm is used to generate a plurality of reverse point prompt feature coordinates according to the reverse correlation feature matrix.

[0029] Optionally, clustering the plurality of forward point prompt feature coordinates and the plurality of reverse point prompt feature coordinates based on a mean clustering algorithm to determine a target background point prompt coordinate set and a target foreground point prompt coordinate set includes:

[0030] Taking the forward point prompt feature coordinates associated with any reverse point prompt feature coordinates not in the reference mask corresponding to the industrial product reference image as background point prompt coordinates, and taking the forward point prompt feature coordinates associated with any reverse point prompt feature coordinates in the reference mask as foreground point prompt coordinates;

[0031] Using a plurality of the background point prompt coordinates, constructing a background point prompt coordinate set;

[0032] Using a plurality of the foreground point prompt coordinates to construct a foreground point prompt coordinate set;

[0033] Based on multiple preset clustering numbers, clustering the foreground point prompt coordinate set using a mean clustering algorithm, and outputting cluster center coordinates corresponding to each of the preset clustering numbers and multiple sample coordinate points associated with the cluster center coordinates corresponding to each of the preset clustering numbers;

[0034] Calculating the silhouette coefficient corresponding to each of the preset cluster numbers according to a plurality of sample coordinate points associated with the cluster center coordinates corresponding to each of the preset cluster numbers;

[0035] Selecting the preset cluster number corresponding to the largest silhouette coefficient as the target cluster number, and using the cluster center coordinates corresponding to the target cluster number to construct a target foreground point prompt coordinate set;

[0036] A preset random extraction function is used to determine a target background point prompt coordinate set according to the background point prompt coordinate set.

[0037] Optionally, the adopting of the segmentation network point prompt encoder and the segmentation network image decoder to generate a target defect segmentation mask according to the industrial product image to be detected, the target background point prompt coordinate set and the target foreground point prompt coordinate set, and generating a target cropped image based on the target defect segmentation mask, comprises:

[0038] splicing the target background point prompt coordinate set and the target foreground point prompt coordinate set, and outputting a spliced ​​point prompt coordinate set;

[0039] Using the splicing point cue coordinate set as the input of the segmentation network point cue encoder, and outputting point cue features;

[0040] Decoding the point hint features using a segmentation network image decoder and outputting an initial mask;

[0041] Using the initial mask as input to a mask proposal generator, and outputting a target defect segmentation mask;

[0042] The target defect segmentation mask is used to crop the image of the industrial product to be inspected to generate a target cropped image.

[0043] Optionally, the preset defect classification network based on the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module includes a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module; the preset defect classification network based on the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module performs image text feature extraction according to the target cropped image and a plurality of the defect categories to be matched, and outputs the image features corresponding to the target cropped image and the text features corresponding to each of the defect categories to be matched, including:

[0044] Using a self-learning defect category prompt module to determine multiple target learnable prompt features according to each defect category to be matched;

[0045] Generate text features corresponding to each of the target learnable prompt features and image features corresponding to the target cropped image according to each of the target learnable prompt features and the target cropped image through a defect image-text cross-modal feature interaction module;

[0046] The self-learning defect category prompt module is used to determine a plurality of target learnable prompt features according to each defect category to be matched, including:

[0047] Initializing each of the defect categories to be matched, and generating a defect text prompt short sentence corresponding to each of the defect categories to be matched;

[0048] Using a preset character dictionary to convert each of the defect text prompt short sentences respectively, and outputting a word index vector corresponding to each of the defect text prompt short sentences;

[0049] Mapping each of the word index vectors, and outputting a continuous word embedding vector corresponding to each of the word index vectors;

[0050] Based on each of the continuous word embedding vectors, determining a target learnable prompt feature corresponding to each of the continuous word embedding vectors;

[0051] The defect image-text cross-modal feature interaction module includes a text feature encoder and an image feature encoder; the defect image-text cross-modal feature interaction module generates text features corresponding to each target learnable prompt feature and image features corresponding to the target cropped image according to each target learnable prompt feature and the target cropped image, including:

[0052] Using each of the target learnable prompt features as the input of a text feature encoder, and outputting text features corresponding to each of the target learnable prompt features;

[0053] The target cropped image is used as an input of an image feature encoder, and image features are output.

[0054] The second aspect of the present invention provides a device for unsupervised semantic segmentation of product defects based on a pre-trained model, comprising:

[0055] An acquisition module, used to acquire an industrial product image to be detected, an industrial product reference image corresponding to the industrial product image to be detected, and a plurality of defect categories to be matched;

[0056] An input module is used to input the industrial product image to be detected, the industrial product reference image and the multiple defect categories to be matched into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network based on a self-learning point generation prompter of defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module;

[0057] An adopting module is used to use a preset segmentation network of a self-learning point generation prompter based on defect latent space features to generate a target defect segmentation mask according to the industrial product image to be detected and the industrial product reference image, and to generate a target cropped image based on the target defect segmentation mask;

[0058] An output module, configured to extract image and text features according to the target cropped image and the plurality of defect categories to be matched, using a preset defect classification network based on a self-learning defect category prompting module and a defect image-text cross-modal feature interaction module, and output image features corresponding to the target cropped image and text features corresponding to each defect category to be matched;

[0059] A calculation module, used to calculate the matching degree between the image feature and each of the text features, and output the matching degree corresponding to each of the text features;

[0060] The selection module is used to select the defect category associated with the text feature corresponding to the maximum matching degree as the defect classification result corresponding to the industrial product image to be detected.

[0061] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a method for unsupervised semantic segmentation of product defects based on a pre-trained model as described in any one of the above items.

[0062] It can be seen from the above technical solutions that the present invention has the following advantages:

[0063] The above technical scheme of the present invention provides a method for unsupervised semantic segmentation of product defects based on a pre-trained model. First, an industrial product image to be detected, an industrial product reference image corresponding to the industrial product image to be detected, and multiple defect categories to be matched are obtained; then, the industrial product image to be detected, the industrial product reference image, and the multiple defect categories to be matched are input into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network of a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompting module and a defect image-text cross-modal feature interaction module; the preset segmentation network of the self-learning point generation prompter based on defect latent space features is used to generate a target defect segmentation mask according to the industrial product image to be detected and the industrial product reference image, and a target cropped image is generated based on the target defect segmentation mask; the preset defect classification network based on the self-learning defect category prompting module and the defect image-text cross-modal feature interaction module is used to perform image-text feature segmentation on the target cropped image and multiple defect categories to be matched. Feature extraction, output the image features corresponding to the target cropped image and the text features corresponding to each defect category to be matched; calculate the matching degree between the image features and each text feature respectively, and output the matching degree corresponding to each text feature; finally, select the defect category associated with the text feature corresponding to the largest matching degree as the defect classification result corresponding to the industrial product image to be detected; based on the above scheme, the acquired industrial product image to be detected and the industrial product reference image are processed by a preset segmentation network of a self-learning point generation prompter based on defect latent space features to generate a target defect segmentation mask, and based on the target defect segmentation mask, a target cropped image is generated, without the need for manual feature selection and adjustment, and a target cropped image with visual prompts can be automatically generated, so that the generated image features can cover accurate defect category information, and then the target cropped image and multiple defect categories to be matched are processed by a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module, so as to combine segmentation and classification, thereby improving the defect classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0065] Figure 1 A flowchart of the steps of a method for unsupervised semantic segmentation of product defects based on a pre-trained model provided in Example 1 of the present invention;

[0066] Figure 2A schematic diagram of the structure of a preset unsupervised segmentation network framework provided in the first embodiment of the present invention, which is composed of a preset segmentation network of a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module;

[0067] Figure 3 A processing framework diagram of a positive correlation feature matrix provided in the first embodiment of the present invention;

[0068] Figure 4 A schematic diagram of the structure of a segmentation network image encoder provided in Embodiment 1 of the present invention;

[0069] Figure 5 A schematic diagram of the structure of a self-learning defect category prompt module provided in the first embodiment of the present invention;

[0070] Figure 6 A schematic diagram of the structure of a defect image-text cross-modal feature interaction module provided in the first embodiment of the present invention;

[0071] Figure 7 A schematic diagram of a model training process of a preset unsupervised segmentation network framework provided in the second embodiment of the present invention, which is composed of a preset segmentation network of a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module;

[0072] Figure 8 A structural block diagram of a device for unsupervised semantic segmentation of product defects based on a pre-trained model provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0073] The embodiment of the present invention provides a method and device for unsupervised semantic segmentation of product defects based on a pre-trained model, which is used to solve the technical problem that the existing unsupervised semantic segmentation method for industrial defects leads to unsatisfactory defect classification accuracy.

[0074] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1 , Figure 1 A flowchart of the steps of a method for unsupervised semantic segmentation of product defects based on a pre-trained model provided in Example 1 of the present invention.

[0076] The present invention provides a method for unsupervised semantic segmentation of product defects based on a pre-trained model, comprising:

[0077] Step 101: Obtain an industrial product image to be inspected, an industrial product reference image corresponding to the industrial product image to be inspected, and multiple defect categories to be matched.

[0078] It should be noted that the industrial product image to be inspected and the industrial product reference image corresponding to the industrial product image to be inspected are industrial product images obtained after image preprocessing, and the defect categories to be matched are the common defect types of industrial products that have been manually sorted in advance; among which, the image preprocessing includes preliminary positioning and cropping of the image detection area.

[0079] Step 102: input the industrial product image to be inspected, the industrial product reference image and multiple defect categories to be matched into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network based on a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image and text cross-modal feature interaction module.

[0080] Please note that Figure 2The preset unsupervised segmentation network framework is composed of a preset segmentation network of a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module. The preset segmentation network of the self-learning point generation prompter based on defect latent space features is an unsupervised segmentation network based on AP-SAM (Auto-Prompt SAM). The preset segmentation network of the self-learning point generation prompter based on defect latent space features is composed of a self-learning point generation prompter based on defect latent space features, a segmentation network image decoder, a segmentation network point prompt encoder and a mask proposal generator; the preset defect classification network based on the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module is a defect classification network based on IA-CLIP (Industrial-Adapted CLIP). Aiming at the problem that the SAM model cannot output segmentation results with defect category information, the present invention proposes an IA-CLIP network based on CLIP to classify the unsupervised segmentation results of the defect area into different defect categories. Among them, the preset defect classification network based on the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module is composed of the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module, and the defect image-text cross-modal feature interaction module is composed of a text feature encoder and an image feature encoder. The improved module proposed in the present invention includes a self-learning defect category prompt module (Category Prompt Module, abbreviated as CPM) and a defect image-text cross-modal feature interaction module (Cross-modal Feature Interaction Module, abbreviated as CFIM), and the purpose is to enable IA-CLIP to generalize to the industrial field with zero samples so as to classify defects.

[0081] Furthermore, an Automatic Prompt Generator (APG) based on defect latent space features is used to automatically generate visual prompts, thereby generating a defect area segmentation result (target defect segmentation mask) without defect category information, and then cropping the original image (industrial product image to be inspected) according to the segmentation result to obtain a cropped image (target cropped image); the self-learning defect category prompt module is used to automatically learn the sentences describing the category, and the cross-modal feature interaction module is used to better learn the relevant features of the industrial data set.

[0082] Step 103, using a preset segmentation network of a self-learning point generation prompter based on defect latent space features to generate a target defect segmentation mask according to the industrial product image to be detected and the industrial product reference image, and generating a target cropped image based on the target defect segmentation mask.

[0083] Specifically, step 103 may include the following sub-steps S31-S32:

[0084] Step S31, using a self-learning point generation prompter based on defect latent space features to determine a target background point prompt coordinate set and a target foreground point point prompt coordinate set according to the industrial product image to be detected and the industrial product reference image;

[0085] Optionally, a self-learning point generation prompter based on defect latent space features is used to determine a target background point prompt coordinate set and a target foreground point prompt coordinate set according to the industrial product image to be detected and the industrial product reference image, that is, the processing process of the self-learning point generation prompter based on defect latent space features for the input industrial product image to be detected and the industrial product reference image includes:

[0086] Generate defect features, product inspection image reconstruction features, and product reference image reconstruction features according to the industrial product image to be inspected and the industrial product reference image;

[0087] The Hungarian algorithm is used to output multiple positive point prompt feature coordinates and multiple reverse point prompt feature coordinates according to defect features, product inspection image reconstruction features and product reference image reconstruction features;

[0088] Based on the mean clustering algorithm, multiple positive point prompt feature coordinates and multiple reverse point prompt feature coordinates are clustered to determine the target background point prompt coordinate set and the target foreground point prompt coordinate set.

[0089] Furthermore, the process of generating defect features, product inspection image reconstruction features and product reference image reconstruction features according to the industrial product image to be inspected and the industrial product reference image includes the following sub-steps S311-S313:

[0090] Step S311, respectively encode the industrial product image to be detected and the industrial product reference image, and output the features of the industrial product image to be detected and the features of the industrial product reference image;

[0091] It should be noted that the features of the reference image (industrial product reference image) and the test image (industrial product image to be detected) extracted by the image encoder (segmentation network image encoder) of the SAM model are respectively the features of the industrial product image to be detected and industrial product reference image features ,in, , , H is the height of the input image, W is the width of the input image, and C is the number of channels of the input image.

[0092] Step S312, reshape the image features of the industrial product to be inspected and the industrial product reference image features respectively to generate product inspection image reshape features and product reference image reshape features;

[0093] It should be noted that the image features of the industrial product to be detected and the reference image features of the industrial product are reshaped into the reshaped features of the product detection image. and product reference image reshape features , , .

[0094] Step S313: perform spatial multiplication on the reference mask corresponding to the industrial product reference image and the industrial product reference image feature, and output the defect feature.

[0095] It should be noted that the mask M corresponding to the industrial product reference image is used R Reference image features of industrial products Perform spatial multiplication to obtain defect characteristics , L is the number of features corresponding to the defect area in the mask.

[0096] Furthermore, the process of outputting a plurality of positive point prompt feature coordinates and a plurality of reverse point prompt feature coordinates according to the defect feature, the product inspection image reconstruction feature and the product reference image reconstruction feature by using the Hungarian algorithm includes the following sub-steps S321-S324:

[0097] Step S321, performing matrix multiplication on defect features and product inspection image reconstruction features to generate a positive correlation feature matrix;

[0098] It should be noted that the defect characteristics and product inspection image reconstruction features Do matrix multiplication to get the positive correlation feature matrix , whose size is L*H*W, which means that the image features of the defective area in the reference image are used to calculate the similarity of the test image features. The positive correlation feature matrix Each row of represents the similarity between each feature of the defect area in the reference image and the feature of the test image.

[0099] It is worth mentioning that see Figure 3 , the image encoder (segmentation network image encoder) of the SAM model is used to extract the features of the reference image (industrial product reference image) and the test image (industrial product image to be detected) as the features of the industrial product image to be detected. and industrial product reference image features Then the image features of the industrial product to be detected and the reference image features of the industrial product are reshaped into the reshaped features of the product detection image. and product reference image reshape features , using the mask M corresponding to the industrial product reference image R Reference image features of industrial products Perform spatial multiplication to obtain defect features. and product inspection image reconstruction features Do matrix multiplication to get the positive correlation feature matrix , the above process can be expressed by the following expression:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, It is the image encoder of SAM model; To reshape; is spatial multiplication; is matrix multiplication.

[0105] For further information, see Figure 4 The process of image feature extraction by the segmentation network image encoder is as follows: first, the input image is processed by the Transformer Layer to obtain deep features and shallow features. The shallow features are convolved by the convolution layer (Conv) and then concatenated with the deep features, that is, the shallow features in the image encoder are saved. Then, the dimensions are adjusted to be consistent with the deep features through the convolution layer, and finally, they are added with the deep features as the output of the image encoder, thereby outputting the final image features.

[0106] Step S322: using the Hungarian algorithm to output multiple positive point prompt feature coordinates according to the positive correlation feature matrix;

[0107] It should be noted that the similarity between the defect features and the test image features is maximized based on the binary matching. At this time, L defect features will match L different features in the test image to achieve the highest overall similarity. The coordinates of the preliminary point prompts are converted according to the index, that is, the Hungarian algorithm is used to output multiple positive point prompt feature coordinates according to the positive correlation feature matrix; the processing process of the positive point prompt feature coordinates can be expressed as:

[0108] ;

[0109] Among them, x is the horizontal coordinate of the point prompt feature coordinates extracted from the feature map by the image encoder; y is the vertical coordinate of the point prompt feature coordinates extracted from the feature map by the image encoder, and the point prompt feature coordinates include forward point prompt feature coordinates and reverse point prompt feature coordinates; index is the coordinate index of each defect feature in the test image (industrial product image to be tested) that matches the reference image (industrial product reference image); Hungarian(.) is the Hungarian algorithm, which is a classic algorithm for solving bipartite matching.

[0110] It is worth mentioning that if you want to convert the coordinates extracted to the feature map by the image encoder into the original image coordinates, you need to make further conversions based on the size ratio of the original image and the feature map.

[0111] Step S323, selecting the positive features corresponding to the coordinates of each positive point prompt feature in the product inspection image reconstruction features, and performing matrix multiplication using each positive feature and the product reference image reconstruction features to output a reverse correlation feature matrix;

[0112] It should be noted that, after the above process, a positive point prompt feature coordinate set consisting of a plurality of positive point prompt feature coordinates is obtained. , The feature coordinates of the i-th positive point are indicated, where there are L coordinates in total. At this time, the feature Z is reshaped in the product inspection image according to these L coordinates. T Take the L positive points corresponding to the position in the prompt feature coordinates and the product reference image to reshape the feature Z R Perform matrix multiplication to obtain the reverse correlation feature matrix .

[0113] Step S324: using the Hungarian algorithm to generate multiple reverse point prompt feature coordinates according to the reverse correlation feature matrix.

[0114] It should be noted that in Using the Hungarian algorithm again for reverse matching will yield L coordinate sets. , Prompt feature coordinates for the i-th reverse point.

[0115] Furthermore, the process of clustering the plurality of forward point prompt feature coordinates and the plurality of reverse point prompt feature coordinates based on the mean clustering algorithm to determine the target background point prompt coordinate set and the target foreground point prompt coordinate set includes the following sub-steps S331-S337:

[0116] Step S331: taking the forward point prompt feature coordinates associated with any reverse point prompt feature coordinates not in the reference mask corresponding to the industrial product reference image as the background point prompt coordinates, and taking the forward point prompt feature coordinates associated with any reverse point prompt feature coordinates in the reference mask as the foreground point prompt coordinates;

[0117] It should be noted that if the reverse matching coordinate set Any reverse point in the reference mask M indicates that the feature coordinates are not in the reference mask M. R In the example, the positive point prompt feature coordinates in the positive matching coordinate set (positive point prompt feature coordinate set) corresponding to the reverse point prompt feature coordinates are filtered out, and each positive point prompt feature coordinate after filtering is obtained as the foreground point prompt coordinate, and each positive point prompt feature coordinate filtered out is used as the background point prompt coordinate.

[0118] Step S332, using multiple background point prompt coordinates to construct a background point prompt coordinate set;

[0119] Step S333, using multiple foreground point prompt coordinates to construct a foreground point prompt coordinate set;

[0120] It should be noted that based on the above foundation, the filtered point prompt coordinate set is obtained, that is, the foreground point prompt coordinate set , The i-th foreground point prompt coordinates are obtained to obtain the filtered background point prompt coordinate set , Prompt the coordinates of the i-th background point. The above process can be expressed as:

[0121]

[0122] in, The processing flow of prompting feature coordinates for forward points and reverse points.

[0123] Step S334: Based on multiple preset clustering numbers, a mean clustering algorithm is used to cluster the foreground point prompt coordinate set, and the cluster center coordinates corresponding to each preset clustering number and multiple sample coordinate points associated with the cluster center coordinates corresponding to each preset clustering number are output;

[0124] It should be noted that the k-means++ clustering algorithm (mean clustering algorithm) is used to cluster the foreground point coordinates into Cluster into k clusters, and then use the cluster center coordinates as the final foreground point prompt. Randomly prompt the background point coordinate set i points are selected as the final background point prompts, and together with the above final foreground point prompts, they are used as the input of the prompt encoder (segmentation network point prompt encoder) in SAM.

[0125] The method of selecting cluster centers is as follows: first, randomly select a sample from the sample set as the first cluster center; second, traverse the remaining samples and calculate the distance from each sample to the nearest existing cluster center; third, select the sample with the farthest distance as the next cluster center. Repeat the second and third steps until k cluster centers are selected.

[0126] Furthermore, the number of foreground point hints should be greater than or equal to the number of defective areas in the test image. If the condition is not met, the foreground point hints after clustering may not be located on the defective area in the test image, and the segmentation effect of the defective area will be significantly reduced. The k value in the k-means++ algorithm is the number of foreground point hints, so the k value cannot be a fixed value, but varies according to different test images. In the present invention, the silhouette coefficient is used to determine the k value (preset number of clustering clusters).

[0127] It is worth mentioning that the rules for determining the number of preset clusters are as follows, and a range of values ​​is artificially set. Clustering is performed under each k value, and the silhouette coefficient of the clustering result is finally calculated, and the k value corresponding to the maximum silhouette coefficient is taken as the final k value. In the field of defect detection, each test image contains few defect areas, so in the present invention, the upper limit of the k value is set to 4.

[0128] Step S335, calculating the silhouette coefficient corresponding to each preset number of clusters according to a plurality of sample coordinate points associated with the cluster center coordinates corresponding to each preset number of clusters;

[0129] It should be noted that the silhouette coefficient is a commonly used indicator for evaluating clustering effects, which comprehensively considers the cohesion and separation of clusters. Cohesion reflects the similarity between samples in the same cluster. The smaller the value, the closer the samples in the cluster. Separation reflects the degree of distinction between samples in different clusters. The larger the value, the more distant the samples between clusters. The calculation formula of the silhouette coefficient is as follows:

[0130]

[0131] in, is the average distance within the class of sample i; n is the total number of samples in the class where sample i belongs; i is the other sample points in the same class as sample point j; j is the other sample points in the same class as sample point i; dist is the distance between i and j; is the minimum average distance within the class of sample i; k is the sample point that is not in the same class as sample i; is the silhouette coefficient of sample i; N is the total number of samples; S is the silhouette coefficient of the entire sample set, and its value range is , the larger the value, the better the clustering effect; m is the total number of samples in the class where sample k belongs.

[0132] Step S336, selecting the preset cluster number corresponding to the largest silhouette coefficient as the target cluster number, and using the cluster center coordinates corresponding to the target cluster number to construct a target foreground point prompt coordinate set;

[0133] Step S337: using a preset random extraction function to determine the target background point prompt coordinate set according to the background point prompt coordinate set.

[0134] Step S32, using a segmentation network point prompt encoder and a segmentation network image decoder to generate a target defect segmentation mask according to the industrial product image to be detected, a target background point prompt coordinate set, and a target foreground point prompt coordinate set, and based on the target defect segmentation mask, generate a target cropped image;

[0135] Further, a segmentation network point prompt encoder and a segmentation network image decoder are used to generate a target defect segmentation mask according to the industrial product image to be detected, the target background point prompt coordinate set and the target foreground point prompt coordinate set, and a target cropped image is generated based on the target defect segmentation mask, including the following sub-steps S341-S345:

[0136] Step S341, splicing the target background point prompt coordinate set and the target foreground point prompt coordinate set, and outputting the spliced ​​point prompt coordinate set;

[0137] It should be noted that the processing process of the splicing point prompt coordinate set can be expressed as:

[0138]

[0139] in, Provide a coordinate set for the target foreground sight point; It is the k-means++ algorithm; Prompt coordinate sets for target background points; To preset random extraction function; is splicing; P is the splicing point prompt coordinate set.

[0140] Step S342, using the splicing point prompt coordinate set as the input of the segmentation network point prompt encoder, and outputting the point prompt feature;

[0141] It should be noted that the splicing point cue coordinate set is used as the input of the segmentation network point cue encoder (SAM cue encoder) to output point cue features.

[0142] Step S343, using a segmentation network image decoder to decode the point prompt feature and output an initial mask;

[0143] Step S344: using the initial mask as input to the mask proposal generator, and outputting a target defect segmentation mask;

[0144] It should be noted that the initial mask is used as the input of the Mask Proposal Generator, which outputs the target defect segmentation mask.

[0145] Step S345: cropping the image of the industrial product to be inspected using the target defect segmentation mask to generate a target cropped image.

[0146] It should be noted that the image of the industrial product to be inspected is cropped according to the target defect segmentation mask to obtain a cropped image (target cropped image).

[0147] Step 104: Use a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module to extract image and text features according to the target cropped image and multiple defect categories to be matched, and output image features corresponding to the target cropped image and text features corresponding to each defect category to be matched.

[0148] Specifically, step 104 may include the following sub-steps S41-S42, including:

[0149] Step S41, using a self-learning defect category prompt module to determine multiple target learnable prompt features according to each defect category to be matched;

[0150] Optionally, the process of using the self-learning defect category prompt module to determine multiple target learnable prompt features according to each defect category to be matched, that is, the process of the self-learning defect category prompt module for each defect category to be matched, includes:

[0151] Initialize each defect category to be matched, and generate a defect text prompt short sentence corresponding to each defect category to be matched;

[0152] Use a preset character dictionary to convert each defect text prompt short sentence respectively, and output the word index vector corresponding to each defect text prompt short sentence;

[0153] Map each word index vector and output the continuous word embedding vector corresponding to each word index vector;

[0154] Based on each continuous word embedding vector, determine the target learnable prompt feature corresponding to each continuous word embedding vector;

[0155] Please note that Figure 5 , for the self-learning defect category prompt module, it includes learnable prompts for defect categories (vecD) and good categories (vecG). The data processing process of the self-learning defect category prompt module is: first, each defect category to be matched is initialized into a short sentence form of category prompt (defect text prompt short sentence), such as "X X... Xclassi ”, where X is equivalent to a placeholder text prompt, which will be replaced by a learnable defect text prompt later. Then the defect text prompt is converted into a word index (word index vector) through a character dictionary (preset character dictionary). For example, the index of the placeholder character X in the character dictionary is 343. At this time, the defect category prompt will become a word index vector with a dimension of [k,77]. By default, all text prompts are composed of 77 words, and the missing part is replaced by index 0. Then map the word index vector to a continuous word embedding vector, that is, map each word index to a word index vector with a dimension of D t The vector of dimension [k,77,D t ]’s continuous word embedding vector, this process can be expressed as:

[0156] ;

[0157] in, is a continuous word embedding vector; Prompt short sentences for defective texts; To preset the character dictionary, the character dictionary is converted into a word index vector of dimension [k,77]. By default, all text prompts are composed of 77 words, and the insufficient part is replaced by index 0; E is the mapping function; k is the number of defect categories.

[0158] Furthermore, each continuous word embedding vector obtained above is sliced ​​into [k, 1, D t ],[k,Num v ,D t ],[k,76-Num v ,D t ]Three consecutive word embedding sub-vectors, where Num v is the number of defect text prompts that can be learned, and uses the function Define a dimension [k,Num v ,D t ] is a learnable continuous vector used to replace the continuous word embedding sub-vector of the same dimension. Finally, the replaced continuous word embedding sub-vector is concat at the position of dim = 1 to obtain a learnable continuous word embedding vector of the defect category with a dimension of [k,77,D t ], the process can be expressed as:

[0159]

[0160] in, is the first continuous word embedding sub-vector; is the second continuous word embedding sub-vector; is the third continuous word embedding sub-vector; For slicing; This is a function provided by the deep learning framework pytorch. The input of Para is an empty tensor with a certain shape. The function is to convert the empty tensor into a trainable parameter, that is, a learnable continuous vector. is a continuous vector that can be learned; Embedding vector for the target continuous word; is a continuous word embedding vector; is the number of learnable defect text prompt words.

[0161] Furthermore, for the good product category, the hint can be learned, and the function is directly used Define a dimension [1,77,D t ] A continuous vector that can be learned. Finally, the learnable prompts of the defective category and the good category are concat at the position of dim = 0 to obtain the final input of the text encoder, that is, the target learnable prompt feature .

[0162]

[0163] in, is a dimension [1,77,D t ]Learnable continuous vectors.

[0164] Step S42: Generate text features corresponding to each target learnable prompt feature and image features corresponding to the target cropped image through the defect image-text cross-modal feature interaction module according to each target learnable prompt feature and the target cropped image;

[0165] Optionally, the process of generating text features corresponding to each target learnable prompt feature and image features corresponding to the target cropped image according to each target learnable prompt feature and the target cropped image through the defect image-text cross-modal feature interaction module includes:

[0166] Taking each target learnable prompt feature as the input of the text feature encoder, and outputting the text feature corresponding to each target learnable prompt feature;

[0167] The target cropped image is used as the input of the image feature encoder and the image features are output.

[0168] Please note that Figure 6 The defect image-text cross-modal feature interaction module consists of a text feature encoder and an image feature encoder. The obtained target cropped image and each target learnable prompt feature are used as the input of the defect image-text cross-modal feature interaction module, and the image feature and multiple text features are output. Specifically, each transformer block in the text encoder first introduces L learnable vectors , until Layer. At this time, the input of each transformer block follows of the form, where Corresponding to the original input vector of the text encoder. At this time, the entire process of the text encoder can be described by the following formula:

[0169]

[0170] in, For connection operation; is the Transformer Block of the text feature encoder; z is the text feature; The fully connected layer used for the final output of the CLIP text encoder (text feature encoder); is the output of the i-th layer, As input, that is , ; is the learnable vector generated in the i-1th layer text feature encoder, which represents the text feature vector after being encoded by the i-1th layer Transformer; D is the dimension of the feature vector; is the output of the j-th layer text feature encoder; Add P to the text encoder after layer D T The additional text feature vector brought by j>D+1; Add a new learnable vector P for the D-th layer text encoder T The additional text feature vector brought by, j-1=D; k is the total number of layers in the text feature encoder in CLIP; It is the raw output of the last layer of the text feature encoder of CLIP.

[0171] It is worth mentioning that after the Dth layer conversion, the subsequent layers will directly process the output vector of the previous layer, and finally use The function calculates the final text features.

[0172] Furthermore, similar to the text feature encoder and , each transformer block in the image feature encoder also introduces L learnable vectors Until the At this point, the entire process of the image feature encoder can be described by the following formula:

[0173]

[0174] in, is the image feature encoder; x is the image feature; is the output of the i-th layer, and the target cropped image As input, that is ; It is the additional picture feature part of the output of the CLIP image encoder; Add P to the text encoder after layer D T The additional text feature vector brought by j>D+1; is the learnable vector generated in the i-1th layer image feature encoder, representing the image feature vector after being encoded by the i-1th layer Transformer; The fully connected layer used for the final output of the CLIP image encoder (image feature encoder).

[0175] Furthermore, in the above text feature encoder and image feature encoder, the number of added learnable vectors is the same, and the layers in which they are added are also the same. The additionally added learnable vectors in the text encoder and image encoder of the same layer are coordinated with each other through an interaction module, specifically:

[0176]

[0177] in, is the i-th vector added additionally in the image feature encoder; is the i-th vector added additionally in the text feature encoder; is the cross attention function; is the text-to-image output of the criss-cross attention function at the i-th layer; Image to Text output of the i-th layer cross attention function; is the image feature encoder; is a text feature encoder.

[0178] Furthermore, the implementation process can be described as follows:

[0179]

[0180] in, is the first trainable parameter; is the second trainable parameter, Their roles are similar to the query, value, and output linear layers in the attention mechanism; is the image feature P I Query vector in cross-modal attention mechanism; is the text feature P T Query vector in cross-modal attention mechanism; is the image feature; is the text feature; d is the dimension of the feature; is the image feature P I The value vector in the cross-modal attention mechanism; is the text feature P T The value vector in the cross-modal attention mechanism; is the Text to Image output of the cross-attention function; ImagetoText output of the cross-attention function.

[0181] In this embodiment, after adding the learnable vector P, the text encoder will originally output a vector [W, P] in layers 1 to D. The present invention will retain W, and then each layer will replace the previous layer with a new P (part) and output it. In the D+1th layer to the kth layer, [W, P] will also be output. The present invention retains both W and P outputted by these layers until the output of the last layer cuts off the W part, that is, W nk ; Slightly different from the text encoder, the original image feature encoder will produce two partial outputs [cls, E]. After adding the learnable vector P, [cls, E, P] will be generated in the original image feature encoder layer. In the image encoder, the present invention only uses cls in the final output.

[0182] Step 105: Calculate the matching degree between the image feature and each text feature respectively, and output the matching degree corresponding to each text feature.

[0183] It should be noted that the process of calculating the matching degree can refer to the prior art, and the present invention will not go into details.

[0184] Step 106: Select the defect category associated with the text feature corresponding to the maximum matching degree as the defect classification result corresponding to the industrial product image to be detected.

[0185] It should be noted that the image features and the text features are matched, and the category corresponding to the text feature with the highest matching degree is taken as the defect category of the image, that is, the defect category associated with the text feature corresponding to the largest matching degree is selected as the defect classification result corresponding to the industrial product image to be detected.

[0186] As a comparison of technical effects, it can be combined with existing technologies for reference. Surface defect detection is an important task in the industrial field. At present, the most widely used deep learning technology in this field is image classification, that is, classifying the input image into defects and good products. However, the classification task provides limited information, and in the industrial field, it is more desirable to obtain detailed information on product defects. Unlike the classification task, semantic segmentation can classify each pixel in the image, thereby segmenting the defective area from the image and providing more detailed defect information. Traditional methods require a large amount of labeled data for supervised model training. The key to unsupervised semantic segmentation is to design effective tasks and loss functions, use low-level features of the image (such as color, texture, etc.) or high-level semantic features (such as objects), guide the model to autonomously learn the potential structure and model of the data, and finally achieve pixel-level segmentation of the image and output the defective area. In the above process, no labeled data is required to train the unsupervised semantic segmentation model.

[0187] Based on the above foundation, the present invention proposes an unsupervised semantic segmentation method for product defects based on a pre-trained model, which can accurately locate the defect area and accurately identify the defect category without any labeled samples for training, and solves the two major shortcomings of the SAM model in the field of surface defects. One is that manual prompts are required; the other is that the defect segmentation result of SAM has no defect category information. Therefore, the model of the present invention is divided into two parts, an unsupervised segmentation network based on AP-SAM (Auto-Prompt SAM) and a defect classification network based on IA-CLIP (Industrial-AdaptedCLIP). Specifically, for the input image, the AP-SAM network will first automatically generate visual prompts through the self-learning point prompt generator (Automatic Prompt Generator, APG) based on the defect latent space feature proposed in this paper, thereby generating a defect area segmentation result without defect category information. Then the original image is cropped according to the segmentation result, and finally the cropped image is input into the IA-CLIP network to obtain the category information of the defect area segmentation result. The IA-CLIP network proposes a Category Prompt Module (CPM) and a Cross-modal Feature Interaction Module (CFIM). The Category Prompt Module is used to automatically learn sentences describing categories, and the Cross-modal Feature Interaction Module is used to better learn relevant features of industrial data sets. The ultimate goal of the two improved modules is to make the IA-CLIP network applicable to industrial data sets, thereby obtaining higher defect classification accuracy.

[0188] In an embodiment of the present invention, the present invention provides a method for unsupervised semantic segmentation of product defects based on a pre-trained model. First, an industrial product image to be detected, an industrial product reference image corresponding to the industrial product image to be detected, and multiple defect categories to be matched are obtained; then, the industrial product image to be detected, the industrial product reference image, and the multiple defect categories to be matched are input into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network of a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompting module and a defect image-text cross-modal feature interaction module; the preset segmentation network of the self-learning point generation prompter based on defect latent space features is used to generate a target defect segmentation mask according to the industrial product image to be detected and the industrial product reference image, and a target cropped image is generated based on the target defect segmentation mask; the preset defect classification network based on the self-learning defect category prompting module and the defect image-text cross-modal feature interaction module is used to perform image text according to the target cropped image and multiple defect categories to be matched. Feature extraction, output the image features corresponding to the target cropped image and the text features corresponding to each defect category to be matched; calculate the matching degree of the image features with each text feature respectively, and output the matching degree corresponding to each text feature; finally, select the defect category associated with the text feature corresponding to the largest matching degree as the defect classification result corresponding to the industrial product image to be detected; based on the above scheme, the acquired industrial product image to be detected and the industrial product reference image are processed by the preset segmentation network of the self-learning point generation prompter based on the defect latent space feature to generate a target defect segmentation mask, and based on the target defect segmentation mask, a target cropped image is generated. There is no need to manually select and adjust the features, and the target cropped image with visual prompts can be automatically generated, so that the generated image features can cover accurate defect category information, and then the target cropped image and multiple defect categories to be matched are processed by the preset defect classification network based on the self-learning defect category prompt module and the defect image and text cross-modal feature interaction module, combining segmentation and classification, thereby improving the defect classification accuracy.

[0189] For better explanation, refer to Figure 7 , a schematic diagram of the model training process of a preset unsupervised segmentation network framework provided by the second embodiment of the present invention, which is composed of a preset segmentation network of a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module, is shown. The process may include the following steps:

[0190] Step S1: Collect industrial data sets and preprocess the data sets;

[0191] Step S2: Construct a self-learning point generation prompter based on defect latent space features to provide auxiliary supervision information for SAM and establish an unsupervised segmentation network (AP-SAM for short).

[0192] Step S3: construct a self-learning defect category prompt module as the input of the CLIP text encoder;

[0193] Step S4: Establish a defect image-text cross-modal feature interaction module between the text encoder and the image encoder of CLIP;

[0194] Step S5: Based on steps S3 and S4, the dataset obtained in step S1 is used for fine-tuning and migration to establish a defect classification network (IA-CLIP for short).

[0195] Step S6: Use the segmentation result output by AP-SAM as the input of the IA-CLIP image encoder to obtain an unsupervised segmentation network framework with category information.

[0196] It should be noted that the unsupervised segmentation network (AP-SAM for short) is a preset segmentation network of a self-learning point generation prompter based on defect latent space features. The defect classification network to be trained is fine-tuned and migrated through the acquired data set, thereby obtaining a trained defect classification network (preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module). The preset defect classification network based on the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module and the preset segmentation network of the self-learning point generation prompter based on defect latent space features are used to construct a preset unsupervised segmentation network framework; for the fine-tuning migration process, the original weight parameters of CLIP are frozen, and the data set obtained in step S1 is used to train the weight parameters of the modules added in steps S3 and S4. Specifically, the present invention uses a model training strategy for fine-tuning migration, that is, only a small-scale data set is used to fine-tune part of the added weight parameters, so that the model can learn the visual features of defect category prompts and defect fields. The original weight parameters are fixed unchanged, that is, the original visual features are kept. At this point, the model can learn visual features in other image areas, namely, defect visual features, without destroying the visual features originally learned by pre-training with large-scale datasets. Finally, the trained model is used to perform zero-sample generalization on the target defect dataset. The hyperparameter settings for model fine-tuning are shown in Table 1.

[0197] Table 1 Hyperparameter settings

[0198]

[0199] In this embodiment, the improvement of SAM is to construct a self-learning point generation prompter based on defect latent space features, including a point prompt generation module, a prompt sampler and a feature fusion module; the point prompt generation module generates point prompt coordinate information of areas similar to the defect area in the reference image; the prompt sampler uses the k-means++ clustering algorithm to cluster the points obtained by the point prompt generation module; the feature fusion module fuses the shallow features and deep features in the image encoder as the output of the image encoder; the improvement of CLIP is to construct a self-learning defect category prompt module to generate text vectors corresponding to the defect category; a defect image-text cross-modal feature interaction module is constructed for better feature extraction of image data and text data; an industrial defect data set is collected to fine-tune the construction category prompt module and the cross-module feature interaction module in CLIP to obtain a trained CLIP model. The present invention can realize unsupervised surface defect segmentation in industrial scenarios.

[0200] See also Figure 8 , Figure 8 A structural block diagram of a device for unsupervised semantic segmentation of product defects based on a pre-trained model provided in Example 3 of the present invention.

[0201] The present invention provides a product defect unsupervised semantic segmentation device based on a pre-trained model, comprising:

[0202] An acquisition module 801 is used to acquire an industrial product image to be detected, an industrial product reference image corresponding to the industrial product image to be detected, and a plurality of defect categories to be matched;

[0203] An input module 802 is used to input the industrial product image to be detected, the industrial product reference image and multiple defect categories to be matched into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network based on a self-learning point generation prompter based on defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module;

[0204] Using module 803, it is used to generate a target defect segmentation mask according to the industrial product image to be inspected and the industrial product reference image using a preset segmentation network of a self-learning point generation prompter based on defect latent space features, and generate a target cropped image based on the target defect segmentation mask;

[0205] An output module 804 is used to extract image and text features according to a target cropped image and a plurality of defect categories to be matched using a preset defect classification network based on a self-learning defect category prompting module and a defect image-text cross-modal feature interaction module, and output image features corresponding to the target cropped image and text features corresponding to each defect category to be matched;

[0206] A calculation module 805 is used to calculate the matching degree between the image feature and each text feature, and output the matching degree corresponding to each text feature;

[0207] The selection module 806 is used to select the defect category associated with the text feature corresponding to the maximum matching degree as the defect classification result corresponding to the industrial product image to be detected.

[0208] Further, the preset segmentation network of the self-learning point generation prompter based on defect latent space features includes a self-learning point generation prompter based on defect latent space features, a segmentation network image decoder, a segmentation network point prompt encoder and a mask proposal generator; module 803 is adopted, including:

[0209] The first submodule is used to determine a target background point prompt coordinate set and a target foreground point prompt coordinate set according to the industrial product image to be detected and the industrial product reference image by using a self-learning point generation prompter based on defect latent space features;

[0210] The second submodule is used to generate a target defect segmentation mask according to the industrial product image to be detected, the target background point prompt coordinate set and the target foreground point prompt coordinate set by using a segmentation network point prompt encoder and a segmentation network image decoder, and generate a target cropped image based on the target defect segmentation mask;

[0211] Furthermore, the first submodule includes:

[0212] The first unit is used to generate defect features, product inspection image reconstruction features and product reference image reconstruction features according to the industrial product image to be inspected and the industrial product reference image;

[0213] The second unit is used to output a plurality of positive point prompt feature coordinates and a plurality of reverse point prompt feature coordinates according to defect features, product inspection image reconstruction features and product reference image reconstruction features by using the Hungarian algorithm;

[0214] The third unit is used to cluster the multiple forward point prompt feature coordinates and the multiple reverse point prompt feature coordinates based on the mean clustering algorithm to determine a target background point prompt coordinate set and a target foreground point prompt coordinate set.

[0215] Furthermore, the first unit is specifically used for:

[0216] Encode the image of the industrial product to be detected and the reference image of the industrial product respectively, and output the features of the image of the industrial product to be detected and the features of the reference image of the industrial product;

[0217] Reshape the image features of the industrial product to be inspected and the reference image features of the industrial product respectively to generate the reshaped features of the product inspection image and the reshaped features of the product reference image;

[0218] Perform spatial multiplication on the reference mask corresponding to the industrial product reference image and the industrial product reference image features to output the defect features.

[0219] Furthermore, the second unit is specifically used for:

[0220] Perform matrix multiplication on defect features and product inspection image reconstruction features to generate a positive correlation feature matrix;

[0221] The Hungarian algorithm is used to output multiple positive point prompt feature coordinates according to the positive correlation feature matrix;

[0222] Select the positive features corresponding to the feature coordinates of each positive point prompt in the product inspection image reconstruction features, and use each positive feature and the product reference image reconstruction features to perform matrix multiplication to output a reverse correlation feature matrix;

[0223] The Hungarian algorithm is used to generate multiple reverse point prompt feature coordinates according to the reverse correlation feature matrix.

[0224] Furthermore, the third unit is specifically used for:

[0225] The forward point prompt feature coordinates associated with any reverse point prompt feature coordinates not in the reference mask corresponding to the industrial product reference image are used as background point prompt coordinates, and the forward point prompt feature coordinates associated with any reverse point prompt feature coordinates in the reference mask are used as foreground point prompt coordinates;

[0226] Using multiple background point prompt coordinates to construct a background point prompt coordinate set;

[0227] Using multiple foreground point prompt coordinates to construct a foreground point prompt coordinate set;

[0228] Based on multiple preset clustering numbers, a mean clustering algorithm is used to cluster the foreground point prompt coordinate set, and the cluster center coordinates corresponding to each preset clustering number and multiple sample coordinate points associated with the cluster center coordinates corresponding to each preset clustering number are output;

[0229] Calculate the silhouette coefficient corresponding to each preset number of clusters according to a plurality of sample coordinate points associated with the cluster center coordinates corresponding to each preset number of clusters;

[0230] The preset cluster number corresponding to the largest silhouette coefficient is selected as the target cluster number, and the cluster center coordinates corresponding to the target cluster number are used to construct the target foreground point prompt coordinate set;

[0231] A preset random extraction function is used to determine the target background point prompt coordinate set based on the background point prompt coordinate set.

[0232] Furthermore, the second submodule is specifically used for:

[0233] splicing the target background point prompt coordinate set and the target foreground point prompt coordinate set, and outputting the spliced ​​point prompt coordinate set;

[0234] The splicing point cue coordinate set is used as the input of the segmentation network point cue encoder, and the point cue features are output;

[0235] Use the segmentation network image decoder to decode the point hint features and output the initial mask;

[0236] The initial mask is used as input to the mask proposal generator, which outputs the target defect segmentation mask.

[0237] The target defect segmentation mask is used to crop the image of the industrial product to be inspected to generate a target cropped image.

[0238] Further, the preset defect classification network based on the self-learning defect category prompting module and the defect image-text cross-modal feature interaction module includes the self-learning defect category prompting module and the defect image-text cross-modal feature interaction module; the output module 804 includes:

[0239] The third submodule is used to determine multiple target learnable prompt features according to each defect category to be matched by using a self-learning defect category prompt module;

[0240] The fourth submodule is used to generate text features corresponding to each target learnable prompt feature and image features corresponding to the target cropped image according to each target learnable prompt feature and the target cropped image through the defect image-text cross-modal feature interaction module;

[0241] Furthermore, the third submodule is specifically used for:

[0242] Initialize each defect category to be matched, and generate a defect text prompt short sentence corresponding to each defect category to be matched;

[0243] Use a preset character dictionary to convert each defect text prompt short sentence respectively, and output the word index vector corresponding to each defect text prompt short sentence;

[0244] Map each word index vector and output the continuous word embedding vector corresponding to each word index vector;

[0245] Based on each continuous word embedding vector, determine the target learnable prompt feature corresponding to each continuous word embedding vector;

[0246] Furthermore, the defect image-text cross-modal feature interaction module includes a text feature encoder and an image feature encoder; the fourth submodule is specifically used for:

[0247] Taking each target learnable prompt feature as the input of the text feature encoder, and outputting the text feature corresponding to each target learnable prompt feature;

[0248] The target cropped image is used as the input of the image feature encoder and the image features are output.

[0249] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules, sub-modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0250] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of a method for unsupervised semantic segmentation of product defects based on a pre-trained model as in any of the above embodiments.

[0251] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for unsupervised semantic segmentation of product defects based on a pre-trained model, characterized in that: include: Acquire an industrial product image to be detected, an industrial product reference image corresponding to the industrial product image to be detected, and multiple defect categories to be matched; Input the industrial product image to be detected, the industrial product reference image and the plurality of defect categories to be matched into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network based on a self-learning point generation prompter of defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module; A preset segmentation network of a self-learning point generation prompter based on defect latent space features is used to generate a target defect segmentation mask according to the industrial product image to be inspected and the industrial product reference image, and a target cropped image is generated based on the target defect segmentation mask; Using a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module to extract image and text features according to the target cropped image and a plurality of defect categories to be matched, and outputting image features corresponding to the target cropped image and text features corresponding to each defect category to be matched; Calculate the matching degree between the image feature and each text feature respectively, and output the matching degree corresponding to each text feature; The defect category associated with the text feature corresponding to the maximum matching degree is selected as the defect classification result corresponding to the industrial product image to be detected.

2. According to the method of unsupervised semantic segmentation of product defects based on a pre-trained model in claim 1, it is characterized in that: The preset segmentation network of the self-learning point generation prompter based on defect latent space features includes a self-learning point generation prompter based on defect latent space features, a segmentation network image decoder, a segmentation network point prompt encoder and a mask proposal generator; the preset segmentation network using the self-learning point generation prompter based on defect latent space features generates a target defect segmentation mask according to the industrial product image to be detected and the industrial product reference image, and generates a target cropped image based on the target defect segmentation mask, including: Using a self-learning point generation prompter based on defect latent space features to determine a target background point prompt coordinate set and a target foreground point point prompt coordinate set according to the industrial product image to be detected and the industrial product reference image; A segmentation network point prompt encoder and a segmentation network image decoder are used to generate a target defect segmentation mask according to the industrial product image to be detected, the target background point prompt coordinate set and the target foreground point prompt coordinate set, and a target cropped image is generated based on the target defect segmentation mask.

3. The method for unsupervised semantic segmentation of product defects based on a pre-trained model according to claim 2, characterized in that: The self-learning point generation prompter based on defect latent space features is used to determine the target background point prompt coordinate set and the target foreground point point prompt coordinate set according to the industrial product image to be detected and the industrial product reference image, including: Generate defect features, product inspection image reconstruction features, and product reference image reconstruction features according to the industrial product image to be inspected and the industrial product reference image; The Hungarian algorithm is used to output a plurality of positive point prompt feature coordinates and a plurality of reverse point prompt feature coordinates according to the defect feature, the product inspection image reconstruction feature and the product reference image reconstruction feature; Based on the mean clustering algorithm, the plurality of forward point prompt feature coordinates and the plurality of reverse point prompt feature coordinates are clustered to determine a target background point prompt coordinate set and a target foreground point prompt coordinate set.

4. The method for unsupervised semantic segmentation of product defects based on a pre-trained model according to claim 3, characterized in that: The generating of defect features, product inspection image reconstruction features and product reference image reconstruction features according to the industrial product image to be inspected and the industrial product reference image comprises: Encode the to-be-detected industrial product image and the industrial product reference image respectively, and output features of the to-be-detected industrial product image and features of the industrial product reference image; Reshaping the image features of the industrial product to be detected and the industrial product reference image features respectively to generate product detection image reshaping features and product reference image reshaping features; Perform spatial multiplication on the reference mask corresponding to the industrial product reference image and the feature of the industrial product reference image to output defect features.

5. The method for unsupervised semantic segmentation of product defects based on a pre-trained model according to claim 3, characterized in that: The Hungarian algorithm is used to output a plurality of positive point prompt feature coordinates and a plurality of reverse point prompt feature coordinates according to the defect feature, the product inspection image reconstruction feature and the product reference image reconstruction feature, including: Performing matrix multiplication on the defect feature and the product inspection image reconstruction feature to generate a positive correlation feature matrix; Using the Hungarian algorithm, according to the forward correlation feature matrix, output a plurality of forward point prompt feature coordinates; Selecting positive features corresponding to the positive point prompt feature coordinates from the product inspection image reconstruction features, and performing matrix multiplication using the positive features and the product reference image reconstruction features to output a reverse correlation feature matrix; A Hungarian algorithm is used to generate a plurality of reverse point prompt feature coordinates according to the reverse correlation feature matrix.

6. The method for unsupervised semantic segmentation of product defects based on a pre-trained model according to claim 3, characterized in that: The method of clustering the plurality of forward point prompt feature coordinates and the plurality of reverse point prompt feature coordinates based on the mean value clustering algorithm to determine a target background point prompt coordinate set and a target foreground point prompt coordinate set includes: Taking the forward point prompt feature coordinates associated with any reverse point prompt feature coordinates not in the reference mask corresponding to the industrial product reference image as background point prompt coordinates, and taking the forward point prompt feature coordinates associated with any reverse point prompt feature coordinates in the reference mask as foreground point prompt coordinates; Using a plurality of the background point prompt coordinates, constructing a background point prompt coordinate set; Using a plurality of the foreground point prompt coordinates to construct a foreground point prompt coordinate set; Based on multiple preset clustering numbers, clustering the foreground point prompt coordinate set using a mean clustering algorithm, and outputting cluster center coordinates corresponding to each of the preset clustering numbers and multiple sample coordinate points associated with the cluster center coordinates corresponding to each of the preset clustering numbers; Calculating the silhouette coefficient corresponding to each of the preset cluster numbers according to a plurality of sample coordinate points associated with the cluster center coordinates corresponding to each of the preset cluster numbers; Selecting the preset cluster number corresponding to the largest silhouette coefficient as the target cluster number, and using the cluster center coordinates corresponding to the target cluster number to construct a target foreground point prompt coordinate set; A preset random extraction function is used to determine a target background point prompt coordinate set according to the background point prompt coordinate set.

7. The method for unsupervised semantic segmentation of product defects based on a pre-trained model according to claim 3, characterized in that: The method uses a segmentation network point prompt encoder and a segmentation network image decoder to generate a target defect segmentation mask according to the industrial product image to be detected, the target background point prompt coordinate set and the target foreground point prompt coordinate set, and generates a target cropped image based on the target defect segmentation mask, including: splicing the target background point prompt coordinate set and the target foreground point prompt coordinate set, and outputting a spliced ​​point prompt coordinate set; Using the splicing point cue coordinate set as the input of the segmentation network point cue encoder, and outputting point cue features; Decoding the point hint features using a segmentation network image decoder and outputting an initial mask; Using the initial mask as input to a mask proposal generator, and outputting a target defect segmentation mask; The target defect segmentation mask is used to crop the image of the industrial product to be inspected to generate a target cropped image.

8. The method for unsupervised semantic segmentation of product defects based on a pre-trained model according to claim 1, characterized in that: The preset defect classification network based on the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module includes a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module; the preset defect classification network based on the self-learning defect category prompt module and the defect image-text cross-modal feature interaction module extracts image text features according to the target cropped image and multiple defect categories to be matched, and outputs image features corresponding to the target cropped image and text features corresponding to each defect category to be matched, including: Using a self-learning defect category prompt module to determine multiple target learnable prompt features according to each defect category to be matched; Generate text features corresponding to each of the target learnable prompt features and image features corresponding to the target cropped image according to each of the target learnable prompt features and the target cropped image through a defect image-text cross-modal feature interaction module; The self-learning defect category prompt module is used to determine a plurality of target learnable prompt features according to each defect category to be matched, including: Initializing each of the defect categories to be matched, and generating a defect text prompt short sentence corresponding to each of the defect categories to be matched; Using a preset character dictionary to convert each of the defect text prompt short sentences respectively, and outputting a word index vector corresponding to each of the defect text prompt short sentences; Mapping each of the word index vectors, and outputting a continuous word embedding vector corresponding to each of the word index vectors; Based on each of the continuous word embedding vectors, determining a target learnable prompt feature corresponding to each of the continuous word embedding vectors; The defect image-text cross-modal feature interaction module includes a text feature encoder and an image feature encoder; the defect image-text cross-modal feature interaction module generates text features corresponding to each target learnable prompt feature and image features corresponding to the target cropped image according to each target learnable prompt feature and the target cropped image, including: Using each of the target learnable prompt features as the input of a text feature encoder, and outputting text features corresponding to each of the target learnable prompt features; The target cropped image is used as an input of an image feature encoder, and image features are output.

9. A device for unsupervised semantic segmentation of product defects based on a pre-trained model, characterized in that: include: An acquisition module, used to acquire an industrial product image to be detected, an industrial product reference image corresponding to the industrial product image to be detected, and a plurality of defect categories to be matched; An input module is used to input the industrial product image to be detected, the industrial product reference image and the multiple defect categories to be matched into a preset unsupervised segmentation network framework; the preset unsupervised segmentation network framework is composed of a preset segmentation network based on a self-learning point generation prompter of defect latent space features and a preset defect classification network based on a self-learning defect category prompt module and a defect image-text cross-modal feature interaction module; An adopting module is used to use a preset segmentation network of a self-learning point generation prompter based on defect latent space features to generate a target defect segmentation mask according to the industrial product image to be detected and the industrial product reference image, and to generate a target cropped image based on the target defect segmentation mask; An output module, configured to extract image and text features according to the target cropped image and the plurality of defect categories to be matched, using a preset defect classification network based on a self-learning defect category prompting module and a defect image-text cross-modal feature interaction module, and output image features corresponding to the target cropped image and text features corresponding to each defect category to be matched; A calculation module, used to calculate the matching degree between the image feature and each of the text features, and output the matching degree corresponding to each of the text features; The selection module is used to select the defect category associated with the text feature corresponding to the maximum matching degree as the defect classification result corresponding to the industrial product image to be detected.

10. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a method for unsupervised semantic segmentation of product defects based on a pre-trained model as described in any one of claims 1 to 6.

Images