An industrial defect new class discovery method based on mask guided contrastive learning

Abstract

The present application relates to the technical field of computer vision, and provides an industrial defect new class discovery method based on mask guided contrast learning.The method comprises the following steps: preprocessing a to-be-classified unlabeled data set and an auxiliary labeled data set to obtain corresponding masks; using mask guided contrast learning; and weighting and merging subgraph predictions based on the area of the defect area in the mask to obtain a classification result of the original image.The present application can effectively guide the model to focus on the corresponding defect area by using the mask based on mask guided contrast learning, and then improve the network model's ability to distinguish different types of defects through contrast learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an industrial defect new class discovery method based on mask guided contrastive learning. BACKGROUND

[0002] There are many factors affecting the industrial production process, so even the same batch of products will have diversified defects induced by different reasons, such as cracks or dents on the product surface due to collision, or product contamination due to production equipment failure and aging of foreign matter. Detecting and repairing these defects is an important part of the industrial production process. The difficulty in completing this task is that traditional anomaly detection algorithms can only detect whether there is a defect in the product or the location of the defect, and cannot further classify the product containing the defect. Therefore, the task of industrial defect new class discovery is proposed, which aims to classify the product samples under the condition of lacking product defect class labels, i.e. dividing industrial products into normal products and samples containing defects, and dividing the defects in industrial products containing defects into different types.

[0003] The existing industrial defect new class discovery method is as follows: locate the defect position, use a pre-trained network model to extract the image features of the defect position, and finally apply an unsupervised clustering algorithm (such as K-means, hierarchical clustering, Gaussian mixture model) to complete the classification of the classes. However, the challenge of this approach is that the same type of defect pattern sometimes also has large variations, such as shape, color texture, and position, which have high diversity and randomness, making the output features of the pre-trained network model insufficient to distinguish different types of defects.

[0004] Currently, the new class discovery task is studied more in the natural scene field, and its main idea is to introduce a class-labeled dataset (with no intersection with the class-labeled dataset to be classified) to assist the classification of the unlabeled dataset. Training the network with a class-labeled dataset can provide the network with certain semantic classification standards, thereby improving the feature separability of different classes in the unlabeled dataset, which provides a new solution for the industrial defect new class discovery task. However, directly applying existing new class discovery algorithms in the natural scene field does not yield good results, mainly due to the following difficulties: firstly, there are significant differences between natural scene images and industrial scene images (natural scene images are often in the middle of the image and occupy a large area of the image, while industrial scene images have defects randomly located in the image and often occupy a small area of the image), and secondly, the semantic nature of defects in industrial scenes is lower than that of objects and objects in natural scenes, making it difficult for the network model to learn to focus on the defect area for classification.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for discovering new categories of industrial defects based on mask-guided contrastive learning.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for discovering new categories of industrial defects based on mask-guided contrastive learning, comprising:

[0009] Unlabeled datasets to be classified and auxiliary labeled datasets Preprocessing is performed to obtain the corresponding mask; among which, and These are the unlabeled image to be classified and the auxiliary labeled data image, respectively. and Images Category labels and defect annotation masks, N l and N u These represent the number of images in the labeled dataset and the unlabeled dataset, respectively.

[0010] Use masks to guide comparative learning;

[0011] The sub-image predictions are weighted and merged based on the area of ​​the defect region in the mask to obtain the classification result of the original image.

[0012] Preferably, the unlabeled dataset to be classified and auxiliary labeled datasets Preprocessing is performed, specifically including:

[0013] Using an anomaly detection model to Make a prediction and obtain the prediction result {A} i |i∈[1,N u ]}, where A i For image The anomaly score map corresponding to the prediction;

[0014] For all outlier score graphs {A i |i∈[1,N u Binarization is performed to obtain The mask {M i |i∈[1,N u ]};

[0015] Calculate mask M i The bounding box corresponding to each connected component in the image is determined based on the bounding box pair of the mask M.i and images Perform trimming to obtain a trimming sub-image centered on the defect. and cutting mask

[0016] For the auxiliary labeled dataset, the sub-images are cropped using a real defect annotation mask in the manner described above, resulting in the cropped sub-images. and the cut mask

[0017] Preferably, the above applies to all anomaly score graphs {A} i |i∈[1,N u Binarization is performed to obtain The mask {M i |i∈[1,N u Specifically, it includes:

[0018] Calculate {A i |i∈[1,N u The outlier score {s} for each outlier score plot in} i |i∈[1,N u ]}, where s i For outlier score graph A i Pixel value A i The maximum value of (p) is further calculated to obtain The search space for setting the threshold is [s]. min ,s max ];

[0019] From the threshold interval [s] min ,s max ]Uniformly sample 64 threshold values ​​to obtain {ε j |j∈[1,64]}, using the threshold ε j For A i Binarization is performed, and values ​​greater than or equal to the threshold ε are included. j The pixel value is set to 1, which is less than the threshold ε. j The pixel value is set to 0 to obtain a binary image.

[0020] right After performing an image erosion operation, count the number of connected components in the resulting erosion graph.

[0021] Incrementally traversing all thresholds yields a sequence of connected component counts. Count the frequency of all non-zero connected components, and use the mode of the non-zero connected component counts. As A i Estimate of the number of defective areas in the data;

[0022] Find the estimated number of connected components The corresponding longest stable threshold interval [ε p ,ε q If the length of the interval is less than 4, i.e., q-p+1<4, then the threshold estimate is taken as 1; otherwise, the left endpoint ε of the longest threshold interval is taken. p As a threshold estimate;

[0023] Using threshold estimate ε p For A i Binarization is performed to obtain the image. The corresponding mask M i For all abnormal score graphs {A i |i∈[1,N u Repeat the above steps to obtain The mask {M i |i∈[1,N u ]}.

[0024] Preferably, the mask-guided contrastive learning specifically includes:

[0025] Subgraphs of unlabeled images and sub-images with labels Perform data augmentation to generate two randomly augmented views. and The corresponding binarized mask is enhanced using the same random parameters;

[0026] Prepare a visual Transformer network pre-trained on natural scene images as a feature extractor f, and employ a linear classifier. Following the feature extractor f, the labels of the input subgraph are obtained;

[0027] Contrastive learning is used to enable network models to learn more discriminative representations of defects, and it is divided into supervised contrastive loss and self-supervised contrastive loss.

[0028] We use a label-supervised learning approach to learn classification tasks.

[0029] Preferably, the sub-image of the unlabeled image and sub-images with labels Perform data augmentation to generate two randomly augmented views. and The corresponding binarized mask is enhanced using the same random parameters, specifically including:

[0030] For the cropped sub-image x i Use data augmentation strategies from BYOP and RandAug to generate two randomly augmented views. and The corresponding binarized mask is enhanced using the same random parameters.

[0031] Preferably, preparing a visual Transformer network pre-trained in a natural scene image as a feature extractor f specifically includes:

[0032] The ViT-Base8 model, which is pre-trained on the ImageNet dataset using the DINO method, has 12 layers, a local image region size of 8×8, an input image size of 224×224, and a number of local image regions N of 784.

[0033] For the cropped and enhanced mask image, average pooling is used and it is flattened to... Insert constant 1 into The first element is selected to ensure that the global features of the image retain the knowledge learned from the previous layer, and the transformation is performed using the following formula:

[0034]

[0035] Using the transformed mask image, the self-attention mechanism of the r-th layer is adjusted as follows:

[0036]

[0037] Where X r This represents the output feature of the r-th layer of ViT. They represent X respectively r-1 After linear projection φ q ,φ k ,φ v The characteristics after, and The self-attention mechanism of the last 9 layers of the ViT-Base8 model was adjusted according to the formula.

[0038] Preferably, a linear classifier is used. Following the feature extractor f, the labels of the input subgraphs are obtained, specifically including:

[0039] For the two randomly augmented views input and Its output is and

[0040] Simultaneously, a three-layer multilayer perceptron is used as the projection layer g for two randomly augmented input views. and Its output is and

[0041] right sub-images in The labels of the two randomly augmented views are the real labels;

[0042] for sub-images in Pseudo-labels are generated using the output of a linear classifier; the network outputs of the two augmented views are respectively... and Before Set it to negative infinity, then input the temperature coefficient τ. t The softmax function is used to obtain the output of the pseudo-labels. and During training, τ t The value gradually changes from an initial value of 0.07 to 0.04;

[0043] Next, the outlier scores of the subgraphs are used to correct the false labels, using the following formula:

[0044] in It is a length of The zero vector, w i =max(0.5-s) i ,0), w i The weight of the normal label, s i It is a cropped sub-image x i The maximum abnormal score.

[0045] Preferably, the method of using contrastive learning to enable the network model to learn more discriminative defect representations is divided into supervised contrastive loss and self-supervised contrastive loss, specifically including:

[0046] The formula for self-supervised contrastive loss is:

[0047] Where B represents the data in the same batch, and |B| represents the number of images in the batch. For the same batch and image I i Different image indexes, τ u The value is 0.07;

[0048] The formula for supervised contrast loss is:

[0049] Among them B l The labeled data is in B. It is in the same batch as I i Image index with the same label, τ c The value is 1.

[0050] Preferably, the label-supervised learning of the classification task specifically includes:

[0051] For each pair of images in the batch and Using a student network to achieve a smoother temperature coefficient τ s Generate predictions and Where τ s The value is 0.1; for unlabeled data, pseudo-labels are generated using a teacher network, and for labeled images, the real labels are used. and

[0052] The loss function for classification tasks is divided into classification loss for labeled images and classification loss for unlabeled images. The specific formula is as follows: Where L CE For the standard cross-entropy loss function, B u The data in B is unlabeled. A regularization term is added at the end, and the average predicted entropy of samples from the same batch is used as the loss function. The formula is as follows: in

[0053] The overall loss function is Where λ and μ are the weights of the loss function, with values ​​of 0.3 and 4 respectively.

[0054] Preferably, the weighted merging of sub-image predictions based on the area of ​​the defect region in the mask to obtain the classification result of the original image specifically includes:

[0055] For unlabeled images Sub-image obtained by cropping The corresponding prediction result is in It represents the number of connected components and also indicates the image. The number of cropped sub-images. Calculate the defect area of ​​each sub-image based on the corresponding mask. For the k-th subgraph Its prediction results Corresponding weights and final image Corresponding prediction results The calculation formula is:

[0056] Where τ α This is the temperature coefficient, with a value of 100.

[0057] Secondly, the present invention also provides a method for discovering new categories of industrial defects based on mask-guided contrastive learning, for implementing the method for discovering new categories of industrial defects based on mask-guided contrastive learning described in the first aspect, wherein the apparatus includes:

[0058] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the mask-guided contrastive learning-based method for discovering new categories of industrial defects as described in the first aspect.

[0059] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect.

[0060] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in memory, performing the method as described in the first aspect.

[0061] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the method as described in the first aspect.

[0062] This invention is based on mask-guided contrastive learning, which effectively guides the model to focus on corresponding defect regions using masks, thereby improving the network model's ability to distinguish different types of defects through contrastive learning. This invention also proposes a novel mask-guided contrastive learning method, which effectively guides the model to focus on corresponding defect regions using masks, thereby improving the network model's ability to distinguish different types of defects through contrastive learning. Furthermore, it proposes a novel method for merging the prediction results of cropped sub-images. This method weights and sums the prediction results of different cropped sub-images based on the defect area to obtain the prediction result of the original image, achieving results superior to traditional averaging or extreme value methods. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0064] Figure 1 This is a flowchart illustrating a method for discovering new categories of industrial defects based on mask-guided contrastive learning, provided in an embodiment of the present invention.

[0065] Figure 2 This is a flowchart illustrating a method for discovering new categories of industrial defects based on mask-guided contrastive learning, provided in an embodiment of the present invention.

[0066] Figure 3 This is a flowchart illustrating a method for discovering new categories of industrial defects based on mask-guided contrastive learning, provided in an embodiment of the present invention.

[0067] Figure 4 This is a flowchart illustrating a method for discovering new categories of industrial defects based on mask-guided contrastive learning, provided in an embodiment of the present invention.

[0068] Figure 5 This is a schematic diagram of a method for discovering new categories of industrial defects based on mask-guided contrastive learning, provided by an embodiment of the present invention.

[0069] Figure 6 This is a schematic diagram of a device for discovering new categories of industrial defects based on mask-guided contrastive learning, provided in an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0071] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0072] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0073] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0074] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0075] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0076] Example 1:

[0077] Embodiment 1 of this invention provides a method for discovering new categories of industrial defects based on mask-guided contrastive learning, such as... Figure 1 As shown, it includes:

[0078] In step 201, the unlabeled dataset to be classified and auxiliary labeled datasets Preprocessing is performed to obtain the corresponding mask; in this embodiment, an unlabeled dataset to be classified is given in advance. (corresponding category set C) u ) and auxiliary labeled datasets (corresponding category set C) l ),in and These are the unlabeled image to be classified and the auxiliary labeled data image, respectively. and Images Category labels and defect annotation masks, N l and Nu Represent the number of images in the labeled dataset and the unlabeled dataset, respectively, and the set of categories C. u and C l No overlap. The goal of the method described in this embodiment is to transform unlabeled datasets... According to set C u The categories are classified according to the data.

[0079] In step 202, mask-guided contrastive learning is used.

[0080] In step 203, the sub-image predictions are weighted and merged based on the area of ​​the defect region in the mask to obtain the classification result of the original image.

[0081] This embodiment is based on mask-guided contrastive learning, which can use masks to effectively guide the model to focus on the corresponding defect areas, and then improve the network model's ability to distinguish different types of defects through contrastive learning.

[0082] In specific application scenarios, the unlabeled dataset to be classified and auxiliary labeled datasets Preprocessing, such as Figure 2 As shown, it specifically includes:

[0083] In step 301, an anomaly detection model is used to... Make a prediction and obtain the prediction result {A} i |i∈[1,N u ]}, where A i For image The anomaly score map is obtained corresponding to the prediction; the anomaly detection model can be a MuSc model, an EfficientAD model, or a CPR model, etc.

[0084] In step 302, for all anomaly score graphs {A} i |i∈[1,N u Binarization is performed to obtain The mask {M i |i∈[1,N u ]}.

[0085] In step 303, the mask M is calculated. i The bounding box corresponding to each connected component in the image is determined based on the bounding box pair of the mask M. i and images Perform trimming to obtain a trimming sub-image centered on the defect. and cutting mask

[0086] In step 304, for the auxiliary labeled dataset (i.e., the auxiliary labeled dataset mentioned above), the sub-image is cropped using a real defect labeling mask in the manner described above (i.e., step 303) to obtain the cropped sub-image. and the cut mask

[0087] In this embodiment, a novel method for binarizing anomaly score maps is also provided for step 302 above, namely, the binarization of all anomaly score maps {A} i |i∈[1,N u Binarization is performed to obtain The mask {M i |i∈[1,N u ]},like Figure 3 As shown, it specifically includes:

[0088] In step 401, calculate {A} i |i∈[1,N u The outlier score {s} for each outlier score plot in} i |i∈[1,N u ]}, where s i For outlier score graph A i Pixel value A i The maximum value of (p) is further calculated to obtain The search space for setting the threshold is [s]. min ,s max ].

[0089] In step 402, from the threshold interval [s] min ,s max ]Uniformly sample 64 threshold values ​​to obtain {ε j |j∈[1,64]}, using the threshold ε j For A i Binarization is performed, and values ​​greater than or equal to the threshold ε are included. j The pixel value is set to 1, which is less than the threshold ε. j The pixel value is set to 0 to obtain a binary image.

[0090] In step 403, for After performing an image erosion operation (6×6 kernel size), the number of connected components in the resulting erosion graph is counted.

[0091] In step 404, the sequence of connected component counts is obtained by incrementally traversing all thresholds. Count the frequency of all non-zero connected components, and use the mode of the non-zero connected component counts. As A i The estimated number of defective areas in the sample.

[0092] In step 405, the estimated number of connected components is found. The corresponding longest stable threshold interval [ε p ,ε q (Where "stable" means that the number of connected components in the binarization result corresponding to any threshold within the threshold interval is a constant value.) If the length of the interval is less than 4 (i.e., q-p+1<4), then the threshold estimate is 1; otherwise, the left endpoint ε of the longest threshold interval is taken. p As a threshold estimate.

[0093] In step 406, the threshold estimate ε is used. p For A i Binarization is performed to obtain the image. The corresponding mask M i For all abnormal score graphs {A i |i∈[1,N u Repeat the above steps to obtain The mask {M i |i∈[1,N u ]}.

[0094] The novel anomaly fraction image binarization method proposed in this embodiment can adaptively obtain the binarization threshold of each anomaly fraction image, giving the approximate location of defects in the original image, and has a strong ability to suppress false positives and false negatives of defects.

[0095] In some embodiments, the use of mask-guided contrastive learning, such as Figure 4 As shown, it specifically includes:

[0096] In step 501, the sub-image of the unlabeled image is... and sub-images with labels Perform data augmentation to generate two randomly augmented views. and The corresponding binarized mask is enhanced using the same random parameters.

[0097] In step 502, a visual Transformer (ViT) network pre-trained in natural scene images is prepared as a feature extractor f, and a linear classifier is used. Following the feature extractor f, the labels of the input subgraph are obtained.

[0098] In step 503, contrastive learning is used to enable the network model to learn more discriminative representations of defects, which is divided into supervised contrastive loss and self-supervised contrastive loss.

[0099] In step 504, a label-supervised learning method is used to learn the classification task.

[0100] Among them, the sub-image of the unlabeled image and sub-images with labels Perform data augmentation to generate two randomly augmented views. and The corresponding binarized mask is enhanced using the same random parameters, specifically including: enhancing the cropped sub-image x. i Two randomly augmented views are generated using data augmentation strategies from BYOP (which includes random cropping, random flipping, color dithering, and Gaussian blur) and RandAug (which includes random rotation, random bit depth, and random sharpness). and The corresponding binarized mask is enhanced using the same random parameters (including random cropping, random flipping, and random rotation).

[0101] In practical applications, preparing a pre-trained visual Transformer network in a natural scene image as a feature extractor f can employ any existing ViT network architecture and pre-trained network parameters. In an optional implementation, preparing a pre-trained visual Transformer network in a natural scene image as a feature extractor f specifically includes:

[0102] The ViT-Base8 model, pre-trained on the ImageNet dataset using the DINO method, has 12 layers, an 8×8 patch size, an input image size of 224×224, and 784 local image patches.

[0103] For the cropped and enhanced mask image, average pooling is used and it is flattened to... Insert constant 1 into The first element is selected to ensure that the global features of the image retain the knowledge learned from the previous layer, and the transformation is performed using the following formula:

[0104]

[0105] Using the transformed mask image, the self-attention mechanism of the r-th layer is adjusted as follows:

[0106]

[0107] Where X r This represents the output feature of the r-th layer of ViT. They represent X respectively r-1 After linear projection φ q ,φ k ,φv The characteristics after, and The self-attention mechanism of the last 9 layers of the ViT-Base8 model was adjusted according to the formula.

[0108] In a preferred embodiment, a linear classifier is employed. Following the feature extractor f, the labels of the input subgraphs are obtained, specifically including:

[0109] For the two randomly augmented views input and Its output is and

[0110] Simultaneously, a three-layer multilayer perceptron is used as the projection layer g for two randomly augmented input views. and Its output is and

[0111] right sub-images in The labels of the two randomly augmented views are the real labels.

[0112] for sub-images in Pseudo-labels are generated using the output of a linear classifier; the network outputs of the two augmented views are respectively... and Before Set it to negative infinity, then input the temperature coefficient τ. t The softmax function is used to obtain the output of the pseudo-labels. and During training, τ t It gradually changes from an initial value of 0.07 to 0.04.

[0113] Next, the sub-image (i.e., the defect-centered trimming sub-image from step 303 above) is used. The formula for correcting pseudo-labels for outlier scores is:

[0114] in It is a length of The zero vector, w i =max(0.5-s) i ,0), w i The weight of the normal label, s i It is a cropped sub-image x i The maximum abnormal score.

[0115] In one optional implementation, the use of contrastive learning to enable the network model to learn more discriminative defect representations is divided into supervised contrastive loss and self-supervised contrastive loss, specifically including:

[0116] The formula for self-supervised contrastive loss is:

[0117] Where B represents the data in the same batch, and |B| represents the number of images in the batch. For the same batch and image I i Different image indexes, τ u The value is 0.07.

[0118] The formula for supervised contrast loss is:

[0119] Among them B l The labeled data is in B. It is in the same batch as I i Image index with the same label, τ c The value is 1.

[0120] The label-supervised learning of the classification task specifically includes:

[0121] For each pair of images in the batch and Using a student network to achieve a smoother temperature coefficient τ s Generate predictions and Where τ s The value is 0.1; for unlabeled data, pseudo-labels are generated using a teacher network, and for labeled images, the real labels are used. and Among them, the use of a teacher network to generate pseudo-labels and the use of a linear classifier are mentioned. Following the feature extractor f, the labeling of the input subgraph is obtained based on the same concept.

[0122] The loss function for classification tasks is divided into classification loss for labeled images and classification loss for unlabeled images. The specific formula is as follows: Where L CE For the standard cross-entropy loss function, B u The data in B is unlabeled. A regularization term is added at the end, and the average predicted entropy of samples from the same batch is used as the loss function. The formula is as follows: in

[0123] The overall loss function is Where λ and μ are the weights of the loss function, with values ​​of 0.3 and 4 respectively.

[0124] The sub-image predictions based on the area of ​​the defect region in the mask are weighted and merged to obtain the classification result of the original image, specifically including:

[0125] For unlabeled images Sub-image obtained by cropping The corresponding prediction result is in It represents the number of connected components and also indicates the image. The number of cropped sub-images. Calculate the defect area (number of pixels) of each sub-image based on the corresponding mask. For the k-th subgraph Its prediction results Corresponding weights and final image Corresponding prediction results The calculation formula is:

[0126] Where τ α This is the temperature coefficient, with a value of 100.

[0127] Example 2:

[0128] Based on the method described in Embodiment 1, this invention combines specific application scenarios and uses technical descriptions in relevant scenarios to illustrate the implementation process of the features of this invention in those scenarios.

[0129] This invention provides a novel industrial defect category discovery method based on mask-guided contrastive learning. This method utilizes anomaly scores output by an anomaly detection model. Figure Two The valued mask is used to provide defect location information for the unlabeled image to be classified, thereby further guiding contrastive learning to enable the network model to better distinguish the features of different types of defect images, achieving a more accurate discovery of new categories of industrial defects.

[0130] First, we define the dataset used in this task to better explain the methodology. This task provides an unlabeled dataset to be classified. (corresponding category set C) u ) and auxiliary labeled datasets (corresponding category set C) l ),in and These are the unlabeled image to be classified and the auxiliary labeled data image, respectively. and Images Category labels and defect annotation masks, N l and N uRepresent the number of images in the labeled dataset and the unlabeled dataset, respectively, and the set of categories C. u and C l There is no overlap. The goal of the task is to integrate the unlabeled dataset. According to set C u The categories are classified according to the data.

[0131] To achieve the above objectives, the present invention provides the following technical solutions, such as... Figure 5 As shown, it includes the following steps:

[0132] Step S1, the unlabeled dataset to be classified and auxiliary labeled datasets Preprocessing is performed, and the specific operations include:

[0133] Step S1-1: Prepare an anomaly detection model (such as MuSc, EfficientAD, CPR, etc.) for testing. Make a prediction and obtain the prediction result {A} i |i∈[1,N u ]}, where A i For image The corresponding anomaly score map obtained from the prediction.

[0134] Step S1-2, calculate {A} i |i∈[1,N u The outlier score {s} for each outlier score plot in} i |i∈[1,N u ]}, where s i For outlier score graph A i Pixel value A i The maximum value of (p) is further calculated to obtain The search space for setting the threshold is [s]. min ,s max ].

[0135] This invention proposes a novel adaptive threshold binarization method, the specific operation steps of which are as follows:

[0136] From the threshold interval [s] min ,s max ]Uniformly sample 64 threshold values ​​to obtain {ε j |j∈[1,64]}, using the threshold ε j For A i Binarization is performed, and values ​​greater than or equal to the threshold ε are included. j The pixel value is set to 1, which is less than the threshold ε. j The pixel value is set to 0 to obtain a binary image. right After performing an image erosion operation (6×6 kernel size), the number of connected components in the resulting erosion graph is counted. Incrementally traversing all thresholds yields a sequence of connected component counts. Count the frequency of all non-zero connected components, and use the mode of the non-zero connected component counts. As A i The estimated number of defective regions is obtained. Next, the estimated number of connected components is found. The corresponding longest stable threshold interval [ε p ,ε q (Where "stable" means that the number of connected components in the binarization result corresponding to any threshold within the threshold interval is a constant value.) If the length of the interval is less than 4 (i.e., q-p+1<4), then the threshold estimate is taken as 1; otherwise, the left endpoint ε of the longest threshold interval is taken. p This is used as a threshold estimate. Finally, the threshold estimate ε is used... p For A i Binarization is performed to obtain the image. The corresponding mask M i For all abnormal score graphs {A i |i∈[1,N u Repeat the above steps to obtain The mask {M i |i∈[1,N u ]}.

[0137] Steps S1-3: Calculate the mask M i The bounding box corresponding to each connected component in the image is determined based on the bounding box pair of the mask M. i and images Perform trimming to obtain a trimming sub-image centered on the defect. and cutting mask

[0138] Steps S1-4: For the auxiliary labeled dataset, use a real defect annotation mask to crop the sub-images according to the method described in steps S1-3 above, to obtain the cropped sub-images. and the cut mask

[0139] Step S2, using a mask to guide contrastive learning, specifically includes:

[0140] Step S2-1, here we are working on the sub-image of the unlabeled image. and sub-images with labels Using the same operation, and for simplicity, the following uses cropped sub-images x. i Describe the cropped sub-image x. iTwo randomly augmented views are generated using data augmentation strategies from BYOP (which includes random cropping, random flipping, color dithering, and Gaussian blur) and RandAug (which includes random rotation, random bit depth, and random sharpness). and The corresponding binarized mask is enhanced using the same random parameters (including random cropping, random flipping, and random rotation). Next, a Visual Transformer (ViT) network pre-trained on a natural scene image is prepared as the feature extractor f. Any existing ViT network architecture and pre-trained network parameters can be used. In this invention, the ViT-Base8 model pre-trained on the ImageNet dataset using the DINO method is used. It has 12 layers, a local image region (patch) size of 8×8, an input image size of 224×224, and the number of local image regions N is 784. For the cropped and enhanced mask image, average pooling is used and it is flattened to... Insert constant 1 into The first element is selected to ensure that the global image features (class tokens) retain the knowledge learned from the previous layer, and the transformation is performed using the following formula:

[0141]

[0142] Using the transformed mask image, the self-attention mechanism of the r-th layer is adjusted as follows:

[0143]

[0144] Where X r This represents the output feature of the r-th layer of ViT. They represent X respectively r-1 After linear projection φ q ,φ k ,φ v The characteristics after, and The self-attention mechanism of the last 9 layers of the ViT-Base8 model was adjusted according to the formula.

[0145] Step S2-2, using a linear classifier Following the feature extractor f, the labels of the input subgraph are obtained. For the two randomly augmented views of the input... and Its output is and Simultaneously, a three-layer multilayer perceptron is used as the projection layer g for two randomly augmented input views. and Its output is and right sub-images in The labels of the two randomly augmented views are the real labels. For sub-images in Pseudo-labels are generated using the output of a linear classifier; the network outputs of the two augmented views are respectively... and Before Set it to negative infinity, then input the temperature coefficient τ. t The softmax function is used to obtain the output of the pseudo-labels. and During training, τ t The value is gradually changed from an initial value of 0.07 to 0.04. Then, the outlier scores of the subplots from steps S1-3 are used to correct the false labels, as shown in the following formula:

[0146]

[0147] in It is a length of The zero vector, w i =max(0.5-s) i ,0), w i The weight of the normal label, s i It is a cropped sub-image x i The maximum abnormal score.

[0148] Step S2-3 employs contrastive learning to enable the network model to learn more discriminative defect representations. This involves supervised contrastive loss and self-supervised contrastive loss. The formula for self-supervised contrastive loss is as follows:

[0149]

[0150] Where B represents the data in the same batch, and |B| represents the number of images in the batch. For the same batch and image I i Different image indexes, τ u The value is 0.07.

[0151] The formula for supervised comparison loss is as follows:

[0152]

[0153] Among them B l The labeled data is in B. It is in the same batch as I i Image index with the same label, τ c The value is 1.

[0154] Steps S2-4 involve learning the classification task using label-supervised learning. For each pair of images in the batch... and Using a student network to achieve a smoother temperature coefficient τ s Generate predictions and Where τ s The value is 0.1. For unlabeled data, pseudo-labels are generated using the teacher network according to step S2-2; for labeled images, the real labels are used. and

[0155] The loss function for classification tasks is divided into classification loss for labeled images and classification loss for unlabeled images. The specific formulas are as follows:

[0156]

[0157] Where L CE For the standard cross-entropy loss function, B u The data in B is unlabeled. A regularization term is added at the end, and the average predicted entropy of samples from the same batch is used as the loss function, as shown in the following formula:

[0158]

[0159] in

[0160] The overall loss function is as follows:

[0161]

[0162] Where λ and μ are the weights of the loss function, with values ​​of 0.3 and 4 respectively.

[0163] Step S3 involves weighted merging of sub-image predictions based on the area of ​​the defect region in the mask to obtain the classification result of the original image. The specific steps are as follows:

[0164] For unlabeled images Sub-image obtained by cropping The corresponding prediction result is in It represents the number of connected components and also indicates the image. The number of cropped sub-images. Calculate the defect area (number of pixels) of each sub-image based on the corresponding mask. For the k-th subgraph Its prediction results Corresponding weights and final image Corresponding prediction results The calculation formula is as follows:

[0165]

[0166] Where τ α This is the temperature coefficient, with a value of 100.

[0167] In summary, the technical solutions conceived by this invention, compared with existing methods, have the following advantages:

[0168] Beneficial effects:

[0169] (1) This invention proposes a novel binarization method for anomaly score images, which can adaptively obtain the binarization threshold of each anomaly score image, give the approximate location of defects in the original image, and has a strong ability to suppress false detections and missed detections of defects.

[0170] (2) This invention proposes a novel mask-guided contrastive learning method, which can use a mask to effectively guide the model to focus on the corresponding defect area, thereby improving the network model's ability to distinguish different types of defects through contrastive learning.

[0171] (3) This invention proposes a novel method for merging the prediction results of cropped sub-images. This method can obtain the prediction result of the original image by weighted summation of the prediction results of different cropped sub-images based on the defect area. The result is better than the traditional method of taking the average or the maximum value.

[0172] Example 3:

[0173] like Figure 6 The diagram shown is an architectural schematic of an industrial defect new category discovery device based on mask-guided contrastive learning according to an embodiment of the present invention. This embodiment of the industrial defect new category discovery device based on mask-guided contrastive learning includes one or more processors 21 and a memory 22. Figure 6 Take a processor 21 as an example.

[0174] Processor 21 and memory 22 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0175] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the mask-guided contrastive learning-based method for discovering new categories of industrial defects in Embodiment 1. The processor 21 executes the mask-guided contrastive learning-based method for discovering new categories of industrial defects by running the non-volatile software programs and instructions stored in the memory 22.

[0176] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0177] The program instructions / modules are stored in the memory 22 and, when executed by one or more processors 21, execute the mask-guided contrastive learning-based method for discovering new categories of industrial defects in Embodiment 1.

[0178] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0179] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0180] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An industrial defect new class discovery method based on mask-guided contrastive learning, characterized in that, The method comprises a test flow and a training flow: The training process comprises: preprocessing a to-be-classified unlabeled data set and an auxiliary labeled data set to obtain corresponding masks; wherein, and are a to-be-classified unlabeled image and an auxiliary labeled data image respectively, and are a class label and a defect label mask of the image respectively, and N l and N u represent the number of images of the labeled data set and the unlabeled data set respectively. Includes: using anomaly detection models to Make a prediction and obtain the prediction result {A} i |i∈[1,N u ]}, where A i For image Corresponding to the predicted anomaly score map; for all anomaly score maps {A i |i∈[1,N u Binarization is performed to obtain The mask {M i |i∈[1,N u ]};Calculate mask M i The bounding box corresponding to each connected component in the image is determined based on the bounding box pair of the mask M. i and images Perform trimming to obtain a trimming sub-image centered on the defect. and cutting mask For the auxiliary labeled dataset, the sub-images are cropped using a real defect annotation mask in the manner described above, resulting in the cropped sub-images. and the cut mask Using mask guided contrastive learning; including: subgraphs of unlabelled images and subgraphs of labelled images Using the same operation; subgraphs of unlabelled images Data augmentation to generate randomly augmented views Subgraphs of labelled images Data augmentation to generate randomly augmented views Their corresponding binary masks are augmented using the same random parameters; prepare a visual transformer network pre-trained in natural scene images as a feature extractor f, using a linear classifier After the feature extractor f, get the label of the input subgraph; use contrastive learning to make the network model learn more discriminative representations of defects, divided into supervised contrastive loss and self-supervised contrastive loss; learn the classification task in the form of label supervision; The trained model is used to perform a test procedure, which includes: weighted merging of subgraph predictions based on the area of the defect region in the mask to obtain a classification result of the original image; and includes: for unannotated images The subgraph obtained by cropping The corresponding prediction result is Wherein is the number of connected domains, also representing the number of cropped subgraphs of the image . Calculate the defect area of each subgraph according to the corresponding mask For the kth subgraph Its prediction result The corresponding weight And the final image The corresponding prediction result The calculation formula is: where τ α is the temperature coefficient, which has a value of 100.

2. The industrial defect new class discovery method based on mask-guided contrastive learning according to claim 1, wherein, The binaryzation of all the abnormal score maps {A i |i∈[1,N u ]} obtains a mask {M |i∈[1,N u ]}. i ​ Compute {A i |i∈[1,N u ]} the anomaly score s i |i∈[1,N u ]} of each anomaly score map A i s i max(A i (p)) for each pixel p in A The search space for setting the threshold is [s min ,s max ]. 64 threshold values are uniformly sampled from the threshold interval [s min ,s max ] to obtain {ε j |j∈[1,64]}. The threshold value ε j is used to binarize A i , and the pixel value greater than or equal to the threshold value ε j is set to 1, and the pixel value less than the threshold value ε j is set to 0, to obtain a binary image To Statistical erosion result map connected domain number after image erosion operation Incrementally traverse all thresholds ε j Obtain the connected component number sequence Count the frequency of occurrence of all non-zero connected component numbers, and take the mode of the non-zero connected component numbers As A i The defect region number estimation value in Finding the connected component number estimate The corresponding longest stable threshold interval [ε p , εq], if the length of this interval is less than 4, i.e. q - p + 1 < 4, then take the threshold estimate to be 1; otherwise, take the left endpoint ε p of the longest threshold interval as the threshold estimate; Using threshold estimate ε p For A i Binarization is performed to obtain the image. The corresponding mask M i For all abnormal score graphs {A i |i∈[1,N u Repeat the above steps to obtain The mask {M i |i∈[1,N u ]}.

3. The industrial defect new class discovery method based on mask-guided contrastive learning according to claim 1, wherein, The pair of subgraphs of unannotated images and subgraphs of annotated images Data augmentation is performed to generate two randomly augmented views and The corresponding binary masks thereof are augmented using the same random parameters, including specifically: cropping subgraph x i Using the data augmentation strategies in BYOP and RandAug, two randomly augmented views are generated and Their corresponding binarized masks are augmented using the same random parameters.

4. The industrial defect new class discovery method based on mask-guided contrastive learning according to claim 1, wherein, The method comprises the following steps: A ViT-Base8 model pre-trained in an ImageNet dataset is used, which has 12 layers in total, a local image region size of 8*8, an input image size of 224*224, and a local image region number N of 784; For the cropped and enhanced mask image, average pooling is used and flattened into a constant 1 is inserted at the first position to enable the image global feature to retain the knowledge learned from the previous layer, and converted using the following formula: ​ The self-attention mechanism of the rth layer is adjusted to the following formula by using the converted mask image: where X r denotes the output feature of the r-th layer of ViT, denotes X r-1 the feature after linear projection φ q ,φ k ,φ v , and The self-attention mechanism of the last 9 layers of the ViT-Base8 model is adjusted according to the formula.

5. The industrial defect new class discovery method based on mask-guided contrastive learning according to claim 1, wherein, Said employing a linear classifier After the feature extractor f, the label of the input subgraph is obtained, specifically including: for the input two random augmented views and whose output is and At the same time, a multi-layer perceptron with three layers is used as the projection layer g for the input of two randomly enhanced views and whose output is and to sub-images in both random augmented views have labels that are true labels; The sub-images in The pseudo labels are generated by using the outputs of the linear classifier, and the network outputs of the two enhanced views are and The former is set to negative infinity, and then input to the softmax function with temperature τ t , to obtain the output of the pseudo label and During the training process, τ t is gradually changed from the initial value 0.07 to 0.04;​ Then the pseudo labels are corrected by the anomaly scores of sub-graphs, and the formula is: wherein is a zero vector of length , w i = max(0.5 - s i , 0), w i is the weight of the normal label, s i is the abnormal score maximum of the cropped subgraph x i .

6. The industrial defect new class discovery method based on mask-guided contrastive learning according to claim 1, wherein, The network model learns a more distinctive defect representation through contrast learning, which includes supervised contrast loss and self-supervised contrast loss, and specifically comprises the following steps: The formula of self-supervised contrastive loss is: where B is the data of the same batch, |B| is the number of images in the batch, is the image I i with a different image index, τ u has a value of 0.07; The formula of supervised contrastive loss is: where B l is the annotated data in B, is the image index in the same batch as I i τ c has a value of 1.

7. The industrial defect new class discovery method based on mask-guided contrastive learning according to claim 1, wherein, The classification task is learned in a label supervision manner, and specifically comprises the following steps: For each pair of images in a batch and Using the student network with a smoother temperature coefficient τ s Generating predictions and where the value of τ s is 0.1; for unlabelled data, using the teacher network to generate pseudo labels, for labelled images, using the true labels and The loss function of the classification task is divided into a classification loss of labeled images and a classification loss of unlabeled images, and the specific formula is: Where L CE is a standard cross-entropy loss function, B u is the unlabeled data in B, and finally a regularization term is added, and the entropy of the average prediction of the same batch of samples is used as the loss function, and the formula is Where The overall loss function is wherein λ and μ are the weights of the loss function, taking values of 0.3 and 4, respectively.

Images