Industrial defect new category discovery method based on mask guided contrast learning

By adopting mask-guided comparison learning method in the discovery of new industrial defect categories, and using anomaly detection model and data augmentation technology, the problem of insufficient distinction ability caused by industrial defect diversity is solved, and more accurate defect classification and distinction is achieved.

CN119027385AActive Publication Date: 2024-11-26HUAZHONG UNIV OF SCI & TECH

Patent Information

Application Number
CN202411093692.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-26
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The existing new category discovery method for industrial defects has difficulties in detecting and classifying industrial defects, especially the shape, color and position diversity of the same type of defect pattern leads to insufficient discrimination ability of pre-trained network models, and directly applies algorithms in the field of natural scenes. Not effective.

Method used

Using a mask-guided comparison learning method, the abnormality score graph is predicted through an abnormality detection model, and the mask is obtained by binarizing and cropping. Combining data enhancement and self-supervising comparison losses, the network's ability to distinguish defect areas is improved, and mask guidance is used. The model focuses on the defect area and weights the prediction results.

Benefits of technology

It effectively improves the ability of network models to distinguish different types of defects, improves the accuracy and robustness of discovering new categories of industrial defects, and is better than the prediction results of traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027385A_ABST
    Figure CN119027385A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computer vision, and provides an industrial defect new category discovery method based on mask guided contrast learning. The method comprises the following steps: preprocessing a to-be-classified unlabeled data set and an auxiliary labeled data set to obtain a corresponding mask; using a mask to guide contrast learning; and performing weighted merging on the sub-image prediction based on the area of the defect region in the mask to obtain a classification result of the original image. According to the invention, based on mask-guided comparative learning, the model can be effectively guided to pay attention to the corresponding defect area by using the mask, so that the capability of distinguishing different types of defects by the network model is improved through comparative learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method for discovering new categories of industrial defects based on mask-guided contrast learning. Background Art

[0002] There are many factors affecting the industrial production process, so that even the same batch of products may have various defects caused by different reasons, such as cracks or dents on the product surface due to collisions, or foreign matter contaminating the product due to failures and aging of production equipment. Detecting and repairing these defects in a targeted manner is an important part of the industrial production process. The difficulty in completing this task lies in the fact that traditional anomaly detection algorithms can only detect whether there are defects or the location of defects in the product, and cannot further classify the products containing defects. For this reason, the task of discovering new categories of industrial defects is proposed. The goal of this task is to complete the classification of the product samples in the absence of labels for the same product defect category, that is, to divide industrial products into normal products and samples with defects, and to divide the defects in industrial products with defects into different types.

[0003] The existing method of discovering new categories of industrial defects is to locate the defect location, use a pre-trained network model to extract the image features of the defect location, and finally apply an unsupervised clustering algorithm (such as K-means, hierarchical clustering, Gaussian mixture model) to complete the classification. However, the challenge of this idea is that the same type of defect pattern sometimes has large changes. For example, the shape, color texture, and location of the defect are highly diverse and random, making the output features of the pre-trained network model insufficient to distinguish different types of defects.

[0004] At present, there are many studies on new category discovery in the field of natural scenes. The main idea is to introduce a dataset with category annotation (with no overlap with the unlabeled dataset to be classified) to assist in the classification of the unlabeled dataset. Using the dataset with category annotation to train the network can provide the network with certain semantic classification standards, thereby improving the network's feature separability for samples of different categories in the unlabeled dataset. This provides a new solution for the task of discovering new categories of industrial defects. However, directly applying the existing new category discovery algorithms in the field of natural scenes does not work well. The main difficulty is that, on the one hand, there is a big difference between the image patterns of natural scenes and industrial scenes (images in natural scenes are often located in the middle of the image and occupy a larger area of ​​the image, while image defects in industrial scenes are randomly located in the image and often only occupy a smaller area of ​​the image). On the other hand, the defects in industrial scenes are less semantic than those in natural scenes, making it very difficult for the network model to learn to focus on the defective area for classification.

[0005] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for discovering new categories of industrial defects based on mask-guided contrast learning.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for discovering new categories of industrial defects based on mask-guided contrastive learning, comprising:

[0009] Unlabeled datasets to be classified and auxiliary annotation datasets Perform preprocessing to obtain the corresponding mask; among them, and They are the unlabeled images to be classified and the auxiliary labeled data images, and Respectively, images The category labels and defect annotation masks of N l and N u Represents the number of images in the labeled dataset and the unlabeled dataset respectively;

[0010] Using masks to guide contrastive learning;

[0011] The sub-image predictions are weighted and merged based on the area of ​​the defective region in the mask to obtain the classification result of the original image.

[0012] Preferably, the unlabeled dataset to be classified and auxiliary annotation datasets Preprocessing is performed, including:

[0013] Using anomaly detection models Make a prediction and get the prediction result {A i |i∈[1,N u ]}, where A i For images The corresponding predicted anomaly score map;

[0014] For all anomaly score maps {A i |i∈[1,N u ]} Binarization is performed to obtain The mask {M i |i∈[1,N u ]};

[0015] Calculate the mask M i The bounding box corresponding to each connected domain in is based on the bounding box mask Mi and images Perform cropping to obtain a cropped subgraph centered on the defect and clipping mask

[0016] For the auxiliary annotated dataset, use the real defect annotation mask to crop the sub-image in the above way to obtain the cropped sub-image and the cropped mask

[0017] Preferably, for all abnormal score maps {A i |i∈[1,N u ]} Binarization is performed to obtain The mask {M i |i∈[1,N u ]}, including:

[0018] Calculate {A i |i∈[1,N u The anomaly score of each anomaly score map in i |i∈[1,N u ]}, where s i is the anomaly score map A i Pixel value A i The maximum value of (p) is further calculated to obtain The search space for setting the threshold is [s min ,s max ];

[0019] From the threshold interval [s min ,s max ] uniformly sample 64 thresholds to obtain {ε j |j∈[1,64]}, using the threshold ε j A i Binarize and set the j The pixel value is set to 1, which is less than the threshold ε j The pixel value is set to 0 to obtain a binary image

[0020] right After performing the image corrosion operation, count the number of connected domains in the corrosion result graph

[0021] Incrementally traverse all thresholds to obtain the number of connected domains Count the frequency of all non-zero connected domains, taking the mode of the non-zero connected domains As A i An estimate of the number of defective areas in ;

[0022] Find an estimate of the number of connected domains The corresponding longest stable threshold interval [ε p ,ε q ], if the length of the interval is less than 4, that is, q-p+1<4, then the threshold estimate is taken as 1; otherwise, the left endpoint ε of the longest threshold interval is taken p as a threshold estimate;

[0023] Using the threshold estimate ε p A i Binarize and get the image The corresponding mask M i , for all anomaly score maps {A i |i∈[1,N u ]} Repeat the above steps to get The mask {M i |i∈[1,N u ]}.

[0024] Preferably, the mask-guided contrastive learning specifically includes:

[0025] Sub-images of unlabeled images and sub-images with labeled images Perform data augmentation and generate two randomly enhanced views and The corresponding binarized mask is enhanced using the same random parameters;

[0026] Prepare a visual Transformer network pre-trained in natural scene images as the feature extractor f and use a linear classifier Connected to the feature extractor f, get the label of the input sub-graph;

[0027] Contrastive learning is used to allow the network model to learn more discriminative defect representations, which are divided into supervised contrast loss and self-supervised contrast loss;

[0028] The classification task is learned using label supervision.

[0029] Preferably, the sub-image of the unlabeled image and sub-images with labeled images Perform data augmentation and generate two randomly enhanced views and The corresponding binarized mask is enhanced using the same random parameters, including:

[0030] To crop subgraph x i Using the data augmentation strategies in BYOP and RandAug, two randomly augmented views are generated. and The corresponding binarized mask is enhanced using the same random parameters.

[0031] Preferably, the step of preparing a visual Transformer network pre-trained in a natural scene image as a feature extractor f specifically includes:

[0032] The ViT-Base8 model pre-trained by the DINO method on the ImageNet dataset is used. It has a total of 12 layers, the local image region size is 8×8, the input image size is 224×224, and the number of local image regions N is 784;

[0033] For the cropped and enhanced mask images, average pooling is used and flattened into Insert the constant 1 into The first layer of the image global features can retain the knowledge learned from the previous layer and is converted using the following formula:

[0034]

[0035] Using the converted mask image, the self-attention mechanism of the rth layer is adjusted to the following formula:

[0036]

[0037] Where X r represents the output features of the rth layer of ViT, Respectively represent X r-1 After linear projection φ q ,φ k ,φ v The following characteristics, and Adjust the self-attention mechanism of the last 9 layers of the ViT-Base8 model according to the formula.

[0038] Preferably, the method uses a linear classifier After the feature extractor f, the label of the input sub-graph is obtained, including:

[0039] For two randomly augmented views of the input and Its output is and

[0040] At the same time, a three-layer multilayer perceptron is used as the projection layer g, for the two randomly enhanced views of the input and Its output is and

[0041] right Sub-image in The labels of the two randomly augmented views are the true labels;

[0042] for Sub-image in The output of the linear classifier is used to generate pseudo labels. The network outputs of the two enhanced views are and Before Set to negative infinity, then input to the temperature coefficient τ t The softmax function is used to obtain the output of the pseudo label and During the training process, τ t It gradually changes from the initial value of 0.07 to 0.04;

[0043] Then use the anomaly score of the subgraph to correct the pseudo-label, the formula is:

[0044] in Is the length The zero vector, w i =max(0.5-s i ,0),w i is the weight of the normal label, s i is the cropped subgraph x i The maximum anomaly score.

[0045] Preferably, the contrastive learning is used to allow the network model to learn a more discriminative representation of defects, which is divided into supervised contrast loss and self-supervised contrast loss, specifically including:

[0046] The formula for the self-supervised contrastive loss is:

[0047] Where B is the data of the same batch, |B| is the number of images in the batch, is the same batch as image I i Different image indices, τ u The value of is 0.07;

[0048] The formula for supervised contrast loss is:

[0049] Among them B l is the labeled data in B, Is the same batch as I i The image index with the same label, τ c The value of is 1.

[0050] Preferably, the learning of the classification task in a label-supervised manner specifically includes:

[0051] For each pair of images in the batch and Use the student network to smooth the temperature coefficient τ s Generate predictions and where τ s The value of is 0.1; for unlabeled data, the teacher network is used to generate pseudo labels, and for labeled images, the real labels are used and

[0052] The loss function of the classification task is divided into the classification loss of labeled images and the classification loss of unlabeled images. The specific formula is: Where L CE is the standard cross entropy loss function, B u is the unlabeled data in B. Finally, the regularization term is added, and the average predicted entropy of samples in the same batch is used as the loss function. The formula is in

[0053] The overall loss function is Where λ and μ are the weights of the loss function, which take values ​​of 0.3 and 4 respectively.

[0054] Preferably, the weighted merging of sub-image predictions based on the area of ​​the defective region in the mask to obtain the classification result of the original image specifically includes:

[0055] For unlabeled images Cropped sub-image The corresponding prediction result is in is the number of connected domains and also represents the image The defect area of ​​each sub-image is calculated according to the corresponding mask. For the kth subgraph Its prediction results The corresponding weight and the final image Corresponding prediction results The calculation formula is:

[0056] where τ α is the temperature coefficient, and its value is 100.

[0057] In a second aspect, the present invention further provides a method for discovering new categories of industrial defects based on mask-guided contrastive learning, which is used to implement the method for discovering new categories of industrial defects based on mask-guided contrastive learning described in the first aspect, and the device includes:

[0058] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for discovering new categories of industrial defects based on mask-guided contrast learning as described in the first aspect.

[0059] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0060] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect.

[0061] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0062] The present invention is based on mask-guided contrastive learning, and can use masks to effectively guide the model to focus on the corresponding defect area, thereby improving the network model's ability to distinguish different types of defects after contrastive learning. The present invention also proposes a novel mask-guided contrastive learning method, which can use masks to effectively guide the model to focus on the corresponding defect area, thereby improving the network model's ability to distinguish different types of defects after contrastive learning. A novel method for merging the prediction results of cropped sub-images is also proposed. This method can obtain the prediction result of the original image by weighted summing the prediction results of different cropped sub-images according to the defect area, and the result is better than the traditional method of taking the average or maximum value. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a flow chart of a method for discovering new categories of industrial defects based on mask-guided contrast learning provided by an embodiment of the present invention;

[0065] Figure 2 It is a flow chart of a method for discovering new categories of industrial defects based on mask-guided contrast learning provided by an embodiment of the present invention;

[0066] Figure 3 It is a flow chart of a method for discovering new categories of industrial defects based on mask-guided contrast learning provided by an embodiment of the present invention;

[0067] Figure 4 It is a flow chart of a method for discovering new categories of industrial defects based on mask-guided contrast learning provided by an embodiment of the present invention;

[0068] Figure 5 is a schematic diagram of a method for discovering new categories of industrial defects based on mask-guided contrast learning provided by an embodiment of the present invention;

[0069] Figure 6 4 is a schematic diagram of a device for discovering new categories of industrial defects based on mask-guided contrast learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0071] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0072] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0073] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0074] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0075] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0076] Embodiment 1:

[0077] Embodiment 1 of the present invention provides a method for discovering new categories of industrial defects based on mask-guided contrast learning, such as Figure 1 As shown, including:

[0078] In step 201, the unlabeled dataset to be classified and auxiliary annotation datasets Preprocessing is performed to obtain the corresponding mask; wherein, in this embodiment, an unlabeled data set to be classified is given in advance (corresponding to category set C u ) and auxiliary annotation datasets (corresponding to category set C l ),in and They are the unlabeled images to be classified and the auxiliary labeled data images, and Respectively, images The category labels and defect annotation masks of N l and Nu Represents the number of images in the labeled dataset and the unlabeled dataset, and the category set C u and C l The method described in this embodiment aims to convert the unlabeled dataset According to set C u categorize the categories in .

[0079] In step 202, mask-guided contrastive learning is performed.

[0080] In step 203, weighted merging of sub-image predictions is performed based on the area of ​​the defective region in the mask to obtain a classification result of the original image.

[0081] This embodiment is based on mask-guided contrastive learning, and can use masks to effectively guide the model to focus on corresponding defect areas, thereby improving the network model's ability to distinguish different types of defects through contrastive learning.

[0082] In a specific application scenario, the unlabeled dataset to be classified and auxiliary annotation datasets Perform preprocessing, such as Figure 2 As shown, specifically including:

[0083] In step 301, an anomaly detection model is used to Make a prediction and get the prediction result {A i |i∈[1,N u ]}, where A i For images The anomaly score map corresponding to the prediction; the anomaly detection model can be a MuSc model, an EfficientAD model or a CPR model, etc.

[0084] In step 302, all anomaly score graphs {A i |i∈[1,N u ]} Binarization is performed to obtain The mask {M i |i∈[1,N u ]}.

[0085] In step 303, the mask M is calculated. i The bounding box corresponding to each connected domain in is based on the bounding box mask M i and images Perform cropping to obtain a cropped subgraph centered on the defect and clipping mask

[0086] In step 304, for the auxiliary annotated data set (i.e., the auxiliary annotated data set), the sub-image is cropped according to the above method (i.e., step 303) using the real defect annotation mask to obtain the cropped sub-image. and the cropped mask

[0087] In this embodiment, a novel method for binarizing anomaly score maps is provided for the above step 302, that is, for all anomaly score maps {A i |i∈[1,N u ]} Binarization is performed to obtain The mask {M i |i∈[1,N u ]},like Figure 3 As shown, specifically including:

[0088] In step 401, calculate {A i |i∈[1,N u The anomaly score of each anomaly score map in i |i∈[1,N u ]}, where s i is the anomaly score map A i Pixel value A i The maximum value of (p) is further calculated to obtain The search space for setting the threshold is [s min ,s max ].

[0089] In step 402, from the threshold interval [s min ,s max ] uniformly sample 64 thresholds to obtain {ε j |j∈[1,64]}, using the threshold ε j A i Binarize and set the j The pixel value is set to 1, which is less than the threshold ε j The pixel value is set to 0 to obtain a binary image

[0090] In step 403, After performing image corrosion (corrosion kernel size is 6×6), the number of connected domains in the corrosion result graph is counted.

[0091] In step 404, all thresholds are incrementally traversed to obtain a sequence of the number of connected domains. Count the frequency of all non-zero connected domains, taking the mode of the non-zero connected domains As A i Estimated number of defective areas in .

[0092] In step 405, find the estimated value of the number of connected domains The corresponding longest stable threshold interval [ε p ,ε q ](“Stable” means that the number of connected domains in the binarization result corresponding to any threshold within the threshold interval is a constant value ), if the length of the interval is less than 4, that is, q-p+1<4, then the threshold estimate is taken as 1; otherwise, the left endpoint ε of the longest threshold interval is taken p as a threshold estimate.

[0093] In step 406, the threshold estimate ε is used p A i Binarize and get the image The corresponding mask M i , for all anomaly score maps {A i |i∈[1,N u ]} Repeat the above steps to get The mask {M i |i∈[1,N u ]}.

[0094] The novel anomaly score image binarization method proposed in this embodiment can adaptively obtain the binarization threshold of each anomaly score image, give the approximate location of the defect in the original image, and has a strong ability to suppress false omissions of defects.

[0095] In some embodiments, the use of mask-guided contrastive learning, such as Figure 4 As shown, specifically including:

[0096] In step 501, the sub-image of the unlabeled image and sub-images with labeled images Perform data augmentation and generate two randomly enhanced views and The corresponding binarized mask is enhanced using the same random parameters.

[0097] In step 502, a visual Transformer (ViT) network pre-trained in natural scene images is prepared as a feature extractor f, and a linear classifier is used Connected to the feature extractor f, get the label of the input sub-image.

[0098] In step 503, contrastive learning is used to allow the network model to learn a more discriminative representation of defects, which is divided into supervised contrast loss and self-supervised contrast loss.

[0099] In step 504, the classification task is learned using label supervision.

[0100] Among them, the sub-image of the unlabeled image and sub-images with labeled images Perform data augmentation and generate two randomly enhanced views and The corresponding binarized mask is enhanced using the same random parameters, including: i Generate two randomly enhanced views using data augmentation strategies from BYOP (including random cropping, random flipping, color jittering, and Gaussian blur) and RandAug (including random rotation, random bits, and random sharpness). and The corresponding binarized mask is enhanced using the same random parameters (including random cropping, random flipping and random rotation).

[0101] In actual use, the preparation of a visual Transformer network pre-trained in a natural scene image as a feature extractor f can adopt any existing ViT network architecture and pre-trained network parameters. In an optional embodiment, the preparation of a visual Transformer network pre-trained in a natural scene image as a feature extractor f specifically includes:

[0102] The ViT-Base8 model pre-trained by the DINO method on the ImageNet dataset is used. It has a total of 12 layers, the local image area (patch) size is 8×8, the input image size is 224×224, and the number of local image areas N is 784.

[0103] For the cropped and enhanced mask images, average pooling is used and flattened into Insert the constant 1 into The first layer of the image global features can retain the knowledge learned from the previous layer and is converted using the following formula:

[0104]

[0105] Using the converted mask image, the self-attention mechanism of the rth layer is adjusted to the following formula:

[0106]

[0107] Where X r represents the output features of the rth layer of ViT, Respectively represent X r-1 After linear projection φ q ,φ k ,φv The following characteristics, and Adjust the self-attention mechanism of the last 9 layers of the ViT-Base8 model according to the formula.

[0108] In a preferred embodiment, the method uses a linear classifier After the feature extractor f, the label of the input sub-graph is obtained, including:

[0109] For two randomly augmented views of the input and Its output is and

[0110] At the same time, a three-layer multilayer perceptron is used as the projection layer g, for the two randomly enhanced views of the input and Its output is and

[0111] right Sub-image in The labels of the two randomly augmented views are the true labels.

[0112] for Sub-image in The output of the linear classifier is used to generate pseudo labels. The network outputs of the two enhanced views are and Before Set to negative infinity, then input to the temperature coefficient τ t The softmax function is used to obtain the output of the pseudo label and During the training process, τ t It gradually changes from the initial value of 0.07 to 0.04.

[0113] Then, the sub-image (i.e., the cropped sub-image centered on the defect in step 303) is used. )’s anomaly score corrects the pseudo-label, and the formula is:

[0114] in Is the length The zero vector, w i =max(0.5-s i ,0),w i is the weight of the normal label, s i is the cropped subgraph x i The maximum anomaly score.

[0115] In an optional implementation, the contrastive learning is used to allow the network model to learn a more discriminative representation of defects, which is divided into supervised contrast loss and self-supervised contrast loss, specifically including:

[0116] The formula for the self-supervised contrastive loss is:

[0117] Where B is the data of the same batch, |B| is the number of images in the batch, is the same batch as image I i Different image indices, τ u The value of is 0.07.

[0118] The formula for supervised contrast loss is:

[0119] Among them B l is the labeled data in B, Is the same batch as I i The image index with the same label, τ c The value of is 1.

[0120] The method of learning the classification task by label supervision specifically includes:

[0121] For each pair of images in the batch and Use the student network to smooth the temperature coefficient τ s Generate predictions and where τ s The value of is 0.1; for unlabeled data, the teacher network is used to generate pseudo labels, and for labeled images, the real labels are used and Among them, the use of the teacher network to generate pseudo labels is similar to the use of a linear classifier After the feature extractor f, obtaining the label of the input sub-graph is implemented based on the same concept.

[0122] The loss function of the classification task is divided into the classification loss of labeled images and the classification loss of unlabeled images. The specific formula is: Where L CE is the standard cross entropy loss function, B u is the unlabeled data in B. Finally, the regularization term is added, and the average predicted entropy of samples in the same batch is used as the loss function. The formula is in

[0123] The overall loss function is Where λ and μ are the weights of the loss function, which take values ​​of 0.3 and 4 respectively.

[0124] The weighted merging of sub-image predictions based on the area of ​​the defective region in the mask to obtain the classification result of the original image specifically includes:

[0125] For unlabeled images Cropped sub-image The corresponding prediction result is in is the number of connected domains and also represents the image The number of cropped sub-images. Calculate the defect area (number of pixels) of each sub-image according to the corresponding mask For the kth subgraph Its prediction results The corresponding weight and the final image Corresponding prediction results The calculation formula is:

[0126] where τ α is the temperature coefficient, and its value is 100.

[0127] Embodiment 2:

[0128] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0129] The present invention provides a method for discovering new categories of industrial defects based on mask-guided contrastive learning. The method uses the anomaly score output by the anomaly detection model Figure 2 The mask obtained by valuating is used to provide the defect location information of the unlabeled image to be classified, thereby further guiding contrastive learning to enable the network model to better distinguish the features of different types of defect images, thereby achieving a more accurate discovery of new categories of industrial defects.

[0130] First, we give the definition of the dataset in the task to facilitate a better explanation of the method process. This task gives an unlabeled dataset to be classified. (corresponding to category set C u ) and auxiliary annotation datasets (corresponding to category set C l ),in and They are the unlabeled images to be classified and the auxiliary labeled data images, and Respectively, images The category labels and defect annotation masks of N l and N uRepresents the number of images in the labeled dataset and the unlabeled dataset, and the category set C u and C l There is no intersection. The goal of the task is to convert the unlabeled dataset According to set C u categorize the categories in .

[0131] To achieve the above object, the present invention provides the following technical solutions: Figure 5 As shown, the following steps are included:

[0132] Step S1: unlabeled dataset to be classified and auxiliary annotation datasets Perform preprocessing, the specific operations include:

[0133] Step S1-1, prepare an anomaly detection model (such as MuSc, EfficientAD, CPR, etc.) Make a prediction and get the prediction result {A i |i∈[1,N u ]}, where A i For images Corresponding predicted anomaly score map.

[0134] Step S1-2, calculate {A i |i∈[1,N u The anomaly score of each anomaly score map in i |i∈[1,N u ]}, where s i is the anomaly score map A i Pixel value A i The maximum value of (p) is further calculated to obtain The search space for setting the threshold is [s min ,s max ].

[0135] The present invention proposes a novel adaptive threshold binarization method, and its specific operation steps are as follows:

[0136] From the threshold interval [s min ,s max ] uniformly sample 64 thresholds to obtain {ε j |j∈[1,64]}, using the threshold ε j A i Binarize and set the j The pixel value is set to 1, which is less than the threshold ε j The pixel value is set to 0 to obtain a binary image right After performing image corrosion (corrosion kernel size is 6×6), the number of connected domains in the corrosion result graph is counted. Incrementally traverse all thresholds to obtain the number of connected domains Count the frequency of all non-zero connected domains, taking the mode of the non-zero connected domains As A i Next, find the estimated number of connected domains. The corresponding longest stable threshold interval [ε p ,ε q ](“Stable” means that the number of connected domains in the binarization result corresponding to any threshold within the threshold interval is a constant value ), if the length of the interval is less than 4, that is, q-p+1<4, then the threshold estimate is taken as 1, otherwise, the left endpoint ε of the longest threshold interval is taken p As the threshold estimate. Finally, the threshold estimate ε p A i Binarize and get the image The corresponding mask M i , for all anomaly score maps {A i |i∈[1,N u ]} Repeat the above steps to get The mask {M i |i∈[1,N u ]}.

[0137] Step S1-3, calculate the mask M i The bounding box corresponding to each connected domain in is based on the bounding box mask M i and images Perform cropping to obtain a cropped subgraph centered on the defect and clipping mask

[0138] Step S1-4: for the auxiliary annotated data set, use the real defect annotation mask to crop the sub-image in the manner of step S1-3 above to obtain the cropped sub-image. and the cropped mask

[0139] Step S2, using mask to guide contrastive learning, the specific operations include:

[0140] Step S2-1, here is the sub-image of the unlabeled image and sub-images with labeled images Using the same operation, to simplify the expression, the following uses the cropped subgraph x i Describe. For the cropped subgraph x iGenerate two randomly enhanced views using data augmentation strategies from BYOP (including random cropping, random flipping, color jittering, and Gaussian blur) and RandAug (including random rotation, random bits, and random sharpness). and The corresponding binarized mask is enhanced using the same random parameters (including random cropping, random flipping, and random rotation). Then, a visual Transformer (ViT) network pre-trained in natural scene images is prepared as a feature extractor f. Any existing ViT network architecture and pre-trained network parameters can be used. The present invention uses the ViT-Base8 model pre-trained on the ImageNet dataset using the DINO method. It has a total of 12 layers, a local image region (patch) size of 8×8, an input image size of 224×224, and a number of local image regions N of 784. For the cropped and enhanced mask image, average pooling is used and it is flattened to Insert the constant 1 into The first position of the image global feature (classtoken) can retain the knowledge learned from the previous layer and is converted using the following formula:

[0141]

[0142] Using the converted mask image, the self-attention mechanism of the rth layer is adjusted to the following formula:

[0143]

[0144] Where X r represents the output features of the rth layer of ViT, Respectively represent X r-1 After linear projection φ q ,φ k ,φ v The following characteristics, and Adjust the self-attention mechanism of the last 9 layers of the ViT-Base8 model according to the formula.

[0145] Step S2-2, using a linear classifier After the feature extractor f, the label of the input sub-image is obtained. For two randomly enhanced views of the input and Its output is and At the same time, a three-layer multilayer perceptron is used as the projection layer g, for the two randomly enhanced views of the input and Its output is and right Sub-image in The labels of the two randomly augmented views are the true labels. Sub-image in The output of the linear classifier is used to generate pseudo labels. The network outputs of the two enhanced views are and Before Set to negative infinity, then input to the temperature coefficient τ t The softmax function is used to obtain the output of the pseudo label and During the training process, τ t The initial value is gradually changed from 0.07 to 0.04. Then, the pseudo-label is corrected using the abnormal score of the sub-graph in step S1-3. The formula is as follows:

[0146]

[0147] in Is the length The zero vector, w i =max(0.5-s i ,0),w i is the weight of the normal label, s i is the cropped subgraph x i The maximum anomaly score.

[0148] In step S2-3, contrastive learning is used to allow the network model to learn a more discriminative representation of defects, which is divided into supervised contrast loss and self-supervised contrast loss. The formula of self-supervised contrast loss is as follows:

[0149]

[0150] Where B is the data of the same batch, |B| is the number of images in the batch, is the same batch as image I i Different image indices, τ u The value of is 0.07.

[0151] The formula for supervised contrast loss is as follows:

[0152]

[0153] Among them B l is the labeled data in B, Is the same batch as I i The image index with the same label, τ c The value of is 1.

[0154] Step S2-4, use label supervision to learn the classification task. For each pair of images in the batch and Use the student network to smooth the temperature coefficient τ s Generate predictions and where τ s The value of is 0.1. For unlabeled data, use the teacher network to generate pseudo labels according to step S2-2. For labeled images, use the real labels and

[0155] The loss function of the classification task is divided into the classification loss of labeled images and the classification loss of unlabeled images. The specific formula is as follows:

[0156]

[0157] Where L CE is the standard cross entropy loss function, B u is the unlabeled data in B. Finally, the regularization term is added, and the average predicted entropy of samples in the same batch is used as the loss function. The formula is as follows:

[0158]

[0159] in

[0160] The overall loss function is as follows:

[0161]

[0162] Where λ and μ are the weights of the loss function, which take values ​​of 0.3 and 4 respectively.

[0163] Step S3, weighted merging of sub-image predictions based on the area of ​​the defective region in the mask to obtain the classification result of the original image, and the specific steps are as follows:

[0164] For unlabeled images Cropped sub-image The corresponding prediction result is in is the number of connected domains and also represents the image The number of cropped sub-images. Calculate the defect area (number of pixels) of each sub-image according to the corresponding mask For the kth subgraph Its prediction results The corresponding weight and the final image Corresponding prediction results The calculation formula is as follows:

[0165]

[0166] where τ α is the temperature coefficient, and its value is 100.

[0167] In general, the above technical solution conceived by the present invention has the following advantages compared with the existing method:

[0168] Beneficial effects:

[0169] (1) The present invention proposes a novel binarization method for anomaly score images, which can adaptively obtain the binarization threshold of each anomaly score image, give the approximate location of the defect in the original image, and has a strong ability to suppress false omissions of defects.

[0170] (2) The present invention proposes a novel mask-guided contrastive learning method, which can use masks to effectively guide the model to focus on the corresponding defect areas, and then improve the network model's ability to distinguish different types of defects through contrastive learning.

[0171] (3) The present invention proposes a novel method for merging the prediction results of cropped sub-images. This method can obtain the prediction result of the original image by weighted summing up the prediction results of different cropped sub-images according to the defect area. The result is better than the traditional method of taking the average or maximum value.

[0172] Embodiment 3:

[0173] like Figure 6 , is a schematic diagram of the architecture of a device for discovering new types of industrial defects based on mask-guided contrastive learning according to an embodiment of the present invention. The device for discovering new types of industrial defects based on mask-guided contrastive learning according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0174] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0175] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for discovering new categories of industrial defects based on mask-guided contrast learning in Example 1. The processor 21 executes the method for discovering new categories of industrial defects based on mask-guided contrast learning by running the non-volatile software programs and instructions stored in the memory 22.

[0176] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The program instructions / modules are stored in the memory 22 , and when executed by the one or more processors 21 , the method for discovering new categories of industrial defects based on mask-guided contrast learning in the above-mentioned embodiment 1 is executed.

[0178] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0179] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0180] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for discovering new categories of industrial defects based on mask-guided contrastive learning, characterized in that: include: Unlabeled datasets to be classified and auxiliary annotation datasets Perform preprocessing to obtain the corresponding mask; among them, and They are the unlabeled images to be classified and the auxiliary labeled data images, and Respectively, images The category labels and defect annotation masks of N l and N u Represents the number of images in the labeled dataset and the unlabeled dataset respectively; Using masks to guide contrastive learning; The sub-image predictions are weighted and merged based on the area of ​​the defective region in the mask to obtain the classification result of the original image.

2. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 1, characterized in that: The unlabeled dataset to be classified and auxiliary annotation datasets Preprocessing is performed, including: Using anomaly detection models Make a prediction and get the prediction result {A i |i∈[1,N u ]}, where A i For images The corresponding predicted anomaly score map; For all anomaly score maps {A i |i∈[1,N u ]} Binarization is performed to obtain The mask {M i |i∈[1,N u ]}; Calculate the mask M i The bounding box corresponding to each connected domain in is based on the bounding box mask M i and images Perform cropping to obtain a cropped subgraph centered on the defect and clipping mask For the auxiliary annotated dataset, use the real defect annotation mask to crop the sub-image in the above way to obtain the cropped sub-image and the cropped mask 3. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 2, characterized in that: For all anomaly score maps {A i |i∈[1,N u ]} Binarization is performed to obtain The mask {M i |i∈[1,N u ]}, including: Calculate {A i |i∈[1,N u The anomaly score of each anomaly score map in i |i∈[1,N u ]}, where s i is the anomaly score map A i Pixel value A i The maximum value of (p) is further calculated to obtain The search space for setting the threshold is [s min ,s max ]; From the threshold interval [s min ,s max ] uniformly sample 64 thresholds to obtain {ε j |j∈[1,64]}, using the threshold ε j A i Binarize and set the j The pixel value is set to 1, which is less than the threshold ε j The pixel value is set to 0 to obtain a binary image right After performing the image corrosion operation, count the number of connected domains in the corrosion result graph Incrementally traverse all thresholds to obtain the number of connected domains Count the frequency of all non-zero connected domains, taking the mode of the non-zero connected domains As A i An estimate of the number of defective areas in ; Find an estimate of the number of connected domains The corresponding longest stable threshold interval [ε p ,ε q ], if the length of the interval is less than 4, that is, q-p+1<4, then the threshold estimate is taken as 1; otherwise, the left endpoint ε of the longest threshold interval is taken p as a threshold estimate; Using the threshold estimate ε p A i Binarize and get the image The corresponding mask M i , for all anomaly score maps {A i |i∈[1,N u ]} Repeat the above steps to get The mask {M i |i∈[1,N u ]}.

4. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 1, characterized in that: The mask-guided contrastive learning specifically includes: Sub-images of unlabeled images and sub-images with labeled images Perform data augmentation and generate two randomly enhanced views and The corresponding binarized mask is enhanced using the same random parameters; Prepare a visual Transformer network pre-trained in natural scene images as the feature extractor f and use a linear classifier Connected to the feature extractor f, get the label of the input sub-graph; Contrastive learning is used to allow the network model to learn more discriminative defect representations, which are divided into supervised contrast loss and self-supervised contrast loss; The classification task is learned using label supervision.

5. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 4, characterized in that: The sub-image of the unlabeled image and sub-images with labeled images Perform data augmentation and generate two randomly enhanced views and The corresponding binarized mask is enhanced using the same random parameters, including: To crop subgraph x i Using the data augmentation strategies in BYOP and RandAug, two randomly augmented views are generated. and The corresponding binarized mask is enhanced using the same random parameters.

6. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 4, characterized in that: The step of preparing a visual Transformer network pre-trained in a natural scene image as a feature extractor f specifically includes: The ViT-Base8 model pre-trained by the DINO method on the ImageNet dataset is used. It has a total of 12 layers, the local image region size is 8×8, the input image size is 224×224, and the number of local image regions N is 784; For the cropped and enhanced mask images, average pooling is used and flattened into Insert the constant 1 into The first layer of the image global feature can retain the knowledge learned from the previous layer and is converted using the following formula: Using the converted mask image, the self-attention mechanism of the rth layer is adjusted to the following formula: Where X r represents the output features of the rth layer of ViT, Respectively represent X r-1 After linear projection φ q ,φ k ,φ v The following characteristics, and Adjust the self-attention mechanism of the last 9 layers of the ViT-Base8 model according to the formula.

7. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 4, characterized in that: The method of using a linear classifier H connected to the feature extractor f to obtain the label of the input sub-graph specifically includes: For two randomly augmented views of the input and Its output is and At the same time, a three-layer multilayer perceptron is used as the projection layer g, for the two randomly enhanced views of the input and Its output is and right Sub-image in The labels of the two randomly augmented views are the true labels; for Sub-image in The output of the linear classifier is used to generate pseudo labels. The network outputs of the two enhanced views are and Before Set to negative infinity, then input to the temperature coefficient τ t The softmax function is used to obtain the output of the pseudo label and During the training process, τ t It gradually changes from the initial value of 0.07 to 0.04; Then use the anomaly score of the subgraph to correct the pseudo-label, the formula is: in Is the length The zero vector, w i =max(0.5-s i ,0),w i is the weight of the normal label, s i is the cropped subgraph x i The maximum anomaly score.

8. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 4, characterized in that: The contrastive learning method allows the network model to learn a more discriminative representation of defects, which is divided into supervised contrast loss and self-supervised contrast loss, specifically including: The formula for the self-supervised contrastive loss is: Where B is the data of the same batch, |B| is the number of images in the batch, is the same batch as image I i Different image indices, τ u The value of is 0.07; The formula for supervised contrast loss is: Among them B l is the labeled data in B, Is the same batch as I i The image index with the same label, τ c The value of is 1.

9. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 1, characterized in that: The label-supervised learning classification task specifically includes: For each pair of images in the batch and Use the student network to smooth the temperature coefficient τ s Generate predictions and where τ s The value of is 0.1; for unlabeled data, the teacher network is used to generate pseudo labels, and for labeled images, the real labels are used and The loss function of the classification task is divided into the classification loss of labeled images and the classification loss of unlabeled images. The specific formula is: Where L CE is the standard cross entropy loss function, B u is the unlabeled data in B. Finally, the regularization term is added, and the average predicted entropy of samples in the same batch is used as the loss function. The formula is in The overall loss function is Where λ and μ are the weights of the loss function, which take values ​​of 0.3 and 4 respectively.

10. The method for discovering new categories of industrial defects based on mask-guided contrastive learning according to claim 1, characterized in that: The weighted merging of sub-image predictions based on the area of ​​the defective region in the mask to obtain the classification result of the original image specifically includes: For unlabeled images Cropped sub-image The corresponding prediction result is in is the number of connected domains and also represents the image The number of cropped sub-images; Calculate the defect area of ​​each sub-image according to the corresponding mask For the kth subgraph Its prediction results The corresponding weight and the final image Corresponding prediction results The calculation formula is: where τ α is the temperature coefficient, and its value is 100.

Citation Information

Patent Citations

  • Industrial defect detection method, device and equipment and readable storage medium

    CN115239638A

  • Semi-supervised image segmentation method based on uncertain false label correction

    CN116206106A

  • Self-supervised facial expression recognition method combining comparative learning with mask image modeling

    CN116386104A

  • Electrical equipment defect detection method and system based on self-supervised learning

    CN117671587A

  • Industrial image defect detection method based on momentum contrast learning

    CN118037708A

Cited By

  • Contrast learning and foreground extraction-based dense scene target detection method and system, and medium

    CN120783019A

  • A dense scene target detection method, system and medium based on contrast learning and foreground extraction

    CN120783019B