A method for few-shot classification of screen defects

Abstract

This invention discloses a method for few-shot classification of screen defects, comprising a feature extractor with multiple transformer blocks, a class prototype generator (CPG), and a context-based classifier (CBC). As shown in the abstract figures, firstly, the feature extractor takes the image as input and outputs the image features extracted by the feature extractor. Secondly, the CPG, based on image features and image labels, introduces channel-based relationships between image features to generate corresponding prototypes. Finally, the CBC registers the class prototypes and uses global context information for all classes to perform classification. This invention designs a feature extractor with multiple transformer blocks. This feature extractor extracts features that highlight defects by constructing cross-sample attention and cross-region interaction. The introduction of CPG generates more representative class prototypes. Finally, a CBC applying global context information is proposed, which can make better classification decisions across all classes.

Description

Technical Field

[0001] This invention relates to the field of image classification, and more particularly to a method for classifying screen defects using a small number of samples. Background Technology

[0002] The quality of a mobile phone screen affects both the phone's appearance and the consumer experience, but defects are inevitable during the complex manufacturing process. Traditional manual inspection, relying on skilled workers, is inefficient and time-consuming. Therefore, some research has focused on vision-based automated defect detection, achieving significant success. These automated methods fall into two main categories: methods based on human-defined features and methods based on convolutional neural networks (CNNs). The former is typically designed for specific defects and therefore lacks generalization ability. In contrast, the latter has strong high-level feature learning capabilities and outperforms the former in defect detection. Consequently, CNN-based methods have recently attracted increasing attention in the field of defect detection.

[0003] However, most deep learning methods require a large amount of labeled training data, thus facing performance degradation when only a small amount of labeled data is available for training. In actual mobile phone screen production lines, the output of normal products is relatively high, making it difficult to collect data on some defects (such as watermarks). Secondly, real-world industrial scenarios not only face known defects but also sometimes new ones. These unknown categories cannot be sampled in large quantities in a short time. These two issues necessitate that the model be able to handle the limited sample problem.

[0004] To liberate CNN-based models from massive training data and enable them to classify unknown classes in sparsely labeled training images, few-shot learning methods and few-shot incremental learning methods have been proposed. Even with insufficient training samples, humans learn new concepts well. These few-shot learning methods aim to endow models with human learning capabilities. However, most other work focuses on natural images. Unlike natural images, defective images of mobile phone screens have many unique features to consider. First, defective images often exhibit high similarity due to similar backgrounds and small defect areas. This high similarity makes it difficult to distinguish defect categories. Additionally, mobile phone screen imaging typically includes reflective areas. These bright areas are unavoidable due to the curved surface. Tin gray defects and scratches are more likely to appear around these bright bands than other defects, potentially misleading the strong binding of bright band features and these two types of features. In reflective areas, normal images or other defective images may be incorrectly classified as scratches or tin gray. Therefore, it is important to investigate how to effectively reduce background information. Furthermore, some screen defects may lack certain category features and be noisy. Unlike defects, noise is usually too small to be identified. However, to improve the accuracy of defect detection, noise needs to be considered and addressed. When unknown categories appear and training samples are scarce, it is crucial to generate more representative prototypes and avoid overwhelming the limited but important features with noise. Summary of the Invention

[0005] The purpose of this invention is to provide a method for classifying screen defects using a small number of samples.

[0006] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0007] This invention comprises a feature extractor with multiple transformer blocks, a CPG, and a CBC, and includes the following steps:

[0008] S1: First, the feature extractor takes the image as input and outputs the image features extracted by the feature extractor. To address the problem of classifying defects in mobile phone screens, a few-shot network is proposed, capable of simultaneously identifying known categories with sufficient training data and new categories with a small number of labeled training samples. Cross-sample attention and cross-region interaction are designed to enhance attention and interaction between different samples and different regions, thereby improving feature extraction. CPG is proposed, utilizing the channel clustering degree of features to avoid drowning out limited important features in noise. CBC is proposed, aiming to apply global contextual information across all classes and generate better classification results across all classes.

[0009] Cross-sample attention: Cross-sample attention is established between support and query using some link tokens. These link tokens are trainable and update themselves by simultaneously absorbing all query and support tokens. In the subsequent transformer block, the updated state of the link tokens is absorbed by the query and support tokens respectively. Therefore, it facilitates the interaction between query and support with relatively low computational cost.

[0010] Cross-region interaction: In each transformer block, a cross-region interaction layer is attached on top of the original self-attention layer. The original self-attention layer encourages interaction between similar patch markers, while cross-region interaction promotes interaction between different regions in the background.

[0011] S2: Based on image features and image labels, CPG introduces channel-based relationships between image features to generate corresponding prototypes; CPG: Taking image features and image labels as input, the generator outputs prototypes. Since there are few annotated samples for the new category, the designed module utilizes the channel aggregation degree of features and makes full use of the features most representative of that category.

[0012] S3: Class prototypes are registered via CBC, and classification is performed using global context information for all classes.

[0013] CBC: Known and new categories are always learned independently. Therefore, while good classification results can be produced within known categories, confusion can easily arise between known and new categories. For example, scratches and floating debris are both linear defects. Based on the number of trainable samples, the former is classified as a known category, and the latter as a new category. A common classifier that only learns the difference between scratches and block defects may not produce good classification results between linear defects (i.e., scratches and floating debris). CBC can output better classification results across all categories. CBC measures the similarity between all known and new categories across all feature channels. It then generates attention based on channel importance. As attention increases, discriminative representations between categories are highlighted, while similar feature points between categories account for a relatively small proportion of the final score composition.

[0014] The beneficial effects of this invention are:

[0015] This invention presents a method for few-shot classification of screen defects. Compared to existing technologies, this invention designs a feature extractor with multiple transformer blocks. This module applies ordinary images to attenuate irrelevant background information, thereby extracting features that highlight defects. Then, to avoid obscuring the limited number of important features in noise, a class prototype generator (CPG) is introduced to produce more representative class prototypes. Finally, a context-based classifier (CBC) that applies global context information is proposed. This module can make better classification decisions across all classes. Attached Figure Description

[0016] Figure 1 Data partitioning for the K-shot learning task.

[0017] Figure 2 This provides an overview framework for the few-shot method proposed in this invention.

[0018] Figure 3 This is the architecture for CBC. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] This invention addresses a multi-class single-label image classification task that simultaneously classifies new and old labels. However, due to the limited number of defective categories, the task of this invention has only one incremental session, instead of eight or ten. That is, the task of this invention includes one basic session and one incremental session. Furthermore, to mitigate the randomness of the results, this incremental session is repeated several times, and the training samples for the incremental session are not fixed.

[0021] like Figure 1 As shown, the total defect dataset is divided into two parts according to class, namely the basic part D. base And new part D novel The categories of these two parts do not overlap. Assume D base All classes in the dataset have rich labeled images for training, while D... novel The number of trainable samples for each class is severely limited. base Divided into and These are used for model pre-training, hyperparameter fine-tuning, and model performance testing, respectively. D novel Randomly split into a support set (i = 1, 2, ..., 100) and a query set (i = 1, 2, ..., 100) 100 times. For K-shot learning, Each category contains K images. The remaining images are used for... In the basic session, A few-shot model was trained. In incremental sessions, the pre-trained model was fine-tuned, and... and The tests were conducted on [the platform]. 100 incremental sessions were performed independently. The average results were used to demonstrate the performance of the fine-tuning model across all involved classes.

[0022] The overall framework of the model of this invention is as follows: Figure 2 As shown, it mainly consists of three modules: a feature extractor with multiple transformer blocks, CPG, and CBC. The feature extractor takes the image as input and outputs the image features extracted by the feature extractor. CPG introduces channel-based relationships between image features based on image features and image labels to generate corresponding prototypes. CBC registers class prototypes and uses global context information for all classes to perform classification.

[0023] The entire model learning process can be divided into two stages: the basic training stage and the incremental classifier learning stage. In the first stage, a contextual meta-learning strategy is used to train the model. The classifier learning stage learns the prototype of the new class and updates the class attention of the classifier. After these two stages, the proposed model can classify both the base class and the new class simultaneously.

[0024] Defective images often have a large number of similar background pixels. Furthermore, heterogeneity (i.e., intra-class diversity in the original image) can be considered a key challenge hindering the compactness of intra-class features. Heterogeneity can be categorized into three levels from coarse to fine: 1) Sample level: The unavoidable distributional gap between the support and query images creates heterogeneity between them. 2) Region level: The background in the feature extractor actually contains multiple regions with different semantics. 3) Patch level: There can be significant differences between some adjacent patches belonging to the same category. Therefore, cross-sample attention and cross-region interaction can be constructed to enhance attention and interaction between different samples (query and support) and different regions, respectively.

[0025] Specifically, use the link tag X link ∈R C×D Interactive queries and support are performed using D-dimensional embedding vectors. Two link tags are initialized in the input block. These are the average features of the foreground and background regions supporting the sample, respectively. Then, the link labels are updated block by block using cross-attention. This process can be described as follows:

[0026]

[0027] in, The linked list represents the updated tag, and {,} represents the join operation. Que, Key, and Val are the supporting tags computed in the i-th transformer block, respectively. Query tags and link tags Operations on query embedding, key embedding, and value embedding. With the updated link tags, support and query patch tags can be updated using the following methods:

[0028]

[0029]

[0030] in, It is the D-dimensional embedding of the supporting tags and query tags in the i-th block. This indicates the link tag embedding that is updated block by block based on cross attention.

[0031] Since the linking tags have absorbed information from all query and support tags, they propagate this absorbed information to both support and query tags, facilitating mediating interactions between support and query samples. This linking tag solution offers a significant efficiency advantage compared to directly building "patch-to-patch" attention between query and support samples. Specifically, patch-to-patch attention incurs quadratic complexity related to the number of patch tags, while using linking tags results in only linear complexity.

[0032] Cross-region interactions are appended to each transformer layer to facilitate interactions between different regions in the background. To do this, the background in each query image is segmented into multiple regions using ground truth labels. Specifically, some regions in the background actually belong to certain annotated foreground categories, but are merged into the background because the current training focuses on different foreground categories. These regions are called temporary background, and the remainder (without foreground annotations) is called constant background. Accordingly, x is used... [tb] and x [cb] Distinguish the markers in the temporary background and constant background regions separately, and use x [f] This indicates a marker in the foreground region of interest within the current scene.

[0033] Cross-regional interaction comparison after the attention layer x [tb] and x [cb] The cosine distance between them, and x is made in the following way [cb] Absorption from different x [tb] ∈X [tb] Information:

[0034]

[0035] Where x [cb]′ This indicates the updated constant background tag embedding.

[0036] In addition to the aforementioned cross-regional interaction that smooths the background across different regions through attention, a region-level triplet loss is further used to bring the constant background marker and temporal background marker closer together on the last transformer block. The region-level triplet loss is enforced on the final output state of the background marker in the following way:

[0037] L tri =max(D(x) [tb] ,x [cb] )-D(x [tb] ,x [f] ),0)

[0038] Where D(.,.) is the cosine distance between the two markers.

[0039] CPG takes image features and image labels as input and extracts class prototypes for subsequent classification. To enable the module to generate powerful representations even when dealing with new classes with only a few training samples, the generator is designed to focus on features that are more representative along the channel axis.

[0040] In the forward propagation, CPG generates a class prototype for each class i.

[0041] P(i)=g(F s ")

[0042] in, K represents the image features of the selected supporting samples. s This is the number of samples selected. At this point, the label of the selected sample is equal to i.

[0043] Some defects may lack certain category features and contain noise. New classes have only a limited number of labeled training samples. The goal is for the prototype to be more representative, and to prevent limited, important features from being buried in noise. Therefore, channel-based relationships between image features are calculated. Based on these relationships, each dimension of the prototype considers more important features. The implementation of g(·) can be summarized as follows:

[0044]

[0045] W1=σ(f1(sim1))

[0046]

[0047] in This is a relationship. This relationship is positively correlated with the contribution of the feature vectors to the similarity. `dist(·)` represents the distance metric used, the same one used to calculate the classification score. Since cosine similarity is chosen as the distance metric, `sim1` and `dist(·)` are positively correlated. σ represents a sigmoid function, and `f1(·)` is a fully connected layer with the same input and output dimensions. `f1(·)` enables `g(·)` to study the relationship between `sim1` and importance-based weights. The relationship between them. Finally, the prototype is sent to a classifier for further classification.

[0048] The base class or the new category can be distinguished well by common classifiers. However, these two categories are learned independently. Common classifiers may not produce good classification results for all the classes involved. To alleviate this problem, this invention proposes a CBC (Discriminative Class Layout) method. Utilizing global contextual information of all classes, this module can highlight discriminative class representations and output better classifications for all classes. Figure 3 The structure of the module is shown, and its important component is context-based attention.

[0049] After registering the class prototypes output by the generator, the classifier calculates the relationships between the registered prototypes, which can be written as the following formula:

[0050]

[0051] Here, dist(·) represents the similarity function used, which is the same as the function used to calculate the classification score. `i` is a function that outputs `if l≠i`. `n` represents the number of class prototypes registered in the classifier. This refers to the average relationship. sim2 is positively correlated with the contribution of the input vector to the similarity.

[0052] Then, channel importance-based attention W2 is generated through a fully connected layer and channel subtraction, as shown below:

[0053] W2 = 1 - f2(sim2)

[0054] Here, f2(·) is implemented by a fully connected layer with the same input and output dimensions. f2(·) allows the model to learn the relationship between sim2 and attention. The design of this invention considers two factors: 1) the discriminative representation of each class is to be highlighted; 2) the designed classifier is equivalent to a regular classifier when all elements of W2 are equal to 1. Therefore, the higher the similarity of a prototype to other prototypes in one dimension, the less than 1 the W2 value will be in that dimension.

[0055] Ultimately, the module uses category-related information (including category prototype P and attention W2) and query features F′ q Output the classification score S for each query image. The prediction score for the i-th class is calculated as follows:

[0056]

[0057] Cosine similarity is used to outperform other distance metrics in few-shot learning to achieve dist(·). Cross-entropy loss is used as the classification loss L. CE .

[0058] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A method for few-sample classification of screen defects, characterized in that: It consists of a feature extractor with multiple transformer blocks, CPG, and CBC, and includes the following steps: S1: First, the feature extractor takes the image as input and outputs the image features extracted by the feature extractor; S2: Based on image features and image labels, CPG introduces channel-based relationships between image features to generate corresponding prototypes; S3: Class prototypes are registered via CBC, and classification is performed using global context information for all classes; Step S1 specifically involves: using link tags. Interactive queries and support, the link tags The embedding vector has D dimensions; in the input block, two link tags are initialized. These are the average features of the foreground and background regions supporting the sample, respectively; then, the link markers are updated block by block through cross-attention; step S1 can be described as follows: in, The linked list represents the updated link markers; {,} represents the join operation; Que, Key, and Val are the support markers used to compute the i-th transformer block, respectively. Query tags and link tags Operations on query embedding, key embedding, and value embedding; with updated link tags, support and query patch tags are updated via the following methods: in, , It is the D-dimensional embedding of the supporting tags and query tags in the i-th block; This represents the link tag embedding that is updated block by block based on cross attention; Cross-region interactions are attached to each transformer layer to facilitate interactions between different regions in the background. The background in each query image is segmented into multiple regions using ground truth labels. Some regions in the background belong to certain annotated foreground categories. The current training focuses on different foreground categories and these regions are merged into the background; these regions are called temporary background. The remaining unannotated foreground regions are called constant background. Accordingly, using... and Distinguish the markers in the temporary background and constant background areas separately, and use This indicates a marker in the foreground region of interest within the current scene; Cross-regional interaction comparison after the attention layer and The cosine distance between them, and made in the following way Absorption from different Information: in Indicates the updated constant background tag embedding; Further, region-level triplet loss is used to bring the constant background marker and temporal background marker closer together on the last transformer block; the region-level triplet loss is enforced on the final output state of the background marker in the following way: Where D(.,.) is the cosine distance between the two markers.

2. The method for classifying screen defects using few samples according to claim 1, characterized in that: CPG takes image features and image labels as input, extracts class prototypes for subsequent classification. In the forward pass, CPG is used for each class... Generate class prototype : in, This represents the image features of the selected supporting samples; This is the number of samples selected; at this point, the label of the selected samples equals... ; The implementation is as follows: in It's a relationship; This indicates that the distance metric used is the same as the distance metric used to calculate the classification score; and It is positively correlated; Represents an sigmoid function. It is a fully connected layer with the same input and output dimensions; make Able to study Weights based on importance The relationship between the prototype and the classifier is then established; finally, the prototype is sent to the classifier for further classification.

3. The method for classifying screen defects using a small sample size according to claim 2, characterized in that: After registering the class prototypes output by the generator, the classifier calculates the relationships between the registered prototypes, which can be written as the following formula: in, This indicates that the similarity function used is the same as the function used to calculate the classification score; It is a function that outputs 1 iff ; This indicates the number of class prototypes registered in the classifier; This refers to the average relationship; sim2 is positively correlated with the contribution of the input vector to the similarity. Then, channel importance-based attention is generated through a fully connected layer and channel subtraction. As shown below: in, It is implemented by a fully connected layer with the same input and output dimensions; Allow the model to learn The relationship between attention and attention; Ultimately, the CBC uses category-related information, including category prototypes. and attention and query features Output the classification score for each query image. The prediction score for the i-th class is calculated as follows: Cosine similarity is used to outperform other distance metrics in few-shot learning. Cross-entropy loss is used as the classification loss. .

Images