LCD screen defect classification method based on fusion attention mechanism and storage medium

Abstract

The application discloses a LCD screen defect classification method based on a fusion attention mechanism and a storage medium, and relates to the technical field of surface defect classification. The method comprises the following steps: acquiring an LCD screen image to be classified and preprocessing the LCD screen image, taking the preprocessed LCD screen image as a query set image; acquiring a sample image labeled with defect information and preprocessing the sample image, taking the preprocessed sample image as a support set image for small sample learning; inputting the query set image and the support set image into a pre-trained prototype network to obtain a distance measurement result between the query set image and the support set image; and classifying according to the distance measurement result to generate a LCD screen defect classification result. Compared with the prior art, the application improves the accuracy of the distance measurement result through an attention generation module, and further improves the precision and stability of the LCD screen defect classification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of surface defect classification, and more particularly to an LCD screen defect classification method based on a fusion attention mechanism and a storage medium. BACKGROUND

[0002] Different types of defects are produced in the production process of LCD screens due to process problems, and different defect treatment methods are required for different defects. With the improvement of production automation, algorithm classification is gradually replacing manual classification.

[0003] Traditional defect classification algorithms usually judge defect types according to typical features such as defect area and shape, rely on manual design by researchers, and are highly specialized, with one defect corresponding to one algorithm, high development cost, and large number of parameters. In addition, the same type of defect may also have large differences in image features, which makes it difficult to use only traditional image processing techniques to identify defect types.

[0004] In recent years, deep learning methods have been increasingly applied to defect classification. Models represented by CNN and Transformer are widely used in image recognition. However, these models rely on large amounts of data. Due to the particularity of LCD screen product defects, there are many types of defects and few defect samples. The above models do not perform well in defect recognition, and pre-trained models have low generalization ability and severe overfitting in small sample defect classification tasks. Defects of similar types are easily misclassified.

[0005] Existing technologies propose small sample image classification methods based on metrics, aiming to reduce the distance between samples of the same class and increase the distance between samples of different classes, thereby making the classes more separable. For example, the prototype network is trained using meta-learning, and the class average is used as the class representation. However, due to the small number of class samples, the average of a small number of samples cannot well represent the entire class, resulting in blurred class boundaries, weak stability, and susceptibility to extreme sample interference. SUMMARY

[0006] To overcome the low classification accuracy and weak stability of the existing technology, the present application provides an LCD screen defect classification method based on a fusion attention mechanism and a storage medium.

[0007] To solve the above technical problems, the technical solution of the present application is as follows:

[0008] In a first aspect, an LCD screen defect classification method based on a fusion channel attention mechanism includes:

[0009] Obtain an LCD screen image to be classified and pre-process it, and take the pre-processed LCD screen image as a query set image;

[0010] Obtain a sample image labeled with defect information and pre-process it, and take the pre-processed sample image as a support set image for small sample learning; the defect information includes the belonging category of the sample image;

[0011] Input the query set image and the support set image into a pre-trained prototype network to obtain a distance measurement result between the query set image and the support set image; wherein the prototype network is provided with an attention generation module, including a series of intra-class attention units and global attention units;

[0012] Classify according to the distance measurement result to generate an LCD screen defect classification result.

[0013] In a second aspect, a computer readable storage medium is provided, and the storage medium stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement the method of the first aspect.

[0014] Compared with the prior art, the technical scheme of the present application has the beneficial effects that:

[0015] By setting the attention generation module in the prototype network, the present application utilizes the feature of the fusion attention mechanism to focus on the key area: through the intra-class attention, the attention degree to the representative features of the category is enhanced, and through the global attention, the attention degree to the features of weak inter-class separability is reduced, so that the distinguishability of different category samples is greater. Compared with the prior art, the present application improves the accuracy of the distance measurement result through the attention generation module, and further improves the precision and stability of the LCD screen defect classification result. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is an LCD screen defect classification method flowchart of embodiment 1 of the present application;

[0017] Figure 2 It is an example diagram of the LCD screen defect category of embodiment 1 of the present application;

[0018] Figure 3 It is a prototype network structure diagram of embodiment 1 of the present application;

[0019] Figure 4 It is an example diagram of the channel measurement attention generation process of embodiment 1 of the present application;

[0020] Figure 5 It is an example diagram of the second embedded feature map processing process of embodiment 1 of the present application;

[0021] Figure 6 Figure 1 is a schematic diagram of the process of reconstructing the feature map according to Example 1. DETAILED DESCRIPTION

[0022] The terms "first", "second", and the like in the description and in the claims of the present application and in the above drawings are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the present application are capable of functioning in other sequences, except where it is inherent from the procedure. Furthermore, the terms "comprise", "comprising", "include", "including", and the like are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, includes, or includes elements or steps that are not listed is not excluded from the scope of the embodiments of the present application.

[0023] The accompanying drawings are used to illustrate the embodiments of the present application only and should not be understood as limiting the present application;

[0024] In order to better illustrate the embodiments, some components in the drawings can be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0025] It is understandable for those skilled in the art that some well-known structures and their descriptions in the drawings can be omitted.

[0026] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0027] Example 1

[0028] The present embodiment proposes a LCD screen defect classification method based on a fusion channel attention mechanism, referring to Figure 1 , comprising:

[0029] Obtaining the LCD screen image to be classified and pre-processing it, taking the pre-processed LCD screen image as the query set image;

[0030] Obtaining the sample image labeled with defect information and pre-processing it, taking the pre-processed sample image as the support set image for small sample learning; the defect information includes the attribution category of the sample image;

[0031] Inputting the query set image and the support set image into the pre-trained prototype network to obtain the distance measurement result between the query set image and the support set image; wherein the prototype network is provided with an attention generation module, including a series of intra-class attention unit and global attention unit;

[0032] Classifying according to the distance measurement result to generate the LCD screen defect classification result.

[0033] In some examples, the attribution category of the sample image includes normal (i.e. no defect), point defect, line defect, mura defect.

[0034] Those skilled in the art should understand that the point defect is an abnormal point appearing after the LCD screen is lighted, the line defect is a bright line or a dark line on the LCD screen, and the mura defect is an irregular mass defect appearing after the LCD screen is lighted, refer to Figure 2 .

[0035] In a preferred embodiment, the preprocessing includes at least one of sliding window cropping, random cropping, rotation, stretching, mask, etc.

[0036] The preprocessing further includes defect region cropping and random sampling of the defective sample in the sample image and mixing the defective sample into the non-defective sample.

[0037] The preprocessing further includes converting the image into a tensor and performing standardization processing (i.e. normalization) on the tensor.

[0038] It should be noted that the sliding window cropping, random cropping, rotation, stretching, mask, etc. can realize data enhancement and increase sample diversity; the above-mentioned methods can be set by those skilled in the art according to actual conditions.

[0039] In addition, by performing defect region cropping and random sampling of the defective sample in the sample image and mixing the defective sample into the non-defective sample, the defect features (i.e. adding noise) are added to the normal sample to a certain extent, which can alleviate the overfitting problem in small sample learning.

[0040] In some examples, the sliding window cropping, random cropping, stretching, tensor conversion, and adding noise are used as the preprocessing method.

[0041] In some examples, the random cropping, rotation, mask, tensor conversion, and adding noise are used as the preprocessing method.

[0042] In another example, the LCD screen image to be classified has a size of 9216x7000, a window of 512x512 is used for sliding window cropping, and then the image is converted into a tensor and standardized.

[0043] In a preferred embodiment, refer to Figure 3 , the prototype network further includes an embedding module, a prototype generation module, and a metric module, and the attention generation module is arranged between the embedding module and the prototype generation module.

[0044] The query set image and the support set image are input into a pre-trained prototype network to obtain a distance measurement result between the query set image and the support set image, and the method comprises the following steps:

[0045] The query set image and the support set image are input into an embedding module to respectively extract a first embedding feature map about the query set image and a second embedding feature map about the support set image, wherein the embedding module is constructed based on a CONV4 (four-layer convolutional network) or a ResNet (Residual Network).

[0046] The second embedding feature map is input into an attention generation module to obtain a reconstructed feature map and a channel measurement attention fused with intra-class attention and global attention.

[0047] The reconstructed feature map and the channel measurement attention are input into a prototype generation module to obtain a reconstructed prototype.

[0048] The first embedding feature map, the reconstructed prototype and the channel measurement attention are taken as inputs of a measurement module to obtain the distance measurement result.

[0049] The preferred embodiment is based on a conventional prototype network, extracts corresponding embedding feature maps through an embedding module, obtains a reconstructed feature map about the support set image and a channel measurement attention based on a fused attention mechanism through an attention generation module, reconstructs the extracted embedding feature maps through a prototype generation module to obtain a reconstructed prototype of each class, and obtains the similarity (i.e. the distance measurement result) of each query set image to each class in the support set image through a measurement module.

[0050] In some examples, the embedding module is constructed based on a CONV4.

[0051] In some examples, the embedding module is constructed based on a shallow network ResNet12.

[0052] In other examples, the embedding module is constructed based on a ResNet18.

[0053] In an optional embodiment, referring to Figure 4 The second embedding feature map is input into an attention generation module to obtain a reconstructed feature map and a channel measurement attention fused with intra-class attention and global attention, and the method comprises the following steps:

[0054] The second embedding feature map is taken as an input of an intra-class attention unit to obtain an intra-class attention weight.

[0055] Based on a broadcast mechanism, the corresponding second embedding feature map is reconstructed according to the intra-class attention weight to obtain a reconstructed feature map.

[0056] The reconstructed feature map is input into a global attention unit to obtain channel measurement attention.

[0057] Those skilled in the art should understand that in LCD screen defect classification, each category should have its unique separable characteristics, i.e., "intra-class commonality". The intra-class attention unit extracts the intra-class commonality to generate intra-class attention weights, which can better represent the importance of the feature channels (i.e., the second embedded feature map) of the corresponding category; the global attention unit extracts inter-class commonality; the two attention units have opposite effects, the intra-class attention unit enhances useful features, and the global attention unit weakens useless features, and the series connection of the two attention units forms channel measurement attention.

[0058] Further, the intra-class attention unit is constructed based on an SE-Net (Squeeze-and-Excitation Networks).

[0059] The second embedded feature map belonging to the same class is input into the intra-class attention unit to perform average pooling operation in the spatial dimension to obtain a first representation of each second embedded feature map.

[0060] The mean representation and the variance representation of each first representation are calculated, and the intra-class attention weights are generated through a Bottleneck structure; wherein the Bottleneck structure is composed of fully connected layers.

[0061] After superimposing all the intra-class attention weights, the intra-class attention weights are obtained through a Sigmod function.

[0062] It should be noted that the intra-class attention unit inputs multiple sample features within the same class, and focuses on intra-class information, which is equivalent to extracting intra-class common features. Those skilled in the art should understand that the category of the second embedded feature map refers to the belonging category of the sample image.

[0063] In some examples, the Bottleneck structure is composed of two fully connected layers.

[0064] In some examples, for two second embedded feature maps x1 and x2 belonging to the same class c1, referring to Figure 5 x1 and x2 are subjected to average pooling operation in the spatial dimension to obtain a representation of each channel (i.e., a first representation), and the mean representation f m and the variance representation f s, the correlation between channels is modeled by a Bottleneck structure, so that the number of output feature weights is the same as the number of input second embedding feature maps, two kinds of intra-class attention weights are generated, and the two kinds of attention weights are superimposed to obtain an intra-class attention weight w c The skilled person in the art should understand that the larger the intra-class attention weight is, the more important the channel of the category is. Multiplying the weight with each second embedding feature map can obtain the reconstructed feature map.

[0065] Further, the global attention unit is constructed based on an SE-Net; and the reconstructed feature map is input into the global attention unit to obtain channel measurement attention, which includes:

[0066] All reconstructed feature maps are input into the global attention unit to perform average pooling operation in the spatial dimension to obtain a second representation of each reconstructed feature map.

[0067] The mean representation and the variance representation of each second representation are calculated, and global attention weights are generated through a Bottleneck structure; wherein the Bottleneck structure is composed of a fully connected layer.

[0068] The channel measurement attention is obtained by superimposing all global attention weights and passing them through a Sigmod function.

[0069] It should be noted that the global attention unit inputs multiple sample features of different categories, and focuses on global information, weakens some noise or background information between different categories, enhances channel features more useful for classification, weakens similar channel features between different categories, and is equivalent to traversing all categories to extract inter-class common features, thereby reducing the influence of background features and inter-class difficult features on the classification result.

[0070] In some examples, for two reconstructed feature maps x1 and x2 belonging to different categories c1 and c2, referring to Figure 6 , average pooling operation is performed on x1 and x2 in the spatial dimension to obtain a representation (i.e., a second representation) of each channel, the mean representation f m and the variance representation f s are calculated, and two global attention weights are obtained through a Bottleneck structure, and the channel measurement attention 1-w g is obtained by superimposing the two global attention weights and passing them through a Sigmod function.

[0071] In some examples, the network structure of the intra-class attention unit and the global attention unit is the same, and the parameters are shared.

[0072] In an optional embodiment, the first embedding feature map, the reconstructed prototype and the channel measurement attention are taken as inputs of the measurement module to obtain a distance measurement result, including:

[0073] The first embedding feature map is unfolded and the feature dimension is reserved to obtain a first unfolded feature vector q = [q1, q2,..., q c ]; wherein the first embedding feature map q c = (q c1 ,..., q c(w×h) ) of the cth channel.

[0074] The reconstructed prototype is unfolded and the feature dimension is reserved to obtain a second unfolded feature vector s = [s1, s2,..., s c ]; wherein the reconstructed prototype s c = (s c1 ,..., s c(w×h) ) of the cth channel.

[0075] The first unfolded feature vector and the second unfolded feature vector are channel-weighted and measured by the channel measurement attention, and a distance measurement result D(S, Q) is obtained based on the Euclidean distance as follows:

[0076]

[0077] In the formula, w j represents the channel measurement attention of the jth channel, c, w and h respectively represent the channel number, width and height of the corresponding image, s i represents the ith feature point of the reconstructed prototype, and q i represents the ith feature point of the first embedding feature map.

[0078] It should be noted that in the above formula, the represents the feature points of two feature maps with a size of w*h, and the square is obtained after point-by-point subtraction.

[0079] In an optional embodiment, the training process of the prototype network includes:

[0080] The embedding module and the attention generation module are pre-trained using a specified general training data set to complete the parameter initialization of the embedding module and the attention generation module.

[0081] The parameters of the embedding module are fixed, the parameters of the attention generation module are fine-tuned using a training data set related to the LCD screen image, cross-entropy is used as a loss function, and the attention generation module parameters are updated through back propagation.

[0082] The optional embodiment adopts a pre-training and fine-tuning manner for training, which can reduce the training time of the prototype network, improve the performance of the prototype network, and adapt to the LCD screen defect classification task through a small number of rounds of fine-tuning of the attention generation module.

[0083] In some examples, the miniImageNet dataset is used as the specified general training dataset, so that the prototype network converges faster during training and improves its generalization ability.

[0084] In a preferred embodiment, the classification according to the distance measurement result generates an LCD screen defect classification result, which includes:

[0085] The Softmax classifier is used to predict according to the distance measurement result, and the class with the highest prediction probability is taken as the classification result of the corresponding query set image.

[0086] Those skilled in the art should understand that the support set image corresponds to different defect categories, and based on the distance measurement result of the query set image to the support set image, the class with the shortest distance to the query set image has the highest prediction probability, i.e. the corresponding defect category of the query set sample.

[0087] Embodiment 2

[0088] This embodiment eliminates an LCD screen defect classification system based on a fusion channel attention mechanism, which includes:

[0089] An image import module is used to obtain an LCD screen image to be classified and pre-process it, and the pre-processed LCD screen image is taken as a query set image; it is also used to obtain a sample image labeled with defect information and pre-process it, and the pre-processed sample image is taken as a support set image for small sample learning; the defect information includes the attribution category of the sample image;

[0090] An image processing module is used to carry a pre-trained prototype network; it is also used to process the query set image and the support set image to obtain the distance measurement result between the query set image and the support set image; wherein the prototype network is provided with an attention generation module, including a series of intra-class attention units and global attention units;

[0091] A classification module is used to classify according to the distance measurement result, and generate an LCD screen defect classification result.

[0092] It can be understood that the optional items in the above embodiment 1 are also applicable to this embodiment, so they will not be described here.

[0093] Embodiment 3

[0094] The embodiment provides a computer readable storage medium, and at least one instruction, at least one program, a code set or an instruction set is stored on the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method in the embodiment 1.

[0095] Exemplarily, the storage medium includes but is not limited to a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various storage medium capable of storing program codes.

[0096] Exemplarily, the instruction, the program, the code set or the instruction set can be implemented by using a programming language such as Java, Python, C++, R or Golang.

[0097] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device and the like, and is used for executing all or part of steps of the method in the embodiment 1.

[0098] It can be understood that the optional items in the above embodiment 1 are also applicable to the embodiment, and thus are not repeatedly described herein.

[0099] It can be understood that the device of the embodiment corresponds to the method in the above embodiment 1, and the optional items in the above embodiment 1 are also applicable to the embodiment, and thus are not repeatedly described herein.

[0100] The same or similar reference numerals correspond to the same or similar components;

[0101] The terms for describing the positional relationship in the drawings are only used for exemplary illustration, and should not be understood as a limitation on the patent;

[0102] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the implementation manner of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art, and each function module or unit can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. It is not necessary or possible to exhaust all the implementation manners. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A LCD screen defect classification method based on a fusion channel attention mechanism, characterized in that, The method comprises the following steps: acquire and pre-process LCD screen images to be classified, and take the pre-processed LCD screen images as query set images; acquire and pre-process sample images labeled with defect information, and take the pre-processed sample images as support set images for small sample learning; the defect information includes the attribution category of the sample images; input the query set images and the support set images into a pre-trained prototype network to obtain distance measurement results between the query set images and the support set images; wherein the prototype network is provided with an attention generation module, which comprises a class-intra attention unit and a global attention unit connected in series; classify according to the distance measurement results to generate LCD screen defect classification results; the prototype network further comprises an embedding module, a prototype generation module and a measurement module, and the attention generation module is arranged between the embedding module and the prototype generation module; the method further comprises the following steps: input the query set images and the support set images into the embedding module to respectively extract a first embedding feature map about the query set images and a second embedding feature map about the support set images; wherein the embedding module is constructed based on CONV4 or ResNet; input the second embedding feature map into the attention generation module to obtain a reconstructed feature map and a channel measurement attention fused with class-intra attention and global attention; input the reconstructed feature map and the channel measurement attention into the prototype generation module to obtain a reconstructed prototype; input the first embedding feature map, the reconstructed prototype and the channel measurement attention into the measurement module as inputs to obtain distance measurement results.

2. The LCD screen defect classification method based on fusion channel attention mechanism according to claim 1, characterized in that, the pre-processing comprises at least one of sliding window cropping, random cropping, rotation, stretching and mask; the pre-processing further comprises defect region cropping and random sampling of defective samples in the sample images and mixing the defective samples into non-defective samples; the pre-processing further comprises converting the images into tensor tensors and performing standardization processing thereon.

3. The LCD screen defect classification method based on fusion channel attention mechanism according to claim 1, characterized in that, the method further comprises the following steps: input the second embedding feature map into the class-intra attention unit as an input to obtain class-intra attention weights; reconstruct the corresponding second embedding feature map based on the broadcast mechanism according to the class-intra attention weights to obtain a reconstructed feature map; input the reconstructed feature map into the global attention unit to obtain a channel measurement attention.

4. The LCD screen defect classification method based on the fusion channel attention mechanism according to claim 3, characterized in that, the class-intra attention unit is constructed based on SE-Net; the method further comprises the following steps: input the second embedding feature map belonging to the same class into the class-intra attention unit to perform average pooling operation in the spatial dimension and obtain a first representation of each second embedding feature map; calculate the mean representation and the variance representation of each first representation to generate class-intra attention weights through a Bottleneck structure; wherein the Bottleneck structure is composed of a fully connected layer. The intra-class attention weights are superimposed and input into a Sigmoid function to obtain intra-class attention weights.

5. The LCD screen defect classification method based on fusion channel attention mechanism according to claim 3, characterized in that, The global attention unit is constructed based on an SE-Net. The reconstructed feature maps are input into the global attention unit to obtain channel measurement attention, including: All the reconstructed feature maps are input into the global attention unit to perform average pooling operation in the spatial dimension to obtain a second representation of each of the reconstructed feature maps. The mean representation and the variance representation of each second representation are calculated, and the global attention weights are generated through a Bottleneck structure; wherein the Bottleneck structure is composed of a fully connected layer. The global attention weights are superimposed and input into a Sigmoid function to obtain channel measurement attention.

6. The LCD screen defect classification method based on fusion channel attention mechanism according to claim 1, characterized in that, The first embedded feature map, the reconstructed prototype and the channel measurement attention are input into the distance measurement module to obtain distance measurement results, including: unfolding the first embedded feature map and keeping the feature dimension to obtain a first unfolded feature vector ; wherein the first embedded feature map of the cth channel ; unfolding the reconstruction prototype and retaining feature dimensions to obtain a second unfolded feature vector ; wherein the reconstruction prototype of the cth channel ; The first unfolded feature vector and the second unfolded feature vector are channel-weighted measured by using channel measurement attention, and a distance measurement result is obtained based on Euclidean distance As follows: In the formula, Indicates the first Channel attention is measured in channels, where c, w, and h represent the number of channels, width, and height of the corresponding image, respectively. The first part represents the reconstructed prototype. i One feature point, The first embedded feature map represents the first... i 1 feature point.

7. The LCD screen defect classification method based on fusion channel attention mechanism according to claim 1, characterized in that, The training process of the prototype network includes: The embedding module and the attention generation module are pre-trained using a specified general training data set to complete the parameter initialization of the embedding module and the attention generation module. The parameters of the embedding module are fixed, and the parameters of the attention generation module are fine-tuned using a training data set related to LCD screen images, using cross-entropy as a loss function, and updating the attention generation module parameters through back propagation.

8. The LCD screen defect classification method based on the fusion channel attention mechanism according to any one of claims 1-7, characterized in that, The distance measurement results are used for classification to generate LCD screen defect classification results, including: A Softmax classifier is used to predict the distance measurement results, and the class with the highest prediction probability is taken as the classification result of the corresponding query set image.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1-8.

Images