Surface defect classification method for glass thermal process based on depth residual error and attention enhancement mechanism

Abstract

The invention discloses a glass thermal process surface defect classification method based on a depth residual error and an attention enhancement mechanism, and belongs to the technical field of glass defect detection, and the method comprises the following steps: (1) in a glass thermal process, collecting a semi-finished product glass surface defect image sample, carrying out image processing, and adding a real label of a defect type to the sample, constructing to obtain a data set; (2) constructing a network model taking a deep residual network and an attention network as backbones, wherein the attention network comprises a channel attention enhancement module and a space attention enhancement module; performing supervised training based on a label prediction result and a real label on the network model by using the data set constructed in the step (1), and optimizing parameters of the network model; and detecting a semi-finished product glass surface defect image of an unknown category by using the network model optimized in the step (2) to realize defect category classification.

Description

Technical Field

[0001] The present invention relates to glass thermal process defect detection, and in particular to a surface defect classification method for glass thermal process based on depth residual and attention enhancement mechanism. Background Art

[0002] Electronic glass is one of the key basic materials for the information display industry, mainly including TFT-LCD glass substrates, OLED glass substrates, cover glass, flexible glass, etc. In recent years, the display industry has developed towards large size, ultra-high definition and thinness, which has put forward higher requirements on the quality of electronic glass substrates. Glass surface defect detection is a key part of quality inspection and is also one of the very important research topics in the field of industrial vision. Glass surface defect detection plays an important role in maintaining product quality, reducing scrap rate, and feeding back production information to improve production processes. Therefore, it is necessary to develop glass surface defect detection technology.

[0003] Existing glass defect detection methods can be divided into traditional manual visual inspection and machine vision inspection. Visual inspection relies on the experience of inspectors, lacks objectivity, has poor detection accuracy, and low execution efficiency. Traditional machine vision-based defect detection methods are mainly based on explicit feature extraction, which designs complex feature representations and then performs recognition and classification. Although good results have been achieved in specific tasks and scenarios, there are problems such as insufficient generalization ability and difficulty and complexity in defect feature extraction.

[0004] The glass thermal process is the upstream process for preparing display glass. Semi-finished glass obtained by the glass thermal process needs to be further processed before it can be obtained as display glass. The surface defect types of semi-finished glass obtained by the glass thermal process are significantly different from those of display glass, and the corresponding surface defect types may change during the further processing. In order to improve the quality of display glass and evaluate and guide the glass production process, it is necessary to perform surface defect detection on semi-finished glass obtained by the glass thermal process.

[0005] The Chinese patent document with publication number CN113298765A discloses a method for detecting surface defects of glass panels based on image processing and deep learning, which includes the following steps: S1 collects samples of surface defects of glass panels; S2 performs image processing on the samples collected in step S1; S3 uses the samples processed in step S2 to train a deep learning neural network model (based on the UNet model of EfficientNet) to obtain a trained deep learning neural network model for identifying surface defects of glass panels; S4 uses the trained deep learning neural network model to detect surface defects of glass panels. However, the invention is aimed at defect detection of glass post-processing, i.e., mobile phone glass cover plates. The corresponding surface defect categories are quite different from those in the glass thermal process. The invention uses image enhancement to expand data and improve the training effect of the model, but in fact its defect recognition ability will not be improved, and it is just repeated learning of the current features.

[0006] The Chinese patent document with publication number CN111652883A discloses a method for glass surface defect detection based on deep learning, which includes using residual network ResNet18 and residual network ResNet101 to train network models. After the network model training is completed, the trained network model is used to detect and distinguish defects in sample images, and defect specifications are controlled at the same time. This invention is also aimed at defect detection of finished glass products. Defect detection and discrimination are divided into two steps, which are implemented with two networks. The calculation speed is slow and occupies a lot of resources. In addition, the post-processing using traditional image algorithms is not very effective for glass thermal defects that are too small, similar in size, and highly similar.

[0007] Therefore, it is necessary to propose a method for detecting surface defects of semi-finished glass in thermal processes with high efficiency, high accuracy, high stability, low labor intensity and wide application range. Summary of the invention

[0008] In the glass thermal process, the surface defects of semi-finished glass are of various types and high similarity, which makes feature extraction difficult, and the small scale of defects causes insufficient single-scale feature semantic information, resulting in the inability to achieve efficient and accurate detection of its surface defects. The present invention provides a surface defect classification method for the glass thermal process based on deep residual and attention enhancement mechanism, which has high accuracy in defect classification.

[0009] The specific technical solutions adopted are as follows:

[0010] The present invention provides a surface defect classification method for glass thermal process based on depth residual and attention enhancement mechanism, comprising the following steps:

[0011] (1) During the glass thermal process, we collected defect image samples on the surface of semi-finished glass, processed the images, and added real labels of defect types to the samples to construct a dataset.

[0012] (2) constructing a network model, wherein the network model uses a deep residual network and an attention network as the backbone network model, and the attention network includes a channel attention enhancement module and a spatial attention enhancement module to enhance the discriminative learning ability of the deep residual network; using the data set constructed in step (1) to perform supervised training on the network model based on label prediction results and true labels, and optimize the network model parameters;

[0013] (3) Using the network model optimized in step (2), the surface defect images of semi-finished glass of unknown categories are detected to achieve defect type classification.

[0014] Specifically, in step (1), local clear images of surface defects of semi-finished glass in various types of glass thermal processes under different lighting methods are collected, so that a surface defect image of semi-finished glass in the sample corresponds to only one type of defect, and the defect types include: stones, tin spots, tin stains, bubbles, deformations, tin stains, etc.

[0015] Optionally, the image processing method in step (1) is: normalizing the pixel value of the collected semi-finished glass surface defect image from [0,255] to [0,1], further standardizing the data to the [-1,1] interval, and limiting the image size to 384×64 pixels.

[0016] Specifically, the network model structure includes a convolution layer, a channel attention enhancement module, a spatial attention enhancement module and a deep residual module; the convolution layer is used to perform a convolution operation on the input image to obtain a convolution layer output feature map; the channel attention enhancement module and the spatial attention enhancement module are used to increase the weight of the features of the convolution layer output feature map, and then the feature is input into the deep residual module, and the unit composed of the deep residual module, the channel attention enhancement module and the spatial attention enhancement module is repeatedly passed through the unit multiple times to extract the fine features of the surface defects (in the process of each repetition of the unit operation, the deep residual module is repeated separately to extract more feature information of the defect image), and the extracted fine features are classified.

[0017] The deep residual module includes a convolution transformation path and a self-mapping residual path. The convolution transformation path uses two convolution kernels to process the information of the surface defect image. The self-mapping residual path is used for self-residual learning to better preserve the shallow information of the surface defect image. The deep residual module is repeated to extract more feature information of the defect image.

[0018] The channel attention enhancement module first performs average pooling and maximum pooling on the input feature map (convolutional layer output feature map) to obtain two sets of channel description vectors. After the two sets of channel description vectors are convolved, they are activated by the ReLU function, and then convolved again and added element by element along the channel. Finally, they are activated by the Sigmoid function to generate a channel weight coefficient. Each eigenvalue in the input feature map is multiplied by the channel weight coefficient to obtain a new feature map, that is, a weighted feature map.

[0019] The spatial attention enhancement module first performs average pooling and maximum pooling on the feature map output by the channel attention enhancement module to obtain two sets of spatial description vectors, which are then concatenated (the values ​​of the same spatial position of each channel are concatenated), and then activated by the Sigmoid function after a convolution operation to generate a spatial weight coefficient (different weights are assigned to pixels at different positions). Each eigenvalue in the feature map input to the module is multiplied by the spatial weight coefficient to obtain a new feature map, i.e., a weighted feature map.

[0020] In step (2), the data set is divided into a training set and a test set in proportion, the training set is used to train the network model, and the test set is used to evaluate the goodness of the network model.

[0021] When the method of the present invention is actually applied, surface defect images of semi-finished glass on a glass thermal production line within a certain time period are collected, and defect image classification results are obtained through an optimized network model. According to the defect type distribution of the detection results, the glass production process is analyzed and evaluated. If a certain defect increases abnormally in the current time period, it indicates that there is a problem in a certain process in the current production process and the production process needs to be improved.

[0022] The present invention also provides a storage medium, in which a program is stored, wherein when the program is run, the surface defect classification method of the glass thermal process based on the depth residual and attention enhancement mechanism is executed.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] (1) The method of the present invention utilizes a convolutional neural network based on a deep residual and attention enhancement mechanism, pays attention to the importance of feature channels through a channel attention enhancement mechanism, and pays attention to the importance of spatial positions through a spatial attention enhancement mechanism, so that the model training focuses on defective parts and improves the accuracy of grasping defective semi-finished glass. On the one hand, the network model of the present invention occupies less computing resources, has fast calculation and high efficiency; on the other hand, the method of the present invention improves the defect classification accuracy of the test set to 93.5%, and the defect classification accuracy of the entire initial data set reaches 99.4%.

[0025] (2) The method of the present invention can realize automatic machine detection of defects on the surface of semi-finished glass and identification of defect types, and analyze, evaluate and guide the glass production process based on the defect type distribution of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flow chart of the surface defect classification method for glass thermal process based on depth residual and attention enhancement mechanism;

[0027] Figure 2 This is a representative image of surface defects on semi-finished glass;

[0028] Figure 3 This is a schematic diagram of the structure of the constructed network model;

[0029] Figure 4 It is a schematic diagram of the structure of the deep residual module;

[0030] Figure 5 This is a schematic diagram of the structure of the channel attention enhancement module;

[0031] Figure 6 Schematic diagram of the structure of the spatial attention enhancement module. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with the embodiments and drawings. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0033] Example

[0034] In the prior art, Convolution Neural Network (CNN) has shown absolute advantages in solving image classification and target detection problems, and therefore has also received great attention in the field of defect detection. Convolutional neural networks can realize automatic feature extraction, learn robust and high-level feature representation of images, and then obtain feature data of images. However, in the glass thermal process, the size of glass defects is small (<80μm), the shape is complex, and the similarity is high. The detection effect of glass surface defects using convolutional neural networks is not good. Therefore, the present invention has developed a surface defect classification method for glass thermal process based on deep residual and attention enhancement mechanism. The flow chart of this method is as follows: Figure 1 As shown, the specific steps include:

[0035] (1) During the glass thermal process, we collected defect image samples on the surface of semi-finished glass, processed the images, and added real labels of defect types to the samples to construct a dataset.

[0036] Specifically: The image acquisition device of the ISRA defect detection equipment is used to scan the semi-finished glass on the conveyor belt to obtain the glass surface image. The data collected by the image acquisition device is a local clear image of the defects in the glass thermal process (multiple images are taken at the same defect under different lighting methods). The defect categories are manually labeled. A semi-finished glass surface defect image corresponds to only one defect. The defect types include stones, tin spots, tin stains, tin stains, bubbles and deformations. The defect images are placed in six folders according to their categories. Representative semi-finished glass surface defect images are as follows: Figure 2 As shown;

[0037] The pixel values ​​of the collected semi-finished glass surface defect images are normalized from [0, 255] to [0, 1], and the data are further standardized to the [-1, 1] interval, and the image size is limited to 384 × 64 pixels.

[0038] (2) constructing a network model, wherein the network model uses a deep residual network and an attention network as the backbone network model, and the attention network includes a channel attention enhancement module and a spatial attention enhancement module to enhance the discriminative learning ability of the deep residual network; using the data set constructed in step (1) to perform supervised training on the network model based on label prediction results and true labels, and optimize the network model parameters;

[0039] Build a network model based on the Pytorch framework, such as Figure 3 As shown, in summary, the network model structure includes a convolution layer, a channel attention enhancement module, a spatial attention enhancement module and a deep residual module; the convolution layer is used to perform a convolution operation on the input image to obtain a convolution layer output feature map; the channel attention enhancement module and the spatial attention enhancement module are used in turn to increase the weight of the features of the convolution layer output feature map, which is then input into the deep residual module, and the unit composed of the deep residual module, the channel attention enhancement module and the spatial attention enhancement module is repeatedly passed through the unit multiple times to extract fine features of surface defects, and the extracted fine features are classified.

[0040] Specifically, the structural diagram of the deep residual module is as follows Figure 4 As shown in the figure, it includes a convolution transformation path (on the left side of the figure) and a self-mapping residual path (on the right side of the figure). The convolution transformation path uses two convolution kernels to process the information of the defect image on the glass surface. The self-mapping residual path is used for self-residual learning, so that it can better preserve the shallow information of the defect image. ReLU and LeakyReLU are both activation functions, which are used after convolution to increase the nonlinear layer of the model.

[0041] Specifically, the schematic diagram of the channel attention enhancement module is as follows: Figure 5As shown in the figure, firstly, the input feature map (convolutional layer output feature map) is average pooled and maximum pooled to obtain two sets of channel description vectors, simplify the information of different channels, and then through the operations of convolution, activation, and convolution, the difference of information of different channels is increased, that is, the two sets of channel description vectors are convolved, activated by the ReLU function, and then convolved again and added element by element along the channel, and finally activated by the Sigmoid function to generate the channel weight coefficient (different weights are given to the feature maps of different channels), and finally the value of the weight coefficient is activated to be between [0,1], and each feature value in the input feature map of the module is multiplied by the channel weight coefficient to obtain a weighted feature map.

[0042] Specifically, the schematic diagram of the spatial attention enhancement module is as follows: Figure 6 As shown in the figure, the spatial attention enhancement module first performs average pooling and maximum pooling on the feature map output by the channel attention enhancement module to obtain two sets of spatial description vectors, simplify the information of different spatial pixels, concatenate the two sets of spatial description vectors in the channel dimension (cat), and then after the convolution operation, reduce them to one channel, activate them through the Sigmoid function, generate a spatial weight coefficient (different weights are given to pixels at different positions), make the value of the weight coefficient between [0,1], multiply each eigenvalue in the feature map input by the module by the spatial weight coefficient, and obtain a weighted feature map.

[0043] It can be understood that the present application uses a channel attention enhancement mechanism to assign different weights to feature maps of different channels; and uses a spatial attention enhancement mechanism to assign different weights to pixels at different positions. The two are performed successively, so that the model training focuses on the defective part and improves the accuracy of defective glass capture.

[0044] Specifically, the deep residual module is repeated to extract more features of the image. The unit composed of the deep residual module, the channel attention enhancement module and the spatial attention enhancement module is repeated 4 times to extract fine features of surface defects. In each repetition, the deep residual module is repeated 3, 4, 6 and 3 times respectively.

[0045] More specifically, in the network model, a batch normalization layer is connected after the convolution layer to ensure the effectiveness of the transmitted feature information. The classification loss is calculated based on the multi-classification cross entropy loss function, the class probability is calculated using the Logits loss function, and the Logit score is softmaxed to obtain the predicted category corresponding to the defect image. A custom optimizer grouping method is used, and different optimization methods are used for weights (using weight decay), biases, and BN layers (using SGD). The cosine annealing learning rate is used to reduce the learning rate through the cosine function, and the sample batch size is 64. The model structure is shown in the following table.

[0046] Table 1 Model structure

[0047]

[0048] The data set is divided into a training set and a test set in a ratio of 8:2. The training set is used to train the network model, and the test set is used to evaluate the goodness of the network model.

[0049] (3) Using the network model optimized in step (2), the surface defect images of semi-finished glass of unknown categories are detected to achieve defect type classification.

[0050] It can be understood that by inputting the defect images of semi-finished glass collected in a certain time period into the defect detection model for identification and classification, and performing defect detection through the pre-trained defect recognition model, efficient and accurate detection of surface defects in the glass thermal process can be achieved, and the glass production process can be analyzed and evaluated based on the defect type distribution of the detection results.

[0051] The embodiments described above provide a detailed description of the technical solutions of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements or similar substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A surface defect classification method for glass thermal process based on depth residual and attention enhancement mechanism, characterized in that: The following steps are involved: (1) During the glass thermal process, we collected defect image samples of semi-finished glass surfaces, processed the images, and added true labels of defect types to the samples to construct a dataset. (2) constructing a network model, wherein the network model uses a deep residual network and an attention network as the backbone network model, and the attention network includes a channel attention enhancement module and a spatial attention enhancement module to enhance the discriminative learning ability of the deep residual network; Using the data set constructed in step (1), the network model is supervised and trained based on the label prediction results and the real labels to optimize the network model parameters; (3) Using the network model optimized in step (2), the surface defect images of semi-finished glass of unknown categories are detected to achieve defect type classification.

2. The surface defect classification method of glass thermal process based on depth residual and attention enhancement mechanism according to claim 1 is characterized in that: One image of a semi-finished glass surface defect in the sample corresponds to only one type of defect, and the defect types include stones, tin spots, tin stains, bubbles, deformations, or tin stains.

3. The surface defect classification method of glass thermal process based on depth residual and attention enhancement mechanism according to claim 1 is characterized in that: The image processing method in step (1) is: normalize the pixel value of the collected semi-finished glass surface defect image from [0, 255] to [0, 1], further standardize the data to the [-1, 1] interval, and limit the image size to 384×64 pixels.

4. The surface defect classification method of glass thermal process based on depth residual and attention enhancement mechanism according to claim 1 is characterized in that: The network model structure includes a convolution layer, a channel attention enhancement module, a spatial attention enhancement module and a deep residual module; the convolution layer is used to perform a convolution operation on the input image to obtain a convolution layer output feature map; the channel attention enhancement module and the spatial attention enhancement module are used to increase the weight of the features of the convolution layer output feature map, and then it is input into the deep residual module, and the unit composed of the deep residual module, the channel attention enhancement module and the spatial attention enhancement module is repeatedly passed through the unit multiple times to extract the fine features of the surface defects, and the extracted fine features are classified.

5. The surface defect classification method of glass thermal process based on depth residual and attention enhancement mechanism according to claim 1 is characterized in that: The deep residual module includes a convolution transformation path and a self-mapping residual path. The convolution transformation path uses two convolution kernels to process the information of the surface defect image, and the self-mapping residual path is used for self-residual learning. The deep residual module is repeated to extract more feature information of the defect image.

6. The surface defect classification method of glass thermal process based on depth residual and attention enhancement mechanism according to claim 1 is characterized in that: The channel attention enhancement module first performs average pooling and maximum pooling on the input feature map to obtain two sets of channel description vectors. The two sets of channel description vectors are convolved and activated by the ReLU function. Then, they are convolved again and summed element by element along the channel. Finally, they are activated by the Sigmoid function to generate a channel weight coefficient. Each eigenvalue in the input feature map of this module is multiplied by the channel weight coefficient to obtain a weighted feature map.

7. The surface defect classification method of glass thermal process based on depth residual and attention enhancement mechanism according to claim 1 is characterized in that: The spatial attention enhancement module first performs average pooling and maximum pooling on the feature map output by the channel attention enhancement module to obtain two sets of spatial description vectors. The two sets of spatial description vectors are concatenated and then activated by the Sigmoid function after a convolution operation to generate a spatial weight coefficient. Each eigenvalue in the feature map input to the module is multiplied by the spatial weight coefficient to obtain a weighted feature map.

8. The surface defect classification method of glass thermal process based on depth residual and attention enhancement mechanism according to claim 1 is characterized in that: The data set is divided into a training set and a test set in proportion, the training set is used to train the network model, and the test set is used to evaluate the goodness of the network model.

9. A storage medium, characterized in that: The storage medium stores a program, wherein when the program is run, the surface defect classification method for glass thermal process based on depth residual and attention enhancement mechanism described in any one of claims 1-8 is executed.

Images