A glass defect detection method based on variational autoencoder
Through the combination of variational autoencoder, projection method and MAD method, unsupervised glass defect detection is achieved, solving the problem of detecting complex and variable glass defect types in the existing technology, reducing training costs, and improving detection accuracy and robustness.
Patent Information
- Application Number
- CN202110716109.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-25
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2041-06-25
AI Technical Summary
The existing glass defect detection methods rely on manual or traditional feature extraction, and cannot effectively detect complex and variable glass defect types. The deep learning-based methods are expensive to train, require a large number of labeled samples, and the model generalization ability is insufficient.
Using a detection method based on a variational autoencoder, the grayscale image of glass products is reconstructed through the VAE reconstruction model, and the projection method and MAD method are used to remove noise, so as to realize unsupervised defect detection and detect various types of glass defects. .
It realizes multiple glass defect detection under unsupervised conditions, reduces training costs, improves the robustness and accuracy of the detection, and can detect multiple glass defect types without labeling and training on defect samples.
Smart Images
Figure CN113344903B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a glass defect detection method based on a variational autoencoder, and belongs to the technical field of glass defect detection. Background Art
[0002] As a common industrial material, glass products may have various defects during the production and processing, such as bubbles, scratches, dirt, cracks, uneven edges, and burrs. These defects affect the performance and appearance of glass products. Therefore, it is necessary to set up a detection mechanism in the glass production process.
[0003] At present, most methods use manual or computer vision to detect glass defects. Manual detection is time-consuming and labor-intensive, and does not conform to the development trend of industrial intelligence; in the field of computer vision, defect detection methods can generally be divided into two categories. The first category is the traditional feature extraction method. For defects of a specified category, different operators can be set manually to extract features and then detect them. However, the feature extraction operators of this type of method are too dependent on people's subjective experience, and the types of defects produced cannot be predicted during the glass production process. The second category is a method based on deep learning, which uses convolutional neural networks to automatically extract the features of defects. These features have better versatility and transferability than the features extracted by traditional methods, but this feature extraction is mainly learned in a data-driven way, and obtaining labeled samples requires a lot of manpower.
[0004] In summary, it can be seen that the manual detection method obviously cannot meet the current requirements for glass defect detection. Both the traditional feature extraction method and the deep learning-based detection method have some problems in glass defect detection. Among them, the feature extraction operator based on the traditional feature extraction detection method is too dependent on human subjective experience, and this method is only effective for a limited number of defect detection categories. With the popularization of deep technology, the deep learning-based method can accurately detect different types of defects by manually annotating samples, but this is a method with high training costs. It requires a large amount of defect data of different categories for training, which is often difficult in practical problems. If there is a lack of sufficient abnormal data information, the generalization ability of the model will be affected and it cannot meet the requirements of detecting complex and changeable glass defect types. Summary of the invention
[0005] In order to solve the above problems, the present invention provides a glass defect detection method based on a variational autoencoder, the method comprising:
[0006] Acquire a grayscale image of the glass product to be inspected;
[0007] Input the grayscale image of the glass product to be inspected into the trained VAE reconstruction model to obtain a reconstructed image;
[0008] Compare the pixels of the reconstructed image with the original test image, find the different areas, and obtain the difference segmentation map;
[0009] Use projection method and MAD method to remove noise points in the difference segmentation map to obtain a final segmentation map;
[0010] Whether the glass product to be inspected has defects is determined based on whether there are defective pixels in the final segmentation image.
[0011] Optionally, the VAE reconstruction model includes an encoder and a decoder; the encoder of the VAE reconstruction model encodes the grayscale image into a latent variable Z, and the decoder reconstructs a reconstructed image similar to the grayscale image according to the latent variable Z;
[0012] The encoder is composed of a three-layer neural network, the input layer has 16384 neurons, the middle layer has 400 neurons, and the output layer is composed of two parallel fully connected neural networks; the two parallel fully connected neural networks of the output layer output a 20-dimensional mean feature vector and a 20-dimensional variance feature vector according to the grayscale image of the glass product to be detected input by the input layer;
[0013] The decoding network consists of two layers of neural networks, the first layer has 400 neurons, and the second layer has 16384 neurons, and finally outputs a segmented binary image, that is, a reconstructed image similar to a grayscale image.
[0014] Optionally, the encoder encodes the grayscale image into a latent variable Z that obeys a d-dimensional Gaussian distribution:
[0015]
[0016] The mean of the distribution is μ z , the variance is I is the feature matrix sampled from the N(0,1) distribution, and the latent variable Z is expressed as:
[0017]
[0018] The mean μ of each latent variable z and standard deviation
[0019] μ z =f1(x i ; W μ ) (3)
[0020] σ z =f2(x i ; W σ ) (4)
[0021] Among them, x i Represents a grayscale image; W μ represents the learnable weights of the fully connected neural network that outputs the mean feature vector in the output layer of the encoder; W σ represents the learnable weights of the fully connected neural network that outputs the variance eigenvector in the output layer of the encoder; f1 represents the Sigmoid nonlinear activation function of the fully connected neural network that outputs the mean eigenvector in the output layer, and f2 represents the Sigmoid nonlinear activation function of the fully connected neural network that outputs the variance eigenvector in the output layer of the encoder;
[0022] The decoder generates a reconstructed image by sampling from the latent variable Z through the decoding network
[0023]
[0024] Among them, z i represents the feature of the i-th dimension of the latent variable Z, W γ represents the learnable weights of the decoder neural network; g represents the Sigmoid nonlinear activation function of the decoder neural network.
[0025] Optionally, the method further includes training a VAE reconstruction model; during the training process, the total loss function of the VAE reconstruction model is defined as:
[0026] L T =L D +L KL (6)
[0027] Where L T is the total loss function, L D is the reconstruction loss, L KL is the KL divergence between the sample data distribution used to train the VAE reconstruction model and the standard normal distribution; where L D The reconstruction loss is the mean square error loss between the reconstructed image and the input grayscale image:
[0028]
[0029] KL divergence L between the sample data distribution and the standard normal distribution used to train the VAE reconstruction model KL for:
[0030]
[0031] Where d is the dimension of the latent variable Z, μ (j) and σ (j) They represent the j-th component of the mean vector and variance vector of the Gaussian distribution respectively; the training is completed by minimizing the loss function using the back propagation algorithm.
[0032] Optionally, the sample data used to train the VAE reconstruction model is flawless image data of glass products; before training, the sample data is expanded by randomly flipping, randomly rotating, and randomly changing brightness and darkness.
[0033] Optionally, the using of the projection method and the MAD method to remove noise points in the difference segmentation map to obtain a final segmentation map includes:
[0034] The pixels of the defective area in the difference segmentation map are projected to the x and y directions respectively using the projection method and the pixel values are counted. The outliers are found using the MAD method. The calculation method of MAD is as follows:
[0035]
[0036] X k ' is the adjusted value, parameter X m is the median of all elements; bias m =|X k -X m | is the median of the absolute deviations between all elements and the median; parameter MAD is the median of absolute deviations, MAD = bias m ; Parameter λ is the adjustment variable.
[0037] When the pixel projection value in the x or y direction is not in [X m -λMAD,X m +λMAD] interval, the pixel point is determined as a noise point, and the points other than the noise points in the x-direction and y-direction in the segmentation map are removed respectively to obtain the final segmentation map after removing the noise interference.
[0038] Optionally, during training, the adjustment variable λ is set to 3.
[0039] Optionally, during the training process, the sample data used to train the VAE reconstruction model is a grayscale image of 128*128 pixels.
[0040] The present application also provides an application method of the above-mentioned glass defect detection method based on variational autoencoder in the field of glass production technology, and the method uses the above-mentioned method to detect whether the produced glass has defects.
[0041] The beneficial effects of the present invention are:
[0042] By using variational autoencoders to detect various defects in glass products under unsupervised conditions, the essence of variational autoencoders is to generate models. By modeling glass flawless samples through VAE encoders, random samples can be reconstructed to obtain reconstructed images close to flawless samples. During detection, the defective glass image is input into the trained VAE encoding and decoding network to generate a reconstructed image close to the flawless sample, and then the difference pixels, i.e. the defective area, are determined by comparing the difference area between the reconstructed image and the original image. There will be noise interference in the obtained defective area. The present invention reduces noise interference by combining the projection method with the MAD method. For the noise existing in the VAE model generation process, the non-target area is removed by combining the projection method with the MAD method to improve the segmentation accuracy. The method of the present application detects defects by adopting an unsupervised learning method, and there is no need to label the defective samples for training, which meets the needs of actual industrial production. Moreover, the generated model has a certain robustness and can detect a variety of glass defect types. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0044] Figure 1 The present invention is a flow chart of a glass defect detection method based on a variational autoencoder.
[0045] Figure 2 Schematic diagram of the VAE method in an embodiment of the present invention.
[0046] Figure 3 This is a picture of a flawless glass sample used for VAE training in the present invention.
[0047] Figure 4 This is a decoded and reconstructed diagram of the test sample in the present invention.
[0048] Figure 5 Difference segmentation map between the VAE decoded reconstruction image and the test image.
[0049] Figure 6 This is the segmentation result after removing background interference.
[0050] Figure 7 Segmentation results of different glass defect types. DETAILED DESCRIPTION
[0051] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0052] Embodiment 1:
[0053] This embodiment provides a glass defect detection method based on a variational autoencoder, the method comprising:
[0054] Acquire a grayscale image of the glass product to be inspected;
[0055] Input the grayscale image of the glass product to be inspected into the trained VAE reconstruction model to obtain a reconstructed image;
[0056] Compare the pixels of the reconstructed image with the original test image, find the different areas, and obtain the difference segmentation map;
[0057] Use projection method and MAD method to remove noise points in the difference segmentation map to obtain a final segmentation map;
[0058] Whether the glass product to be inspected has defects is determined based on whether there are defective pixels in the final segmentation image.
[0059] The VAE reconstruction model includes two parts: an encoder and a decoder; the encoder of the VAE reconstruction model encodes the grayscale image into a latent variable Z, and the decoder reconstructs the latent variable Z to obtain a reconstructed image similar to the grayscale image;
[0060] The encoder is composed of a three-layer neural network, the input layer has 16384 neurons, the middle layer has 400 neurons, and the output layer is composed of two parallel fully connected neural networks; the two parallel fully connected neural networks of the output layer output a 20-dimensional mean feature vector and a 20-dimensional variance feature vector according to the grayscale image of the glass product to be detected input by the input layer;
[0061] The decoding network consists of two layers of neural networks, the first layer has 400 neurons, and the second layer has 16384 neurons, and finally outputs a segmented binary image, that is, a reconstructed image similar to a grayscale image.
[0062] Embodiment 2:
[0063] This embodiment provides a glass defect detection method based on a variational autoencoder, see Figure 1 , the method comprising:
[0064] Step 1: Image data set division: 17,000 collected 128*128 pixel grayscale images of glass products are divided into training data set and test data set. The training data set uses 1,000 flawless image data, and the test data set contains flawless image data and 16,000 images with various defects.
[0065] Step 2: Train the reconstruction model based on VAE, and use the back propagation algorithm to minimize the loss function composed of the reconstruction error and the KL divergence between distributions to complete the training, obtain the VAE model and save it. In order to increase the generalization ability of the model, the data enhancement method is used here for the training data.
[0066] Furthermore, the VAE model used in step 2 is specifically:
[0067] The variational autoencoder consists of two parts: the encoder and the decoder. The encoder encodes the image into a latent variable Z through a neural network. The variable obeys a Gaussian distribution, which produces two vectors: one is the mean vector and the other is the standard deviation vector. Z obeys a d-dimensional Gaussian distribution:
[0068]
[0069] The mean of the distribution is μ z , the variance is I is the feature matrix sampled from the N(0,1) distribution, and the latent variable Z is expressed as:
[0070]
[0071] The decoder reconstructs the latent variable Z into a new sample similar to the original image. Structurally, the output layer of the encoding network of the variational autoencoder consists of two parallel fully connected neural networks. The network obtains the mean μ of each latent variable through learning. z and standard deviation
[0072] μ z =f1(x i ; W μ ) (3)
[0073] σ z =f2(x i ; W σ ) (4)
[0074] Among them, x i Represents a grayscale image; W μ represents the learnable weights of the fully connected neural network that outputs the mean feature vector in the output layer of the encoder; W σrepresents the learnable weights of the fully connected neural network that outputs the variance eigenvector in the output layer of the encoder; f1 represents the Sigmoid nonlinear activation function of the fully connected neural network that outputs the mean eigenvector in the output layer, and f2 represents the Sigmoid nonlinear activation function of the fully connected neural network that outputs the variance eigenvector in the output layer of the encoder.
[0075] The decoder generates a reconstructed image by sampling from the latent variable Z through the decoding network:
[0076]
[0077] Among them, z i represents the feature of the i-th dimension of the latent variable Z, W γ represents the learnable weights of the decoder neural network; g represents the Sigmoid nonlinear activation function of the decoder neural network.
[0078] The total loss function is defined as:
[0079] L T =L D +L KL (6)
[0080] Where L T is the total loss function, L D is the reconstruction loss, L KL is the KL divergence between the sample data distribution and the standard normal distribution. D The reconstruction loss is the mean square error loss between the reconstructed sample and the original sample pixel:
[0081]
[0082] Where L KL is the KL divergence between the sample data distribution and the standard normal distribution:
[0083]
[0084] Where d is the dimension of the latent variable Z, μ (j) and σ (j) They represent the j-th component of the mean vector and variance vector of the Gaussian distribution respectively; the training is completed by minimizing the loss function using the back propagation algorithm.
[0085] The encoding network used in step 2 consists of a three-layer neural network. The input image is a 128*128 single-channel grayscale image, that is, the input layer has 16384 neurons, the middle layer has 400 neurons, and the output layer consists of two parallel fully connected neural networks, one outputs a 20-dimensional mean feature vector, and the other outputs a 20-dimensional variance feature vector. The decoding network consists of two layers of neural networks, the first layer has 400 neurons, the second layer has 16384 neurons, and finally outputs a 128*128 segmented binary image. The VAE principle diagram is as follows Figure 2 shown.
[0086] The specific method of using data enhancement in step 2 is: in order to increase the generalization performance of the model, random flipping, random rotation, random light and dark changes and other transformation methods are used on the training data, which is equivalent to expanding the training data.
[0087] Step 3: Use the VAE model to infer the test data and obtain the reconstructed image.
[0088] In the model inference stage, the test image can be reconstructed through the trained VAE model. The obtained reconstructed image and the original test image are as follows: Figure 3 , Figure 4 shown.
[0089] Step 4: Compare the pixels of the reconstructed image with the original test image to find the different areas and obtain the difference segmentation map as shown in Figure 5 shown.
[0090] Step 5: For the noise points in the difference segmentation map, use the projection method and MAD (Median Absolute Deviation) method to remove the interference of the noise points.
[0091] The projection method projects the pixels of the defective area in the segmentation map to the x and y directions respectively and counts the pixel values. The MAD method is used to find outliers (i.e. noise points), and the noise points in the non-defective area are removed to obtain the final segmentation map. If there are no defective pixels in the final segmentation map, the original image is judged to be defect-free.
[0092] The specific process of using the projection method in step 5 and selecting the threshold value using MAD to remove noise is as follows: for the segmented binary image obtained, the pixel statistics in the x direction and y direction are calculated for each pixel, and the MAD method is used to obtain its outliers. In statistics, MAD is a robust measure of sample deviation of univariate numerical data. This method determines whether each element is an outlier by finding whether the deviation between an element and the median is within a reasonable range. The calculation method of MAD is as follows:
[0093]
[0094] X k ' is the adjusted value, parameter X m is the median of all elements. bias m =|X k -X m | is the median of the absolute deviations between all elements and the median. The parameter MAD is the median of the absolute deviations, MAD = bias m The parameter λ is a regulating variable, and here λ is set to 3. In this method, when the pixel projection value in the x direction or y direction is not in [X m -λMAD,X m +λMAD] interval, these points are judged as outliers, and the points other than the outliers in the x-direction and y-direction of the segmentation map are removed respectively to obtain the result after removing the noise interference. The final segmentation result is shown in the figure below. Figure 6 shown.
[0095] The evaluation index of the model is composed of the weighted miss detection rate (escape), overkill rate (overkill) and mean intersection over union (mIOU). Among them, the miss detection rate refers to the percentage of samples with defects that are not detected in the total number of all defective samples; the overkill rate refers to the percentage of flawless samples that are detected as flawless in the total number of all flawless samples; the miss detection rate and overkill rate are defective or flawless detection indicators, which examine the defect classification detection ability. The mean intersection over union examines the overlap accuracy of the defect area between the segmented image and the original image. The score calculation formula is as follows:
[0096] score=0.5*(1-escape)+0.3*(1-overkill)+0.2*(mIOU) (10)
[0097] The method provided in this application can detect different types of glass defects. After testing 16,000 test images, the final score is 93. The detection results for different defects are as follows: Figure 7 As shown; Figure 7 In the figure, a is the original detection image of the inside of the glass surface, which contains various types of defects inside the glass, and b is the detection result image corresponding to a; c is the original detection image of the edge of the glass surface, which contains various types of defects on the edge of the glass, and d is the detection result image corresponding to c. It can be seen that this method can accurately detect the defect areas inside and on the edge of the glass, and the model has good robustness in detecting different types of glass defects.
[0098] The results show that the glass defect detection method based on variational autoencoder can achieve good detection effect in an unsupervised learning mode. The model has a certain robustness and can detect a variety of glass defect types. The VAE model trained with flawless samples can simultaneously complete the recognition and segmentation tasks. For the noise existing in the VAE model generation process, the non-target area is removed by combining the projection method and the MAD method to improve the segmentation accuracy.
[0099] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.
[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A glass defect detection method based on variational autoencoder, characterized in that: The method comprises: Acquire a grayscale image of the glass product to be inspected; Input the grayscale image of the glass product to be inspected into the trained VAE reconstruction model to obtain a reconstructed image; Compare the reconstructed image with the original test image by pixel, find the different areas, and obtain the difference segmentation map; Use projection method and MAD method to remove noise points in the difference segmentation map to obtain a final segmentation map; Determine whether the glass product to be inspected has defects according to whether there are defective pixels in the final segmentation image; The use of the projection method and the MAD method to remove the noise points existing in the difference segmentation map to obtain the final segmentation map includes: The pixels of the defective area in the difference segmentation map are projected to the x and y directions respectively using the projection method and the pixel values are counted. The outliers are found using the MAD method. The calculation method of MAD is as follows: X k ' is the adjusted value, parameter X m is the median of all elements; bias m =|X k -X m | is the median of the absolute deviations between all elements and the median; parameter MAD is the median of absolute deviations, MAD = bias m ;Parameter λ is the adjustment variable; When the pixel projection value in the x or y direction is not in [X m -λMAD,X m +λMAD] interval, the pixel point is determined as a noise point, and the points other than the noise points in the x-direction and y-direction in the segmentation map are removed respectively to obtain the final segmentation map after removing the noise interference.
2. The method according to claim 1, characterized in that The VAE reconstruction model includes two parts: an encoder and a decoder; the encoder of the VAE reconstruction model encodes the grayscale image into a latent variable Z, and the decoder reconstructs the latent variable Z to obtain a reconstructed image similar to the grayscale image; The encoder is composed of a three-layer neural network, the input layer has 16384 neurons, the middle layer has 400 neurons, and the output layer is composed of two parallel fully connected neural networks; the two parallel fully connected neural networks in the output layer output a 20-dimensional mean feature vector and a 20-dimensional variance feature vector according to the grayscale image of the glass product to be detected input into the input layer.
3. The method according to claim 2, characterized in that The encoder encodes the grayscale image into a latent variable Z that follows a d-dimensional Gaussian distribution: The mean of the distribution is μ z , the variance is I is the feature matrix sampled from the N(0,1) distribution, and the latent variable Z is expressed as: The mean μ of each latent variable z and standard deviation μ z =f1(x i ;W μ ) (3) σ z =f2(x i ;W σ ) (4) Among them, x i Represents a grayscale image; W μ represents the learnable weights of the fully connected neural network that outputs the mean feature vector in the output layer of the encoder; W σ represents the learnable weights of the fully connected neural network that outputs the variance eigenvector in the output layer of the encoder; f1 represents the Sigmoid nonlinear activation function of the fully connected neural network that outputs the mean eigenvector in the output layer, and f2 represents the Sigmoid nonlinear activation function of the fully connected neural network that outputs the variance eigenvector in the output layer of the encoder; The decoder generates a reconstructed image by sampling from the latent variable Z through the decoding network Among them, z i represents the feature of the i-th dimension of the latent variable Z, W γ represents the learnable weights of the decoder neural network; g represents the Sigmoid nonlinear activation function of the decoder neural network; The decoding network consists of two layers of neural networks, the first layer has 400 neurons, and the second layer has 16384 neurons, and finally outputs a segmented binary image, that is, a reconstructed image similar to a grayscale image.
4. The method according to claim 3, characterized in that The method further includes training the VAE reconstruction model; during the training process, the total loss function of the VAE reconstruction model is defined as: L T =L D +L KL (6) Where L T is the total loss function, L D is the reconstruction loss, L KL is the KL divergence between the sample data distribution used to train the VAE reconstruction model and the standard normal distribution; where L D The reconstruction loss is the mean square error loss between the reconstructed image and the input grayscale image: KL divergence L between the sample data distribution and the standard normal distribution used to train the VAE reconstruction model KL for: Where d is the dimension of the latent variable Z, μ (j) and σ (j) They represent the j-th component of the mean vector and variance vector of the Gaussian distribution respectively; the training is completed by minimizing the loss function using the back propagation algorithm.
5. The method according to claim 4, characterized in that The sample data used to train the VAE reconstruction model is flawless image data of glass products; before training, the sample data is expanded by randomly flipping, randomly rotating, and randomly changing light and dark.
6. The method according to claim 5, characterized in that During training, the adjustment variable λ is set to 3.
7. The method according to claim 6, characterized in that During the training process, the sample data used to train the VAE reconstruction model is a grayscale image of 128*128 pixels.
8. An application method of the glass defect detection method based on variational autoencoder as described in any one of claims 1 to 7 in the field of glass production technology, wherein the application method uses the above method to detect whether the produced glass has defects.