Defect detection method and equipment based on unsupervised learning

By adding artificial noise to normal images to generate abnormal images, and using unsupervised learning methods to train the defect detection network, the problem of unbalanced positive and negative samples in supervised learning is solved, and the accuracy and adaptability of defect detection are improved.

CN114862838BActive Publication Date: 2024-11-26SHENZHEN HUAHAN WEIYE TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210623633.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-02
Publication Date
2024-11-26
Estimated Expiration
2042-06-02

AI Technical Summary

Technical Problem

The existing defect detection method based on supervised learning is due to the scarcity of defective product images, which leads to unbalanced positive and negative samples in the training center, resulting in poor detection results.

Method used

Using an unsupervised learning method, an abnormal image is generated by adding artificial noise to normal images, and a training data set is constructed, and the teacher network, feature compression network, student network and projection network are used for training. The training goal is to minimize normal image features and reconstruction. The distance between features and the difference between abnormal image features and reconstruction features avoid leakage of feature representation capabilities.

Benefits of technology

It realizes that without manual annotation, the detection effect of the defect detection network is improved, false detection is reduced, and the accuracy of defect detection is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114862838B_ABST
    Figure CN114862838B_ABST
Patent Text Reader

Abstract

The present invention provides a defect detection method and device based on unsupervised learning, the method comprising: obtaining a normal image of a product, adding artificial noise to the normal image to generate an abnormal image paired with the normal image; training a defect detection network according to the normal image and the abnormal image paired with the normal image to obtain a trained defect detection network; using the trained defect detection network to perform defect detection on the image to be detected of the product to obtain a defect detection result. By adding artificial noise to the normal image to obtain the abnormal image, the positive and negative samples in the training set are balanced, and no manual labeling is required, which not only realizes defect detection based on unsupervised learning, but also improves the defect detection effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and specifically to a defect detection method and device based on unsupervised learning. Background Art

[0002] In the production process of products, due to the limitation of process level or the influence of environmental factors, defective products with defects will inevitably be produced. Defect detection is an effective means to ensure the yield rate of products. Automatic defect detection by analyzing product images has many advantages such as low cost, high efficiency, strong real-time performance and high reliability, and has been widely used in industrial production.

[0003] With the rapid development of deep learning in the field of target detection, defect detection as a branch of target detection has also achieved rapid development. At present, defect detection methods based on deep learning often adopt supervised learning, that is, normal images of normal products are collected as positive samples and added to the training set, defect images of defective products are collected as negative samples and added to the training set, the samples in the training set are manually labeled, and the defect detection model is trained based on the supervised learning method using the training set. However, in the actual production process, the number of defective products is relatively scarce compared to normal products, which increases the difficulty of collecting defective images, resulting in an imbalance of positive and negative samples in the training set, which will eventually lead to poor defect detection results based on supervised learning. Summary of the invention

[0004] The embodiments of the present invention provide a defect detection method and device based on unsupervised learning, which are used to solve the problem that the existing defect detection method based on supervised learning has poor effect.

[0005] In a first aspect, an embodiment of the present invention provides a defect detection method based on unsupervised learning, comprising:

[0006] Obtain a normal image of the product, add artificial noise to the normal image to generate an abnormal image paired with the normal image;

[0007] The defect detection network is trained according to normal images and abnormal images paired with normal images to obtain a trained defect detection network. The defect detection network includes a teacher network, a feature compression network, a student network and a projection network. The network structure of the student network is the same as that of the teacher network. The training of the defect detection network includes:

[0008] Inputting the normal image and the abnormal image paired with the normal image into the defect detection network respectively;

[0009] Extract features of a normal image through a teacher network to obtain an original feature map of the normal image; compress features of the original feature map of the normal image through a feature compression network to obtain a compressed feature map of the normal image; reconstruct features of the compressed feature map of the normal image through a student network to obtain a reconstructed feature map of the normal image; determine a first loss function according to the original feature map of the normal image and the reconstructed feature map of the normal image, and the first loss function is negatively correlated with the similarity between the original feature map of the normal image and the reconstructed feature map of the normal image;

[0010] The teacher network is used to extract features of the abnormal image to obtain an original feature map of the abnormal image; the feature compression network is used to compress the original feature map of the abnormal image to obtain a compressed feature map of the abnormal image; the student network is used to reconstruct the compressed feature map of the abnormal image to obtain a reconstructed feature map of the abnormal image; the projection network is used to project and reduce the dimensions of the original feature map of the abnormal image and the reconstructed feature map of the abnormal image to obtain an original feature vector of the abnormal image and a reconstructed feature vector of the abnormal image respectively; a second loss function is determined according to the original feature vector of the abnormal image and the reconstructed feature vector of the abnormal image, and the second loss function is positively correlated with the similarity between the original feature vector of the abnormal image and the reconstructed feature vector of the abnormal image;

[0011] The network parameters of the feature compression network, the student network and the projection network are iteratively updated with the goal of minimizing the sum of the first loss function and the second loss function until the iteration termination condition is met to obtain a trained defect detection network;

[0012] The trained defect detection network is used to perform defect detection on the image to be inspected of the product to obtain the defect detection result.

[0013] In one embodiment, a teacher network is used to extract multi-scale features of a normal image at different levels to obtain an n-layer (n≥2) original feature map of the normal image; a feature compression network is used to compress the n-layer original feature map of the normal image to obtain a compressed feature map of the normal image; a student network is used to reconstruct the features of the compressed feature map of the normal image in reverse order layer by layer to obtain an n-layer reconstructed feature map of the normal image;

[0014] The teacher network is used to extract multi-scale features of the abnormal image at different levels to obtain the n-layer original feature map of the abnormal image; the feature compression network is used to compress the n-layer original feature map of the abnormal image to obtain the compressed feature map of the abnormal image; the student network is used to reconstruct the features of the compressed feature map of the abnormal image layer by layer in reverse order to obtain the n-layer reconstructed feature map of the abnormal image; the projection network is used to project and reduce the dimension of the n-layer original feature map of the abnormal image and the n-layer reconstructed feature map of the abnormal image to obtain the n-layer original feature vector of the abnormal image and the n-layer reconstructed feature vector of the abnormal image respectively.

[0015] In one embodiment, feature compression is performed on n layers of original feature maps of a normal image through a feature compression network to obtain a compressed feature map of the normal image, including: performing convolution, batch normalization and activation processing on each layer of original feature maps of the first n-1 layers of original feature maps of the normal image, converting it into an original feature map with the same feature resolution as the nth layer of original feature map of the normal image, to obtain n layers of original feature maps of the normal image with uniform resolution; channel splicing is performed on the n layers of original feature maps of the normal image with uniform resolution on the channel to obtain a fused original feature map of the normal image; spatial compression and channel compression are performed on the fused original feature map of the normal image in sequence to obtain a compressed feature map of the normal image;

[0016] The feature compression network is used to compress the n-layer original feature maps of the abnormal image to obtain the compressed feature map of the abnormal image, including: performing convolution, batch normalization and activation processing on each layer of the original feature map of the first n-1 layers of the abnormal image, converting it into an original feature map with the same feature resolution as the n-th layer of the original feature map of the abnormal image, and obtaining the n-layer original feature map of the abnormal image with uniform resolution; channel splicing is performed on the n-layer original feature map of the abnormal image with uniform resolution on the channel to obtain the fused original feature map of the abnormal image; spatial compression and channel compression are performed on the fused original feature map of the abnormal image in sequence to obtain the compressed feature map of the abnormal image.

[0017] In one embodiment, the first loss function is determined according to the following expression:

[0018]

[0019] Among them, L pull represents the first loss function, α k Represents the k-th layer original feature map of the normal image, γ k Represents the k-th layer reconstructed feature map of the normal image, v k Indicates (α k ) T γ k The number of elements in .

[0020] In one embodiment, the second loss function is determined according to the following expression:

[0021]

[0022] Among them, L push represents the second loss function, ε k represents the k-th layer original feature vector of the abnormal image, σ k Represents the k-th layer reconstructed feature vector of the abnormal image.

[0023] In one embodiment, a trained defect detection network is used to perform defect detection on an image of a product to be detected, and the defect detection result obtained includes:

[0024] Input the image to be detected into the trained defect detection network to obtain n-layer original feature maps and n-layer reconstructed feature maps of the image to be detected;

[0025] Calculate the distance between the original feature map and the reconstructed feature map of the corresponding layer of the image to be detected, and obtain n layers of abnormal prediction sub-maps;

[0026] The n-layer anomaly prediction sub-images are upsampled to the resolution of the image to be detected, and fused at the corresponding pixel positions to obtain an anomaly prediction image of the same size as the image to be detected. The elements in the anomaly prediction image are used to represent the probability that the corresponding pixel in the image to be detected is an abnormal pixel;

[0027] Determine the defect detection results based on the anomaly prediction graph.

[0028] In one embodiment, determining a defect detection result according to an abnormality prediction graph includes:

[0029] The maximum value in the anomaly prediction map is determined as the anomaly score of the image to be detected;

[0030] When the anomaly score is greater than or equal to the preset anomaly threshold, the image to be detected is determined to be an abnormal image;

[0031] When the abnormality score is less than the preset abnormality threshold, the image to be detected is determined to be a normal image.

[0032] In one embodiment, determining a defect detection result according to an abnormality prediction graph includes:

[0033] Binarization is performed on the abnormal prediction image according to a preset binarization threshold to obtain an abnormal segmentation image of the image to be detected;

[0034] Connectivity analysis is performed on the abnormal segmentation graph, and defect detection results are determined according to the connectivity results.

[0035] In a second aspect, an embodiment of the present invention provides a defect detection device based on unsupervised learning, comprising: at least one processor and a memory;

[0036] Memory stores computer-executable instructions;

[0037] At least one processor executes computer-executable instructions stored in the memory, and the processor is used to execute the defect detection method based on unsupervised learning as described in any one of the first aspects.

[0038] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement a defect detection method based on unsupervised learning as described in any one of the first aspects.

[0039] The defect detection method and device based on unsupervised learning provided by the embodiment of the present invention generate abnormal images paired with normal images by adding artificial noise to normal images, and construct a training data set by pairing normal images with abnormal images one by one without manual labeling. The defect detection network is trained based on unsupervised learning, and the balance of positive and negative samples in the training data set helps to improve the detection effect of the defect detection network; the compressed feature map obtained by feature compression of the original feature map based on the feature compression network is reconstructed, which can effectively avoid the leakage of the feature representation ability of the abnormal image compared with the direct feature transfer in a short-circuit manner between the teacher network and the student network through knowledge distillation; training is performed with the goal of minimizing the sum of the first loss function and the second loss function, which can reduce the distance between the original feature map of the normal image and the reconstructed feature map, and at the same time increase the distance between the original feature vector and the reconstructed feature vector of the abnormal image paired with it. The defect detection network trained in this embodiment can reduce the feature reconstruction capability of abnormal images while maintaining the refined feature reconstruction of normal images, thereby increasing the distance between the original feature vector and the reconstructed feature vector of the abnormal images. Therefore, using the defect detection network trained in this embodiment to perform defect detection on the image to be detected can reduce false detections and improve the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a defect detection method based on unsupervised learning provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a normal image training process provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a training process for abnormal images provided by an embodiment of the present invention;

[0043] Figure 4 A defect segmentation schematic diagram provided by an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the structure of a defect detection device based on unsupervised learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0046] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0047] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0048] Defect detection technology based on machine vision has been widely used in industrial production due to its advantages of safety, reliability and high degree of automation. However, in the actual production process, images of defective products are relatively scarce in number compared to images of normal products. It is difficult to collect a large amount of image data containing all defects, and the labeling cost is high. Therefore, the positive and negative samples in the training set are unbalanced, resulting in poor defect detection results based on supervised learning, narrow usage scenarios, and difficulty in large-scale promotion and use. In contrast, normal images of normal products are easy to obtain. Therefore, this application obtains abnormal images by adding artificial noise to normal images, and implements defect detection by feature reconstruction. It can implement defect detection based on unsupervised learning when there are only normal images, and can adapt to many scenarios of industrial production, and realize large-scale application of surface abnormality detection of industrial product images and improvement of detection performance. The following will be explained in detail through specific embodiments.

[0049] Please refer to Figure 1 , a defect detection method based on unsupervised learning provided by an embodiment of the present invention may include:

[0050] S101. Obtain a normal image of a product, and add artificial noise to the normal image to generate an abnormal image paired with the normal image.

[0051] The product in this embodiment can be any product in industrial production, for example, mobile phone screen glass, battery, car shell, etc. It can be understood that the trained defect detection network can be used to detect defects of any product using images of the product for training.

[0052] Normal images are obtained by collecting images of normal products. Since normal products are easy to obtain in large quantities, normal images are also easy to obtain. The artificial noise in this embodiment can be obtained from large-scale natural image data by random sampling. It should be noted that artificial noise is not used to simulate the appearance of one or several real defects in an industrial production environment, but to generate an appearance outside the distribution of normal images. The abnormal image generated by adding artificial noise to the normal image has a different data distribution from the normal image.

[0053] A large number of normal images are collected, and paired training samples are formed by adding artificial noise to form a training data set. The training samples in the training data set appear in pairs, that is, the normal image and its paired abnormal image are trained as a group. In this embodiment, a training data set can be constructed when there are only normal images, and the positive and negative samples in the training data set appear in pairs, without manual labeling, which not only reduces the cost of data collection, but also the balanced training samples help improve the defect detection performance of the defect detection network.

[0054] S102: training a defect detection network according to normal images and abnormal images paired with the normal images to obtain a trained defect detection network.

[0055] The defect detection network (F) in this embodiment can be built based on a deep neural network, and can include a teacher network (T), a feature compression network (C), a student network (S) and a projection network (P). Among them, the teacher network is used to extract features from the image, the feature compression network is used to compress the features extracted by the teacher network, the student network is used to reconstruct the features compressed by the feature compression network, and the projection network is used to project and reduce the dimensionality of the features of the abnormal image. The student network has the same network structure as the teacher network. During the training process, the network parameters of the teacher network remain unchanged, and the features reconstructed by the student network have the same dimensions as the features extracted by the teacher network.

[0056] During training, normal images and their paired abnormal images are used as a set of input defect detection networks for training. For normal images, the training goal is to reduce the distance between extracted features and reconstructed features; for abnormal images, the training goal is to increase the difference between extracted features and reconstructed features. Since normal images and abnormal images have different distributions, training can enable the defect detection network to identify defects by deviating from the normality of normal images.

[0057] The normal image and the abnormal image paired with the normal image are respectively input into the defect detection network to train it. The following will explain in detail how to train the defect detection network through the training process of normal images and the training process of abnormal images.

[0058] The teacher network is used to extract features from the normal image to obtain the original feature map of the normal image; the feature compression network is used to compress the original feature map of the normal image to obtain the compressed feature map of the normal image; the student network is used to reconstruct the compressed feature map of the normal image to obtain the reconstructed feature map of the normal image; the first loss function is determined according to the original feature map of the normal image and the reconstructed feature map of the normal image, and the first loss function is negatively correlated with the similarity between the original feature map of the normal image and the reconstructed feature map of the normal image. For the normal image, the training goal is to reduce the distance between the original feature map and the reconstructed feature map of the normal image. That is, under the control of the first loss function, the distance between the original feature map of the normal image and the reconstructed feature map of the normal image is made as small as possible, and the student network is trained to learn the ability to restore the features of the normal image. When the distance between the original feature map of the normal image and the reconstructed feature map of the normal image is smaller, it means that the similarity between the two is higher, and then the value of the first loss function is smaller, so the training can be carried out by minimizing the first loss function.

[0059] The teacher network is used to extract features from the abnormal image to obtain the original feature map of the abnormal image; the feature compression network is used to compress the original feature map of the abnormal image to obtain the compressed feature map of the abnormal image; the student network is used to reconstruct the compressed feature map of the abnormal image to obtain the reconstructed feature map of the abnormal image; the projection network is used to project and reduce the dimension of the original feature map of the abnormal image and the reconstructed feature map of the abnormal image to obtain the original feature vector of the abnormal image and the reconstructed feature vector of the abnormal image respectively; the second loss function is determined according to the original feature vector of the abnormal image and the reconstructed feature vector of the abnormal image, and the second loss function is positively correlated with the similarity between the original feature vector of the abnormal image and the reconstructed feature vector of the abnormal image. It should be noted that not all pixels in the abnormal image are abnormal pixels, and abnormal pixels are usually only located in some areas, so they cannot be trained in a point-to-point manner. Therefore, in this embodiment, the original feature map and the reconstructed feature map of the abnormal image are projected and reduced in dimension by the projection network to obtain the corresponding feature vector representation. The low-dimensional feature vector represents whether the image is abnormal as a whole, and the difference between the original feature vector and the reconstructed feature vector of the abnormal image can be as large as possible under the control of the second loss function. When the distance between the original feature vector of the abnormal image and the reconstructed feature vector is larger, it means that the similarity between the two is lower, and the value of the second loss function is smaller. Therefore, training can be performed by minimizing the second loss function.

[0060] Normal images and abnormal images are paired one by one and input into the defect detection network for training, and the network is optimized together. In order to improve the feature reconstruction ability of the defect detection network for normal images and reduce the feature reconstruction ability of the defect detection network for abnormal images, the network parameters of the feature compression network, the student network and the projection network can be iteratively updated with the goal of minimizing the sum of the first loss function and the second loss function until the iteration termination condition is met to obtain a trained defect detection network. When the trained defect detection network is tested, the network parameters remain unchanged.

[0061] The feature reconstruction in this embodiment is based on the compressed feature map obtained by feature compression network after feature compression of the original feature map. Compared with the direct feature transfer between the teacher network and the student network in a short-circuit manner through knowledge distillation, it can effectively avoid the leakage of the feature representation ability of abnormal images, thereby avoiding excessive enhancement of the generalization of the student network. The trained defect detection network in this embodiment can reduce the feature reconstruction ability of abnormal images while maintaining the refined feature reconstruction of normal images, and increase the distance between the original feature vector and the reconstructed feature vector of the abnormal image, which can effectively reduce the false detection of normal images and improve the accuracy of defect detection.

[0062] S103: Use the trained defect detection network to perform defect detection on the image to be detected of the product to obtain a defect detection result.

[0063] In this embodiment, after obtaining a trained defect detection network, the trained defect detection network can be used to perform defect detection on the image to be detected of the product. Specifically, the image to be detected can be input into the trained defect detection network, and the defect detection result can be determined based on the distance between the original feature map and the reconstructed feature map of the image to be detected. The defect detection result can be, for example, to determine whether the image to be detected as a whole is an abnormal image based on the distance between the original feature map and the reconstructed feature map; or the distance between all pixels calculated in a point-to-point manner between the original feature map and the reconstructed feature map can be used to achieve accurate segmentation of defects.

[0064] The defect detection method based on unsupervised learning provided in the present embodiment generates an abnormal image paired with the normal image by adding artificial noise to the normal image, and the normal image and the abnormal image are paired one by one to construct a training data set without manual labeling. The defect detection network is trained based on an unsupervised learning method, and the balance of positive and negative samples in the training data set helps to improve the detection effect of the defect detection network; the compressed feature map obtained by feature compression of the original feature map based on the feature compression network is reconstructed, which can effectively avoid the leakage of the feature representation ability of the abnormal image compared to the direct feature transfer in a short-circuit manner between the teacher network and the student network through knowledge distillation; training is performed with the goal of minimizing the sum of the first loss function and the second loss function, which can reduce the distance between the original feature map of the normal image and the reconstructed feature map, and at the same time increase the distance between the original feature vector and the reconstructed feature vector of the abnormal image paired with it. The defect detection network trained in this embodiment can reduce the feature reconstruction capability of abnormal images while maintaining the refined feature reconstruction of normal images, thereby increasing the distance between the original feature vector and the reconstructed feature vector of the abnormal images. Therefore, using the defect detection network trained in this embodiment to perform defect detection on the image to be detected can reduce false detections and improve the accuracy of defect detection.

[0065] The defects of the product include both large targets with larger sizes and small targets with smaller sizes. In order to detect defects of different sizes and shapes, based on the above embodiments, the method provided in this embodiment can perform multi-scale feature extraction of normal images at different levels through a teacher network to obtain n (n≥2) layers of original feature maps of the normal image; perform feature compression on the n layers of original feature maps of the normal image through a feature compression network to obtain a compressed feature map of the normal image; and perform reverse order layer-by-layer feature reconstruction on the compressed feature map of the normal image through a student network to obtain n layers of reconstructed feature maps of the normal image.

[0066] Figure 2 A schematic diagram of a normal image training process provided by an embodiment of the present invention. Figure 2 Take n=3 as an example for explanation. Figure 2 As shown in the figure, the normal image x is input into the defect detection network. First, the teacher network T in the defect detection network F extracts multi-scale features of the normal image x at different levels, and obtains the first-layer original feature map α1, the second-layer original feature map α2, and the third-layer original feature map α3 from low to high levels. The higher the level, the lower the resolution of the feature map and the more channels. In other words, the higher the level, the larger the receptive field of the pixel is, which is more friendly to large targets; the lower the level, the smaller the receptive field of the pixel is, which is more friendly to small targets. α1, α2, and α3 constitute the multi-scale original feature map of the normal image x. Then, the feature compression network C in the defect detection network F compresses the multi-scale original feature map of the normal image x to obtain the compressed feature map β of the normal image x. Finally, the student network S in the defect detection network F performs reverse feature reconstruction of the compressed feature map β of the normal image x, and obtains the third-layer reconstructed feature map γ3, the second-layer reconstructed feature map γ2, and the first-layer reconstructed feature map γ1 from high to low levels. Among them, the dimensions of α1 and γ1, α2 and γ2, and α3 and γ3 are equal.

[0067] When performing multi-scale feature extraction at different levels on a normal image, the feature compression network is used to compress the n layers of original feature maps of the normal image to obtain a compressed feature map of the normal image, which may include: performing convolution, batch normalization and activation processing on each layer of the original feature map of the first n-1 layers of the normal image, converting it into an original feature map with the same feature resolution as the nth layer of the original feature map of the normal image, to obtain n layers of original feature maps with uniform resolution of the normal image; performing channel splicing on the n layers of original feature maps of the normal image with uniform resolution on the channel to obtain a fused original feature map of the normal image; performing spatial compression and channel compression on the fused original feature map of the normal image in turn to obtain a compressed feature map of the normal image. Figure 2 Taking the extraction of 3 layers of original feature maps as shown in the figure as an example, the feature compression network C can perform convolution, batch normalization BN and Relu activation processing on α1 and α2 according to the size of α3, so as to reduce the feature resolution and increase the number of channels, and convert α1 and α2 into original feature maps with the same feature resolution as α3, so as to obtain 3 layers of original feature maps with uniform resolution of normal image x. Channel concatenation is performed on the 3 layers of original feature maps with uniform resolution to obtain the fused original feature map of normal image x. The fused original feature map is further compressed in space, for example, by using a convolution layer to further reduce the feature resolution, and then performing a 1x1 convolution to reduce the number of feature channels, completing channel compression, and finally obtaining the compressed feature map β of normal image x.

[0068] When multi-scale feature extraction of different levels is performed on a normal image, the first loss function is determined according to the n-layer original feature map of the normal image and its corresponding n-layer reconstructed feature map. Minimizing the first loss function is to minimize the distance between the original feature map of each layer of the normal image and its corresponding reconstructed feature map of each layer. In an optional implementation, the first loss function can be determined according to the following expression:

[0069]

[0070] Among them, L pull represents the first loss function, α k Represents the k-th layer original feature map of the normal image, γ k Represents the k-th layer reconstructed feature map of the normal image, v k Indicates (α k ) T γ k As shown in the above formula, when α k With γ k The higher the similarity between them, that is, the smaller the distance between them, the smaller the value of the first loss function. By calculating the average of the cosine loss of each layer, the quality of the reconstructed features can be effectively controlled.

[0071] As for abnormal images, the method provided in this embodiment can further perform multi-scale feature extraction of abnormal images at different levels through a teacher network on the basis of the above-mentioned embodiments to obtain n-layer original feature maps of the abnormal images; perform feature compression on the n-layer original feature maps of the abnormal images through a feature compression network to obtain compressed feature maps of the abnormal images; perform reverse layer-by-layer feature reconstruction on the compressed feature maps of the abnormal images through a student network to obtain n-layer reconstructed feature maps of the abnormal images; perform projection and dimensionality reduction on the n-layer original feature maps of the abnormal images and the n-layer reconstructed feature maps of the abnormal images through a projection network, respectively, to obtain n-layer original feature vectors of the abnormal images and n-layer reconstructed feature vectors of the abnormal images, respectively.

[0072] Figure 3 A schematic diagram of a training process for abnormal images provided by an embodiment of the present invention. Figure 3 The following description will be made using n=3 as an example. Figure 3 As shown in the figure, the abnormal image x is obtained by adding artificial noise to the normal image x. + Input defect detection network, firstly use teacher network T in defect detection network F to detect abnormal image x + Perform multi-scale feature extraction at different levels to obtain the first layer of original feature maps from low level to high level. The second layer original feature map And the third layer original feature map The higher the level, the lower the resolution of the feature map and the more channels it has. In other words, the higher the level of the feature map, the larger the receptive field of the pixels, which is more friendly to large objects; the lower the level of the feature map, the smaller the receptive field of the pixels, which is more friendly to small objects. and Constitutes an abnormal image x + Then, the feature compression network C in the defect detection network F is used to compress the abnormal image x + The multi-scale original feature map is compressed to obtain the abnormal image x + The compressed feature map β + Then, the student network S in the defect detection network F is used to detect the abnormal image x + The compressed feature map β + Reconstruct the features layer by layer in reverse order to obtain the third-layer reconstructed feature map from high level to low level. The second layer reconstructs the feature map And the first layer reconstructed feature map Finally, the abnormal images x are respectively + The multi-scale original feature map and abnormal image x + The multi-scale reconstructed feature map is projected and reduced in dimension to obtain the abnormal image x + The multi-scale original feature vectors ε1, ε2 and ε3 of , as well as the abnormal image x + The multi-scale reconstruction feature vectors σ1, σ2 and σ3.

[0073] When performing multi-scale feature extraction at different levels on an abnormal image, the feature compression network is used to compress the n-layer original feature maps of the abnormal image to obtain a compressed feature map of the abnormal image, which may include: performing convolution, batch normalization and activation processing on each layer of the original feature map of the first n-1 layers of the abnormal image, converting it into an original feature map with the same feature resolution as the n-th layer of the original feature map of the abnormal image, and obtaining n-layer original feature maps with uniform resolution of the abnormal image; performing channel splicing on the n-layer original feature maps of the abnormal image with uniform resolution on the channel to obtain a fused original feature map of the abnormal image; and sequentially performing spatial compression and channel compression on the fused original feature map of the abnormal image to obtain a compressed feature map of the abnormal image. Figure 3 Taking the extraction of 3 layers of original feature maps as an example, the feature compression network C can be The size of and Convolution, batch normalization BN and ReLU activation are performed to reduce feature resolution and increase the number of channels. and Convert to The original feature map with the same feature resolution is used to obtain the abnormal image x + The three layers of original feature maps with uniform resolution are concatenated to obtain the abnormal image x + The fused original feature map is further compressed in space, for example, using a convolution layer to further reduce the feature resolution, and then performing a 1x1 convolution to reduce the number of feature channels and complete channel compression, and finally the abnormal image x is obtained. + The compressed feature map β + .

[0074] When multi-scale feature extraction of different levels is performed on the abnormal image, the second loss function is determined according to the n-layer original feature vector of the abnormal image and its corresponding n-layer reconstructed feature vector. Minimizing the second loss function is to maximize the distance between the original feature map of each layer of the abnormal image and its corresponding reconstructed feature map of each layer. In an optional implementation, the second loss function can be determined according to the following expression:

[0075]

[0076] Among them, L push represents the second loss function, ε k represents the k-th layer original feature vector of the abnormal image, σ k represents the k-th layer reconstructed feature vector of the abnormal image. As shown in the above formula, when ε k With σ k The larger the distance between them, the smaller the second loss function.

[0077] In an optional implementation, the overall loss function L of the defect detection network F can be constructed according to the following expression: total :

[0078] L total =L pull +L push

[0079] That is, the defect detection network F is trained with the goal of minimizing the sum of the first loss function and the second loss function.

[0080] The defect detection method based on unsupervised learning provided in this embodiment is based on the above-mentioned embodiment. It further extracts and reconstructs multi-scale features at different levels and comprehensively considers the distance between the original features and the reconstructed features at each level. It can detect defects of different sizes and shapes, is more sensitive to subtle defects, and improves the detection capability of subtle defects.

[0081] Based on any of the above embodiments, in the method provided in this embodiment, a trained defect detection network is used to perform defect detection on an image to be detected of a product, and obtaining a defect detection result may include:

[0082] S1031. Input the image to be detected into the trained defect detection network to obtain n layers of original feature maps and n layers of reconstructed feature maps of the image to be detected.

[0083] After the defect detection network F is trained, the projection network P does not participate in the detection process. After the trained defect detection network is obtained, it is only necessary to input the collected image to be detected into the trained defect detection network to obtain the n-layer original feature map α of the image to be detected. k (k=1,2,…,n) and n-layer reconstructed feature map γ k (k=1,2,…,n).

[0084] S1032. Calculate the distance between the original feature map and the reconstructed feature map of the corresponding layer of the image to be detected to obtain n layers of abnormality prediction sub-maps.

[0085] The greater the distance between the original feature map and the reconstructed feature map, the greater the probability that the image to be detected is abnormal. k With γ k The distance between them is used to obtain the corresponding abnormal prediction subgraph M k In an optional implementation, M can be determined point-by-point according to the following expression: k :

[0086]

[0087] When α k With γ k The larger the distance between them, the greater the k The larger the value of the element in , the greater the probability that it is an abnormal pixel.

[0088] S1033. Upsample the n-layer anomaly prediction sub-images to the resolution size of the image to be detected, and fuse them at the corresponding pixel positions to obtain an anomaly prediction image of the same size as the image to be detected. The elements in the anomaly prediction image are used to represent the probability that the corresponding pixel in the image to be detected is an abnormal pixel.

[0089] It can be understood that, although the abnormal prediction sub-images of each layer determined according to the original feature map and the reconstructed feature map of different scales have different resolutions, they still retain the position information of the defect. In order to realize the pixel-level positioning of the defect in the image to be detected, in this embodiment, the n-layer abnormal prediction sub-image M kUpsample to the resolution of the image to be detected, and fuse at the corresponding pixel positions to obtain an anomaly prediction map A with the same size as the image to be detected. Specifically, the calculation formula of the pixel-level anomaly prediction map A is as follows:

[0090]

[0091] A is the same size as the image to be detected. Each element in A is used to represent the probability that the corresponding pixel in the image to be detected is an abnormal pixel. The larger the value, the higher the probability of an abnormal pixel. Upsample means upsampling. The sampling rate of upsampling can be determined based on the image to be detected and M. k The ratio of the resolution sizes is used to determine the value.

[0092] S1034. Determine the defect detection result according to the abnormality prediction graph.

[0093] According to the abnormal prediction graph, both defect classification and defect segmentation can be performed.

[0094] In an optional implementation, the image to be detected can be quickly classified according to the maximum value in the anomaly prediction map. Specifically, the maximum value in the anomaly prediction map can be determined as the anomaly score of the image to be detected, that is, the anomaly score Score = Max (A). When the anomaly score is greater than or equal to the preset anomaly threshold, the image to be detected is determined to be an abnormal image; when the anomaly score is less than the preset anomaly threshold, the image to be detected is determined to be a normal image. By comparing the anomaly score with the preset anomaly threshold, the image to be detected can be quickly classified.

[0095] In an optional implementation, the pixel-level segmentation of the image to be detected can also be achieved based on the anomaly prediction map. Specifically, the anomaly prediction map can be binarized according to a preset binarization threshold to obtain an anomaly segmentation map of the image to be detected, and connectivity analysis can be performed on the anomaly segmentation map, and the defect detection result can be determined based on the connectivity result. It can be understood that by adjusting the size of the binarization threshold, the anomaly segmentation map can include all possible abnormal pixels; further, in order to reduce the false detection rate, connectivity analysis can be performed on the anomaly segmentation map, and the defect detection result can be determined based on the area and confidence of the connectivity result. For example, for some isolated points in the anomaly segmentation map, it may be that normal pixels are mistakenly identified as anomalies. By screening the area of ​​the connectivity result, the false detection rate can be reduced. Figure 4 As shown in the figure, by inputting the image to be detected on the left into the defect detection network, the pixel-level defect detection result on the right can be obtained, thus achieving accurate positioning of the defects.

[0096] The defect detection method based on unsupervised learning provided in this embodiment, on the basis of the above embodiment, further calculates the distance between the original feature map of each layer of the image to be detected and the reconstructed feature map to obtain n layers of abnormality prediction sub-maps, and then upsamples the n layers of abnormality prediction sub-maps to the resolution size of the image to be detected, and fuses them at the corresponding pixel positions to obtain an abnormality prediction map with the same size as the image to be detected. According to the abnormality prediction map, not only the rapid classification of the image to be detected can be achieved, but also the pixel-level segmentation of the image to be detected can be achieved.

[0097] The embodiment of the present invention also provides a defect detection device based on unsupervised learning, see Figure 5 As shown, the embodiment of the present invention is only Figure 5 This is just an example for explanation, and it does not mean that the present invention is limited to this. Figure 5 A schematic diagram of the structure of a defect detection device based on unsupervised learning provided by an embodiment of the present invention. Figure 5 As shown, the defect detection device 50 based on unsupervised learning provided in this embodiment may include: a memory 501, a processor 502 and a bus 503. The bus 503 is used to realize the connection between various components.

[0098] The memory 501 stores a computer program, and when the computer program is executed by the processor 502 , the technical solution of the defect detection method based on unsupervised learning provided by any of the above method embodiments can be implemented.

[0099] The memory 501 and the processor 502 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines, such as through bus 503. The memory 501 stores a computer program for implementing a defect detection method based on unsupervised learning, including at least one software function module that can be stored in the memory 501 in the form of software or firmware. The processor 502 executes various functional applications and data processing by running the software program and module stored in the memory 501.

[0100] Understandably, Figure 5 The structure is only for illustration and may also include Figure 5 More or fewer components as shown, or with Figure 5 Different configurations are shown. Figure 5 The components shown in the figure can be implemented by hardware and / or software. For example, an image acquisition device for acquiring an image to be detected can also be included.

[0101] This document is described with reference to various exemplary embodiments. However, those skilled in the art will recognize that changes and modifications may be made to the exemplary embodiments without departing from the scope of this document. For example, various operating steps and components for performing the operating steps may be implemented in different ways (e.g., one or more steps may be deleted, modified, or incorporated into other steps) depending on the specific application or considering any number of cost functions associated with the operation of the system.

[0102] In addition, as will be appreciated by those skilled in the art, the principles of the present invention may be reflected in a computer program product on a computer-readable storage medium, which is pre-installed with a computer-readable program code. Any tangible, non-temporary computer-readable storage medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu Ray disks, etc.), flash memory, and / or the like. These computer program instructions may be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to form a machine, so that these instructions executed on a computer or other programmable data processing device may generate a device that implements a specified function. These computer program instructions may also be stored in a computer-readable memory, which may instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory may form a manufactured product, including an implementation device that implements a specified function. The computer program instructions may also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device may provide steps for implementing a specified function.

[0103] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A defect detection method based on unsupervised learning, characterized in that: include: Acquire a normal image of a product, and add artificial noise to the normal image to generate an abnormal image paired with the normal image; The defect detection network is trained according to the normal image and the abnormal image paired with the normal image to obtain a trained defect detection network, wherein the defect detection network includes a teacher network, a feature compression network, a student network and a projection network, and the student network has the same network structure as the teacher network. The training of the defect detection network includes: Inputting the normal image and the abnormal image paired with the normal image into the defect detection network respectively; Performing feature extraction on the normal image through the teacher network to obtain an original feature map of the normal image; performing feature compression on the original feature map of the normal image through the feature compression network to obtain a compressed feature map of the normal image; performing feature reconstruction on the compressed feature map of the normal image through the student network to obtain a reconstructed feature map of the normal image; determining a first loss function according to the original feature map of the normal image and the reconstructed feature map of the normal image, wherein the first loss function is negatively correlated with the similarity between the original feature map of the normal image and the reconstructed feature map of the normal image; The teacher network is used to extract features of the abnormal image to obtain an original feature map of the abnormal image; the feature compression network is used to compress features of the original feature map of the abnormal image to obtain a compressed feature map of the abnormal image; the student network is used to reconstruct features of the compressed feature map of the abnormal image to obtain a reconstructed feature map of the abnormal image; the projection network is used to project and reduce the dimensions of the original feature map of the abnormal image and the reconstructed feature map of the abnormal image to obtain an original feature vector of the abnormal image and a reconstructed feature vector of the abnormal image respectively; a second loss function is determined according to the original feature vector of the abnormal image and the reconstructed feature vector of the abnormal image, and the second loss function is positively correlated with the similarity between the original feature vector of the abnormal image and the reconstructed feature vector of the abnormal image; Iteratively updating the network parameters of the feature compression network, the student network, and the projection network with the goal of minimizing the sum of the first loss function and the second loss function until an iteration termination condition is met to obtain a trained defect detection network; The trained defect detection network is used to perform defect detection on the image to be detected of the product to obtain a defect detection result.

2. The method according to claim 1, characterized in that Performing multi-scale feature extraction of different levels on the normal image through the teacher network to obtain n (n≥2) layers of original feature maps of the normal image; performing feature compression on the n layers of original feature maps of the normal image through the feature compression network to obtain a compressed feature map of the normal image; Reconstructing the compressed feature map of the normal image layer by layer in reverse order through the student network to obtain an n-layer reconstructed feature map of the normal image; The teacher network is used to perform multi-scale feature extraction at different levels on the abnormal image to obtain an n-layer original feature map of the abnormal image; the feature compression network is used to perform feature compression on the n-layer original feature map of the abnormal image to obtain a compressed feature map of the abnormal image; the student network is used to perform reverse layer-by-layer feature reconstruction on the compressed feature map of the abnormal image to obtain an n-layer reconstructed feature map of the abnormal image; the projection network is used to perform projection dimensionality reduction on the n-layer original feature map of the abnormal image and the n-layer reconstructed feature map of the abnormal image to obtain an n-layer original feature vector of the abnormal image and an n-layer reconstructed feature vector of the abnormal image, respectively.

3. The method according to claim 2, characterized in that The feature compression network is used to perform feature compression on the n-layer original feature maps of the normal image to obtain the compressed feature map of the normal image, including: performing convolution, batch normalization and activation processing on each layer of the original feature map of the first n-1 layers of the normal image, converting it into an original feature map with the same feature resolution as the n-th layer of the original feature map of the normal image, to obtain the n-layer original feature map of the normal image with uniform resolution; performing channel splicing on the n-layer original feature map of the normal image with uniform resolution on the channel to obtain the fused original feature map of the normal image; performing spatial compression and channel compression on the fused original feature map of the normal image in sequence to obtain the compressed feature map of the normal image; The method of performing feature compression on the n-layer original feature maps of the abnormal image through the feature compression network to obtain the compressed feature map of the abnormal image includes: performing convolution, batch normalization and activation processing on each layer of the original feature map of the first n-1 layers of the abnormal image, converting it into an original feature map with the same feature resolution as the n-th layer of the original feature map of the abnormal image, and obtaining the n-layer original feature map of the abnormal image with uniform resolution; performing channel splicing on the n-layer original feature map of the abnormal image with uniform resolution on the channel to obtain the fused original feature map of the abnormal image; and performing spatial compression and channel compression on the fused original feature map of the abnormal image in sequence to obtain the compressed feature map of the abnormal image.

4. The method according to claim 2, characterized in that The first loss function is determined according to the following expression: Among them, L pull represents the first loss function, α k Represents the k-th layer original feature map of the normal image, γ k Represents the k-th layer reconstructed feature map of the normal image, v k Indicates (α k ) T γ k The number of elements in .

5. The method according to claim 2, characterized in that The second loss function is determined according to the following expression: Among them, L push represents the second loss function, ε k represents the k-th layer original feature vector of the abnormal image, σ k Represents the k-th layer reconstructed feature vector of the abnormal image.

6. The method according to any one of claims 2 to 5, characterized in that: The defect detection is performed on the image to be detected of the product by using the trained defect detection network to obtain the defect detection result, which includes: Inputting the image to be detected into the trained defect detection network to obtain n-layer original feature maps and n-layer reconstructed feature maps of the image to be detected; Calculate the distance between the original feature map and the reconstructed feature map of the corresponding layer of the image to be detected to obtain n layers of abnormality prediction sub-maps; The n-layer anomaly prediction sub-images are upsampled to the resolution size of the image to be detected, and are fused at the corresponding pixel positions to obtain an anomaly prediction image of the same size as the image to be detected, wherein the elements in the anomaly prediction image are used to represent the probability that the corresponding pixel in the image to be detected is an abnormal pixel; The defect detection result is determined according to the abnormality prediction graph.

7. The method according to claim 6, characterized in that Determining the defect detection result according to the abnormality prediction graph includes: Determine the maximum value in the abnormal prediction map as the abnormal score of the image to be detected; When the abnormality score is greater than or equal to a preset abnormality threshold, the image to be detected is determined to be an abnormal image; When the abnormality score is less than a preset abnormality threshold, it is determined that the image to be detected is a normal image.

8. The method according to claim 6, characterized in that Determining the defect detection result according to the abnormality prediction graph includes: Binarizing the abnormal prediction map according to a preset binarization threshold to obtain an abnormal segmentation map of the image to be detected; Connectivity analysis is performed on the abnormal segmentation graph, and defect detection results are determined according to the connectivity results.

9. A defect detection device based on unsupervised learning, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, and the processor is used to execute the defect detection method based on unsupervised learning as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the defect detection method based on unsupervised learning as described in any one of claims 1 to 8.