Defect detection method and device based on semi-supervised learning

Abstract

The present application provides a defect detection method and apparatus based on semi-supervised learning. The defect detection method based on semi-supervised learning includes: obtaining an image of a detection object, wherein defects in a portion of the image are marked; using a first classifier to identify the image as defects and background, wherein the first classifier is trained using data from the portion of the image based on a semi-supervised learning method; and using a second classifier to classify the defects into multiple types of defects, wherein the second classifier is trained using data from the portion of the image based on a semi-supervised learning method.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a defect detection method and device based on semi-supervised learning. Background Art

[0002] In the era of robotics and automation, artificial intelligence (AI) is helping to solve many problems that are beyond human reach. Glass inspection is a major challenge facing the glass manufacturing industry, as the quality of mobile phone display glass panels can impact user experience and device durability. Because glass is extremely fragile to impact, it must be inspected at all stages of manufacturing. Defects such as pits and cracks can appear on the glass surface. In practice, glass manufacturers face significant financial losses due to time and labor constraints. In China, for example, glass inspection is primarily performed by over 100,000 skilled workers who visually detect various defects in glass. However, some minor defects are difficult to detect with the human eye, and these can also affect the quality of smart devices. Many glass manufacturers are currently showing great interest in automated systems and cutting-edge technologies that can help them overcome these challenges, thereby improving production line efficiency and increasing sales profits.

[0003] There are already various methods available for detecting glass defects. For detecting defects in satin and float glass, researchers have used machine learning techniques. Several image processing-based frameworks have been proposed for satin glass and glass bottles. Furthermore, researchers have proposed optical methods for detecting microcracks in glass, detecting surface defects in touch panel glass, and inspecting window glass. There is also research on detecting defects in automotive glass.

[0004] With the rapid growth of smart device production in the past decade, many technology companies producing smart devices (such as mobile phones, tablets, laptops, and smartwatches) need to assemble tens of thousands of pieces of glass each year. Glass defect detection is not only labor-intensive but also time-sensitive. Due to precision limitations, the human eye can only detect defects larger than 0.1 mm, and the average time required to inspect a piece of smartphone glass is about 1-2 minutes. Therefore, it is difficult to increase production speed without using robotics or automation to solve this problem.

[0005] Accuracy and speed are crucial factors for smartphone glass inspection. Current state-of-the-art methods, such as support vector machines (SVMs), principal component analysis (PCA), or adaptive boosting (AdaBoost), are proposed to detect defects in smartphone cover glass. However, these methods are currently limited to detecting scratches and are not capable of effectively classifying light leaks and stains on smartphone glass.

[0006] With the rise of deep learning in recent years, we've seen significant success in certain visual recognition tasks, surpassing most machine learning methods in terms of accuracy and robustness. This success stems from deep learning methods' exceptional ability to extract high-level features and efficiently utilize data. Object detection and semantic segmentation are two major research areas within visual recognition. Object detection involves detecting objects in an image and assigning bounding boxes to all objects, while semantic segmentation provides pixel-level mappings for all objects or semantic patches in an image. Object detection combines object classification with bounding box location and size regression, while semantic segmentation simply classifies all pixels in an image. Since Girshick et al. invented the Region-based Convolutional Neural Network (R-CNN), the R-CNN family has consistently represented the state-of-the-art in terms of accuracy and speed. Faster object detection algorithms include YOLO (You Only Look Once) and SSD (Single Shot Multi-Box Detector). However, for glass detection applications, we desire to detect more detail in the detection image. However, the bounding boxes output by object detection algorithms lack precise defect location and type information, which makes it impossible to accurately locate the defects.

[0007] Since the introduction of the Fully Convolutional Network (FCN), classic semantic segmentation methods based on edge detection, texture theory, etc. have been surpassed by deep learning-based methods. This deep learning-based method applies a series of convolutional layers and a series of deconvolutional layers to infer semantic segmentation results. However, due to the loss of information within the deconvolution layer, the recovered segmentation map usually has blurred edges. To overcome such shortcomings, researchers have designed more complex network configurations to reduce information loss within the FCN, such as the work of SegNet and dilated convolutional networks. Following the idea of ​​dilated convolution, Chen et al. used bilinear interpolation to enlarge the feature map to the original resolution, thereby adding multi-scale processing of large images, and applied conditional random field (CRF) post-processing to smooth the segmentation. Summary of the Invention

[0008] In one aspect of the present disclosure, a defect detection method based on semi-supervised learning is provided, comprising: obtaining an image of a detection object, wherein defects in a portion of the image are marked; identifying the image as defects and background using a first classifier, wherein the first classifier is trained based on a semi-supervised learning method using data of the portion of the image; and classifying the defects into multiple types of defects using a second classifier, wherein the second classifier is trained based on a semi-supervised learning method using data of the portion of the image.

[0009] According to an embodiment of the present disclosure, before the step of using a first classifier to identify the image as a defect and a background, the defect detection method further includes: detecting a white continuous area on the image; cropping the white continuous area from the image along a bounding box of the white continuous area; adjusting the size of the cropped image of the white continuous area; and performing feature extraction on the adjusted image of the white continuous area using a pre-trained convolutional neural network.

[0010] According to an embodiment of the present disclosure, detecting white continuous areas on the image includes: converting the image into a binary image; performing a dilation operation on the binary image; and determining a bounding box of each white continuous area by non-maximum suppression.

[0011] According to an embodiment of the present disclosure, the first classifier is a segmented K-means classifier.

[0012] According to an embodiment of the present disclosure, the second classifier is a random forest classifier.

[0013] According to an embodiment of the present disclosure, the multiple types of defects include: dirt, scratches, and light leakage.

[0014] Another aspect of the present disclosure provides a defect detection device based on semi-supervised learning, including: an image acquisition unit, which is configured to obtain an image of a detection object, wherein defects in a portion of the image are marked; a first classifier, which is configured to identify the image as defects and background, wherein the first classifier is trained based on a semi-supervised learning method using data from the portion of the image; and a second classifier, which is configured to classify the defects into multiple types of defects, wherein the second classifier is trained based on a semi-supervised learning method using data from the portion of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of a glass detection system according to an embodiment of the present application;

[0016] Figure 2 is a schematic diagram of an image captured by a glass inspection system according to an embodiment of the present application;

[0017] Figures 3a to 3e is a schematic diagram of an example of a glass image defect according to an embodiment of the present application;

[0018] Figure 4 Schematic diagram of feature extraction using transfer learning according to an embodiment of the present application;

[0019] Figure 5 Schematic diagram of defect / background classification using semi-supervised learning according to an embodiment of the present application;

[0020] Figure 6 is a schematic diagram of a glass image including multiple defects according to an embodiment of the present application;

[0021] Figure 7 is a schematic diagram of test results of glass including multiple defects using SmartInspect according to an embodiment of the present application;

[0022] Figure 8 Flowchart of a defect detection method based on semi-supervised learning according to an embodiment of the present application;

[0023] Figure 9a and Figure 9b Schematic diagram of the test results of a positive sample using SmartInspect according to an embodiment of the present application;

[0024] Figure 10 4 is a block diagram of a defect detection device based on semi-supervised learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] This application improves the current visual recognition method and solves the problem of defect detection and positioning on glass images. Due to the large image size, the current deep learning-based algorithm cannot be directly applied. To address this situation, this application provides an algorithm called SmartInspect. SmartInspect is an automatic detection algorithm that contains two classifiers to handle classification. The SKM (Segmented K-Means) classifier is a binary classifier that can classify defects into two categories: defects and background. These defects are then projected into a Random Forest (RF) classifier, and the defects are further classified into three types of defects: dirt, scratches, and light leakage.

[0027] Figure 1 is a schematic diagram of a glass detection system according to an embodiment of the present application; Figure 2 is a schematic diagram of an image captured by a glass inspection system according to an embodiment of the present application; Figures 3a to 3e Schematic diagram of an example of glass image defects according to an embodiment of the present application.

[0028] like Figure 1 As shown, the glass detection system includes a glass placement unit with an LED light source, a linear scan camera, and an image processing unit (not shown), wherein the linear scan camera is used to capture glass images (such as Figure 2 As shown in FIG, the linear scan camera is selected to capture images with a resolution of 16384 pixels, for example. The resolution of the glass image captured by the camera is, for example, 16384*24576, and the glass inspection system can detect defects with a minimum size of 10 microns or less. In an embodiment according to the present application, the glass inspection system can capture glass images with the following three defects: scratches, light leakage, and dirt. Figure 3a and Figure 3b is a schematic diagram of an example of a glass image containing scratches according to an embodiment of the present application. Figure 3c is a schematic diagram of an example of a glass image containing light leakage according to an embodiment of the present application, and Figure 3d and Figure 3e Schematic diagram of an example of a glass image containing dirt according to an embodiment of the present application.

[0029] In this application, the current visual recognition method is improved to enhance the detection and location of defects on glass images. Due to the large size of glass images, the current deep learning-based methods cannot be directly applied. The present application provides a method of smart inspection (SmartInspect), which is an automatic inspection method. In the SmartInspect method, the Segmented K-Means (SKM) classifier is a binary classifier that divides the image into the following two categories: defects and background. The defects are then projected to the Random Forest (RF) classifier to further classify the defects into the following three types of defects: dirt, scratches, and light leakage.

[0030] Figure 4 Schematic diagram of feature extraction using transfer learning according to an embodiment of the present application. Figure 5 Schematic diagram of defect / background classification using semi-supervised learning according to an embodiment of the present application. Figure 4 and Figure 5 As shown, in an embodiment of the present application, the SmartInspect method includes the following four preprocessing stages: stage I, stage II, stage III, and stage IV.

[0031] In stage I, all suspicious white continuous regions on each image are cropped along the contour box. These regions are detected by the continuous region selection algorithm in Table 1:

[0032] Table 1: Continuous region selection algorithm

[0033]

[0034] In stage II, a pre-trained Convolutional Neural Network (CNN) is used to extract features from the images cropped in stage I.

[0035] In Phase III, a background defect classifier is trained using a semi-supervised approach (described below) based on the few labeled defects. A k-means clustering algorithm is applied, discarding clusters that are excluded or contain a relatively low percentage of defects in each iteration until the number of discarded images falls below a preset threshold. This phase eliminates redundant background images, significantly reducing the number of defects.

[0036] In stage IV, a three-level Random Forest classifier is trained based on the labeled defects.

[0037] The following describes this application from five aspects: data set, defect location, finding the real defect, classification of three types of defects, and test plan.

[0038] Dataset

[0039] In this example, 76 images of smartphone glass were selected, each with three types of defects: scratches, light leakage, and dirt. Each image is a combination of a glass screen image and a defect image, both acquired from a regional camera. The glass screen images are nearly perfect, but they still contain unlabeled defects. All defects artificially added to the glass images are labeled with their location and category. The real glass screen is nearly perfect, with some scratches and unlabeled defects.

[0040]

Defect Location

[0041] First, the image is converted into a binary image. Then, the binary image is dilated to enhance the connectivity of the region. After that, the bounding box of each white continuous region and the size and original position of the bounding box are estimated by non-maximum suppression (NMS). All white continuous regions are cropped along the bounding box, and the cropped bounding box is adjusted to 224×224 pixels. For the entire dataset, approximately M=29,000 segmented images can be obtained, as shown in Figure 4 This scheme is further explained in the continuous region selection algorithm described above. The IoU (intersection over union) is estimated between each bounding box and the other bounding boxes, and these boxes are merged into a single bounding box based on a non-maximum suppression threshold. Finally, all boxes corresponding to objects of a single class are resized to 224×224 pixels.

[0042] Finding the real flaw

[0043] Transfer learning is used to process all cropped images for classification. A pre-trained convolutional neural network, ResNet18, is used as a feature extractor. After passing through ResNet18, each image is converted into a 512-dimensional feature vector.

[0044] In order to distinguish actual defects from background, the SKM classifier is trained. Due to the limited amount of labeled data, it is not possible to directly train the background defect classifier using a supervised method. Therefore, this application provides a semi-supervised method to train the background defect classifier. In this application, the k-means method is iteratively used and some clusters are filtered based on the labeled data until the algorithm converges. Figure 5 shown.

[0045] For each k-means iteration, all data points (feature vectors for each image) are divided into 10 clusters. Then, based on the labeled data, the top three clusters containing the highest proportion of labeled data are retained and projected into the next iteration (profiles of some clusters are shown in the next section). Training is stopped when the number of discarded data falls below a preset threshold.

[0046]

Classification of three types of defects

[0047] After removing all background data, approximately 7,000 actual defects (154 labeled data points) remained. A classifier was trained to classify the actual defects into three distinct categories: dirt, scratches, and light leakage. For the dataset described in this article, a random forest algorithm was selected to train a three-class classifier using the 154 labeled data points. The random forest algorithm achieved better performance than the support vector machine (SVM).

[0048]

Test plan

[0049] In order to evaluate the robustness of the SmartInspect method, in the examples of this application, tests were performed on glass images with many different defects and on some positive glass samples. Figure 6 This figure shows a glass image containing multiple defects, used as a test image, according to an embodiment of the present application. First, all continuous regions are preprocessed. Then, the test algorithm in Table 2 is used to mark all regions, identify defects, and separate them from the background.

[0050] Table 2: Test algorithms

[0051]

[0052]

[0053] After pre-processing all regions in the image, all pre-processed regions are passed to the SKM classifier. If they are observed to be defect regions, the RF classifier is used to further classify the defects, such as scratches, dust particles, or light leakage areas. Figure 7 The performance of the glass image by inspecting the defect-filled area is shown in Figure 7 In this image, defects are separated using bounding boxes, where the red marked areas are scratches, green indicates light leaks, and orange indicates dust particles.

[0054] From the experimental results, we can see that SmartInspect scans the entire image and intelligently identifies the background area. In other frameworks, all white areas would be considered defects. However, SmartInspect does not mark the background area as a defect. Figure 7Background areas such as the QR code, speaker, buttons, sensors, and camera are clearly shown. Some light reflections are observed in these areas, which are shown in Figure 7 in the green box.

[0055] Figure 8 FIG is a flow chart of a defect detection method based on semi-supervised learning according to an embodiment of the present application. Figure 8 As shown, in an embodiment of the present application, the defect detection method based on semi-supervised learning includes steps 301 to 306.

[0056] In step 301, using Figure 1 The line scan camera shown in FIG. 1 acquires an image of the device to be inspected.

[0057] In step 302, the obtained image is converted into a binary image based on a preset threshold.

[0058] In step 303, a dilation operation is performed on the binary image to enhance the connectivity of the white areas.

[0059] In step 304 , the bounding box of each white continuous area, its size, and its original position are determined by non-maximum suppression, and the size of the bounding box is adjusted, for example, the bounding box is adjusted to 224*224 pixels.

[0060] In step 305, features of the image representing the continuous white area are extracted using a pre-trained convolutional neural network (CNN) based on transfer learning. For example, a pre-trained convolutional neural network (ResNet18) is used as the feature extractor. After passing through ResNet18, each image is converted into a 512-dimensional feature vector.

[0061] In step 306 , the labeled data is used to train an SKM classifier based on a semi-supervised learning method, and the first classifier is used to classify the image of the white continuous area as a defect or background.

[0062] In step 307, the identified defects are projected into an intra-class classifier to identify the type of the defect. In this example, an intra-class classifier is trained by training the labeled data using a random forest method.

[0063] On the other hand, the method of the present application can be tested on glass samples that have been marked as positive after manual inspection. Figure 9a and Figure 9b Two different glass test results for positive samples are shown. Figure 9a and Figure 9bAs shown in Figure 2, SmartInspect outperforms manual inspection in assessing the quality of glass samples. Three defects have been located and classified: two light leaks (D-1 and D-3) and one dust particle (D-2) were detected, Figure 9a SmartInspect inspected another positive sample that was completely different from the one above, as shown in the orange and green boxes. Figure 9b The method of the present application is also superior to other methods in determining the fouling area (D-1).

[0064] Table 3 shows a comparison of SmartInspect with current state-of-the-art methods. This demonstrates that our approach is highly intelligent in identifying background areas. It also reliably achieves high accuracy in detecting defects up to 2 microns, which is crucial in smart device glass, as defects such as light leakage or contamination can degrade the image quality of the screen.

[0065] Table 3: Robustness comparison of various methods

[0066]

[0067] Figure 10 Block diagram of a defect detection device based on semi-supervised learning according to an embodiment of the present application. Figure 10 As shown, in the embodiment of the present application, the defect detection device 10 includes an image acquisition unit 101 , a first classifier 102 and a second classifier 103 .

[0068] The image acquisition unit 101 is configured to obtain an image of the inspection object, wherein defects in a portion of the image are marked. For example, the image acquisition unit 101 may be a line scan camera.

[0069] The first classifier 102 is configured to identify the image as defects and background, wherein the first classifier is trained using the data of the partial image based on a semi-supervised learning method. For example, the first classifier 102 can be an SKM classifier.

[0070] The second classifier 103 is configured to classify the defects into multiple types of defects, wherein the second classifier is trained using the data of the partial image based on a semi-supervised learning method. For example, the second classifier 103 can be the above-mentioned intra-class classifier or random forest classifier.

[0071] It should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, path construction method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, path construction method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0072] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0076] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0077] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A defect detection method based on semi-supervised learning, comprising: An image of the inspection object is obtained, wherein defects in a portion of the image are marked; using a first classifier to identify the image as defects and background, wherein the first classifier is trained based on a semi-supervised learning method using data of the partial image, and the first classifier is a segmented K-means classifier; and The defects are classified into multiple types of defects using a second classifier, wherein the second classifier is trained based on a semi-supervised learning method using data of the partial image, and the second classifier is a random forest classifier. The method of training the first classifier using the data of the partial image based on a semi-supervised learning method includes: The k-means method is used iteratively. For each k-means iteration, based on the labeled data, the n clusters containing the highest proportion of labeled data are retained, and these n clusters are projected to the next k-means iteration, and the remaining clusters are discarded until the number of discarded clusters is less than a preset threshold.

2. The defect detection method according to claim 1, further comprising: Before the step of using a first classifier to identify the image into defects and background, detecting white continuous areas on the image; cropping the white continuous area from the image along a boundary box of the white continuous area; Adjusting the size of the cropped image of the white continuous area; as well as A pre-trained convolutional neural network is used to perform feature extraction on the adjusted image of the white continuous area.

3. The defect detection method according to claim 2, wherein: Detecting a continuous white area on the image includes: converting the image into a binary image; performing a dilation operation on the binary image; and The bounding box of each white continuous region is determined by non-maximum suppression.

4. The defect detection method according to claim 1, wherein The various types of defects include: Dirt, scratches and light leaks.

5. A defect detection device based on semi-supervised learning, comprising an image acquisition unit configured to obtain an image of the inspection object, wherein defects in a portion of the image are marked; a first classifier configured to identify the image as defects and background, wherein the first classifier is trained based on a semi-supervised learning method using data of the partial image, and the first classifier is a segmented K-means classifier; and a second classifier configured to classify the defect into multiple types of defects, wherein the second classifier is trained based on a semi-supervised learning method using data of the partial image, and the second classifier is a random forest classifier; The first classifier is trained by iteratively using the k-means method, and for each k-means iteration, based on the labeled data, retaining the n clusters containing the highest proportion of labeled data, projecting these n clusters to the next k-means iteration, and discarding the remaining clusters until the number of discarded clusters is less than a preset threshold.

Images