An unsupervised defect detection method based on deep learning feature correlation

Abstract

An unsupervised defect detection method based on deep learning feature correlation, constructing a defect detection model, using a deep learning pre-trained network as an image feature extraction module, abstractly describing the input image, training the defect detection model with normal samples, obtaining a feature template library, extracting features from the image to be detected during detection, comparing the features of the image to be detected with the features of normal samples, performing k-nearest neighbor retrieval on the features of the image to be detected in the feature template library, measuring the distance between the two vectors as the Euclidean distance, measuring the abnormal score of the sample to be detected based on the distance, and judging whether there is a defect. The present invention performs defect detection by means of unsupervised feature correlation, without the need for cumbersome manual labeling, and only requires a small number of normal samples to complete defect detection and segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision and image processing, and relates to an unsupervised defect detection method based on deep learning feature correlation. Background Art

[0002] In the field of industrial automation, defect detection plays an important role, including the determination of the presence and location of product defects in images. Automatic defect detection of products through machine vision algorithms can greatly improve the efficiency of industrial assembly lines, reduce labor costs, and improve quality.

[0003] Nowadays, deep learning has reached and even exceeded the accuracy of human recognition in certain specific image recognition tasks, and has been widely used in the industrial field. Using deep learning for pixel-level defect detection often requires a large amount of labeled data and a large amount of supervised training such as the existing semantic segmentation method. However, the yield rate of products on actual industrial assembly lines is very high, which means that it is difficult to obtain enough labeled defect data, and a lot of manpower costs are required for labeling. Summary of the invention

[0004] The problem to be solved by the present invention is that in existing industrial inspections, when using deep learning for image recognition, a large amount of labeled data is required for learning, and the learning cost of training the inspection model is high.

[0005] The technical solution of the present invention is: an unsupervised defect detection method based on deep learning feature correlation, constructing a defect detection model, using a deep learning pre-trained network as an image feature extraction module, abstractly describing the input image, training the defect detection model with normal samples, and obtaining a feature template library. During detection, features are extracted from the image to be detected, the features of the image to be detected are compared with the features of normal samples, and k-nearest neighbor retrieval is performed on the features of the image to be detected in the feature template library. The distance between the two vectors is measured as Euclidean distance, and the abnormal score of the sample to be detected is measured according to the distance to determine whether there is a defect.

[0006] Furthermore, some features are selected from the template feature library to obtain a core template feature library.

[0007] Step 1: Get the template feature library and initialize the core template feature library to be empty;

[0008] Step 2: Calculate the Euclidean distance between each template feature and all other template features in the template feature library, and take the minimum value of these Euclidean distances to represent the weight of the template feature;

[0009] Step 3: Select the template feature with the largest weight from all weighted template features calculated in the previous step, add it to the core template feature library, and remove the template feature from the template feature library;

[0010] Step 4: Repeat steps 2 and 3 until the number of core template feature libraries reaches a preset value M to obtain the final core template feature library.

[0011] Furthermore, extracting image features includes the following steps:

[0012] Step 1: Input image with width and height size (H, W);

[0013] Step 2: Use the preset residual neural network to extract features from the input image, and extract intermediate features of two different scales: intermediate feature 1 and intermediate feature 2, with sizes of (H / 8, W / 8, C) and (H / 16, W / 16, 2*C) respectively, where C is the preset number of features;

[0014] Step 3: Use bilinear interpolation to scale the intermediate feature 2 so that its width and height are consistent with those of the intermediate feature 1. After interpolation, the size of the intermediate feature 2 becomes (H / 8, W / 8, 2*C);

[0015] Step 4: Fuse the extracted intermediate feature 1 and the intermediate feature 2 after interpolation in step 3, and splice them along the feature channel, that is, the last dimension. The size after splicing is (H / 8, W / 8, 3*C);

[0016] Step 5: Output the matrix obtained in step 4, which is the depth feature of the original image, with a size of (H / 8, W / 8, 3*C).

[0017] Furthermore, the training of the defect detection model includes the following steps:

[0018] Step 1: Input a total of N training images, all of which are normal samples with width and height dimensions of (H, W);

[0019] Step 2: Extract deep features from each training image through the feature extraction model and combine them into the same matrix with a size of (N, H / 8, W / 8, 3*C);

[0020] Step 3: Reintegrate the matrix shape obtained in step 2 to transform it into (N*H*W / 64,3*C). The entire (N*H*W / 64,3*C) matrix is ​​the template feature library, which is regarded as having N*H*W / 64 template features, and the dimension of each template feature vector is 3*C.

[0021] Furthermore, the defect detection of the image to be inspected is as follows:

[0022] Step 1: Input the image to be detected, with the width and height scale being (H, W);

[0023] Step 2: Extract the depth features of the image to be detected;

[0024] Step 3: Get the size of the deep feature as (H / 8, W / 8, 3*C), reshape the feature to the size of (H*W / 64, 3*C), and regard it as H*W / 64 feature vectors to be tested, each of which is 3*C in size;

[0025] Step 4: Perform k-nearest neighbor search for each feature vector to be tested in the core template feature library. The distance between two vectors is measured as Euclidean distance. Each feature vector to be tested obtains the distance between k nearest template features in the core template feature library and obtains the weight matrix (H*W / 64, k).

[0026] Step 5: Calculate the anomaly score and take the average of each weight vector to get the weight (H*W / 64,1), that is, the anomaly score of each position is the average of the k nearest neighbors. Reshape the matrix to (H / 8,W / 8);

[0027] Step 6: Bidirectionally interpolate the matrix obtained in step 4 to the size of (H, W) to obtain the final pixel-level anomaly score segmentation heat map.

[0028] The present invention performs defect detection through unsupervised feature correlation, without the need for tedious manual labeling, and only requires a small number of normal samples to complete defect detection and segmentation. Specifically, the use of a deep learning pre-training model to extract features can effectively abstractly describe the image, while using the k-nearest neighbor method to calculate the score of the image area, comparing the extracted features of the abnormal image with the features of the normal image, and measuring the score of the test sample based on the distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the feature extraction process of the present invention.

[0030] Figure 2 Schematic diagram of the process of selecting the core feature library of the present invention.

[0031] Figure 3 Schematic diagram of the training process of the defect detection model of the present invention.

[0032] Figure 4 It is a schematic diagram of the detection process of the present invention. DETAILED DESCRIPTION

[0033] The detection method of the present invention constructs a defect detection model, uses a deep learning pre-trained network as an image feature extraction module, performs an abstract description of the input image, trains the defect detection model with normal samples, obtains a feature template library, and during detection, extracts features from the image to be detected, compares the features of the image to be detected with the features of the normal sample, performs a k-nearest neighbor search on the features of the image to be detected in the feature template library, and measures the distance between the two vectors as the Euclidean distance. The abnormality score of the sample to be detected is measured according to the distance to determine whether there is a defect. The present invention extracts features from the image through a pre-trained model to obtain a feature template library, selects a certain number of representative features from the template feature library, and obtains a core template feature library, which can greatly improve the efficiency of the subsequent test process; the abnormality score image is calculated by using the k-nearest neighbor method kNN, which can effectively distinguish normal and abnormal samples. The implementation of the present invention is described in detail below in conjunction with the accompanying drawings.

[0034] like Figure 1 As shown, it is a feature extraction module of the defect detection model of the present invention, and its implementation includes the following steps:

[0035] Step 1: Input image with width and height size (H, W);

[0036] Step 2: Use the preset residual neural network to extract features from the input image, and extract intermediate features of two different scales: intermediate feature 1 and intermediate feature 2, with sizes of (H / 8, W / 8, C) and (H / 16, W / 16, 2*C) respectively, where C is the preset number of features;

[0037] Step 3: Use bilinear interpolation to scale the intermediate feature 2 so that its width and height are consistent with those of the intermediate feature 1. After interpolation, the size of the intermediate feature 2 becomes (H / 8, W / 8, 2*C);

[0038] Step 4: Fuse the extracted intermediate feature 1 and the intermediate feature 2 after interpolation in step 3, and splice them along the feature channel (that is, the last dimension). The size after splicing is (H / 8, W / 8, 3*C);

[0039] Step 5: Output the matrix obtained in step 4, which is the depth feature of the original image, with a size of (H / 8, W / 8, 3*C).

[0040] like Figure 2 As shown, the core feature library selection of the present invention includes the following steps:

[0041] Step 1: The feature size of each image is (H / 8, W / 8, 3*C), the size of the template feature library of N images is (N*H*W / 64, 3*C), and the core template feature library is initialized to be empty;

[0042] Step 2: Calculate the Euclidean distance between each template feature and all other template features in the template feature library, and take the minimum value of these Euclidean distances to represent the weight of the template feature;

[0043] Step 3: Select the template feature with the largest weight from all weighted template features calculated in the previous step, add it to the core template feature library, and remove the template feature from the template feature library;

[0044] Step 4: Repeat steps 2 and 3 until the number of core template feature libraries reaches the preset value M, and the final core template feature library size is (M, 3*C).

[0045] like Figure 3 As shown in the figure, it is a defect detection model training flow chart, which includes the following steps:

[0046] Step 1: Input a total of N training images, all of which are normal samples with width and height dimensions of (H, W);

[0047] Step 2: Pass Figure 1 The feature extraction module shown extracts deep features for each training image and combines them into the same matrix with a size of (N, H / 8, W / 8, 3*C);

[0048] Step 3: Reintegrate the matrix shape obtained in step 2 to transform it into (N*H*W / 64, 3*C). The entire (N*H*W / 64, 3*C) matrix is ​​called a template feature library, which can be regarded as having N*H*W / 64 template features, and each template feature vector dimension is 3*C;

[0049] Step 4: Use Figure 2 The core feature selection module shown selects representative template features from the template feature library;

[0050] Step 5: Output the core template feature library after filtering in step 4.

[0051] The training process of the present invention only uses a pre-trained deep neural network to extract features, which are then stored in a template feature library for subsequent detection. Only a small number of normal samples are required, and no manual labeling is required, thus avoiding a large number of training processes and reducing training costs.

[0052] like Figure 4 As shown, it is a defect detection flow chart of the present invention, which includes the following steps:

[0053] Step 1: Input the image to be detected, with the width and height scale being (H, W);

[0054] Step 2: Adoption Figure 1 The feature extraction module shown extracts deep features;

[0055] Step 3: Get the size of the deep feature as (H / 8, W / 8, 3*C), reshape the feature into the size of (H*W / 64, 3*C), which can be regarded as H*W / 64 feature vectors to be tested, each of which is 3*C in size;

[0056] Step 4: Perform k-nearest neighbor search for each feature vector to be tested in the core template feature library. The distance between two vectors is measured by Euclidean distance. Each feature vector to be tested can obtain k distances to the nearest template features in the core template feature library, and the weight matrix is ​​(H*W / 64, k).

[0057] Step 5: Calculate the anomaly score. Take the mean of each weight vector, that is, the weight is (H*W / 64,1), that is, the anomaly score of each position is the average of the k nearest neighbors, and reshape the matrix to (H / 8,W / 8);

[0058] Step 6: Bidirectionally interpolate the matrix obtained in step 4 to the size of (H, W) to obtain the final segmentation heat map of pixel-level anomaly scores, showing the defects in the image to be detected.

Claims

1. An unsupervised defect detection method based on deep learning feature correlation, characterized by Construct a defect detection model, use a deep learning pre-trained network as an image feature extraction module, perform an abstract description of the input image, train the defect detection model with normal samples, and obtain a feature template library. During detection, extract features from the image to be detected, compare the features of the image to be detected with the features of normal samples, perform a k-nearest neighbor search on the features of the image to be detected in the feature template library, and measure the distance between two vectors using the Euclidean distance. Measure the abnormality score of the sample to be detected based on the distance to determine whether there is a defect. The extraction of image features includes the following steps: Step 1: Input image with dimensions (H, W). Step 2: Use the preset residual neural network to extract features from the input image, and extract two intermediate features of different scales: intermediate feature 1 and intermediate feature 2, with sizes of (H / 8, W / 8, C) and (H / 16, W / 16, 2*C) respectively, where C is the preset number of features; Step 3: Use bilinear interpolation to scale the intermediate feature 2 so that its width and height are consistent with those of the intermediate feature 1. After interpolation, the size of the intermediate feature 2 becomes (H / 8, W / 8, 2*C); Step 4: Fuse the extracted intermediate feature 1 and the intermediate feature 2 after interpolation in step 3, and splice them along the feature channel, that is, the last dimension. The size after splicing is (H / 8, W / 8, 3*C); Step 5: Output the matrix obtained in step 4, which is the depth feature of the original image, with a size of (H / 8, W / 8, 3*C); The training of the defect detection model consists of the following steps: Step 1: Input a total of N training images, all images are normal samples and have a width and height size of (H, W); Step 2: Extract deep features from each training image through the feature extraction model and combine them into the same matrix with a size of (N, H / 8, W / 8, 3*C); Step 3: Reintegrate the matrix shape obtained in step 2 to transform it into (N*H*W / 64, 3*C). The entire (N*H*W / 64, 3*C) matrix is ​​the template feature library, which is regarded as having N*H*W / 64 template features, and the dimension of each template feature vector is 3*C; The defect detection of the image to be inspected is as follows: Step 1: The image to be detected is input into the defect detection model, and the image width and height scale is (H, W); Step 2: Extract the depth features of the image to be detected; Step 3: Get the size of the deep feature as (H / 8, W / 8, 3*C), reshape the feature to the size of (H*W / 64, 3*C), and regard it as H*W / 64 feature vectors to be tested, each of which is 3*C in size; Step 4: For each feature vector to be tested, perform k nearest neighbor search in the template feature library. The distance between two vectors is measured as Euclidean distance. Each feature vector to be tested obtains k distances to the nearest template features in the template feature library, and obtains the weight matrix (H*W / 64, k). Step 5: Calculate the anomaly score and take the average of each weight vector to get the weight (H*W / 64, 1), that is, the anomaly score of each position is the average of the k nearest neighbors. Reshape the matrix to (H / 8, W / 8); Step 6: Bidirectionally interpolate the matrix obtained in step 4 to the size of (H, W) to obtain the final pixel-level anomaly score segmentation heat map.

2. According to the unsupervised defect detection method based on deep learning feature correlation according to claim 1, Its characteristic is that some features are selected from the template feature library to obtain a core template feature library, and the features of the image to be detected are searched by k nearest neighbors in the core template feature library. The core template feature library is obtained as follows: Step 1: Get the template feature library and initialize the core template feature library to be empty; Step 2: Calculate the Euclidean distance between each template feature and all other template features in the template feature library, and take the minimum value of these Euclidean distances to represent the weight of the template feature; Step 3: Select the template feature with the largest weight from all weighted template features calculated in the previous step, add it to the core template feature library, and remove the template feature from the template feature library; Step 4: Repeat steps 2 and 3 until the number of core template feature libraries reaches a preset value M to obtain the final core template feature library.

Images