A high real-time industrial defect detection method, device, electronic device and storage medium

Abstract

The present invention relates to a high-real-time industrial defect detection method, comprising: extracting features from a to-be-detected image through a wide residual network to obtain features to be detected; calculating features to be detected through a re-implemented K nearest neighbor algorithm to obtain an abnormal parameter matrix; wherein the re-implemented K nearest neighbor algorithm is obtained in the following manner: importing a normal feature matrix as a constant into a re-implemented K nearest neighbor algorithm, integrating the re-implemented K nearest neighbor algorithm into the wide residual network, and converting them together into an ONNX format; performing minimum value screening and bilinear interpolation on the abnormal parameter matrix in turn to obtain an adjusted abnormal parameter matrix, and screening the adjusted abnormal parameter matrix through a feature mean to obtain an abnormal detection result. The high-real-time industrial defect detection method of the present invention enables the feature comparison process to be directly accelerated on a GPU, and the final inference speed reaches about 12ms, and its frame rate is close to 100fps.

Description

Technical Field

[0001] The present invention relates to the field of image anomaly detection, and in particular to a high-real-time industrial defect detection method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence and the improvement of industrial manufacturing level, the quality requirements for products are getting higher and higher, so industrial defect detection technology has been popularized and applied. Among them, traditional industrial defect detection technologies include rule-based expert systems, image processing-based methods and machine learning. Although they can detect common defects in most products, they require a lot of manual intervention, such as manual labeling and parameter adjustment. At the same time, they are difficult to apply to relatively rare, irregular and mutant defects. In addition, for manual labeling of defects, it requires a large number of defective products as defect samples for training models, and the cost of making defect samples is high, and some mutant defects still cannot be directly detected and identified.

[0003] In order to solve the scarcity problem of defective samples at low cost and realize the feature of detecting mutation defects, the existing technology analyzes the abnormal pixel area in the image data through the anomaly detection algorithm, and sets a mean threshold according to the normal sample characteristics to determine whether the abnormal pixel area is a defective content.

[0004] However, since the anomaly detection algorithm needs to frequently expand the normal sample feature library, the existing anomaly detection algorithm stores the normal sample feature library in the memory, so that the anomaly detection algorithm can be modularized, highly scalable and adaptable; specifically, after the feature extraction model is used to extract features from the image to be detected, the result is compared with the normal sample feature library in the memory for distance and comparison, and the calculation result is divided by a threshold to obtain the divided defect area.

[0005] Based on this, the existing anomaly detection algorithm needs to calculate the distance between the normal sample feature library and the feature extraction results in the memory, which is prone to delays, making it difficult to meet the real-time detection needs when the industrial production line is fast, which in turn leads to the inability to timely discover and deal with defects during the product production process, ultimately affecting product quality and production efficiency. Summary of the invention

[0006] Based on this, an object of the present invention is to provide a highly real-time industrial defect detection method.

[0007] A highly real-time industrial defect detection method comprises the following steps:

[0008] S1: Extract features of the image to be detected through a wide residual network to obtain the features to be detected;

[0009] S2: The features to be detected are calculated by re-implementing the K nearest neighbor algorithm to obtain the anomaly parameter matrix;

[0010] The re-implemented K nearest neighbor algorithm is obtained as follows:

[0011] Importing a normal feature matrix as a constant into a re-implemented K nearest neighbor algorithm, integrating the re-implemented K nearest neighbor algorithm into the wide residual network, and converting the re-implemented K nearest neighbor algorithm and the wide residual network together into an ONNX format;

[0012] S3: Perform minimum value screening and bilinear interpolation on the abnormal parameter matrix in sequence to obtain an adjusted abnormal parameter matrix, and screen the adjusted abnormal parameter matrix through a characteristic mean to obtain an abnormal detection result.

[0013] Compared with the prior art, the high real-time industrial defect detection method described in the present invention integrates the normal feature library as a constant directly into the K nearest neighbor algorithm, so that the feature comparison process can be accelerated directly on the GPU, and avoids the need to transmit the features to be detected between different hardware, thereby reducing the delay caused by data transmission and loading. The final inference speed reaches about 12ms, and the frame rate is close to 100fps (Frames Per Second).

[0014] Furthermore, the normal feature matrix and the feature mean are obtained by the following steps:

[0015] T1: extracting features from a normal map of a normal sample training set through the wide residual network to obtain a shallow normal feature map and a deep normal feature map;

[0016] The normal images of the normal sample training set are normal samples in industrial samples;

[0017] T2: Upsample the deep normal feature map and merge the upsampled deep normal feature map with the shallow normal feature map to obtain a fused feature matrix;

[0018] T3: Determine whether the normal graph of the current normal sample training set has been traversed. If yes, execute step T4A and step T4B; if no, execute step T1;

[0019] T4A: Merge all fusion feature matrices to obtain a merged feature matrix;

[0020] T4B: Calculate the mean of all fused feature matrices to obtain the feature mean;

[0021] T5: Process the merged feature matrix through sparse random projection to obtain a sparse random feature matrix;

[0022] T6: Calculate the Euler distance between the merged feature matrix and the sparse random feature matrix to obtain the difference feature matrix;

[0023] T7: Take the maximum value of the difference feature matrix row as the scaling factor, and filter the merged feature matrix by the scaling factor to obtain the normal feature matrix.

[0024] Among them, by processing the merged feature matrix through sparse random matrix projection, the merged feature matrix can maintain the original structure of the data while significantly reducing the dimension of the feature data, thereby reducing the risk of overfitting caused by the dimensionality curse. At the same time, by calculating the Euler distance between the sparse random feature matrix and the merged feature matrix, the deviation between the features after dimensionality reduction and the original features can be quantified, thereby evaluating the quality and impact of the sparse random projection, and then ensuring that key feature information is not lost during the dimensionality reduction process.

[0025] Furthermore, the step S2 comprises the following steps:

[0026] S21A: adding the square sum of the column vector elements of the feature to be detected to the square sum of the column vector elements of a normal feature matrix to obtain a fused feature length vector;

[0027] S21B: multiplying the feature to be detected by a transposed matrix of a normal feature matrix to obtain a fused feature matrix to be detected;

[0028] S22: by fusing the feature length vector and the feature matrix to be detected, the Euclidean distance between the feature to be detected and the normal feature matrix is ​​calculated to obtain the abnormal parameter matrix;

[0029] Furthermore, step S3 includes the following steps:

[0030] S31: Calculate and integrate the minimum parameters in the abnormal parameter matrix to obtain the minimum abnormal parameter matrix;

[0031] S32: performing bilinear interpolation calculation on the minimum abnormal parameter matrix to obtain an adjustment abnormal parameter matrix;

[0032] S33: Screening the adjusted abnormal parameter matrix through a feature mean to obtain an abnormality detection result.

[0033] Among them, since the acquisition of the abnormal parameter matrix is ​​based on the square sum of the column vector elements of the feature, the minimum parameter of the abnormal parameter matrix in the row direction is obtained to simplify and highlight the significant abnormal points in the abnormal parameter matrix, so that the anomaly can be quickly located during anomaly detection.

[0034] In addition, bilinear interpolation can largely maintain the spatial relationship of abnormal parameters, while ensuring the accurate calibration of abnormal regions in the image to be detected, making it easier to directly mark abnormal regions on the image to be detected.

[0035] A highly real-time industrial defect detection device, comprising a feature extraction unit, a similarity calculation unit and a defect area calculation unit;

[0036] The feature extraction unit extracts features of the image to be detected through a wide residual network to obtain features to be detected;

[0037] The similarity calculation unit calculates the features to be detected by re-implementing the K nearest neighbor algorithm to obtain an abnormal parameter matrix;

[0038] The re-implemented K nearest neighbor algorithm is obtained as follows:

[0039] Importing a normal feature matrix as a constant into a re-implemented K nearest neighbor algorithm, integrating the re-implemented K nearest neighbor algorithm into the wide residual network, and converting the re-implemented K nearest neighbor algorithm and the wide residual network together into an ONNX format;

[0040] The defect area calculation unit is used to perform minimum value screening and bilinear interpolation on the abnormal parameter matrix in sequence to obtain an adjusted abnormal parameter matrix, and screen the adjusted abnormal parameter matrix through a characteristic mean to obtain an abnormality detection result.

[0041] Furthermore, the normal feature matrix and the feature mean are obtained by the following steps:

[0042] T1: extracting features from a normal map of a normal sample training set through the wide residual network to obtain a shallow normal feature map and a deep normal feature map;

[0043] The normal images of the normal sample training set are normal samples in industrial samples;

[0044] T2: Upsample the deep normal feature map and merge the upsampled deep normal feature map with the shallow normal feature map to obtain a fused feature matrix;

[0045] T3: Determine whether the normal graph of the current normal sample training set has been traversed. If yes, execute step T4A and step T4B; if no, execute step T1;

[0046] T4A: Merge all fusion feature matrices to obtain a merged feature matrix;

[0047] T4B: Calculate the mean of all fused feature matrices to obtain the feature mean;

[0048] T5: Process the merged feature matrix through sparse random projection to obtain a sparse random feature matrix;

[0049] T6: Calculate the Euler distance between the merged feature matrix and the sparse random feature matrix to obtain the difference feature matrix;

[0050] T7: Take the maximum value of the difference feature matrix row as the scaling factor, and filter the merged feature matrix by the scaling factor to obtain the normal feature matrix.

[0051] Further, the similarity calculation unit includes a feature vector calculation module, a feature matrix calculation module and an abnormal parameter matrix calculation module;

[0052] The feature vector calculation module adds the square sum of the column vector elements of the feature to be detected to the square sum of the column vector elements of a normal feature matrix to obtain a fused feature length vector;

[0053] The feature matrix calculation module is used to multiply the feature to be detected by a transposed matrix of a normal feature matrix to obtain a fused feature matrix to be detected;

[0054] The abnormal parameter matrix calculation module calculates the Euclidean distance between the feature to be detected and the normal feature matrix by fusing the feature length vector and the feature matrix to be detected, and obtains the abnormal parameter matrix;

[0055] Further, the defect area calculation unit includes a minimum abnormality matrix acquisition module, a matrix adjustment module and an abnormality detection module;

[0056] The minimum abnormal matrix acquisition module is used to calculate and integrate the minimum parameters in the abnormal parameter matrix to obtain the minimum abnormal parameter matrix;

[0057] The matrix adjustment module is used to perform bilinear interpolation calculation on the minimum abnormal parameter matrix to obtain an adjusted abnormal parameter matrix;

[0058] The anomaly detection module screens the adjustment anomaly parameter matrix through a characteristic mean to obtain an anomaly detection result.

[0059] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a structural diagram of the highly real-time industrial defect detection device of the present invention;

[0061] Figure 2 This is a flow chart of the high real-time industrial defect detection method of the present invention;

[0062] Figure 3A simple flowchart for re-implementing the K nearest neighbor algorithm;

[0063] Figure 4 It is a specific flow chart of the normal feature matrix and feature mean acquisition method;

[0064] Figure 5 This is a comparison chart between the high real-time industrial defect detection device of the present invention and a general industrial defect detection device. DETAILED DESCRIPTION

[0065] In existing anomaly detection algorithms, when calculating the difference between the normal sample feature library and the feature extraction results, the Facebook AI Similarity Search (Faiss) is usually used to save and load the two-dimensional matrix of the normal sample feature library. However, the loading and calling speed of this library in memory is slow, resulting in a significant reduction in the real-time performance of reasoning after the final deployment of the model.

[0066] In order to solve the delay problem caused by the calculation of the difference between the normal sample feature library and the feature extraction result in the memory of the anomaly detection algorithm, the present invention sets the normal sample feature library as a constant in a similarity algorithm. Then, the similarity algorithm is re-implemented through a function, and the re-implemented function is integrated into the feature extraction network in the anomaly detection algorithm, and finally the feature network is converted into a serialized data structure in the ONNX open format (Open Neural Network Exchange), so that the anomaly detection algorithm can use the streaming parallel computing of the GPU (Graphics Processing Unit) hardware to achieve inference calculation acceleration during the inference process, greatly improving the real-time performance of defect detection.

[0067] However, due to the large number of features in the normal sample feature library, the normal sample feature library is directly deployed as a constant in the feature network of the ONNX development format, which takes up too many parameters and makes deployment difficult. At the same time, when the number of features is large, it is easy to increase unnecessary calculations and easily cause overfitting. Accordingly, the present invention further reduces the dimension of the normal sample features through a sparse random matrix, and uses the normal sample feature library after dimension reduction as a constant to significantly reduce the number of features in the normal sample feature library.

[0068] Based on the above design, the present invention proposes a highly real-time industrial defect detection method, and based on the method, proposes a highly real-time industrial defect detection device.

[0069] Please also see Figure 1 and Figure 2 , Figure 1 This is a structural diagram of the highly real-time industrial defect detection device of the present invention. Figure 2 This is a flow chart of the high real-time industrial defect detection method described in the present invention.

[0070] The highly real-time industrial defect detection device comprises a feature extraction unit 1 , a similarity calculation unit 2 and a defect area calculation unit 3 .

[0071] The feature extraction unit 1 is used to execute step S1: extracting features from the image to be detected through a wide residual network to obtain features to be detected.

[0072] Specifically, the feature extraction is to obtain the features to be detected by sequentially performing convolution, wide residual block processing, normalization and activation on the image to be detected. The wide residual block processing is to perform convolution on the convolved feature image multiple times, and connect the features to each other through residual connections after each convolution.

[0073] Among them, the width (i.e., the number of channels) of the wide residual block of the wide residual network (WRN) is larger than that of the residual network (ResNet), and more levels and more complex features can be extracted, so that the network can extract the features of irregular defect shapes in defects. Accordingly, the wide residual network has better feature extraction capability for detecting tiny cracks or irregular defects in industrial images, and can extract more abnormal features.

[0074] Since the wide residual block can be set to different execution times according to different needs to balance the depth of feature extraction and the computational complexity of the overall network, the present invention is not specifically limited here. At the same time, convolution, normalization and activation are common knowledge in the field, and the present invention will not be described in detail here.

[0075] The similarity calculation unit 2 is used to execute step S2: calculate the features to be detected by re-implementing the K nearest neighbor algorithm to obtain an abnormal parameter matrix.

[0076] Specifically, the similarity calculation unit 2 includes a feature vector calculation module 21A, a feature matrix calculation module 21B and an abnormal parameter matrix calculation module 22 .

[0077] The feature vector calculation module 21A is used to execute step S21A: adding the square sum of the column vector elements of the feature to be detected and the square sum of the column vector elements of a normal feature matrix to obtain a fused feature length vector.

[0078] Specifically, the sum of squares of all elements of each column vector of the feature to be detected is calculated to obtain the length vector of the feature to be detected, and the sum of squares of all elements of the column vector of the normal feature matrix is ​​calculated to obtain the length vector of the normal feature.

[0079] Next, the feature length vector to be detected and the normal feature length vector are added to obtain a fused feature length vector.

[0080] The length is the sum of squares of the norms; the calculation of the length simplifies the identification of the original features and optimizes the efficiency of the subsequent calculation of the distance (ie, similarity) between feature vectors.

[0081] Take the feature fmX to be detected of an input image extracted by the wise resnet50 network as an example:

[0082] Calculate the sum of the squares of all column elements in each column of the feature to be detected fmX, and obtain the length vector Fd of the feature to be detected with the sum of the squares of the column elements as the row vector elements, Fd = [F1, F2, ..., Fn];

[0083] Calculate the sum of squares of all column elements in each column of the normal feature matrix fm6 to obtain a normal feature length vector Sd with the sum of squares of column elements as row vector elements, Sd = [S1, S2, ..., Sn];

[0084] The sum of the length vector Fd of the feature to be detected and the length vector Sd of the normal feature is calculated to obtain the fused feature length vector SFd = [SF1, SF2, ..., SFn].

[0085] The feature matrix calculation module 21B is used to execute step S21B: multiply the feature to be detected by a transposed matrix of a normal feature matrix to obtain a fused feature matrix to be detected.

[0086] Specifically, matrix cross multiplication is performed between the feature to be detected and the transposed normal feature matrix, and the similarity is calculated to obtain a fused feature matrix to be detected.

[0087] The transposed matrix corresponds the rows of the normal feature matrix to the columns of the feature to be detected, thereby ensuring that the feature to be detected and the normal feature matrix can be matrix multiplied; and the larger the value of each element in the fused matrix to be detected, the closer the two vectors are in direction, that is, the more similar they are. Accordingly, matrix multiplication is used to ensure that the feature to be detected and the normal feature matrix can be correctly corresponded and compared, thereby effectively calculating the similarity between the two.

[0088] Take the feature fmX to be detected of an input image extracted by the wise resnet50 network as an example:

[0089] Obtain the transposed matrix of the normal feature matrix fm6 to obtain the normal feature transposed matrix fm6T;

[0090] Calculate the product of the feature to be detected fmX and the normal feature transposed matrix fm6T to obtain the fused feature matrix M to be detected;

[0091] The abnormal parameter matrix calculation module 22 is used to execute step S22: by fusing the feature length vector and the feature matrix to be detected, the Euclidean distance between the feature to be detected and the normal feature matrix is ​​calculated to obtain the abnormal parameter matrix.

[0092] Specifically, according to the Euclidean distance formula, it is expressed as follows:

[0093]

[0094] Among them, Fd+Sd is the fused feature length vector, Fd and Sd represent the feature length vector to be detected and the normal feature length vector (i.e., the sum of the squares of each column vector element), respectively; M is the fused feature matrix (the cross product matrix of the feature to be detected and the normal feature matrix); d is the abnormal parameter matrix, which represents the Euclidean distance between the feature elements to be detected and the elements of the normal feature matrix, and is used to measure the similarity.

[0095] Take the feature fmX to be detected of an input image extracted by the wise resnet50 network as an example:

[0096] Obtain the doubled fusion feature matrix to be detected 2M;

[0097] Calculate the Euclidean distance between the feature to be detected fmX and the normal sample feature matrix fm6, that is, subtract the fused feature length vector SFd from the twice fused feature matrix to be detected 2M, and perform square root to obtain the similarity between the feature to be detected fmX and the normal sample feature matrix fm6, that is, the abnormal feature matrix Rt.

[0098] The defect area calculation unit 3 is used to execute step S3: perform minimum value screening and bilinear interpolation on the abnormal parameter matrix in sequence to obtain an adjusted abnormal parameter matrix, and screen the adjusted abnormal parameter matrix through a characteristic mean to obtain an abnormality detection result.

[0099] Specifically, the defect area calculation unit 3 includes a minimum abnormality matrix acquisition module 31 , a matrix adjustment module 32 and an abnormality detection module 33 .

[0100] The minimum abnormality matrix acquisition module 31 is used to execute step S31: calculate and integrate the minimum parameters in the abnormality parameter matrix to obtain the minimum abnormality parameter matrix.

[0101] Specifically, the minimum value in each row of the abnormal parameter matrix is ​​screened out, and all minimum values ​​are constructed into a one-dimensional matrix with minimum values, and the one-dimensional matrix with minimum values ​​is adjusted to a two-dimensional matrix according to 1 / 8 of the width and height of the image to be detected to obtain the minimum abnormal parameter matrix.

[0102] Among them, the minimum abnormal parameter matrix is ​​obtained to highlight the minimum distance between each eigenvector in the abnormal parameter matrix and the normal feature; since the width of the matrix obtained after selecting the minimum value of each row is 1, which is equivalent to a one-dimensional array, it is necessary to further adjust its size to two dimensions, that is, the width and height of the minimum abnormal parameter matrix are 1 / 8 of the width and height of the image to be detected.

[0103] The matrix adjustment module 32 is used to execute step S32: performing bilinear interpolation calculation on the minimum abnormal parameter matrix to obtain an adjusted abnormal parameter matrix.

[0104] Specifically, the size of the minimum anomaly parameter matrix is ​​further adjusted to be consistent with the size of the image to be detected through bilinear interpolation to obtain an adjusted anomaly parameter matrix.

[0105] Among them, the spatial relationship of the abnormal parameters can be largely maintained through bilinear interpolation, while ensuring the accurate calibration of the abnormal area in the image to be detected, which is convenient for directly marking the abnormal area on the image to be detected.

[0106] The anomaly detection module 33 is used to execute step S33: screening the adjustment anomaly parameter matrix through a characteristic mean to obtain an anomaly detection result.

[0107] Specifically, all elements in the adjustment anomaly parameter matrix that are greater than the feature mean are screened out, and the matrix coordinates of the elements are retained to obtain anomaly detection results.

[0108] The matrix coordinates of the elements correspond to the pixel coordinates of the image to be inspected, and accordingly, the abnormal area, ie, the defect area, can be directly located and visualized on the image to be inspected through the retained matrix coordinates.

[0109] See also Figure 3 , Figure 3 A simple process for re-implementing the K nearest neighbor algorithm.

[0110] In order to enable the similarity calculation unit and the feature extraction unit of the present invention to share the hardware parallel calculation of the GPU to achieve the effect of calculation acceleration, the present invention integrates the wide residual network of the feature extraction unit through the following steps:

[0111] SA: A normal feature matrix is ​​introduced as a constant into a reimplemented K nearest neighbor algorithm.

[0112] Specifically, a normal feature matrix is ​​used as a constant of the K nearest neighbor algorithm, and is reimplemented into a function that meets the ONNX format conversion requirements to obtain the reimplemented K nearest neighbor algorithm.

[0113] Among them, by statically embedding the normal feature matrix into the K nearest neighbor algorithm as a fixed reference point of the K nearest neighbor algorithm, the re-implemented K nearest neighbor algorithm can directly access the pre-loaded normal feature matrix, i.e., the fixed reference point, during the feature comparison process, thereby accelerating the matching speed.

[0114] SB: The re-implemented K nearest neighbor algorithm is integrated with the wide residual network and converted into ONNX format together to obtain the highly real-time industrial defect detection method described in the present invention.

[0115] Specifically, the re-implemented K nearest neighbor algorithm is integrated as the regression and detection parts of the wide residual network, and the integrated network is packaged into the ONNX format through the torch to ONNX function in the Pytorch library to obtain the high real-time industrial defect detection method described in the present invention.

[0116] Among them, the ONNX format has cross-platform and cross-framework compatibility, so that the integrated network can maintain high performance when deployed in different environments. In addition, the ONNX format is suitable for deep learning reasoning using GPU, thereby ensuring that the entire detection process of the integrated network can efficiently utilize the GPU hardware acceleration characteristics and significantly improve the processing speed.

[0117] Accordingly, after successfully re-implementing the K nearest neighbor algorithm and testing its performance in a conventional GPU environment, the present invention integrates the re-implemented K nearest neighbor algorithm and the wide residual network into a unified deep learning model, and converts the entire model into the ONNX format through the Pytorch tool, ensuring that the GPU hardware accelerated computing can be efficiently utilized in the abnormal reasoning stage to maintain the high real-time performance of the detection.

[0118] See also Figure 4 , Figure 4 This is a specific flow chart of the method for obtaining the normal feature matrix and feature mean.

[0119] In order to obtain a normal feature matrix and feature mean adapted to the re-implemented K nearest neighbor algorithm, the present invention classifies the industrial sample data set, divides the industrial sample data set into normal images and abnormal images, and divides the normal images into a normal sample training set and a normal test sample at an 8:2 ratio; wherein the normal test sample and the abnormal image constitute the test sample. Then, training is performed by executing the following steps to obtain a normal feature matrix and feature mean:

[0120] T1: A wide residual network is used to extract features from the normal map of the normal sample training set to obtain shallow normal feature maps and deep normal feature maps.

[0121] Specifically, the size of the normal image is H in height and W in width; the normal image is input into the wide residual network, and after convolution, several wide residual blocks, normalization and activation in sequence, an 8-fold downsampled shallow feature map fm1 and a 16-fold downsampled deep normal feature map fm2 are obtained.

[0122] Among them, the size of the shallow feature map fm1 is H / 8 in height, W / 8 in width, and 512 in number of layers; the size of the deep normal feature map fm2 is H / 16 in height, W / 16 in width, and 1024 in number of layers.

[0123] Since the wide residual network is consistent with the wide residual network used in the high real-time industrial defect detection method described in the present invention, the above content (feature extraction unit 1) has introduced the content about the wide residual network, and the present invention will not go into details here.

[0124] T2: Upsample the deep normal feature map and merge the upsampled deep normal feature map with the shallow normal feature map to obtain a fused feature matrix.

[0125] Specifically, the deep normal feature map fm2 is upsampled by 2 times the nearest neighbor, and the upsampled feature map is merged with the shallow normal feature map fm1, and the merged feature map is converted into a two-dimensional matrix to obtain the fused feature matrix fm3.

[0126] Among them, the size of the fused feature map fm3 is 1536 in height and H / 8×W / 8 in width; the feature loss of the deep normal feature map is reduced by nearest neighbor upsampling, and feature fusion is performed with the shallow normal feature map to ensure that the semantic information of the deep feature map is fully fused, so as to enhance the fusion of the semantic information and position information of the final fused feature matrix.

[0127] T3: Determine whether the normal graph of the current normal sample training set has been traversed. If yes, execute step T4A and step T4B; if no, execute step T1.

[0128] Specifically, through the current fusion feature matrix, determine whether all normal images in the current normal sample training set have completed feature extraction and obtained the fusion feature matrix corresponding to the normal image? If yes, it is determined that the traversal is completed and step T4A is executed; if not, it is determined that the traversal is not completed and step T1 is continued.

[0129] T4A: Merge all fused feature matrices to obtain a merged feature matrix.

[0130] Specifically, after completing the traversal of the normal sample training set, the normal Figure 1A corresponding fused feature matrix, based on which, all fused feature matrices are merged to obtain a merged feature matrix fm4.

[0131] Here, assuming that the number of all feature matrices is N, the size of the merged feature matrix fm4 is 1536 in height and H / 8×W / 8×N in width.

[0132] T4B: Calculate the mean of all fused feature matrices to obtain the feature mean.

[0133] Specifically, the mean is calculated by summing up all elements in all fused feature matrices and dividing the sum by the total number of elements to obtain the global feature average value, that is, the feature mean.

[0134] Among them, since the feature mean can be a weighted average, an average of the sum of all features or an adaptive weighted average according to the needs of different sample images, such as a normal image in which the background occupies most of the area but the product (detection target) is relatively fixed, the relevant target feature parts can be weighted and the average can be taken; and for a normal image in which the product (detection target) occupies most of the image area, the average of the sum of all features can be directly taken; if the background occupies most of the area but the position of the product (detection target) is random, the features can be adaptively weighted averaged. Accordingly, the present invention does not specifically limit the calculation method of the feature mean.

[0135] In this embodiment, the feature mean is calculated by executing steps T1 to T3 on the test sample set and combining the mean of the resulting fusion feature map fm3 to obtain the threshold parameters of the entire test sample set and the normal training sample set, that is, the feature mean. At the same time, the feature mean needs to remain modifiable during the deployed inference process.

[0136] T5: Process the merged feature matrix through sparse random projection to obtain a sparse random feature matrix.

[0137] Specifically, the merged feature matrix fm4 is linearly transformed by a sparse random matrix to obtain a sparse random feature matrix sm. Most of the elements in the sparse random matrix are 0, and non-zero elements are randomly distributed throughout the matrix.

[0138] Among them, by processing the merged feature matrix through sparse random matrix projection, the merged feature matrix can maintain the original structure of the data while significantly reducing the dimension of the feature data, thereby reducing the overfitting risk caused by the dimensionality curse.

[0139] Since the feature volume of the normal sample feature library is large, directly deploying it to the ONNX format will take up a large number of parameters, making it difficult to deploy and increasing the amount of calculation and the risk of overfitting. Therefore, feature dimensionality reduction through sparse random matrix projection can effectively reduce the feature volume.

[0140] T6: Calculate the Euler distance between the merged feature matrix and the sparse random feature matrix to obtain the difference feature matrix.

[0141] Specifically, the difference feature matrix fm5 is obtained by calculating the Euler distance between the merged feature matrix fm4 and the sparse random feature matrix sm, and combining the corresponding Euler distances as elements into a matrix.

[0142] Among them, by calculating the Euler distance between the sparse random feature matrix and the merged feature matrix, the deviation between the features after dimensionality reduction and the original features can be quantified, so as to evaluate the quality and impact of the sparse random projection, and then ensure that key feature information is not lost in the dimensionality reduction process.

[0143] Meanwhile, the calculation of the Euler distance is common knowledge in the art, which is equivalent to the L2 norm. In addition, the method for calculating the distance also includes the Manhattan distance. The present invention does not specifically limit the method for calculating the distance.

[0144] T7: Take the maximum value of the difference feature matrix row as the scaling factor, and filter the merged feature matrix by the scaling factor to obtain the normal feature matrix.

[0145] Specifically, the maximum value of each row in the difference feature matrix fm5 is calculated, and its maximum value is used as the scale factor M. The scale factor M is used to filter out the columns corresponding to the scale factor M in the merged feature matrix fm4. Finally, all the filtered columns are matrix-constructed to obtain the normal feature matrix fm6.

[0146] Among them, the size of the normal feature matrix fm6 is 1536 in height and H / 8×W / 8×N×M in width; the most significant feature in the merged feature matrix fm4 is screened out through the scale factor M, that is, the most important feature in the merged feature matrix. The significant feature highlights the key information in the feature information. At the same time, the key information can be used to characterize the feature information defined as normal in the normal image, which can be used as a normal feature matrix.

[0147] At the same time, the evaluated difference feature matrix is ​​used as a screening condition to ensure that key features are not lost during the feature dimensionality reduction process, thereby ensuring the effectiveness of the normal feature matrix.

[0148] Compared with the prior art, the present invention converts the normal sample feature library in the anomaly detection algorithm into a matrix table data structure and embeds it into the model as a constant, thereby avoiding the frequent dynamic loading and calculation of normal samples in the memory during the inference stage, thereby reducing the delay caused by data transmission and loading. In addition, the re-implemented k-nearest neighbor algorithm is integrated with the wide residual network and converted into the ONNX format, so that the overall network fully utilizes the parallel computing capabilities of the GPU hardware, significantly improving the real-time performance of the inference stage, and finally reaching an inference speed of about 12ms, with a frame rate close to 100fps (Frames Per Second). For specific effect comparison charts, please refer to Figure 5 .

[0149] In addition, the present invention ensures that the key feature information of the normal feature library is kept intact during the feature dimensionality reduction process through sparse random projection and Euler distance calculation, and screens out important features based on the row maximum value in the difference feature graph after Euler distance calculation, constructs a low-parameter and effective normal feature matrix, and solves the deployment difficulties and overfitting risks caused by the excessive feature quantity of the normal sample feature library.

[0150] Based on the same inventive concept, the present application also provides an electronic device, which may be a terminal device such as a server, a desktop computing device or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement a high-real-time industrial defect detection method according to an embodiment of the present invention; and the memory is used to store a computer program executable by the processor.

[0151] Based on the same inventive concept, the present application also provides a computer-readable storage medium, corresponding to the embodiment of the aforementioned high real-time industrial defect detection method, wherein the computer-readable storage medium stores a computer program thereon, and when the program is executed by the processor, the steps of a high real-time industrial defect detection method recorded in any of the above embodiments are implemented.

[0152] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0153] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A highly real-time industrial defect detection method, characterized in that: The following steps are involved: S1: Extract features of the image to be detected through a wide residual network to obtain the features to be detected; S2: The features to be detected are calculated by re-implementing the K nearest neighbor algorithm to obtain the anomaly parameter matrix; The re-implemented K nearest neighbor algorithm is obtained as follows: Importing a normal feature matrix as a constant into a re-implemented K nearest neighbor algorithm, integrating the re-implemented K nearest neighbor algorithm into the wide residual network, and converting the re-implemented K nearest neighbor algorithm and the wide residual network together into an ONNX format; The re-implemented K nearest neighbor algorithm specifically includes the following sub-steps: S21A: adding the square sum of the column vector elements of the feature to be detected and the square sum of the column vector elements of the normal feature matrix to obtain a fused feature length vector; S21B: multiplying the feature to be detected by the transposed matrix of the normal feature matrix to obtain a fused feature matrix to be detected; S22: by fusing the feature length vector and the feature matrix to be detected, the Euclidean distance between the feature to be detected and the normal feature matrix is ​​calculated to obtain the abnormal parameter matrix; S3: Perform minimum value screening and bilinear interpolation on the abnormal parameter matrix in sequence to obtain an adjusted abnormal parameter matrix, and screen the adjusted abnormal parameter matrix through a characteristic mean to obtain an abnormality detection result.

2. The high real-time industrial defect detection method according to claim 1, characterized in that: The normal feature matrix and the feature mean are obtained by the following steps: T1: extracting features from a normal map of a normal sample training set through the wide residual network to obtain a shallow normal feature map and a deep normal feature map; The normal images of the normal sample training set are normal samples in industrial samples; T2: Upsample the deep normal feature map and merge the upsampled deep normal feature map with the shallow normal feature map to obtain a fused feature matrix; T3: Determine whether the normal graph of the current normal sample training set has been traversed. If yes, execute step T4A and step T4B; if no, execute step T1; T4A: Merge all fusion feature matrices to obtain a merged feature matrix; T4B: Calculate the mean of all fused feature matrices to obtain the feature mean; T5: Process the merged feature matrix through sparse random projection to obtain a sparse random feature matrix; T6: Calculate the Euler distance between the merged feature matrix and the sparse random feature matrix to obtain the difference feature matrix; T7: Take the maximum value of the difference feature matrix row as the scaling factor, and filter the merged feature matrix by the scaling factor to obtain the normal feature matrix.

3. The high real-time industrial defect detection method according to claim 2, characterized in that: The step S3 comprises the following steps: S31: Calculate and integrate the minimum parameters in the abnormal parameter matrix to obtain the minimum abnormal parameter matrix; S32: performing bilinear interpolation calculation on the minimum abnormal parameter matrix to obtain an adjustment abnormal parameter matrix; S33: Screening the adjusted abnormal parameter matrix through a feature mean to obtain an abnormality detection result.

4. A highly real-time industrial defect detection device, characterized in that: It includes a feature extraction unit, a similarity calculation unit and a defect area calculation unit; The feature extraction unit extracts features of the image to be detected through a wide residual network to obtain features to be detected; The similarity calculation unit calculates the features to be detected by re-implementing the K nearest neighbor algorithm to obtain an abnormal parameter matrix; The re-implemented K nearest neighbor algorithm is obtained as follows: Importing a normal feature matrix as a constant into a re-implemented K nearest neighbor algorithm, integrating the re-implemented K nearest neighbor algorithm into the wide residual network, and converting the re-implemented K nearest neighbor algorithm and the wide residual network together into an ONNX format; Wherein, the similarity calculation unit also includes a feature vector calculation module, a feature matrix calculation module and an abnormal parameter matrix calculation module; The feature vector calculation module adds the square sum of the column vector elements of the feature to be detected and the square sum of the column vector elements of the normal feature matrix to obtain a fused feature length vector; The feature matrix calculation module is used to multiply the feature to be detected by the transposed matrix of the normal feature matrix to obtain a fused feature matrix to be detected; The abnormal parameter matrix calculation module calculates the Euclidean distance between the feature to be detected and the normal feature matrix by fusing the feature length vector and the feature matrix to be detected, and obtains the abnormal parameter matrix; The defect area calculation unit is used to perform minimum value screening and bilinear interpolation on the abnormal parameter matrix in sequence to obtain an adjusted abnormal parameter matrix, and screen the adjusted abnormal parameter matrix through a characteristic mean to obtain an abnormality detection result.

5. The high real-time industrial defect detection device according to claim 4 is characterized in that: The normal feature matrix and the feature mean are obtained by the following steps: T1: extracting features from a normal map of a normal sample training set through the wide residual network to obtain a shallow normal feature map and a deep normal feature map; The normal images of the normal sample training set are normal samples in industrial samples; T2: Upsample the deep normal feature map and merge the upsampled deep normal feature map with the shallow normal feature map to obtain a fused feature matrix; T3: Determine whether the normal graph of the current normal sample training set has been traversed. If yes, execute step T4A and step T4B; if no, execute step T1; T4A: Merge all fusion feature matrices to obtain a merged feature matrix; T4B: Calculate the mean of all fused feature matrices to obtain the feature mean; T5: Process the merged feature matrix through sparse random projection to obtain a sparse random feature matrix; T6: Calculate the Euler distance between the merged feature matrix and the sparse random feature matrix to obtain the difference feature matrix; T7: Take the maximum value of the difference feature matrix row as the scaling factor, and filter the merged feature matrix by the scaling factor to obtain the normal feature matrix.

6. The high real-time industrial defect detection device according to claim 5 is characterized in that: The defect area calculation unit includes a minimum abnormality matrix acquisition module, a matrix adjustment module and an abnormality detection module; The minimum abnormal matrix acquisition module is used to calculate and integrate the minimum parameters in the abnormal parameter matrix to obtain the minimum abnormal parameter matrix; The matrix adjustment module is used to perform bilinear interpolation calculation on the minimum abnormal parameter matrix to obtain an adjusted abnormal parameter matrix; The anomaly detection module screens the adjustment anomaly parameter matrix through a characteristic mean to obtain an anomaly detection result.

7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a high real-time industrial defect detection method as described in any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used for a high real-time industrial defect detection method as described in any one of claims 1 to 3.

Images