Industrial defect classification detection method, device, medium and electronic equipment

Abstract

The present invention provides an industrial defect classification detection method, device, medium, and electronic equipment, including: obtaining a reference image, collecting an image of an industrial product as an image to be detected; extracting the original features of the image to be detected and the reference image, performing local image block matching, and calculating a first anomaly score. When the first anomaly score is greater than a preset first classification threshold, the image to be detected is classified as a complex image; extracting shallow features of the complex image and the reference image, performing fine-grained feature matching, and calculating a second anomaly score for the complex image. When the second anomaly score is greater than a preset second classification threshold, the complex image is classified as an abnormal image. This method is used to design models with different performance to respectively complete the preliminary screening of images and complex sample judgment. By using a collaborative working method, it can meet the dual requirements of industrial defect detection for real-time performance and high precision, while avoiding resource waste. At the equipment layout level, it can also effectively reduce the hardware cost of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial image defect detection, and in particular to an industrial defect classification detection method, device, medium and electronic equipment. Background Art

[0002] In recent years, with the continuous advancement of intelligent manufacturing, the number of products produced on highly automated production lines has skyrocketed, and manual testing of product quality is no longer sufficient. Consequently, industrial defect detection based on deep learning has garnered increasing attention. Research in this field aims to apply advanced artificial intelligence techniques based on deep neural networks to improve the accuracy of product defect detection, thereby ensuring production efficiency and product reliability.

[0003] However, while deep neural networks achieve high-precision defect detection, they also bring about more complex network structures and place higher demands on hardware performance, making them difficult to deploy on edge devices with limited computing power and storage resources. The inference speed cannot meet the real-time requirements of industrial environments, limiting their promotion in the industrial field.

[0004] In actual industrial production, unlike the defect detection indicators in academic research (Pixel-AP, AUPRO, Image-AUC, etc.), defect detection usually focuses more on the binary classification problem of images, that is, whether the image is a defective sample or a non-defective sample.

[0005] In addition, most industrial sites generally use expensive dedicated defect detection equipment. These devices integrate high-performance hardware and customized software, and are relatively expensive. However, most products in modern industrial production are normal or have obvious defects, and only a small number have subtle defects. Therefore, most samples do not require complex processing. For those simple samples, if high-performance detection equipment is used for detection, it is easy to cause a waste of computing resources.

[0006] Therefore, it is necessary to provide an industrial defect classification detection method that can meet the real-time and high-precision requirements of industrial defect detection while avoiding resource waste. Summary of the Invention

[0007] The purpose of the present invention is to provide an industrial defect classification detection method, device, medium and electronic equipment to achieve the goal of avoiding resource waste while meeting the real-time and high-precision requirements of industrial defect detection.

[0008] In a first aspect, the industrial defect classification detection method provided by the present invention includes: obtaining a reference image and collecting an image of an industrial product as an image to be detected; extracting the original features of the image to be detected and the original features of the reference image to perform local image block matching and calculate a first anomaly score, and when the first anomaly score is greater than a preset first classification threshold, classifying the image to be detected as a complex image; extracting shallow features of the complex image and the shallow features of the reference image and performing fine-grained matching of features, calculating a second anomaly score of the complex image based on the fine-grained matching results of the features of the complex image and the reference image, and when the second anomaly score is greater than a preset second classification threshold, classifying the complex image as an abnormal image.

[0009] The industrial defect classification and detection method provided by this invention has the beneficial effect of fully utilizing a cascaded architecture to design models with different performance characteristics to perform preliminary image screening and complex sample judgment, respectively. This collaborative approach not only meets the dual requirements of real-time and high-precision industrial defect detection, but also avoids resource waste. Furthermore, it effectively reduces the hardware cost of the system at the equipment layout level.

[0010] In a possible embodiment, extracting original features of the image to be detected and original features of the reference image, performing local image block matching, and calculating a first anomaly score includes: extracting original features of the image to be detected and original features of the reference image; performing local feature block matching on the original features of the image to be detected and original features of the reference image to match the feature blocks of the image to be detected with nearest neighbor reference image feature blocks in the reference image; calculating a pixel-level image anomaly score of the image to be detected based on the nearest neighbor reference image feature blocks that match the reference image and the image to be detected; and calculating a first anomaly score of the image to be detected based on the pixel-level image anomaly score of the image to be detected.

[0011] In a possible embodiment, shallow features of the complex image and shallow features of the reference image are extracted and fine-grained matching of the features is performed, and a second anomaly score of the complex image is calculated based on the fine-grained matching results of the features of the complex image and the reference image, including: extracting shallow features of the complex image and shallow features of the reference image; combining the shallow features of the complex image with the original features of the complex image and extracting fine-grained features of the complex image, and combining the shallow features of the reference image with the original features of the reference image and extracting fine-grained features of the reference image; flattening the fine-grained features of the complex image and the fine-grained features of the reference image to obtain a feature block set of the complex image and a feature block set of the reference image; matching the feature block set of the complex image with the feature block set of the reference image to retrieve local neighboring feature blocks of the complex image; calculating an initial anomaly score of the complex image based on the feature block set of the complex image and the local neighboring feature blocks of the complex image; and calculating a second anomaly score of the complex image based on the initial anomaly score of the complex image and foreground information of the complex image.

[0012] In another possible embodiment, an edge defect classification model is constructed to determine whether the image to be detected is a complex image. The edge defect classification model includes: a backbone network, which is used to extract the original features of the image; an image matching module, which is used to perform local feature block matching on the original features of the image to be detected and the original features of the reference image; and a calculation module, which is used to calculate a first anomaly score of the image to be detected.

[0013] In other possible embodiments, the backbone network is a lightweight neural network, and a knowledge distillation technique is designed to transfer the learning capability of a dense convolutional network to the lightweight neural network.

[0014] A server-side defect classification model is constructed to determine whether the complex image is an abnormal image. The server-side defect classification model includes: a shallow feature encoder for extracting shallow features of the image; a zero convolution layer for introducing the shallow features into a local retrieval branch; a local retrieval branch for extracting fine-grained features of the image; and a computing unit for calculating a second anomaly score of the complex image.

[0015] In a second aspect, the present invention further provides an industrial defect classification detection device, comprising:

[0016] An image acquisition unit is used to acquire a reference image and collect an image of an industrial product as an image to be detected; a first classification unit is used to extract original features of the image to be detected and the original features of the reference image, perform local image block matching and calculate a first anomaly score, and when the first anomaly score is greater than a preset first classification threshold, classify the image to be detected as a complex image; a second classification unit is used to extract shallow features of the complex image and the shallow features of the reference image and perform fine-grained feature matching, calculate a second anomaly score of the complex image based on the fine-grained feature matching results of the complex image and the reference image, and when the second anomaly score is greater than a preset second classification threshold, classify the complex image as an abnormal image.

[0017] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned industrial defect classification and detection method is implemented.

[0018] In a fourth aspect, the present invention also provides an electronic device comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned industrial defect classification and detection method.

[0019] For the beneficial effects of the second to fourth aspects, please refer to the description of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a process flow of an industrial defect classification detection method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a distillation process provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a generalization training process for a bottleneck injection layer provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a server-side defect classification model detection process provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the structure of a shallow feature encoder provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of a server-side defect classification model training process provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of an industrial defect classification and detection device provided by an embodiment of the present invention;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with ordinary skills in the field to which the present invention belongs. The words "including" and similar words used in this article mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0029] In response to the problems existing in the prior art, embodiments of the present invention provide an industrial defect classification detection method, device, medium and electronic equipment.

[0030] See the instructions attached Figure 1 This embodiment provides a method for industrial defect classification detection. The method includes:

[0031] S101: Acquire a reference image and collect an image of an industrial product as an image to be detected.

[0032] In a possible embodiment, the reference image is a normal image of an industrial product, and the captured industrial product image is an image of the industrial product to be inspected that is captured during the industrial production process.

[0033] In a specific embodiment, an image acquisition camera is built at an industrial production site by selecting suitable industrial cameras, lenses, light sources, etc., and the image acquisition camera is installed on a three-axis motion platform to obtain images of industrial products at the industrial production site. The image acquisition camera is connected to an edge device, and the acquired images can be directly transmitted to the edge device for image detection.

[0034] S102: extracting original features of the image to be detected and original features of the reference image, performing local image block matching, and calculating a first anomaly score. When the first anomaly score is greater than a preset first classification threshold, classifying the image to be detected as a complex image.

[0035] In a possible embodiment, extracting original features of an image to be detected and original features of a reference image, performing local image block matching, and calculating a first anomaly score includes: extracting original features of the image to be detected and original features of the reference image; performing local feature block matching on the original features of the image to be detected and original features of the reference image to match the feature blocks of the image to be detected with nearest neighbor reference image feature blocks in the reference image; calculating a pixel-level image anomaly score of the image to be detected based on the nearest neighbor reference image feature blocks that match the reference image and the image to be detected; and calculating a first anomaly score of the image to be detected based on the pixel-level image anomaly score of the image to be detected.

[0036] In a specific embodiment, a preliminary detection of whether the image to be detected is a complex image is performed on the edge device. Exemplarily, the detection process specifically includes: extracting the original features of the image to be detected and the original features of the reference image, and performing local feature block matching on the original features of the image to be detected and the original features of the reference image. Specifically, the local feature block matching of the original features of the image to be detected and the original features of the reference image can be achieved using local retrieval, that is, extracting finer fine features contained in the original features of the detection image and the original features of the reference image, flattening the fine features of the image to be detected and the fine features of the reference image respectively, and matching the nearest neighbor reference image feature block for each feature block of the image to be detected in the flattened fine features. The cosine similarity between the feature blocks of the image to be detected and the reference image feature block to be matched to the nearest neighbor is calculated based on the feature vectors of the two feature blocks, and the calculated result is the pixel-level image anomaly score of the image to be detected. For an image to be detected, its first anomaly score is the sum of the first T pixel-level image anomaly scores on the image to be detected.

[0037] In a possible embodiment, the preset first classification threshold is determined by sorting the first anomaly scores of all images to be detected to obtain , calculate the average of the first anomaly scores of each two images to be detected as the candidate classification threshold, that is, For each candidate classification threshold , divide the positive and negative samples according to the threshold and calculate the corresponding recall rate Finally, according to the candidate classification threshold and the recall rate threshold of the edge classifier on the validation set Select an optimal classification threshold as the first classification threshold. The first classification threshold is the maximum value of all classification thresholds that can meet the recall rate requirements of the validation set: When the first anomaly score is less than or equal to the first classification threshold, the corresponding image to be detected is determined to be a normal image. When the first anomaly score is greater than the first classification threshold, the corresponding image to be detected is determined to be a complex image. The complex image needs to be transmitted to the cloud server for further detection.

[0038] In a specific embodiment, a set of images to be detected is input The first anomaly score of the i-th sample calculated by the edge defect classification model is the sum of the first T pixel-level image anomaly scores on the image. The specific calculation satisfies the following formula: .in, , H represents the height of the picture, W represents the width of the picture, Indicates the sorting of all pixel values ​​in the pixel-level anomaly score, and T is a positive integer.

[0039] In one possible embodiment, an edge defect classification model is constructed to determine whether the image to be detected is a complex image. The edge defect classification model includes: a backbone network, which is used to extract the original features of the image; an image matching module, which is used to perform local feature block matching on the original features of the image to be detected and the original features of the reference image; and a calculation module, which is used to calculate a first anomaly score of the image to be detected.

[0040] In one possible embodiment, the backbone network is a lightweight neural network, and a knowledge distillation technique is designed to transfer the learning capability of the dense convolutional network to the lightweight neural network.

[0041] In one possible embodiment, due to the limited performance of edge devices, it is not ideal to directly use a dense convolutional network as the backbone network application, so a lightweight neural network is used as the backbone network. At the same time, in order to improve the performance of the edge defect classification model, the knowledge distillation technology is used to migrate the learning ability of the dense convolutional network to the lightweight neural network, so as to improve the model performance while maintaining a certain inference speed. In addition, during the distillation process, due to the differences in network structure between the dense convolutional network and the lightweight neural network, especially the inconsistency in the dimension of the output features of the target intermediate layer, it is not feasible to directly migrate the features of the dense convolutional network to the lightweight neural network. To this end, the present invention proposes a bottleneck injection method, which reduces the dimension of the output features of the dense convolutional network by introducing a small convolution and a convolution layer with a small step size, so that the target intermediate layer features of the dense convolutional network and the lightweight neural network remain consistent in the channel dimension.

[0042] In a specific embodiment, the dense convolutional network is DenseNet201, and the lightweight neural network is MobileNetV2. In this embodiment, DenseNet201 and MobileNetV2 are used as examples to illustrate how to reduce the dimensionality of the output features of the dense convolutional network so that the target intermediate layer features of the dense convolutional network and the lightweight neural network are consistent in the channel dimension. The specific selection of the dense convolutional network and the lightweight neural network may also be other networks known to those skilled in the art.

[0043] For example, a 1×1 bottleneck injection layer is added to the output of two dense blocks (features.block1 and features.block2) of DenseNet201 to compress the channel dimension to align with the feature layers (features.2 and features.3) of MobileNetV2. The process satisfies the following formula: ; ; .in, , , represents the input image, represents DenseNet201, represents MobileNetV2, Represents the intermediate layer features extracted from DenseNet201, Represents the intermediate layer features extracted from MobileNetV2, the bottleneck injection layer Will The number of feature channels Adjust to .

[0044] See the instructions attached Figure 2 The output features of DenseNet201 are extracted from its two dense blocks (features.block1, features.block2), denoted as D1 and D2 respectively, while the output features of MobileNetV2 are extracted from (features.2, features.3), denoted as M1 and M2 respectively. By distilling the features of the above network layers separately, the consistency of the features extracted from the two in width and height is guaranteed, while the difference in channel dimension is eliminated by the bottleneck injection layer, ensuring more accurate information transfer and matching during the distillation process, and effectively transferring knowledge from DenseNet201 to MobileNetV2. The distillation process satisfies the following formula: ; ; ; .in represents the output features of the teacher model, represents the output features of the student model, is the divergence of the features. In order to comprehensively consider the contribution of different levels of features to the distillation effect, the weighted KL divergence is used as the distillation loss function. Indicates the The weight coefficient of each level reflects the contribution of different feature layers to the distillation loss.

[0045] After distillation, MobileNetV2 serves as the backbone network to extract image features. The extracted results are the original features of the image. Following the backbone network, a module consistent with the CPR local retrieval branch performs local image block matching. This module specifically performs local feature block matching between the original features of the image to be detected and the original features of the reference image.

[0046] See the instructions attached Figure 3 In one possible embodiment, in order to prevent overfitting problems during the distillation process, the bottleneck injection layer is generalized. Specifically, the MVTec 3D-AD dataset is selected and the bottleneck injection layer is trained using the standard CPR training process to obtain its weights. The dataset contains 10 subsets, and the training samples and test samples of all subsets are merged. This can effectively help the bottleneck injection layer learn universal feature representations and improve its generalization ability under different datasets and tasks. During generalization training, the local retrieval branch is trained based on metric learning and an improved contrast loss function. , the training samples are randomly selected of one of the As a reference sample for metric learning. Extracted fine-grained features and Three groups of corresponding local feature blocks at different positions are selected to establish sample pairs for metric learning, and the loss of the three sample pairs is calculated based on the improved contrast loss function for back propagation. represents the first K global neighbor reference image set of the training sample, represents any reference image a in the first K global neighbor reference image set, Represents the features of a obtained by the local retrieval branch.

[0047] S103: Extract shallow features of the complex image and shallow features of the reference image and perform fine-grained matching of the features. Calculate a second anomaly score of the complex image based on the fine-grained matching results of the features of the complex image and the reference image. When the second anomaly score is greater than a preset second classification threshold, classify the complex image as an abnormal image.

[0048] In a possible embodiment, shallow features of a complex image and shallow features of a reference image are extracted and fine-grained matching of the features is performed, and a second anomaly score of the complex image is calculated based on the fine-grained matching results of the features of the complex image and the reference image, including: extracting shallow features of the complex image and shallow features of the reference image; combining the shallow features of the complex image with the original features of the complex image and extracting fine-grained features of the complex image, and combining the shallow features of the reference image with the original features of the reference image and extracting fine-grained features of the reference image; flattening the fine-grained features of the complex image and the fine-grained features of the reference image to obtain a feature block set of the complex image and a feature block set of the reference image; matching the feature block set of the complex image with the feature block set of the reference image to retrieve local neighbor feature blocks of the complex image; calculating an initial anomaly score of the complex image based on the feature block set of the complex image and the local neighbor feature blocks of the complex image; and calculating a second anomaly score of the complex image based on the initial anomaly score of the complex image and foreground information of the complex image.

[0049] In a specific embodiment, the detection of whether a complex image is an abnormal image is performed on the server-side defect classification model. Figure 4 The specific detection process includes: extracting shallow features of complex images and shallow features of reference images. Shallow features refer to features containing low-level texture and edge information in the image. Shallow features can make up for the lack of detail representation ability of high-level semantic features of the backbone network. Shallow features are new information for the model. In order to avoid destroying the original ability of the local retrieval branch and prevent instability in model training, shallow features are introduced into the local retrieval branch through the zero convolution layer. Zero convolution layer Shallow features The processing can be expressed as the following function: The shallow features corresponding to the global neighbor reference image of the complex image are Processed , the shallow features of complex images are Processed ,Will and The original features corresponding to and After combining, the local search branch Extract fine-grained features, The feature extraction process can be expressed by the following formula: ;right Flatten to get the set of eigenvectors: ,in , express The height of the extracted features, the subscript L represents the identifier of the corresponding network, similarly, express The height of the extracted features, express The width of the extracted features. In a similar way, we get The set of eigenvectors of : . Each image feature block in Perform regional restricted matching between feature block sets in order to retrieve The local neighbor feature block of , the retrieval process is as follows: ,in represents the cosine similarity between two feature vectors, Indicates the size of the local search range. Then calculate the complex image The initial anomaly score : . Estimating branches by foreground Calculated Prospect information : The initial anomaly score of the complex image is multiplied element-wise with the foreground information to obtain the second anomaly score of the complex image: .

[0050] In a specific embodiment, foreground estimation involves predicting the foreground and background of an image. The resulting foreground information (background pixel values ​​are 0, foreground pixel values ​​are 1) is element-wise multiplied by the initial anomaly score family, concentrating the prediction region on the foreground of the image and ignoring incorrect predictions in the background. The foreground estimation branch performs the operations required to obtain this foreground information.

[0051] In one possible embodiment, a shallow feature encoder is constructed based on the Inception module to extract shallow features of the image. Figure 5 , the shallow feature encoder contains three consecutive Inception modules, each of which has the same structure as the Inception module in the local retrieval branch in CPR, but the stride of the convolution and pooling operations is 2. Shallow feature encoder The feature extraction process can be represented by the following function: .

[0052] In one possible embodiment, the global search branch Split Each feature in is composed of multiple feature blocks and constructed based on Feature block cluster center set , It is also decomposed into feature blocks. Global neighbors of Reference Image Set Its original characteristics When, through Calculate separately and The histogram vector of the feature block is then calculated based on and The spatial distance between the histogram vectors is obtained.

[0053] In one possible embodiment, a server-side defect classification model is constructed to determine whether a complex image is an abnormal image. The server-side defect classification model includes: a shallow feature encoder for extracting shallow features of the image; a zero convolution layer for introducing the shallow features into a local retrieval branch; the local retrieval branch for extracting fine-grained features of the image; and a computing unit for calculating a second anomaly score of the complex image.

[0054] In a specific embodiment, see the attached Figure 6 In order to obtain a better shallow feature encoder, zero convolution layer and local retrieval branch, the parameter learning process is optimized based on metric learning and contrast loss function. Specifically, the training samples of the server classifier are a subset of the training samples used by the edge classifier. Given a training image , in its global neighbor reference image set Randomly select one As a reference image for metric learning. extract and The original characteristics and ,go through The shallow features processed are and , the shallow features of the two are summed with the corresponding original features and then pass through the local retrieval branch Extracting finer-grained features and , and further flatten to construct the corresponding feature vector set and .

[0055] For example, for metric learning training, this study sets up three different sample pairs, namely normal sample pairs, abnormal sample pairs, and distant sample pairs. The three sample pairs are selected as follows: Normal sample pair: Normal pixel points and the same position Pixels, Abnormal sample pairs: Abnormal pixels and the same position Pixels, Distant sample pairs: Normal pixels and Any pixel point, and the coordinate distance between the pixel points is greater than the set local search range Among them, the training set contains normal samples and abnormal samples. The abnormal samples mark the abnormal areas in the image, which can clearly know which pixels are abnormal or normal.

[0056] The contrast loss function used by the server classifier is:

[0057] ,

[0058] The training is completed when the loss function is minimized. and The inner product of represents the cosine similarity of the sample pair, and are the boundary thresholds of positive samples and negative samples respectively, is a weight constant to balance positive and negative samples, is the weight of the distant sample pair, which is calculated by and The distance between them is obtained, Represents the label of the sample, the positive sample is 1, the negative sample is 0, and the abnormal sample pairs and distant sample pairs are both negative samples. Indicates the number of sample pairs.

[0059] The industrial defect classification and detection method provided by the present invention faces the challenges of industrial defect detection in image binary classification, cost control and real-time performance. A cascade classification scheme is designed based on cascade classifiers and edge computing. Through the collaborative work of edge devices and servers, it not only meets the dual requirements of industrial defect detection for real-time performance and high precision, but also effectively reduces the hardware cost of the system. In this solution, the application of edge devices can replace a large number of traditional expensive defect detection machines. An edge-end defect classification model is designed on the edge device to complete the preliminary screening of most images that do not require complex processing. A server-side defect classification model with better performance is deployed on the server side to process complex samples that cannot be accurately judged by the edge device side. Through the technical solution of the present invention, many traditional expensive defect detection machines can be reduced through the application of edge devices, significantly reducing hardware investment and maintenance costs.

[0060] Applying the industrial defect classification and detection method of this invention to inspect images of industrial products fully leverages the cascaded architecture design of models with different performance characteristics. Through collaborative work, this method not only meets the dual requirements of real-time and high-precision industrial defect detection, but also avoids resource waste. Furthermore, it effectively reduces the hardware cost of the system at the equipment deployment level.

[0061] See the instructions attached Figure 7 This embodiment also provides an industrial defect classification detection device, which is used to implement the above method embodiment. The device includes:

[0062] The image acquisition unit 201 is used to acquire a reference image and collect an image of an industrial product as an image to be detected.

[0063] The first classification unit 202 is used to extract original features of the image to be detected and the original features of the reference image, perform local image block matching, and calculate a first anomaly score. When the first anomaly score is greater than a preset first classification threshold, the image to be detected is classified as a complex image.

[0064] The second classification unit 203 is used to extract shallow features of the complex image and shallow features of the reference image and perform fine-grained feature matching, calculate a second anomaly score of the complex image based on the fine-grained feature matching results of the complex image and the reference image, and classify the complex image as an abnormal image when the second anomaly score is greater than a preset second classification threshold.

[0065] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0066] In other embodiments of the present application, the present application discloses an electronic device, such as Figure 8 As shown, the electronic device 300 may include: one or more processors 301; a memory 302; a display 303; one or more applications (not shown); and one or more computer programs 304. The above components may be connected via one or more communication buses 305. The one or more computer programs 304 are stored in the above memory and configured to be executed by the one or more processors 301. The one or more computer programs 304 include instructions, which may be used to execute the following: Figure 1 and each step in the corresponding embodiment.

[0067] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0068] The functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0069] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk, and other media that can store program code.

[0070] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for detecting industrial defects by classification, characterized in that: include: Obtain reference images and collect industrial product images as images to be detected; Extracting original features of the image to be detected and original features of the reference image to perform local image block matching and calculate a first anomaly score, and when the first anomaly score is greater than a preset first classification threshold, classifying the image to be detected as a complex image; Extracting shallow features of the complex image and shallow features of the reference image and performing fine-grained feature matching, calculating a second anomaly score of the complex image based on the fine-grained feature matching results of the complex image and the reference image, and classifying the complex image as an abnormal image when the second anomaly score is greater than a preset second classification threshold; The method comprises: extracting the original features of the image to be detected and the original features of the reference image, performing local image block matching, and calculating a first anomaly score, including: extracting the original features of the image to be detected and the original features of the reference image; performing local feature block matching on the original features of the image to be detected and the original features of the reference image, so as to match the feature blocks of the image to be detected with the nearest neighbor reference image feature blocks in the reference image; calculating the pixel-level image anomaly score of the image to be detected based on the nearest neighbor reference image feature blocks that match the reference image and the image to be detected; and calculating the first anomaly score of the image to be detected based on the pixel-level image anomaly score of the image to be detected. Extracting shallow features of the complex image and shallow features of the reference image and performing fine-grained matching of the features, and calculating a second anomaly score of the complex image based on the fine-grained matching results of the features of the complex image and the reference image, including: extracting shallow features of the complex image and shallow features of the reference image; combining the shallow features of the complex image with the original features of the complex image and extracting fine-grained features of the complex image, and combining the shallow features of the reference image with the original features of the reference image and extracting fine-grained features of the reference image; flattening the fine-grained features of the complex image and the fine-grained features of the reference image to obtain a feature block set of the complex image and a feature block set of the reference image; matching the feature block set of the complex image with the feature block set of the reference image to retrieve local neighboring feature blocks of the complex image; calculating an initial anomaly score of the complex image based on the feature block set of the complex image and the local neighboring feature blocks of the complex image; and calculating a second anomaly score of the complex image based on the initial anomaly score of the complex image and foreground information of the complex image.

2. The method according to claim 1, characterized in that An edge defect classification model is constructed to determine whether the image to be detected is a complex image. The edge defect classification model includes: A backbone network, wherein the backbone network is used to extract original features of the image; An image matching module, configured to perform local feature block matching on the original features of the image to be detected and the original features of the reference image; A calculation module is used to calculate a first anomaly score of the image to be detected.

3. The method according to claim 2, characterized in that The backbone network is a lightweight neural network, and the knowledge distillation technology is designed to transfer the learning ability of the dense convolutional network to the lightweight neural network.

4. The method according to claim 1, wherein A server-side defect classification model is constructed to determine whether the complex image is an abnormal image. The server-side defect classification model includes: Shallow feature encoder, used to extract shallow features of the image; A zero convolution layer, used to introduce the shallow features into the local retrieval branch; Local retrieval branch, used to extract fine-grained features of the image; A calculation unit is configured to calculate a second anomaly score of the complex image.

5. An industrial defect classification detection device, characterized in that: The device comprises: An image acquisition unit is used to acquire a reference image and collect an image of an industrial product as an image to be detected; a first classification unit, configured to extract original features of the image to be detected and original features of the reference image, perform local image block matching, and calculate a first anomaly score, and classify the image to be detected as a complex image when the first anomaly score is greater than a preset first classification threshold; a second classification unit, configured to extract shallow features of the complex image and shallow features of the reference image and perform fine-grained feature matching, calculate a second anomaly score of the complex image based on the fine-grained feature matching results of the complex image and the reference image, and classify the complex image as an abnormal image when the second anomaly score is greater than a preset second classification threshold; The method comprises: extracting the original features of the image to be detected and the original features of the reference image, performing local image block matching, and calculating a first anomaly score, including: extracting the original features of the image to be detected and the original features of the reference image; performing local feature block matching on the original features of the image to be detected and the original features of the reference image, so as to match the feature blocks of the image to be detected with the nearest neighbor reference image feature blocks in the reference image; calculating the pixel-level image anomaly score of the image to be detected based on the nearest neighbor reference image feature blocks that match the reference image and the image to be detected; and calculating the first anomaly score of the image to be detected based on the pixel-level image anomaly score of the image to be detected. Extracting shallow features of the complex image and shallow features of the reference image and performing fine-grained matching of the features, and calculating a second anomaly score of the complex image based on the fine-grained matching results of the features of the complex image and the reference image, including: extracting shallow features of the complex image and shallow features of the reference image; combining the shallow features of the complex image with the original features of the complex image and extracting fine-grained features of the complex image, and combining the shallow features of the reference image with the original features of the reference image and extracting fine-grained features of the reference image; flattening the fine-grained features of the complex image and the fine-grained features of the reference image to obtain a feature block set of the complex image and a feature block set of the reference image; matching the feature block set of the complex image with the feature block set of the reference image to retrieve local neighboring feature blocks of the complex image; calculating an initial anomaly score of the complex image based on the feature block set of the complex image and the local neighboring feature blocks of the complex image; and calculating a second anomaly score of the complex image based on the initial anomaly score of the complex image and foreground information of the complex image.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the industrial defect classification and detection method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: include: processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so as to enable the electronic device to perform the industrial defect classification and detection method according to any one of claims 1 to 4.

Images