Defect classification method and device

Abstract

The invention discloses a defect classification method and device, and belongs to the technical field of industrial detection. The defect classification method comprises the following steps: acquiring a feature similarity between a collected to-be-detected image corresponding to a to-be-detected object and each first prompt image in a prompt image library; based on the feature similarities, obtaining a target defect category corresponding to a target prompt image in the plurality of first prompt images; and determining the target defect category as the defect category of the to-be-detected object. The defect classification method can adapt to diversified and complicated defect modes, can reduce storage requirements and calculation time on the basis of keeping feature diversity, uses the prompt image library to perform defect classification, does not need to depend on a large amount of annotation data, consumes few calculation resources, is stable in performance, and is high in practicability. And the accuracy of defect classification is high.

Description

Technical Field

[0001] This application belongs to the technical field of industrial inspection, and particularly relates to a defect classification method and device. Background Art

[0002] In fields such as manufacturing, healthcare, and transportation, it is necessary to automatically identify and classify defect images. In related technologies, there are methods for defect classification based on feature engineering or deep learning. However, defect classification based on feature engineering requires manual design of feature extraction to obtain defect features, lacks generality, and is difficult to adapt to diverse and complex defect patterns. Defect classification based on deep learning requires a large amount of labeled data for model training, with a high labeling cost, consumes a large amount of computing resources during the training process, and the performance of the model is unstable when the data is insufficient, resulting in low accuracy of defect classification. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems existing in the related technologies. For this purpose, this application provides a defect classification method and device, which can adapt to diverse and complex defect patterns, can reduce storage requirements and computing time while retaining feature diversity, and uses a hint image library for defect classification without relying on a large amount of labeled data, consuming less computing resources, having relatively stable performance, and higher accuracy of defect classification.

[0004] In a first aspect, this application provides a defect classification method, which includes:

[0005] Obtaining the feature similarity between the acquired test image corresponding to the object to be tested and each first hint image in the hint image library; the hint image library includes first hint images corresponding to multiple defect categories, and the first hint images corresponding to each defect category are obtained by clustering multiple initial hint images under the defect category;

[0006] Based on each of the feature similarities, obtaining the target defect category corresponding to the target hint image among the multiple first hint images;

[0007] Determining the target defect category as the defect category of the object to be tested.

[0008] According to the defect classification method provided by the embodiments of the present application, by constructing a hint image library including hint images corresponding to each defect category, and then obtaining the target hint image by calculating the feature similarity between the to-be-tested image and each first hint image in the hint image library, the defect category of the to-be-tested object can be obtained according to the target hint image. The defect categories of the hint images included in the hint image library are relatively diverse, so that the versatility of the hint image library is relatively strong, and it can adapt to diverse and complex defect patterns. In addition, by clustering the initial hint images in the hint image library, multiple most representative first hint images can be obtained, which can reduce the storage requirements and calculation time on the basis of retaining feature diversity. Moreover, using the hint image library for defect classification does not rely on a large amount of labeled data, consumes less computing resources, has relatively stable performance, and has a relatively high accuracy of defect classification.

[0009] For the defect classification method according to an embodiment of the present application, obtaining the target defect category corresponding to the target hint image among multiple first hint images based on each of the feature similarities includes:

[0010] Based on each of the feature similarities, using a machine learning algorithm to obtain the target defect category corresponding to the target hint image among multiple first hint images; the machine learning algorithm includes the nearest neighbor algorithm or the K-nearest neighbor algorithm.

[0011] For the defect classification method according to an embodiment of the present application, obtaining the target defect category corresponding to the target hint image among multiple first hint images by using a machine learning algorithm based on each of the feature similarities includes:

[0012] Determine the first hint image corresponding to the maximum feature similarity among multiple first hint images as the target hint image, and determine the defect category of the first hint image corresponding to the maximum feature similarity as the target defect category;

[0013] Or,

[0014] Obtain at least one second hint image whose feature similarity is greater than the first similarity threshold from multiple first hint images;

[0015] Based on the defect categories corresponding to each of the second hint images, determine the defect category with the largest number of corresponding second hint images as the target defect category.

[0016] For the defect classification method according to an embodiment of the present application, the first hint image corresponding to each defect category is obtained by clustering multiple initial hint images under the defect category, including:

[0017] Obtain the feature similarity between any two initial prompt images under the defect category; the feature similarity between any two of the initial prompt images is calculated based on the initial feature vectors corresponding to any two of the initial prompt images;

[0018] Based on the feature similarities under the defect category and multiple initial prompt images, obtain at least one clustering cluster under the defect category; the clustering cluster includes at least one initial prompt image;

[0019] Based on the initial feature vectors corresponding to the initial prompt images in the clustering cluster, obtain the first prompt image.

[0020] In a defect classification method according to an embodiment of the present application, the obtaining the first prompt image based on the initial feature vectors corresponding to the initial prompt images in the clustering cluster includes:

[0021] Obtain the average feature vector of the initial feature vectors corresponding to all the initial prompt images in the clustering cluster;

[0022] Based on the average feature vector, obtain the first prompt image.

[0023] In a defect classification method according to an embodiment of the present application, the obtaining at least one clustering cluster under the defect category based on the feature similarities under the defect category and multiple initial prompt images includes:

[0024] Obtain a target number of cluster centers from the initial feature vectors corresponding to the multiple initial prompt images;

[0025] Based on the feature similarity between each of the remaining initial feature vectors and the target cluster center among at least one of the cluster centers, obtain the clustering cluster corresponding to the target cluster center; the remaining initial feature vectors are the remaining initial feature vectors among the multiple initial feature vectors under the defect category except for the target number of cluster centers.

[0026] In a defect classification method according to an embodiment of the present application, the obtaining the feature similarity between the to-be-tested image corresponding to the to-be-tested object collected and each first prompt image in the prompt image library includes:

[0027] Obtain the first feature vector in the to-be-tested image and the second feature vectors in each of the first prompt images;

[0028] Based on the first feature vector and each of the second feature vectors, obtain the feature similarity between the to-be-tested image and each of the first prompt images.

[0029] In a second aspect, the present application provides a defect classification device, and the device includes:

[0030] The first processing module is configured to obtain the feature similarity between the acquired test image corresponding to the object to be tested and each first prompt image in the prompt image library; the prompt image library includes first prompt images corresponding to multiple defect categories, and the first prompt images corresponding to each defect category are obtained by clustering multiple initial prompt images under the defect category;

[0031] The second processing module is configured to obtain the target defect category corresponding to the target prompt image among the multiple first prompt images based on each feature similarity;

[0032] The third processing module is configured to determine the target defect category as the defect category of the test image.

[0033] According to the defect classification device provided by the embodiments of the present application, by constructing a prompt image library including prompt images corresponding to each defect category, and then calculating the feature similarity between the acquired test image and each first prompt image in the prompt image library to obtain the target prompt image, so that the defect category of the object to be tested can be obtained according to the target prompt image. The defect categories of the prompt images included in the prompt image library are relatively diverse, making the universality of the prompt image library relatively strong, and it can adapt to diverse and complex defect patterns; in addition, by clustering the initial prompt images in the prompt image library, multiple most representative first prompt images can be obtained, which can reduce the storage requirement and calculation time on the basis of retaining feature diversity. Moreover, using the prompt image library for defect classification does not require relying on a large amount of labeled data, consumes less computing resources, has relatively stable performance, and has a relatively high accuracy of defect classification.

[0034] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the defect classification method described in the first aspect above is implemented.

[0035] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the defect classification method described in the first aspect above is implemented.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the defect classification method described in the first aspect above is implemented.

[0037] One or more of the above technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0038] By constructing a hint image library including hint images corresponding to each defect category, and then calculating the feature similarity between the to-be-tested image obtained and each first hint image in the hint image library to obtain a target hint image, the defect category of the to-be-tested object can be obtained according to the target hint image. The defect categories of the hint images included in the hint image library are relatively diverse, making the hint image library highly versatile and adaptable to diverse and complex defect patterns. In addition, by clustering the initial hint images in the hint image library, multiple most representative first hint images can be obtained, which can reduce the storage requirement and calculation time on the basis of retaining feature diversity. Moreover, using the hint image library for defect classification does not rely on a large amount of labeled data, consumes less computing resources, has relatively stable performance, and has a high accuracy of defect classification.

[0039] Furthermore, by calculating the feature similarity between the to-be-tested image and each first hint image, and then adopting a machine learning algorithm to obtain the target defect category corresponding to the target hint image, the target defect category is used as the defect category of the to-be-tested object. The operation is simple, and it can achieve efficient classification without a large amount of labeled data. In addition, defect classification based on the combination of feature extraction and similarity calculation can improve the accuracy of defect classification.

[0040] Even further, by clustering multiple initial hint images under each defect category in the hint image library to obtain at least one first hint image corresponding to each defect category, the storage requirement and calculation time are reduced on the basis of retaining feature diversity, and the accuracy of the clustering result is improved by adopting a hierarchical clustering method for clustering.

[0041] Still further, by using a pre-trained model to extract the first feature vector corresponding to the to-be-tested image and the second feature vector corresponding to the first hint image, there is no need to train or fine-tune the model for a specific data set, the annotation cost is relatively low, and the consumed computing resources are less. The feature similarity between the to-be-tested image and the first hint image can be measured based on the cosine similarity between the first feature vector and the second feature vector, so that defect classification can be performed based on the feature similarity, and the calculation is simple and the robustness is relatively strong.

[0042] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where:

[0044] Figure 1It is one of the flow diagrams of the defect classification method provided by the embodiments of the present application;

[0045] Figure 2 It is the second of the flow diagrams of the defect classification method provided by the embodiments of the present application;

[0046] Figure 3 It is the principle diagram of the defect classification method provided by the embodiments of the present application;

[0047] Figure 4 It is the structural diagram of the defect classification device provided by the embodiments of the present application;

[0048] Figure 5 It is the structural diagram of the electronic device provided by the embodiments of the present application. Detailed Description of the Preferred Embodiments

[0049] Next, the technical solutions in the embodiments of the present application will be clearly described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0050] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order different from those illustrated or described herein, and the objects distinguished by "first", "second", etc. generally belong to the same category, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.

[0051] Next, with reference to the accompanying drawings, the defect classification method, defect classification device, electronic device, and readable storage medium provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0052] Among them, the defect classification method can be applied to a terminal, and specifically can be executed by hardware or software in the terminal.

[0053] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablet computers having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad).

[0054] In each of the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0055] The defect classification method provided in the embodiments of this application may be executed by an electronic device or a functional module or entity in the electronic device that can implement the defect classification method. The electronic devices mentioned in the embodiments of this application include, but are not limited to, mobile phones, tablet computers, computers, cameras, and wearable devices, etc. Hereinafter, taking the electronic device as the execution subject, the defect classification method provided in the embodiments of this application will be described.

[0056] As Figure 1 shown, the defect classification method includes: step 110, step 120, and step 130.

[0057] It should be noted that the defect classification method can be applied to the field of machine vision detection. For example, it can be used to monitor the product quality on the production line, or can be used for fault diagnosis, or can also be applied to other scenarios, which are not limited in this application.

[0058] Step 110: Obtain the feature similarity between the collected test image corresponding to the object to be tested and each first prompt image in the prompt image library;

[0059] In this step, the object to be tested is a workpiece or sample that needs to be tested or inspected during industrial inspections such as quality inspection.

[0060] The object to be tested may include metal parts, plastic products, electronic components, or textiles, etc.

[0061] The test image is an image of the surface of the object to be tested, and the test image may be a grayscale image.

[0062] The prompt image library includes first prompt images corresponding to multiple defect categories.

[0063] There is at least one first prompt image corresponding to each defect category, and the image features of each first prompt image are different.

[0064] The defect categories may include scratches, spots, holes, and damages, etc., and can be customized based on industrial inspection requirements, which are not limited in this application.

[0065] The first prompt image corresponding to each defect category is obtained by clustering multiple initial prompt images under the defect category.

[0066] Before the clustering process, there are multiple initial prompt images corresponding to each defect category. The image features in each initial prompt image are different, and the defect morphologies corresponding to each initial prompt image are inconsistent.

[0067] For multiple initial prompt images under each defect category, clustering processing can be performed on the multiple prompt images to obtain M clusters, that is, the multiple prompt images are reduced to M first prompt images.

[0068] Multiple initial prompt images related to the defect category can be obtained from a professional database or other public image libraries. Each initial prompt image can reflect its respective defect category, and there is at least one initial prompt image under each defect category.

[0069] Preprocessing can be performed on the initial prompt images. For example, the initial prompt images can be grayscaled, normalized, noise-eliminated, and image-enhanced, etc.

[0070] Key features such as edges, textures, and shapes can be extracted from the preprocessed initial prompt images. A unique label can be assigned to each defect category, and the label is used to retrieve and identify relevant prompt images in the prompt image library.

[0071] A relational database (such as MySQL or PostgreSQL, etc.) or a non-relational database (such as MongoDB, etc.) can be used to store and manage the prompt image library. The data table structure can be designed according to requirements. For example, a table containing fields such as image ID, defect category label, and image path can be created to store image information.

[0072] In some embodiments, a deep learning model pre-trained on a large-scale dataset, such as a Residual Neural Network (ResNet), or an architecture-based deep learning model (Visual Transformer, ViT), etc., can be used to extract high-dimensional feature vectors of each first prompt image and store the high-dimensional feature vectors of each first prompt image.

[0073] The feature similarity between the image to be tested and the first prompt image is used to characterize the similarity and difference degree between the two.

[0074] By calculating the distance or similarity degree between the features in the image to be tested and the features in the first prompt image, a value reflecting the similarity between the two, that is, the feature similarity, can be obtained.

[0075] By calculating the Euclidean distance, cosine similarity, or correlation coefficient, etc., between the feature vector of the image to be tested and the feature vector of the first prompt image, the feature similarity between the two can be obtained.

[0076] Step 120: Based on each feature similarity, obtain the target defect category corresponding to the target prompt image among the multiple first prompt images;

[0077] In this step, according to the similarity of each feature, the first prompt image most similar to the image to be measured can be obtained from multiple first prompt images, and it can be determined as the target prompt image;

[0078] Or the first prompt image with the largest number of similar image features to the image to be measured can be obtained from multiple first prompt images, and it can be determined as the target prompt image.

[0079] The number of target prompt images can be one or more. When the number of target prompt images is one, based on the defect categories corresponding to each prompt image stored in the prompt image library, the target defect category corresponding to the target prompt image can be obtained; when the number of target prompt images is multiple, the defect categories corresponding to the multiple target prompt images are the same, that is, the target defect category.

[0080] Step 130: Determine the defect category of the object to be measured as the target defect category.

[0081] In this step, the defect category label associated with the target prompt image can be retrieved from the prompt image library, and then this defect category label can be used as the defect category of the object to be measured.

[0082] In the actual execution process, experts or technicians can conduct manual inspections on the object to be measured to confirm the accuracy of the defect category.

[0083] Other types of features (such as size, position, or severity, etc.) can be combined to improve the accuracy of defect recognition.

[0084] The defect category of the object to be measured can be recorded, and response processing can be performed as needed, such as recording defect information, triggering an alarm, generating a report, arranging maintenance or replacement, etc.

[0085] In this application, the prompt image library can be dynamically expanded, which can adapt to changing application scenarios and newly added defect categories, improving the flexibility of defect classification based on the prompt image library.

[0086] According to the defect classification method provided by the embodiments of the present application, by constructing a prompt image library including prompt images corresponding to each defect category, and then calculating the feature similarity between the image to be measured and each first prompt image in the prompt image library to obtain a target prompt image, the defect category of the object to be measured can be obtained according to the target prompt image. The defect categories of the prompt images included in the prompt image library are diverse, so that the versatility of the prompt image library is strong and it can adapt to diverse and complex defect patterns. In addition, by clustering the initial prompt images in the prompt image library, multiple most representative first prompt images can be obtained, which can reduce the storage requirements and calculation time on the basis of retaining feature diversity. Moreover, using the prompt image library for defect classification does not rely on a large amount of labeled data, consumes less computing resources, has stable performance, and has high accuracy in defect classification.

[0087] In some embodiments, step 120 may include:

[0088] Based on each feature similarity, a machine learning algorithm is used to obtain the target defect category corresponding to the target prompt image among multiple first prompt images.

[0089] In this embodiment, the machine learning algorithm may include the Nearest Neighbor Algorithm or the K-Nearest Neighbors (KNN) algorithm.

[0090] For example, the first prompt image with the highest similarity to the image to be measured can be obtained from multiple first prompt images and determined as the target prompt image.

[0091] The feature similarity can be measured by calculating a certain distance between feature vectors. The smaller the distance, the higher the similarity.

[0092] Alternatively, K first prompt images with the highest feature similarity to the image to be measured can be selected, and then the final target prompt image can be determined according to a certain form of decision (such as majority voting or weighted average, etc.) of these images. K can be determined through experiments or cross-validation.

[0093] As Figure 2 shown, in some embodiments, based on each feature similarity, using a machine learning algorithm to obtain the target defect category corresponding to the target prompt image among multiple first prompt images may include:

[0094] The first prompt image corresponding to the largest feature similarity among multiple first prompt images is determined as the target prompt image, and the defect category of the first prompt image corresponding to the largest feature similarity is determined as the target defect category.

[0095] In this embodiment, a certain similarity measurement method (such as Euclidean distance, cosine similarity, or Manhattan distance, etc.) can be used to calculate the feature similarity between each first prompt image and the image to be measured.

[0096] The feature similarity reflects the degree of closeness between images in the feature space.

[0097] Among all the first prompt images, the first prompt image with the largest feature similarity can be selected as the target prompt image.

[0098] The defect category of the first prompt image corresponding to the largest feature similarity is the target defect category.

[0099] In the actual execution process, the feature similarities between the image to be measured and all the first prompt images can be sorted, and then the first prompt image with the highest feature similarity can be selected as the target prompt image.

[0100] Continue to refer to Figure 2 , in some embodiments, based on each feature similarity, using a machine learning algorithm to obtain the target prompt image from multiple first prompt images, and the target defect category corresponding to the target prompt image, may include:

[0101] Obtain at least one second prompt image with a feature similarity greater than the first similarity threshold from multiple first prompt images;

[0102] Based on the defect categories corresponding to each second prompt image, determine the defect category with the largest number of corresponding second prompt images as the target defect category.

[0103] In this embodiment, the first similarity threshold can be user-defined. For example, the first similarity threshold can be 98%, 98.5%, or 99%, etc., and the present application does not make a limitation.

[0104] At least one second prompt image is an image that is relatively similar to the image to be measured.

[0105] The filtered second prompt images and the defect categories corresponding to the second prompt images can be obtained. The number of second prompt images under each defect category can be counted. For example, all the filtered second prompt images can be traversed, and counting can be performed according to the defect categories of each second prompt image.

[0106] The defect category with the largest number can be determined as the target defect category. For example, multiple first prompt images can be sorted according to the feature similarity, and the top K first prompt images with the highest similarity can be selected as the nearest neighbors. Voting can be performed on the defect categories of the K nearest neighbors. Each defect category obtains votes according to the number of times it appears, and then the category with the most votes can be selected as the predicted category of the image to be classified, that is, the target defect category.

[0107] According to the defect classification method provided by the embodiments of the present application, the feature similarity between the image to be measured and each first prompt image is calculated, and then a machine learning algorithm is used to obtain the target defect category corresponding to the target prompt image, so as to use the target defect category as the defect category of the object to be measured. The operation is simple, and high-efficiency classification can be achieved without a large amount of labeled data. Moreover, defect classification is performed by combining feature extraction and similarity calculation, which can improve the accuracy of defect classification.

[0108] In some embodiments, the first prompt image corresponding to each defect category is obtained by clustering multiple initial prompt images under the defect category, and may include:

[0109] Obtain the feature similarity between any two initial prompt images under the defect category;

[0110] Based on the feature similarities under the defect category and multiple initial prompt images, obtain at least one clustering cluster under the defect category;

[0111] Based on the initial feature vectors corresponding to the initial prompt images in the clustering cluster, obtain the first prompt image.

[0112] In this embodiment, a feature vector can be generated for each initial prompt image. For example, image processing techniques (such as convolutional neural networks or feature extraction algorithms, etc.) can be used to extract the features of each initial prompt image and convert them into feature vectors.

[0113] The feature similarity between any two initial prompt images is calculated based on the initial feature vectors corresponding to any two initial prompt images.

[0114] The feature similarity between any two initial prompt images can be obtained by calculating the cosine similarity, Euclidean distance or other metrics between any two initial feature vectors.

[0115] In the actual execution process, a set list can be initialized. The set list includes a randomly obtained initial feature vector. The set list is used to store different clustering clusters. For each initial prompt image, it can be checked whether it belongs to a certain clustering cluster (it can be checked by comparing the feature similarity with each image in the set list). In the case where it is determined that the initial prompt image does not belong to any set, a new set can be created and the initial image can be added to the new set; in the case where it is determined that the initial prompt image belongs to a certain set (that is, the feature similarity with an image in the set is greater than the second similarity threshold), the initial prompt image is also added to the set.

[0116] A representative feature vector can be obtained based on the initial feature vectors of each initial prompt image in each cluster to obtain a first prompt image. For example, various strategies can be used to obtain the representative feature vector. For example, the average value (or median) of the feature vectors in the set can be obtained.

[0117] In some embodiments, obtaining the first prompt image based on the initial feature vectors corresponding to each initial prompt image in the cluster may include:

[0118] Obtain the average feature vector of the initial feature vectors corresponding to all the initial prompt images in the cluster;

[0119] Obtain the first prompt image based on the average feature vector.

[0120] In this embodiment, for a cluster, the average feature vector corresponding to the initial feature vectors corresponding to all the initial prompt images in the cluster can be obtained. When the dimension of the initial feature vector is d, the average feature vector is also a d-dimensional vector, and each element of the average feature vector is the average value of all the feature vector elements in the corresponding dimension.

[0121] The average feature vector is used to represent the common features of all the initial prompt images in the cluster. After calculating the average feature vector corresponding to the cluster, the average feature vector can be visualized as the first prompt image. For example, a mapping function from the feature vector to the image can be used to map the average feature vector back to the image space; or a correspondence relationship between the average feature vector and the first prompt image can be established.

[0122] In some embodiments, obtaining at least one cluster under the defect category based on the feature similarities under the defect category and multiple initial prompt images includes:

[0123] Obtain the target number of cluster centers from the initial feature vectors corresponding to multiple initial prompt images;

[0124] Based on the feature similarities between the remaining initial feature vectors and the target cluster centers among at least one cluster center, obtain the cluster corresponding to the target cluster center.

[0125] In this embodiment, the target number can be user-defined.

[0126] For example, the target number of cluster centers can be obtained from multiple initial feature vectors by using random selection, feature-based heuristic selection, or other clustering initialization methods.

[0127] The remaining initial feature vectors are the remaining initial feature vectors among the multiple initial feature vectors under the defect category except for the target number of cluster centers.

[0128] For all initial feature vectors (i.e., data points) except the cluster centers, based on their feature similarity to each cluster center, all initial feature vectors assigned to that cluster center can be obtained.

[0129] The above steps can be repeated, recalculating the cluster centers in each iteration and reassigning the data points to the nearest cluster center, and terminating the iteration when conditions such as reaching the set number of iterations, the change in the cluster centers being less than a certain threshold, or the clustering quality metric reaching the optimum are met.

[0130] In the actual execution process, in some embodiments, when the number of initial prompt images in the prompt image library is small, the initial prompt images can be directly determined as the first prompt images.

[0131] When the number of initial prompt images in the prompt image library is large, hierarchical clustering can be performed on the initial prompt images corresponding to each defect category in the prompt image library to obtain M clustering clusters, and then the average of all initial feature vectors within each cluster can be taken to reduce to a representative feature (i.e., the average feature vector).

[0132] As Figure 3 shown, the k-means clustering algorithm can be used to perform clustering processing on the prompt image library, and the cosine similarity between the initial feature vectors can be used as the distance metric.

[0133] First, k initial cluster centers {c1, c2, …, ck} can be randomly obtained. Then, for each initial feature vector, it can be assigned to the nearest cluster according to the cosine similarity. Finally, the cluster centers are updated to the average of the features within the cluster, and the above process is repeated until the cluster assignment no longer changes.

[0134] For example, if there are 8 initial feature vectors corresponding to defect category 1 and they need to be clustered into 4 clusters, the 8 initial feature vectors can be first divided into 2 clusters, and then clustering is performed again within the 2 clusters to obtain two more clusters, resulting in a total of 4 clusters.

[0135] According to the defect classification method provided by the embodiments of the present application, by performing clustering processing on multiple initial prompt images under each defect category in the prompt image library to obtain at least one first prompt image corresponding to each defect category, while retaining the feature diversity, the storage requirements and calculation time are reduced, and the accuracy of the clustering result is improved by using the hierarchical clustering method for clustering.

[0136] As Figure 2 shown, in some embodiments, step 110 may include:

[0137] Obtain the first feature vector in the image to be measured, and the second feature vectors in each first prompt image;

[0138] Based on the first feature vector and each second feature vector, obtain the feature similarity between the image to be measured and each first prompt image.

[0139] In this embodiment, a pre-trained deep learning model (such as a convolutional neural network CNN) or a custom feature extraction algorithm can be used to extract the first feature vector from the image to be measured and the second feature vector from the first prompt image.

[0140] The first feature vector and the second feature vector are usually representations in a high-dimensional space.

[0141] Any similarity measurement method can be selected to characterize the feature similarity between the image to be measured and each first prompt image. For example, Euclidean distance, cosine similarity, Manhattan distance, etc.

[0142] For example, the feature similarity between the first feature vector and the second feature vector can be calculated by the following formula:

[0143]

[0144] Among them, sim is the feature similarity between the first feature vector and the second feature vector, u is the first feature vector, and v is the second feature vector.

[0145] According to the defect classification method provided by the embodiments of the present application, the first feature vector corresponding to the image to be measured and the second feature vector corresponding to the first prompt image are extracted through a pre-trained model, without model training or fine-tuning for a specific data set, the annotation cost is low, and the consumed computing resources are less. The feature similarity between the image to be measured and the first prompt image can be measured based on the cosine similarity between the first feature vector and the second feature vector, so that defect classification can be performed based on the feature similarity, the calculation is simple, and the robustness is strong.

[0146] Next, the defect classification device provided by the present application will be described. The defect classification device described below can be correspondingly referred to the defect classification method described above.

[0147] For the defect classification method provided by the embodiments of the present application, the execution subject can be a defect classification device. In the embodiments of the present application, taking the defect classification device executing the defect classification method as an example, the defect classification device provided by the embodiments of the present application is described.

[0148] The embodiments of the present application also provide a defect classification device.

[0149] Such as Figure 4As shown, the defect classification device includes: a first processing module 410, a second processing module 420, and a third processing module 430.

[0150] The first processing module 410 is configured to obtain the feature similarity between the acquired test image corresponding to the object to be tested and each first prompt image in the prompt image library; the prompt image library includes first prompt images corresponding to multiple defect categories, and the first prompt images corresponding to each defect category are obtained by clustering multiple initial prompt images under the defect category;

[0151] The second processing module 420 is configured to obtain the target defect category corresponding to the target prompt image among the multiple first prompt images based on each feature similarity;

[0152] The third processing module 430 is configured to determine the target defect category as the defect category of the test image.

[0153] According to the defect classification device provided by the embodiments of the present application, by constructing a prompt image library including prompt images corresponding to each defect category, and then obtaining the target prompt image by calculating the feature similarity between the acquired test image and each first prompt image in the prompt image library, so that the defect category of the object to be tested can be obtained according to the target prompt image. The defect categories of the prompt images included in the prompt image library are relatively diverse, making the universality of the prompt image library relatively strong, and it can adapt to diverse and complex defect patterns; in addition, by clustering the initial prompt images in the prompt image library, multiple most representative first prompt images can be obtained, which can reduce the storage requirements and calculation time on the basis of retaining feature diversity, and using the prompt image library for defect classification does not require relying on a large amount of labeled data, consumes less computing resources, has relatively stable performance, and has relatively high defect classification accuracy.

[0154] In some embodiments, the second processing module 420 may further be configured to:

[0155] Based on each feature similarity, use a machine learning algorithm to obtain the target defect category corresponding to the target prompt image among the multiple first prompt images; the machine learning algorithm includes the nearest neighbor algorithm or the K-nearest neighbor algorithm.

[0156] In some embodiments, the second processing module 420 may further be configured to:

[0157] Determine the first prompt image corresponding to the largest feature similarity among the multiple first prompt images as the target prompt image, and determine the defect category of the first prompt image corresponding to the largest feature similarity as the target defect category;

[0158] Or,

[0159] Obtain at least one second prompt image from the multiple first prompt images, where the feature similarity of the second prompt image is greater than the first similarity threshold;

[0160] Based on the defect categories corresponding to each second prompt image, determine the defect category with the largest number of corresponding second prompt images as the target defect category.

[0161] In some embodiments, the defect classification device may further include a fourth processing module for:

[0162] Obtain the feature similarity between any two initial prompt images under the defect category; the feature similarity between any two initial prompt images is calculated based on the initial feature vectors corresponding to any two initial prompt images;

[0163] Based on the feature similarities under the defect category and multiple initial prompt images, obtain at least one clustering cluster under the defect category; the clustering cluster includes at least one initial prompt image;

[0164] Based on the initial feature vectors corresponding to each initial prompt image in the clustering cluster, obtain the first prompt image.

[0165] In some embodiments, the fourth processing module may further be used for:

[0166] Obtain the average feature vector of the initial feature vectors corresponding to all the initial prompt images in the clustering cluster;

[0167] Based on the average feature vector, obtain the first prompt image.

[0168] In some embodiments, the fourth processing module may further be used for:

[0169] Obtain a target number of cluster centers from the initial feature vectors corresponding to the multiple initial prompt images;

[0170] Based on the feature similarities between the remaining initial feature vectors and the target cluster center among at least one cluster center, obtain the clustering cluster corresponding to the target cluster center; the remaining initial feature vectors are the remaining initial feature vectors among the multiple initial feature vectors under the defect category except for the target number of cluster centers.

[0171] In some embodiments, the first processing module 410 may further be used for:

[0172] Obtain the first feature vector in the image to be measured and the second feature vectors in each of the first prompt images;

[0173] Based on the first feature vector and each of the second feature vectors, obtain the feature similarity between the image to be measured and each of the first prompt images.

[0174] The defect classification device in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than terminals. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, an in-vehicle electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and may also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0175] The defect classification device in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0176] The defect classification device provided in the embodiments of the present application can implement Figures 1 to 3 each process implemented by the method embodiments. To avoid repetition, it will not be elaborated here.

[0177] In some embodiments, as Figure 5 shown, the embodiments of the present application further provide an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements each process of the above-mentioned defect classification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0178] It should be noted that the electronic device in the embodiments of the present application includes the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0179] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute each process of the above-mentioned defect classification method embodiment and achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0180] On another aspect, the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the execution of each process of the above-mentioned defect classification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0181] On yet another aspect, the embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to realize each process of the above-mentioned defect classification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0182] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-a-chip, etc.

[0183] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0184] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the related technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A defect classification method, characterized in that, Including: Obtaining the feature similarity between the to-be-tested image corresponding to the to-be-tested object collected and each first prompt image in the prompt image library; The prompt image library includes first prompt images corresponding to multiple defect categories, and the first prompt images corresponding to each defect category are obtained by clustering multiple initial prompt images under the defect category; Based on each of the feature similarities, obtaining the target defect category corresponding to the target prompt image among the multiple first prompt images; Determining the target defect category as the defect category of the to-be-tested object.

2. The defect classification method according to claim 1, wherein The obtaining the target defect category corresponding to the target prompt image among the multiple first prompt images based on each of the feature similarities includes: Based on each of the feature similarities, using a machine learning algorithm to obtain the target defect category corresponding to the target prompt image among the multiple first prompt images; the machine learning algorithm includes the nearest neighbor algorithm or the K-nearest neighbor algorithm.

3. The defect classification method according to claim 2, wherein The obtaining the target defect category corresponding to the target prompt image among the multiple first prompt images based on each of the feature similarities includes: Determining the first prompt image corresponding to the maximum feature similarity among the multiple first prompt images as the target prompt image, and determining the defect category of the first prompt image corresponding to the maximum feature similarity as the target defect category; Or, Obtaining at least one second prompt image with a feature similarity greater than a first similarity threshold from the multiple first prompt images; Based on the defect categories corresponding to each of the second prompt images, determining the defect category with the largest number of corresponding second prompt images as the target defect category.

4. The defect classification method according to any one of claims 1-3, characterized in that, The first prompt images corresponding to each of the defect categories are obtained by clustering multiple initial prompt images under the defect category, including: Obtaining the feature similarity between any two initial prompt images under the defect category; the feature similarity between any two initial prompt images is calculated based on the initial feature vectors corresponding to any two initial prompt images; Based on each of the feature similarities under the defect category and the multiple initial prompt images, obtaining at least one clustering cluster under the defect category; the clustering cluster includes at least one initial prompt image; Based on the initial feature vectors corresponding to each initial prompt image in the clustering cluster, obtaining the first prompt image.

5. The defect classification method according to claim 4, characterized in that, The obtaining the first prompt image based on the initial feature vectors corresponding to each initial prompt image in the clustering cluster includes: Obtaining the average feature vector of the initial feature vectors corresponding to all the initial prompt images in the clustering cluster; Obtaining the first prompt image based on the average feature vector.

6. The defect classification method according to claim 4, wherein The obtaining at least one clustering cluster under the defect category based on each of the feature similarities under the defect category and the multiple initial prompt images includes: Obtaining the target number of cluster centers from the initial feature vectors corresponding to the multiple initial prompt images; Obtain the clustering cluster corresponding to the target cluster center based on the feature similarity between the remaining initial feature vectors and the target cluster center among at least one of the cluster centers; the remaining initial feature vectors are the remaining initial feature vectors among the multiple initial feature vectors under the defect category except for the target number of cluster centers.

7. The defect classification method according to any one of claims 1-3, characterized in that, The obtaining of the feature similarity between the to-be-tested image corresponding to the to-be-tested object collected and each first prompt image in the prompt image library includes: Obtain the first feature vector in the to-be-tested image and the second feature vectors in each of the first prompt images; Based on the first feature vector and each of the second feature vectors, obtain the feature similarity between the to-be-tested image and each of the first prompt images.

8. A defect classification device, characterized in that, Includes: A first processing module, configured to obtain the feature similarity between the to-be-tested image corresponding to the to-be-tested object collected and each first prompt image in the prompt image library; The prompt image library includes first prompt images corresponding to multiple defect categories, and the first prompt images corresponding to each defect category are obtained by performing clustering processing on multiple initial prompt images under the defect category; A second processing module, configured to obtain the target defect category corresponding to the target prompt image among the multiple first prompt images based on each of the feature similarities; A third processing module, configured to determine the target defect category as the defect category of the to-be-tested image.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the defect classification method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the defect classification method according to any one of claims 1-7.

Images