Defect classification method and device and electronic equipment
By using the matching method of target image and standard image and the prediction method of defect classification model on the wafer detection machine, the problem of low accuracy of defect image recognition in the prior art is solved, and more efficient wafer detection is achieved.
Patent Information
- Application Number
- CN202311836067.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-27
- Publication Date
- 2025-06-27
AI Technical Summary
The existing wafer detection machines have low accuracy in identifying defect categories of defect images, resulting in low manual recognition efficiency.
By obtaining the target image, local enlarged image and standard image of the target to be classified, match the local enlarged image and standard image. If the matching fails, input the target image to the defect classification model for feature extraction and prediction, and finally determine the final defect category of the target to be classified.
It improves the accuracy of defect category identification, enhances the efficiency of wafer detection, and saves time and labor costs for relevant personnel in semiconductor factories.
Smart Images
Figure CN120219785A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and particularly to a defect classification method, apparatus, and electronic device. Background Art
[0002] The process of manufacturing wafers in a semiconductor factory includes multiple steps such as slicing, grinding, polishing, chemical vapor deposition, lithography, etching, ion implantation, chemical mechanical polishing, etc., and finally realizes the processing and production of several levels of circuits and components on the wafers.
[0003] In the actual manufacturing process of the factory, to ensure that the final wafers meet the standards, it is necessary to use on-line measurement tools for inspection after multiple process steps of wafer manufacturing are completed, and monitor the abnormal conditions on the dies. Then, according to the detected abnormal positions, wafer defect images are taken to identify the defect categories.
[0004] However, the existing wafer inspection machines have the problem of low recognition accuracy for the defect categories of defect images. Summary of the Invention
[0005] This application provides a defect classification method, apparatus, and electronic device to solve the problem that the existing wafer inspection machines have low recognition accuracy for the defect categories of defect images.
[0006] In a first aspect, this application provides a defect classification method, including:
[0007] Obtain a target image, a locally magnified image, and a standard image of the target to be classified;
[0008] When it is determined that the locally magnified image is not a sub-image of the standard image, input the target image into a defect classification model for feature extraction to obtain a feature vector of the target to be classified, and predict a predicted defect category according to the feature vector;
[0009] Determine a target distance of the predicted defect category according to the feature vector and the predicted defect category of the target to be classified;
[0010] Determine the final defect category of the target to be classified according to the predicted defect category and the target distance.
[0011] Optionally,
[0012] The determination that the locally magnified image is not a sub-image of the standard image specifically includes:
[0013] Identify the image scale of the locally magnified image;
[0014] Determine a scaling ratio according to the image scale of the locally magnified image and a preset scale of the standard image;
[0015] Scale the local enlarged image according to the scaling ratio to obtain a scaled image, and the scaled image has the same scale as the standard image;
[0016] Use a preset template matching algorithm to match the scaled image and the standard image;
[0017] If the target image fails to match the scaled image, it is determined that the local enlarged image is not a sub-image of the standard image.
[0018] Optionally, the method further includes:
[0019] If the scaled image is a sub-image of the target image, it is determined that the target to be classified has no defect.
[0020] Optionally, input the target image into a defect classification model for feature extraction to obtain the feature vector of the target to be classified, and predict the predicted defect category according to the feature vector. Specifically, it includes:
[0021] Input the target image into the feature hierarchy of the defect classification model for feature extraction to obtain a feature vector;
[0022] Input the feature vector into the classification hierarchy of the defect classification model for prediction to obtain the predicted defect category.
[0023] Optionally, determine the target distance of the predicted defect category according to the feature vector of the target to be classified and the predicted defect category. Specifically, it includes:
[0024] Determine the cluster center corresponding to the predicted defect category according to the predicted defect category;
[0025] Calculate the target distance of the target to be classified according to the feature vector and the cluster center.
[0026] Optionally, the calculation process of the cluster center includes:
[0027] Obtain the target samples in the sample set of the defect classification model whose actual defect category is consistent with the predicted defect category, and the feature vector corresponding to each target sample;
[0028] Classify the target samples into multiple defect classes according to the actual defect category;
[0029] Calculate the mean value of the feature vectors corresponding to the target samples in each defect class to obtain the cluster center corresponding to the actual defect category.
[0030] Optionally, according to the predicted defect category and the target distance, determine the final defect category of the target to be classified, specifically including:
[0031] According to the predicted defect category, determine the confidence distance corresponding to the predicted defect category;
[0032] Compare the target distance and the confidence distance;
[0033] When the target distance is less than the confidence distance, determine that the predicted category is the final defect category of the target to be classified;
[0034] When the target distance is greater than or equal to the confidence distance, determine that the final defect category of the target to be classified is other defects.
[0035] Optionally, the calculation process of the confidence distance includes:
[0036] According to the feature vectors corresponding to each target sample in each defect category and the clustering center of the defect category, calculate the sample distances of each target sample;
[0037] Sort the sample distances of the target samples in ascending order to obtain a sample distance queue;
[0038] Obtain the sample distance at a preset position in the sample distance queue as the confidence distance.
[0039] Optionally, the step of inputting the target image into a defect classification model for feature extraction to obtain the feature vector of the target to be classified further includes:
[0040] Obtain the text information of the target to be classified;
[0041] Encode the text information to obtain encoded information;
[0042] Input the encoded information into the defect classification model for feature extraction to obtain a text vector;
[0043] Input the stacked feature vector and the text vector into the defect classification model, and fuse and extract to obtain a new feature vector.
[0044] In a second aspect, the present application provides a defect classification device, including:
[0045] An acquisition module, configured to acquire a target image, a locally magnified image, and a standard image of a target to be classified;
[0046] An identification module, which is configured to, when determining that the locally magnified image is not a sub - image of the standard image, input the target image into a defect classification model for feature extraction to obtain a feature vector of the target to be classified, and predict a predicted defect category based on the feature vector; determine a target distance of the predicted defect category according to the feature vector and the predicted defect category of the target to be classified; and determine an actual defect category of the target to be classified according to the predicted defect category and the target distance.
[0047] Optionally, the identification module is specifically configured to:
[0048] Identify the image scale of the locally magnified image;
[0049] Determine a scaling ratio according to the image scale of the locally magnified image and a preset scale of the standard image;
[0050] Scale the locally magnified image according to the scaling ratio to obtain a scaled image, and the scaled image has the same scale as the standard image;
[0051] Use a preset template matching algorithm to match the scaled image and the standard image;
[0052] If the target image fails to match the scaled image, determine that the locally magnified image is not a sub - image of the standard image.
[0053] Optionally, the identification module is further configured to:
[0054] If the scaled image is a sub - image of the target image, determine that the target to be classified has no defect.
[0055] Optionally, the identification module is specifically configured to:
[0056] Determine a cluster center corresponding to the predicted defect category according to the predicted defect category;
[0057] Calculate the target distance of the target to be classified according to the feature vector and the cluster center.
[0058] Optionally, the identification module is specifically configured to:
[0059] Obtain target samples in the sample set of the defect classification model, where the actual defect category is consistent with the predicted defect category, and the feature vector corresponding to each target sample;
[0060] Classify the target samples into multiple defect classes according to the actual defect category;
[0061] Calculate the mean value of the feature vectors corresponding to each of the target samples in each of the defect classes to obtain the clustering center corresponding to the actual defect class.
[0062] Optionally, the recognition module is specifically configured to:
[0063] Determine the confidence distance corresponding to the predicted defect class according to the predicted defect class;
[0064] Compare the target distance and the confidence distance;
[0065] When the target distance is less than the confidence distance, determine that the predicted class is the final defect class of the target to be classified;
[0066] When the target distance is greater than or equal to the confidence distance, determine that the final defect class of the target to be classified is other defects.
[0067] Optionally, the recognition module is specifically configured to:
[0068] Calculate the sample distances of each of the target samples according to the feature vectors corresponding to each of the target samples in each of the defect classes and the clustering center of the defect class;
[0069] Sort the sample distances of the target samples in ascending order to obtain a sample distance queue;
[0070] Obtain the sample distance at a preset position in the sample distance queue as the confidence distance.
[0071] Optionally, the recognition module is further configured to:
[0072] Obtain the text information of the target to be classified;
[0073] Encode the text information to obtain encoded information;
[0074] Input the encoded information into the defect classification model to perform feature extraction to obtain a text vector;
[0075] Input the stacked feature vectors and the text vector into the defect classification model, and fuse and extract to obtain a new feature vector.
[0076] In a third aspect, the present application provides an electronic device, including: a memory and a processor;
[0077] The memory is used to store a computer program; the processor is used to execute the defect classification method in the first aspect and any possible design of the first aspect according to the computer program stored in the memory.
[0078] Fourthly, the present application provides a computer-readable storage medium storing a computer program, and when at least one processor of an electronic device executes the computer program, the electronic device executes the defect classification method in the first aspect and any possible design of the first aspect.
[0079] Fifthly, the present application provides a computer program product including a computer program, and when at least one processor of an electronic device executes the computer program, the electronic device executes the defect classification method in the first aspect and any possible design of the first aspect.
[0080] The defect classification method, device and electronic device provided by the present application obtain a target image, a partial enlarged image and a standard image of a target to be detected and classified currently; match the partial enlarged image and the standard image; if the match is successful, it indicates that the partial enlarged image is a sub-image of the standard image and there are no defects in the target to be classified; if the match fails, it indicates that the partial enlarged image is not a sub-image of the standard image and there are defects in the target to be classified; when the match fails, the target image of the target to be classified is input into a defect classification model to extract a feature vector of the target to be classified, and a predicted defect category of the target to be classified is obtained through the feature vector; according to the predicted defect category, a clustering center and a confidence distance are determined; a target distance is determined by calculating the distance between the feature vector and the clustering center; the confidence distance and the target distance are compared; and the final defect category of the target to be classified finally obtained is determined according to the comparison result, thereby achieving the effect of improving the accuracy of defect category recognition. Description of the Drawings
[0081] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0082] Figure 1 It is a flowchart of a defect classification method provided by an embodiment of the present application;
[0083] Figure 2 It is a flowchart of matching between a target image and a standard image provided by an embodiment of the present application;
[0084] Figure 3 It is a schematic structural diagram of a defect classification model provided by an embodiment of the present application;
[0085] Figure 4 It is a flowchart of a defect classification example provided by an embodiment of the present application;
[0086] Figure 5 The structural schematic diagram of a defect classification device provided by an embodiment of the present application;
[0087] Figure 6 The hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners
[0088] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the accompanying drawings in the present application. Apparently, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.
[0089] Terms such as "first", "second", "third", "fourth", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances. For example, without departing from the scope of this article, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.
[0090] The process of manufacturing wafers in a semiconductor factory includes multiple steps such as slicing, grinding, polishing, chemical vapor deposition, lithography, etching, ion implantation, chemical mechanical polishing, etc., and finally the processing and manufacturing of several layers of circuits and components are realized on the wafers. In the actual manufacturing process of the factory, to ensure that the final wafers meet the standards, it is necessary to use on-line measurement tools to inspect the wafers after multiple process steps of wafer manufacturing are completed, so as to monitor the abnormal conditions on the dies.
[0091] For wafers with detected die anomalies 78, wafer defect images can be taken according to the detected abnormal positions. Furthermore, the images of the wafers can be analyzed by manual methods or machine vision methods (Machine Vision Applications MVA), and the defect categories can be identified by extracting effective information such as the size and shape of the defect images. Currently, the more commonly used machine vision methods can include difference methods, semantic segmentation methods, filtering methods, morphological methods, etc.
[0092] However, existing wafer inspection machines have problems such as low recognition rate of defect image categories and low manual recognition efficiency. To solve the above problems, an embodiment of the present application proposes a defect classification method. The defect classification method is a semiconductor defect open-set classification method based on multi-dimensional information. First, the method can use a machine learning method to preliminarily identify whether there are defects in the scaled image through a defect discrimination module. If there are defects, the method can continue to use the constructed dual-input neural network to combine the scaled image and its production information for specific defect category division. Among them, the production information can specifically include information such as product model, layer, and process step. The model for realizing defect category division in the present application can be an open-set classification network. This model can identify unseen defect categories as other categories. The present application can more quickly and accurately achieve defect classification of the target to be classified through this method, thereby improving the inspection efficiency of wafers and saving the time and labor costs of relevant personnel in semiconductor factories.
[0093] The present application can use an electronic device as the execution subject to execute the defect classification method of the following embodiments. Specifically, the execution subject can be a hardware device of the electronic device, or a software application in the electronic device that implements the following embodiments, or a computer-readable storage medium installed with the software application that implements the following embodiments, or the code of the software application that implements the following embodiments. Optionally, the electronic device can also be a single-chip microcomputer, a chip, etc. provided with the code of the software application. Optionally, the electronic device can also be a terminal device such as a computer or a server.
[0094] The technical solution of the present application will be described in detail below with specific embodiments. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0095] Figure 1 The flowchart of a defect classification method provided by an embodiment of the present application is shown. As Figure 1 shown, with an electronic device as the execution subject, the method of this embodiment can include the following steps:
[0096] S101. Obtain the target image, local enlarged image, and standard image of the target to be classified.
[0097] In this embodiment, the electronic device obtains the target image of the target to be classified currently being detected. The target image can be the overall image of the target to be classified. Optionally, the target to be classified can be a wafer or a local part of the wafer. Optionally, the target image can be denoted as image1.
[0098] The electronic device can also obtain a locally magnified image of the target to be classified. The locally magnified image may include the area where defects are most likely to occur in the target to be classified. Optionally, the locally magnified image of the target to be classified obtained by the electronic device may be one or more. Optionally, the scale of the locally magnified image is different from that of the target image. Optionally, the locally magnified image may be denoted as image2.
[0099] The electronic device can also obtain a standard image of the target to be classified. The standard image may be an image obtained by the electronic device in advance for a target without defects. That is, the standard image is a standard reference image (golden image) taken of a target without defects. There may be one standard image corresponding to a type of target to be classified. After the electronic device obtains the standard image of the target, the electronic device can directly obtain the standard image when performing defect detection on this type of target subsequently. Optionally, the standard image and the target image are images of the same angle of the target. Optionally, the standard image and the target image have the same scale.
[0100] S102. When it is determined that the locally magnified image is not a sub - image of the standard image, input the target image into the defect classification model for feature extraction to obtain the feature vector of the target to be classified, and predict the predicted defect category according to the feature vector.
[0101] In this embodiment, the electronic device first needs to determine whether there are defects on the target to be classified. The judgment principle of this application is that there must be no defects on the standard image. For better comparison, this embodiment uses the magnified locally magnified image to compare with the standard image, so as to more accurately determine whether there are defects. This embodiment can use the locally magnified image to match with the standard image to determine whether the locally magnified image is a sub - image of the standard image.
[0102] If the locally magnified image is a sub - image of the standard image, it means that there are no defects in the locally magnified image. The electronic device can mark its defect category as the defect - free category.
[0103] If the locally magnified image is not a sub - image of the standard image, it means that there is a difference between the target in the locally magnified image and the target without defects. That is, there are defects in the area corresponding to the locally magnified image. If there are defects in the locally magnified image, the electronic device can input the target image into the defect classification model. The defect classification model can extract the feature vector when performing feature extraction on the target image. The defect classification model can predict the predicted defect category of the target to be classified according to the feature vector.
[0104] In one example, such as Figure 2As shown, the process by which the electronic device matches the locally magnified image with the standard image may include the following steps:
[0105] Step 1: Identify the image scale of the locally magnified image.
[0106] In this step, before this step is executed, the electronic device needs to obtain the locally magnified image and the standard image.
[0107] Subsequently, the electronic device can perform a scale identification operation through this step.
[0108] Specifically, the electronic device can identify the image content of the scale area in the locally magnified image to obtain the image scale of the locally magnified image.
[0109] Since the scale on the locally magnified image has a high brightness and usually appears at a fixed position in the locally magnified image. Therefore, the electronic device can perform local cropping on the position where the scale is located to obtain a scale slice. For example, the lower 10% height part of the locally magnified image can be cropped as the scale slice.
[0110] The electronic device can use existing algorithms such as the connected component algorithm, or improved algorithms of existing algorithms, to identify the scale slice. The electronic device can identify multiple connected components with the longest horizontal length and use the pixel lengths corresponding to these multiple connected components as the pixel length of the scale. For example, the number of these multiple connected components can be 8.
[0111] The electronic device can adopt a text recognition model constructed based on open source libraries such as EasyOCR, PaddleOCR, and CnOCR to identify the scale value in the scale slice. For example, the scale value can be 100nm. In order to improve the recognition accuracy of the scale value, when training the text recognition model, only the numbers 0-9 and the three letters u, n, and m can be used. This setting can improve the training efficiency and recognition accuracy of the text recognition model by reducing the classification quantity of the text recognition model.
[0112] The electronic device can determine the image scale of the locally magnified image according to the scale value and the pixel length of the scale.
[0113] Step 2: Determine the scaling ratio according to the image scale of the locally magnified image and the preset scale of the standard image. Scale the locally magnified image according to the scaling ratio to obtain a scaled image, and the scaled image has the same scale as the standard image.
[0114] In this step, the electronic device can scale the locally magnified image according to the image scale ratio identified in Step 1.
[0115] Specifically, the electronic device can scale the locally magnified image to a scaled image with the same ratio as the standard image according to the image ratio of the locally magnified image.
[0116] Among them, the calculation of the scaling ratio (resizeRatio) can be achieved through the following formula:
[0117]
[0118] Among them, number image2 represents the scale value of the locally magnified image, and lengtg image2 represents the pixel length of the scale of the locally magnified image. number golden image represents the scale value of the standard image, and length golden image represents the pixel length of the scale of the standard image. Among them, the scale value of the standard image and the pixel length of the scale are preset scales determined according to the standard image. The preset scale is a known parameter.
[0119] The electronic device can scale the locally magnified image according to the scaling ratio to obtain a scaled image. The scaled image obtained after scaling has the same image scale as the standard image.
[0120] Step 3: Use the preset template matching algorithm to match the scaled image and the standard image.
[0121] In this step, the electronic device can determine whether the scaled image is a sub-image of the standard image through template matching.
[0122] Specifically, the electronic device can be pre-set with a template matching algorithm. For example, the template matching algorithm can be methods such as matchTemplate. The electronic device can use this template matching algorithm to match the scaled image and the standard image. If the match is successful, it means that the scaled image is a sub-image of the standard image. Otherwise, if the match fails, it means that there are defects in the scaled image.
[0123] Step 4: If the local magnification image and the scaled image do not match, it is determined that the local magnification image is not a sub-image of the standard image.
[0124] In this step, when it is determined that the local magnification image and the scaled image do not match, the electronic device can determine that the local magnification image does not match the standard image. At this time, the electronic device can confirm that there are defects in the local magnification image. That is, there are defects in the target to be classified.
[0125] In one example, based on the previous example, when the electronic device determines that there are defects in the target image, such asFigure 3 As shown, the electronic device can predict the defect category by inputting the target image into the defect classification model. The specific process may include:
[0126] Step 5: Input the target image into the feature layer of the defect classification model for feature extraction to obtain a feature vector.
[0127] In this step, after obtaining the target image, the electronic device can input the target image into the defect classification model. The defect classification model can perform feature extraction on the target image and obtain a feature vector.
[0128] Specifically, the electronic device can use the first feature network layer to perform feature extraction on the target image to obtain the first feature. The first feature network may include a Convolutional Neural Networks (CNN). The CNN has multiple layers such as convolution, pooling, and fully connected layers. When the target image is input into the CNN, the CNN can perform operations such as convolution and pooling on the target image to achieve feature extraction of the target image. As Figure 3 shown, the input target image can be extracted after passing through the CNN and then mapped through a linear layer to obtain the first feature. The first feature can be M*1 dimensional.
[0129] In another implementation, the defect classification model can be a dual-input network. In addition to inputting the target image, the defect classification model can also input text information. The text information may include various description information such as the product model, level, and process steps of the target to be classified. Adding the features of the text information to the feature vector can increase the amount of information in the feature vector, thereby improving the accuracy of predicting the defect category in the following Step 6.
[0130] The specific steps of fusing the text information into the feature vector may include:
[0131] Step 5.1: Obtain the text information of the target to be classified.
[0132] In this step, the text information may include a large amount of description information of the target to be classified. The amount of data in the text information is usually large and the data content is relatively messy. To better use the text information, the text information can be encoded before being input into the defect classification model, thereby improving data validity and reducing the amount of data for easier model processing.
[0133] Step 5.2: Encode the text information to obtain encoded information.
[0134] In this step, the electronic device can summarize and extract the text information by encoding the text information. Specifically, different encodings corresponding to words and phrases can be set in the controller. The controller can traverse the text to determine the words and phrases contained in the text, and replace the words and phrases with the encoding to obtain the final encoded information.
[0135] Step 5.3: Input the encoded information into the defect classification model for feature extraction to obtain a text vector.
[0136] In this step, the electronic device can input the encoded information into the second feature network. The second feature network can be a multi-layer perceptron network. As Figure 3 shown, the encoded information can be input into the MLP to obtain a text vector. The text vector can be N*1 dimensional.
[0137] Step 5.4: Input the stacked feature vectors and the text vector into the defect classification model, and fuse and extract to obtain a new feature vector.
[0138] In this step, the electronic device can stack the feature vectors obtained in step 5 and the text vector obtained in step 5.3. Optionally, as Figure 3 shown, the stacking can be determined by calculating the exclusive OR value of the feature vector and the text vector. The electronic device can input the stacked features into the third feature network layer for feature extraction to obtain a new feature vector. As Figure 3 shown, the third feature network can be an MLP.
[0139] Step 6: Input the feature vector into the classification layer of the defect classification model for prediction to obtain the predicted defect category.
[0140] In this step, the electronic device can input the feature vector into the classification layer of the defect classification model. The classification layer is the last linear layer of the defect classification model network, which is used to map the deep features to the number of defect categories. For example, when the number of defect categories is 1000, the resulting vector after mapping is 1000*1 dimensional. In the 1000*1 dimension, each dimension can correspond to a defect category. The classification layer can use the softmax activation function to obtain the probability that the feature vector belongs to each defect category. The electronic device can determine the predicted defect category corresponding to the feature vector according to the defect category corresponding to the dimension with the highest probability in the resulting vector.
[0141] S103: Determine the target distance of the predicted defect category according to the feature vector of the target to be classified and the predicted defect category.
[0142] In this embodiment, the electronic device can determine the clustering center corresponding to the predicted defect category according to the predicted defect category. The electronic device can determine the target distance of the predicted defect category by calculating the distance between the feature vector and the clustering center. Among them, the distance can be calculated by methods such as Euclidean distance, Hamming distance, Manhattan distance, and cosine distance.
[0143] In one example, the calculation process of the above target distance can be divided into the following two steps:
[0144] Step 1: Determine the clustering center corresponding to the predicted defect category according to the predicted defect category.
[0145] In this step, the electronic device first needs to determine the clustering center corresponding to the defect category according to the predicted defect category. Optionally, the clustering center of the defect category can be calculated when the defect classification model is completed. Optionally, the calculation process of the clustering center can include the following steps:
[0146] Step 1.1: Obtain the target samples in the sample set of the defect classification model where the actual defect category is the same as the predicted defect category, and the feature vector corresponding to each target sample.
[0147] In this step, after the defect classification model is completed, the electronic device can predict the samples in the sample set. The electronic device can obtain the target samples where the predicted defect category is the same as the actual defect category. Optionally, the sample set can include training samples in the training set and validation samples in the validation set.
[0148] Step 1.2: Divide the target samples into multiple defect categories according to the actual defect category.
[0149] In this step, the electronic device can divide the target samples into multiple defect categories according to their actual defect categories. The electronic device can obtain the feature vectors of each target sample in each defect category.
[0150] Step 1.3: Calculate the mean value of the feature vectors corresponding to each target sample in each defect category to obtain the clustering center corresponding to the actual defect category.
[0151] In this step, the electronic device can calculate the mean value of the feature vectors of multiple target samples corresponding to a defect classification. This mean value is the clustering center of the defect classification.
[0152] For example, for the defect category d1, after the defect classification model is completed, the electronic device can obtain the target samples where the actual defect category and the predicted defect category are both d1, and form a target sample set dd1. The electronic device can obtain the feature vectors of the target samples in the target sample set dd1 and calculate the mean value of these feature vectors to obtain the clustering center of the defect category d1.
[0153] Step 2: Calculate the target distance of the target to be classified according to the feature vector and the clustering center.
[0154] In this step, the electronic device can use a preset distance calculation method to calculate the distance between the feature vector and the clustering center, and obtain the target distance.
[0155] S104: Determine the final defect category of the target to be classified according to the predicted defect category and the target distance.
[0156] In this embodiment, the electronic device can determine the confidence distance according to the preset defect category. The electronic device can compare the confidence distance and the target distance. The electronic device can determine the final defect category of the target to be classified according to the comparison result.
[0157] In one example, the determination process of the final defect category may include the following steps:
[0158] Step 1: Determine the confidence distance corresponding to the predicted defect category according to the predicted defect category.
[0159] In this step, the electronic device first needs to determine the confidence distance corresponding to the defect category according to the predicted defect category. Optionally, the confidence distance of the defect category can be calculated when the defect classification model is completed. Optionally, the calculation process of the confidence distance may include the following steps:
[0160] Step 1.1: Calculate the sample distances of each target sample according to the feature vectors of each target sample in each defect category and the clustering center of the defect category.
[0161] In this step, the electronic device can obtain multiple target samples in each defect category and the clustering center corresponding to each defect category. The electronic device can calculate the sample distance from the feature vector of each target sample in a defect category to the clustering center.
[0162] Step 1.2: Sort the sample distances of the target samples in ascending order to obtain a sample distance queue.
[0163] In this step, the electronic device can sort the sample distances of multiple target samples in each defect category to obtain a sample distance queue. In the sample distance queue, the target samples are sorted in ascending order of the sample distance.
[0164] Step 1.3: Obtain the sample distance at a preset position in the sample distance queue as the confidence distance of the defect category.
[0165] In this step, the thr in the thr quantile can be preset in the electronic device. For example, the thr can be 50%, 95%, etc. The electronic device can obtain the sample distance corresponding to the thr quantile as the confidence distance. For example, when the thr is 95%, the 95% quantile is the sample distance at the 95% position from the front to the back in the sample distance queue. For example, when there are 100 sample distances in the queue, the sample distance at the 95% position is the 95th sample distance. Another example is that when there are 20 sample distances in the queue, the sample distance at the 95% position is the 19th sample distance.
[0166] For example, after the electronic device obtains the target original sample set dd1 composed of multiple target samples with the defect class d1, it can obtain the feature vector of each target sample and the clustering center of the defect class d1. The electronic device can calculate the sample distance from each feature vector to the clustering center. The electronic device can sort these sample distances from smallest to largest. The electronic device can obtain the 0.95 quantile as the confidence distance.
[0167] Step 2: Compare the target distance and the confidence distance. When the target distance is less than the confidence distance, determine that the predicted category is the actual defect category of the target to be classified. When the target distance is greater than or equal to the confidence distance, determine that the actual defect category of the target to be classified is other defects.
[0168] In this step, the electronic device can compare the target distance and the confidence distance.
[0169] If the target distance is less than the confidence distance, it means that the final defect category of the target to be classified is the predicted defect category.
[0170] If the target distance is greater than or equal to the confidence distance, it means that the target to be classified does not actually belong to the predicted defect category.
[0171] Optionally, other defects, that is, other classes, can be set in the electronic device. The electronic device can set the defect category of the target to be classified with an incorrect predicted defect category to this other class. This other class is used to indicate that the defect of the target to be classified belongs to an unseen defect category.
[0172] Optionally, when the electronic device determines that the target to be classified does not belong to the predicted defect category, it can obtain the second-largest probability from the result vector of the defect classification model. If the second-largest probability is greater than the probability threshold, the electronic device can obtain the defect category corresponding to the dimension of the second-largest probability and use this defect category as the new predicted defect category. Here, the probability threshold can be a preset value. For example, the probability threshold can be 0.95. The electronic device can use the new predicted defect category to re-execute step S103 and step 104. By analogy, when the predicted defect category determined by the second-largest probability is incorrect, the electronic device can obtain the third-largest probability. The electronic device can sequentially obtain the predicted defect categories corresponding to each probability until the probability is less than or equal to the probability threshold and then stop. If the electronic device still has not determined the final defect category of the target to be classified when the probability is less than or equal to the probability threshold, the electronic device can set the final defect category of the target to be classified as other category.
[0173] In the defect classification method provided in this application, the electronic device can obtain the target image and the standard image of the target to be detected currently. The electronic device can match the target image and the standard image. If the match is successful, it indicates that the target image is a sub-image of the standard image and there are no defects in the target to be classified. If the match fails, it indicates that the target image is not a sub-image of the standard image and there are defects in the target to be classified. When the match fails, the electronic device can input the target image of the target to be classified into the defect classification model to extract the feature vector of the target to be classified, and use the feature vector to predict the predicted defect category of the target to be classified. The electronic device can determine the clustering center and the confidence distance according to the predicted defect category. The electronic device can determine the target distance by calculating the distance between the feature vector and the clustering center. The electronic device can compare the confidence distance and the target distance. The electronic device can determine the final defect category of the target to be classified finally obtained according to the comparison result. In this application, by fusing feature extraction and confidence judgment, the accuracy of defect category recognition is improved.
[0174] Figure 4 shows a flowchart of a defect classification example provided by an embodiment of this application. In Figures 1 to 3 Based on the shown embodiment, as Figure 4 shown, with the electronic device as the execution subject, the example of this embodiment can include the following steps:
[0175] Step 1: The electronic device obtains the externally input image and text information. Optionally, the image may include a target image, a partial enlarged image, and a standard image of the target to be classified. The target image and the partial enlarged image can be obtained by the electronic device from the outside. The standard image can be an image of a defect-free target pre-obtained by the electronic device. The text information can be the description information of the current target to be classified obtained by the electronic device. Optionally, the text information may include information such as product model, level, and process steps.
[0176] Step 2: The electronic device can achieve the matching of the partial enlarged image and the standard image through a defect discrimination module based on machine learning. In this application, it is default that the standard image does not include defects. Therefore, when the partial enlarged image is a sub-image of the standard image, it indicates that there are no defects in the partial enlarged image. Otherwise, when the partial enlarged image is not a sub-image of the standard image, it indicates that there are defects in the partial enlarged image.
[0177] Step 3: The electronic device can determine whether there are defects in the partial enlarged image. If there are no defects, step 4 is executed. If there are defects, step 5 is executed.
[0178] Step 4: The electronic device can set the defect category of the target to be classified as the defect-free category. The defect-free category is used to indicate that there are no defects in the target to be classified.
[0179] Step 5: The electronic device can input the target image and text information into a dual-input open-set classification network model. The dual-input open-set classification network model may include a defect classification model. The defect classification model in the dual-input open-set classification network model can perform fusion extraction on the target image and text information to obtain a feature vector. The defect classification model in the dual-input open-set classification network model can also classify the feature vector to obtain a predicted defect category.
[0180] Step 6: The electronic device can use the cluster center and the confidence distance to verify the predicted defect category according to the feature vector. If the distance between the feature vector and the cluster center is greater than or equal to the confidence distance, the electronic device can execute step 7. Otherwise, if the distance between the feature vector and the cluster center is less than the confidence distance, the electronic device can execute step 8.
[0181] Step 7: The electronic device can set the final defect category of the target to be classified as the other category. The other category is used to indicate that there are defects in the target to be classified, but the defect category is not currently included in the defect classification model.
[0182] Step 8: The electronic device can set the final defect category of the target to be classified as the predicted defect number category.
[0183] Figure 5 The structural schematic diagram of a defect classification device provided by an embodiment of the present application is shown. As Figure 5 shown, the defect classification device 10 of this embodiment is used to implement the operations corresponding to the electronic device in any of the above method embodiments. The defect classification device 10 of this embodiment includes:
[0184] An acquisition module 11, configured to acquire a target image, a partial enlarged image, and a standard image of a target to be classified.
[0185] An identification module 12, configured to, when it is determined that the partial enlarged image is not a sub-image of the standard image, input the target image into a defect classification model for feature extraction to obtain a feature vector of the target to be classified, and predict a predicted defect category according to the feature vector; determine a target distance of the predicted defect category according to the feature vector and the predicted defect category of the target to be classified; and determine a final defect category of the target to be classified according to the predicted defect category and the target distance.
[0186] Optionally, the identification module 12 is specifically configured to:
[0187] Identify the image scale of the partial enlarged image;
[0188] Determine a scaling ratio according to the image scale of the partial enlarged image and a preset scale of the standard image;
[0189] Scale the partial enlarged image according to the scaling ratio to obtain a scaled image, and the scaled image has the same scale as the standard image;
[0190] Use a preset template matching algorithm to match the scaled image and the standard image;
[0191] If the target image fails to match the scaled image, it is determined that the partial enlarged image is not a sub-image of the standard image.
[0192] Optionally, the identification module 12 is further configured to:
[0193] If the scaled image is a sub-image of the target image, it is determined that the target to be classified has no defect.
[0194] Optionally, the identification module 121 is specifically configured to:
[0195] Input the target image into the feature layer of the defect classification model for feature extraction to obtain a feature vector.
[0196] Input the feature vector into the classification layer of the defect classification model for prediction to obtain a predicted defect category.
[0197] Optionally, the identification module 12 is specifically configured to:
[0198] Determine the clustering center corresponding to the predicted defect category according to the predicted defect category;
[0199] Calculate the target distance of the target to be classified according to the feature vector and the clustering center.
[0200] Optionally, the recognition module 12 is specifically used for:
[0201] Obtain the target samples in the sample set of the defect classification model whose actual defect category is consistent with the predicted defect category, and the feature vector corresponding to each target sample;
[0202] Divide the target samples into multiple defect categories according to the actual defect category;
[0203] Calculate the mean value of the feature vectors corresponding to each target sample in each defect category to obtain the clustering center corresponding to the actual defect category.
[0204] Optionally, the recognition module 12 is specifically used for:
[0205] Determine the confidence distance corresponding to the predicted defect category according to the predicted defect category.
[0206] Compare the target distance and the confidence distance.
[0207] When the target distance is less than the confidence distance, determine that the predicted category is the final defect category of the target to be classified.
[0208] When the target distance is greater than or equal to the confidence distance, determine that the final defect category of the target to be classified is other defects.
[0209] Optionally, the recognition module 12 is specifically used for:
[0210] Calculate the sample distance of each target sample according to the feature vector corresponding to each target sample in each defect category and the clustering center of the defect category;
[0211] Sort the sample distances of the target samples in ascending order to obtain a sample distance queue;
[0212] Obtain the sample distance at a preset position in the sample distance queue as the confidence distance.
[0213] Optionally, the recognition module 12 is further used for:
[0214] Obtain the text information of the target to be classified;
[0215] Encode the text information to obtain encoded information;
[0216] Input the encoded information into the defect classification model for feature extraction to obtain a text vector;
[0217] Input the stacked feature vectors and text vectors into the defect classification model, and fuse and extract to obtain new feature vectors.
[0218] The defect classification device 10 provided by the embodiments of the present application can execute the above method embodiments. For its specific implementation principle and technical effects, please refer to the above method embodiments, which will not be elaborated here in this embodiment.
[0219] Figure 6 Shows a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. As Figure 6 shown, the electronic device 20 is used to implement the operations corresponding to the electronic device in any of the above method embodiments. The electronic device 20 in this embodiment may include: a memory 21, a processor 22, and a communication interface 24.
[0220] The memory 21 is used to store computer programs. The memory 21 may include a high-speed random access memory (Random Access Memory, RAM), and may also include non-volatile storage (Non-Volatile Memory, NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc, etc.
[0221] The processor 22 is used to execute the computer program stored in the memory to implement the defect classification method in the above embodiments. Specifically, please refer to the relevant descriptions in the foregoing method embodiments. The processor 22 may be a central processing unit (Central Processing Unit, CPU), or may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor.
[0222] Optionally, the memory 21 may be either independent or integrated with the processor 22.
[0223] When the memory 21 is a device independent of the processor 22, the electronic device 20 may further include a bus 23. The bus 23 is used to connect the memory 21 and the processor 22. The bus 23 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.
[0224] A communication interface 24 can be connected to the processor 21 through the bus 23. The communication interface 24 can be connected to an external device and is used to obtain a target image and a partially enlarged image of a target to be classified from the external device. The communication interface 24 can also be connected to a production line and is used to send the identified actual defect category to the production line, so as to facilitate the production line to classify and transfer the target to be classified.
[0225] The electronic device provided in this embodiment can be used to execute the above-mentioned defect classification method, and its implementation manner and technical effect are similar, which will not be elaborated here in this embodiment.
[0226] This application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the methods provided by the above various embodiments.
[0227] Among them, the computer-readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, the computer-readable storage medium is coupled to the processor, so that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can be located in an Application Specific Integrated Circuits (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the computer-readable storage medium can also exist as discrete components in a communication device.
[0228] Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically-erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk. The storage medium can be any available medium accessible by a general-purpose or special-purpose computer.
[0229] The present application also provides a computer program product. The computer program product includes a computer program stored in a computer-readable storage medium. At least one processor of the device can read the computer program from the computer-readable storage medium, and the execution of the computer program by the at least one processor causes the device to implement the methods provided by the above various embodiments.
[0230] The embodiments of the present application also provide a chip. The chip includes a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device installed with the chip executes the methods in the above various possible embodiments.
[0231] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or modules can be in an electrical, mechanical, or other form.
[0232] Among them, each module can be physically separated. For example, it can be installed at different positions of a device, or installed on different devices, or distributed to multiple network units, or distributed to multiple processors. Each module can also be integrated together. For example, it can be installed in the same device, or integrated in a set of code. Each module can exist in the form of hardware, or can also exist in the form of software, or can also be implemented in the form of software plus hardware. This application can select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment.
[0233] When the integrated module is implemented in the form of a software functional module, it can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) or a processor to execute some steps of the methods of various embodiments of this application.
[0234] It should be understood that although the steps in the flowcharts in the above embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit and can be executed in other orders. Moreover, at least some of the steps in the figure may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times, and their execution order does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.
[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features. And these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of various embodiments of this application.
Claims
1. A defect classification method, characterized in that, The method includes: Obtaining a target image, a locally magnified image, and a standard image of the target to be classified; When it is determined that the locally magnified image is not a sub - image of the standard image, inputting the target image into a defect classification model for feature extraction to obtain a feature vector of the target to be classified, and predicting a predicted defect category based on the feature vector; Determining a target distance of the predicted defect category according to the feature vector and the predicted defect category of the target to be classified; Determining a final defect category of the target to be classified according to the predicted defect category and the target distance.
2. The method according to claim 1, wherein The determination that the locally magnified image is not a sub - image of the standard image specifically includes: Identifying an image scale of the locally magnified image; Determining a scaling ratio according to the image scale of the locally magnified image and a preset scale of the standard image; Scaling the locally magnified image according to the scaling ratio to obtain a scaled image, and the scaled image has the same scale as the standard image; Using a preset template matching algorithm to match the scaled image and the standard image; If the target image fails to match the scaled image, it is determined that the locally magnified image is not a sub - image of the standard image.
3. The method according to claim 2, characterized in that, The method further includes: If the scaled image is a sub - image of the target image, it is determined that the target to be classified has no defect.
4. The method according to any one of claims 1 to 3, characterized in that, The step of inputting the target image into a defect classification model for feature extraction to obtain a feature vector of the target to be classified, and predicting a predicted defect category based on the feature vector specifically includes: Inputting the target image into the feature hierarchy of the defect classification model for feature extraction to obtain a feature vector; Inputting the feature vector into the classification hierarchy of the defect classification model for prediction to obtain a predicted defect category.
5. The method according to any one of claims 1 to 3, characterized in that, The step of determining a target distance of the predicted defect category according to the feature vector and the predicted defect category of the target to be classified specifically includes: Determining a clustering center corresponding to the predicted defect category according to the predicted defect category; Calculating a target distance of the target to be classified according to the feature vector and the clustering center.
6. The method according to claim 5, wherein The calculation process of the clustering center includes: Obtaining target samples in the sample set of the defect classification model whose actual defect category is the same as the predicted defect category, and the feature vector corresponding to each target sample; Dividing the target samples into multiple defect classes according to the actual defect category; Calculating the mean value of the feature vectors corresponding to the target samples in each defect class to obtain the clustering center corresponding to the actual defect category.
7. The method according to any one of claims 1 to 3, characterized in that, The step of determining a final defect category of the target to be classified according to the predicted defect category and the target distance specifically includes: Determining a confidence distance corresponding to the predicted defect category according to the predicted defect category; Comparing the target distance and the confidence distance; When the target distance is less than the confidence distance, determining that the predicted category is the final defect category of the target to be classified; When the target distance is greater than or equal to the confidence distance, determine that the final defect category of the target to be classified is other defects.
8. The method according to claim 7, wherein The calculation process of the confidence distance includes: Calculate the sample distance of each target sample according to the feature vector corresponding to each target sample in each defect category and the clustering center of the defect category; Sort the sample distances of the target samples in ascending order to obtain a sample distance queue; Obtain the sample distance at a preset position in the sample distance queue as the confidence distance of the defect category.
9. The method according to any one of claims 1 to 3, characterized in that The step of inputting the target image into a defect classification model for feature extraction to obtain the feature vector of the target to be classified further includes: Obtain the text information of the target to be classified; Encode the text information to obtain encoded information; Input the encoded information into the defect classification model for feature extraction to obtain a text vector; Input the stacked feature vector and the text vector into the defect classification model, and fuse and extract to obtain a new feature vector.
10. A defect classification device, characterized in that, The device includes: An acquisition module, configured to acquire a target image, a locally magnified image, and a standard image of a target to be classified; An identification module, configured to, when determining that the locally magnified image is not a sub-image of the standard image, input the target image into a defect classification model for feature extraction to obtain the feature vector of the target to be classified, and predict the predicted defect category according to the feature vector; determine the target distance of the predicted defect category according to the feature vector and the predicted defect category of the target to be classified; and determine the actual defect category of the target to be classified according to the predicted defect category and the target distance.
11. An electronic device, characterized in that, The electronic device includes: a memory, a processor; The memory is used to store a computer program; the processor is configured to implement the defect classification method according to any one of claims 1 to 9 based on the computer program stored in the memory.
12. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it is used to implement the defect classification method according to any one of claims 1 to 9.
13. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the defect classification method according to any one of claims 1 to 9.