Defect category detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a defect category detection method and device, electronic equipment and a storage medium. The defect category detection method comprises the following steps: acquiring a plurality of to-be-detected defect images of a target object; classifying the plurality of to-be-detected defect images for the defects of the target object by using the trained large model to obtain a classification result; according to a classification result, determining a defect category of a first defect image in the plurality of defect images to be detected; the plurality of second defect images are clustered, the defect category of each second defect image is determined based on a clustering result, and the plurality of second defect images are defect images except the first defect image in the plurality of to-be-detected defect images. And zero sample learning is supported, the generalization ability is strong, and unknown defect categories can be identified. And clustering the defect images which fail to classify the large model. Automatic defect classification of the to-be-detected defect image is realized, manual participation is not needed, and manpower is saved; and the accuracy of the obtained detection result is ensured.

Description

Technical Field

[0001] The present application relates to the field of industrial inspection, and more specifically to a defect category inspection method, a defect category inspection device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] In the industrial field, with the continuous development of production technology, people have put forward higher requirements for the production quality and efficiency of industrial products. With the development of image processing, more and more product inspections are based on images.

[0003] For example, in the semiconductor industry, for the wafer inspection process, the relevant technology usually uses automated optical inspection (AOI) equipment to perform appearance inspection on the wafer to determine the type of defects on the wafer. However, the different defects on the wafer are highly random, complex and changeable. In some existing defect detection methods, high-magnification color cameras are used to collect color images of the wafer for manual judgment by staff, which is inefficient. In other defect detection methods, a low-magnification camera is used to collect a grayscale image of the wafer and match it with the ideal template collected in advance to perform defect detection on the wafer. The accuracy of detection by this method is difficult to guarantee. Summary of the invention

[0004] The present invention has been made in view of the above-mentioned problems.

[0005] According to one aspect of the present invention, a method for detecting defect categories is provided, comprising:

[0006] Acquire multiple defect images to be tested of a target object; classify the multiple defect images to be tested according to the defects of the target object using a trained large model to obtain a classification result; determine the defect category of a first defect image among the multiple defect images to be tested based on the classification result; and cluster the multiple second defect images to determine the defect category of each second defect image based on the clustering result, wherein the multiple second defect images are defect images other than the first defect image among the multiple defect images to be tested.

[0007] The large model is used to classify the defect images to be tested. It supports zero-sample learning and has strong generalization ability, and can identify unknown defect categories. Then, the defect images that failed to be classified by the large model are clustered. In this way, not only the defect classification of the defect images to be tested is realized automatically, without manual participation, saving manpower; but also the accuracy of the obtained test results is guaranteed.

[0008] Exemplarily, obtaining multiple defect images to be tested of the target object includes: obtaining multiple grayscale images of the target object; determining the location information of the defects of the target object according to the multiple grayscale images; obtaining multiple color images of the corresponding target object according to the location information of the defects of the target object as multiple defect images to be tested. This realizes the detection of defects of the target object using grayscale images first, and then re-judging the defects of the target object using color images. This can reduce the amount of calculation of the algorithm and improve the accuracy of defect category detection.

[0009] Exemplarily, according to the classification result, the defect category of the first defect image among the multiple defect images to be tested is determined, including: for each defect image to be tested, judging whether the confidence in the classification result of the defect image to be tested is greater than a first threshold; and if it is greater than the first threshold, determining the defect image to be tested as the first defect image, and determining the defect category in the classification result of the defect image to be tested as the defect category of the first defect image. This method is simple and reliable, and can ensure the accuracy and efficiency of defect category detection.

[0010] Exemplarily, the defect category detection method further includes: in response to a first operation of a user on a classified first defect image, correcting the defect category of the first defect image; and / or in response to a second operation of a user on a clustered second defect image, determining the defect category of the second defect image. Thus, the accuracy of defect category detection is further improved, and missed or misjudgment is reduced.

[0011] Exemplarily, the defect category detection method further includes: retraining the large model according to the defect category of each first defect image and / or the defect category of each second defect image. This can improve the accuracy of the large model in detecting defect categories and improve the large model's ability to classify more categories of defects.

[0012] Exemplarily, clustering the plurality of second defect images includes: clustering the plurality of second defect images using a density-based spatial clustering algorithm. This method has good robustness and can more accurately determine the defect categories of the second defect images.

[0013] Exemplarily, clustering the plurality of second defect images includes: extracting features from the plurality of second defect images respectively to obtain respective first feature vectors; and clustering the first feature vectors to obtain clustering results. Further, the accuracy of defect category detection of the second defect images is improved, and the amount of calculation of the clustering algorithm can be reduced, thereby increasing the speed of defect category detection.

[0014] According to another aspect of the present invention, a defect category detection device is also provided, comprising an acquisition module, a classification module, a first determination module and a second determination module. The acquisition module is used to acquire multiple defect images to be tested of a target object; the classification module is used to classify the multiple defect images to be tested according to the defects of the target object using a trained large model to obtain a classification result; the first determination module is used to determine the defect category of a first defect image among the multiple defect images to be tested according to the classification result; and the second determination module is used to cluster the multiple second defect images to determine the defect category of each second defect image based on the clustering result, wherein the multiple second defect images are defect images other than the first defect image among the multiple defect images to be tested.

[0015] According to another aspect of the present invention, there is also provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned defect category detection method when the processor is executed.

[0016] According to yet another aspect of the present invention, a storage medium is provided, on which program instructions are stored. When the program instructions are run, they are used to execute the defect category detection method as described above.

[0017] According to another aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions are used to execute the defect category detection method as described above when running.

[0018] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention.

[0020] Figure 1 A schematic flow chart of a defect category detection method according to an embodiment of the present invention is shown;

[0021] Figure 2 A schematic diagram of a large model iterative training process according to an embodiment of the present invention is shown;

[0022] Figure 3A schematic diagram showing a defect category of a second defect image according to an embodiment of the present invention is shown;

[0023] Figure 4 A schematic diagram showing a defect category detection method according to another embodiment of the present invention;

[0024] Figure 5 A schematic block diagram of a defect category detection device according to an embodiment of the present invention is shown;

[0025] Figure 6 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more obvious, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention.

[0027] With the continuous development of science and technology, people can manufacture more and more industrial products. Products can be inspected based on images, product defects can be identified, and defect categories can be determined. Based on the defect category, the product production process can be optimized to improve production quality and efficiency. However, in related technologies, some methods use manual recognition methods, which are inefficient; other methods use ideal templates collected in advance for matching, which can only identify known defects, and the accuracy is difficult to guarantee.

[0028] In order to at least solve the above technical problems, the present invention provides a defect category detection method, which uses a trained large model to classify the defect images to be tested, and clusters the defect images corresponding to unknown defects in the classification results to achieve defect type detection. Figure 1 FIG. 1 is a schematic flow chart of a defect category detection method according to an embodiment of the present invention. Figure 1 As shown, the method includes step S110, step S120, step S130 and step S140.

[0029] In step S110, a plurality of defect images to be detected of the target object are acquired.

[0030] The target object can be various industrial products, such as liquid crystal panels, wafers, automobile body panels, etc. Taking wafers as an example, an imaging device such as an optical microscope or a scanning electron microscope can be used to take pictures of the wafers to obtain an image of each wafer. It can be understood that the way to obtain images of different industrial products may be different.

[0031] The image of the target object can be any suitable image. Exemplarily, the image of the target object can be a color image or a grayscale image. The image of the target object can be an image of any suitable size and resolution. The image can be an original image directly captured by an imaging device, or it can be an image after a preprocessing operation is performed on the original image. The preprocessing operation can include all operations to improve the visual effect of the image, improve its clarity, or highlight certain features in the image. Exemplarily and not restrictively, the preprocessing operation can include operations such as digitization, geometric transformation, normalization, filtering, etc. of the original image. The image can also be a synthesized image without affecting subsequent image processing.

[0032] For the acquired image of the target object, existing or future developed technologies such as template matching-based methods, threshold-based methods, edge detection-based methods, and machine learning-based methods can be used to determine whether the target object in the image has defects, and the image of the target object with defects is determined as the defect image to be tested. Defects refer to abnormal parts in the target object, which can be damaged, dirty, or have foreign matter. In this way, multiple defect images to be tested can be obtained.

[0033] In step S120, the trained large model is used to classify a plurality of defect images to be tested according to the defects of the target object to obtain a classification result.

[0034] A large model is a machine learning model with a large number of parameters, usually in the millions or even billions. For example, models such as groundingDINO and CLIP can be used. These large models can be multimodal models. In other words, large models can be trained based on text and images. Before performing defect category detection, large models can be pre-trained to capture a wide range of useful features by learning a large amount of general data, thereby improving the generalization ability of large models.

[0035] For the detection of defect categories, the large model can be pre-trained using the first sample defect image. Then, the pre-trained large model can be further trained using the second sample defect image with fewer labeled defect categories to achieve fine-tuning of the large model. A labeling standard can be formulated based on the existing sample defect images with determined defect categories. The labeling standard can be a defect classification rule formulated based on the cause of the defect, the morphological characteristics of the defect, the color of the defect, and other characteristics. The sample defect image is labeled according to the labeling standard, and the text attributes are defined. The text attribute can be a description text of the defect written according to the labeling standard, such as color, morphology, surface defect or non-surface defect. The labeled sample defect image includes an image label and a corresponding text attribute label. After the pre-trained large model is trained based on the first sample defect image that has been labeled, the large model can be tested and fine-tuned based on the test results. The training is iterated in this way until the result meets the requirements for the detection of the defect category, and the large model can be exported and deployed to perform the detection of the defect category.

[0036] Figure 2 FIG. 2 is a schematic diagram showing a large model iterative training process according to an embodiment of the present invention. Figure 2 As shown in the figure, after the sample defect images are labeled, the large model is trained, the trained large model is tested, and the large model is adjusted according to the test results to achieve model iterative optimization. When the large model meets the detection requirements of the defect category, the large model can be exported and then deployed to perform defect category detection.

[0037] The multiple defect images to be tested of the target object obtained in the above step S110 are input into the trained large model. The trained large model can classify the multiple defect images to be tested according to the defects of the target object in the defect images to be tested. The classification result corresponding to each defect image of the large model can be obtained. The defect categories of the target objects in different categories of defect images to be tested are different. The classification result may include confidence information that each defect image to be tested belongs to different categories. In a specific example, for the defect image to be tested of a certain wafer, the classification result may be that the confidence that the defect category of the wafer is a foreign body defect is 82%; for the defect image to be tested of another wafer, the classification result may be that the confidence that the defect category of the wafer is a scratch defect is 42%.

[0038] Optionally, the position information of the defect of the target object in the defect image to be tested can be input into the large model together with the defect image to be tested. The position information can indicate the position of the defect of the target object in the defect image, thereby enabling the large model to receive more information about the detection of defect categories of the defect image to be tested, thereby improving the detection accuracy of defect categories of the large model.

[0039] In step S130, the defect category of the first defect image among the plurality of defect images to be tested is determined according to the classification result.

[0040] The large model has learning ability. Therefore, the trained large model has a certain defect category detection ability, that is, it has the ability to understand defect images. In particular, for the defect categories that already exist in the sample defect image, its detection ability is stronger. Therefore, according to the classification results output by the large model, the defect categories of some defect images among the multiple defect images to be tested can be determined. For the convenience of description, this part of defect images can be referred to as the first defect image.

[0041] In the above specific example, for the defect image to be tested whose confidence level of defect category detection is 82%, it can be determined that its defect category is a foreign body defect.

[0042] In step S140, clustering is performed on the plurality of second defect images to determine the defect category of each second defect image based on the clustering result. The plurality of second defect images are defect images other than the first defect image among the plurality of defect images to be tested.

[0043] For defect categories that appear less frequently or not at all in the sample defect images, the large model also has the ability to generalize, and it can also handle these defect categories. In other words, for new defect categories of target objects that are constantly emerging, the large model also has a certain detection capability. This is because the large model can transfer the defect category knowledge learned from the sample defect images to other defect images. Therefore, according to an embodiment of the present invention, the large model is used to detect various known and unknown defect categories.

[0044] The second defect image is a defect image other than the first defect image among the multiple defect images to be tested. In other words, for the second defect image, the defect category of the target object of the second defect image in the classification result output by the large model may be inaccurate. It can be that the large model does not give an accurate defect category for the second defect image, or it can be said that it gives an open defect category.

[0045] According to an embodiment of the present application, all second defect images can be clustered. The clustering algorithm does not require images to have labels, and can directly cluster images, avoiding manual intervention and saving labor costs. The second defect images can be clustered using existing or future developed clustering methods such as K-means clustering, hierarchical clustering, and spatial clustering. The category of each second defect image can be determined based on the clustering results. Thus, the defect category of each defect image to be tested in multiple defect images to be tested can be determined.

[0046] In the above technical solution, the defect images to be tested are classified by a large model, and the defect categories of some of the defect images are determined according to the classification results. The other defect images are clustered to determine their defect categories according to the clustering results. This technical solution uses a large model to classify the defect images to be tested, which supports zero-sample learning and has strong generalization capabilities, and can identify unknown defect categories. Furthermore, the defect images that fail to be classified by the large model are clustered. As a result, not only is the automatic defect classification of the defect images to be tested realized without manual participation, saving manpower; but also the accuracy of the obtained detection results is guaranteed.

[0047] Exemplarily, step S110 acquires a plurality of defect images to be tested of the target object, including: step S111 , step S112 , step S113 , and step S114 .

[0048] In step S111, a plurality of grayscale images of the target object are acquired.

[0049] In some embodiments, a plurality of target objects may be photographed separately by a grayscale camera to obtain a plurality of grayscale images. The resolution of the grayscale camera may be relatively low, so that the accuracy of the grayscale images photographed by the camera may be relatively low. In this way, the cost of the system may be reduced.

[0050] In step S112, location information of defects of the target object is determined based on the multiple grayscale images.

[0051] Optionally, in step S111, grayscale images of all target objects may be obtained. In this step S112, defect detection of target objects may be performed on all grayscale images. The location information of defects of target objects may be determined based on grayscale images by using existing or future developed methods such as template matching, threshold segmentation, edge detection-based methods, and machine learning-based methods. Specifically, the location information of defects may include information indicating the location of defects in the target object and the size and shape of the defects.

[0052] It can be understood that if there are defects in some target objects, the position information of the defects in the grayscale images of these target objects can be determined through the above-mentioned defect detection steps. Some other target objects may not have defects, and the position information of the defects will not be determined from the grayscale images of these target objects. Therefore, through defect detection, target objects with defects can be screened out from all target objects.

[0053] In step S113, a plurality of color images of the corresponding target object are acquired according to the position information of the defects of the target object, as a plurality of defect images to be tested.

[0054] The color image can be taken at the same viewing angle and distance as the grayscale image of the target object. The color image can have the same size as the grayscale image. The resolution of the color camera can be relatively high, so the accuracy of the color image taken by it can be relatively high. In this way, the accuracy of the defect category detection of the target object can be guaranteed.

[0055] In this step S113, the color image obtained as the defect image to be tested corresponds to the position information of the defect of the target object. In other words, the color image includes the image information of the area where the defect of the target object is located. If there is no defect in the target object, the color image of the target object may not be included in the defect image to be tested.

[0056] Exemplarily, a color camera may be used to photograph the target object in which the position information of the defect is determined in step S112 to obtain a color image and the obtained color image is used as the defect image to be tested. Alternatively, a color camera may be used to photograph all target objects to obtain color images of all target objects. Then, according to the position information of the defect of the target object determined in step S112, the color image of the corresponding target object is screened out from the color images of all target objects to serve as the defect image to be tested.

[0057] In some implementation examples, the color image may be cropped according to the position information of the defect corresponding to the color image, the image area at the defect position is retained, and the image area at the non-defect position is cropped. In other embodiments, a mask may be set in the color image according to the position information of the defect to mark the defect position in the color image. Alternatively, the color image may be framed according to the position information of the defect to frame the position of the defect in the color image. The color image subjected to the above operation may be used as a defect image to be tested to detect the defect category of the target object.

[0058] The above technical solution determines the position information of the defect based on the grayscale image, and obtains a color image based on the corresponding position information of the defect as the defect image to be tested. The grayscale image can reduce the amount of algorithm calculation, and the position information of the defect can be roughly determined based on the grayscale image. Then, based on the position information, a color image is obtained as the defect image to be tested, and the defect category of the target object is subsequently detected on the defect image to be tested. In this way, the defects of the target object are first detected using the grayscale image, and then the defects of the target object are re-judged using the color image. This can not only reduce the amount of algorithm calculation, but also improve the accuracy of defect category detection.

[0059] Exemplarily, step S130 determines the defect category of the first defect image among the multiple defect images to be tested according to the classification result output by the large model, including: step S131 and step S132.

[0060] In step S131, for each defect image to be tested, it is determined whether the confidence level in the classification result of the defect image to be tested is greater than a first threshold.

[0061] Confidence refers to the credibility or probability estimate of the classification result of the large model for the defect image to be tested. In some embodiments, after the large model classifies the defect image to be tested, it can output the confidence that the target object in the defect image to be tested belongs to at least one defect category. The confidence can represent the probability that the defect in the defect image to be tested is of the defect category. The confidence can be represented by a numerical value between 0 and 1. For each defect image to be tested, if the confidence in the classification result of the defect image to be tested is greater than the first threshold, it can be considered that the defect type corresponding to the confidence is accurate. Further, the defect category of the target object in the defect image to be tested can be a known defect category. For example, still taking the confidence represented by a numerical value between 0 and 1 as an example, the first threshold can be 0.7. If the confidence in the classification result of the defect image to be tested is less than or equal to the first threshold, it can be considered that the classification result is unreliable, and then it can be considered that the defect type corresponding to the confidence in the classification result is an unknown defect type. In other words, the unknown defect category can be an open category, that is, an undetermined defect category.

[0062] In step S132, if the defect image to be tested is greater than the first threshold, the defect image to be tested is determined as a first defect image, and the defect category in the classification result of the defect image to be tested is determined as the defect category of the first defect image.

[0063] As described above, if the confidence in the classification result of the defect image to be tested is greater than the first threshold, it means that the defect category in the classification result is likely to be relatively accurate, for example, a known defect category. According to the classification result, the defect image to be tested can be determined as a first defect image, and the defect category in the classification result of the defect image to be tested can be determined as the defect category of the first defect image. In other words, the defect category of the first defect image is a relatively accurate classification result and can be a known defect type.

[0064] The above technical solution determines the defect category of the first defect image based on the classification result of the defect image to be tested whose confidence is greater than the first threshold. This method is simple and reliable and can ensure the accuracy and efficiency of defect category detection.

[0065] Exemplarily, step S140 clusters the plurality of second defect images, including: step S141 clusters the plurality of second defect images using a density-based spatial clustering algorithm.

[0066] The second defect image may correspond to an unknown defect category, or an open defect category. In other words, based on step S130, it is impossible to determine how many defect categories the multiple second defect images correspond to in total. The density-based spatial clustering algorithm does not require the pre-specification of the number of clusters, where each cluster corresponds to a defect category, so it is suitable for processing second defect images with unknown defect categories. The density-based spatial clustering algorithm can dynamically determine the number of clusters based on the density of data points corresponding to the second defect image. It can be an existing or future-developed density-based spatial clustering algorithm such as the DBSCAN algorithm and the OPTICS algorithm.

[0067] The above technical solution uses a density-based spatial clustering algorithm to cluster the second defect image. This method has good robustness and can more accurately determine the defect category of the second defect image.

[0068] Exemplarily, step S140 clusters the plurality of second defect images, including step S142 and step S143.

[0069] In step S142, feature extraction is performed on the plurality of second defect images respectively to obtain respective first feature vectors.

[0070] In some embodiments, a DINO backbone network, a Swin-Transformer model, etc. may be used as a feature extractor. Multiple second defect images may be input into the feature extractor to extract the first feature vector of each second defect image. The first feature vector may reflect the shape, texture, color, and other features of the defect in the second defect image. Alternatively, before the second defect image is input into the feature extractor, the second defect image may be preprocessed, for example, each second defect image may be size-standardized, noise removed, etc., to ensure the consistency of the second defect image.

[0071] In step S143, clustering is performed on the first feature vector to obtain a clustering result.

[0072] All first eigenvectors may be input into a clustering algorithm to obtain a clustering result. The clustering algorithm may be an existing or future developed clustering method such as K-means clustering, hierarchical clustering, spatial clustering, etc. All first eigenvectors are clustered. Alternatively, before all first eigenvectors are input into the clustering algorithm, the first eigenvectors may be adjusted, for example, by using dimensionality reduction, pooling, etc. to adjust the first eigenvectors to a fixed length before clustering.

[0073] The above technical solution extracts the first feature vector of the second defect image and clusters the first feature vector. The first feature vector can accurately represent the characteristics of the defects in the second defect image, improve the accuracy of clustering, and further improve the accuracy of defect category detection of the second defect image, and can reduce the calculation amount of the clustering algorithm and improve the speed of defect category detection.

[0074] Exemplarily, after determining the defect category of the first defect image among the plurality of defect images to be detected according to the classification result in step S130, the defect category detection method further includes step S135.

[0075] In step S135 , in response to a first operation of the user on the classified first defect image, the defect category of the first defect image is corrected.

[0076] In some embodiments, for the classified first defect images, the defect categories of some of the first defect images may be incorrectly detected. In other words, the defect categories of the first defect images determined according to the classification results of the large model may not be correct. The user may perform a first operation on the first defect images whose defect categories are determined to be incorrect, so as to correct the defect categories of the first defect images.

[0077] For example, for each defect category, first defect images belonging to the same defect category may be displayed together, so that the user can find first defect images with incorrect defect category detection. The user can use an input device such as a mouse or keyboard to adjust the defect category of the first defect image with incorrect defect category, that is, perform a first operation to correct the defect category of the first defect image targeted by the first operation.

[0078] It can be understood that the above step S135 only needs to be performed after step S130. After the defect category of the first defect image is determined in step S130, it can be performed at any time in response to the first operation of the user.

[0079] The above technical solution allows the user to correct the defect category of the first defect image obtained through automatic detection with an inaccurate defect category, thereby further improving the accuracy of defect category detection and reducing missed or misjudgment.

[0080] Exemplarily, after clustering the plurality of second defect images to obtain a clustering result in step S140 , the defect category detection method further includes step S145 .

[0081] In step S145 , in response to a second operation of the user on the clustered second defect images, a defect category of each second defect image is determined.

[0082] Clustering can determine the second defect images with the same features as the same category, but cannot describe their specific defect categories. In some embodiments, each second defect image that has been clustered can correspond to a defect category label. For example, class1, class2... etc. It can be understood that these defect category labels cannot describe the defect category well. The user can name the label of the defect category according to the features of the second defect image corresponding to the same defect category label. For example, the features of the second defect images corresponding to the defect category label class1 are all dirt, and the defect category of these second defect images can be determined as dirt, that is, the defect category corresponding to the defect category label class1 is dirt. Figure 3 FIG. 2 shows a schematic diagram of determining the defect category of the second defect image according to an embodiment of the present invention. Figure 3 As shown, the second defect image with the same defect category label is displayed in the same row. For simplicity, Figure 3 Only 7 second defect images corresponding to the three defect category labels class1, class2 and class3 are shown. The defect category of the second defect image corresponding to the defect category label class1 is determined to be particle damage; the defect category of the second defect image corresponding to the defect category label class2 is determined to be tin residue; the defect category of the second defect image corresponding to the defect category label class3 is determined to be collision scratch.

[0083] It can be understood that the above step S145 can be executed at any time after step S140 in response to the second operation of the user.

[0084] In the above technical solution, the user can manually determine the defect category of the clustered second defect image, thereby further improving the accuracy of defect category detection and reducing missed or misjudgment.

[0085] Exemplarily, the defect category detection method further includes step S150, retraining the large model according to the defect category of each first defect image and / or the defect category of each second defect image.

[0086] In some embodiments, the first defect image and the corresponding defect category determined according to the embodiment of the present invention may be input into the large model to train the large model again. Specifically, the first defect image and its defect category corrected after executing the above step S135 may be input into the large model to train the large model. In other words, the retraining of the large model may be based on the first defect image and the corresponding corrected defect category. Thus, the defect classification capability of the large model for the first defect image with inaccurate defect category detection may be improved.

[0087] In other embodiments, the second defect image and the corresponding defect category after the above step S145 is executed can be input into the large model to retrain the large model. In other words, the retraining of the large model is based on the second defect image and the corresponding defect category manually determined by the user. The defect images of these defect categories may not exist or exist in a small number in the sample defect images used in the previous training and learning of the large model. In this way, the ability of the large model to identify these defect categories can be improved.

[0088] After both step S135 and step S145 are executed, the large model may be trained again based on the first defect image and the corresponding defect category and the second defect image and the corresponding defect category.

[0089] The above technical solution retrains the large model according to the defect category of each first defect image and / or the defect category of each second defect image, which can improve the accuracy of the large model in detecting defect categories and improve the large model's ability to classify more categories of defects.

[0090] For example, Figure 4 FIG. 2 is a schematic diagram showing a method for detecting defect categories according to another embodiment of the present invention. Figure 4 As shown, firstly, a grayscale image of the target object is captured by a grayscale camera. Then, the defect location in the grayscale image is performed to obtain the location information of the defect. According to the location information of the defect, a color image of the defect to be detected is captured by a color camera. The location information and the defect image to be detected can be input into a defect category detection module to determine the defect category of the target object in the defect image to be detected. It can be understood that the detection module corresponds to Figure 4 The content in the dotted box in the figure. The position information and the defect image to be tested are input into the large model for classification. The classification result is threshold-controlled according to the confidence of each defect category, and the defect image to be tested is divided into two parts, namely the first defect image whose classification confidence is greater than the first threshold and the second defect image whose classification confidence is less than or equal to the first threshold. Optionally, the defect category of the classification result corresponding to the first defect image is manually reviewed. The second defect image is clustered. Optionally, the defect category of the clustered second defect image is manually reviewed. Determine whether the defect category of the defect image to be tested is correct through manual review. If correct, determine the defect category of the defect image to be tested, output the result, and the defect image to be tested and the corresponding defect category can be used to train the large model with a small sample. If incorrect, the defect category of the defect image to be tested is corrected and marked, and the defect image to be tested and the corresponding defect category after correction and marking can output the result, and can be used to train the clustering algorithm and the large model with a small sample.

[0091] Illustratively, according to yet another aspect of the present invention, a device for detecting defect categories is also provided. Figure 5 FIG. 5 shows a schematic block diagram of a defect category detection device 500 according to an embodiment of the present invention. Figure 5 As shown, the defect category detection device 500 includes an acquisition module 510 , a classification module 520 , a first determination module 530 and a second determination module 540 .

[0092] The acquisition module 510 is used to acquire a plurality of defect images to be tested of a target object. The classification module 520 is used to classify the plurality of defect images to be tested according to the defects of the target object using a trained large model to obtain a classification result. The first determination module 530 is used to determine the defect category of a first defect image among the plurality of defect images to be tested according to the classification result. The second determination module 540 is used to cluster a plurality of second defect images to determine the defect category of each second defect image based on the clustering result, wherein the plurality of second defect images are defect images other than the first defect image among the plurality of defect images to be tested.

[0093] Exemplarily, the acquisition module 510 includes a first acquisition submodule, a first determination submodule, and a second acquisition submodule. The first acquisition submodule is used to acquire multiple grayscale images of the target object. The first determination submodule is used to determine the location information of the defects of the target object based on the multiple grayscale images. The second acquisition submodule is used to acquire multiple color images of the target object corresponding to the location information of the defects of the target object as the multiple defect images to be tested.

[0094] The first determination module 530 includes a judgment submodule and a second determination submodule. The judgment submodule is used to judge, for each defect image to be tested, whether the confidence in the classification result of the defect image to be tested is greater than a first threshold. The second determination submodule is used to determine the defect image to be tested as a first defect image if the confidence in the classification result of the defect image to be tested is greater than the first threshold, and determine the defect category in the classification result of the defect image to be tested as the defect category of the first defect image.

[0095] Exemplarily, the defect category detection device 500 further includes a first correction module. The first correction module is used to correct the defect category of the first defect image in response to a first operation of a user on the classified first defect image.

[0096] Exemplarily, the defect category detection device 500 further includes a second correction module. The second correction module is used to determine the defect category of the second defect image in response to a second operation of the user on the clustered second defect image.

[0097] Exemplarily, the defect category detection device 500 further includes a training module. The training module is used to retrain the large model according to the defect category of each first defect image and / or the defect category of each second defect image.

[0098] Exemplarily, the second determination module 540 includes a first clustering submodule. The first clustering submodule is used to cluster the plurality of second defect images using a density-based spatial clustering algorithm.

[0099] Exemplarily, the second determination module 540 includes a third acquisition submodule and a second clustering submodule. The third acquisition submodule is used to perform feature extraction on the plurality of second defect images respectively to obtain respective first feature vectors. The second clustering submodule is used to cluster the first feature vectors to obtain the clustering result.

[0100] Illustratively, according to yet another aspect of the present invention, an electronic device is also provided. Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of the present invention is shown. The electronic device 600 includes a processor 610 and a memory 620. The memory 620 stores computer program instructions, which are used by the processor 610 to execute the above-mentioned defect category detection method when executed.

[0101] Exemplarily, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned defect category detection method when running. The storage medium may include, for example, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media.

[0102] Illustratively, according to yet another aspect of the present invention, there is further provided a computer program product, comprising computer program instructions, wherein the computer program instructions are used to execute the defect category detection method as described above when being run.

[0103] A person skilled in the art can understand the specific implementation schemes and beneficial effects of the above-mentioned defect category detection devices, electronic devices, storage media and computer program products by reading the above-mentioned descriptions of the defect category detection methods. For the sake of brevity, they will not be elaborated here.

[0104] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0105] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0107] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0108] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0109] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0110] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0111] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the defect category detection device according to the embodiment of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0112] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0113] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A defect category detection method, characterized in that: include: Acquire multiple defect images to be tested of the target object; Using the trained large model, the plurality of defect images to be tested are classified according to the defects of the target object to obtain a classification result; Determining a defect category of a first defect image among the plurality of defect images to be tested according to the classification result; as well as Clustering is performed on a plurality of second defect images to determine a defect category of each second defect image based on a clustering result, wherein the plurality of second defect images are defect images other than the first defect image among the plurality of defect images to be tested.

2. The defect type detection method according to claim 1, characterized in that: The step of acquiring a plurality of defect images to be detected of the target object comprises: Acquire multiple grayscale images of the target object; Determining location information of defects of the target object according to the multiple grayscale images; According to the position information of the defect of the target object, a plurality of corresponding color images of the target object are acquired as the plurality of defect images to be tested.

3. The defect type detection method according to claim 1, characterized in that: The step of determining the defect category of the first defect image among the plurality of defect images to be tested according to the classification result comprises: For each defect image to be tested, determining whether the confidence in the classification result of the defect image to be tested is greater than a first threshold; and For a case where the defect image to be tested is greater than the first threshold, the defect image to be tested is determined as a first defect image, and the defect category in the classification result of the defect image to be tested is determined as the defect category of the first defect image.

4. The defect type detection method according to any one of claims 1 to 3, characterized in that: The defect category detection method further includes: In response to a first operation of a user on the classified first defect image, correcting the defect category of the first defect image; and / or In response to a second operation of the user on the clustered second defect image, a defect category of the second defect image is determined.

5. The defect type detection method according to claim 4, characterized in that: The defect category detection method further includes: The large model is retrained according to the defect category of each first defect image and / or the defect category of each second defect image.

6. The defect type detection method according to any one of claims 1 to 3, characterized in that: The clustering of the plurality of second defect images comprises: The plurality of second defect images are clustered using a density-based spatial clustering algorithm.

7. The defect type detection method according to any one of claims 1 to 3, characterized in that: The clustering of the plurality of second defect images comprises: Performing feature extraction on the plurality of second defect images respectively to obtain respective first feature vectors; Clustering is performed on the first feature vector to obtain the clustering result.

8. A defect type detection device, characterized in that: include: An acquisition module, used for acquiring multiple defect images to be tested of a target object; A classification module, used for classifying the plurality of defect images to be tested according to the defects of the target object by using the trained large model to obtain a classification result; A first determination module, configured to determine a defect category of a first defect image among the plurality of defect images to be tested according to the classification result; as well as The second determination module is used to cluster the plurality of second defect images to determine the defect category of each second defect image based on the clustering result, wherein the plurality of second defect images are defect images other than the first defect image among the plurality of defect images to be tested.

9. An electronic device, comprising: A processor and a memory, characterized in that The memory stores computer program instructions, which are used by the processor to execute the defect category detection method according to any one of claims 1 to 7 when the processor executes the computer program instructions.

10. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the defect category detection method according to any one of claims 1 to 7 when running.

11. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the defect category detection method according to any one of claims 1 to 7 when being executed.

Images