Method, apparatus, and computer-readable storage medium for detecting defects

Abstract

According to example embodiments of the present disclosure, a method, apparatus, and computer-readable storage medium for detecting defects are provided. After receiving a candidate image set to be used for training a defect detection model, a set of historical images is selected from a historical image set that has been used to train the defect detection model. The candidate image set and the images in the historical image set include defects related to objects in the images. A set of representative images is determined from the candidate image set. The representative images can reflect image features of multiple different images in the candidate image set. Then, based on at least the selected set of historical images and the set of representative images, a defect detection model is trained. The trained defect detection model is used to detect defects related to objects in the images to be detected. Embodiments of the present disclosure can accelerate incremental training of defect detection models.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, apparatuses, and computer-readable storage media for detecting defects. Background Art

[0002] Automated defect detection (also known as defect inspection) is widely used in various fields, such as semiconductor manufacturing, flat panel display manufacturing, and other manufacturing industries. In this type of defect detection, potential defects are typically detected from an image of the object being inspected. For example, potential defects can be detected from an image of a semiconductor chip.

[0003] To automate the defect detection process, machine learning methods (e.g., deep learning methods) are often used to detect defects, classify detected defects, and / or mark defect locations, etc. Machine learning methods significantly reduce the workload of manual inspection and avoid human errors caused by fatigue. Summary of the Invention

[0004] According to an example embodiment of the present disclosure, a scheme for detecting defects is provided.

[0005] In a first aspect of the present disclosure, a method for training a defect detection model is provided. The method includes: in response to receiving a candidate image set to be used for training the defect detection model, selecting a set of historical images from a historical image set that has been used to train the defect detection model, wherein the candidate image set and images in the historical image set include defects related to an object in the image; determining a set of representative images from the candidate image set, wherein each image in the set of representative images reflects image features of multiple different images in the candidate image set; and training the defect detection model based on at least the set of historical images and the set of representative images to detect defects related to the object in the image to be detected.

[0006] In a second aspect of the present disclosure, a method for detecting defects is provided, comprising: obtaining an image to be detected, wherein an object in the image to be detected has a defect; and determining information related to the defect using a defect detection model trained according to the method of the first aspect.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the device to perform actions. The actions include: in response to receiving a candidate image set to be used for training a defect detection model, selecting a set of historical images from a historical image set that has been used to train the defect detection model, the candidate image set and images in the historical image set including defects related to an object in the image; determining a set of representative images from the candidate image set, each image in the set of representative images reflecting image features of multiple different images in the candidate image set; and training the defect detection model based on at least the set of historical images and the set of representative images to detect defects related to the object in the image to be detected.

[0008] In some embodiments, determining a group of representative images from a candidate image set includes: dividing the candidate image set into multiple groups of candidate images based on image features of each candidate image in the candidate image set; and selecting at least one candidate image from each of the multiple groups to combine into a group of representative images, wherein the at least one candidate image selected in each group reflects the image features of the candidate images in the corresponding group.

[0009] In some embodiments, dividing the candidate image set into multiple groups includes: performing a data dimensionality reduction operation on the image features of each candidate image in the candidate image set to obtain reduced-dimensionality image features; clustering the candidate images in the candidate image set based on the reduced-dimensionality image features; and determining multiple clusters corresponding to the multiple groups based on the clustering results.

[0010] In some embodiments, selecting at least one candidate image from each of a plurality of groups to combine into a set of representative images includes: for a first group in the plurality of groups, determining a center and an edge of a first cluster corresponding to the first group; and selecting at least one of the following to combine into a set of representative images: a candidate image whose distance from the center in the first cluster is within a first threshold range, or a candidate image whose distance from the edge in the first cluster is within a second threshold range.

[0011] In some embodiments, the actions further include: adding a set of representative images to the historical image set for subsequent training of the defect detection model.

[0012] In a fourth aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein. When executed by the processor, the instructions cause the device to perform actions. The actions include: obtaining an image to be inspected, wherein an object in the image to be inspected has a defect; and determining information related to the defect using a defect detection model trained according to the method of the first aspect.

[0013] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0014] In a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the second aspect of the present disclosure is implemented.

[0015] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0017] Figure 1 A schematic diagram showing a conventional scheme for training a defect detection model;

[0018] Figure 2 A schematic diagram showing another conventional approach for training a defect detection model;

[0019] Figure 3 A schematic block diagram of a model training system according to some embodiments of the present disclosure is shown;

[0020] Figure 4 A schematic diagram showing clustering results of a candidate image set according to some embodiments of the present disclosure is shown;

[0021] Figure 5 shows a schematic block diagram of a defect detection system according to some embodiments of the present disclosure;

[0022] Figure 6 A flowchart illustrating an example method of training a defect detection model according to an embodiment of the present disclosure is shown;

[0023] Figure 7 A flowchart illustrating an example method of detecting defects according to an embodiment of the present disclosure; and

[0024] Figure 8 A block diagram is shown of a computing device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0027] In the embodiments of the present disclosure, the term "model" is capable of processing inputs and providing corresponding outputs. Taking a neural network model as an example, it generally includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. The models used in deep learning applications (also referred to as "deep learning models") generally include many hidden layers, thereby extending the depth of the network. The layers of the neural network model are connected in sequence so that the output of the previous layer is used as the input of the next layer, wherein the input layer receives the input of the neural network model, and the output of the output layer serves as the final output of the neural network model. Each layer of the neural network model includes one or more nodes (also referred to as processing nodes or neurons), each node processing the input from the previous layer. In this document, the terms "neural network", "model", "network" and "neural network model" are used interchangeably.

[0028] As briefly mentioned above, machine learning methods have been used for defect detection. Machine learning methods require training of machine learning models. Training a machine learning-based defect detection model requires a training dataset, typically consisting of a set of images and defect labels. The resulting model depends on: the training dataset (e.g., images and defect labels); the training process (e.g., batch size, number of iterations); and the model type (e.g., neural network architecture). When the training process and model type are the same, the resulting model is determined solely by the training dataset.

[0029] refer to Figure 1 , which shows a schematic diagram of a conventional scheme for training defect detection models. Figure 1In the example, model training device 120 uses training data sets 101, 102, and 103 to train defect detection models 111, 112, and 113, respectively. The trained defect detection models 111, 112, and 113 can only be applied to their corresponding data sets. In other words, different data types may require different models.

[0030] When new image types or new defect types that were not used for training appear, the defect detection model needs to be updated (i.e., retrained) to classify the new images and / or detect the new defect types. The new images or new defect types may be caused by drift or changes in the manufacturing process.

[0031] refer to Figure 2 , which shows a schematic diagram of another conventional scheme for training defect detection models. Figure 2 As shown, training datasets 201, 202, and 203 are obtained sequentially. To enable a new defect detection model to recognize old images, the previously used training datasets need to be included in the training of the new defect detection model. Model training device 220 uses the first-acquired training dataset 201 to obtain defect detection model 211. After obtaining new training dataset 202, model training device 220 uses training datasets 201 and 202 to obtain defect detection model 212. Similarly, after obtaining new training dataset 203, model training device 220 uses training datasets 201, 202, and 203 to obtain defect detection model 213.

[0032] The capabilities of machine learning-based defect detection models are highly correlated with the training dataset. A trained defect detection model cannot recognize unknown (or untrained) image types or defect types. Figure 2 In conventional approaches, when new training data becomes available, historical training datasets are required to train new defect detection models. Recording all historical datasets requires significant storage costs. Furthermore, training all historical datasets consumes significant computational time and resources.

[0033] According to an embodiment of the present disclosure, a defect detection scheme is proposed. In this scheme, after receiving a candidate image set to be used for training a defect detection model, a set of historical images is selected from a historical image set that has already been used to train the defect detection model. Images in the candidate image set and the historical image set include defects related to objects in the images. A set of representative images is determined from the candidate image set. The representative images reflect image features of multiple different images in the candidate image set. Then, a defect detection model is trained based on at least the selected set of historical images and the set of representative images. The trained defect detection model is used to detect defects related to objects in the images to be detected.

[0034] In embodiments of the present disclosure, a defect detection model is trained using a portion of historical data and newly acquired representative data, enabling the trained defect detection model to identify a variety of defects from multiple image types. In other words, the defect detection model trained in this manner can detect both old defect types from old image types and new defect types from new image types. This accelerates the training process of the defect detection model when new data, unknown or not previously used for training, becomes available. Therefore, embodiments of the present disclosure can accelerate incremental training of defect detection models.

[0035] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0036] Example model training system and operations

[0037] Figure 3 FIG. 3 is a schematic block diagram illustrating a model training system 300 according to some embodiments of the present disclosure. Figure 3 As shown, the model training system 300 (also referred to as "system 300") may include a candidate image selection device 330, a historical image selection device 340, a model training device 350, and a storage device 370, which operate in conjunction to train a defect detection model 360. In some embodiments, the above-mentioned multiple devices may be implemented in different physical devices. Alternatively, at least some of the above-mentioned multiple devices may be implemented in the same physical device. For example, the candidate image selection device 330, the historical image selection device 340, and the model training device 350 may be implemented in the same physical device, while the storage device 370 may be implemented in another physical device.

[0038] A candidate image set 301 to be used for training a defect detection model 360 is input to the system 300. The candidate image set 301 includes at least a plurality of candidate images 310-1, 310-2, 310-3, 310-4, ..., 310-N, which are also collectively or individually referred to as candidate images 310 or images 310. N is an integer greater than 1. The candidate images 310 include defects related to objects in the image. For example, in an embodiment where the defect detection model 360 is used to detect defects in a semiconductor chip, the candidate image 310 may be an image of at least a portion of the semiconductor chip, and the device or connection in the at least portion includes a defect. For another example, in an embodiment where the defect detection model 360 is used to detect defects in a flat panel display, the candidate image 310 may be an image of at least a portion of the flat panel display, and the component or connection in the at least portion includes a defect. For another example, in an embodiment where the defect detection model 360 is used to detect defects in a packaging inspection process, the candidate image 310 may be an image of at least a portion of the packaging inspection process, and the component or connection in the at least portion includes a defect. The candidate images 310 may include, but are not limited to, optical microscope images, scanning electron microscope images (SEM), and the like.

[0039] Compared to the historical image set 302 described below, the candidate image set 301 may be obtained from a new process or production line. The candidate image set 301 may also be obtained from the same process or production line at a later time than the historical image set 302. The candidate image set 301 may also be from a different batch of images than the historical image set 302. The above is merely an example, and embodiments of the present disclosure are not limited in this respect.

[0040] like Figure 3 As shown, in some embodiments, candidate image set 301 additionally includes a plurality of labels 311-1, 311-2, 311-3, 311-4, ..., 311-N corresponding to a plurality of candidate images 310-1, 310-2, 310-3, 310-4, ..., 310-N, respectively, which are also collectively or individually referred to as labels 311. Labels 311 indicate defects in corresponding images 310, including but not limited to the type of defect, the location of the defect, etc. If defect detection model 360 is based on supervised training, candidate image set 301 includes labels 311. If defect detection model 360 is based on unsupervised training, candidate image set 301 may not include labels 311.

[0041] The candidate image selection device 330 is used to determine a set of representative images from the candidate image set 301 as the representative image set 303. The selected representative images reflect the image characteristics of multiple different images in the candidate image set. In other words, the representative images can reflect the common characteristics of a class of images in the candidate image set.

[0042] exist Figure 3 In the example of , candidate images 310 - 1 , 310 - 3 , . . . , 310 -M are determined as representative images, where M is an integer less than or equal to N. In the case where the candidate image set 301 includes the label 311 , labels 311 - 1 , 311 - 3 , . . . , 311 -M corresponding to the candidate images 310 - 1 , 310 - 3 , . . . , 310 -M are also added to the representative image set 303 .

[0043] In some embodiments, to determine a representative image, the candidate image selection device 330 may divide the candidate image set 301 into multiple groups of candidate images based on the image features of each candidate image 310 in the candidate image set 301. In this case, the candidate images in the same group have similar or similar image features. The candidate image selection device 330 may then select at least one candidate image from each group as a representative image. The at least one candidate image selected in each group reflects the image features of the candidate images in the corresponding group. In other words, the at least one candidate image selected in each group reflects the common features of the candidate images in that group.

[0044] In some embodiments, the number of candidate images selected from each group is related to the number of candidate images in the corresponding group. Assuming that the second group includes more candidate images than the first group, candidate image selection device 330 can select more candidate images from the second group as representative images. In other words, more representative images are selected from the group with more candidate images.

[0045] The candidate image selection device 330 may utilize any suitable method to group the candidate images 310 in the candidate image set 301 based on image features. In some embodiments, the candidate image selection device 330 may cluster the candidate images 310. Figure 4 , which shows a schematic diagram of a clustering result 400 of a candidate image set 301 according to some embodiments of the present disclosure.

[0046] Figure 4 Clusters 401, 402, and 403 obtained by clustering the candidate image set 101 are schematically shown. Each cluster corresponds to a group of candidate images. Cluster 401 includes at least candidate images 310-1 and 310-2, that is, candidate images 310-1 and 310-2 belong to the same group. Cluster 402 includes at least candidate images 310-3, 310-4, 310-5, and 310-8, that is, these candidate images belong to the same group. Cluster 403 includes at least candidate images 310-6, 310-7, and 310-M, that is, candidate images 310-6, 310-7, and 310-M belong to the same group. It should be understood that Figure 4 The clustering result 400 shown in FIG. 4 is merely illustrative and is not intended to limit the scope of the present disclosure. The clustering result 400 may further include more or fewer clusters, and the clusters 401 , 402 , and 403 may further include more or fewer images 310 .

[0047] In some embodiments, in order to better group the candidate images 310, a data dimensionality reduction operation may be applied. Specifically, the candidate image selection device 330 may perform a data dimensionality reduction operation on the image features of each candidate image 310 to obtain image features that have been reduced in dimension. As an example, the candidate image selection device 330 may perform principal component analysis (PCA) on the image features of the candidate image 310 to achieve dimensionality reduction. As another example, the candidate image selection device 330 may perform nearest neighbor embedding, such as t-distributed random neighbor embedding (t-SNE), on the image features of the candidate image 310 to achieve dimensionality reduction. It should be understood that the principal component analysis and nearest neighbor embedding described above are merely exemplary and are not intended to limit the scope of the present disclosure. In the embodiments of the present disclosure, any suitable data dimensionality reduction method may be utilized.

[0048] After obtaining the reduced-dimensional image features, the candidate image selection device 330 may cluster the candidate images 310 in the candidate image set 301 based on the reduced-dimensional image features. Figure 4 The clustering results are shown.

[0049] In this embodiment, redundant features can be removed by performing a data dimensionality reduction operation on the image features of candidate images 310, thereby achieving better clustering results. In this way, candidate images with similar defect features are more easily grouped into the same group, which is conducive to accurately identifying representative images.

[0050] Continue to refer Figure 4 According to the clustering results, the candidate images 310 are divided into a plurality of clusters 401, 402, and 403. That is, the candidate images 310 are divided into a plurality of groups. In some embodiments, the candidate image selection device 330 may randomly select one or more candidate images from each cluster as representative images.

[0051] In some embodiments, the candidate image selection device 330 may select a representative image based on the location of the candidate image 310 in the corresponding cluster. For example, the candidate image selection device 330 may determine the center and edge of each cluster and then select a representative image based on the distance of the candidate image 310 from the center or edge.

[0052] As an example, candidate image 310 that is close to the center of the cluster can be selected as a representative image. For example, candidate image 310-3 is close to the center of cluster 402 and is thus selected as a representative image. Alternatively or additionally, candidate image 310 that is close to the edge of the cluster can be selected as a representative image. For example, candidate image 310-1 is close to the edge of cluster 401, and candidate image 310-M is close to the edge of cluster 403, so that candidate images 310-1 and 310-M are selected as representative images. It should be understood that, as used herein, "close to the center" means that the distance from the center is within a threshold range. Similarly, "close to the edge" means that the distance from the edge is within a threshold range.

[0053] Return Reference Figure 3 By using the candidate image selection device 303 , a representative image set 303 can be obtained, which includes a part of the candidate image set 301 .

[0054] The storage device 370 stores the historical image set 302 that has been used to train the defect detection model 360. The storage device 370 can be any type of non-volatile storage device, and the embodiments of the present disclosure are not limited thereto. In some embodiments, the storage device 370 can be implemented on a cloud platform.

[0055] Historical image set 302 includes at least a plurality of historical images 320-1, 320-2, 320-3, 320-4, ..., 320-P, which are collectively or individually referred to as historical images 320 or images 320. P is an integer greater than 1. Historical images 320 include defects associated with objects in the images. For example, in an embodiment where defect detection model 360 is used to detect defects in semiconductor chips, historical images 320 may be images of at least a portion of the semiconductor chip, and a device or connection in the at least a portion includes a defect.

[0056] Similar to candidate image set 301, in some embodiments, historical image set 302 may additionally include a plurality of tags 321-1, 321-2, 321-3, 321-4, ..., 321-P corresponding to a plurality of historical images 320-1, 320-2, 320-3, 320-4, ..., 320-P, respectively. These tags are collectively or individually referred to as tags 321. Tags 321 indicate defects in corresponding images 320, including but not limited to the type of defect, the location of the defect, etc. If defect detection model 360 is based on supervised training, historical image set 302 includes tags 321. If defect detection model 360 is based on unsupervised training, historical image set 302 may not include tags 321.

[0057] The historical image selection device 340 is used to select a group of historical images from the historical image set 302 as the historical image subset 304 for retraining the defect detection model 360. Figure 3 In the example of FIG, the historical image selection device 340 selects historical images 320-1, 320-3, ..., 320-Q, where Q is an integer less than or equal to P. In the case where the historical image set 302 includes the tag 321, the tags 321-1, 321-3, ..., 321-Q corresponding to the images 320-1, 320-3, ..., 320-Q are also added to the historical image subset 304.

[0058] In some embodiments, historical images 320 in historical image set 302 are representative images selected from a historical candidate image set. For example, historical images 320 are representative images selected from the historical candidate image set during previous training of defect detection model 360 using the operations described above with reference to candidate image selection device 330. In such embodiments, historical image selection device 340 may randomly select a group of historical images from the historical image set for retraining defect detection model 360.

[0059] Alternatively, in some embodiments, historical image selection device 340 may determine a group of historical images from historical image set 302 as representative images, as described above with reference to candidate image selection device 330. For example, historical image selection device 340 may group historical images 320 based on their image features and then select at least one representative image from each group. This embodiment is similar to that described with respect to candidate image set 301 and will not be further described.

[0060] The selected representative images and historical images are input to the model training device 350. The model training device 350 then trains the defect detection model 360 based on the selected representative images and historical images. In the training of the defect detection model 360, the representative images and historical images are continuously fed to the model training device 350 until the defect detection model 360 converges.

[0061] In embodiments of the present disclosure, defect detection model 360 may have any suitable network structure and may be based on supervised or unsupervised training. The scope of the present disclosure is not limited in this respect. In embodiments where defect detection model 360 is based on supervised training, labels 311-1, 311-3, ..., 311-M and labels 321-1, 321-3, ..., 321-Q are used together with the corresponding images to train defect detection model 360.

[0062] In this way, defect detection model 360 is retrained based on the new training data and the historical training data. The retrained defect detection model 360 is able to detect defects in images similar to the historical images as well as in images similar to the new images. In this way, rapid incremental training of defect detection model 360 is achieved.

[0063] like Figure 3 As shown, representative images in the representative image set can be added to the historical image set 302 for subsequent training of the defect detection model 360. For example, the representative images are added to the historical image set 302, and the resulting updated historical image set is stored in the storage device 370. After receiving a subsequent new candidate image set, images are selected from the updated historical image set for retraining the defect detection model 360 again.

[0064] Example model application system and operation

[0065] Figure 5 1 shows a schematic block diagram of a defect detection system 500 according to some embodiments of the present disclosure. The defect detection system 500 includes an image processing device 510. The image processing device 510 may include or be deployed with a trained defect detection model 360. It should be understood that the image processing device 510 may also include or be deployed with other models.

[0066] like Figure 5 As shown, image processing apparatus 510 may receive an image to be inspected 501. Image to be inspected 501 may include potential defects related to an object in the image. For example, image to be inspected 501 may be an image of a portion of a semiconductor chip, where some devices or connections in the portion may have defects.

[0067] The image processing device 510 can generate and provide a detection result 502 based on the image to be detected 501 according to the trained defect detection model 360. The detection result 502 may include information related to the defects in the image to be detected 501, such as the number of defects, the type of defects, the location of the defects, etc.

[0068] Example Method

[0069] Figure 6 FIG. 6 is a flow chart showing an example method 600 for training a defect detection model according to an embodiment of the present disclosure. For example, the method 600 may be performed by Figure 3 At least a portion of the system 300 shown (eg, candidate image selection device 330, historical image device 340, model training device 350) is executed. Figure 36. It should be understood that the method 600 may include additional blocks not shown and / or may omit some of the blocks shown. The scope of the present disclosure is not limited in this respect.

[0070] At block 610, in response to receiving the candidate image set 301 to be used for training the defect detection model 360, a set of historical images, such as historical images 320-1, 320-3, ..., 320-Q, is selected from the historical image set 302 that has been used to train the defect detection model 360. The images in the candidate image set 301 and the historical image set 302 include defects related to objects in the images.

[0071] In some embodiments, the historical images 320 in the historical image set 302 are representative images selected from the historical candidate image set. A group of historical images may be randomly selected from the historical image set 302.

[0072] At block 620 , a set of representative images, such as images 310 - 1 , 310 - 3 . . . , 310 -M, is determined from the candidate image set 301. Each image in the set of representative images reflects image characteristics of a plurality of different images in the candidate image set.

[0073] In some embodiments, candidate image set 301 is divided into a plurality of groups of candidate images based on image features of each candidate image 320 in candidate image set 301. At least one candidate image may be selected from each of the plurality of groups to form a set of representative images. The at least one candidate image selected in each group reflects the image features of the candidate images in the corresponding group.

[0074] In some embodiments, to divide the candidate image set into multiple groups, a data dimensionality reduction operation may be performed on the image features of each candidate image in the candidate image set 301 to obtain reduced-dimensionality image features. Based on the reduced-dimensionality image features, the candidate images in the candidate image set 301 may be clustered. Based on the clustering results, multiple clusters corresponding to the multiple groups may be determined, such as clusters 401, 402, 403, etc.

[0075] In some embodiments, for a first group among the plurality of groups, a center and an edge of a first cluster corresponding to the first group may be determined, and at least one of the following may be selected to be combined into a set of representative images: a candidate image whose distance from the center of the first cluster is within a first threshold range, or a candidate image whose distance from the edge of the first cluster is within a second threshold range.

[0076] In some embodiments, the number of the at least one candidate image selected from a second group of the plurality of groups is determined based on the number of candidate images in the second group.

[0077] At block 630 , the defect detection model 360 is trained based on at least a set of historical images and a set of representative images to detect defects associated with objects in the image to be detected.

[0078] Figure 7 FIG. 7 is a flow chart showing an example method 700 for detecting defects according to an embodiment of the present disclosure. For example, the method 700 may be performed by: Figure 5 The following is performed by the image processing device 510 shown in FIG. Figure 5 7. The method 700 is described in detail. It should be understood that the method 700 may include additional blocks not shown and / or may omit some of the blocks shown. The scope of the present disclosure is not limited in this respect.

[0079] At block 710 , an image to be inspected 501 is obtained. The object in the image to be inspected 501 has defects. For example, the image to be inspected 501 may be an image of a portion of a semiconductor chip, and some devices or connections in the portion may have defects.

[0080] At block 720, information related to the defects is determined using the defect detection model 360 trained according to the method 600. The information related to the defects is, for example, included in the detection result 502. The information related to the defects may include, for example, the number of defects, the type of defects, the location of the defects, etc.

[0081] Example device

[0082] Figure 8 8 shows a schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure. Figure 3 One or more devices in the system 300 shown or provided by Figure 5 The image processing apparatus 510 in the illustrated system 500 can be implemented by a device 800. As shown, the device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0083] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0084] The processing unit 801 performs the various methods and processes described above, such as any of the methods 600 and 700. For example, in some embodiments, any of the methods 600 and 700 may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the CPU 801, one or more steps of any of the methods 600 and 700 described above may be performed. Alternatively, in other embodiments, the CPU 801 may be configured to perform any of the methods 600 and 700 by any other suitable means (e.g., by means of firmware).

[0085] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0086] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0088] In addition, although adopting specific order to describe each operation, this should be understood as requiring such operation to be carried out in the specific order shown or in sequential order, or requiring that all illustrated operations should be carried out to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although comprising some specific implementation details in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation also can be implemented in a plurality of implementations individually or in the mode of any suitable subcombination.

[0089] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for training a defect detection model, comprising: In response to receiving a candidate image set to be used for training the defect detection model, selecting a set of historical images from a historical image set that has been used to train the defect detection model, the candidate image set and images in the historical image set including defects associated with objects in the images; determining a set of representative images from the candidate image set, each image in the set of representative images reflecting image features of a plurality of different images in the candidate image set; as well as training the defect detection model based on at least the set of historical images and the set of representative images to detect defects associated with objects in the image to be detected, Wherein determining the group of representative images from the candidate image set comprises: dividing the candidate image set into a plurality of groups of candidate images based on an image feature of each candidate image in the candidate image set; as well as selecting at least one candidate image from each of the plurality of groups to combine into the set of representative images, The number of candidate images selected from each group is related to the number of candidate images in the corresponding group. 2 . The method according to claim 1 , wherein the at least one candidate image selected in each group reflects image features of the candidate images in the corresponding group.

3. The method of claim 2, wherein dividing the set of candidate images into the plurality of groups comprises: performing a data dimensionality reduction operation on the image features of each candidate image in the candidate image set to obtain reduced-dimensionality image features; clustering the candidate images in the candidate image set based on the dimensionally reduced image features; as well as Based on the clustering result, a plurality of clusters corresponding to the plurality of groups are determined.

4. The method of claim 3 , wherein selecting the at least one candidate image from each of the plurality of groups to combine into the set of representative images comprises: For a first group among the plurality of groups, determining a center and an edge of a first cluster corresponding to the first group; as well as Select at least one of the following to form the set of representative images: a candidate image in the first cluster whose distance from the center is within a first threshold range, or A candidate image in the first cluster whose distance from the edge is within a second threshold range.

5. The method according to claim 1, further comprising: The set of representative images is added to the historical image set for subsequent training of the defect detection model.

6. A method for detecting defects, comprising: Obtaining an image to be detected, wherein an object in the image to be detected has a defect; as well as Using the defect detection model trained by the method according to any one of claims 1 to 5, information related to the defect is determined.

7. An electronic device comprising: processor; as well as a memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, causing the device to perform actions, the actions comprising: In response to receiving a candidate image set to be used for training the defect detection model, selecting a set of historical images from a historical image set that has been used to train the defect detection model, the candidate image set and images in the historical image set including defects associated with objects in the images; determining a set of representative images from the candidate image set, each image in the set of representative images reflecting image features of a plurality of different images in the candidate image set; and training the defect detection model based on at least the set of historical images and the set of representative images to detect defects associated with objects in the image to be detected, Wherein determining the group of representative images from the candidate image set comprises: dividing the candidate image set into a plurality of groups of candidate images based on image features of each candidate image in the candidate image set; and selecting at least one candidate image from each of the plurality of groups to combine into the set of representative images, The number of candidate images selected from each group is related to the number of candidate images in the corresponding group.

8. An electronic device comprising: processor; as well as a memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, causing the device to perform actions, the actions comprising: Obtaining an image to be detected, wherein an object in the image to be detected has a defect; as well as Using the defect detection model trained by the method according to any one of claims 1 to 5, information related to the defect is determined.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for training a defect detection model according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for detecting defects as claimed in claim 6 is implemented.

Images