Defect classification method and defect classification apparatus

Abstract

The application discloses a defect classification method and a defect classification device, and relates to the technical field of semiconductor manufacturing. The method comprises the following steps: acquiring an image set of a sample to be classified, wherein the image set comprises at least one group of images, each group of images comprises a defect image and a cropped image, the defect image is an image collected by an image collection device on at least one defect of the sample to be classified, the cropped image is an image obtained by image cropping on a target defect in the defect image, and the target defect is one of the at least one defect; and inputting the image set into a defect classification model to obtain a defect category of the target defect. The technical scheme provided by the application can increase the attention of the defect classification model to the defect and improve the accuracy of defect classification.

Description

Technical Field

[0001] This application relates to the field of semiconductor manufacturing technology, and in particular to a defect classification method and defect classification equipment. Background Technology

[0002] In semiconductor manufacturing, SEM images of defects on samples (such as wafers) are acquired using a scanning electron microscope (SEM), and these defects are then classified based on these images. Current defect classification methods typically input a single SEM image to be classified into a pre-trained classification network to obtain the classification result. However, since a single SEM image contains limited information, and the classification network derives its results based on image information, the accuracy of the classification results is relatively low. Therefore, it is necessary to improve the accuracy of defect classification methods. Summary of the Invention

[0003] This application provides a defect classification method and a defect classification device, which can increase the attention of the defect classification model to defects and improve the accuracy of defect classification.

[0004] Firstly, this application provides a defect classification method, which includes:

[0005] A set of images of a sample to be classified is obtained. The set of images includes at least one set of images. Each set of images includes a defect image and a cropped image. The defect image is an image acquired by an image acquisition device based on at least one defect of the sample to be classified. The cropped image is an image obtained by cropping a target defect from the defect image. The target defect is one of the at least one defect.

[0006] The defect category of the target defect is obtained by inputting the image set into the defect classification model.

[0007] In one embodiment, the defect classification model includes a first neural network; the step of obtaining the defect category of the target defect based on the image set input into the defect classification model includes:

[0008] The image set is input into the first neural network to obtain the corresponding first feature vector;

[0009] Based on the first feature vector, a classification process is performed to obtain a probability vector;

[0010] The defect category corresponding to the highest probability value in the probability vector is taken as the defect category of the target defect.

[0011] In one embodiment, the defect classification model further includes a second neural network; the step of obtaining the defect category of the target defect based on the image set input into the defect classification model further includes:

[0012] At least one of the cropped images from the image set is input into the second neural network to obtain the corresponding second feature vector;

[0013] The step of performing classification processing based on the first feature vector to obtain a probability vector includes:

[0014] The first feature vector and the second feature vector are fused to obtain a fused feature vector.

[0015] The fused feature vector is then classified to obtain a probability vector.

[0016] In one embodiment, the first neural network is a multi-channel neural network, and the number of channels of the first neural network is the same as the total number of images in the image set; the image set includes multiple sets of images, each set of images includes a cropped image, and the second neural network is a multi-channel neural network, and the number of channels of the second neural network is the same as the total number of cropped images.

[0017] In one embodiment, acquiring the image set of the samples to be classified includes:

[0018] Based on the image acquisition device, at least one image is acquired on at least one defect of the sample to be classified under a preset first field of view, so as to obtain at least one first defect image;

[0019] Defect detection is performed on each of the first defect images to obtain the location information of the target defect in the first defect image;

[0020] Based on the location information, an image region including the target defect is cropped at a preset size to obtain the cropped image corresponding to the first defect image;

[0021] A set of images is obtained based on each of the first defect images and the corresponding cropped images, and the image set is obtained based on all the sets of images corresponding to the target defect.

[0022] In one embodiment, the step of cropping an image region including the target defect according to a preset size based on the location information to obtain a cropped image corresponding to the first defect image includes:

[0023] Based on the location information, an image region including the target defect is cropped from the first defect image at a preset size to obtain the cropped image corresponding to the first defect image;

[0024] The step of obtaining a set of images based on each of the first defect images and the corresponding cropped images includes:

[0025] Each of the first defective images and the corresponding cropped image are treated as a group of images.

[0026] In one embodiment, the location information is first location information; the step of cropping an image region including the target defect according to a preset size based on the location information to obtain a cropped image corresponding to the first defect image includes:

[0027] Based on the image acquisition device and the first position information, a second defect image of the target defect is acquired under a preset second field of view. The second field of view is smaller than the first field of view, and one first defect image corresponds to one second defect image.

[0028] Defect detection is performed on each of the second defect images to obtain the second location information of the target defect in the second defect image;

[0029] For each of the second defect images, based on the second location information, an image region including the target defect is cropped from the second defect image according to the preset size to obtain the cropped image corresponding to the second defect image;

[0030] The step of obtaining a set of images based on each of the first defect images and the corresponding cropped images includes:

[0031] Each of the first defect image and / or the corresponding second defect image, as well as the cropped image corresponding to the second defect image, are respectively regarded as a group of images.

[0032] In one embodiment, the image acquisition device includes a scanning electron microscope or an optical microscope, the image set includes multiple sets of images, and the defect images in the image set include images acquired by a first detector and a second detector for at least one defect of the sample to be classified, respectively, with the first detector and the second detector located in different spatial positions.

[0033] In one embodiment, the image acquisition device includes a scanning electron microscope, wherein the first detector is a secondary electron detector and the second detector is a backscattered electron detector.

[0034] In one embodiment, before inputting the image set into the defect classification model, the method further includes:

[0035] The defective image and / or the cropped image in the image set are subjected to image scaling processing to obtain the defective image and the cropped image of the same size;

[0036] The defect classification model is trained in the following manner:

[0037] Multiple training image sets and labels corresponding to each training image set are obtained. Each training image set includes at least one set of training images. The number of sets of training images in each training image set is the same. Each set of training images includes a defect training image and a cropped training image. The defect training image is an image acquired based on at least one defect of the sample by an image acquisition device. The cropped training image is an image obtained by cropping an image of a defect in the defect training image.

[0038] The defect classification model is obtained by training a neural network based on the multiple training image sets and the labels corresponding to each training image set.

[0039] Secondly, this application provides a defect classification device, the defect classification device comprising:

[0040] Image acquisition device as described in any embodiment of this application;

[0041] At least one processor; and a memory communicatively connected to said at least one processor;

[0042] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the defect classification method described in any embodiment of this application.

[0043] Firstly, to address the shortcomings of existing technologies in the background art, embodiments of this application provide a defect classification method. Implementing this method yields the following beneficial effects: Compared to methods that use a single image of a sample to be classified for defect classification, this application uses an image set of the sample to be classified for defect classification. The image set includes at least one set of images, each set including a defect image and a cropped image. The defect image is an image acquired by an image acquisition device for at least one defect in the sample to be classified. The cropped image is an image obtained by cropping a target defect from the defect image; the target defect is at least one defect. The cropped image is used to highlight the area of ​​interest in the defect image. By adding a cropped image as input to the defect classification model based on the defect image, the model's attention to defects can be increased, thereby improving the accuracy of defect classification.

[0044] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the defect classification device, or it may be packaged separately from the processor of the defect classification device; this application does not impose any limitations on this.

[0045] The description of the second aspect in this application can be referred to the detailed description of the first aspect; and the beneficial effects of the description of the second aspect can be referred to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0046] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description.

[0047] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a defect classification method provided in an embodiment of this application;

[0050] Figure 2 A flowchart illustrating defect classification of samples to be classified, provided as an embodiment of this application;

[0051] Figure 3 Another flowchart for classifying defects in samples to be classified, provided as an embodiment of this application;

[0052] Figure 4 This is a block diagram of a defect classification device used to implement a defect classification method according to an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0054] It should be noted that the terms "first," "second," "target," and "original," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein. Furthermore, the terms "comprising," "having," and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0055] Figure 1 This is a flowchart illustrating a defect classification method provided in an embodiment of this application. This embodiment is applicable to scenarios where defect images of samples to be classified are acquired and classified during semiconductor manufacturing. The defect classification method provided in this embodiment can be executed by a defect classification device provided in this application. This device can be implemented through software and / or hardware and integrated into the electronic device executing the method. Preferably, the electronic device in this application embodiment can be a defect classification device.

[0056] See Figure 1 The method in this embodiment includes, but is not limited to, the following steps S110-S120.

[0057] S110. Obtain the image set of the samples to be classified.

[0058] The sample to be classified refers to the sample that needs to be classified, such as a wafer or a photomask. The image set includes at least one set of images, each set of images including a defect image and a cropped image. The defect image is an image acquired by the image acquisition device of at least one defect of the sample to be classified, and the cropped image is an image obtained by cropping the target defect from the defect image. The target defect is one of the at least one defects of the sample to be classified.

[0059] Regarding the selection of image acquisition devices and detectors, in one embodiment, the image acquisition device includes a SEM or an optical microscope, and the image set includes multiple sets of images. The defect images in the image set include images acquired using a first detector and a second detector, respectively, of at least one defect in the sample to be classified. The first detector and the second detector are located in different spatial positions. The advantage of this arrangement is that it provides defect images from different detectors for the defect classification model, thereby improving the generalization performance of the defect classification model. The number of defects in the images acquired using the first detector and the number of defects in the images acquired using the second detector can be the same or different.

[0060] In one embodiment, the image acquisition device includes a SEM, wherein the first detector and the second detector are of different types, specifically referring to the different types of signals collected from the sample by the detectors. For example, the first detector is a secondary electron detector used to collect secondary electrons to obtain an image, and the second detector is a backscattered electron detector used to collect backscattered electrons to obtain an image.

[0061] The advantage of this setup is that primary electrons emitted by the SEM interact with the sample at the surface, generating secondary electrons, and interact with the sample at deeper depths, generating backscattered electrons. Since different types of detectors (such as secondary electron detectors and backscattered electron detectors) can detect different types of electrons generated at different depths within the sample, images reflecting information at different depths are obtained, resulting in multiple sets of images. Using these images containing information at different depths as input to the defect classification model allows the model to learn more comprehensive features of the sample, including defects at different depths. Compared to using images acquired by a single type of detector as input, the model can better adapt to various situations, thus improving its generalization performance.

[0062] Optionally, defect images from the same set of images can be acquired using detectors of the same type. For example: Figure 2 and Figure 3 As shown, taking an image set with three sets of images as an example, the defect image in the first set of images is acquired by detector 1, the defect image in the second set of images is acquired by detector 2, and the defect image in the third set of images is acquired by detector 3. Detector 1, detector 2, and detector 3 are all secondary electron detectors or backscattered electron detectors. Detector 1, detector 2, and detector 3 include both secondary electron detectors and backscattered electron detectors. For example, detector 1 and detector 2 are both secondary electron detectors, and detector 3 is a backscattered electron detector. The spatial positions of detector 1, detector 2, and detector 3 are all different.

[0063] In one embodiment, for acquiring the image set, each image set includes a first defect image and a cropped image. The cropped image is obtained by cropping the target defect based on the location information of the target defect obtained by defect detection of the first defect image. For example, the cropped image is an image cropped from the first defect image, and the image size of the cropped image is smaller than the image size of the corresponding first defect image. The number of images in each image set is 2.

[0064] In one embodiment, acquiring an image set of a sample to be classified includes: acquiring at least one image of at least one defect of the sample to be classified using an image acquisition device under a preset first field of view to obtain at least one first defect image; performing defect detection on each first defect image to obtain the location information of the target defect in the first defect image; cropping an image region including the target defect according to a preset size based on the location information to obtain a cropped image corresponding to the first defect image; obtaining a set of images based on each first defect image and the corresponding cropped image; and obtaining an image set based on all sets of images corresponding to the target defect.

[0065] In one embodiment, cropping an image region including the target defect at a preset size based on location information to obtain a cropped image corresponding to a first defect image includes: cropping an image region including the target defect from the first defect image at a preset size based on location information to obtain a cropped image corresponding to the first defect image. In another embodiment, cropping an image region including the target defect from each first defect image at a preset size based on location information to obtain a corresponding cropped image.

[0066] For example, the specific process of obtaining the cropped image can be as follows: using SEM to acquire at least one image of the defects of the sample to be classified under the first field of view (denoted as FOV1), obtaining at least one defect image, and then using these defect images as one first defect image in each group of images. The number of defects in the sample to be classified can be at least one, and the target defect is one of the defects; for each first defect image, defect detection is performed based on a defect detection algorithm to obtain the location information of the defect. Optionally, the location information is, for example, the location information of the defect bounding box, and the defect bounding box is a rectangular area including the defect; select one defect as the target defect among the detected defects, and any image in the image set includes the target defect; then, according to the location information of the target defect, crop the image area including the target defect from the first defect image according to a preset size to obtain the cropped image corresponding to each first defect image.

[0067] Optionally, the preset size and the size of the defect bounding box can be the same or different. For example, when cropping an image from the first defect image, including the image area covered by the defect bounding box, the preset size is greater than or equal to the size of the defect bounding box.

[0068] In one embodiment, a set of images is obtained based on each first defect image and its corresponding cropped image, including: treating each first defect image and its corresponding cropped image as a set of images. For example, each set of images includes one first defect image and one cropped image.

[0069] In another embodiment, each set of images includes a first defect image, a second defect image, and a cropped image. The cropped image is an image cropped from the second defect image, and the image size of the cropped image is smaller than the image size of the corresponding second defect image. For example, the number of images in each set of images is 3.

[0070] In another embodiment, a second defect image can be obtained by acquiring a defect image under a second field of view based on the first location information of the target defect, performing defect detection on the second defect image to obtain the second location information of the target defect, and cropping an image region including the target defect in the second defect image according to a preset size based on the second location information to obtain a cropped image.

[0071] In another embodiment, acquiring an image set of samples to be classified includes: acquiring at least one image of at least one defect of the sample to be classified using an image acquisition device under a preset first field of view to obtain at least one first defect image; performing defect detection on each first defect image to obtain the position information of the target defect in the first defect image; the position information is first position information; acquiring a second defect image of the target defect under a preset second field of view based on the image acquisition device and the first position information; performing defect detection on each second defect image to obtain the second position information of the target defect in the second defect image; for each second defect image, cropping an image region including the target defect from the second defect image according to a preset size based on the second position information to obtain a cropped image corresponding to the second defect image; and taking each first defect image and / or the corresponding second defect image, as well as the cropped image corresponding to the second defect image, as a group of images; for example, taking each first defect image, the second defect image corresponding to the first defect image, and the cropped image corresponding to the second defect image as a group of images, in which case each group of images includes 3 images.

[0072] In this system, the second field of view is smaller than the first field of view, and one first defect image corresponds to one second defect image. For the same defect (i.e., the target defect), compared to using only one image, using first and second defect images acquired with different fields of view can provide defect background data with different amounts of information, thus improving the accuracy of classification results.

[0073] For example, the specific process of acquiring the cropped image can be as follows: using SEM to acquire at least one image of the defects of the sample to be classified under the first field of view (denoted as FOV1), obtaining at least one defect image, and then using these defect images as one first defect image in each group of images. The number of defects in the sample to be classified can be at least one, and the target defect is one of the defects; for each first defect image, a defect detection algorithm is used to detect the defect, obtaining the location information of the defect, denoted as the first location information. Optionally, the first location information is, for example, the location information of the defect bounding box, where the defect bounding box is a rectangular region including the defect; upon detection... One defect is selected as the target defect, and any image in the image set includes the target defect. A second defect image of the target defect is acquired using SEM in the second field of view (FOV2) based on the first position information. For each second defect image, a defect detection algorithm is used to detect the defect, and the position information of the target defect is obtained again, which is denoted as the second position information. Optionally, the second position information is, for example, the position information of the defect bounding box, which is a rectangular area including the target defect. Based on the second position information, the image area including the target defect is cropped from each second defect image according to a preset size to obtain the cropped image corresponding to each second defect image.

[0074] Optionally, the defect detection algorithm used to determine the location information of the defect can be an existing detection algorithm. The defect detection algorithm used for the first defect image and the defect detection algorithm used for the second defect image can be the same or different.

[0075] In another embodiment, each set of images includes a second defect image and a cropped image, wherein the cropped image is an image cropped from the second defect image. Based on the above embodiments, a second defect image and a cropped image obtained by cropping the second defect image can be obtained, and each second defect image and its corresponding cropped image can be used as a set of images.

[0076] In another embodiment, each set of images includes a first defect image and a cropped image, wherein the cropped image is an image cropped from the second defect image. Based on the above embodiments, a first defect image and a cropped image obtained by cropping the second defect image can be obtained, and each first defect image and its corresponding cropped image can be used as a set of images.

[0077] S120. Input the image set into the defect classification model to obtain the defect category of the target defect.

[0078] In one embodiment, the defect classification model includes a first neural network. The first neural network is a multi-channel neural network, such as a multi-channel convolutional neural network (CNN). The number of channels in the first neural network is the same as the total number of images in the image set. For example, if the image set includes 9 images, then the first neural network is a 9-channel neural network.

[0079] In one embodiment, the defect category of a target defect is obtained by inputting an image set into a defect classification model, including: inputting the image set into a first neural network to obtain a corresponding first feature vector; performing classification processing based on the first feature vector to obtain a probability vector; and taking the defect category corresponding to the highest probability value in the probability vector as the defect category of the target defect.

[0080] In one embodiment, classification processing based on a first feature vector to obtain a probability vector includes: classifying the first feature vector to obtain a probability vector. Wherein, as... Figure 2 As shown, the defect classification model also includes a classification layer, which is connected to the first neural network. The classification layer is used to calculate a probability vector. Optionally, the classification layer can be a Softmax function layer. In the first neural network, the convolutional layers can be configured with the same number of convolutional kernels as the number of images in the image set. These convolutional kernels as a whole can be called multi-channel convolutional kernels, and each convolutional kernel can be the same or different.

[0081] The existing technology for obtaining feature vectors using neural networks exemplifies this. For instance, each convolutional kernel is convolved with an image, and the results are summed. This summation, obtained from the multi-channel convolutional kernels, is then input into the pooling layer and the fully connected layer of the first neural network to output a first feature vector. A classification layer then performs classification processing on the first feature vector to obtain a probability vector. By searching for the maximum value (i.e., the highest probability value) in the probability vector, the index position of this maximum value within the vector is found. This index position corresponds one-to-one with a preset defect category, thus determining the corresponding defect category.

[0082] For example, using a probability vector of [0.1, 0.2, 0.7], the probability that the target defect belongs to three different defect categories is 10%, 20%, and 70%, respectively. Taking indices 0, 1, and 2 in the probability vector as examples, the defect category corresponding to index 2 is a scratch. Since the third element in the probability vector (index 2) has the largest value (0.7), the defect category of the target defect can be determined to be a scratch.

[0083] In another embodiment, such as Figure 3As shown, the defect classification model includes a first neural network, a second neural network, a fully connected layer, and a classification layer. The first neural network outputs a first feature vector, and the second neural network outputs a second feature vector. Both the first and second neural networks are connected to the fully connected layer, which outputs a fused feature vector. The fully connected layer is also connected to the classification layer. The second neural network is a multi-channel neural network, with the number of channels equal to the total number of images captured. The first and second neural networks can be the same or different neural networks, such as different CNN networks.

[0084] For example, in the second neural network, the convolutional layers can be configured with the same number of convolutional kernels as the total number of images being cropped. These convolutional kernels as a whole can be called multi-channel convolutional kernels. Each convolutional kernel can be the same or different. Each convolutional kernel is convolved with an image, and the results of the convolutions are summed to obtain a summation result. Thus, a summation result is obtained based on the multi-channel convolutional kernels. This summation result is then input into the pooling layer and the fully connected layer in the second neural network, and a second feature vector can be output.

[0085] In another embodiment, the defect category of the target defect is obtained by inputting an image set into a defect classification model, including: inputting the image set into a first neural network to obtain a corresponding first feature vector; inputting at least one cropped image from the image set into a second neural network to obtain a corresponding second feature vector; fusing the first feature vector and the second feature vector to obtain a fused feature vector; classifying the fused feature vector to obtain a probability vector; and taking the defect category corresponding to the highest probability value in the probability vector as the defect category of the target defect.

[0086] For example, the image set includes multiple sets of images, each set including a cropped image. The cropped image is used to highlight the region of interest in the defect image. Based on the defect image, at least one cropped image (e.g., all cropped images) from the image set is used as input to a second neural network. The second neural network extracts a second feature vector corresponding to the cropped image, which can further highlight the defect features of the defect of interest (i.e., the target defect), increase the degree of attention the defect classification model pays to the target defect, and further improve the accuracy of defect classification.

[0087] Furthermore, before inputting the image set into the defect classification model, the defect classification method also includes: performing image scaling on the defect images and / or cropped images in the image set to obtain defect images and cropped images of the same size, thereby setting each image in the image set to the same image size, the size of which depends on the needs.

[0088] For example, the first defect image and the second defect image have the same image size. The cropped image is enlarged so that the cropped image has the same image size as the second defect image. The first defect image, the second defect image, and the cropped image all contain the same defect (i.e., the target defect).

[0089] The technical solution provided in this embodiment acquires an image set of a sample to be classified. The image set includes at least one set of images, each set including a defect image and a cropped image. The defect image is an image acquired by an image acquisition device of at least one defect of the sample to be classified. The cropped image is an image obtained by cropping a target defect from the defect image. The target defect is one of the at least one defects. The image set is input into a defect classification model to obtain the defect category of the target defect. This application uses an image set of the sample to be classified instead of a single image of the sample to classify defects. The cropped image is used to highlight the area of ​​interest in the defect image. By adding the cropped image as input to the defect classification model based on the defect image, the model's attention to defects can be increased, thereby improving the accuracy of defect classification.

[0090] In one embodiment, the defect classification model is trained as follows:

[0091] The first step is to obtain multiple training image sets and the corresponding labels for each training image set. Each training image set includes at least one set of training images, and the number of sets of training images in each training image set is the same. Each set of training images includes a defect training image and a cropped training image. The defect training image is an image acquired based on at least one defect of the sample by the image acquisition device, and the cropped training image is an image obtained by cropping an image of a defect from the defect training image.

[0092] For example, each training image set includes three sets of training images. Each set of training images includes a first defect training image, a second defect training image, and a cropped training image. Therefore, each training image set includes 9 training images. Please refer to the relevant content for the first defect image, the second defect image, and the cropped image.

[0093] Each training image set includes multiple images of the same defect. A label can be manually assigned to each training image set, representing the category of the defect. Because each training image set corresponds to one label, there are multiple defect categories for multiple training image sets.

[0094] The second step is to train a neural network based on multiple sets of training images and the corresponding labels for each set of training images to obtain a defect classification model.

[0095] For example, the neural network includes a first neural network, a second neural network, a fully connected layer, and a classification layer. Each set of training images is used as the input to the first neural network, and all cropped training images in each set are used as the input to the second neural network. The network architecture of the neural network can be referred to the relevant content of the defect classification model in this application, and will not be repeated here. The neural network can be trained using existing methods to obtain a trained neural network, which is the defect classification model.

[0096] In one practical application embodiment, Figure 3 This is a flowchart illustrating the process of classifying defects in samples to be categorized. Figure 3 The diagram illustrates that the image set contains a total of 3 groups of images, i.e., 3 groups of images. It also illustrates that each group of images includes one first defect image, one second defect image, and one cropped image. All images (e.g., 9 images) include the same defect (i.e., the target defect), meaning that the image set includes multiple images of the same defect.

[0097] The first defect image includes at least one defect. For example, the first defect image includes two defects. After defect detection is performed on the first defect image, the location information of each of the two defects is determined. One of the defects is selected as the target defect. Based on the location information of the target defect (i.e., the first location information), an image of the target defect is acquired to obtain a second defect image. The image region in the second defect image that includes the target defect is cropped to obtain a cropped image. Therefore, it can be seen that the first defect image, the second defect image, and the cropped image all include the same defect. For example, both the second defect image and the cropped image include only one defect.

[0098] For example, the defect classification model includes a first neural network, a second neural network, a fully connected layer, and a classification layer. The image set is used as the input of the first neural network, and all cropped images in the image set are used as the input of the second neural network. The first feature vector and the second feature vector are fused through the fully connected layer to obtain a fused feature vector. The fused feature vector is classified through the classification layer to obtain a probability vector. The defect category corresponding to the highest probability value in the probability vector is taken as the defect category of the target defect.

[0099] Figure 4 This is a block diagram of a defect classification device used to implement a defect classification method according to an embodiment of this application.

[0100] like Figure 4As shown, the defect classification device 10 includes an image acquisition device (not shown) as described in any of the above embodiments, at least one processor 11, and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores a computer program executable by the at least one processor. The processor 11 can execute the defect classification method described in any of the above embodiments according to the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the defect classification device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0101] Multiple components in the defect classification device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the defect classification device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as defect classification methods.

[0103] In some embodiments, the defect classification method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the defect classification device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the defect classification method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the defect classification method by any other suitable means (e.g., by means of firmware).

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] To provide user interaction, the systems and techniques described herein can be implemented on a defect classification device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the defect classification device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0109] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0110] Note that the above are merely preferred embodiments and technical principles applied in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. For example, those skilled in the art can use the various forms of processes shown above to reorder, add, or delete steps; the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution of this application can be achieved, and no limitations are imposed herein.

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A defect classification method, characterized in that, The method includes: A set of images of a sample to be classified is obtained. The set of images includes at least one set of images. Each set of images includes a defect image and a cropped image. The defect image is an image acquired by an image acquisition device based on at least one defect of the sample to be classified. The cropped image is an image obtained by cropping a target defect from the defect image. The target defect is one of the at least one defect. The defect category of the target defect is obtained by inputting the image set into the defect classification model.

2. The defect classification method according to claim 1, characterized in that, The defect classification model includes a first neural network; the step of inputting the image set into the defect classification model to obtain the defect category of the target defect includes: The image set is input into the first neural network to obtain the corresponding first feature vector; Based on the first feature vector, a classification process is performed to obtain a probability vector; The defect category corresponding to the highest probability value in the probability vector is taken as the defect category of the target defect.

3. The defect classification method according to claim 2, characterized in that, The defect classification model further includes a second neural network; the step of inputting the image set into the defect classification model to obtain the defect category of the target defect further includes: At least one of the cropped images from the image set is input into the second neural network to obtain the corresponding second feature vector; The step of performing classification processing based on the first feature vector to obtain a probability vector includes: The first feature vector and the second feature vector are fused to obtain a fused feature vector. The fused feature vector is then classified to obtain a probability vector.

4. The defect classification method according to claim 3, characterized in that, The first neural network is a multi-channel neural network, and the number of channels of the first neural network is the same as the total number of images in the image set; the image set includes multiple sets of images, and each set of images includes a cropped image; the second neural network is a multi-channel neural network, and the number of channels of the second neural network is the same as the total number of cropped images.

5. The defect classification method according to claim 1, characterized in that, The acquisition of the image set of the samples to be classified includes: Based on the image acquisition device, at least one image is acquired on at least one defect of the sample to be classified under a preset first field of view, so as to obtain at least one first defect image; Defect detection is performed on each of the first defect images to obtain the location information of the target defect in the first defect image; Based on the location information, an image region including the target defect is cropped at a preset size to obtain the cropped image corresponding to the first defect image; A set of images is obtained based on each of the first defect images and the corresponding cropped images, and the image set is obtained based on all the sets of images corresponding to the target defect.

6. The defect classification method according to claim 5, characterized in that, The step of cropping an image region including the target defect according to a preset size based on the location information to obtain the cropped image corresponding to the first defect image includes: Based on the location information, an image region including the target defect is cropped from the first defect image at a preset size to obtain the cropped image corresponding to the first defect image; The step of obtaining a set of images based on each of the first defect images and the corresponding cropped images includes: Each of the first defective images and the corresponding cropped image are treated as a group of images.

7. The defect classification method according to claim 5, characterized in that, The location information is first location information; the step of cropping an image region including the target defect according to a preset size based on the location information to obtain a cropped image corresponding to the first defect image includes: Based on the image acquisition device and the first position information, a second defect image of the target defect is acquired under a preset second field of view. The second field of view is smaller than the first field of view, and one first defect image corresponds to one second defect image. Defect detection is performed on each of the second defect images to obtain the second location information of the target defect in the second defect image; For each of the second defect images, based on the second location information, an image region including the target defect is cropped from the second defect image according to the preset size to obtain the cropped image corresponding to the second defect image; The step of obtaining a set of images based on each of the first defect images and the corresponding cropped images includes: Each of the first defect image and / or the corresponding second defect image, as well as the cropped image corresponding to the second defect image, are respectively regarded as a group of images.

8. The defect classification method according to claim 1, characterized in that, The image acquisition device includes a scanning electron microscope or an optical microscope, the image set includes multiple sets of images, and the defect images in the image set include images acquired by a first detector and a second detector for at least one defect of the sample to be classified, respectively, with the first detector and the second detector located in different spatial positions.

9. The defect classification method according to claim 8, characterized in that, The image acquisition device includes a scanning electron microscope, the first detector is a secondary electron detector, and the second detector is a backscattered electron detector.

10. The defect classification method according to claim 1, characterized in that, Before inputting the image set into the defect classification model, the method further includes: The defective image and / or the cropped image in the image set are subjected to image scaling processing to obtain the defective image and the cropped image of the same size; The defect classification model is trained in the following manner: Multiple training image sets and labels corresponding to each training image set are obtained. Each training image set includes at least one set of training images. The number of sets of training images in each training image set is the same. Each set of training images includes a defect training image and a cropped training image. The defect training image is an image acquired based on at least one defect of the sample by an image acquisition device. The cropped training image is an image obtained by cropping an image of a defect in the defect training image. The defect classification model is obtained by training a neural network based on the multiple training image sets and the labels corresponding to each training image set.

11. A defect classification device, characterized in that, The defect classification equipment includes: The image acquisition device as described in any one of claims 1 to 10; At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the defect classification method according to any one of claims 1 to 10.

Images