Pattern inspection device

Abstract

To provide a novel classification device.SOLUTION: The classification device includes a storage unit, a processing unit, and a classifier. A plurality of pieces of image data and an identification model are stored in the storage unit. Each of the plurality of pieces of image data is image data determined to contain a defect. The identification model includes an input layer, an intermediate layer, and an output layer. First to n-th (n is an integer greater than or equal to 2) image data of the plurality of pieces of image data are supplied to the processing unit. The processing unit has a function of outputting respective feature quantities of the first to n-th image data (first to n-th feature quantities) based on the identification model. The feature quantities output from the processing unit are numerical values of a neuron included in the intermediate layer. The first to n-th feature quantities output from the processing unit are supplied to the classifier. The classifier has a function of performing clustering of the first to n-th image data based on the first to n-th feature quantities.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] 1. Field of the Invention An aspect of the present invention relates to a classification device, an image classification method, and a pattern inspection device. [Background technology]

[0002] Visual inspection is one of the defect detection methods in the semiconductor manufacturing process. A pattern inspection device is one example of a device (visual inspection device) for automatically performing visual inspection. Defects are detected and identified on an image acquired using the visual inspection device. When defect detection and identification are performed visually, there may be individual differences in the accuracy of defect detection and identification. Furthermore, when the number of image data is large, it takes a huge amount of time to detect and identify defects.

[0003] In recent years, techniques for identifying defects using neural networks have been reported. For example, Patent Document 1 discloses a defect type determination device that identifies defects using a neural processing unit that is trained to input information about defects and output defect identification results. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 8-21803 Summary of the Invention [Problem to be solved by the invention]

[0005] To train the neuro-processing unit, it is necessary to associate information about defects with the types of defects in advance. In Patent Document 1, the information about defects includes the area, shape, and position of the defects, and is acquired using an image processing device. However, when multiple defects overlap or exist in the same area, the accuracy of the information about the defects may decrease.

[0006] When a defect is detected, it is necessary to decide whether or not to perform the rework process. The decision on whether or not to perform the rework process must be made based on not only the type of defect, but also the overall occurrence frequency of each defect, the distribution of defects within the lot / board, etc.

[0007] In view of the above, an object of one embodiment of the present invention is to provide a novel classification device.Another object of one embodiment of the present invention is to provide a novel image classification method.Another object of one embodiment of the present invention is to provide a novel pattern inspection device.

[0008] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract problems other than these from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0009] An aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. The storage unit stores a plurality of image data and a discrimination model. The discrimination model has a plurality of convolution layers, a plurality of pooling layers, a first fully connected layer, a second fully connected layer, a third fully connected layer, a fourth fully connected layer, and a fifth fully connected layer. The fifth fully connected layer is an output layer. The fourth fully connected layer is connected to the fifth fully connected layer. The third fully connected layer is connected to the fourth fully connected layer. The second fully connected layer is connected to the third fully connected layer. The first fully connected layer is connected to the second fully connected layer. The processing unit is supplied with first to n-th (n is an integer of 2 or more) image data out of the plurality of image data. The processing unit has a function of outputting the k-th feature amount of the k-th (k is an integer of 1 to n) image data based on the discrimination model. The feature values ​​output from the processing unit are the numerical values ​​of the neurons in the first fully connected layer, the numerical values ​​of the neurons in the second fully connected layer, or the numerical values ​​of the neurons in the third fully connected layer. The first to n-th feature values ​​output from the processing unit are supplied to a classifier. The classifier has a function of performing clustering of the first to n-th image data based on the first to n-th feature values.

[0010] Another aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. The storage unit stores a plurality of image data and a discrimination model. The discrimination model has a plurality of convolution layers, a plurality of pooling layers, a first fully connected layer, a second fully connected layer, and a third fully connected layer. The third fully connected layer is an output layer. The second fully connected layer is connected to the third fully connected layer. The first fully connected layer is connected to the second fully connected layer. The first to n-th (n is an integer of 2 or more) image data of the plurality of image data are supplied to the processing unit. The processing unit has a function of outputting the k-th feature amount of the k-th (k is an integer of 1 to n) image data based on the discrimination model. The feature amount output from the processing unit is the numerical value of the neuron in the first fully connected layer or the numerical value of the neuron in the second fully connected layer. The first to n-th feature amounts output from the processing unit are supplied to the classifier. The classifier has a function of performing clustering of the first to n-th image data based on the first to n-th feature amounts.

[0011] Another aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. The storage unit stores a plurality of image data and a discrimination model. The discrimination model has a plurality of convolution layers, a plurality of pooling layers, and a fully connected layer. The fully connected layer is an output layer. The first to n-th (n is an integer equal to or greater than 2) image data of the plurality of image data are supplied to the processing unit. The processing unit has a function of outputting a k-th feature amount of the k-th (k is an integer equal to or greater than 1 and equal to or less than n) image data based on the discrimination model. The feature amount output from the processing unit is a numerical value of a neuron in any one of the plurality of convolution layers, or a numerical value of a neuron in any one of the plurality of pooling layers. The first to n-th feature amounts output from the processing unit are supplied to the classifier. The classifier has a function of clustering the first to n-th image data based on the first to n-th feature amounts.

[0012] In the above classification device, it is preferable that each of the plurality of image data is image data that has been determined to include a defect.

[0013] Another aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. The storage unit stores a plurality of image data and a discrimination model. Each of the plurality of image data is image data that is determined to include a defect. The discrimination model has an input layer, an intermediate layer, and an output layer. The first to n-th (n is an integer of 2 or more) image data of the plurality of image data are supplied to the processing unit. The processing unit has a function of outputting a k-th feature amount of the k-th (k is an integer of 1 to n) image data based on the discrimination model. The feature amount output from the processing unit is a numerical value of a neuron in the intermediate layer. The first to n-th feature amounts output from the processing unit are supplied to the classifier. The classifier has a function of clustering the first to n-th image data based on the first to n-th feature amounts.

[0014] Another aspect of the present invention is a classification device having a storage unit, a processing unit, a processing section, and a classifier. The storage unit stores a plurality of image data and a discrimination model. Each of the plurality of image data is image data that is determined to include a defect. The discrimination model has an input layer, an intermediate layer, and an output layer. The processing unit is supplied with first to n-th (n is an integer of 2 or more) image data among the plurality of image data. The processing unit has a function of generating (n+k)-th image data by removing a part of the k-th (k is an integer of 1 to n) image data. The processing unit is supplied with (n+1) to (2n)-th image data. The processing unit has a function of outputting the k-th feature value of the (n+k)-th image data based on the discrimination model. The feature value output from the processing unit is a value of a neuron in the intermediate layer. The classifier is supplied with the first to n-th feature values ​​output from the processing unit. The classifier has a function of performing clustering of the first to n-th image data based on the first to n-th feature amounts.

[0015] In the above classification device, the number of dimensions of the feature output from the processing unit is preferably 32 or more and 256 or less.

[0016] In the above classification device, it is preferable that the discrimination model undergoes supervised learning so that the type of defect can be inferred for image data determined to contain a defect, and that a hierarchical method is used for clustering.

[0017] In addition, the classification device preferably has an output unit in addition to the storage unit, the processing unit, and the classifier, and the output unit has a function of displaying the result of the clustering performed by the classifier.

[0018] Another aspect of the present invention is a pattern inspection device including the classification device, an imaging unit, and an inspection device. The imaging unit has a function of imaging an object to be inspected. The inspection device has a function of determining whether or not a defect is included in image data acquired by imaging the object by the imaging unit.

[0019] Another aspect of the present invention is an image classification method which includes a first step of supplying 1st to nth (n is an integer equal to or greater than 2) image data to a processing unit, a second step of using the processing unit to extract 1st to nth feature amounts for the 1st to nth image data based on a discrimination model, a third step of supplying the 1st to nth feature amounts to a classifier, and a fourth step of using the classifier to cluster the 1st to nth image data based on the 1st to nth feature amounts, wherein each of the 1st to nth image data is image data that has been determined to contain a defect, the discrimination model has an input layer, an intermediate layer, and an output layer, and the feature amount output from the processing unit is a numerical value of a neuron in the intermediate layer.

[0020] In the above image classification method, it is preferable that the discrimination model undergoes supervised learning so as to infer the type of defect for image data determined to contain a defect, and that a hierarchical method is used for clustering.

[0021] In the above image classification method, the number of dimensions of the feature output from the processing unit is preferably 32 or more and 256 or less.

[0022] In the above image classification method, it is preferable that in the fifth step, the result of the clustering performed by the classifier is supplied to an output unit, and in the sixth step, the result is displayed. Effect of the Invention

[0023] According to one aspect of the present invention, a novel classification device can be provided. Also, according to one aspect of the present invention, a novel image classification method can be provided. Also, according to one aspect of the present invention, a novel pattern inspection device can be provided.

[0024] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from descriptions in the specification, drawings, etc., and can be appropriately extracted from these descriptions. One embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a block diagram illustrating an example of a classification device. [Diagram 2] FIG. 2 is a diagram showing an example of image data and a label. [Diagram 3] 3A and 3B are schematic diagrams showing examples of the configuration of a neural network. [Figure 4] 4A and 4B are block diagrams illustrating an example of a classification device. [Diagram 5] FIG. 5 is a flow chart illustrating an example method for classifying image data. [Figure 6] FIG. 6 is a diagram illustrating an example of image data and feature amounts. [Figure 7]FIG. 7 is a flow chart illustrating an example of a method for classifying image data. [Figure 8] FIG. 8 is a flowchart illustrating an example of a method for generating a trained discriminative model. [Figure 9] FIG. 9 is a diagram illustrating an example of hardware of the classification device. [Figure 10] FIG. 10 is a diagram illustrating an example of hardware of a classification device. [Figure 11] FIG. 11 is a block diagram showing an example of a pattern inspection apparatus. [Figure 12] FIG. 12 is a diagram for explaining the result of clustering. [Figure 13] 13A to 13D are diagrams showing an example of image data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be easily understood by those skilled in the art that the modes and details of the present invention can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below.

[0027] In the configurations of the invention described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and repeated explanations thereof will be omitted.

[0028] In addition, for ease of understanding, the position, size, range, etc. of each component shown in the drawings may not represent the actual position, size, range, etc. Therefore, the disclosed invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings.

[0029] In addition, it should be noted that the ordinal numbers "first," "second," and "third" used in this specification are used to avoid confusion of components and are not intended to limit the numbers.

[0030] In addition, in the present specification, when upper and lower limit values ​​are specified, it is understood that any combination of these values ​​is also disclosed.

[0031] In this specification, a data set used for learning and evaluating a machine learning model is referred to as a learning data set. When learning and evaluating a machine learning model, the learning data set is divided into learning data (also called training data) and test data (also called evaluation data). Furthermore, the learning data may be divided into learning data and verification data. Note that the test data may be divided in advance from the learning data set.

[0032] Training data is data used to train a machine learning model. Validation data is data used to evaluate the learning results of a machine learning model. Test data is data used to evaluate a machine learning model. When machine learning is supervised learning, the training data, validation data, and test data are assigned labels.

[0033] In this specification and the like, a semiconductor element refers to an element that can function by utilizing semiconductor characteristics. An example is a semiconductor element such as a transistor, a diode, a light-emitting element, or a light-receiving element. Another example of a semiconductor element is a passive element such as a capacitance, a resistor, or an inductor that is formed by a conductive film or an insulating film. Another example of a semiconductor element is a semiconductor device that includes a circuit having a semiconductor element or a passive element.

[0034] (Embodiment 1) In this embodiment, a classification device of one embodiment of the present invention will be described with reference to FIGS.

[0035] The classification device can be provided in an information processing device such as a personal computer used by a user, or the classification device can be provided in a server and used from a client PC via a network.

[0036] The classification device has a function of clustering image data. In this embodiment, the classification device is described as clustering defects detected in a semiconductor manufacturing process. In other words, the image data is image data including defects detected in the semiconductor manufacturing process.

[0037] Defects detected in the semiconductor manufacturing process include, for example, foreign matter contamination, film loss, pattern defects, remaining film, film floating, and broken wires. Foreign matter contamination refers to defects caused by foreign matter originating from workers, materials, manufacturing equipment, the working environment, etc., adhering to a substrate (semiconductor substrate such as silicon wafer, glass substrate, plastic substrate, metal substrate, SOI substrate, etc.) during the semiconductor manufacturing process. Film loss refers to defects caused by peeling off of a normal pattern. Pattern defects refer to defects caused by a pattern not being formed as designed.

[0038] The image data is image data obtained by photographing an area where a semiconductor film, insulating film, wiring, or other pattern (hereinafter simply referred to as a pattern) is not normal in a semiconductor device in the middle of a manufacturing process or in a semiconductor device where a manufacturing process has been completed. In other words, the image data is image data obtained by photographing an area where a defect is observed. The image data may be simply referred to as image data including a defect.

[0039] <Classification device 100> An example of a classification device according to an embodiment of the present invention is shown in Fig. 1. Fig. 1 is a diagram showing a configuration of a classification device 100. As shown in Fig. 1, the classification device 100 includes a storage unit 101, a processing unit 102, a classifier 103, and an output unit 104.

[0040] [Storage section 101] Image data is stored in the storage unit 101. The image data includes defects.

[0041] Here, the image data stored in the storage unit 101 will be described with reference to Fig. 2. The storage unit 101 stores a plurality of image data 50. As shown in Fig. 2, the plurality of image data 50 includes image data 51_1 to image data 51_s (s is an integer equal to or greater than 1) and image data 52_1 to image data 52_t (t is an integer equal to or greater than 1).

[0042] A label is assigned to each of the image data 51_1 to 51_s. In FIG. 2, a label 61A is assigned to the image data 51_1 and the image data 51_2. A label 61F is assigned to the image data 51_s. In this embodiment, the labels assigned to the image data 51_1 to 51_s correspond to defects detected in the semiconductor manufacturing process. In other words, each of the labels 61A, 61F, etc. corresponds to one or more of the defects detected in the semiconductor manufacturing process. The type of defect detected in the semiconductor manufacturing process is given as a numerical array.

[0043] No label corresponding to a defect is assigned to the image data 52_1 to 52_t. Note that "-" in Fig. 2 indicates that no label is assigned to the image data.

[0044] [Processing section 102] The processing unit 102 has a function of performing processing using a trained discrimination model. Specifically, the processing unit 102 has a function of extracting features from image data using a trained discrimination model. The processing unit 102 is supplied with image data 53_1 to 53_n (n is an integer equal to or greater than 2) from the storage unit 101. Here, the image data 53_1 to 53_n are part or all of the multiple image data 50 stored in the storage unit 101. At this time, the processing unit 102 extracts features for each of the image data 53_1 to 53_n.

[0045] For the identification model, it is preferable to use a neural network, and more preferably a convolutional neural network (CNN). Examples of CNN include VGG11, VGG16, GoogLeNet, and ResNet.

[0046] 3A is a diagram showing an example of the configuration of a neural network 300. The neural network 300 has layers 301_1 to 301_k (k is an integer equal to or greater than 3).

[0047] The layers 301_1 to 301_k each have neurons, and the neurons in each layer are connected to each other. For example, the neurons in the layer 301_1 are connected to the neurons in the layer 301_2. The neurons in the layer 301_2 are connected to the neurons in the layer 301_1 and the neurons in the layer 301_3. The same is true for the neurons in the layers 301_3 to 301_k. That is, the layers 301_1 to 301_k form a hierarchical neural network.

[0048] Image data is input to layer 301_1, which outputs data corresponding to the input image data. The data is input to layer 301_2, which outputs data corresponding to the input data. Data output from layer 301_k-1 is input to layer 301_k, which outputs data corresponding to the input data. From the above, layer 301_1 can be regarded as an input layer, layers 301_2 to 301_k-1 as intermediate layers, and layer 301_k as an output layer. A neural network having two or more intermediate layers is also called deep learning.

[0049] The neural network 300 is trained in advance so that data output from, for example, layers 301_1 to 301_k corresponds to the features of image data input to the neural network 300. The training can be performed by unsupervised training, supervised training, or the like. Regardless of whether the training is performed by unsupervised training or supervised training, a backpropagation method or the like can be used as a training algorithm. In this embodiment, the training is preferably performed by supervised training.

[0050] An example of a CNN is shown in FIG. 3B. FIG. 3B is a diagram showing the configuration of a CNN 310. As shown in FIG. 3B, the CNN 310 has a plurality of convolution layers (convolution layer 311_1 to convolution layer 311_m (m is an integer equal to or greater than 1)), a plurality of pooling layers (pooling layer 312_1 to pooling layer 312_m), and a fully connected layer 313. FIG. 3B shows an example in which the fully connected layer 313 has three layers, namely, a fully connected layer 313_1, a fully connected layer 313_2, and a fully connected layer 313_3. Note that the CNN 310 may have only one or two fully connected layers 313, or may have four or more fully connected layers.

[0051] The convolution layer has a function of performing convolution on data input to the convolution layer. For example, the convolution layer 311_1 has a function of performing convolution on input image data. The convolution layer 311_2 has a function of performing convolution on data output from the pooling layer 312_1. The convolution layer 311_m has a function of performing convolution on data output from the pooling layer 312_m-1.

[0052] The convolution is performed by repeating a multiplication and accumulation operation between the data input to the convolution layer and the weight filter. By the convolution in the convolution layer, the features of the image corresponding to the image data input to the CNN 310 are extracted.

[0053] The convolved data is converted by an activation function and then output to the pooling layer. As the activation function, ReLU (Rectified Linear Units) or the like can be used. ReLU is a function that outputs "0" when the input value is negative, and outputs the input value as is when the input value is "0" or greater. In addition, a sigmoid function, a tanh function, or the like can also be used as the activation function.

[0054] The pooling layer has a function of performing pooling on the data input from the convolution layer. Pooling is a process of dividing data into multiple regions, extracting predetermined data for each region, and arranging them in a matrix. Pooling can reduce the amount of data while retaining the features extracted by the convolution layer. It can also increase robustness against small deviations in the input data. Note that maximum pooling, average pooling, Lp pooling, etc. can be used as pooling.

[0055] The fully connected layer 313 has a function of transforming input data by an activation function and outputting the transformed data. Specifically, when the CNN 310 has the configuration shown in FIG. 3B, the fully connected layer 313_1 has a function of transforming data output from the pooling layer 312_m by an activation function. The fully connected layer 313_2 has a function of transforming data output from the fully connected layer 313_1 by an activation function. The fully connected layer 313_3 has a function of transforming data output from the fully connected layer 313_2 by an activation function. As the activation function, ReLU, sigmoid function, tanh function, etc. can be used. The fully connected layer 313 has a configuration in which all nodes of a certain layer are connected to all nodes of the next layer. The data output from the convolution layer or the pooling layer is a two-dimensional feature map, and is expanded into one dimension when input to the fully connected layer 313. Then, a vector obtained by inference by the fully connected layer 313 is output from the fully connected layer 313.

[0056] In the CNN 310, one of the layers of the fully connected layer 313 can be the output layer. For example, in the CNN 310 shown in FIG. 3B, the fully connected layer 313_3 can be the output layer. Here, in the CNN 310 shown in FIG. 3B, the fully connected layer 313_1 and the fully connected layer 313_2 can be the intermediate layers. Alternatively, when the CNN 310 has only the fully connected layer 313_1 as the fully connected layer 313, the fully connected layer 313_1 can be the output layer. Alternatively, when the CNN 310 has the fully connected layer 313_1 and the fully connected layer 313_2, the fully connected layer 313_2 can be the output layer, and the fully connected layer 313_1 can be the intermediate layer. Similarly, when the CNN 310 has four or more fully connected layers 313, one fully connected layer 313 can be the output layer, and the remaining fully connected layers 313 can be the intermediate layers.

[0057] The configuration of the CNN 310 is not limited to the configuration of FIG. 3B. For example, each of the multiple convolution layers (convolution layer 311_1 to convolution layer 311_m) may have two or more convolution layers. That is, the number of convolution layers in the CNN 310 may be greater than the number of pooling layers. In addition, if it is desired to retain as much position information of the extracted features as possible, it is not necessary to provide a pooling layer.

[0058] By performing learning, the CNN 310 can optimize the filter values ​​of the weight filters, the weight coefficients of the fully connected layer, and the like.

[0059] The above-mentioned identification model is trained to input image data and output a defect identification result. That is, when image data is input to the input layer of the neural network, a defect identification result is output from the output layer of the neural network. For example, when the neural network has the configuration shown in FIG. 3A, when image data including a defect is input to layer 301_1, which is the input layer, a defect identification result is output from layer 301_k, which is the output layer. Also, when the neural network has the configuration shown in FIG. 3B, when image data including a defect is input to convolution layer 311_1, which is the input layer, a defect identification result is output from fully connected layer 313_3, which is the output layer.

[0060] The processing unit 102 has a function of outputting the numerical values ​​of neurons in the intermediate layer of the discrimination model. The numerical values ​​of neurons in the intermediate layer include data (also called feature amounts) corresponding to the features of image data input to the discrimination model. In other words, by outputting the numerical values ​​of neurons in the intermediate layer, it is possible to extract the feature amounts of the image data input to the discrimination model.

[0061] The number of dimensions of the extracted feature is preferably a certain number or more. If the number of dimensions is small, the accuracy of the clustering may be insufficient. On the other hand, if the number of dimensions is large, the amount of calculation for the clustering increases, the time required for the clustering increases, or the computer resources may be insufficient. For example, the number of dimensions is preferably larger than the number of dimensions of the fully connected layer that is the output layer. Specifically, the number of dimensions is preferably 32 to 1024, more preferably 32 to 256.

[0062] When the neural network has the configuration shown in FIG. 3A, for example, the numerical value of the neuron in layer 301_k-1 is output. Here, the numerical value of the neuron output from layer 301_k-1 is taken as feature amount 305. The feature amount 305 includes data corresponding to the features of the image data. Note that, although FIG. 3A shows a configuration in which the feature amount 305 is output from layer 301_k-1, this is not limiting. For example, the feature amount 305 may be output from any one of layers 301_2 to 301_k-2.

[0063] Also, when the neural network has the configuration shown in FIG. 3B, for example, the numerical value of the neuron of the fully connected layer 313_2 is output. Here, the numerical value of the neuron output from the fully connected layer 313_2 is set as the feature amount 315. The feature amount 315 includes data corresponding to the feature of the image data. Note that, in FIG. 3B, the feature amount 315 is output from the fully connected layer 313_2, but is not limited to this. For example, the feature amount 315 may be output from any one of the convolution layer 311_1 to the convolution layer 311_m, the pooling layer 312_1 to the pooling layer 312_m, and the fully connected layer 313_1. Note that the feature amount output from the convolution layer or the pooling layer may be called a feature map.

[0064] The processing unit 102 can perform processing using a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. In particular, since the processing unit 102 is configured with a neural network, it is preferable to use a GPU since it can perform processing at high speed.

[0065] The trained discrimination model may be stored in a storage unit (not shown in FIG. 1) included in the processing unit 102, or may be stored in the storage unit 101. When the trained discrimination model is stored in the storage unit 101, the training discrimination model is supplied to the processing unit 102 from the storage unit 101. Alternatively, the trained discrimination model may be supplied to the processing unit 102 via an input unit, a storage medium, communication, or the like.

[0066] [Classifier 103] The classifier 103 has a function of performing clustering (cluster analysis). Specifically, the classifier 103 has a function of performing clustering of image data based on feature amounts. The feature amounts of each of the image data 53_1 to 53_n extracted by the processing unit 102 are supplied to the classifier 103. At this time, the classifier 103 clusters the image data 53_1 to 53_n based on the feature amounts of each of the image data 53_1 to 53_n.

[0067] As a clustering (cluster analysis) method, either hierarchical or non-hierarchical methods can be used. Hierarchical methods are methods that form clusters by grouping together similar data. Examples of hierarchical methods include the shortest distance method, the longest distance method, the group average method, and Ward's method. Also, non-hierarchical methods are methods that divide the entire data so that similar data belong to the same cluster. Examples of non-hierarchical methods include the k-means method.

[0068] In this embodiment, it is preferable to use a hierarchical method as the clustering (cluster analysis) method. By using the hierarchical method, when image data including defects not confirmed so far is included, it is possible to prevent the image data from being classified into a set classified into confirmed defects. Furthermore, when the data distribution of the image data to be processed is unknown, the hierarchical method is preferable since there is no initial setting for the number of clusters. In the hierarchical method, the number of clusters is determined by setting a threshold value. For example, the threshold value may be determined to be a value with a high matching rate by preparing sample data.

[0069] When the total number of image data is large, it may be preferable to use the k-means method as a clustering method. When the total number of image data is large (for example, more than 2000), the k-means method may be able to perform clustering with fewer calculations than the hierarchical method. When using the k-means method, the number of clusters may be automatically estimated using the x-means method, or may be determined in advance by preparing sample data.

[0070] The classifier 103 may have a storage unit (not shown in FIG. 1). In this case, a program related to the clustering technique is stored in the storage unit. Alternatively, the program related to the clustering technique may be stored in the storage unit 101. When the program is stored in the storage unit 101, the program is supplied to the classifier 103 from the storage unit 101. Alternatively, the program related to the clustering technique may be supplied to the classifier 103 via an input unit, a storage medium, communication, or the like.

[0071] [Output section 104] The output unit 104 has a function of supplying the result of the clustering performed by the classifier 103. The output unit 104 may have a function of displaying the result. For example, the output unit 104 is an output device such as a display or a speaker.

[0072] The classification device 100 may have an input unit (not shown in FIG. 1). The image data may be stored in the storage unit 101 via the input unit. In addition, a discrimination model, a program related to a clustering technique, and the like may also be stored in the storage unit 101 via the input unit. Note that the image data may be stored in the storage unit 101 via a storage medium, communication, or the like.

[0073] The above is the description of the classification device 100. Note that, in FIG. 1, the classification device 100 is shown to have a configuration including a storage unit 101, a processing unit 102, a classifier 103, and an output unit 104, but is not limited to this. Modifications of the classification device 100 are described below. The modifications of the classification device described below can be appropriately combined with other classification devices shown in this specification and the like.

[0074] <Classification device 100A> Fig. 4A shows a classification device 100A, which is a modified example of the classification device 100 shown in Fig. 1. As shown in Fig. 4A, the classification device 100A has a processing unit 105 in addition to a storage unit 101, a processing unit 102, a classifier 103, and an output unit 104.

[0075] The processing unit 105 has a function of processing image data. Details of the processing of image data will be described later. The processing unit 105 may also have a function of expanding data.

[0076] In the classification device 100A, the image data 53_1 to 53_n are supplied to a processing unit 105. The image data 53_1 to 53_n are processed by the processing unit 105 to generate image data 53a_1 to 53a_n different from the image data 53_1 to 53_n. The image data 53a_1 to 53a_n generated by the processing unit 105 are supplied to a processing unit 102. The processing unit 102 extracts feature amounts of the image data 53a_1 to 53a_n. The multiple feature amounts extracted by the processing unit 102 are supplied to a classifier 103. The classifier 103 clusters the image data 53_1 to 53_n based on the multiple feature amounts.

[0077] By including the processing unit 105, the feature amount extracted by the processing unit can include more defect features contained in the image data, thereby improving the accuracy of clustering.

[0078] This concludes the description of classification device 100A.

[0079] <Classification device 100B> 4B shows a classification device 100B, which is a modified example of the classification device 100A shown in FIG 4A. As shown in FIG 4B, the classification device 100B includes a storage unit 101, a classifier 103, an output unit 104, a processing unit 105, a first processing unit 106, and a second processing unit 107.

[0080] Second processing unit 107 corresponds to processing unit 102 of classification device 100A. Therefore, for second processing unit 107, the contents described for processing unit 102 can be referred to.

[0081] The first processing unit 106 has a function of learning a discrimination model. By using the first processing unit 106, a learned discrimination model can be generated. Alternatively, by using the first processing unit 106, the discrimination model can be re-learned. The discrimination model may be re-learned, for example, after labeled image data is stored in the storage unit 101, or after labels are added to one or more pieces of unlabeled image data (image data 52_1 to image data 52_t shown in FIG. 2) stored in the storage unit 101. By re-learning the discrimination model, a discrimination model with improved discrimination accuracy can be used, and the accuracy of clustering can be improved.

[0082] This concludes the description of classification device 100B.

[0083] By utilizing the clustering results obtained by using the classification device according to one embodiment of the present invention, it is possible to reduce the time required to determine the type of defect even when the defect is identified visually for image data in which the accuracy of inference by a discrimination model is insufficient. Also, even a user who is not sufficiently skilled in identifying defects can identify defects with high accuracy while reducing the time required for the identification.

[0084] Furthermore, by clustering image data including defects using the classification device according to one embodiment of the present invention, the user can quickly and appropriately determine whether or not to perform a rework process. This is particularly effective when image data including defects is associated with the photographing position of the image data (position within the lot / substrate). Furthermore, it is possible to easily grasp defects that have never been confirmed before or defects whose type has not been identified.

[0085] As described above, by using the classification device according to one embodiment of the present invention, work efficiency can be improved.

[0086] Note that the image data clustered by the classification device of one embodiment of the present invention is not limited to image data including defects detected in a semiconductor manufacturing process. For example, the image data may include deterioration or damage to a building. Deterioration of a building includes cracks, peeling, foreign matter attachment, corrosion, and the like. By clustering the image data including deterioration or damage to a building, the building can be repaired quickly and appropriately. Note that the image data may be acquired by photographing the building using a fixed camera, a surveillance camera, or the like.

[0087] This embodiment mode can be appropriately combined with other embodiment modes, examples, etc. In addition, in the case where a plurality of configuration examples are shown in one embodiment mode in this specification, the configuration examples can be appropriately combined.

[0088] (Embodiment 2) In this embodiment, a method for classifying image data (image classification method) and a method for generating a trained discrimination model will be described with reference to Figures 5 to 8. Note that the method for classifying image data in this embodiment can be performed using the classification device described in the first embodiment.

[0089] <How to classify image data> A method for classifying image data according to one embodiment of the present invention will be described. Note that the classification of image data described in this embodiment refers to dividing a set of image data into a plurality of subsets. In other words, the classification of image data in this embodiment can be rephrased as clustering of image data.

[0090] 5 is a flowchart showing an example of a method for classifying image data, which is also a flowchart explaining the flow of processing executed by the classification device explained in the first embodiment.

[0091] The method for classifying image data includes steps S001 to S005 as shown in FIG.

[0092] [Step S001] Step S001 is a step of supplying a plurality of image data to a processing unit. The plurality of image data corresponds to the image data 53_1 to the image data 53_n described in the first embodiment. The processing unit corresponds to the processing unit 102 or the second processing unit 107 described in the first embodiment.

[0093] [Step S002] Step S002 is a process of extracting feature amounts of each of the plurality of image data in the processing unit. That is, feature amounts 62_1 to 62_n are extracted from image data 53_1 to 53_n, respectively. The feature amounts can be extracted using the trained discrimination model described in the first embodiment. That is, the feature amounts correspond to the feature amount 305 or the feature amount 315 described in the first embodiment.

[0094] The feature of each image data is given by a numerical array with u dimensions. That is, each of the feature 62_1 to feature 62_n is given by a numerical array with u dimensions. For example, as shown in FIG. 6, the feature 62_1 extracted from the image data 53_1 is composed of values ​​63_1[1] to 63_1[u]. Similarly, the feature 62_2 extracted from the image data 53_2 is composed of values ​​63_2[1] to 63_2[u]. Similarly, the feature 62_n extracted from the image data 53_n is composed of values ​​63_n[1] to 63_n[u].

[0095] [Step S003] Step S003 is a step of supplying the features (feature values ​​62_1 to 62_n) extracted by the processing unit to a classifier. The classifier corresponds to the classifier 103 described in the first embodiment.

[0096] [Step S004] Step S004 is a process of clustering the plurality of image data (image data 53_1 to image data 53_n) based on the feature amounts (feature amounts 62_1 to 62_n) by the classifier. For example, the hierarchical method described in the first embodiment can be used for clustering the image data.

[0097] [Step S005] Step S005 is a process of displaying the result of the clustering performed by the classifier on an output unit. The output unit corresponds to the output unit 104 described in the first embodiment. When a hierarchical method is used for clustering image data, for example, a dendrogram is created as a result of the clustering. Therefore, the output unit displays, for example, a dendrogram.

[0098] In this way, the image data can be classified.

[0099] The method of classifying the image data is not limited to the above-mentioned method, and another example of the method of classifying the image data is shown below.

[0100] Fig. 7 is a flowchart showing another example of a method for classifying image data. As shown in Fig. 7, the method for classifying image data may include steps S011 to S014 and steps S003 to S005. For steps S003 to S005, the above-mentioned contents can be referred to.

[0101] [Step S011] Step S011 is a process of supplying a plurality of image data to a processing unit. The plurality of image data corresponds to the image data 53_1 to the image data 53_n described in the first embodiment. The processing unit corresponds to the processing unit 105 described in the first embodiment.

[0102] [Step S012] Step S012 is a process of processing each of the plurality of image data in the processing unit. Specifically, step S012 is a process of processing the image data 53_1 to 53_n to generate the image data 53a_1 to 53a_n. More specifically, step S012 is a process of generating the image data 53a_1 to 53a_n by cutting out areas including defects from the image data 53_1 to 53_n. Note that cutting out areas including defects from the image data can be rephrased as removing at least a part of areas not including defects from the image data.

[0103] In the following, a process of processing the image data 53_1 to generate image data 53a_1 will be described.

[0104] The shape of the image data 53a_1 may be a rectangle. The rectangle has a long side with a length of a1 and a short side with a length of a2. The length a1 and the length a2 are specified so that the image data 53a_1 fits within the image data 53_1. Therefore, at least the length a1 is set to be equal to or less than the length of the long side of the image data 53_1, and the length a2 is set to be equal to or less than the length of the short side of the image data 53_1. Furthermore, the length a1 and the length a2 are specified so that the defect fits within the image data 53a_1.

[0105] The ratio of the length a1 to the length a2 is preferably equal to the ratio of the length of the long side of the image data 53_1 to the length of the short side of the image data 53_1. When the ratio of the length of the long side of the image data 53_1 to the length of the short side of the image data 53_1 is 4:3, for example, the length a1 may be set to 640 pixels and the length a2 may be set to 480 pixels.

[0106] The ratio of the length a1 to the length a2 does not necessarily have to match the ratio of the long side of the image data 53_1 to the short side of the image data 53_1. For example, the ratio of the length a1 to the length a2 of the rectangle may be different from the ratio of the long side of the image data 53_1 to the short side of the image data 53_1. Alternatively, the shape of the image data 53a_1 may be a square.

[0107] Alternatively, the long side of the rectangle may be parallel to the short side of the image data 53_1, or the short side of the rectangle may be parallel to the long side of the image data 53_1. Alternatively, the long side of the rectangle may not be parallel or perpendicular to the long side of the image data 53_1.

[0108] The position of the image data 53a_1 is determined so that the image data 53a_1 fits within the image data 53_1. The position of the image data 53a_1 may be determined based on the center of gravity of the image data 53a_1 or on one of the vertices of the image data 53a_1. For example, the center of gravity of the image data 53a_1 is determined by a uniform random number. A uniform random number is a random number that follows a continuous uniform distribution in which all real numbers appear with the same probability within a specified interval or range.

[0109] In the above, the process of determining the position of the image data 53a_1 after specifying the length a1 and the length a2 has been described, but the process is not limited to this. After specifying the position of the image data 53a_1, the length a1 and the length a2 may be determined so that the image data 53a_1 fits into the image data 53_1. Alternatively, the position of the image data 53a_1 and the length a1 and the length a2 may be determined simultaneously so that the image data 53a_1 fits into the image data 53_1.

[0110] It is preferable that the length of the long side and the length of the short side of the image data 53a_1 are equal to the length of the long side and the short side of the other image data 53a_1, respectively. This can improve the accuracy of identifying defects as described above.

[0111] The above is a description of the process of processing the image data 53_1 to generate the image data 53a_1. Note that the image data 53a_2 to 53a_n can be generated by carrying out similar processes.

[0112] Step S012 may be performed by a user or automatically using a classification device. When using a classification device, for example, a difference between image data including defects and image data not including defects may be obtained, and an area with a large difference and its surrounding area may be cut out.

[0113] From the above, the image data 53a_1 to 53a_n can be generated. By cutting out the area including the defect from each of the image data 53_1 to 53_n, the ratio of the area (area) occupied by the part to be identified to the area (area) of the entire image data can be increased. This allows the feature amount extracted by the processing unit to include more of the features of the defect contained in the image data. Therefore, the accuracy of clustering can be improved. Note that the image data 53a_1 to 53a_n are assigned the labels assigned to the image data 53_1 to 53_n, respectively.

[0114] [Step S013] Step S013 is a step of supplying the image data 53a_1 to 53a_n to a processing unit. This processing unit corresponds to the processing unit 102 or the second processing unit 107 described in the first embodiment.

[0115] [Step S014] Step S014 is a process of extracting feature amounts of each of the image data 53a_1 to 53a_n in the processing unit. That is, feature amounts 62_1 to 62_n are extracted from the image data 53a_1 to 53a_n, respectively. Since the image data 53a_1 to 53a_n are created by processing the image data 53_1 to 53_n, respectively, the feature amounts 62_1 to 62_n can be rephrased as feature amounts of the image data 53_1 to 53_n, respectively. The feature amounts can be extracted using the trained discrimination model described in the first embodiment.

[0116] After step S014, steps S003, S004, and S005 are performed in this order. In this manner, the image data can be classified.

[0117] The above is an explanation of the method for classifying image data. By clustering image data including defects, the time required for a user to identify defects can be shortened. Furthermore, even a user who is not sufficiently skilled in identifying defects can identify defects with high accuracy. Furthermore, a user can quickly and appropriately decide whether or not to perform a rework process. Furthermore, defects that have not been confirmed before, defects whose types have not been identified, etc. can be easily grasped.

[0118] <How to generate a trained discrimination model> Here, a method for generating a trained discriminative model according to one aspect of the present invention will be described. The method for generating a trained discriminative model can be rephrased as a method for training a discriminative model. The method for generating a trained discriminative model can be rephrased as a method for retraining a trained discriminative model.

[0119] Fig. 8 is a flowchart illustrating an example of a method for generating a trained discrimination model. Fig. 8 is also a flow diagram illustrating the flow of processing executed by the classification device described in the first embodiment.

[0120] The method for generating a trained discrimination model includes steps S021 to S026, as shown in FIG.

[0121] [Step S021] Step S021 is a process of supplying a plurality of image data to a processing unit. The plurality of image data are image data that can be used as learning data, verification data, or test data. The processing unit corresponds to the processing unit 105 described in the first embodiment.

[0122] Each of the plurality of image data includes a defect. Furthermore, the image data including the defect is assigned a label corresponding to the defect included in the image data. That is, the plurality of image data includes some or all of the image data 51_1 to 51_s. Here, the plurality of image data are described as image data 54_1 to image data 54_p (p is an integer between 2 and s).

[0123] [Step S022] Step S022 is a process in which the processing unit processes the plurality of image data to generate a plurality of image data different from the plurality of image data. Specifically, this is a process in which the image data 54_1 to 54_p are processed to generate the image data 54a_1 to 54a_p. More specifically, this is a process in which the image data 54a_1 to 54a_p are generated by cutting out areas including defects from the image data 54_1 to 54_p. The explanation of step S012 can be referred to for this process.

[0124] It is preferable that the area of ​​the image data to be identified occupies a large proportion of the area of ​​the entire image data. For example, in the case of a pattern inspection result image, it is effective to cut out defective parts. By the above processing, the feature amount extracted by the processing unit can include more of the defective features contained in the image data. Therefore, the accuracy of clustering can be improved. The image data 54a_1 to 54a_p are assigned the labels assigned to the image data 54_1 to 54_p, respectively.

[0125] [Step S023] Step S023 is a process of performing data extension in the processing unit. Methods of data extension include rotation, inversion, noise addition, blurring, gamma conversion, and the like for image data. For data extension, some or all of the image data 54a_1 to 54a_p may be used. By data extension, q (q is an integer equal to or greater than 1) pieces of image data (image data 54a_p+1 to image data 54a_p+q) are generated.

[0126] It is preferable that the number of image data created for each defect is approximately equal. For example, it is preferable that the number of image data to which a label corresponding to a foreign substance is assigned, the number of image data to which a label corresponding to a film hole is assigned, the number of image data to which a label corresponding to a pattern defect is assigned, etc. are approximately equal. This makes it possible to suppress overfitting (overlearning) for a specific defect.

[0127] The method of data extension, the number of image data generated by data extension, etc. may be selected randomly or may be specified by a user. Also, the classification device may automatically select the method of data extension based on the labels assigned to the image data 54a_1 to 54a_p.

[0128] There are cases where data expansion is not necessary. For example, this is the case when sufficient training data is available to generate a classification model capable of classification with high accuracy. In this case, step S023 may be omitted.

[0129] By performing steps S022 and S023, a learning dataset can be generated. Input data for the learning dataset is p pieces of image data (image data 54a_1 to image data 54a_p) generated in step S022 and q pieces of image data (image data 54a_p+1 to image data 54a_p+q) generated in step S023.

[0130] The correct labels of the learning dataset are the labels assigned to each of the image data 54a_1 through 54a_p+q.

[0131] From the above, the learning dataset is composed of (p+q) pieces of image data and labels assigned to the image data.

[0132] [Step S024] Step S024 is a process of supplying the learning data set generated by the processing unit to a processing unit. The learning data set includes image data 54a_1 to 54a_p+q. The processing unit corresponds to the first processing unit 106 described in the first embodiment.

[0133] [Step S025] Step S025 is a step of causing the processing unit to train a discrimination model using the training data set.

[0134] When training the discriminative model, it is advisable to divide the training data set into training data, validation data, and test data. For example, the discriminative model is trained using the training data, the validation data is used to evaluate the training results, and the test data is used to evaluate the trained discriminative model. This makes it possible to confirm the accuracy of the trained discriminative model. Hereinafter, the ratio of the number of correct discrimination results to the number of test data may be referred to as the accuracy rate.

[0135] The learning data is composed of a part of the image data 54a_1 to 54a_p+q. The verification data is composed of a part of the image data that is not used in the learning data. The test data is composed of image data that is not used in the learning data and the verification data.

[0136] Methods for dividing a learning data set into learning data, validation data, and test data include, for example, the hold-out method, the cross-validation method, and the leave-one-out method.

[0137] The learning may be terminated when a predetermined number of trials is reached. Alternatively, the learning may be terminated when the accuracy rate exceeds a predetermined threshold. Alternatively, the learning may be terminated when the accuracy rate is saturated to a certain extent. It is preferable to prepare a constant in advance for the number of trials or the threshold. Alternatively, the user may specify the timing for termination while attempting the learning.

[0138] Through the above learning, a trained discrimination model is generated.

[0139] [Step S026] Step S026 is a process of storing the trained discrimination model generated in step S025 in a storage unit. The storage unit is the storage unit 101 described in embodiment 1. Note that the storage unit may be a storage unit included in the first processing unit 106, a storage unit included in the processing unit 102 or the second processing unit 107, a storage medium connected to the classification device, or the like.

[0140] The above is an example of a method for generating a trained discrimination model. By training the discrimination model based on the training data set, a discrimination model with high defect discrimination accuracy can be generated.

[0141] According to one aspect of the present invention, a method for classifying image data can be provided.

[0142] This embodiment mode can be appropriately combined with other embodiment modes, examples, etc. In addition, in the case where a plurality of configuration examples are shown in one embodiment mode in this specification, the configuration examples can be appropriately combined.

[0143] (Embodiment 3) In this embodiment, a classification device of one embodiment of the present invention will be described with reference to FIGS.

[0144] <Classification device configuration example 1> 9 shows a block diagram of the classification device 200. In the drawings attached to this specification, the components are classified by function and shown as independent blocks in the block diagram, but it is difficult to completely separate the components by function in reality, and one component may be involved in multiple functions. Also, one function may be involved in multiple components, and for example, the processing performed by the processing unit 202 may be executed by different servers depending on the processing.

[0145] The classification device 200 shown in FIG. 9 includes an input unit 201, a processing unit 202, a storage unit 203, a database 204, a display unit 205, and a transmission path 206.

[0146] [Input section 201] Image data is supplied to the input unit 201 from outside the classification device 200. The image data corresponds to the labeled image data and the unlabeled image data described in the above embodiment. The image data supplied to the input unit 201 is supplied to the processing unit 202, the storage unit 203, or the database 204 via the transmission path 206.

[0147] [Processing section 202] The processing unit 202 has a function of performing processing using data supplied from the input unit 201, the storage unit 203, the database 204, etc. The processing unit 202 can supply the processing results to the storage unit 203, the database 204, the display unit 205, etc.

[0148] The processing unit 202 includes the processing unit 102 or the second processing unit 107, and the classifier 103, which are described in the previous embodiment. That is, the processing unit 202 has a function of performing processing using a trained discrimination model, a function of performing clustering, and the like. The processing unit 202 may also include the processing unit 105 and the first processing unit 106, which are described in the previous embodiment. In this case, the processing unit 202 has a function of processing image data, a function of performing data expansion, a function of generating a learning dataset, a function of training a discrimination model, and the like.

[0149] The processing unit 202 may be a transistor having a metal oxide in a channel formation region. Since the off-state current of the transistor is extremely small, the transistor can be used as a switch for holding charge (data) flowing into a capacitor element functioning as a memory element, thereby ensuring a long data holding period. By using this characteristic in at least one of the register and the cache memory of the processing unit 202, the processing unit 202 can be operated only when necessary, and in other cases, the processing unit 202 can be turned off by saving information of the immediately preceding process in the memory element. In other words, normally-off computing is possible, and the power consumption of the classification device can be reduced.

[0150] Note that in this specification and the like, a transistor whose channel formation region is formed using an oxide semiconductor is referred to as an oxide semiconductor transistor (OS transistor).The channel formation region of an OS transistor preferably contains a metal oxide.

[0151] The metal oxide in the channel formation region preferably contains indium (In). When the metal oxide in the channel formation region contains indium, the carrier mobility (electron mobility) of the OS transistor is increased. The metal oxide in the channel formation region preferably contains an element M. The element M is preferably aluminum (Al), gallium (Ga), or tin (Sn). Other elements applicable to the element M include boron (B), titanium (Ti), iron (Fe), nickel (Ni), germanium (Ge), yttrium (Y), zirconium (Zr), molybdenum (Mo), lanthanum (La), cerium (Ce), neodymium (Nd), hafnium (Hf), tantalum (Ta), and tungsten (W). However, the element M may be a combination of a plurality of the above elements. The element M is, for example, an element having a high bond energy with oxygen. The element M is, for example, an element having a higher bond energy with oxygen than indium. The metal oxide in the channel formation region preferably contains zinc (Zn), since zinc-containing metal oxides may tend to crystallize.

[0152] The metal oxide contained in the channel formation region is not limited to a metal oxide containing indium. The metal oxide contained in the channel formation region may be, for example, a metal oxide containing zinc but not indium, such as zinc tin oxide or gallium tin oxide, a metal oxide containing gallium, or a metal oxide containing tin.

[0153] A transistor containing silicon in a channel formation region (Si transistor) may be used in the processing section 202. A transistor containing a semiconductor material having a band gap, such as graphene, silicene, or chalcogenide (transition metal chalcogenide), may be used in a channel formation region.

[0154] The processing portion 202 may include a combination of a transistor including an oxide semiconductor in a channel formation region and a transistor including silicon in a channel formation region.

[0155] The processing unit 202 includes, for example, an arithmetic circuit or a central processing unit (CPU: Central Processing Unit).

[0156] The processing unit 202 may have a microprocessor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The microprocessor may be realized by a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or an FPAA (Field Programmable Analog Array). The processing unit 202 can perform various data processing and program control by interpreting and executing instructions from various programs using the processor. Programs that can be executed by the processor are stored in at least one of a memory area of ​​the processor and the storage unit 203.

[0157] The processing unit 202 may include a main memory. The main memory includes at least one of a volatile memory such as a RAM and a non-volatile memory such as a ROM.

[0158] The RAM may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM), and a virtual memory space is allocated and used as a working space for the processing unit 202. The operating system, application programs, program modules, program data, lookup tables, and the like stored in the storage unit 203 are loaded into the RAM for execution. The data, programs, and program modules loaded into the RAM are each directly accessed and operated by the processing unit 202.

[0159] ROM can store BIOS (Basic Input / Output System) and firmware that do not require rewriting. Examples of ROM include mask ROM, OTPROM (One Time Programmable Read Only Memory), EPROM (Erasable Programmable Read Only Memory), etc. Examples of EPROM include UV-EPROM (Ultra-Violet Erasable Programmable Read Only Memory), which allows the erasure of stored data by exposure to ultraviolet light, EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory, etc.

[0160] In addition, a product-sum operation is performed in a neural network. When the product-sum operation is performed by hardware, it is preferable that the processing unit 202 has a product-sum operation circuit. As the product-sum operation circuit, a digital circuit or an analog circuit may be used. When an analog circuit is used for the product-sum operation circuit, it is possible to reduce the circuit scale of the product-sum operation circuit or to reduce the number of accesses to the memory, thereby improving the processing speed and reducing power consumption. In addition, the product-sum operation may be performed on software using a program.

[0161] The product-sum circuit may be formed using either Si transistors or OS transistors. In particular, OS transistors have an extremely small off-state current and are therefore suitable as transistors forming the analog memory of the product-sum circuit. Note that the product-sum circuit may be formed using both Si transistors and OS transistors.

[0162] [Storage section 203] The storage unit 203 has a function of storing a program executed by the processing unit 202. The storage unit 203 has a function of storing, for example, a discrimination model, a program related to a clustering technique, etc. The storage unit 203 may also have a function of storing image data supplied to the input unit 201, etc.

[0163] The storage unit 203 has at least one of a volatile memory and a non-volatile memory. The storage unit 203 may have a volatile memory such as a DRAM or an SRAM. The storage unit 203 may have a non-volatile memory such as a ReRAM (Resistive Random Access Memory, also called a resistance change type memory), a PRAM (Phase change Random Access Memory), a FeRAM (Ferroelectric Random Access Memory), an MRAM (Magnetoresistive Random Access Memory, also called a magnetoresistive type memory), or a flash memory. The storage unit 203 may also have a recording media drive such as a hard disk drive (HDD) and a solid state drive (SSD).

[0164] [Database 204] The classification device 200 may include a database 204. For example, the database 204 has a function of storing the image data. Note that the database 204 may store image data related to the learning dataset generated by the processing unit 202, a trained discrimination model, a clustering result, and the like.

[0165] It is to be noted that the storage unit 203 and the database 204 do not have to be separated from each other. For example, the classification device 200 may have a storage unit having the functions of both the storage unit 203 and the database 204.

[0166] The memories included in the processing unit 202, the storage unit 203, and the database 204 can each be considered as an example of a non-transitory computer-readable storage medium.

[0167] [Display section 205] The display unit 205 has a function of displaying the processing results in the processing unit 202. For example, the display unit 205 has a function of displaying the results of clustering.

[0168] The classification device 200 may have an output unit. The output unit has a function of supplying data to an outside device.

[0169] [Transmission Line 206] The transmission path 206 has a function of transmitting various data. Data can be transmitted and received between the input unit 201, the processing unit 202, the storage unit 203, the database 204, and the display unit 205 via the transmission path 206. For example, data such as image data and a trained discrimination model is transmitted and received via the transmission path 206.

[0170] <Classification device configuration example 2> 10 shows a block diagram of a classification device 210. The classification device 210 includes a server 220 and a terminal 230 (such as a personal computer).

[0171] The server 220 includes a processing unit 202, a transmission path 212, a storage unit 213, and a communication unit 217a. Although not shown in FIG. 10, the server 220 may further include an input unit, an output unit, and the like.

[0172] The terminal 230 includes an input unit 201, a storage unit 203, a display unit 205, a transmission path 216, a communication unit 217b, and a processing unit 218. Although not shown in FIG.

[0173] The image data, the identification model, and the like received by the communication unit 217a are stored in the storage unit 213 via the transmission path 212. Alternatively, the image data, the identification model, and the like may be supplied directly to the processing unit 202 from the communication unit 217a.

[0174] As described in the above embodiment, the learning of the discriminative model requires high processing power. The processing unit 202 of the server 220 has a higher processing power than the processing unit 218 of the terminal 230. Therefore, it is preferable that the learning of the discriminative model is performed by the processing unit 202.

[0175] Then, a trained classification model is created by the processing unit 202. The trained classification model is supplied from the processing unit 202 to the communication unit 217a via the transmission path 212 or directly. The trained classification model is transmitted from the communication unit 217a of the server 220 to the communication unit 217b of the terminal 230 and stored in the storage unit 203. Alternatively, the trained classification model may be stored in the storage unit 213 via the transmission path 212.

[0176] [Transmission path 212 and transmission path 216] The transmission paths 212 and 216 have a function of transmitting data. Data can be transmitted and received between the processing unit 202, the storage unit 213, and the communication unit 217a via the transmission path 212. Data can be transmitted and received between the input unit 201, the storage unit 203, the display unit 205, the communication unit 217b, and the processing unit 218 via the transmission path 216.

[0177] [Processing unit 202 and processing unit 218] The processing unit 202 has a function of performing processing using data supplied from the storage unit 213, the communication unit 217a, etc. The processing unit 218 has a function of performing processing using data supplied from the input unit 201, the storage unit 203, the display unit 205, the communication unit 217b, etc. For the processing unit 202 and the processing unit 218, refer to the description of the processing unit 202. It is preferable that the processing unit 202 has a higher processing capacity than the processing unit 218.

[0178] [Storage section 203] The storage unit 203 has a function of storing a program executed by the processing unit 218. The storage unit 203 also has a function of storing the trained discrimination model generated by the processing unit 202, the clustering result generated by the processing unit 218, data input to the communication unit 217b, data input to the input unit 201, and the like.

[0179] [Storage section 213] The storage unit 213 has a function of storing a program executed by the processing unit 202. The storage unit 213 also has a function of storing a discrimination model, data input to the communication unit 217a, etc. The storage unit 213 can refer to the explanation of the storage unit 203.

[0180] [Communication Unit 217a and Communication Unit 217b] Using the communication units 217a and 217b, data can be transmitted and received between the server 220 and the terminal 230. A hub, a router, a modem, or the like can be used as the communication units 217a and 217b. Data can be transmitted and received using either a wired connection or wirelessly (for example, radio waves, infrared rays, etc.).

[0181] In addition, communication between server 220 and terminal 230 may be performed by connecting to a computer network such as the Internet, which is the basis of the World Wide Web (WWW), an intranet, an extranet, a Personal Area Network (PAN), a Local Area Network (LAN), a Campus Area Network (CAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), or a Global Area Network (GAN).

[0182] This embodiment mode can be combined as appropriate with other embodiment modes or examples.

[0183] (Embodiment 4) In this embodiment, a pattern inspection device according to one embodiment of the present invention will be described with reference to Fig. 11. The pattern inspection device of this embodiment includes the classification device described in the previous embodiment.

[0184] Fig. 11 is a diagram showing the configuration of a pattern inspection device 400. As shown in Fig. 11, the pattern inspection device 400 includes an imaging device 401, an inspection device 402, and a classification device 403. The classification device 403 corresponds to the classification device 100 described in the previous embodiment.

[0185] The imaging device 401 has a function of capturing an image of a semiconductor element in the middle of a manufacturing process or a semiconductor element whose manufacturing process has been completed. An example of the imaging device 401 is a camera. By capturing an image of the semiconductor element, image data is obtained that has not been determined to be defective. In other words, the image data is image data that can be a target for classification or clustering.

[0186] The inspection device 402 has a function of determining whether or not a defect is included in the image data acquired by using the imaging device 401. This makes it possible to determine whether or not a defect is included in the image data.

[0187] The determination as to whether or not a defect is included is performed by comparing the image data to be determined with the image data acquired immediately before. Here, the semiconductor element included in the image data to be determined is different from the semiconductor element included in the image data acquired immediately before. For example, first, the difference between the image data to be determined and the image data acquired immediately before is obtained. Then, the determination as to whether or not a defect is included may be performed based on the difference.

[0188] Machine learning may be used to determine whether a defect is included. The number of image data to be used to determine whether a defect is included can be enormous. Therefore, by using machine learning, the time required for the determination can be reduced.

[0189] For example, a method similar to that for detecting an abnormal portion can be used to determine whether a defect is included. For detecting an abnormal portion, unsupervised learning may be used. Therefore, it is preferable to use unsupervised learning for this determination. By using unsupervised learning, even if the number of image data containing defects is small, it is possible to accurately determine whether a defect is included.

[0190] In addition, supervised learning may be used to detect an abnormal portion. Therefore, supervised learning may be used for the judgment. By using supervised learning, it is possible to accurately judge whether or not a defect is included.

[0191] It is preferable to use a neural network (particularly, deep learning) for the above machine learning.

[0192] The image data determined to include a defect is subjected to classification or clustering. In other words, the image data may become one of the multiple image data 50 described in the previous embodiment. The image data is supplied to a classification device 403.

[0193] The pattern inspection device 400 has an imaging device 401, an inspection device 402, and a classification device 403, and is therefore capable of not only clustering image data and / or generating a trained discrimination model, but also acquiring image data and determining whether or not there is a defect.

[0194] The inspection device 402 may be provided in a server different from the server provided with the classification device 403. Alternatively, the inspection device 402 may be provided in a server provided with the classification device 403. Alternatively, a part of the functions of the inspection device 402 and a part of the functions of the classification device 403 may be provided in a server, and another part of the functions of the inspection device 402 and another part of the functions of the classification device 403 may be provided in a server different from the server.

[0195] The above is a description of the configuration of the pattern inspection device 400. By using the pattern inspection device according to one embodiment of the present invention, a series of processes from acquisition of image data to clustering of the image data can be performed with high efficiency. In addition, the series of processes can be performed fully automatically.

[0196] According to one aspect of the present invention, a novel pattern inspection apparatus can be provided.

[0197] This embodiment mode can be combined as appropriate with other embodiment modes or examples. EXAMPLES

[0198] In this embodiment, image data including defects was clustered using a trained discrimination model. The results of the clustering of the image data will be described with reference to Figs. 12 and 13.

[0199] In this embodiment, a CNN was used as the discrimination model. The CNN is composed of seven convolution layers, six pooling layers, and five fully connected layers (fully connected layers 913_1 to 913_5). A neuron provided in the fully connected layer 913_1 is connected to a neuron provided in one pooling layer and a neuron provided in the fully connected layer 913_2. A neuron provided in the fully connected layer 913_2 is connected to a neuron provided in the fully connected layer 913_1 and a neuron provided in the fully connected layer 913_3. A neuron provided in the fully connected layer 913_3 is connected to a neuron provided in the fully connected layer 913_2 and a neuron provided in the fully connected layer 913_4. A neuron provided in the fully connected layer 913_4 is connected to a neuron provided in the fully connected layer 913_3 and a neuron provided in the fully connected layer 913_5. The fully connected layer 913_5 is set as an output layer. Moreover, the fully connected layers 913_1 to 913_4 are included in the intermediate layers. For example, the fact that a neuron provided in the first fully connected layer is connected to a neuron provided in the second fully connected layer can be rephrased as the first fully connected layer being connected to the second fully connected layer.

[0200] In this embodiment, the numerical values ​​of the neurons in the fully connected layer 913_2 are used as the feature quantities of the image data input to the input layer. The number of dimensions of the feature quantities is set to 64.

[0201] In this embodiment, clustering was performed on image data including 344 defects. Specifically, the above-mentioned discrimination model was used to obtain feature values ​​for each of the image data including 344 defects, and cluster analysis was performed based on the feature values. A hierarchical method was used for the cluster analysis.

[0202] The results of the clustering are shown in Figure 12. Figure 12 is a dendrogram that explains the results of the clustering. The vertical axis represents the distance between clusters. Note that the image data is appropriately arranged on the horizontal axis so that the image data are grouped together for each cluster.

[0203] In the dendrogram shown in Fig. 12, 20 clusters were obtained by setting the threshold value of the distance between clusters to 34.1 (the dashed line shown in Fig. 12). A portion of image data included in one of the 20 clusters is shown in Figs. 13A to 13D. It can be seen that each of the image data shown in Figs. 13A to 13D includes the same type of pattern defect. In other words, it can be seen that the cluster in question is a cluster related to a portion of the pattern defect.

[0204] As described above, by using the method shown in this embodiment, it is possible to perform clustering in which similar defects are grouped into clusters.

[0205] This embodiment can be implemented in appropriate combination with the configurations described in the other embodiment modes. [Explanation of symbols]

[0206] 50: image data, 51_s: image data, 51_1: image data, 51_2: image data, 52_t: image data, 52_1: image data, 53_n: image data, 53_1: image data, 53_2: image data, 53a_n: image data, 53a_1: image data, 53a_2: image data, 54_p: image data, 54_1: image data, 54a_p: image data, 54a_1: image data, 61A: label, 6 1F: label, 62_n: feature, 62_1: feature, 62_2: feature, 63_n: value, 63_1: value, 63_2: value, 100: classification device, 100A: classification device, 100B: classification device, 101: memory unit, 102: processing unit, 103: classifier, 104: output unit, 105: processing unit, 106: processing unit, 107: processing unit, 200: classification device, 201: input unit, 202: processing unit, 203: memory unit, 204: database, 205: table display unit, 206: transmission path, 210: classification device, 212: transmission path, 213: storage unit, 216: transmission path, 217a: communication unit, 217b: communication unit, 218: processing unit, 220: server, 230: terminal, 300: neural network, 301_k: layer, 301_k-1: layer, 301_k-2: layer, 301_1: layer, 301_2: layer, 301_3: layer, 305: feature, 310: CNN, 311_m: layer, 311_1: layer, 311_ 2: layer, 312_m: pooling layer, 312_m-1: pooling layer, 312_1: pooling layer, 313: fully connected layer, 313_1: fully connected layer, 313_2: fully connected layer, 313_3: fully connected layer, 315: feature, 400: pattern inspection device, 401: imaging device, 402: inspection device, 403: classification device, 913_1: fully connected layer, 913_2: fully connected layer, 913_3: fully connected layer, 913_4: fully connected layer, 913_5: fully connected layer

Claims

[Claim 1] A pattern inspection device having an imaging unit, an inspection device, and a classification device, the inspection device has a function of determining whether or not a defect is included based on a difference between two images of different elements captured by the imaging unit, The classification device includes a storage unit, a processing unit, and a classifier. The storage unit stores a plurality of image data determined to include defects and a discrimination model, the discriminative model is a convolutional neural network; The processing unit is supplied with the plurality of image data, The processing unit has a function of outputting a feature amount of image data based on the discrimination model, The classifier is supplied with the feature values ​​output from the processing unit, The classifier has a function of performing clustering of the image data based on the feature amount.

Images