Image classification methods

Abstract

The present invention provides a novel classification device, an image classification method, and a pattern inspection device. [Solution] The classification device 100 includes a storage unit 101, a processing unit 102, a classifier 103, and an output unit 104. The storage unit stores a plurality of image data and an identification model. Each of the plurality of image data is image data that has been determined to contain defects. The identification model includes an input layer, an intermediate layer, and an output layer. The processing unit is supplied with the first to nth image data (n is an integer of 2 or more) from the plurality of image data, and the processing unit outputs the first to nth feature quantities, which are the feature quantities of each of the first to nth image data, based on the identification model. The feature quantities output from the processing unit are numerical values ​​of neurons in the intermediate layer. The classifier is supplied with the first to nth feature quantities output from the processing unit, and the classifier performs clustering of the first to nth image data based on the first to nth feature quantities.

Description

Technical Field

[0001] One aspect of the present invention relates to a classification device. Another aspect of the present invention relates to an image classification method. Another aspect of the present invention relates to a pattern inspection device. And.

Background Art

[0002] Appearance inspection is cited as a means for detecting defects in semiconductor manufacturing processes. As a device (appearance inspection device) for automatically performing appearance inspection, for example, a pattern inspection device is cited. For the image acquired using the appearance inspection device, detection and identification of defects are performed. When detection and identification of defects are performed visually, there may be individual differences in the accuracy of detection and identification of defects. Also, when the number of image data is large, it takes a huge amount of time to detect and identify defects. For example, a pattern inspection device. Detection and identification of defects are performed on the image acquired using the appearance inspection device. When detection and identification of defects are performed visually, there may be individual differences in the accuracy of detection and identification of defects. Also, when the number of image data is large, it takes a huge amount of time to detect and identify defects. And there is a case where individual differences occur in the accuracy of detection and identification of defects. Also, when the number of image data is large, it takes a huge amount of time to detect and identify defects.

[0003] In recent years, techniques for identifying defects (defects) using neural networks have been reported. For example, in Patent Document 1, a defect type determination device that identifies defects is disclosed using a neural processing unit that is learned to input information related to defects and output an identification result of the defects. For example, using a neural processing unit that is learned to input information related to defects and output an identification result of the defects, a defect type determination device that identifies defects is disclosed. Is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] To train a neural processing unit, the association between information related to defects and the type of defects This needs to be done in advance. In Patent Document 1, information regarding defects is the area of ​​the defect, the defect These include the shape and location of defects, and are acquired using an image processing device. However, When multiple defects overlap or reside in the same area, the accuracy of defect information decreases. There is a risk of that happening.

[0006] If a defect is detected, it is necessary to decide whether or not to perform a rework process. The decision of whether or not to perform the procedure depends not only on the type of defect, but also on the frequency of each defect, and the lot and board size. This needs to be done from a broad perspective, including the distribution of defects.

[0007] Therefore, one aspect of the present invention aims to provide a novel classification device. One aspect of the present invention aims to provide a novel image classification method. One aspect of the invention aims to provide a novel pattern inspection device.

[0008] Furthermore, the description of these problems does not preclude the existence of other problems. One approach does not require that all of these issues be resolved. The title will become clear from the description in the specification, drawings, claims, etc. It is possible to extract other issues from the descriptions in the drawings, claims, etc. [Means for solving the problem]

[0009] An aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. It stores multiple image data and a classification model. The classification model has multiple convolutions A layer, multiple pooling layers, a first fully connected layer, a second fully connected layer, a third fully connected layer It has a fourth fully coupled layer and a fifth fully coupled layer. The fifth fully coupled layer is the output layer. The fourth fully connected layer is connected to the fifth fully connected layer. The third fully connected layer is connected to the fourth fully connected layer. The second fully connected layer is connected to the third fully connected layer. The first fully connected layer is , connected to the second fully connected layer. The processing unit receives, among multiple image data, the first to the nth ( Image data (where n is an integer greater than or equal to 2) is supplied. The processing unit is based on the identification model. A function that outputs the k-th feature of the k-th image data (where k is an integer between 1 and n, inclusive). It has the following features. The feature quantities output from the processing unit are the numerical values ​​of the neurons in the first fully connected layer. The number of neurons in the second fully connected layer, or the number of neurons in the third fully connected layer. This is the numerical value. The classifier is supplied with the first to nth feature quantities output from the processing unit. The classifier performs clustering of the first to nth image data based on the first to nth feature quantities. It has the function to perform this task.

[0010] Another aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. The memory section stores multiple image data and a recognition model. The recognition model is composed of multiple tatami mats. A filling layer, multiple pooling layers, a first fully bonded layer, a second fully bonded layer, and a third fully bonded layer. It has a coupling layer and a third fully coupled layer which is the output layer. The second fully coupled layer is the third fully coupled layer It is connected to the layer. The first fully connected layer is connected to the second fully connected layer. Multiple processing units are located in the processing unit. Of the image data, the first to the nth (where n is an integer greater than or equal to 2) image data is supplied. The processing unit, based on the identification model, processes the k-th (where k is an integer between 1 and n) image data. It has the function of outputting the kth feature of the data. The feature output from the processing unit is the first The numerical value of the neurons in the fully connected layer or the numerical value of the neurons in the second fully connected layer is. The first to nth feature amounts output from the processing unit are supplied to the classifier. The classifier performs clustering of the first to nth image data based on the first to nth feature amounts and has a function.

[0011] Another aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. In the storage unit, a plurality of image data and an identification model are stored. The identification model has a plurality of convolutional layers, a plurality of pooling layers, and a fully connected layer. The fully connected layer is an output layer . Among the plurality of image data, the first to nth (n is an integer of 2 or more) image data are supplied to the processing unit. The processing unit has a function of outputting the kth (k is an integer of 1 or more and n or less ) feature amount of the kth image data based on the identification model. The feature amount output from the processing unit is the numerical value of the neurons in any one of the plurality of convolutional layers or the numerical value of the neurons in any one of the plurality of pooling layers. The first to nth feature amounts output from the processing unit are supplied to the classifier. The classifier performs clustering of the first to nth image data based on the first to nth feature amounts. and has a function.

[0012] In the above classification device, each of the plurality of image data is preferably image data determined to include defects .

[0013] Another aspect of the present invention is a classification device having a storage unit, a processing unit, and a classifier. In the storage unit, a plurality of image data and an identification model are stored. Each of the plurality of image data These are image data that have been identified as containing defects. The discrimination model consists of an input layer and an intermediate layer. It has a layer and an output layer. The processing unit processes the first to nth (n is 2) of the multiple image data. Image data (which is an integer greater than or equal to the above) is supplied. The processing unit, based on the identification model, determines the k It has the function of outputting the k-th feature of image data (where k is an integer between 1 and n, inclusive). The feature quantities output from the processing unit are numerical values ​​of neurons in the hidden layer. The first to nth feature quantities output from the processing unit are supplied. The classifier uses the first to nth feature quantities. It has the function of clustering the first to nth image data based on the features.

[0014] Another aspect of the present invention is a classification device having a storage unit, a processing unit, a processing unit, and a classifier. The memory unit stores multiple image data and an identification model. Each of the data is image data that has been identified as containing defects. The discrimination model is based on the input. It has a layer, an intermediate layer, and an output layer. The processing unit processes a plurality of image data, from the first to the second Image data n (where n is an integer greater than or equal to 2) is supplied. The processing unit processes the kth (where k is greater than or equal to 1). By removing a portion of the image data (which is an integer less than or equal to n), the (n+k)th image data is obtained. It has the function of generating. The processing unit is supplied with the (n+1) to (2n)th image data. The processing unit, based on the identification model, determines the kth feature of the (n+k)th image data. It has the function of outputting. The feature quantity output from the processing unit is the number of neurons in the hidden layer. This is a value. The classifier is supplied with the first to nth feature quantities output from the processing unit. Classifier This involves clustering the first to nth image data based on the first to nth features. It has the function of [doing something].

[0015] In the above classification device, the number of dimensions of the feature quantities output from the processing unit is between 32 and 256. It is preferable that this be the case.

[0016] In the above classification device, the identification model applies to image data that has been determined to contain defects. Supervised learning is performed so that the type of defect can be inferred, and clustering is performed. It is preferable that a hierarchical approach be used.

[0017] Furthermore, the above-mentioned classification device has an output unit in addition to the storage unit, processing unit, and classifier. The output section preferably has a function to display the clustering results performed by the classifier. It's nice.

[0018] Another aspect of the present invention is a pattern having the above-mentioned classification device, imaging unit, and inspection device. This is a surface inspection device. The imaging unit has the function of imaging the object to be inspected. The inspection device is an imaging unit It has a function to determine whether or not defects are included in the image data acquired by imaging. do.

[0019] Another aspect of the present invention is that in the first step, the first to n (where n is an integer of 2 or more) In the second step, the image data of a certain object is supplied to the processing unit, and in the second step, the processing unit is used to identify Based on the model, the first to nth feature quantities are extracted from the first to nth image data. In the third step, the first to nth feature quantities are supplied to the classifier, and in the fourth step... Then, using a classifier, based on the first to nth features, the first to nth image data An image classification method that performs clustering of first to n image data This is image data that has been identified as containing defects, and the identification model consists of an input layer and an intermediate layer. The system has an output layer and a processing layer, and the feature quantities output from the processing layer are numerical values ​​of the neurons in the hidden layer. This is an image classification method.

[0020] In the above image classification method, the identification model determines that image data contains defects. In contrast, supervised learning is performed so that the type of delinquent is inferred, and clustering It is preferable to use a hierarchical approach for the grading process.

[0021] In the image classification method described above, the number of dimensions of the feature quantities output from the processing unit is between 32 and 256. The following is preferable:

[0022] Furthermore, in the above image classification method, in the fifth step, the class made by the classifier The result of the taring is supplied to the output unit, and in the sixth step, it is preferable to display the result. It's nice. [Effects of the Invention]

[0023] According to one aspect of the present invention, a novel classification device can be provided. Depending on the circumstances, a novel image classification method can be provided. Furthermore, according to one aspect of the present invention, We can provide a novel pattern inspection device.

[0024] The effects of one embodiment of the present invention are not limited to those listed above. This does not preclude the existence of other effects. These other effects are described below. This is an effect not mentioned in the section. An effect not mentioned in this section would be obvious to someone skilled in the art. This can be derived from detailed descriptions, drawings, etc., and can be extracted appropriately from these descriptions. Yes, it is possible. Furthermore, one aspect of the present invention includes, among the effects listed above and / or other effects, at least one of the following: It has at least one effect. Therefore, one aspect of the present invention may, in some cases, It may not always have the effects listed above. [Brief explanation of the drawing]

[0025] [Figure 1] Figure 1 is a block diagram showing an example of a classification device. [Figure 2] Figure 2 shows an example of image data and labels. [Figure 3] Figures 3A and 3B are schematic diagrams illustrating examples of neural network configurations. [Figure 4] Figures 4A and 4B are block diagrams showing an example of a classification device. [Figure 5] Figure 5 is a flowchart showing an example of a method for classifying image data. [Figure 6] Figure 6 shows an example of image data and features. [Figure 7] Figure 7 is a flowchart showing an example of a method for classifying image data. [Figure 8] Figure 8 is a flowchart showing an example of a method for generating a pre-trained discriminative model. [Figure 9] Figure 9 shows an example of the hardware of a classification device. [Figure 10] Figure 10 shows an example of the hardware of a classification device. [Figure 11] Figure 11 is a block diagram showing an example of a pattern inspection device. [Figure 12] Figure 12 illustrates the results of the clustering. [Figure 13] Figures 13A to 13D show examples of image data. [Modes for carrying out the invention]

[0026] Embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description. Without departing from the spirit and scope of the present invention, its form and details may be changed in various ways. Those skilled in the art will readily understand that further improvements are possible. Therefore, the present invention can be implemented as follows: It should not be interpreted as being limited to the description of the form.

[0027] Furthermore, in the configuration of the invention described below, the same part or part having a similar function The same reference numeral is used consistently across different drawings, and explanations of its repetition are omitted.

[0028] Furthermore, the position, size, and scope of each component shown in the drawings are, for the sake of ease of understanding, actual The position, size, and range of the edges may not be shown. Therefore, the disclosed invention is not necessarily However, this is not limited to the location, size, and scope disclosed in the drawings.

[0029] Furthermore, the ordinal numbers "1st," "2nd," and "3rd" used in this specification refer to the constituent elements. This note is added to avoid confusion and does not imply any numerical limitation.

[0030] Furthermore, if upper and lower numerical limits are specified in this specification, they may be freely combined. The configuration is also disclosed.

[0031] In this specification, the datasets used for training and evaluating machine learning models are referred to as training datasets It is referred to as a dataset. When training and evaluating machine learning models, the training dataset is It is divided into training data (also called learning data) and test data (also called evaluation data). It is divided. Furthermore, the training data may be split into training data and validation data. The test data may be pre-split from the training dataset.

[0032] Training data refers to data used to train a machine learning model. Validation data refers to data used to train a machine learning model. This is data used to evaluate the training results of a machine learning model. Test data refers to data used to evaluate the training results of a machine learning model. This is the data used to evaluate Dell. If machine learning is supervised learning, the training data, The validation data and test data are labeled.

[0033] In this specification and elsewhere, a semiconductor device is defined as a device that can function by utilizing semiconductor properties. It refers to semiconductors such as transistors, diodes, light-emitting elements, or light-receiving elements. It is a semiconductor element. Another example of a semiconductor element is a conductive film such as a capacitor, resistor, or inductor. Alternatively, it is a passive element formed by an insulating film or the like. Another example of a semiconductor element is a semiconductor element. This is a semiconductor device comprising a circuit having conductive elements or passive elements.

[0034] (Embodiment 1) In this embodiment, a classification device according to one aspect of the present invention will be described with reference to Figures 1 to 4. ru.

[0035] The classification device is installed in an information processing device such as a personal computer used by the user. Alternatively, a classification device can be installed on the server, and accessed from a client PC via the network. It can be used via [a specific method / platform].

[0036] The classification device has a function to cluster image data. In this embodiment, The classification device is described as one that clusters defects detected in the semiconductor manufacturing process. In other words, the image data in question is image data that includes defects detected in the semiconductor manufacturing process. Let's assume that.

[0037] Defects detected in the semiconductor manufacturing process include, for example, foreign matter contamination, film defects, and pattern defects. These include film residue, film lifting, and disconnection. Foreign matter contamination refers to the contamination of semiconductors by workers during the semiconductor manufacturing process. Foreign matter originating from materials, manufacturing equipment, and the work environment can be found on substrates (such as silicon wafers). Adhesion onto conductive substrates, glass substrates, plastic substrates, metal substrates, SOI substrates, etc. This refers to defects that occur due to [unspecified cause]. Furthermore, film delamination refers to the peeling off of the normal pattern. This refers to a defect. A pattern defect, on the other hand, means that the pattern was not formed according to the design. This refers to defects that occur in that way.

[0038] The above image data shows semiconductor devices in the middle of the manufacturing process, or semiconductor devices after the manufacturing process has been completed. In a physical device, patterns such as semiconductor films, insulating films, and wiring (hereinafter simply referred to as patterns) This is image data in which an abnormal area has been captured. In other words, the image data is defective. The area in question can be rephrased as image data in which the suspected area is captured. This is sometimes referred to as image data containing defects.

[0039] <Classification device 100> Figure 1 shows an example of a classification device according to one aspect of the present invention. Figure 1 shows the configuration of the classification device 100. This is a diagram. The classification device 100 consists of a storage unit 101, a processing unit 102, as shown in Figure 1. It includes a classifier 103 and an output unit 104.

[0040] [Storage section 101] Image data is stored in the memory unit 101. This image data includes image data containing defects. It is Ta.

[0041] Here, the image data stored in the memory unit 101 will be explained using Figure 2. Multiple image data 50 are stored in section 101. As shown in Figure 2, multiple image data Data 50 is image data 51_1 to image data 51_s (where s is an integer greater than or equal to 1). And, image data 52_1 to image data 52_t (where t is an integer of 1 or more) do.

[0042] Each of the image data 51_1 through 51_s has a label assigned to it. In Figure 2, image data 51_1 and image data 51_2 are assigned the label 61A. It has been done. Also, the image data 51_s has been assigned the label 61F. In terms of form, the labels attached to image data 51_1 to image data 51_s are semiconductor-made It addresses defects detected during the manufacturing process. In other words, it addresses defects such as labels 61A and 61F. Each corresponds to one or more defects detected in the semiconductor manufacturing process. The types of defects detected in the semiconductor manufacturing process are given as a numerical array.

[0043] Image data 52_1 to 52_t are assigned labels corresponding to defects. None. Note that the "-" in Figure 2 indicates that the image data has not been labeled.

[0044] [Processing step 102] The processing unit 102 has the function of performing processing using a trained discrimination model. Specifically, The processing unit 102 has the function of extracting features from image data using a trained classification model. The processing unit 102 has image data 53_1 to 53_n (where n is 2 or more). It is an integer of the number .) supplied from the storage unit 101. Here, image data 53_1 to image Data 53_n is part or all of the multiple image data 50 stored in the storage unit 101. At this time, the processing unit 102 processes the image data 53_1 to 53_n. Features are extracted for each of them.

[0045] A neural network is preferred for the discrimination model, and convolutional neural networks are preferred. CNN: Convolutional Neural Network It is preferable to use ). For example, VGG11, VGG16, Go Examples include ogLeNet and ResNet.

[0046] Figure 3A shows an example of the configuration of neural network 300. Workpiece 300 has layers 301_1 to 301_k (where k is an integer greater than or equal to 3). .

[0047] Layers 301_1 to 301_k contain neurons, and each layer is provided with neurons The neurons are connected to each other. For example, the neurons located in layer 301_1 are connected to each other in layer 30 It is connected to the neurons located in layer 1_2. Also, it is located in layer 301_2. The neurons located in layer 301_1 and those located in layer 301_3 are also included. It is connected to the neurons that are located in layers 301_3 through 301_k. The same applies to the neurons located in these layers. That is, layers 301_1 to 30 1_k constitutes a hierarchical neural network.

[0048] Image data is input to layer 301_1, and layer 301_1 corresponds to the input image data. The data is output. This data is input to layer 301_2, and layer 301_2 is input Outputs data corresponding to the data. Layer 301_k outputs the data from layer 301_k-1. When data is input, layer 301_k outputs data corresponding to that input data. Based on the above, layer 301_1 is the input layer, layers 301_2 to 301_k-1 are the hidden layers, and layer 30 1_k can be used as the output layer. Note that this applies to neural networks with two or more hidden layers. This method is also known as deep learning.

[0049] The neural network 300 outputs, for example, from layers 301_1 to 301_k. The data corresponds to the features of the image data input to the neural network 300. It is pre-trained to behave in this way. Learning is done through unsupervised learning, supervised learning, etc. This can be done whether learning is done using unsupervised or supervised methods. However, a backpropagation method or the like can be used as the learning algorithm. Therefore, learning is preferably carried out through supervised learning.

[0050] An example of a CNN is shown in Figure 3B. Figure 3B shows the configuration of CNN310. As shown in Figure 3B, N310 consists of multiple convolutional layers (convolutional layer 311_1 to con Layer 311_m (where m is an integer greater than or equal to 1), multiple pooling layers (pooling layer 3 It has 12_1 to pooling layer 312_m) and a fully bonded layer 313. In Figure 3B, The fully bonded layer 313 consists of fully bonded layer 313_1, fully bonded layer 313_2, and fully bonded layer 313_ An example with 3 layers is shown. Note that CNN310 has one fully connected layer 313 or It may have only two layers, or it may have four or more layers.

[0051] A convolutional layer has the function of performing convolution on the data input to that convolutional layer. For example, the convolutional layer 311_1 performs convolution on the input image data. It has the ability to process data output from pooling layer 312_1. It has the function of performing convolution on the data. In addition, the convolutional layer 311_m is pooling It has the function of performing convolution on the data output from layer 312_m-1.

[0052] Convolution involves repeatedly performing sum-of-products operations on the data input to the convolutional layer and the weight filter. This is done by convolution in the convolutional layer, which is the input to CNN310. Image features and other characteristics corresponding to the image data are extracted.

[0053] The convolved data is transformed by an activation function and then output to a pooling layer. The activation function is ReLU (Rectified Linear Unit). s) etc. can be used. ReLU outputs "0" if the input value is negative. This function outputs the input value as is if the force value is "0" or greater. It also includes an activation function. Alternatively, functions such as the sigmoid function and the tanh function can be used.

[0054] The pooling layer has the function of pooling the data input from the convolutional layer. Pooling divides the data into multiple regions and extracts specific data from each region. This is a process of arranging them in a matrix. Through pooling, the convolutional layer extracts It is possible to reduce the amount of data while retaining the desired characteristics. Also, minute details of the input data This can increase robustness against this. Furthermore, as for pooling, the maximum pooling Average pooling, Lp pooling, etc., can be used.

[0055] The fully connected layer 313 transforms the input data using an activation function and outputs the transformed data. It has the ability to perform the following actions. Specifically, if CNN310 has the configuration shown in Figure 3B, it is fully connected. Layer 313_1 transforms the data output from pooling layer 312_m using an activation function. It has the function of receiving the data output from the fully connected layer 313_1. It has the function of transforming the data using an activation function. In addition, the fully connected layer 313_3 is a fully connected layer It has the function of transforming the data output from 313_2 using an activation function. Activation function For example, ReLU, sigmoid function, tanh function, etc. can be used. Fully connected layer 3 13 has a configuration in which all nodes in one layer are connected to all nodes in the next layer. The data output from the integration layer or pooling layer is a two-dimensional feature map, and the fully connected layer When input to 313, it is expanded into one dimension. Then, inference is performed by the fully connected layer 313. The obtained vector is output from the fully connected layer 313.

[0056] In CNN310, one of the layers in the fully connected layer 313 can be used as the output layer. For example, in CNN310 shown in Figure 3B, the fully connected layer 313_3 can be used as the output layer. Yes, it is possible. Here, in CNN310 shown in Figure 3B, the fully connected layer 313_1 and the fully connected layer 313_2 can be an intermediate layer. Alternatively, CNN310 can be a fully connected layer 313. If only a fully coupled layer 313_1 is present, the fully coupled layer 313_1 can be used as an output layer. Alternatively, CNN310 has fully connected layer 313_1 and fully connected layer 313_2. In this case, the fully coupled layer 313_2 can be used as the output layer and the fully coupled layer 313_1 can be used as the intermediate layer. Similarly, if CNN310 has four or more fully connected layers 313, then it also has one fully connected layer 3 Layer 13 can be used as the output layer, and the remaining fully coupled layer 313 can be used as an intermediate layer.

[0057] Note that the configuration of CNN310 is not limited to the configuration shown in Figure 3B. For example, it can have multiple convolutional layers. Each of the convolutional layers (311_1 to 311_m) consists of two or more convolutional layers. It may have layers. In other words, the number of convolutional layers in CNN310 is equal to the number of pooling layers. More is acceptable. Also, if you want to retain as much of the location information of the extracted features as possible, pooling is used. It is not necessary to create layers.

[0058] CNN310 learns the filter values ​​of the weight filters and the weights of the fully connected layers through training. Numerical values ​​can be optimized.

[0059] The above identification model is trained to take image data as input and output a defect identification result. In other words, when image data is input to the input layer of a neural network, The output layer of the neural network outputs the defect identification result. For example, If the network has the configuration shown in Figure 3A, then the input layer, layer 301_1, contains an image with defects. When data is input, the defect identification result is output from layer 301_k, which is the output layer. Furthermore, if the neural network has the configuration shown in Figure 3B, the input layer is the convolutional layer. When image data containing defects is input to 311_1, the fully connected layer 313_3, which is the output layer, Then, the defect identification result is output.

[0060] The processing unit 102 has the function of outputting numerical values ​​for neurons in the intermediate layer of the discrimination model. The number of neurons in the hidden layer corresponds to the features of the image data input to the discrimination model. It includes corresponding data (also called features). In other words, it contains the numerical values ​​of neurons in the hidden layer. By outputting the data, it is possible to extract the feature quantities of the image data input to the classification model.

[0061] The number of dimensions of the extracted features is preferably above a certain number. Without it, the accuracy of clustering may be insufficient. On the other hand, if the number of dimensions is large, The computational load for clustering increases, and the time required for clustering also increases. This may result in insufficient computing resources. The number of dimensions in question is, for example, the fully connected layer that serves as the output layer. It is preferable that the number of dimensions be greater than the given number of dimensions. Specifically, the number of dimensions is 32 or more and 1024 or less. A lower value is preferable, and a value between 32 and 256 is more preferable.

[0062] If the neural network has the configuration shown in Figure 3A, then for example, layer 301_k-1 is The numerical value of the neurons present is output. Here, the neurons output from layer 301_k-1 Let Ron's value be Feature 305. Feature 305 is the data corresponding to the features of the image data. This includes the following. Note that in Figure 3A, feature quantity 305 is shown as being output from layer 301_k-1. However, it is not limited to this. For example, feature 305 is layer 301_2 to layer 301_ The output may come from any one of the k-2 points.

[0063] Furthermore, if the neural network has the configuration shown in Figure 3B, for example, fully connected layer 31 The number of neurons in 3_2 is output. Here, the output from the fully connected layer 313_2 Let the numerical value of the neuron be defined as feature 315. Feature 315 corresponds to the features of the image data. This includes the data. Note that in Figure 3B, feature 315 is output from the fully connected layer 313_2. The configuration shown is one example, but it is not limited to this. For example, feature 315 is a convolutional layer 31 1_1 or convolutional layer 311_m, pooling layer 312_1 or pooling layer 312_m The output may come from either of the following: the fully connected layer 313_1. Alternatively, the features output from a pooling layer are sometimes called feature maps.

[0064] The processing unit 102 consists of a CPU (Central Processing Unit) and a GP (Programmable Processor). Processing can be done using a U (Graphics Processing Unit), etc. Yes, it is possible. In particular, the processing unit 102 is composed of a neural network, so the GPU Using this method is preferable because it allows for high-speed processing.

[0065] The trained identification model is stored in the memory unit (not shown in Figure 1) of the processing unit 102. The trained identification model may be stored in the memory unit 101. If stored in the memory unit 101, the processing unit 102 receives the learned identification model from the memory unit 101. It is supplied. Alternatively, the trained discriminant model is transmitted via an input unit, storage medium, communication, etc. It may be supplied to the processing unit 102.

[0066] [Classifier 103] The classifier 103 has the function of performing clustering (cluster analysis). Specifically, The classifier 103 has the function of clustering image data based on its features. The classifier 103 receives the image data 53_1 to image data extracted by the processing unit 102. Each of the 53_n features is supplied. At this time, classifier 103 processes the image data 5 Based on the respective features of image data 53_n, Image data 53_n is clustered.

[0067] Clustering (cluster analysis) methods include hierarchical methods and non-hierarchical methods. It can be used. Hierarchical methods group similar data together to form classes. This is a method for forming a grid. Hierarchical methods include, for example, the shortest distance method, the longest distance method, and grouping. Examples include the equality method and Ward's algorithm. Additionally, non-hierarchical methods classify similar data points as belonging to the same class. This is a method of dividing the entire data into categories that belong to a group. As a non-hierarchical method, for example, Examples include the k-means method.

[0068] In this embodiment, a hierarchical method is used as the clustering (cluster analysis) method. It is preferable that they be present. By using a hierarchical method, defects that have not been previously identified can be included. If image data is included, suppress its classification into the set that has been identified as defective. This is possible. Also, if the data distribution of the image data to be processed is unknown, a hierarchical method can be used. This is preferable because there is no initial setting for the number of clusters. In hierarchical methods, a threshold is set. Then, the number of clusters is determined. This threshold is determined, for example, by preparing sample data. It is best to choose a value that provides a high degree of precision.

[0069] When the total number of image data is large, the k-means method is used as a clustering technique. It may be preferable to have it. When the total number of image data becomes large (for example, exceeding 2000) Therefore, the k-means method can perform clustering with fewer computations compared to hierarchical methods. It is sometimes possible. When using the k-means method, the number of clusters is xm You can use the EANS method for automatic estimation, or you can prepare sample data and make a predetermined decision. You can leave it there.

[0070] The classifier 103 may have a memory unit (not shown in Figure 1). This contains programs related to clustering methods. The program relating to the method may be stored in the memory unit 101. If stored in unit 101, the classifier 103 receives the program from the storage unit 101. The following is supplied. Alternatively, the program relating to the clustering method is provided in the input unit and storage medium. The data may be supplied to the classifier 103 via communication or other means.

[0071] [Output section 104] The output unit 104 has the function of supplying the clustering results performed by the classifier 103. The output unit 104 may also have a function to display the above results. For example, the output unit 1 04 includes output devices such as displays and speakers.

[0072] The classification device 100 may have an input unit (not shown in Figure 1). The image data is... It is preferable that the data be stored in the memory unit 101 via the input unit. Also, identification models and clustering Programs related to the method may also be stored in the memory unit 101 via the input unit. Furthermore, the image data may be stored in the storage unit 101 via a storage medium, communication, etc. stomach.

[0073] The above is a description of the classification device 100. Note that in Figure 1, the classification device 100 is... The diagram shows a configuration having a storage unit 101, a processing unit 102, a classifier 103, and an output unit 104. However, this is not the only example. Modifications of the classification device 100 are shown below. The classification described below... Modified versions of the apparatus can be appropriately combined with other classification devices shown in this specification and elsewhere.

[0074] <Classification device 100A> A modified example of the classification device 100 shown in Figure 1, the classification device 100A, is shown in Figure 4A. Figure 4A As shown, the classification device 100A includes a storage unit 101, a processing unit 102, a classifier 103, and In addition to the output unit 104, it also has a processing unit 105.

[0075] The processing unit 105 has the function of processing image data. Further details will be described later. Additionally, the processing unit 105 may have a function for data expansion.

[0076] In the classification device 100A, image data 53_1 to 53_n are processed by the processing unit 105 It is supplied to the processing unit 105, and image data 53_1 to 53_n are processed. By processing, image data 53_1 to image data 53_n are different from image data 53_n. Image data a_1 to 53a_n is generated. Image data generated in processing unit 105 Image data 53a_1 to 53a_n are supplied to the processing unit 102. Then, the feature quantities of each image data 53a_1 through 53a_n are extracted. The multiple features extracted by the processing unit 102 are supplied to the classifier 103. Classifier 10 Using 3, based on the multiple features, image data 53_1 to image data 53_n These are then clustered.

[0077] By having a processing unit 105, the feature quantities extracted by the processing unit include those included in the image data. This allows for better inclusion of defect characteristics. Therefore, it improves the accuracy of clustering. It is possible.

[0078] The above is a description of the classification device 100A.

[0079] <Classification device 100B> Furthermore, a modified example of the classification device 100A shown in Figure 4A, the classification device 100B, is shown in Figure 4B. As shown in Figure 4B, the classification device 100B consists of a storage unit 101, a classifier 103, and an output unit 1 It has a 04, a processing section 105, a first processing section 106, and a second processing section 107.

[0080] The second processing unit 107 corresponds to the processing unit 102 of the classification device 100A. Therefore, the second processing unit Regarding section 107, the contents described for processing unit 102 can be taken into consideration.

[0081] The first processing unit 106 has the function of training the discrimination model. Using the first processing unit 106 By doing so, a trained discrimination model can be generated. Alternatively, the first processing unit 106 By using this, the discriminant model can be retrained. Retraining the discriminant model can be done, for example, After the labeled image data is stored in the storage unit 101, or the storage unit 101 Unlabeled image data stored in (image data 52_ shown in Figure 2) This is done after one or more of the image data (1 to 52_t) have been labeled. Good. Retrain the classification model to use the improved classification model. This allows for improved clustering accuracy.

[0082] The above is a description of the classification device 100B.

[0083] By utilizing the clustering results obtained using a classification device according to one aspect of the present invention, In situations where the accuracy of inference by the discrimination model is insufficient, defects are identified visually in image data. Even if combined, the time required to determine the type of defect can be shortened. Even users who are not sufficiently proficient in identification can reduce the time required to identify defects, and improve defect recognition. It can perform the task with high precision.

[0084] Furthermore, the classification device according to one aspect of the present invention is used to cluster image data containing defects. This allows users to quickly and appropriately decide whether or not to perform a rework process. In particular, image data containing defects and the location where the image data was taken (lot / location within the circuit board). It is valid if it is associated with ) and has not been confirmed to date. It makes it easy to identify defects and defects whose type cannot be determined.

[0085] Based on the above, by using a classification device according to one aspect of the present invention, work efficiency can be improved. It is possible.

[0086] Furthermore, the image data that the classification device according to one aspect of the present invention clusters is used in the semiconductor manufacturing process. This is not limited to image data containing defects detected by the system. For example, it includes images containing deterioration or damage to buildings. Image data is also acceptable. Building deterioration includes cracks, peeling, foreign matter adhesion, corrosion, etc. By clustering image data that includes deterioration and damage to buildings, the repair of buildings can be improved. This can be done quickly and appropriately. The image data can be obtained from fixed-point cameras, surveillance cameras, etc. It is best to obtain this by photographing buildings using a camera or similar device.

[0087] This embodiment can be appropriately combined with other embodiments, examples, etc. In this specification, if multiple configuration examples are shown within a single embodiment, the configuration examples are... It is possible to combine them as appropriate.

[0088] (Embodiment 2) In this embodiment, a method for classifying image data (image classification method), and learned recognition The method for generating a separate model will be explained using Figures 5 to 8. Note that this embodiment... The method for classifying the image data can be performed using the classification device described in Embodiment 1. can.

[0089] <Methods for classifying image data> A method for classifying image data, which is one aspect of the present invention, will be described. Image data classification, explained by morphology, involves dividing a set of image data into multiple subsets. This refers to the classification of image data in this embodiment, which is the classification of image data into classes. This can be rephrased as "taring."

[0090] Figure 5 is a flowchart showing an example of a method for classifying image data. This is also a flowchart illustrating the processing flow performed by the classification device described in Form 1.

[0091] The method for classifying image data is as shown in Figure 5, from step S001 to step S0 It has 05.

[0092] [Step S001] Step S001 is the process of supplying multiple image data to the processing unit. The image data corresponds to the image data 53_1 to 53_n described in Embodiment 1. The processing unit is the processing unit 102 or the second processing unit 10 described in Embodiment 1. It corresponds to 7.

[0093] [Step S002] Step S002 is when the processing unit extracts the feature quantities of each of the multiple image data. This is the process of outputting the results. In other words, from image data 53_1 to image data 53_n, Feature vectors 62_1 to 62_n are extracted. Feature extraction is described in Embodiment 1. This can be done using a pre-trained classification model. In other words, the feature is used for the implementation. This corresponds to feature vectors 305 or 315 as described in Form 1.

[0094] Each image data feature is given by a numerical array with u dimensions. That is, 62 features. Each of the features _1 through 62_n is given by an array of u numerical elements. For example, as shown in Figure 6. As such, the feature 62_1 extracted from the image data 53_1 is the value 63_1[1] or It consists of the value 63_1[u]. Similarly, the feature 6 extracted from the image data 53_2 2_2 is composed of values ​​63_2[1] to 63_2[u]. Similarly, image The feature 62_n extracted from data 53_n is a value 63_n[1] to a value 63_n[u It consists of ].

[0095] [Step S003] Step S003 is performed using the feature quantities extracted in the above processing unit (feature quantities 62_1 to 62_1). This is the process of supplying 2_n) to the classifier. The classifier is the classifier described in Embodiment 1. It corresponds to 103.

[0096] [Step S004] Step S004 is performed by the classifier above, using the above features (features 62_1 to 62_ Based on n), the above multiple image data (image data 53_1 to image data 53_n) This is the process of performing clustering. For example, clustering of image data can be performed The hierarchical method described in Form 1 can be used.

[0097] [Step S005] Step S005 displays the clustering results performed by the classifier in the output section. This is the process shown. The output unit corresponds to the output unit 104 described in Embodiment 1. Oh, when a hierarchical method is used for clustering image data, the clustering As a result of the process, for example, a dendrogram is created. Therefore, the output unit is, for example, , display the dendrogram.

[0098] Based on the above, image data can be classified.

[0099] Furthermore, the methods for classifying image data are not limited to those described above. Another example of this method is shown below.

[0100] Figure 7 is a flowchart illustrating another example of a method for classifying image data. The method for classifying image data is as follows: Steps S011 to S014, and It may also have steps S003 to S005. Regarding S005, the above information can be taken into consideration.

[0101] [Step S011] Step S011 is a process of supplying multiple image data to the processing unit. The image data corresponds to the image data 53_1 to 53_n described in Embodiment 1. Furthermore, this processing section corresponds to the processing section 105 described in Embodiment 1.

[0102] [Step S012] Step S012 is the process in which the processing unit processes each of the multiple image data. Specifically, step S012 is to process image data 53_1 to image data 53_ This is a process of processing n to generate image data 53a_1 to 53a_n. More specifically, the region containing defects is extracted from image data 53_1 to image data 53_n. This is a process of generating image data 53a_1 to 53a_n by extracting the data. Oh, cropping out the area containing defects from the image data, and cropping out the area without defects from the image data. This can be rephrased as at least partial removal.

[0103] The following describes the process of processing image data 53_1 to generate image data 53a_1. I will explain.

[0104] The shape of image data 53a_1 should be a rectangle. The length of the longest side of this rectangle should be a The length is 1, and the length of the shorter side is a2. Lengths a1 and a2 are in image data 53a_ Specify that 1 fits within the image data 53_1. Therefore, at least length a1 is, The length a2 is less than or equal to the length of the longer side of the image data 53_1, and the length a2 is the length of the shorter side of the image data 53_1. The following applies. Also, lengths a1 and a2 are defined as follows, and the defects are contained within the image data 53a_1. Specify that it should be done that way.

[0105] The ratio of length a1 to length a2 is the length of the longer side of image data 53_1 and the length of the longer side of image data 53_1. It is preferable that the ratio is equal to the ratio of the lengths of the shorter sides. The length of the longer side of image data 53_1 and the image data If the ratio of the shorter sides of 53_1 is 4:3, for example, if length a1 is 640 pixels It is best to set the length a2 to 480 pixels.

[0106] Note that the ratio of length a1 to length a2 is the length of the longer side of image data 53_1 and image data 53 The ratio of the lengths of the shorter sides of _1 does not necessarily have to match. For example, the above rectangle has length a1 The ratio of length a2 to the length of the longer side of image data 53_1 is the ratio of the length of the shorter side of image data 53_1. The ratio may be different. Alternatively, the shape of image data 53a_1 may be a square. .

[0107] Alternatively, the above rectangle is such that the longer side of the rectangle is parallel to the shorter side of the image data 53_1. The shorter side of the rectangle may be parallel to the longer side of the image data 53_1. Alternatively, The above rectangle is defined as such that the longer side of the rectangle is parallel or perpendicular to the longer side of the image data 53_1. It's fine.

[0108] The position of image data 53a_1 is such that image data 53a_1 fits within image data 53_1. The position of image data 53a_1 is determined to be the centroid of image data 53a_1. You can decide on a criterion, or you can decide on one of the vertices of image data 53a_1 as the criterion. This is also good. For example, the centroid of image data 53a_1 is determined using a uniform random number. A uniform random number is, It follows a continuous uniform distribution such that all real numbers appear with equal probability within a specified interval or range. It is a random number.

[0109] In the above, after specifying lengths a1 and a2, the position of image data 53a_1 is determined. The process for determining the position has been explained, but the process is not limited to this. After specifying, the length a1 is set so that the image data 53a_1 fits within the image data 53_1. The length a2 may be determined. Alternatively, the image data 53a_1 may be used to determine the image data 53_1. To ensure that it fits, simultaneously adjust the position of image data 53a_1, as well as lengths a1 and a2. You can make the decision.

[0110] Note that the length of the long side and the length of the short side of image data 53a_1 are the same as those of other image data 53a It is preferable that the length of the long side and the length of the short side of _1 are equal. This is how the above As demonstrated, this improves the accuracy of identifying defects.

[0111] The above describes the process of processing image data 53_1 to generate image data 53a_1. This is an explanation. Furthermore, image data 53a_2 to 53a_n are processed in the same manner. It can be generated by doing this.

[0112] Step S012 may be performed by the user or automatically using a classification device. Good. When using a classification device, for example, image data containing defects and image data without defects It is recommended to obtain the difference between the original image and the current image, and then crop out the area with the largest difference, as well as the surrounding areas.

[0113] Based on the above, image data 53a_1 to 53a_n can be generated. Extract the area containing the defect from each of the image data 53_1 through 53_n. Therefore, the area (area) occupied by the part to be identified relative to the entire area (area) of the image data. The proportion of ) can be increased. This allows the features extracted in the processing unit to be image This allows for better inclusion of defects in the data. Therefore, clustering The accuracy can be improved. Note that image data 53a_1 to image data 53a_ n is the label assigned to each of the image data 53_1 through 53_n. It will be granted.

[0114] [Step S013] Step S013 supplies image data 53a_1 to 53a_n to the processing unit. This is the process. The processing unit is the processing unit 102 or the second processing unit described in Embodiment 1. It corresponds to 107.

[0115] [Step S014] Step S014 is performed by the above processing unit, which processes image data 53a_1 to image data 53a_ This is the process of extracting each feature of n. From data 53a_n, feature quantities 62_1 to 62_n are extracted respectively. Data 53a_1 to image data 53a_n are, respectively, image data 53_1 to image data Since it is created by processing -53_n, feature quantities 62_1 to 62_n are, These can be rephrased as the feature quantities of image data 53_1 to 53_n, respectively. Feature extraction is performed using the trained classification model described in Embodiment 1. It is possible.

[0116] After performing step S014, step S003, step S004, and step Perform S005 in order. Based on the above, image data can be classified.

[0117] The above explains how to classify image data. Image data containing defects is clustered. By doing so, the time required for users to identify defects can be reduced. Even users who are not sufficiently proficient in identifying defects can perform defect identification with high accuracy. Furthermore, users can make quick and appropriate decisions about whether or not to perform a rework process. Yes, it is possible. Also, defects that have never been observed before, defects whose type cannot be identified, etc. This can be easily understood.

[0118] <How to generate a pre-trained discriminative model> This section describes a method for generating a trained discrimination model according to one aspect of the present invention. The method for generating a pre-trained discriminative model is the same as the method for training the discriminative model. It can be replaced. Also, the method for generating a trained discriminant model is trained discriminant This can be rephrased as a method for retraining the model.

[0119] Figure 8 is a flowchart showing an example of a method for generating a pre-trained discriminative model. Figure 8 is also a flowchart illustrating the processing flow performed by the classification device described in Embodiment 1. ru.

[0120] The method for generating a trained discriminant model is shown in Figure 8, from step S021 to S It has a step S026.

[0121] [Step S021] Step S021 is a process of supplying multiple image data to the processing unit. Image data can be used as training data, validation data, or test data. This is image data. Furthermore, this processing unit corresponds to the processing unit 105 described in Embodiment 1. .

[0122] Each of the above multiple image data contains defective image data. Furthermore, the defective Image data containing the image data is assigned a label corresponding to the defect contained in that image data. In other words, the above multiple image data are one of the image data 51_1 to 51_s. This refers to a part or all of the above image data. Here, the above multiple image data are referred to as image data 54_1 to image data Write it as -54_p (where p is an integer between 2 and s, inclusive).

[0123] [Step S022] Step S022 involves processing the above-mentioned multiple image data in the processing unit, and then processing the above-mentioned multiple This is a process that generates multiple image data, which are different from the image data. Specifically, it generates image data Image data 54_1 to 54_p are processed to obtain image data 54a_1 to 5 This is the process of generating 4a_p. More specifically, it involves generating image data 54_1 to image data 5 By cutting out the area containing defects from 4_p, image data 54a_1 to image data 54 This is the process of generating a_p. This process can be described in reference to step S012. Cut.

[0124] Image data represents the area of ​​the data to be identified relative to the entire area of ​​the image data. It is preferable that the proportion of the area (region) is large. For example, in the case of a pattern inspection result image. Therefore, cutting out the defective parts is effective. The feature quantities extracted in the processing unit by the above processing are This allows for better inclusion of defect features in the image data. Therefore, class The accuracy of tagging can be improved. 54a_p is assigned to each of the image data 54_1 through 54_p respectively. A label will be assigned.

[0125] [Step S023] Step S023 is a process of performing data augmentation in the above processing unit. As methods of data augmentation, rotation, flipping, noise addition, blurring, gamma conversion, etc. for image data can be mentioned. For data augmentation, it is preferable to use some or all of the image data 54a_1 to image data 54a_p. By data augmentation, q (q is an integer of 1 or more) pieces of image data (image data 54a_p+1 to image data 54a_p+q) are generated.

[0126] Note that it is preferable that the number of image data created for each defect is approximately equal. For example, the number of image data to which labels corresponding to different objects are assigned, the number of image data to which labels corresponding to film peeling are assigned, the number of image data to which labels corresponding to pattern defects are assigned, etc. are preferably approximately equal. Thereby, overfitting (overlearning) with respect to a specific defect can be suppressed.

[0127] The method of data augmentation, the number of image data generated by data augmentation, etc. may be randomly selected, or may be specified by the user. Also, based on the labels given to the image data 54a_1 to image data 54a_p, etc., the classification device may automatically select.

[0128] Note that there may be cases where data augmentation does not need to be performed. For example, it is a case where learning data sufficient to generate an identification model that can be identified with high accuracy is available. At this time, step S02 3 may be omitted.

[0129] By performing step S022 and step S023, a learning data set is generated. ​​​This is possible. The input data for the training dataset is p created in step S022. The image data (image data 54a_1 to image data 54a_p), and step q image data generated in S023 (image data 54a_p+1 to image data 54a (_p+q)

[0130] The correct labels for the training dataset are image data 54a_1 to image data 54a_p These are the labels assigned to each of the +q elements.

[0131] Therefore, the training dataset consists of (p+q) image data and the data set It consists of the assigned labels.

[0132] [Step S024] Step S024 supplies the training dataset generated in the processing unit to the processing unit. This is the process. The training dataset includes image data 54a_1 to image data 54 a_p+q is included. This processing unit corresponds to the first processing unit 106 described in Embodiment 1. do.

[0133] [Step S025] Step S025 is when the above processing unit uses the above training dataset to create a discriminant model. This is the process of training the system.

[0134] When training the above discrimination model, the above training dataset is used as training data and validation data. It is good to split the data into training data and test data. For example, using the training data to develop the classification model... The model is trained, the results of the training are evaluated using the verification data, and the test data is used Next, evaluate the trained classification model. This verifies the accuracy of the trained classification model. This is possible. Hereinafter, the ratio of the number of correct identification results to the number of test data may be referred to as the correct rate. It may be called the correct rate.

[0135] The training data is composed of a part of the image data 54a_1 to the image data 54a_p+q. The verification data is composed of a part of the image data not used for the training data. The test data is composed of the image data not used for the training data and the verification data.

[0136] As a method of dividing the training data set into training data, verification data, and test data, for example, there are the hold-out method, the cross-validation method, the leave-one-out method, etc. For example, there are the hold-out method, the cross-validation method, the leave-one-out method, etc.

[0137] The learning may end when a predetermined number of trials is reached. Or, the learning may end when the correct rate exceeds a predetermined threshold. Or, the learning may end when the correct rate saturates to a certain extent. It should be noted that a constant may be prepared in advance for the number of trials or the threshold. Or, during the learning, the user may be allowed to specify the timing to end. Or, the learning may end when the correct rate exceeds a predetermined threshold. Or, the learning may end when the correct rate saturates to a certain extent. It should be noted that a constant may be prepared in advance for the number of trials or the threshold. Or, during the learning, the user may be allowed to specify the timing to end. Or, the learning may end when the correct rate saturates to a certain extent. It should be noted that a constant may be prepared in advance for the number of trials or the threshold. Or, during the learning, the user may be allowed to specify the timing to end. It should be noted that a constant may be prepared in advance for the number of trials or the threshold. Or, during the learning, the user may be allowed to specify the timing to end. Or, during the learning, the user may be allowed to specify the timing to end.

[0138] Through the above learning, a learned identification model is generated.

[0139] [Step S026] Step S026 is a step of storing the learned identification model generated in step S025 in the storage unit. The storage unit is the storage unit 101 described in Embodiment 1. Note that the storage unit may be stored in the storage unit possessed by the first processing unit 106, the processing unit 102, or the storage unit possessed by the second processing unit 107, a storage medium connected to the classification device, etc. Step S026 is a step of storing the learned identification model generated in step S025 in the storage unit. The storage unit is the storage unit 101 described in Embodiment 1. Note that the storage unit may be stored in the storage unit possessed by the first processing unit 106, the processing unit 102, or the storage unit possessed by the second processing unit 107, a storage medium connected to the classification device, etc. Note that the storage unit may be stored in the storage unit possessed by the first processing unit 106, the processing unit 102, or the storage unit possessed by the second processing unit 107, a storage medium connected to the classification device, etc. Note that the storage unit may be stored in the storage unit possessed by the first processing unit 106, the processing unit 102, or the storage unit possessed by the second processing unit 107, a storage medium connected to the classification device, etc.​​

[0140] The above is one example of a method for generating a pre-trained classification model. The above training dataset is used. Based on this, by training a discrimination model, a discrimination model with high accuracy in identifying defects can be generated. It is possible.

[0141] According to one aspect of the present invention, a method for classifying image data can be provided.

[0142] This embodiment can be appropriately combined with other embodiments, examples, etc. In this specification, if multiple configuration examples are shown within a single embodiment, the configuration examples are... It is possible to combine them as appropriate.

[0143] (Embodiment 3) In this embodiment, a classification device according to one aspect of the present invention will be described with reference to Figures 9 and 10. ru.

[0144] <Example of a Classification Device Configuration 1> Figure 9 shows a block diagram of the classification device 200. Note that in the drawings attached to this specification, the structure is shown. The components are classified by function and shown as independent blocks in the block diagram, In reality, it is difficult to completely separate the components by function, and one component can be used for multiple functions. It may also be related to Noh. Furthermore, one function may be related to multiple components. For example, the processing performed in processing unit 202 may be executed on different servers depending on the process. be.

[0145] The classification device 200 shown in Figure 9 comprises an input unit 201, a processing unit 202, a storage unit 203, and a database. It has a section 204, a display unit 205, and a transmission line 206.

[0146] [Input section 201] Image data is supplied to the input unit 201 from outside the classification device 200. The first example shows image data with the label shown in the previous embodiment, and image data without the label. This corresponds to image data. The image data supplied to the input unit 201 is transmitted through the transmission line 2 It is supplied to the processing unit 202, the storage unit 203, or the database 204 via 06.

[0147] [Processing 202] The processing unit 202 is supplied with power from the input unit 201, storage unit 203, database 204, etc. It has the function of processing using the data. The processing unit 202 stores the processing result in the storage unit 20 3. It can be supplied to the database 204, display unit 205, etc.

[0148] The processing unit 202 is the same as the processing unit 102 or the second processing unit 107 shown in the previous embodiment, This includes a classifier 103. That is, the processing unit 202 performs processing using the trained discriminant model. It has functions to perform the above, and functions to perform clustering. In addition, the processing unit 202 has functions to perform the above The configuration may include a processing section 105 and a first processing section 106 as shown. The 202 unit has functions for processing image data, augmenting data, and training data sets. It has functions such as generating data and training classification models.

[0149] The processing unit 202 may use a transistor having a metal oxide in the channel formation region. i. Because the off-current of the transistor is extremely small, the transistor can be used as a memory element. It is used as a switch to hold the charge (data) that has flowed into a capacitive element that functions in this way. This ensures that data can be retained for a long period of time. This characteristic is utilized by the processing unit 20 By using it in at least one of the registers and cache memory of 2, the necessary The processing unit 202 is activated only in certain cases; otherwise, the information from the previous processing is saved to the memory element. By doing so, the processing unit 202 can be turned off. In other words, normally This enables computing power, allowing for lower power consumption in the classification device.

[0150] Furthermore, in this specification, etc., a transistor using an oxide semiconductor in the channel formation region is defined as It is called an Oxide Semiconductor transistor (OS transistor). The channel formation region of the S-transistor preferably contains a metal oxide.

[0151] The metal oxide in the channel-forming region preferably contains indium (In). If the metal oxide in the channel-forming region is an indium-containing metal oxide, the OS transition The carrier mobility (electron mobility) of the star increases. Also, the metal present in the channel-forming region The oxide preferably contains element M. Element M is aluminum (Al), gallium ( It is preferable that the element M is Ga (Ga) or tin (Sn). Other elements applicable to element M include These are boron (B), titanium (Ti), iron (Fe), nickel (Ni), and germanium (G). e) Yttrium (Y), Zirconium (Zr), Molybdenum (Mo), Lanthanum (L) a) Cerium (Ce), neodymium (Nd), hafnium (Hf), tantalum (Ta), Examples include tungsten (W). However, as element M, multiple elements mentioned above can be combined. In some cases, this is acceptable. Element M is, for example, an element with a high bond energy with oxygen. Element M is, for example, an element whose bond energy with oxygen is higher than that of indium. Also, The metal oxide in the channel-forming region preferably contains zinc (Zn). Some metal oxides may become more prone to crystallization.

[0152] The metal oxides present in the channel-forming region are not limited to indium-containing metal oxides. The metal oxides present in the channel-forming region are, for example, zinc-tin oxide and gallium-tin oxide. Metal oxides that do not contain indium but contain zinc, metal oxides that contain gallium, etc. It is also acceptable if it is a metal oxide containing ions.

[0153] Furthermore, the processing unit 202 includes a transistor (Si transistor) containing silicon in the channel formation region. You may also use a radiator. Alternatively, you may use graphene, silicene, or chalcone in the channel-forming region. This includes semiconductor materials with band gaps, such as genides (transition metal chalcogenites). A transistor may be used.

[0154] Furthermore, the processing unit 202 includes a transistor containing an oxide semiconductor in the channel formation region, and A transistor containing silicon in the channel formation region may be used in combination with this.

[0155] The processing unit 202 is, for example, an arithmetic circuit or a central processing unit (CPU: Central Processing Unit). It has a processing unit, etc.

[0156] The processing unit 202 is a DSP (Digital Signal Processor), G Microprocessors such as PUs (Graphics Processing Units) It may have. Microprocessors are FPGAs (Field Programmers). ble Gate Array), FPAA (Field Programmable PLDs (Programmable Logic Decoders) such as Analog Arrays The configuration may also be implemented by (vice). The processing unit 202 is operated by the processor By interpreting and executing instructions from various programs, various data processing and programs Control can be performed. The programs that can be executed by the processor are those that the processor possesses. It is stored in at least one of the memory area and the storage unit 203.

[0157] The processing unit 202 may have main memory. Main memory may be volatile memory such as RAM. It has at least one of memory and non-volatile memory such as ROM.

[0158] For example, RAM can be DRAM (Dynamic Random Access Memory). memory), SRAM (Static Random Access Memory) These are used, and a memory space is virtually allocated and used as the workspace for the processing unit 202. The operating system and application programs stored in memory unit 203 are... The program modules, program data, and lookup tables are used during execution. These data, programs, and are loaded into RAM. Each program module is directly accessed and operated by the processing unit 202.

[0159] The ROM contains BIOS (Basic Input / Output) which does not require rewriting. It can store the ut System and firmware, etc. As for ROM, Mask ROM, OTPROM (One Time Programmable Read) Only Memory), EPROM (Erasable Programmability) Examples include e Read Only Memory. EPROMs are also available in ultraviolet light. UV-EPROM (Ultra-Violet) allows for the erasure of stored data by irradiation. Erasable Programmable Read Only Memory) , EEPROM (Electrically Erasable Programmable) Examples include Read Only Memory (LEM), flash memory, and others.

[0160] Furthermore, in neural networks, multiply-accumulate operations are performed. When performed by software, it is preferable that the processing unit 202 has a multiply-accumulate circuit. The sum-of-accumulate circuit may be a digital circuit or an analog circuit. When using analog circuits in the sum-of-accumulate operation circuit, the circuit size of the sum-of-accumulate operation circuit can be reduced, or The goal is to improve processing speed and reduce power consumption by reducing the number of memory accesses. Yes, it is possible. Furthermore, this sum-of-products operation may be performed in software using a program.

[0161] The multiply-accumulate circuit may be constructed using Si transistors, or OS transistors. This configuration is also possible. In particular, OS transistors have an extremely small off-current, so multiply-accumulate operations It is suitable as a transistor for constituting the analog memory of the circuit. A multiply-accumulate circuit may be constructed using both a transistor and an OS transistor.

[0162] [Storage section 203] The storage unit 203 has the function of storing the program to be executed by the processing unit 202. 203 stores programs such as identification models and clustering methods. It has the function of recording image data and the like supplied to the input unit 201. It may also have the ability to remember.

[0163] The storage unit 203 includes at least one of volatile memory and non-volatile memory. The memory unit 203 may have, for example, volatile memory such as DRAM or SRAM. Memory section 203 is, for example, ReRAM (Resistive Random Access). Memory (also called resistive random-access memory), PRAM (Phase change Random Access Memory), FeRAM (Ferroelectri c Random Access Memory), MRAM (Magnetoresi (Also known as magnetically resistive random access memory.) , or it may have non-volatile memory such as flash memory. Also, storage unit 20 3 refers to hard disk drives (HDDs) and solids Solid State Drive (SSD) and other recording media It may have a drive.

[0164] [Database 204] The classification device 200 may have a database 204. For example, database 2 04 has the function of storing the above image data. Note that the learning generated by the processing unit 202 Image data related to the dataset, a trained classification model, and clustering results. You may also memorize things like these.

[0165] Note that the memory unit 203 and the database 204 do not necessarily have to be separated from each other. The classification device 200 has the functions of both a storage unit 203 and a database 204. It may have units.

[0166] Furthermore, the memory of the processing unit 202, the storage unit 203, and the database 204 is These can be considered examples of non-temporary computer-readable storage media.

[0167] [Display section 205] The display unit 205 has the function of displaying the processing results in the processing unit 202. For example, a table The display unit 205 has the function of displaying the clustering results.

[0168] The classification device 200 may also have an output unit. The output unit supplies data to the outside. It has the function of doing so.

[0169] [Transmission path 206] The transmission line 206 has the function of transmitting various types of data. Input section 201, Processing section 202, Data transmission and reception between the storage unit 203, the database 204, and the display unit 205 is performed via the transmission path This can be done via 206. For example, data such as image data and trained classification models. The data is transmitted and received via transmission line 206.

[0170] <Example of a Classification Device Configuration 2> Figure 10 shows a block diagram of the classification device 210. The classification device 210 consists of a server 220 and It has a terminal 230 (such as a personal computer).

[0171] Server 220 comprises a processing unit 202, a transmission line 212, a storage unit 213, and a communication unit 217a. It has. Although not shown in Figure 10, the server 220 further has an input section, an output section, etc. It's fine if you do that.

[0172] Terminal 230 consists of an input unit 201, a storage unit 203, a display unit 205, a transmission line 216, and a communication unit 2 It has 17b and a processing unit 218. Although not shown in Figure 10, terminal 230 further has It is acceptable to have a database or similar system.

[0173] The image data, identification model, etc., received by the communication unit 217a are transmitted via the transmission line 212. It is stored in memory unit 213. Alternatively, image data, identification models, etc. are stored in communication unit 217a. Alternatively, it may be supplied directly to the processing unit 202.

[0174] The training of the discrimination model described in the previous embodiment requires high processing power. The processing unit 202 in terminal 220 has higher processing power than the processing unit 218 in terminal 230. Therefore, it is preferable that the training of the discriminant model be performed in the processing unit 202.

[0175] Then, the processing unit 202 creates a trained discrimination model. The signal is supplied from the processing unit 202 to the communication unit 217a via the transmission line 212 or directly. The trained identification model is transmitted from the communication unit 217a of the server 220 to the communication unit of the terminal 230. It is transmitted to 217b and stored in memory unit 203. Alternatively, the trained discriminant model is transmitted The data may be stored in the storage unit 213 via the transmission path 212.

[0176] [Transmission lines 212 and 216] Transmission lines 212 and 216 have the function of transmitting data. Processing unit 202, Data transmission and reception between the memory unit 213 and the communication unit 217a is performed via the transmission line 212. This is possible. Input unit 201, storage unit 203, display unit 205, communication unit 217b, and processing unit. Data transmission and reception between 218 can be performed via the transmission line 216.

[0177] [Processing Units 202 and 218] The processing unit 202 uses the data supplied from the storage unit 213 and the communication unit 217a, etc. , has the function of processing. The processing unit 218 has an input unit 201, a storage unit 203, and a display unit 20 5. It has the function of processing data supplied from the communication unit 217b and other sources. Processing units 202 and 218 can refer to the description of processing unit 202. Processing unit 202 is, It is preferable that the processing capacity is higher than that of the processing unit 218.

[0178] [Storage section 203] The memory unit 203 has the function of storing the program to be executed by the processing unit 218. The memory unit 203 stores the trained identification model generated by the processing unit 202, and the processing unit 218 generates As a result of the clustering, the data input to the communication unit 217b and the data input to the input unit 201 It has the function of storing the data that has been collected.

[0179] [Storage section 213] The memory unit 213 has the function of storing the program to be executed by the processing unit 202. The memory unit 213 has the function of storing the identification model, data input to the communication unit 217a, etc. It has. The memory unit 213 can refer to the description of the memory unit 203.

[0180] [Communication section 217a and communication section 217b] Using communication units 217a and 217b, data is transmitted between the server 220 and the terminal 230. It can send and receive data. Communication units 217a and 217b include a hub, Routers, modems, etc. can be used. Data can be transmitted and received using either a wired or wireless connection. For example, radio waves, infrared rays, etc. may be used.

[0181] Note that communication between server 220 and terminal 230 is via the World Wide Web (WWW). The foundation of ) is the internet, intranet, extranet, PAN (Pers onal Area Network), LAN(Local Area Network) k), CAN (Campus Area Network), MAN (Metropol Itan Area Network), WAN (Wide Area Network) ), computer networks such as GAN (Global Area Network) This can also be done by connecting to [the appropriate network].

[0182] This embodiment can be appropriately combined with other embodiments, examples, etc.

[0183] (Embodiment 4) In this embodiment, a pattern inspection apparatus according to one aspect of the present invention will be described with reference to Figure 11. The pattern inspection apparatus of this embodiment has a classification apparatus as described in the previous embodiment. .

[0184] Figure 11 shows the configuration of the pattern inspection device 400. As shown in Figure 11, The inspection device 400 comprises an imaging device 401, an inspection device 402, and a classification device 403. The classification device 403 corresponds to the classification device 100 described in the previous embodiment.

[0185] The imaging device 401 captures a semiconductor element that is in the middle of the manufacturing process, or one that has completed the manufacturing process. It has the function of imaging semiconductor elements. An imaging device 401 is, for example, a camera. By imaging the semiconductor element, image data is obtained in which the presence or absence of defects has not been determined. In other words, the image data in question is image data that can be used for identification or clustering. That is the case.

[0186] The inspection device 402 detects that the image data acquired using the imaging device 401 contains defects. It has a function to determine whether or not the image data contains defects. It is possible to make a judgment.

[0187] The determination of whether or not a defect is present is made based on the image data subject to the determination and the data obtained immediately before it. This is done by comparing it with the image data that was used for the determination. The semiconductor elements included in this image are different from the semiconductor elements included in the image data acquired just before this one. Yes. For example, first, the image data subject to the judgment and the image acquired immediately before it. The difference between the data and the actual data is obtained. Then, based on that difference, a determination is made as to whether or not defects are present. That's fine.

[0188] Furthermore, machine learning may be used to determine whether or not defects are present. The number of image data points used to determine whether something is true or false can be enormous. Therefore, by using machine learning... This can reduce the time required for making such a decision.

[0189] To determine whether or not a defect is present, a method similar to that used for detecting abnormal areas is used. This is possible. Unsupervised learning is sometimes used to detect abnormal areas. For the analysis, it is preferable to use unsupervised learning. By using unsupervised learning, the defects are included. Even with a small number of image data points, it is possible to accurately determine whether or not defects are present. can.

[0190] Furthermore, supervised learning may be used to detect abnormal areas. Supervised learning may be used for this. By using supervised learning, it is possible to determine whether or not there are defects. This allows for accurate judgment.

[0191] The above machine learning uses neural networks (especially deep learning). This is preferable.

[0192] Image data that is determined to contain defects will be subject to identification or clustering. In other words, the image data in question is one of the multiple image data 50 described in the previous embodiment. This is possible. The image data is supplied to the classification device 403.

[0193] The pattern inspection device 400 includes an imaging device 401, an inspection device 402, and a classification device 403. Having allows for the clustering of image data and / or the raw data of the trained discriminant model. In addition to processing, it can also acquire image data and determine whether or not there are defects.

[0194] Note that the inspection device 402 is installed on a different server than the server on which the classification device 403 is installed. It may be provided as follows: Alternatively, the inspection device 402 may be provided on a server equipped with a classification device 403. It is also possible to combine some of the functions of the inspection device 402 and some of the functions of the classification device 403. The server is equipped with other parts of the functions of the inspection device 402, and a different server is equipped with the same server. The classification device 403 may also be provided with other parts of its functions.

[0195] The above is a description of the configuration of the pattern inspection device 400. This is one aspect of the present invention. By using a pattern inspection device, from image data acquisition to image data clustering This allows for highly efficient execution of the entire process up to that point. Furthermore, this process can be fully automated. It is possible.

[0196] According to one aspect of the present invention, a novel pattern inspection device can be provided.

[0197] This embodiment can be appropriately combined with other embodiments, examples, etc. [Examples]

[0198] In this example, a trained discrimination model is used to cluster image data containing defects. The clustering was performed. The results of the image data clustering are shown in Figures 12 and 13. explain.

[0199] In this example, a CNN was used as the discrimination model. This CNN has seven convolutional layers. It consists of six pooling layers and five fully connected layers (fully connected layers 913_1 to 913_5). This is achieved. Neurons located in the fully connected layer 913_1 are located in one pooling layer. The neurons that are connected, and the neurons located in the fully connected layer 913_2 Neurons located in the fully connected layer 913_2 are located in the fully connected layer 913_1. The neurons that are connected, and the neurons located in the fully connected layer 913_3 Neurons located in the fully connected layer 913_3 are located in the fully connected layer 913_2. The neurons that are connected, and the neurons located in the fully connected layer 913_4 Neurons located in the fully connected layer 913_4 are located in the fully connected layer 913_3. The neurons that are connected, and the neurons located in the fully connected layer 913_5 It is configured such that the fully connected layer 913_5 is used as the output layer. Furthermore, the fully connected layers 913_1 to the full The bonding layer 913_4 is included in the intermediate layer. For example, it is provided in the first fully bonded layer. The neurons are connected to neurons located in the second fully connected layer. In other words, the first fully connected layer is connected to the second fully connected layer.

[0200] In this embodiment, the numerical values ​​of the neurons in the fully connected layer 913_2 are input to the input layer. These were used as features in the image data. The dimensionality of these features was set to 64.

[0201] In this example, clustering was performed on image data containing 344 defects. Specifically Using the above identification model, the features of each of the 344 defective image data are extracted. Based on these features, cluster analysis was performed. This cluster analysis involved hierarchical analysis. A specific method was used.

[0202] The clustering results are shown in Figure 12. Figure 12 illustrates the clustering results. This is an endogram. The vertical axis represents the distance between clusters. The horizontal axis represents image data. The image data is arranged appropriately so that it is grouped by cluster.

[0203] In the dendrogram shown in Figure 12, the threshold for the distance between clusters is 34.1 (Figure 12). By setting it to the dashed line shown in 12, 20 clusters were obtained. 20 clusters A portion of the image data contained in one of the images is shown in Figures 13A to 13D. It was found that each of the 3D image data contained the same type of pattern defect. In other words, the cluster in question is a cluster related to a portion of the pattern defects. I understand.

[0204] Therefore, by using the method shown in this embodiment, similar defects can be clustered together. Rasterization can be performed.

[0205] This embodiment can be implemented in appropriate combination with the configurations described in other embodiments. It is Noh. [Explanation of symbols]

[0206] 50: Image data, 51_s: Image data, 51_1: Image data, 51_2: Image data Ta, 52_t: image data, 52_1: image data, 53_n: image data, 53_1: Image data, 53_2: Image data, 53a_n: Image data, 53a_1: Image data , 53a_2: Image data, 54_p: Image data, 54_1: Image data, 54a_p :Image data, 54a_1:Image data, 61A:Label, 61F:Label, 62_n: Feature, 62_1: Feature, 62_2: Feature, 63_n: Value, 63_1: Value, 63_2 :Value, 100:Classification device, 100A:Classification device, 100B:Classification device, 101:Storage unit, 102: Processing unit, 103: Classifier, 104: Output unit, 105: Processing unit, 106: Processing unit, 107: Processing unit, 200: Classification device, 201: Input unit, 202: Processing unit, 203: Storage unit , 204: Database, 205: Display unit, 206: Transmission line, 210: Classification device, 212 : transmission line, 213: memory unit, 216: transmission line, 217a: communication unit, 217b: communication unit, 2 18: Processing unit, 220: Server, 230: Terminal, 300: Neural network, 30 1_k: layer, 301_k-1: layer, 301_k-2: layer, 301_1: layer, 301_2: Layer, 301_3: Layer, 305: Feature, 310: CNN, 311_m: Layer, 311_1: Layer, 311_2: Layer, 312_m: Pooling layer, 312_m-1: Pooling layer, 31 2_1: Pooling layer, 313: Fully connected layer, 313_1: Fully connected layer, 313_2: Fully connected Layer, 313_3: Fully connected layer, 315: Feature quantity, 400: Pattern inspection device, 401: Imaging Equipment, 402: Inspection equipment, 403: Classification equipment, 913_1: Fully connected layer, 913_2: Fully connected combined layer, 913_3: fully connected layer, 913_4: fully connected layer, 913_5: fully connected layer

Claims

[Claim 1] In the first step, first to nth (where n is an integer of 2 or more) image data are supplied to the processing unit. In the second step, the processing unit is used to extract the first to nth feature quantities from the first to nth image data based on the identification model. In the third step, the first to nth feature quantities are supplied to the classifier. An image classification method comprising, in the fourth step, using the classifier to cluster the first to nth image data based on the first to nth feature quantities, Each of the first to nth image data is image data that has been determined to contain defects. The aforementioned identification model has an input layer, an intermediate layer, and an output layer. The feature quantities output from the processing unit are numerical values ​​of neurons in the intermediate layer. Image classification method.

Images