Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus configured to prevent a measurement image from being misclassified, and an image processing method.SOLUTION: An image processing apparatus (30) for classifying a measurement image (327) of an inspection object includes: a calculation unit (350) which calculates an evaluation value (328) of the measurement image (327) on the basis of a feature quantity of the measurement image (327) and a feature quantity of a learning image; and a determination unit (360) which determines whether the feature quantity of the measurement image (327) is an outlier with respect to the feature quantity of the learning image, on the basis of the evaluation value (328) of the measurement image (327).SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method.

Background Art

[0002] Conventionally, as this type of information processing apparatus, there is known an information processing apparatus for detecting a defect of an object, including: an acquisition unit that acquires a group of images captured by irradiating the object with light from a plurality of directions; a generation unit that generates a luminance profile for each small area of the image based on the luminance information of each image included in the group of images; and a detection unit that detects a defect in the object based on the luminance profile generated for each small area (see Patent Document 1). The information processing apparatus of Patent Document 1 detects a defect of an object without using data obtained by pre-learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, in a measurement image obtained by photographing an inspection object, it may be required to classify the type of the inspection object, for example, whether there is a defect or not, and the type of the defect when there is a defect.

[0005] Therefore, a method for classifying the measurement image based on the feature amount of the measurement image and the feature amount of a learning image whose classification is known in advance has been proposed.

[0006] However, when the feature amount of the measurement image is an outlier compared to the feature amount of the learning image, such a measurement image does not belong to any of the existing classifications, but the above method gives the closest classification. As a result, the measurement image may be misclassified.

[0007] The present invention has been made in view of such circumstances, and one of its objects is to provide an image processing apparatus and an image processing method capable of suppressing misclassification of measurement images.

Means for Solving the Problems

[0008] An image processing apparatus according to an aspect of the present disclosure is an image processing apparatus for classifying a measurement image of an object to be inspected, and includes a calculation unit that calculates an evaluation value of the measurement image based on the feature amount of the measurement image and the feature amount of the learning image, and a determination unit that determines whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image based on the evaluation value of the measurement image.

[0009] According to this aspect, based on the evaluation value of the measurement image calculated based on the feature amount of the measurement image and the feature amount of the learning image, it is determined whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image. Thereby, it becomes possible to exclude the measurement image as, for example, a measurement error or the like outside the existing classification. Therefore, misclassification of the measurement image can be suppressed.

[0010] In the above-described aspect, the calculation unit may calculate the evaluation value of the measurement image based on the distance between the point of the measurement image and the point of the learning image in the feature space represented by a plurality of feature amounts.

[0011] According to this aspect, the evaluation value of the measurement image is calculated based on the distance between the point of the measurement image and the point of the learning image in the feature space represented by a plurality of feature amounts. Thereby, it is possible to easily calculate the evaluation value of the measurement image for evaluating whether the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image.

[0012] In the above-described aspect, the calculation unit may calculate an evaluation value of the measurement image based on the density between the point of the measurement image and the point of the learning image in the feature space represented by a plurality of feature amounts.

[0013] According to this aspect, the evaluation value of the measurement image is calculated based on the density between the point of the measurement image and the point of the learning image in the feature space represented by a plurality of feature amounts. Thereby, an evaluation value of the measurement image with higher accuracy can be calculated simply and easily.

[0014] In the above-described aspect, a classification unit that classifies the measurement image based on the learning image and the classification given to the learning image may be further provided.

[0015] According to this aspect, the measurement image is classified based on the learning image and the classification given to the learning image. Thereby, the classification to which the measurement image belongs can be estimated from the learning image whose classification is known in advance.

[0016] In the above-described aspect, the measurement image and the learning image are each an image of an inspection object that may include a defect, and the classification may include a class indicating that there is no defect and a class indicating the type of the defect when there is a defect.

[0017] According to this aspect, the measurement image and the learning image are each an image of an inspection object that may include a defect, and the classification includes a class indicating that there is no defect and a class indicating the type of the defect when there is a defect. Thereby, it becomes possible to classify the presence or absence and the type of the defect of the measurement image.

[0018] In the above-described aspect, the classification unit may input the feature amount of the measurement image into a learned model learned using the feature amount and the classification of the learning image, and output the classification of the measurement image from the learned model.

[0019] According to this aspect, the feature amount of the measurement image is input to the learned model trained using the feature amount and classification of the learning image, and the classification of the measurement image is output from the learned model. Thereby, a more accurate classification of the measurement image can be obtained easily and simply.

[0020] In the above-described aspect, a learning unit may be further provided that uses the feature amount of the learning image as an input, trains a learning model using the classification of the learning image as label data, and generates a learned model.

[0021] According to this aspect, a learning model is trained using the feature amount of the learning image as an input and the classification of the learning image as label data, and a learned model is generated. Thereby, a learned model can be obtained without a learned model generation device.

[0022] In the above-described aspect, a selection unit may be further provided that selects at least one from a plurality of feature amounts based on respective indexes of the plurality of feature amounts. The learning unit uses the selected feature amount in the learning image as an input, trains a learning model using the classification of the learning image as label data, generates a learned model, and the classification unit inputs the selected feature amount in the measurement image to the learned model and outputs the classification of the measurement image from the learned model.

[0023] According to this aspect, a learned model is generated by training a learning model using, as an input, a feature amount selected from a plurality of feature amounts in a learning image and using the classification of the learning image as label data. The classification unit inputs the selected feature amount in the measurement image to the learned model and outputs the classification of the measurement image from the learned model. Thereby, the learning time can be shortened, and overfitting of the learned model and a decrease in the classification accuracy of the measurement image can be suppressed.

[0024] In the above-described aspect, the calculation unit may calculate an evaluation value of the measurement image based on the selected feature amount in the measurement image and the selected feature amount in the learning image.

[0025] According to this aspect, an evaluation value of the measurement image is calculated based on the selected feature amount in the measurement image and the selected feature amount in the learning image. As a result, the number (dimension) of the feature amounts is reduced, so that an evaluation value of the measurement image with higher accuracy can be calculated with a smaller amount of calculation.

[0026] In the above-described aspect, an output unit that outputs the suitability of the measurement image may be further provided based on the determination result by the determination unit and the classification by the classification unit.

[0027] According to this aspect, the suitability of the measurement image is output based on the determination result by the determination unit and the classification by the classification unit. As a result, a measurement image that is not suitable for classification or includes a defect can be easily excluded.

[0028] In the above-described aspect, the determination unit may determine whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image based on a comparison between the evaluation value of the measurement image and a threshold value.

[0029] According to this aspect, it is determined whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image based on a comparison between the evaluation value of the measurement image and a threshold value. As a result, it is possible to easily determine whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image.

[0030] In the above-described aspect, the calculation unit may calculate an evaluation value of one learning image based on the feature amount of one learning image and the feature amount of another learning image, and the threshold value may be set based on the evaluation value of the learning image.

[0031] According to this aspect, the threshold value is set based on the evaluation value of the learning image. As a result, an appropriate threshold value can be easily set.

[0032] In the above-described aspect, a creation unit that creates a graph in which points of the measurement image and points of the learning image are plotted in a coordinate system based on the feature amount of the measurement image and the feature amount of the learning image may be further provided.

[0033] According to this aspect, a graph is created in which points of the measurement image and points of the learning image are plotted in a coordinate system based on the feature amounts of the measurement image and the feature amounts of the learning image. Thereby, it becomes possible to visually confirm whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image.

[0034] An image processing method according to another aspect of the present disclosure is an image processing method for classifying a measurement image of an inspection object, including: a step of calculating an evaluation value of the measurement image based on the feature amount of the measurement image and the feature amount of the learning image; and a step of determining whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image based on the evaluation value of the measurement image.

[0035] According to this aspect, based on the evaluation value of the measurement image calculated based on the feature amount of the measurement image and the feature amount of the learning image, it is determined whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image. Thereby, it becomes possible to exclude the measurement image as, for example, a measurement error or the like other than existing classifications. Therefore, misclassification of the measurement image can be suppressed.

Advantageous Effects of the Invention

[0036] According to the present invention, misclassification of a measurement image can be suppressed.

Brief Description of the Drawings

[0037]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

DETAILED DESCRIPTION OF THE INVENTION

[0038] Embodiments of the present invention will be described below. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, the drawings are schematic. Therefore, specific dimensions and the like should be determined in light of the following description. Of course, there are also parts where the dimensional relationships and ratios are different between the drawings. Furthermore, the technical scope of the present invention should not be construed as being limited to the embodiments.

[0039] First, with reference to FIG. 1, the configuration of an image inspection system according to an embodiment will be described. FIG. 1 is a configuration diagram illustrating a schematic configuration of an image inspection system 1 in an embodiment.

[0040] As shown in FIG. 1, the image inspection system 1 includes a learned model generation device 10, an image inspection device 20, an image processing device 30, and illumination IL. The learned model generation device 10, the image inspection device 20, and the image processing device 30 are communicably connected to each other via a communication network NW. The illumination IL irradiates the inspection object TA with light L. The image inspection device 20 captures the reflected light R and inspects the inspection object TA based on an image obtained by measuring the inspection object TA (hereinafter also referred to as a "measurement image"). More specifically, the image inspection device 20 determines whether the inspection object TA is a non-defective product or a defective product based on the measurement image, or determines whether the inspection object TA includes a defect.

[0041] The learned model generation device 10 generates a learned model used by the image processing device 30 to classify the measurement image. Further, the learned model generation device 10 may generate a learned model used by the image inspection device 20 to perform the inspection.

[0042] The image processing device 30 is for classifying the measurement image of the inspection object TA. The measurement image is an image of the inspection object TA, and defects may occur in the inspection object TA, for example, in the manufacturing process. The defects are not particularly limited, and include, for example, visible ones such as chips, scratches, light stains, dark stains, cracks, burrs, deposits, foreign substances, dents, unevenness in color, etc., blurring of printing, and misalignment of printing, etc. Therefore, the measurement image may include a partial image of such a defect (hereinafter also simply referred to as a "defect").

[0043] As described above, the image inspection device 20 can determine whether the inspection object TA includes a defect based on the measurement image. On the other hand, the image processing device 30 can classify the presence or absence of a defect in the measurement image and the type of the defect when there is a defect based on the measurement image. Therefore, the image processing device 30 is useful, for example, for analyzing and interpreting the measurement image, and estimating and / or narrowing down the cause of the defect that may be included in the measurement image.

[0044] Next, while referring to FIG. 2, the physical configurations of the learned model generation device, the image inspection device, and the image processing device according to one embodiment will be described. FIG. 2 is a configuration diagram showing the physical configurations of the learned model generation device 10, the image inspection device 20, and the image processing device 30 in one embodiment.

[0045] As shown in FIG. 2, the learned model generation device 10, the image inspection device 20, and the image processing device 30 each include a processor 31, a memory 32, a storage device 33, a communication device 34, an input device 35, and an output device 36. These components are connected to be able to transmit and receive data to and from each other via a bus.

[0046] The processor 31 is configured to control the operations of each part of the learned model generation device 10, the image inspection device 20, and the image processing device 30. The processor 31 includes, for example, an integrated circuit such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), or a SoC (System-on-a-Chip).

[0047] The memory 32 and the storage device 33 are each configured to store programs, data, and the like. The memory 32 is composed of, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and / or a RAM (Random Access Memory). The storage device 33 is composed of, for example, a storage such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), and / or an eMMC (embedded Multi Media Card).

[0048] The communication device 34 is configured to communicate via at least one of a wired and a wireless network. The communication device 34 includes, for example, a network card, a communication module, an interface for connecting to other devices, etc.

[0049] The input device 35 is configured to be able to input information by a user's operation. The input device 35 includes, for example, a keyboard, a touch panel, a mouse, a pointing device, and / or a microphone, etc.

[0050] The output device 36 is configured to output information. The output device 36 includes, for example, a display device such as a liquid crystal display, an EL (Electro Luminescence) display, a plasma display, an LCD (Liquid Crystal Display), and / or a speaker, etc.

[0051] In the example shown in FIG. 2, the case where the learned model generation device 10, the image inspection device 20, and the image processing device 30 are each configured by a single computer will be described, but it is not limited thereto. The learned model generation device 10, the image inspection device 20, and the image processing device 30 may each be realized by a combination of a plurality of computers. Further, the configuration shown in FIG. 2 is an example, and the learned model generation device 10, the image inspection device 20, and the image processing device 30 may each have other configurations, or may not have some of these configurations.

[0052] Next, while referring to FIGS. 3 to 12, the functional blocks of the image processing apparatus according to an embodiment will be described. FIG. 3 is a configuration diagram showing the configuration of the functional blocks of the image processing apparatus 30 in an embodiment. FIG. 4 is a diagram showing an example of the learning data 321. FIG. 5 is a diagram showing an example of the feature amounts and evaluation values 328 of the measurement image 327 and the feature amounts and evaluation values 322 of the learning image 321a. FIG. 6 is a diagram showing an example of the measurement image 327. FIG. 7 is a diagram showing a first example of the output screen OS. FIG. 8 is a diagram showing a second example of the output screen OS. FIG. 9 is a diagram showing a third example of the output screen OS. FIG. 10 is a diagram showing a fourth example of the output screen OS. FIG. 11 is a diagram showing an example of the graph created by the creation unit 380. FIG. 12 is a diagram showing an example of the feature amount selection screen SS.

[0053] As shown in FIG. 3, the image processing apparatus 30 includes a communication unit 310, a storage unit 320, a learning unit 330, a classification unit 340, a calculation unit 350, a determination unit 360, an output unit 370, a creation unit 380, and a selection unit 390.

[0054] The communication unit 310 is configured to be able to transmit and receive various types of information. The communication unit 310 receives, for example, a plurality of learning data 321 from the learned model generation apparatus 10 or another apparatus via the communication network NW. Further, the communication unit 310 receives, for example, the measurement image 327 from the image inspection apparatus 20 or another apparatus via the communication network NW. Furthermore, the communication unit 310 receives, for example, the learned model 325 from the learned model generation apparatus 10 via the communication network NW. The received plurality of learning data 321, learned model 325, and measurement image 327 are written and stored in the storage unit 320. Note that the communication unit 310 may receive only the plurality of learning data 321 and the measurement image 327. When the communication unit 310 receives only the plurality of learning data 321 and the measurement image 327, the learning unit 330 described later generates the learned model 325 using the plurality of learning data 321.

[0055] The storage unit 320 is configured to store various types of information. The storage unit 320 stores, for example, a plurality of learning data 321, a learned model 325, and a measurement image 327. By providing the storage unit 320 that stores the learned model 325 in this way, the learned model can be easily read out.

[0056] The learning unit 330 is configured to perform machine learning of a learning model using the plurality of learning data 321 stored in the storage unit 320 and generate a learned model 325. The generated learned model 325 is written and stored in the storage unit 320.

[0057] More specifically, the learning unit 330 is configured to use the feature amount of the learning image included in the learning data 321 as an input, and learn a learning model using the classification of the learning image as label data, thereby generating a learned model 325.

[0058] As shown in FIG. 4, each of the plurality of learning data 321 includes a learning image 321a and label data 321b. That is, each learning data 321 is composed of a combination of a learning image 321a and label data 321b. The learning image 321a is an image of an inspection object TA that may include defects, similar to the measurement image 327 described above.

[0059] Generally, label data is data that represents the nature of input data input to a learning model during machine learning of the learning model. In other words, label data is data that the learning model should output and is the target data for learning. In the present embodiment, the label data 321b is a classification of the corresponding learning image 321a, strictly speaking, a classification based on the feature amount of the corresponding learning image 321a. Thus, in the learning data 321, the learning image 321a is given a classification. This classification includes a class indicating that the learning image 321a does not include a defect (hereinafter also referred to as "no defect" or "defect-free"), and a class indicating the type of the defect when the learning image 321a includes a defect (hereinafter also referred to as "having a defect" or "defect present").

[0060] More specifically, a class indicating the classification of the corresponding learning image 321a is set in the label data 321b, and the class is represented by an ID. For example, when the label data 321b is "0" (ID: 0), the corresponding learning image 321a is classified into a class indicating "no defect". Also, when the label data 321b is "1" (ID: 1), the corresponding learning image 321a is "defective", and is classified into a class indicating that the type of the defect is "chip". Further, when the label data 321b is "2" (ID: 2), the corresponding learning image 321a is "defective", and is classified into a class indicating that the type of the defect is "scratch". Also, when the label data 321b is "3" (ID: 3), the corresponding learning image 321a is "defective", and is classified into a class indicating that the type of the defect is "heavy stain". Furthermore, when the label data 321b is "4" (ID: 4), the corresponding learning image 321a is "defective", and is classified into a class indicating that the type of the defect is "light stain".

[0061] In this way, the measurement image 327 and the learning image 321a are each an image of the inspection object TA that may include defects, and by including a class indicating no defect and a class indicating the type of the defect when there is a defect in the classification, it becomes possible to classify the presence or absence and the type of the defect in the measurement image 327.

[0062] Here, a feature amount is a variable that quantitatively represents the feature of a thing to be obtained. When performing machine learning of the learning model, the feature amount of the learning image 321a input to the learning model is, for example, what characterizes an image of the inspection object TA that may include defects. Specifically, examples of the feature amount of the learning image 321a include density values represented by area, the x coordinate of the centroid position, the y coordinate of the centroid position, the flatness ratio, the average density of the image, and the like. Generally, it is rare for the learning image 321a to be recognizable as an image of the inspection object TA by only one type of feature amount, and in many cases, it can be recognized as an image of the inspection object TA by using a plurality of types of feature amounts.

[0063] A feature vector is a vector representation with one or more of these feature quantities as components. The number of feature quantities represents the dimension (number of dimensions) of the feature vector. The space represented by the feature vector, in other words, the space spanned by the feature vector, is called the feature space, and the feature vector is represented as a point in the feature space. Therefore, it can be said that the feature space is a coordinate system based on the feature quantities.

[0064] The learning unit 330 calculates one or more feature quantities from the learning image 321a and inputs the calculated feature quantities into the learning model. Note that the feature quantities input into the learning model may be calculated in advance based on the learning image 321a and included in the learning data 321. Further, when a plurality of feature quantities are used for the learning image 321a, at least one feature quantity selected from the plurality of feature quantities may be input into the learning model as described later.

[0065] For the generation of the learned model 325, for example, learning models such as the k-nearest neighbor method, SVM (Support Vector Machine), decision tree, regression model, and neural network can be used. For example, when the learning model is a neural network, the feature quantities of the learning image 321a are input, and based on the difference between the output and the label data, the weights of the neural network are updated by the error backpropagation method. By performing this process for each of the plurality of learning data 321, the learned model 325 is generated. Note that the learning model is not limited to the above-described ones, and the learning unit 330 may generate the learned model 325 using other algorithms.

[0066] In this way, by using the feature quantities of the learning image 321a as input, training the learning model with the classification of the learning image 321a as label data, and generating the learned model 325, the learned model 325 can be obtained even without the learned model generation device 10.

[0067] In addition, when the communication unit 310 receives the learned model 325 from the learned model generation device 10 via the communication network NW, the learned model generation device 10 has the same functions as the learning unit 330. That is, the learned model generation device 10 trains the learning model in the same way as the description of the learning unit 330 to generate the learned model 325. Therefore, the detailed description of the learned model generation device 10 is omitted.

[0068] Returning to the description of FIG. 3, the classification unit 340 is configured to classify the measurement image 327 based on the learning image 321a and the classification assigned to the learning image 321a. Thereby, the classification to which the measurement image 327 belongs can be estimated from the learning image 321a whose classification is known in advance.

[0069] More specifically, the classification unit 340 inputs the feature amount of the measurement image 327 into the learned model 325 trained using the feature amount and classification of the learning image 321a, and is configured to output the classification of the measurement image 327 from the learned model 325.

[0070] Specifically, the classification unit 340 reads out the learned model 325 and the measurement image 327 stored in the storage unit 320 respectively, inputs the feature amount of the measurement image 327 into the learned model 325, and outputs the classification of the measurement image 327. The classification of the measurement image 327 output from the learned model 325 is written and stored in the storage unit 320. In addition, the classification unit 340 outputs the classification of the output measurement image 327 to the output unit 370. Further, the classification unit 340 outputs the classification of the output measurement image 327 to the output device 36 described above, and the classification is displayed on a display device or the like. Note that the classification unit 340 does not have to use the output of the learned model 325 as it is as the classification of the measurement image 327, and may perform any post-processing.

[0071] In this way, by inputting the feature amount of the measurement image 327 into the learned model 325 trained using the feature amount and classification of the learning image 321a, and outputting the classification of the measurement image 327 from the learned model 325, it is possible to easily obtain a more accurate classification of the measurement image 327.

[0072] In this embodiment, an example in which the classification unit 340 obtains the classification of the measurement image 327 using the learned model 325 is shown, but it is not limited thereto. For example, the classification unit 340 may classify the measurement image 327 based on the Mahalanobis distance. In this case, in a predetermined coordinate system, the classification unit 340 calculates the Mahalanobis distance between the points of the measurement image 327 and each point group (group) formed for each classification of the plurality of learning images 321a. Then, the classification unit 340 determines the classification of the measurement image 327 as belonging to the class of the point group with the minimum Mahalanobis distance.

[0073] Further, the classification unit 340 is not limited to classifying the measurement image 327 by one classification method, and may classify the measurement image 327 using one or more classification methods selected from a plurality of classification methods. In this case, for each classification method, the classification of the measurement image 327 according to the classification method is obtained. Further, the storage unit 320 may store a plurality of learned models learned according to each classification method.

[0074] The calculation unit 350 is configured to calculate an evaluation value of the measurement image 327 based on the feature amounts of the measurement image 327 and the learning image 321a. The calculation unit 350 outputs the calculated evaluation value 328 of the measurement image 327 to the determination unit 360. Further, the evaluation value 328 is written and stored in the storage unit 320. The calculation unit 350 may calculate the evaluation value of the measurement image 327 based on at least one feature amount. When the measurement image 327 and the learning image 321a are characterized by a plurality of feature amounts, the evaluation value of the measurement image 327 may be calculated based on the plurality of feature amounts in the measurement image 327 and the learning image 321a. Further, the calculation unit 350 may calculate the evaluation value of the measurement image 327 based on the feature amounts of at least one learning image 321a. When there are a plurality of learning images 321a, the evaluation value of the measurement image 327 may be calculated based on each of the feature amounts of the plurality of learning images 321a.

[0075] Further, the calculation unit 350 is configured to calculate an evaluation value 322 of the one learning image 321a based on the feature amount of the one learning image 321a and the feature amounts of other learning images 321a. The other learning image 321a may be one or a plurality. When there are a plurality of learning images 321a, the calculation unit 350 calculates the evaluation value 322 of each of the plurality of learning images 321a based on the feature amount of the learning image 321a and the feature amounts of other learning images 321a. The calculated evaluation value 322 is written and stored in the storage unit 320.

[0076] Here, as shown in FIG. 5, the measurement image 327 and the plurality of learning images 321a are each characterized by, for example, five feature amounts from the first feature amount to the fifth feature amount. The first feature amount is, for example, the area of the image, and the unit is pixels. The second feature amount is, for example, the x coordinate of the center of gravity of the image, and the unit is pixels. The third feature amount is, for example, the y coordinate of the center of gravity of the image, and the unit is pixels. The fourth feature amount is, for example, the flatness ratio of the image, which is a value from zero to one and dimensionless (unitless). The fifth feature amount is, for example, the shading value of the image, which is the brightness from zero to 255 and dimensionless (unitless).

[0077] As described above, each of the plurality of learning images 321a is given a classification as label data 321b. This classification includes five classes identified by IDs from "0" to "4". Further, the measurement image 327 is given a classification by the classification unit 340. In the example shown in FIG. 5, the classification of the measurement image 327 is the class identified by the ID "4", and in the example described with reference to FIG. 4, it is "with defect", and the type of the defect is "thin stain".

[0078] The measurement image 327 in the example shown in FIG. 5 is, for example, an image as shown in FIG. 6. This measurement image 327 does not include the types of defects classified into the "with defect" classes identified by the above-mentioned IDs from "1" to "4". On the other hand, compared with the learning image 321a shown in FIG. 4, which is classified into the "without defect" class identified by the ID "1", the measurement image 327 shown in FIG. 6 has a notch in the lower right part and a different outer shape. Therefore, it is considered that the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a. Thus, the measurement image 327 with an outlier feature amount may also include, for example, when the inspection object TA is a semi-finished product, when photographing the inspection object TA, the position of the inspection object TA is shifted and a part of the inspection object TA is not included, or when the measurement image 327 does not include an image of the inspection object TA.

[0079] Conventionally, when the feature amount of the measurement image 327 is an outlier compared to the feature amount of the learning image 321a, although the measurement image should not belong to any of the existing classifications in principle, the closest classification among the classifications given to the learning images has been given. As a result, the measurement image 327 may be misclassified.

[0080] In contrast, in the image processing apparatus 30 of the present embodiment, the calculation unit 350 calculates the evaluation value 328 of the measurement image 327 based on the feature amount of the measurement image 327 and the feature amount of the learning image 321a. In the example shown in FIG. 5, the evaluation value 328 of the measurement image 327 is "3.22". Further, for each of the plurality of learning images 321a, the calculation unit 350 calculates the evaluation value 322 of the learning image 321a based on the feature amount of the learning image 321a and the feature amount of the other learning images 321a. In the example shown in FIG. 5, the maximum value among the evaluation values 322 of each learning image 321a is "2.01". The evaluation value 328 of the measurement image 327 is relatively high, for example, when compared with the evaluation values 322 of each of the plurality of learning images 321a.

[0081] The evaluation value of the measurement image 327 is for evaluating whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a. In other words, the evaluation value of the measurement image 327 is a value that serves as a criterion for determining whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a. Therefore, it is preferable that the evaluation value of the measurement image 327 is based on the distribution between the measurement image 327 and the learning image 321a in the feature space represented by the feature amounts of the measurement image 327 and the learning image 321a.

[0082] More specifically, the calculation unit 350 may be configured to calculate the evaluation value of the measurement image 327 based on the distance between the point of the measurement image 327 and the point of the learning image 321a in the feature space represented by the plurality of feature amounts. Specifically, when there are a plurality of learning images 321a, the evaluation value of the measurement image 327 is the distance between the point of the measurement image 327 and the point of the learning image 321a that is closest to the point of the measurement image 327. The distance is not limited to the generally used Euclidean distance, and may be other distances such as the Mahalanobis distance.

[0083] In this way, by calculating the evaluation value of the measurement image 327 based on the distance between the point of the measurement image 327 and the point of the learning image 321a in the feature space represented by a plurality of feature amounts, it is possible to easily calculate the evaluation value of the measurement image 327 for evaluating whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a.

[0084] Further, the evaluation value of the measurement image 327 may preferably be based on, particularly, density among the distributions in the feature space represented by a plurality of feature amounts. More specifically, the calculation unit 350 may be configured to calculate the evaluation value of the measurement image 327 based on the density between the point of the measurement image 327 and the point of the learning image 321a in the feature space represented by a plurality of feature amounts. As a method for estimating the density between the point of the measurement image 327 and the point of the learning image 321a, for example, the Local Outlier Factor (LOF) method is used. When using the Local Outlier Factor method, the evaluation value of the measurement image 327 is calculated based on an index of local density between the point of the measurement image 327 in the feature space and k (k is an integer of 1 or more) points of the learning image 321a in the vicinity of the point of the measurement image 327. By using this Local Outlier Factor method, it is possible to consider the difference in local density, which is particularly effective when the points of the measurement image 327 and the learning image 321a in the feature space are distributed complexly. Note that the density between the point of the measurement image 327 and the point of the learning image 321a in the feature space is not limited to the case of using the Local Outlier Factor method, and for example, methods such as the k-Nearest Neighbor (k-NN) method and One Class SVM may be used.

[0085] In this way, by calculating the evaluation value of the measurement image 327 based on the density between the point of the measurement image 327 and the point of the learning image 321a in the feature space represented by a plurality of feature amounts, it is possible to more easily calculate the evaluation value of the measurement image 327 with higher accuracy.

[0086] Returning to the description of FIG. 3, the determination unit 360 is configured to determine whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a based on the evaluation value 328 of the measurement image 327. The determination unit 360 outputs the determination result to the output unit 370.

[0087] In this way, based on the evaluation value 328 of the measurement image 327 calculated based on the feature amount of the measurement image 327 and the feature amount of the learning image 321a, by determining whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a, it becomes possible to exclude the measurement image 327 as something other than existing classifications, for example, as a measurement error or the like. Therefore, misclassification of the measurement image 327 can be suppressed.

[0088] More specifically, the determination unit 360 is configured to determine whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a based on a comparison between the evaluation value 328 of the measurement image 327 and the threshold value 323. Thereby, it is possible to easily determine whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a.

[0089] The threshold value 323 is set based on the evaluation value 322 of the learning image 321a. The set threshold value 323 is written into and stored in the storage unit 320. When there are a plurality of learning images 321a, the threshold value 323 is set based on each of the evaluation values 322 of the plurality of learning images 321a. The setting of the threshold value is performed, for example, according to an input operation by an operator (user). The operator (user) looks at the evaluation value 322 of the learning image 321a shown in FIG. 5, for example, and sets an appropriate value, for example, "3.00" as the threshold value 323. Alternatively, the calculation unit 350 may set, as the threshold value 323, the maximum value of the calculated evaluation values 322 of the plurality of learning images 321a plus a predetermined value. Note that the threshold value 323 is not limited to being one, and a plurality of threshold values may exist, for example, in the case of an upper limit value and a lower limit value.

[0090] In this way, by setting the threshold value 323 based on the evaluation value 322 of the learning image 321a, an appropriate threshold value 323 can be easily set.

[0091] The output unit 370 is configured to output the suitability of the measurement image 327 based on the determination result by the determination unit 360 and the classification by the classification unit 340. The output unit 370 outputs the suitability of the measurement image 327 to the output device 36 described above, and the suitability of the measurement image 327 is displayed on a display device or the like.

[0092] As shown in FIG. 7, an output screen OS for outputting the suitability of the measurement image 327 is displayed. The output unit 370 generates data necessary for the display of the output screen OS, and outputs the generated data to the output device 36 described above, so that the output screen OS can be displayed on a display device or the like. The output screen OS includes, for example, a first region R11, a second region R12, a third region R13, and a fourth region R14.

[0093] The first region R11 is for displaying the measurement image 327. In the example shown in FIG. 7, a measurement image 327 classified into the class of "no defect" identified by the ID "0" is displayed in the first region R11.

[0094] The second region R12 is for displaying whether the feature amount of the measurement image 327 displayed in the first region R11 is an outlier with respect to the feature amount of the learning image 321a. Based on the determination result input from the determination unit 360, the output unit 370 displays "NG" when the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a, and displays "OK" when the feature amount of the measurement image 327 is not an outlier with respect to the feature amount of the learning image 321a. In the example shown in FIG. 7, since the evaluation value of the measurement image 327 displayed in the first region R11 is, for example, equal to or less than the threshold value 323, "OK" is displayed in the second region R12.

[0095] The third area R13 is for indicating whether the measurement image 327 displayed in the first area R11 includes a defect. The output unit 370 displays "NG" when the measurement image 327 includes a defect and displays "OK" when the measurement image 327 does not include a defect, based on the classification of the measurement image 327 input from the classification unit 340. In the example shown in FIG. 7, since the classification of the measurement image 327 displayed in the first area R11 is the "no defect" class identified by the ID "0", "OK" is displayed in the third area R13.

[0096] The fourth area R14 is for indicating whether the measurement image 327 displayed in the first area R11 is appropriate, that is, for indicating the propriety of the measurement image 327. The output unit 370 displays "NG" when the measurement image 327 is inappropriate and displays "OK" when the measurement image 327 is appropriate, based on the determination result by the determination unit 360 and the classification by the classification unit 340.

[0097] In the present embodiment, the measurement image 327 is appropriate when the feature amount of the measurement image 327 is not an outlier with respect to the feature amount of the learning image 321a and the measurement image 327 does not include a defect. In other words, when the display in the second area R12 is "OK" and the display in the third area R13 is "OK", the measurement image 327 is regarded as appropriate. Conversely, when at least one of the display in the second area R12 and the display in the third area R13 is "NG", the measurement image 327 is regarded as inappropriate. In the example shown in FIG. 7, since the feature amount of the measurement image 327 displayed in the first area R11 is not an outlier with respect to the feature amount of the learning image 321a and its classification is the "no defect" class, "OK" is displayed in the fourth area R14.

[0098] As shown in FIG. 8, when a measurement image 327 identified by ID "1", which has a defect and is classified into the class of "chip" as the type of the defect, is displayed in the first region R11, since the evaluation value of the measurement image 327 is, for example, equal to or less than the threshold value 323, "OK" is displayed in the second region R12. Also, since the classification of the measurement image 327 is the class of "defect" identified by ID "1", "NG" is displayed in the third region R13. Further, although the feature amount of the measurement image 327 is not an outlier with respect to the feature amount of the learning image 321a, since its classification is the class of "defect", "NG" is displayed in the fourth region R14.

[0099] As shown in FIG. 9, when a measurement image 327 identified by ID "2", which has a defect and is classified into the class of "scratch" as the type of the defect, is displayed in the first region R11, since the evaluation value of the measurement image 327 is, for example, equal to or less than the threshold value 323, "OK" is displayed in the second region R12. Also, since the classification of the measurement image 327 is the class of "defect" identified by ID "2", "NG" is displayed in the third region R13. Further, although the feature amount of the measurement image 327 is not an outlier with respect to the feature amount of the learning image 321a, since its classification is the class of "defect", "NG" is displayed in the fourth region R14.

[0100] As shown in FIG. 10, when a measurement image 327 identified by ID "4", which has a defect and is classified into the class of "light stain" as the type of the defect and whose evaluation value is greater than the threshold value, is displayed in the first region R11, since the evaluation value of the measurement image 327 is greater than the threshold value 323, "NG" is displayed in the second region R12. Also, although the classification of the measurement image 327 is the class of "defect" identified by ID "4", since the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a, this classification is a misclassification, and since the measurement image 327 does not contain a defect, "OK" is displayed in the third region R13. Further, since the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a, "NG" is displayed in the fourth region R14.

[0101] In this way, based on the determination result by the determination unit 360 and the classification by the classification unit 340, by outputting the suitability of the measurement image 327, it is possible to easily exclude the measurement image 327 that is not suitable for classification or includes defects.

[0102] Returning to the description of FIG. 3, the creation unit 380 is configured to create a graph in which points of the measurement image 327 and points of the learning image 321a are plotted in a coordinate system based on the feature amounts of the measurement image 327 and the learning image 321a. The creation unit 380 outputs the created graph to the output device 36 described above, and the graph is displayed on a display device or the like.

[0103] Further, the creation unit 380 is configured to be able to change parameters such as an enlargement ratio, a reduction ratio, a rotation angle, and an elevation angle with respect to the graph to be created. Therefore, the operator (user) can arbitrarily change these parameters by an input operation via the input device 35 described above.

[0104] As shown in FIG. 11, the graph is represented by, for example, a three-dimensional (3D) coordinate system. The first coordinate axis in the direction parallel to the X axis is the first feature amount which is, for example, the area of the image, the second coordinate axis in the direction parallel to the Y axis is the second feature amount which is, for example, the x coordinate of the center of gravity of the image, and the third coordinate axis in the direction parallel to the Z axis is the third feature amount which is, for example, the y coordinate of the center of gravity of the image.

[0105] In the example shown in FIG. 11, the point MP of the measurement image 327 indicated by a circle is arranged on the negative side of the second coordinate axis and is arranged at a position away from each point of the plurality of learning images 321a each indicated by a cross. Therefore, it is estimated that the feature amount of the measurement image 327 is an outlier with respect to the feature amounts of the plurality of learning images 321a.

[0106] In addition, if the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a, the points of the measurement image 327 may be plotted at positions far from the points of the learning image 321a and cannot be displayed in the graph. Thus, in order to avoid the points of the measurement image 327 from not being displayed in the graph, the creation unit 380 may perform at least one of normalization and standardization on the feature amount of the measurement image 327 and the feature amount of the learning image 321a, and create a graph in which the points of the measurement image 327 and the points of the learning image 321a are plotted in a coordinate system with the transformed feature amount as the coordinate axis.

[0107] For example, when performing standardization, the creation unit 380 performs standardization on the feature amount of the measurement image 327 and the feature amount of the learning image 321a shown in FIG. 5 based on, for example, the feature amounts of a plurality of learning images 321a, and transforms them into the transformed feature amount of the measurement image 327 and the transformed feature amount of the learning image 321a, respectively.

[0108] More specifically, the creation unit 380 is configured to perform standardization on the feature amount of the measurement image 327 and the feature amount of the learning image 321a using the average and standard deviation calculated based on the feature amounts of a plurality of learning images 321a.

[0109] Specifically, each transformed feature amount Yi (i = 1, 2, 3, 4, 5) of the measurement image 327 can be calculated by the following formula (1) using the i-th feature amount Xi (i = 1, 2, 3, 4, 5) before transformation of the measurement image 327, the average μi calculated based on the i-th feature amount of each learning image 321a, and the standard deviation ρi calculated based on the corresponding i-th feature amount in each learning image 321a. Yi = (Xi - μi) / ρi …(1)

[0110] Similarly, each transformed feature amount Mi (i = 1, 2, 3, 4, 5) of the learning image 321a can be calculated by the following formula (2) using the i-th feature amount Li (i = 1, 2, 3, 4, 5) before transformation of the learning image 321a and the aforementioned average μi and standard deviation ρi. Mi = (Li - μi) / ρi …(2)

[0111] In this way, for the i-th feature amount of the measurement image 327 and the i-th feature amount of the learning image 321a, by performing normalization using the average μi and the standard deviation ρi calculated based on the i-th feature amounts of the plurality of learning images 321a, the average μi becomes "0" and the standard deviation ρi becomes "1". Therefore, even between different feature amounts, it becomes easier to compare the value of the transformed feature amount of the measurement image 327 with the value of the transformed feature amount of the learning image 321a.

[0112] Further, the creation unit 380 is configured to perform normalization on the feature amount of the measurement image 327 and the feature amount of the learning image 321a using the maximum value and the minimum value in the feature amounts of the plurality of learning images 321a instead of, or in addition to, the above-described normalization.

[0113] Specifically, each transformed feature amount Ti (i = 1, 2, 3, 4, 5) of the measurement image 327 can be calculated by the following formula (3) using the i-th feature amount Si (i = 1, 2, 3, 4, 5) before transformation of the measurement image 327, the maximum value MAXi in the i-th feature amount of each learning image 321a, and the minimum value MINi in the i-th feature amount of each learning image 321a. Ti = (Si - MINi) / (MAXi - MINi) …(3)

[0114] Similarly, each transformed feature amount Qi (i = 1, 2, 3, 4, 5) of the learning image 321a can be calculated by the following formula (4) using the i-th feature amount Pi (i = 1, 2, 3, 4, 5) before transformation of the learning image 321a and the above-described maximum value MAXi and minimum value MINi. Qi = (Pi - MINi) / (MAXi - MINi) …(4)

[0115] In this way, for the i-th feature amount of the measurement image 327 and the i-th feature amount of the learning image 321a, by performing normalization using the maximum value MAXi and the minimum value MINi in the i-th feature amounts of the plurality of learning images 321a, the value of the transformed feature amount of the measurement image 327 and the value of the transformed feature amount of the learning image 321a fall within a certain range. Therefore, even between different feature amounts, the comparison between the value of the transformed feature amount of the measurement image 327 and the value of the transformed feature amount of the learning image 321a becomes easy.

[0116] In the above-described standardization and normalization, an example in which it is performed based only on the feature amounts of the plurality of learning images 321a has been shown, but it is not limited thereto. For example, the creation unit 380 may perform at least one of standardization and normalization based further on the feature amount of the measurement image 327. Specifically, the above-described average μi may be calculated from the i-th feature amount of each learning image 321a and the i-th feature amount of the measurement image 327, and the above-described standard deviation ρi may be calculated from the i-th feature amount of each learning image 321a and the i-th feature amount of the measurement image 327. Similarly, for example, the above-described maximum value MAXi may be the maximum value in the i-th feature amounts of each learning image 321a and the i-th feature amount of the measurement image 327, and the above-described minimum value MINi may be the minimum value in the i-th feature amounts of each learning image 321a and the i-th feature amount of the measurement image 327.

[0117] In this way, by performing at least one of the above-described standardization and normalization based further on the i-th feature amount of the measurement image 327, it becomes possible to perform at least one of standardization and normalization more appropriately.

[0118] In this embodiment, although an example in which the graph created by the creation unit 380 is plotted in a three-dimensional coordinate system has been shown, the present invention is not limited thereto. The creation unit 380 may create a graph in which the points of the measurement image 327 and the points of the learning image 321a are plotted in a coordinate system with a dimensionality other than three dimensions, for example, a two-dimensional or four-dimensional or higher coordinate system. Further, the creation unit 380 may display the feature amounts and evaluation values of the measurement image 327 and the learning image 321a in a format other than a graph, for example, in a list format. In this case, at least one of the feature amounts and evaluation values may be used to sort the measurement image 327 and the learning image 321a, and the sorted images may be displayed in descending or ascending order.

[0119] In this way, by creating a graph in which the points of the measurement image 327 and the points of the learning image 321a are plotted in a coordinate system based on the feature amounts of the measurement image 327 and the feature amounts of the learning image 321a, it becomes possible to visually confirm whether the feature amounts of the measurement image 327 are outliers with respect to the feature amounts of the learning image 321a.

[0120] Returning to the description of FIG. 3, the selection unit 390 is configured to select at least one from a plurality of feature amounts based on respective indexes of the plurality of feature amounts. The selected feature amount is written and stored in the storage unit 320.

[0121] As shown in FIG. 12, for example, a feature amount selection screen SS that enables selection of at least one feature amount from a plurality of feature amounts is displayed. The selection unit 390 generates data necessary for displaying the feature amount selection screen SS, and outputs the generated data to the output device 36 described above, so that the feature amount selection screen SS can be displayed on a display device or the like. The feature amount selection screen SS includes, for example, a first region R21, a confirmation button B1, a second region R22, and a third region R23.

[0122] The first area R21 is for selecting at least one feature amount from among a plurality of feature amounts. In the example shown in FIG. 12, the first area R21 includes check boxes for the first to the fifteenth feature amounts. Among them, the check boxes for the sixth to the fifteenth feature amounts that are not to be selected are grayed out and cannot be selected. On the other hand, the check boxes for the first to the fifth feature amounts are "on". The operator (user) can set the selection state or non-selection state of the feature amount by changing these check boxes to "on" or "off" through an input operation.

[0123] The second area R22 is for displaying the correct classification rate of the learning image 321a based on the selected feature amount. When there are a plurality of classification methods, the correct classification rate of the learning image 321a based on the selected feature amount is displayed for each classification method. Specifically, when at least one of the check boxes in the first area R21 is "on" and the operator (user) presses the confirmation button B1, the correct classification rate for each classification method is calculated and displayed in the second area R22.

[0124] The correct classification rate is, for example, for each of a plurality of learning images 321a, based on the selected feature amount, the classification obtained using the classification method is compared with the classification of the label data 321b corresponding to the learning image 321a to determine whether they match. Then, for all the learning images 321a, the ratio (%) of the number of cases where the obtained classification matches the classification of the corresponding label data 321b is calculated. In the example shown in FIG. 12, since all the feature amounts from the first to the fifth feature amounts are selected, the correct classification rate is "100.0" (%) for each of the first to the fourth classification methods.

[0125] The third area R23 is for displaying an index for each feature amount. When there are a plurality of classification methods, the index of the feature amount is displayed for each classification method and for each feature amount.

[0126] The index of the feature quantity is determined, for example, for each of the plurality of learning images 321a, by comparing the classification obtained using the classification method when the feature quantity is not used with the classification of the label data 321b corresponding to the learning image 321a to determine whether they match. Then, for all the learning images 321a, the number of cases where the obtained classification does not match the classification of the corresponding label data 321b is calculated as the index. In this case, when the index is "0", the feature quantity is unnecessary for classification, and when the index is other than "0", the feature quantity is likely to be necessary for classification. In the example shown in FIG. 12, for example, in the first classification method, the indices of the first feature quantity, the second feature quantity, and the third feature quantity are all "0", while the indices of the fourth feature quantity and the fifth feature quantity are both "2". Therefore, in the first classification method, it is presumed that the fourth feature quantity and the fifth feature quantity should be selected.

[0127] Note that the index of the feature quantity is not limited to the example described above and can be calculated in a variety of ways. For example, as a method for selecting a feature quantity, a filter method known for independently evaluating and ranking a plurality of feature quantities one by one, a wrapper method for evaluating a combination of a plurality of feature quantities, or an embedding method for simultaneously performing learning of a learning model and evaluation of a feature quantity can be used to calculate the index of the feature quantity. Also, the index of the feature quantity is not limited to a numerical value or something that can be calculated, and can be, for example, a level, a rank, a class, a grade (rating), etc., as long as it is a criterion, degree, or standard for selecting a feature quantity.

[0128] In this embodiment, an example is shown in which the selection unit 390 displays the feature quantity selection screen SS and selects a feature quantity according to an input operation by an operator (user), but it is not limited to this. For example, the selection unit 390 may automatically select at least one feature quantity from a plurality of feature quantities based on the index of each of the plurality of feature quantities. In the above example described with reference to FIG. 12, in the first classification method, since the indices of the fourth feature quantity and the fifth feature quantity are both other than "0", the selection unit 390 automatically selects the fourth feature quantity and the fifth feature quantity from among the first to fifth feature quantities.

[0129] Further, the creation unit 380 may create a graph based on the feature amount selected by the selection unit 390. That is, the creation unit 380 creates a graph in which points of the measurement image 327 and points of the learning image 321a are plotted in a coordinate system based on the selected feature amount of the measurement image 327 and the selected feature amount of the learning image 321a. Thereby, the number (dimension) of feature amounts can be reduced, and in the coordinate system with the transformed feature amount of the selected feature amount as the coordinate axis, the points of the measurement image 327 and the points of the learning image 321a can be more appropriately distributed.

[0130] Further, the learning unit 330 may input the feature amount selected by the selection unit 390 into a learning model to perform machine learning, and the classification unit 340 may input the feature amount selected by the selection unit 390 into the learned model 325 to output the classification of the measurement image 327. That is, the learning unit 330 uses the selected feature amount in the learning image 321a as an input, and learns the learning model with the classification of the learning image 321a as label data to generate the learned model 325. Then, the classification unit 340 inputs the selected feature amount in the measurement image 327 into this learned model 325, and outputs the classification of the measurement image 327 from the learned model 325. Thereby, the learning time can be shortened, and overfitting of the learned model 325 and a decrease in the classification accuracy of the measurement image 327 can be suppressed.

[0131] Further, the calculation unit 350 may calculate an evaluation value based on the feature amount selected by the selection unit 390. That is, the calculation unit 350 calculates the evaluation value of the measurement image 327 based on the selected feature amount in the measurement image 327 and the selected feature amount in the learning image 321a. Thereby, since the number (dimension) of feature amounts is reduced, a more accurate evaluation value of the measurement image 327 can be calculated with a smaller amount of calculation.

[0132] Note that at least one of the learning unit 330, the classification unit 340, the calculation unit 350, the determination unit 360, the output unit 370, the creation unit 380, and the selection unit 390 may be realized by the processor 31 executing a program stored in the storage device 33. When executing the program, the program may be stored in a storage medium. The storage medium storing the program may be a non-transitory computer readable medium. The non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a USB memory or a CD-ROM (Compact Disc ROM).

[0133] Next, with reference to FIG. 13, the processing procedure performed by the image processing apparatus according to an embodiment will be described. FIG. 13 is a flowchart for explaining an example of the image processing S100 performed by the image processing apparatus 30 in an embodiment.

[0134] Note that in the following example, it is described assuming that a plurality of learning data 321, a learned model 325, and a measurement image 327 are stored in the storage unit 320.

[0135] As shown in FIG. 13, first, the classification unit 340 reads out the learned model 325 and the measurement image 327 stored in the storage unit 320, inputs the feature amount of the measurement image 327 to the learned model 325, and outputs the classification of the measurement image 327 from the learned model 325, thereby classifying the measurement image 327 (S101). The classification of the measurement image 327 is stored in the storage unit 320 and output to the output unit 370.

[0136] Next, the calculation unit 350 reads out each of the plurality of learning data 321 stored in the storage unit 320, and for each of the plurality of learning images 321a, calculates an evaluation value 322 of the learning image 321a based on the feature amount of the learning image 321a and the feature amounts of other learning images 321a other than the learning image 321a (S102). The calculated evaluation value 322 of each learning image 321a is stored in the storage unit 320.

[0137] Next, the calculation unit 350 sets a threshold value 323 based on each of the evaluation values 322 of the plurality of learning images 321a calculated in step S102 (S103). The calculated threshold value 323 is stored in the storage unit 320 and output to the determination unit 360.

[0138] Next, the calculation unit 350 reads out each of the plurality of learning data 321 and the measurement image 327 stored in the storage unit 320, and calculates an evaluation value 328 of the measurement image 327 based on the feature amount of the measurement image 327 and the feature amounts of the plurality of learning images 321a (S104). The calculated evaluation value 328 of the measurement image 327 is stored in the storage unit 320 and output to the determination unit 360.

[0139] Next, the determination unit 360 determines whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a based on a comparison between the evaluation value 328 of the measurement image 327 calculated in step S104 and the threshold value 323 calculated in step S103 (S105). The determination result is output to the output unit 370.

[0140] As a result of the determination in step S105, if the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a, the determination unit 360 determines that the measurement image 327 is a measurement error (S106). In this case, the image processing apparatus 30, for example, regards the classification of the measurement image 327 in step S101 as misclassification and does not use it. On the other hand, as a result of the determination in step S105, if the feature amount of the measurement image 327 is not an outlier with respect to the feature amount of the learning image 321a, the determination unit 360 does not perform step S106.

[0141] Next, the output unit 370 outputs the suitability of the measurement image 327 based on the determination result in step S105 and the classification of the measurement image 327 in step S101 (S107). The suitability of the measurement image 327 is displayed on a display device or the like as described with reference to FIGS. 7 to 10.

[0142] Next, the creation unit 380 reads out the measurement image 327 and the plurality of learning data 321 stored in the storage unit 320, and plots the points of the measurement image 327 and the points of each of the plurality of learning images 321a in a coordinate system based on the feature amounts of the measurement image 327 and the plurality of learning data 321 to create a graph (S108). As described with reference to FIG. 11, the created graph is displayed on a display device or the like.

[0143] After step S108, the image processing apparatus 30 ends the image processing S100.

[0144] In FIG. 13, for simplicity of explanation, an example in which all of the plurality of feature amounts are used when a plurality of feature amounts are used for the measurement image 327 and the learning image 321a is shown, but the present invention is not limited to this. As described above, in the image processing S100, at least one of the plurality of feature amounts may be selected, and steps S101 to S108 may be performed for the selected feature amount.

[0145] Also, the sequences and flowcharts described in the present embodiment may be rearranged as long as there is no contradiction in the processing.

[0146] As described above, exemplary embodiments of the present invention have been described. According to the image processing apparatus 30 and the image processing method in the present embodiment, based on the evaluation value 328 of the measurement image 327 calculated based on the feature amount of the measurement image 327 and the feature amount of the learning image 321a, it is determined whether the feature amount of the measurement image 327 is an outlier with respect to the feature amount of the learning image 321a. Thereby, it becomes possible to exclude the measurement image 327 as, for example, a measurement error or the like other than the existing classification. Therefore, misclassification of the measurement image 327 can be suppressed.

[0147] Note that the embodiments described above are for facilitating the understanding of the present invention and are not for limiting the interpretation of the present invention. The present invention can be modified / improved without departing from its gist, and equivalents thereof are also included in the present invention. That is, even if those skilled in the art make appropriate design changes to the embodiments, as long as they have the features of the present invention, they are included in the scope of the present invention. For example, each element included in the embodiments and its arrangement, material, conditions, shape, size, etc. are not limited to those illustrated and can be changed as appropriate. Also, it goes without saying that the embodiments are illustrative, and partial substitution or combination of the configurations shown in different embodiments is possible, and these are also included in the scope of the present invention as long as they include the features of the present invention.

[0148] [Appendix 1] An image processing apparatus (30) for classifying a measurement image (327) of an object to be inspected (TA), A calculation unit (350) that calculates an evaluation value (328) of the measurement image (327) based on the feature amount of the measurement image (327) and the feature amount of the learning image (321a), A determination unit (360) that determines whether the feature amount of the measurement image (327) is an outlier with respect to the feature amount of the learning image (321a) based on the evaluation value (327) of the measurement image (327), The image processing apparatus (30). [Appendix 14] An image processing method for classifying a measurement image (327) of an object to be inspected (TA), An image processing apparatus (30) for classifying a measurement image (327) of an object to be inspected (TA), A step of calculating an evaluation value (328) of the measurement image (327) based on the feature amount of the measurement image (327) and the feature amount of the learning image (321a), A step of determining whether the feature amount of the measurement image (327) is an outlier with respect to the feature amount of the learning image (321a) based on the evaluation value (327) of the measurement image (327), The image processing method.

Explanation of Signs

[0149] 1…Imaging inspection system, 10…Model generation device, 20…Imaging inspection device, 30…Image processing device, 31…Processor, 32…Memory, 33…Storage device, 34…Communication device, 35…Input device, 36…Output device, 310…Communication unit, 320…Storage unit, 320…Classification unit, 321…Learning data, 321a…Learning image, 321b…Label data, 322…Evaluation value, 323…Threshold value, 325…Trained model, 327…Measured image, 328…Evaluation value, 330…Learning unit, 340…Classification unit, 350…Calculation unit, 360…Judgment unit, 370…Output unit, 380…Creation unit, 390…Selection unit, B1…Confirmation button, IL…Illumination, L…Light, MAXi…Maximum value, MINi…Minimum value, MP…Point, NW…Communication network, OS…Output screen, R…Reflected light, R11…First region, R12…Second region, R13…Third region, R14…Fourth region, R21…First region, R22…Second region, R23…Third region, S100…Image processing, SS…Feature selection screen, TA…Inspection object, μi…Average, ρi…Standard deviation.

Claims

1. An image processing apparatus for classifying a measurement image of an object to be inspected, comprising: a calculation unit that calculates an evaluation value of the measurement image based on a feature amount of the measurement image and a feature amount of a learning image; a determination unit that determines whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image based on the evaluation value of the measurement image; the calculation unit calculates the evaluation value of the measurement image based on a density between a point of the measurement image and a point of the learning image in a feature space represented by a plurality of the feature amounts; the determination unit determines whether or not the feature amount of the measurement image is an outlier with respect to the feature amount of the learning image based on a comparison between the evaluation value of the measurement image and a threshold value; An image processing apparatus.

2. the calculation unit calculates the evaluation value of the measurement image based on a distance between a point of the measurement image and a point of the learning image in a feature space represented by a plurality of the feature amounts; The image processing apparatus according to claim 1.

3. further comprising a classification unit that classifies the measurement image based on the learning image and a classification assigned to the learning image; The image processing apparatus according to claim 1 or 2.

4. the measurement image and the learning image are each an image of the object to be inspected that may include a defect, the classification includes a class indicating that there is no defect and a class indicating a type of the defect when there is a defect; The image processing apparatus according to claim 3.

5. the classification unit inputs the feature amount of the measurement image to a learned model learned using the feature amount of the learning image and the classification, and outputs the classification of the measurement image from the learned model; The image processing apparatus according to claim 3 or 4.

6. further comprising a learning unit that learns a learning model with the feature amount of the learning image as an input and the classification of the learning image as label data, and generates the learned model; The image processing apparatus according to claim 5.

7. further comprising a selection unit that selects at least one from the plurality of feature amounts based on respective indexes of the plurality of feature amounts; the learning unit learns a learning model with the selected feature amount in the learning image as an input and the classification of the learning image as label data, and generates the learned model; The classification unit inputs the selected feature amounts in the measurement image into the learned model, and causes the learned model to output the classification of the measurement image. The image processing apparatus according to claim 6.

8. The calculation unit calculates an evaluation value of the measurement image based on the selected feature amounts in the measurement image and the selected feature amounts in the learning image. The image processing apparatus according to claim 7.

9. The apparatus further includes an output unit that outputs the suitability of the measurement image based on the determination result by the determination unit and the classification by the classification unit. The image processing apparatus according to any one of claims 3 to 8.

10. The calculation unit calculates an evaluation value of one of the learning images based on the feature amounts of one of the learning images and the feature amounts of the other learning images. The threshold value is set based on the evaluation value of the learning image. The image processing apparatus according to claim 1.

11. The apparatus further includes a creation unit that creates a graph in which points of the measurement image and points of the learning image are plotted in a coordinate system based on the feature amounts of the measurement image and the feature amounts of the learning image. The image processing apparatus according to any one of claims 1 to 10.

12. An image processing method for classifying a measurement image of an object to be inspected, calculating an evaluation value of the measurement image based on the feature amounts of the measurement image and the feature amounts of the learning image; and determining whether the feature amounts of the measurement image are outliers with respect to the feature amounts of the learning image based on the evaluation value of the measurement image. The calculating step includes calculating the evaluation value of the measurement image based on a density between a point of the measurement image and a point of the learning image in a feature space represented by a plurality of the feature amounts. The determining step includes determining whether the feature amounts of the measurement image are outliers with respect to the feature amounts of the learning image based on a comparison between the evaluation value of the measurement image and a threshold value. Image processing method. ​

Images