Image processing device and image processing method

Abstract

To provide an image processing apparatus configured to output an appropriate graph without scale adjustment of a coordinate axis, and an image processing method.SOLUTION: An image processing apparatus (30) for classifying a measurement image (327) of an inspection object includes: a conversion unit (340) which performs at least one of standardization and normalization on a feature quantity of the measurement image (327) and a feature quantity of a learning image on the basis of feature quantities of multiple learning images, to be converted into converted feature quantities (322, 328), respectively; a generation unit (350) which generates a graph by plotting points of the measurement image (327) and points of the learning image on a coordinate system which uses the converted feature quantities as coordinate axes; and an output unit (380) which output the graph.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Conventionally, there is known an image processing device that classifies a plurality of images distributed in a feature space into a plurality of image groups using image features so that images with a similarity equal to or greater than a threshold are grouped together (see Patent Document 1). The image processing device of Patent Document 1 plots each point of a plurality of images in a coordinate system with the calculated first feature and second feature as the horizontal and vertical axes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-46902 A Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, in a measurement image obtained by photographing an inspection object, it is sometimes required to classify the type of the inspection object, for example, whether there is a defect or not, and if there is a defect, the type of the defect.

[0005] To achieve this, a method has been proposed in which, for example, in a coordinate system whose coordinate axes are the features, a graph is created in which points of the measurement image and points of training images whose classification is known in advance are plotted, and the measurement image is classified based on the distribution of these points.

[0006] However, for example, when the units are different for each feature, when trying to create a graph of a coordinate system in which such feature values ​​are directly used as coordinate axes, the points of multiple learning images may be plotted densely, making it difficult to classify the measurement images, or the points of the measurement images may be plotted at positions far away from the points of the learning images and may not be displayed on the graph. As a result, the operator (user) must adjust the scale for each coordinate axis to avoid the points being crowded together or the occurrence of points that cannot be displayed on the graph, making the operation cumbersome.

[0007] The present invention has been made in consideration of the above circumstances, and one of its objects is to provide an image processing device and an image processing method that can output an appropriate graph without adjusting the scale of the coordinate axes. [Means for solving the problem]

[0008] An image processing device according to one aspect of the present disclosure is an image processing device for classifying measurement images of an object to be inspected, and includes a conversion unit that converts features of the measurement image and features of the learning image into converted features by at least one of standardization and normalization based on the features of a plurality of learning images, a creation unit that creates a graph in which points of the measurement image and points of the learning image are plotted in a coordinate system whose coordinate axes are the converted features, and an output unit that outputs the graph.

[0009] According to this aspect, the feature amount of the measurement image and the feature amount of the learning image are converted into transformed feature amounts by performing at least one of standardization and normalization based on the feature amounts of the multiple learning images, and a graph is created in which the points of the measurement image and the points of the learning image are plotted in a coordinate system whose coordinate axes are the transformed feature amounts, and the graph is output. As a result, the scaling of each coordinate axis is automatically performed, so that the points of the multiple learning images can be distributed at an appropriate density, making it easy to classify the measurement images. In addition, even if the feature amount of the measurement image has a value that is significantly different from the feature amount of the learning image, it is possible to plot the points of the measurement image in the coordinate system. Therefore, an appropriate graph can be output without adjusting the scale of the coordinate axes.

[0010] In the above-described aspect, the conversion unit may standardize the feature amount of the measurement image and the feature amount of the learning image using an average and a standard deviation calculated based on the feature amounts of a plurality of learning images.

[0011] According to this aspect, the feature of the measurement image and the feature of the learning image are standardized using the average and standard deviation calculated based on the feature of a plurality of learning images. This makes the average "0" and the standard deviation "1", making it easy to compare the values ​​of the converted feature even between different feature values.

[0012] In the above-described aspect, the conversion unit may normalize the feature amount of the measurement image and the feature amount of the learning image by using a maximum value and a minimum value in the feature amounts of the multiple learning images.

[0013] According to this aspect, the feature amount of the measurement image and the feature amount of the learning image are normalized using the maximum and minimum values ​​of the feature amounts of the multiple learning images, so that the value of the transformed feature amount falls within a certain range, making it easy to compare the values ​​of the transformed feature amount even between different feature amounts.

[0014] In the above-described aspect, at least one of the standardization and the normalization may be performed further based on a feature amount of the measurement image.

[0015] According to this aspect, at least one of standardization and normalization is performed further based on the feature amount of the measurement image, which makes it possible to more appropriately perform at least one of standardization and normalization.

[0016] In the above-described aspect, the image processing device may further include a classification unit that classifies the measurement image based on the learning image and the classification assigned to the learning image.

[0017] According to this aspect, the measurement image is classified based on the learning image and the classification assigned to the learning image, whereby the classification of the measurement image estimated based on the visual distribution in the created graph can be compared with the classification of the measurement image by the classification unit.

[0018] In the above-described aspect, the measurement image and the learning image are each images of an object to be inspected that may contain defects, the classification includes a class indicating that there is no defect and a class indicating the type of defect when a defect is present, and the output unit may change the display format of the points plotted on the graph for each class included in the classification.

[0019] According to this aspect, the measurement image and the learning image are images of an object to be inspected that may contain defects, and the classification includes a class indicating the absence of a defect and a class indicating the type of the defect when a defect is present. This makes it possible to classify the presence or absence of a defect in the measurement image and the type of the defect. Also, the display format of the points plotted on the graph is changed based on the classes included in the classification. This makes it possible to output a graph plotted with points in different display formats according to the class.

[0020] In the above-mentioned aspect, the classification unit may input the features of the measurement image to a trained model trained using the features and classification of the training image, and output a classification of the measurement image from the trained model.

[0021] According to this aspect, the feature amount of the measurement image is input to a trained model trained using the feature amount and classification of the training image, and the trained model outputs a classification of the measurement image. This makes it possible to simply and easily obtain a classification of the measurement image with higher accuracy.

[0022] In the above-mentioned aspect, the system may further include a learning unit that uses the feature amounts of training images as input, trains a training model using the classification of the training images as label data, and generates a trained model.

[0023] According to this aspect, the feature amount of the training image is used as an input, and the training model is trained using the classification of the training image as label data, thereby generating a trained model. This makes it possible to obtain a trained model without a trained model generation device.

[0024] In the above-mentioned aspect, the system may further include a selection unit that selects at least one from the multiple features based on an index of each of the multiple features, and the learning unit may input the selected feature in the training image and train a learning model using the classification of the training image as label data to generate a trained model, and the classification unit may input the selected feature in the measurement image to the trained model and output the classification of the measurement image from the trained model.

[0025] According to this aspect, a learning model is trained using selected features in a training image as input and the classification of the training image as label data to generate a trained model, and the selected features in a measurement image are input to the trained model, and the classification of the measurement image is output from the trained model. This makes it possible to shorten the training time and to prevent overtraining of the trained model and a decrease in the classification accuracy of the measurement image.

[0026] In the above-mentioned aspect, the apparatus may further include a selection unit that selects at least one from the plurality of feature amounts based on an index of each of the plurality of feature amounts, and the creation unit may create a graph in which points of the measurement image and points of the training image are plotted in a coordinate system in which the transformed feature amounts of the selected feature amounts serve as coordinate axes.

[0027] According to this aspect, a graph is created in which points of the measurement image and points of the learning image are plotted in a coordinate system having the transformed feature values ​​of the selected feature values ​​as its coordinate axes. This makes it possible to reduce the number of feature values ​​(dimensions), and to more appropriately distribute points of the measurement image and points of the learning image in the coordinate system having the transformed feature values ​​of the selected feature values ​​as its coordinate axes.

[0028] An image processing method according to another aspect of the present disclosure is an image processing method for classifying measurement images of an inspection object, and includes the steps of converting features of the measurement image and features of the learning image into transformed features by performing at least one of standardization and normalization based on the features of a plurality of learning images, creating a graph in which points of the measurement image and points of the learning image are plotted in a coordinate system whose coordinate axes are the transformed features, and outputting the graph.

[0029] According to this aspect, the feature amount of the measurement image and the feature amount of the learning image are converted into transformed feature amounts by performing at least one of standardization and normalization based on the feature amounts of the multiple learning images, and a graph is created in which the points of the measurement image and the points of the learning image are plotted in a coordinate system whose coordinate axes are the transformed feature amounts, and the graph is output. As a result, the scaling of each coordinate axis is automatically performed, so that the points of the multiple learning images can be distributed at an appropriate density, making it easy to classify the measurement images. In addition, even if the feature amount of the measurement image has a value that is significantly different from the feature amount of the learning image, it is possible to plot the points of the measurement image in the coordinate system. Therefore, an appropriate graph can be output without adjusting the scale of the coordinate axes. Effect of the Invention

[0030] According to the present invention, it is possible to output an appropriate graph without adjusting the scale of the coordinate axes. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a configuration diagram illustrating a schematic configuration of an image inspection system according to an embodiment. [Diagram 2] FIG. 2 is a configuration diagram showing the physical configuration of a trained model generation device, an image inspection device, and an image processing device in one embodiment. [Diagram 3] FIG. 3 is a configuration diagram showing a functional block configuration of an image processing device according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of learning data. [Diagram 5] FIG. 5 is a diagram showing an example of the feature amount of the measurement image and the feature amount of the learning image. [Figure 6] FIG. 6 is a diagram showing an example of the feature amounts after conversion for the feature amounts of the measurement image and the feature amounts after conversion for the feature amounts of the learning image. [Figure 7] FIG. 7 is a diagram showing an example of a graph created by the creating unit. [Figure 8] FIG. 8 is a diagram showing an example of the feature amount selection screen. [Figure 9] FIG. 9 is a flowchart illustrating an example of image processing performed by the image processing device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] An embodiment of the present invention will be described below. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, the drawings are schematic. Therefore, specific dimensions, etc. should be judged in light of the following description. In addition, it goes without saying that the drawings include parts with different dimensional relationships and ratios. Furthermore, the technical scope of the present invention should not be interpreted as being limited to the embodiment.

[0033] First, the configuration of an image inspection system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a configuration diagram illustrating a schematic configuration of an image inspection system 1 in an embodiment.

[0034] As shown in FIG. 1, the image inspection system 1 includes a trained model generating device 10, an image inspection device 20, an image processing device 30, and an illumination device IL. The trained model generating device 10, the image inspection device 20, and the image processing device 30 are communicably connected to each other via a communication network NW. The illumination device IL irradiates light L onto an inspection object TA. The image inspection device 20 captures reflected light R and inspects the inspection object TA based on an image (hereinafter also referred to as a "measurement image") obtained by measuring the inspection object TA. More specifically, the image inspection device 20 judges whether the inspection object TA is a good product or a defective product, or judges whether the inspection object TA includes a defect, based on the measurement image.

[0035] The trained model generation device 10 generates a trained model that is used by the image processing device 30 to classify a measurement image. The trained model generation device 10 may also generate a trained model that is used by the image inspection device 20 to perform inspection.

[0036] The image processing device 30 is for classifying the measurement image of the inspection object TA. The measurement image is an image of the inspection object TA, and the inspection object TA may have defects, for example, during the manufacturing process. The defects are not particularly limited, but include visible defects such as chips, scratches, light stains, dark stains, cracks, burrs, deposits, foreign matter, dents, uneven color, faint printing, misalignment of printing, etc. Therefore, the measurement image may include partial images of such defects (hereinafter also simply referred to as "defects").

[0037] As described above, the image inspection device 20 can determine whether or not the inspection target TA includes a defect based on the measurement image. Meanwhile, the image processing device 30 can classify the presence or absence of a defect in the measurement image, and the type of the defect when a defect is present, based on the measurement image. Therefore, the image processing device 30 is useful, for example, for analyzing and interpreting the measurement image, and for estimating and / or narrowing down the cause of defects that may be included in the measurement image.

[0038] Next, the physical configurations of a trained model generation device, an image inspection device, and an image processing device according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a configuration diagram showing the physical configurations of a trained model generation device 10, an image inspection device 20, and an image processing device 30 in an embodiment.

[0039] 2, the trained model generation device 10, the image inspection device 20, and the image processing device 30 each include a processor 31, a memory 32, a storage device 33, a communication device 34, an input device 35, and an output device 36. These components are connected to each other via a bus so as to be able to transmit and receive data to and from each other.

[0040] The processor 31 is configured to control the operation of each part of the trained model generating device 10, the image inspection device 20, and the image processing device 30. The processor 31 is configured to include integrated circuits such as a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and a system-on-a-chip (SoC).

[0041] The memory 32 and the storage device 33 are configured to store programs, data, etc. The memory 32 is configured, for example, of a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), and / or a Random Access Memory (RAM), etc. The storage device 33 is configured, for example, of a storage such as a Hard Disk Drive (HDD), a Solid State Drive (SSD), and / or an embedded Multi Media Card (eMMC).

[0042] The communication device 34 is configured to perform communication via at least one of a wired and wireless network, and is configured to include, for example, a network card, a communication module, an interface for connecting to other devices, and the like.

[0043] The input device 35 is configured to allow a user to input information through an operation by the user. The input device 35 includes, for example, a keyboard, a touch panel, a mouse, a pointing device, and / or a microphone.

[0044] The output device 36 is configured to output information. The output device 36 includes a display device such as a liquid crystal display, an EL (Electro Luminescence) display, a plasma display, or an LCD (Liquid Crystal Display), and / or a speaker.

[0045] In the example shown in Fig. 2, the trained model generation device 10, the image inspection device 20, and the image processing device 30 are each configured with one computer, but the present invention is not limited to this. The trained model generation device 10, the image inspection device 20, and the image processing device 30 may each be realized by combining multiple computers. The configuration shown in Fig. 2 is an example, and the trained model generation device 10, the image inspection device 20, and the image processing device 30 may each include other configurations, or may not include some of these configurations.

[0046] Next, functional blocks of an image processing device according to an embodiment will be described with reference to Figs. 3 to 8. Fig. 3 is a configuration diagram showing a configuration of functional blocks of an image processing device 30 in an embodiment. Fig. 4 is a diagram showing an example of learning data 321. Fig. 5 is a diagram showing an example of a feature amount of a measurement image 327 and a feature amount of a learning image 321a. Fig. 6 is a diagram showing an example of a transformed feature amount 328 for the feature amount of the measurement image 327 and a transformed feature amount 322 for the feature amount of the learning image 321a. Fig. 7 is a diagram showing an example of a graph created by the creation unit 350. Fig. 8 is a diagram showing an example of a feature amount selection screen SS.

[0047] As shown in FIG. 3, the image processing device 30 includes a communication unit 310, a storage unit 320, a learning unit 330, a conversion unit 340, a creation unit 350, a classification unit 360, a selection unit 370, and an output unit 380.

[0048] The communication unit 310 is configured to be able to transmit and receive various information. The communication unit 310 receives a plurality of pieces of training data 321 from the trained model generation device 10 or another device, for example, via a communication network NW. The communication unit 310 also receives a measurement image 327 from the image inspection device 20 or another device, for example, via the communication network NW. The communication unit 310 also receives a trained model 325 from the trained model generation device 10, for example, via the communication network NW. The received plurality of pieces of training data 321 and trained model 325 are written and stored in the storage unit 320. The communication unit 310 may receive only the plurality of pieces of training data 321 and the measurement image 327. When the communication unit 310 receives only the plurality of pieces of training data 321 and the measurement image 327, a learning unit 330 (described later) generates a trained model 325 using the plurality of pieces of training data 321.

[0049] The storage unit 320 is configured to store various types of information. For example, the storage unit 320 stores a plurality of pieces of learning data 321, a trained model 325, and a measurement image 327. In this manner, by providing the storage unit 320 that stores the trained model 325, the trained model can be easily read out.

[0050] The learning unit 330 is configured to use a plurality of pieces of learning data 321 stored in the storage unit 320 to execute machine learning of the learning model, and generate a trained model 325. The generated trained model 325 is written to and stored in the storage unit 320.

[0051] More specifically, the learning unit 330 is configured to input the features of the training images included in the training data 321, train a learning model using the classification of the training images as label data, and generate a trained model 325.

[0052] 4, each of the multiple learning data 321 includes a learning image 321a and label data 321b. That is, each of the learning data 321 is composed of a combination of the learning image 321a and the label data 321b. The learning image 321a is an image of an inspection object TA that may include defects, similar to the above-mentioned measurement image.

[0053] Generally, label data is data that represents the properties of input data input to a learning model during machine learning of the learning model. In other words, label data is data that the learning model should output, and is data that is the goal of learning. In this embodiment, the label data 321b is a classification of the corresponding learning image 321a, more precisely, a classification based on the feature amount of the corresponding learning image 321a. In this way, in the learning data 321, a classification is assigned to the learning image 321a. This classification includes a class indicating that the learning image 321a does not contain a defect (hereinafter, also referred to as "no defect" or "no defect"), and a class indicating the type of the defect when the learning image 321a contains a defect (hereinafter, also referred to as "defective" or "defective").

[0054] More specifically, a class indicating the classification of the corresponding learning image 321a is set in the label data 321b, and the class is represented by an ID. For example, when the label data 321b is "0" (ID:0), the corresponding learning image 321a is classified into a class indicating "no defect". When the label data 321b is "1" (ID:1), the corresponding learning image 321a is classified into a class indicating "defective" and the type of the defect is "chipped". When the label data 321b is "2" (ID:2), the corresponding learning image 321a is classified into a class indicating "defective" and the type of the defect is "scratch". When the label data 321b is "3" (ID:3), the corresponding learning image 321a is classified into a class indicating "defective" and the type of the defect is "dark stain". Furthermore, if the label data 321b is "4" (ID: 4), the corresponding learning image 321a is classified into a class indicating that "a defect is present" and the type of the defect is "light stain".

[0055] In this way, the measurement image 327 and the learning image 321a are each images of the inspection object TA that may contain defects, and the classification includes a class indicating the absence of a defect and a class indicating the type of defect when a defect is present, making it possible to classify the presence or absence and type of defect in the measurement image 327.

[0056] Here, the feature amount is a variable that quantitatively represents the feature of the object to be found. During machine learning of the learning model, the feature amount of the learning image 321a input to the learning model characterizes the image as an image of the inspection object TA that may include a defect, for example. Specifically, the feature amount of the learning image 321a includes the area, the x coordinate of the center of gravity, the y coordinate of the center of gravity, the flattening ratio, and a gray value represented by the average density of the image. Generally, it is rare that the learning image 321a can be recognized as an image of the inspection object TA using only one type of feature amount, and it is often possible to recognize the image as an image of the inspection object TA using multiple types of feature amounts.

[0057] The learning unit 330 calculates one or more feature amounts from the training image 321a, and inputs the calculated feature amounts to the training model. The feature amounts input to the training model may be calculated in advance based on the training image 321a and included in the training data 321. The feature amounts input to the training model may be converted feature amounts, which will be described later. Furthermore, when multiple feature amounts are used for the training image 321a, at least one feature amount selected from the multiple feature amounts may be input to the training model, as will be described later.

[0058] To generate the trained model 325, a training model such as a K-nearest neighbor method, a support vector machine (SVM), a decision tree, a regression model, or a neural network can be used. For example, when the training model is a neural network, the feature amount of the training image 321a is input, and the weight of the neural network is updated using the backpropagation method based on the difference between the output and the label data. This process is performed for each of the multiple training data 321, thereby generating the trained model 325. Note that the training model is not limited to the one described above, and the training unit 330 may generate the trained model 325 using other algorithms.

[0059] In this way, by training a learning model using the features of the training image 321a as input and the classification of the training image 321a as label data, and generating a trained model 325, it is possible to obtain the trained model 325 even without the trained model generation device 10.

[0060] When the communication unit 310 receives the trained model 325 from the trained model generation device 10 via the communication network NW, the trained model generation device 10 has the same functions as the training unit 330. That is, the trained model generation device 10 trains the trained model in the same manner as described for the training unit 330 to generate the trained model 325. Therefore, a detailed description of the trained model generation device 10 will be omitted.

[0061] The conversion unit 340 is configured to convert the feature amount of the measurement image 327 and the feature amount of the training image 321a into the converted feature amount 328 and the converted feature amount 322, respectively, by performing at least one of standardization and normalization based on the feature amounts of the multiple training images 321a. When multiple feature amounts are used for the measurement image 327 and the training image 321a, the conversion unit 340 converts each feature amount into the converted feature amount 328 of the measurement image 327 and the converted feature amount 322 of the training image 321a. When multiple training images 321a exist, the conversion unit 340 converts each of the multiple training images 321a into the converted feature amount 322 of the training image 321a. The converted feature amount 328 for the feature amount of the measurement image 327 and the converted feature amount 322 for the feature amount of the training image 321a are written and stored in the storage unit 320. Moreover, the transformed feature amount 328 and the transformed feature amount 322 are output to the creation unit 350 .

[0062] Here, as shown in FIG. 5, the measurement image 327 and the multiple learning images 321a are each characterized by, for example, five feature amounts, the first feature amount to the fifth feature amount. The first feature amount is, for example, the area of ​​the image, and is expressed in pixels. The second feature amount is, for example, the x coordinate of the center of gravity of the image, and is expressed in pixels. The third feature amount is, for example, the y coordinate of the center of gravity of the image, and is expressed in pixels. The fourth feature amount is, for example, the flattening ratio of the image, and is a value between zero and one and is unitless (dimensionless). The fifth feature amount is, for example, the gray value of the image, and is a brightness between zero and 255 and is unitless (dimensionless).

[0063] Conventionally, when a graph is created in a coordinate system in which the feature quantities are directly used as coordinate axes when the units of the feature quantities are different, the points of the multiple learning images 321a are plotted densely, making it difficult to classify the measurement image 327, or the points of the measurement image 327 are plotted at positions far away from the points of the learning image 321a, making it impossible to display them in the graph. As a result, the operator (user) needs to adjust the scale for each coordinate axis to avoid the points being densely packed or the occurrence of points that cannot be displayed in the graph, which makes the operation complicated.

[0064] In contrast, in the image processing device 30 of this embodiment, the conversion unit 340 standardizes the feature of the measurement image 327 shown in FIG. 5 and the feature of the training image 321a based on the feature of a plurality of training images 321a, as shown in FIG. 6, and converts them into the converted feature 328 and the converted feature 322, respectively.

[0065] More specifically, the conversion unit 340 is configured to standardize the feature amounts of the measurement image 327 and the feature amounts of the training image 321a using an average and a standard deviation calculated based on the feature amounts of the multiple training images 321a.

[0066] Specifically, each transformed feature Yi (i=1, 2, 3, 4, 5) of the measurement image 327 can be calculated by the following formula (1) using the i-th feature Xi (i=1, 2, 3, 4, 5) before transformation of the measurement image 327, the average μi calculated based on the i-th feature of each training image 321a, and the standard deviation ρi calculated based on the corresponding i-th feature in each training image 321a. Yi = (Xi - μi) / ρi … (1)

[0067] Similarly, each transformed feature Mi (i=1, 2, 3, 4, 5) of the training image 321a can be calculated using the i-th feature Li (i=1, 2, 3, 4, 5) before transformation of the training image 321a and the aforementioned average μi and standard deviation ρi according to the following equation (2). Mi = (Li-μi) / ρi … (2)

[0068] In this way, by standardizing the i-th feature of the measurement image 327 and the i-th feature of the training image 321a using the mean μi and standard deviation ρi calculated based on the i-th feature of multiple training images 321a, the mean μi becomes “0” and the standard deviation ρi becomes “1”, making it easy to compare the values ​​of the transformed feature 328 and the transformed feature 322 even between different features.

[0069] Moreover, instead of or in addition to the above-mentioned standardization, the conversion unit 340 is configured to normalize the feature amounts of the measurement image 327 and the feature amounts of the training image 321a by using the maximum and minimum values ​​among the feature amounts of the multiple training images 321a.

[0070] Specifically, each transformed feature Ti (i=1, 2, 3, 4, 5) of the measurement image 327 can be calculated using the i-th feature Si (i=1, 2, 3, 4, 5) before transformation of the measurement image 327, the maximum value MAXi of the i-th feature of each training image 321a, and the minimum value MINi of the i-th feature of each training image 321a, using the following equation (3). Ti=(Si-MINi) / (MAXi-MINi) …(3)

[0071] Similarly, each transformed feature Qi (i = 1, 2, 3, 4, 5) of the training image 321a can be calculated using the i-th feature Pi (i = 1, 2, 3, 4, 5) before transformation of the training image 321a and the aforementioned maximum value MAXi and minimum value MINi using the following equation (4). Qi=(Pi-MINi) / (MAXi-MINi) …(4)

[0072] In this way, by normalizing the i-th feature of the measurement image 327 and the i-th feature of the training image 321a using the maximum value MAXi and minimum value MINi of the i-th feature of the multiple training images 321a, the values ​​of the transformed feature 328 and the transformed feature 322 fall within a certain range, making it easy to compare the values ​​of the transformed feature 328 and the transformed feature 322 even between different features.

[0073] In the above-mentioned standardization and normalization, an example has been shown in which the standardization and normalization are performed based only on the feature amounts of the multiple learning images 321a, but the present invention is not limited thereto. For example, the conversion unit 340 may perform at least one of the standardization and normalization based further on the feature amount of the measurement image 327. Specifically, the above-mentioned average μi may be calculated from the i-th feature amount of each learning image 321a and the i-th feature amount of the measurement image 327, and the above-mentioned standard deviation ρi may be calculated from the i-th feature amount of each learning image 321a and the i-th feature amount of the measurement image 327. Similarly, for example, the above-mentioned maximum value MAXi may be the maximum value between the i-th feature amount of each learning image 321a and the i-th feature amount of the measurement image 327, and the above-mentioned minimum value MINi may be the minimum value between the i-th feature amount of each learning image 321a and the i-th feature amount of the measurement image 327.

[0074] In this manner, at least one of the standardization and normalization described above is performed further based on the i-th feature amount of the measurement image 327, so that at least one of the standardization and normalization can be performed more appropriately.

[0075] 3, the creation unit 350 is configured to create a graph in which points of the measurement image 327 and points of the learning image 321a are plotted in a coordinate system whose coordinate axes are the converted feature amounts converted by the conversion unit 340. The creation unit 350 outputs the created graph to the output unit 380.

[0076] Moreover, the creation unit 350 is configured to be able to change parameters of the graph to be created, such as the magnification ratio, reduction ratio, rotation angle, elevation angle, etc. Therefore, the operator (user) can arbitrarily change these parameters by inputting through the input device 35 described above.

[0077] The output unit 380 is configured to output the graph created by the creation unit 350. Specifically, the output unit 380 outputs the graph input from the creation unit 350 to the above-mentioned output device 36, and the graph is displayed on a display device or the like. Note that the output of the graph is not limited to the output device 36. For example, the output unit 380 may output the graph to an external device.

[0078] 7, the graph is represented, for example, in a three-dimensional (3D) coordinate system. The first coordinate axis parallel to the X-axis is the transformed feature amount of the first feature amount described above, which is, for example, the area of ​​the image, the second coordinate axis parallel to the Y-axis is the transformed feature amount of the second feature amount described above, which is, for example, the x-coordinate of the center of gravity of the image, and the third coordinate axis parallel to the Z-axis is the transformed feature amount of the third feature amount described above, which is, for example, the y-coordinate of the center of gravity of the image.

[0079] 7, the points of the measurement image 327 indicated by circles are located within a group of points of the multiple learning images 321a indicated by crosses, which are located on the positive side of the first coordinate axis and the negative side of the second coordinate axis, so that it can be visually estimated that the classification of the measurement image 327 is the same as the classification assigned to the multiple learning images 321a.

[0080] The graph created by the creation unit 350 is not limited to being plotted in a three-dimensional coordinate system. The creation unit 350 may create a graph in which points of the measurement image 327 and points of the learning image 321a are plotted in a coordinate system of a number of dimensions other than three dimensions, for example, a two-dimensional or four or more dimensional coordinate system.

[0081] In this way, by creating a graph in which points of the measurement image 327 and points of the learning image 321a are plotted in a coordinate system whose coordinate axes are the transformed feature values, and outputting the graph, the scaling of each coordinate axis is automatically performed, so that the points of the multiple learning images 321a can be distributed at an appropriate density, making it easy to classify the measurement image 327. Furthermore, even if the feature values ​​of the measurement image 327 have values ​​that are significantly different from the feature values ​​of the learning image, it becomes possible to plot the points of the measurement image 327 in the coordinate system. Therefore, an appropriate graph can be output without adjusting the scale of the coordinate axes.

[0082] 3, the classification unit 360 is configured to classify the measurement image 327 based on the learning image 321a and the classification assigned to the learning image 321a. This makes it possible to compare the classification of the measurement image 327 estimated based on the visual distribution in the created graph with the classification of the measurement image 327 by the classification unit 360.

[0083] More specifically, the classification unit 360 is configured to input the feature amount of the measurement image 327 to the trained model 325 trained using the feature amount and classification of the training image 321a, and to output the classification of the measurement image 327 from the trained model 325. When the trained model 325 is generated by inputting the converted feature amount 322 of the training image 321a to the trained model during machine learning of the training model, the classification unit 360 inputs the converted feature amount 328 of the measurement image 327 to the trained model 325.

[0084] Specifically, the classification unit 360 reads out the trained model 325 and the converted feature amount 328 of the measured image 327 stored in the storage unit 320, and inputs the converted feature amount 328 to the trained model 325 to output the classification of the measured image 327. The classification of the measured image 327 output from the trained model 325 is written and stored in the storage unit 320. The classification unit 360 also outputs the classification of the measured image 327 output from the trained model 325 to the output unit 380, and also reads out each of the multiple pieces of training data 321 from the storage unit 320 and outputs each classification assigned to the multiple training images 321a to the output unit 380. Note that the classification unit 360 does not need to use the output of the trained model 325 as the classification of the measured image 327 as it is, and may perform any post-processing.

[0085] In this way, by inputting the features of the measurement image 327 into the trained model 325 trained using the features and classification of the training image 321a, and outputting the classification of the measurement image 327 from the trained model 325, it is possible to simply and easily obtain a more accurate classification of the measurement image 327.

[0086] The output unit 380 described above may be configured to change the display format of the points plotted on the graph based on the classes included in the classification of the measurement image 327 and the learning image 321a. The display format may be changed, for example, to a point belonging to a class of a specified ID, in order to confirm which class set the classification of the measurement image 327 is closest to, by (A) changing the color of the point belonging to a class of another ID, for example, to a different color, for example, green, from the color of the point belonging to the class of another ID, for example, white, or (B) changing the shape of the point belonging to the class of another ID, for example, to a different shape, for example, cross, from the shape of the point belonging to the class of another ID, for example, triangle, etc. Alternatively, the output unit 380 may change the color or shape of the plotted point for each class indicated by the ID so that the plotted point and the class are associated. This makes it possible to output a graph plotted with points in different display formats according to the class.

[0087] In the present embodiment, an example has been shown in which the classification unit 360 obtains the classification of the measurement image 327 using the trained model 325, but the present invention is not limited to this. For example, the classification unit 360 may classify the measurement image 327 based on the Mahalanobis distance. In this case, the classification unit 360 calculates the Mahalanobis distance between the points of the measurement image 327 and each of a plurality of point clouds (groups) formed for each classification of a plurality of training images 321a in a predetermined coordinate system. Then, the classification unit 360 determines the classification of the measurement image 327 as belonging to the class of the point cloud for which the Mahalanobis distance is the smallest.

[0088] The classification unit 360 is not limited to classifying the measurement image 327 by one classification method, and may classify the measurement image 327 by using one or more classification methods selected from a plurality of classification methods. In this case, a classification of the measurement image 327 according to each classification method is obtained. The storage unit 320 may store a plurality of trained models trained according to each classification method.

[0089] The selection unit 370 is configured to select at least one of the plurality of feature amounts based on the indexes of each of the plurality of feature amounts. The selected feature amount is written and stored in the storage unit 320.

[0090] 8, for example, a feature quantity selection screen SS is displayed on which at least one feature quantity can be selected from a plurality of feature quantities. The selection unit 370 generates data necessary for displaying the feature quantity selection screen SS, and outputs the generated data to the above-mentioned output device 36, thereby making it possible to display the feature quantity selection screen SS on a display device or the like. The feature quantity selection screen SS includes, for example, a first region R1, a confirmation button B1, a second region R2, and a third region R3.

[0091] The first region R1 is for selecting at least one feature from among a plurality of feature amounts. In the example shown in FIG. 8, the first region R1 includes check boxes for the first to fifteenth feature amounts, and among them, the check boxes for the sixth to fifteenth feature amounts, which are not subject to selection, are grayed out and cannot be selected. On the other hand, the check boxes for the first to fifth feature amounts are "on." The operator (user) can set the selected or unselected state of the feature amounts by changing these check boxes to "on" or "off" by input operations.

[0092] The second area R2 is for displaying the accuracy rate of classification of the learning image 321a based on the selected feature. When multiple classification methods exist, the accuracy rate of classification of the learning image 321a based on the selected feature is displayed for each classification method. Specifically, when at least one of the check boxes in the first area R1 is "on" and the operator (user) presses the confirmation button B1, the accuracy rate of classification for each classification method is calculated and displayed in the second area R2.

[0093] The accuracy rate of classification is, for example, determined by comparing the classification obtained using the classification method based on the selected feature for each of a plurality of training images 321a with the classification of the label data 321b corresponding to the training image 321a, and determining whether they match. Then, the percentage (%) of the number of training images 321a in which the acquired classification matches the classification of the corresponding label data 321b is calculated for all training images 321a. In the example shown in FIG. 8, all the features from the first feature to the fifth feature are selected, so the accuracy rate is "100.0" (%) for each of the first to fourth classification methods.

[0094] The third area R3 is for displaying an index for each feature amount. If multiple classification methods are used, an index for each feature amount is displayed for each classification method.

[0095] The index of the feature is determined by, for example, comparing the classification obtained by using the classification method when the feature is not used for each of the multiple learning images 321a with the classification of the label data 321b corresponding to the learning image 321a, and determining whether they match. Then, for all learning images 321a, the number of times the acquired classification does not match the classification of the corresponding label data 321b is calculated as the index. In this case, when the index is "0", the feature is not necessary for classification, and when the index is other than "0", the feature is highly likely to be necessary for classification. In the example shown in FIG. 8, for example, in the first classification method, the indexes of the first feature, the second feature, and the third feature are all "0", while the indexes of the fourth feature and the fifth feature are each "2". Therefore, it is estimated that the fourth feature and the fifth feature should be selected in the first classification method.

[0096] The index of the feature is not limited to the above-mentioned example, and can be calculated by a wide variety of methods. For example, the index of the feature can be calculated using a filter method that evaluates and ranks multiple feature quantities independently, a wrapper method that evaluates multiple feature quantities in combination, or an embedding method that simultaneously learns a learning model and evaluates the feature quantities, which are known as methods for selecting feature quantities. In addition, the index of the feature is not limited to a numerical value or a calculable one, and may be, for example, a level, rank, class, grade, etc., as long as it is a criterion, degree, or guideline for selecting a feature.

[0097] In the present embodiment, the selection unit 370 displays the feature quantity selection screen SS and selects a feature quantity according to an input operation by an operator (user), but the present invention is not limited to this. For example, the selection unit 370 may automatically select at least one feature quantity from a plurality of feature quantities based on the respective indices of the plurality of feature quantities. In the above example described with reference to FIG. 8, in the first classification method, the indices of the fourth feature quantity and the fifth feature quantity are each other than "0", so the selection unit 370 automatically selects the fourth feature quantity and the fifth feature quantity from the first feature quantity to the fifth feature quantity.

[0098] Furthermore, the creation unit 350 may create a graph based on the feature quantities selected by the selection unit 370. That is, the creation unit 350 creates a graph in which points of the measurement image 327 and points of the learning image 321a are plotted in a coordinate system having the transformed feature quantities of the selected feature quantities as its coordinate axes. This makes it possible to reduce the number of feature quantities (dimensions), and to more appropriately distribute the points of the measurement image 327 and the points of the learning image 321a in the coordinate system having the transformed feature quantities of the selected feature quantities as its coordinate axes.

[0099] Furthermore, the learning unit 330 may input the feature amount selected by the selection unit 370 to a learning model to perform machine learning, and the classification unit 360 may input the feature amount selected by the selection unit 370 to the trained model 325 to output the classification of the measurement image 327. That is, the learning unit 330 trains the learning model using the feature amount selected in the training image 321a as input and the classification of the training image 321a as label data to generate the trained model 325. Then, the classification unit 360 inputs the feature amount selected in the measurement image 327 to the trained model 325 and outputs the classification of the measurement image 327 from the trained model 325. This can shorten the learning time and prevent overtraining of the trained model 325 and a decrease in the classification accuracy of the measurement image 327.

[0100] At least one of the learning unit 330, the conversion unit 340, the creation unit 350, the classification unit 360, the selection unit 370, and the output unit 380 may be realized by the processor 31 executing a program stored in the storage device 33. When executing a program, the program may be stored in a storage medium. The storage medium storing the program may be a non-transitory computer readable storage medium. The non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a USB memory or a CD-ROM (Compact Disc ROM).

[0101] Next, a processing procedure performed by the image processing device according to an embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart for explaining an example of image processing S100 performed by the image processing device 30 in an embodiment.

[0102] In the following example, it is assumed that a plurality of pieces of learning data 321, a trained model 325, and a measurement image 327 are stored in the storage unit 320.

[0103] 9, first, the conversion unit 340 reads out the measurement image 327 and the plurality of learning data 321 stored in the storage unit 320, and performs at least one of standardization and normalization on the feature amount of the measurement image 327 based on the feature amount of the learning image 321a of the plurality of learning data 321, thereby converting it into a converted feature amount 328 (step S101). The converted converted feature amount 328 is stored in the storage unit 320.

[0104] Next, the conversion unit 340 reads out the multiple pieces of learning data 321 stored in the storage unit 320, and performs at least one of standardization and normalization on the feature amounts of the learning images 321a of the multiple pieces of learning data 321 based on the feature amounts of the learning images 321a of the multiple pieces of learning data 321, thereby converting them into converted feature amounts 322 (step S102). The converted converted feature amounts 322 are stored in the storage unit 320.

[0105] Next, the creation unit 350 determines whether or not a parameter has been input based on an input on a parameter setting screen (not shown) (S103). An operator (user) can change the values ​​of parameters such as the magnification ratio, reduction ratio, rotation angle, and elevation angle on the parameter setting screen.

[0106] If it is determined in step S103 that a parameter has been input, the creation unit 350 generates data according to the input and sets the parameter (S104). On the other hand, if it is determined in step S103 that a parameter has not been input, the creation unit 350 does not perform step S104. At this time, the creation unit 350 sets the parameter to data according to, for example, a default.

[0107] Next, the selection unit 370 determines whether or not a feature has been selected based on an input on the feature selection screen SS shown in Fig. 8 (S105). The operator (user) can turn the check box for each feature "on" or "off" in the first area R1 of the feature selection screen SS.

[0108] If the result of the determination in step S105 is that a feature has been selected, the selection unit 370 selects one or more feature quantities from the multiple feature quantities in accordance with the selection, and the creation unit 350 sets the transformed feature quantities of the selected feature quantities on the coordinate axes (S106). On the other hand, if the result of the determination in step S105 is that no feature quantity has been selected, the selection unit 370 and the creation unit 350 do not perform step S106. At this time, the creation unit 350 sets the transformed feature quantities of the default feature quantities, for example, on the coordinate axes.

[0109] Next, the creation unit 350 reads out the measurement image 327 and the plurality of learning data 321 stored in the storage unit 320, and creates a graph in which points of the measurement image 327 and points of each of the plurality of learning images 321a are plotted in a coordinate system whose coordinate axes are the converted features of the set features (S107).Then, the output unit 380 outputs the graph created in step S107 to a display device or the like (S108).

[0110] After step S108, the image processing device 30 repeats steps S103 to S108 until the communication unit 310 receives another measurement image 327 or until another measurement image 327 is selected by an operation of the operator (user).

[0111] The sequence and flow charts described in this embodiment may be rearranged as long as no inconsistency occurs in the processing.

[0112] An exemplary embodiment of the present invention has been described above. According to the image processing device 30 and the image processing method of this embodiment, the feature amount of the measurement image 327 and the feature amount of the learning image 321a are converted into the converted feature amount 328 and the converted feature amount 322 by performing at least one of standardization and normalization based on the feature amounts of the multiple learning images 321a, and a graph is created in which the points of the measurement image 327 and the points of the learning image 321a are plotted in a coordinate system having the converted feature amount as the coordinate axis, and the graph is output. As a result, the scaling of each coordinate axis is automatically performed, so that the points of the multiple learning images 321a can be distributed at an appropriate density, and the measurement image 327 can be easily classified. Furthermore, even if the feature amount of the measurement image 327 has a value that is significantly different from the feature amount of the learning image, it is possible to plot the points of the measurement image 327 in the coordinate system. Therefore, an appropriate graph can be output without adjusting the scale of the coordinate axis.

[0113] The above-described embodiments are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. The present invention may be modified / improved without departing from the spirit thereof, and equivalents are also included in the present invention. In other words, even if a person skilled in the art appropriately modifies the design of the embodiments, the embodiment is included in the scope of the present invention as long as it has the characteristics of the present invention. For example, the elements of the embodiments and their arrangements, materials, conditions, shapes, sizes, etc. are not limited to those exemplified, and can be appropriately modified. In addition, the embodiments are merely examples, and it goes without saying that partial replacement or combination of the configurations shown in different embodiments is possible, and these are also included in the scope of the present invention as long as they include the characteristics of the present invention.

[0114] [Appendix 1] An image processing device (30) for classifying a measurement image (327) of an inspection object (TA), comprising: a conversion unit (340) that converts the feature amount of the measurement image (327) and the feature amount of the learning image (321a) into converted feature amounts (322, 328) by performing at least one of standardization and normalization based on the feature amounts of the multiple learning images (321a); a creation unit (350) that creates a graph in which points of the measurement image (327) and points of the learning image (321a) are plotted in a coordinate system whose coordinate axes are the transformed feature values; and an output unit (380) that outputs a graph. An image processing device (30). [Appendix 11] 1. An image processing method for classifying a measured image (327) of an inspection object (TA), comprising: a step of converting the feature amount of the measurement image (327) and the feature amount of the learning image (321a) into converted feature amounts (322, 328) by performing at least one of standardization and normalization based on the feature amounts of the multiple learning images (321a); creating a graph in which points of the measurement image (327) and points of the learning image (321a) are plotted in a coordinate system whose coordinate axes are the transformed feature values; and outputting the graph. Image processing methods. [Explanation of symbols]

[0115] 1...image inspection system, 10...model generation device, 20...image inspection device, 30...image processing device, 31...processor, 32...memory, 33...storage device, 34...communication device, 35...input device, 36...output device, 310...communication unit, 320...storage unit, 321...learning data, 321a...learning image, 321b...label data, 322...post-conversion feature, 325...trained model, 327...measured image, 328...post-conversion feature quantity, 330...learning section, 340...conversion section, 350...creation section, 360...selection section, 360...classification section, 370...selection section, 380...output section, B1...confirmation button, IL...illumination, L...light, MAXi...maximum value, MINi...minimum value, NW...communication network, R...reflected light, R1...first region, R2...second region, R3...third region, S100...image processing, SS...feature quantity selection screen, TA...inspection object, μi...average, ρi...standard deviation.

Claims

1. 1. An image processing device for classifying measurement images of an inspection object, comprising: a conversion unit that converts the feature amounts of the measurement image and the feature amounts of the learning images into converted feature amounts by performing at least one of standardization and normalization based on a plurality of feature amounts of the learning images; a generating unit that generates a graph in which points of the measurement image and points of the learning image are plotted in a coordinate system whose coordinate axes are the transformed feature values; an output unit that outputs the graph, the conversion unit performs the standardization on the feature amounts of the measurement image and the feature amounts of the learning images by using an average and a standard deviation calculated based on the feature amounts of the plurality of learning images. Image processing device.

2. the conversion unit performs the normalization on the feature amounts of the measurement image and the feature amounts of the learning images by using a maximum value and a minimum value in the feature amounts of the plurality of learning images. The image processing device according to claim 1 .

3. At least one of the standardization and the normalization is further performed based on a feature amount of the measurement image.

3. The image processing device according to claim 1 or 2.

4. a classification unit that classifies the measurement image based on the learning image and a classification assigned to the learning image, The image processing device according to claim 1 .

5. the measurement image and the learning image are each an image of the inspection object that may include a defect, The classification includes a class indicating the absence of the defect and a class indicating a type of the defect when the defect is present, the output unit changes a display format of the points plotted on the graph based on the classes included in the classification. The image processing device according to claim 4.

6. The classification unit inputs the feature amount of the measurement image into a trained model trained using the feature amount and the classification of the training image, and causes the trained model to output the classification of the measurement image.

6. The image processing device according to claim 4 or 5.

7. A learning unit is further provided that uses the feature amount of the training image as an input, trains a training model using the classification of the training image as label data, and generates the trained model. The image processing device according to claim 6.

8. A selection unit that selects at least one of the plurality of feature amounts based on an index of each of the plurality of feature amounts, The learning unit learns a learning model using the selected feature amount in the training image as an input and the classification of the training image as label data, thereby generating the trained model; The classification unit inputs the selected feature amount in the measurement image to the trained model, and outputs the classification of the measurement image from the trained model. The image processing device according to claim 7.

9. A selection unit that selects at least one of the plurality of feature amounts based on an index of each of the plurality of feature amounts, the creation unit creates a graph in which points of the measurement image and points of the learning image are plotted in a coordinate system whose coordinate axes are the transformed feature quantities of the selected feature quantities. The image processing device according to claim 1 .

10. 1. An image processing method for classifying measurement images of an inspection object, comprising: performing at least one of standardization and normalization on the feature amounts of the measurement image and the feature amounts of the learning images based on a plurality of feature amounts of the learning images to convert the feature amounts into converted feature amounts; creating a graph in which points of the measurement image and points of the learning image are plotted in a coordinate system whose coordinate axes are the transformed feature values; and outputting the graph. the converting step includes performing the standardization on the feature amounts of the measurement image and the feature amounts of the learning images by using an average and a standard deviation calculated based on the feature amounts of the plurality of learning images. Image processing methods.

Images