Classification evaluation support apparatus, classification evaluation support method, and classification evaluation support program

Abstract

To make it possible to appropriately evaluate accuracy of a machine learning model used for classification into a plurality of classes.SOLUTION: An apparatus 1 configured to support evaluation of accuracy of a machine learning model 23A that classifies a predetermined image into one of a plurality of classes includes: a calculation unit 23 that calculates a matching degree of each class of multiple classes with respect to each verification image of a plurality of verification images associated with a ground truth class by using the machine learning model 23A; a classification unit 24 that classifies the verification images into any one of the multiple classes based on the matching degree of each class calculated for each verification image; and a display processing unit 25 that displays, on a predetermined display device, a separation degree between a statistic of matching degrees of the ground truth class and a statistic of matching degrees of another class that is a class different from the ground truth class, among the matching degrees of the classes calculated for a verification image group associated with a predetermined ground truth class.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technology for supporting evaluation of the accuracy of a machine learning model that classifies images into multiple classes. [Background technology]

[0002] Recently, image inspection using machine learning models (AI image inspection) has started to become widespread. For example, in anomaly detection, which is one example of AI image inspection, the degree of anomaly of each image is calculated, and pass / fail judgment is made based on whether the degree of anomaly is equal to or greater than a threshold (see, for example, Patent Document 1). In multi-class classification, which is another example of AI image inspection, the degree of agreement of each class for each image is detected, and each image is classified into the class with the highest degree of agreement.

[0003] In AI image inspection, it is necessary to evaluate the extent to which the accuracy of the machine learning model has improved through learning, and whether the accuracy is sufficient to start using the machine learning model for image inspection.

[0004] The accuracy of a machine learning model can be evaluated based on whether the machine learning model can make correct judgments on a verification image for which the correct answer is known. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2023-77059 A Summary of the Invention [Problem to be solved by the invention]

[0006] When evaluating accuracy, even if the accuracy rate for the judgment on the verification images is 100%, it is not possible to judge with what margin of error the judgment can be made correctly (i.e., the stability of the machine learning model's performance) based on the accuracy rate alone. If the stability of performance is low, there is a high possibility that incorrect judgments will be made after the start of operation, so it is also important to evaluate the stability of performance. In the anomaly detection in Patent Document 1, a single value, called the degree of anomaly, is calculated for each image. Therefore, the above stability can be evaluated, for example, based on the degree to which the degree of anomaly for a group of good product images used for verification can be separated from the degree of anomaly for a group of defective product images. In contrast, in machine learning models used for multi-class classification, the degree of match (also called confidence, reliability, score, etc.) of multiple classes is calculated for one image, and it is necessary to consider not only the degree of match of the correct class but also the relative relationship with the degrees of match of other classes. Therefore, unlike anomaly detection, in which only one value, the degree of anomaly, is calculated, there is a problem in multi-class classification in that it is not easy to evaluate the stability of performance.

[0007] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a technology that enables appropriate evaluation of the accuracy, and in particular the performance stability, of a machine learning model used for classifying data into multiple classes. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, a classification evaluation support device according to one aspect is a classification evaluation support device that supports evaluation of accuracy of a machine learning model used in a classification process for classifying a specified image into one of a plurality of classes, and includes: a calculation unit that uses the machine learning model to calculate a degree of coincidence of each of the multiple classes for each of a plurality of verification images associated with a correct class; a classification unit that classifies each of the verification images into one of the multiple classes based on the degree of coincidence of each of the classes calculated for each of the verification images; and a display processing unit that displays, on a specified display device, a degree of separation between a statistical amount of the degree of coincidence of the correct class and a statistical amount of the degree of coincidence of other classes that are classes different from the correct class, among the degrees of coincidence of each class calculated for a group of verification images associated with a specified correct class. Effect of the Invention

[0009] According to the present invention, it is possible to appropriately evaluate the accuracy of a machine learning model used for classification into multiple classes, in particular the stability of performance. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an image inspection device according to an embodiment. [Diagram 2] FIG. 2 is a hardware configuration diagram of an image inspection device according to an embodiment. [Diagram 3] FIG. 3 is a functional configuration diagram of an image inspection device according to an embodiment. [Figure 4] FIG. 4 is a flowchart of a machine learning model setting process according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing the configuration of a machine learning model evaluation screen according to one embodiment. [Figure 6] FIG. 6 is a diagram illustrating another example of a resolution graph according to an embodiment. [Figure 7] FIG. 7 is another example of a verification image display area according to an embodiment. [Figure 8] FIG. 8 is a configuration diagram of a matching degree table according to an embodiment. [Figure 9] FIG. 9 is a diagram showing the configuration of a correction coincidence degree table according to a modified example. [Figure 10] FIG. 10 is a diagram showing the structure of a normalized matching degree table according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0012] FIG. 1 is a diagram showing the overall configuration of an image inspection device according to an embodiment.

[0013] The image inspection device 1 is an example of a classification evaluation support device, and is a device for performing classification processing to classify an inspection object, such as various parts or products, into which of a plurality of classes the inspection object belongs, based on an image of the inspection object, and is used in production sites such as factories. The inspection object in the image inspection device 1 may be the entire inspection object, a part of the inspection object, or multiple parts of the inspection object. Furthermore, the image may include multiple inspection objects.

[0014] The image inspection device 1 includes a control unit 2, an imaging unit 3, a display device 4, and a personal computer 5. The personal computer 5 is not essential and may be omitted. The personal computer 5 may be used instead of the display device 4. The control unit 2 may be realized by the personal computer 5.

[0015] In the example of Fig. 1, the image inspection device 1 has a control unit 2, an imaging unit 3, a display device 4, and a personal computer 5 as separate entities, but any configuration may be combined and integrated. For example, the control unit 2 and the imaging unit 3 may be integrated to configure a so-called smart camera, or the control unit 2 and the display device 4 may be integrated. The control unit 2 may be divided into multiple units, and some of the divided units may be incorporated into other configurations such as the imaging unit 3 and the display device 4. The imaging unit 3 may be divided into multiple units, and some of the divided units may be incorporated into other configurations.

[0016] FIG. 2 is a hardware configuration diagram of an image inspection device according to an embodiment.

[0017] (Configuration of imaging unit 3) The imaging unit 3 includes a camera module 14 and an illumination module 15. The camera module 14 includes an AF (Auto Focus) motor 141 that drives the imaging optical system, and an imaging board 142. The AF motor 141 performs automatic focus adjustment by driving the lens of the imaging optical system. The focus adjustment of the AF motor 141 is controlled by a well-known technique such as contrast autofocus.

[0018] The imaging board 142 includes a CMOS (Complementary Metal-Oxide-Semiconductor) sensor 143 as a light receiving element that receives light incident from an imaging optical system, an FPGA (Field Programmable Gate Array) 144, and a DSP (Digital Signal Processor) 145. The CMOS sensor 143 is an imaging sensor configured to be able to acquire a color image. The CMOS sensor 143 is configured to be able to output a live image, that is, a currently captured image, at a short frame rate at any time. The CMOS sensor 143 inputs an image (image signal) to the FPGA 144 and the DSP 145. Instead of the CMOS sensor 143, a light receiving element such as a CCD (Charged-Coupled Device) sensor may be used. The FPGA 144 and the DSP 145 control the camera module 14 and perform predetermined image processing on the image from the CMOS sensor 143. The CMOS sensor 143 starts imaging in response to an imaging control signal from the control unit 13A of the control unit 2, which will be described later, and adjusts the exposure time to an arbitrary time to perform imaging. That is, the imaging unit 3 images the field of view of the CMOS sensor 143 in response to an imaging control signal output from the control unit 13A. Therefore, if there is an object to be inspected within the field of view, the CMOS sensor 143 images the object to be inspected, and if there is an object other than the object to be inspected within the field of view, the CMOS sensor 143 also images the object to be inspected. For example, when the image inspection device 1 is set up, the imaging unit 3 can capture a plurality of images for assigning (associating) attributes of each of a plurality of classes classified by the user with a correct class. Also, when the image inspection device 1 is in operation, the imaging unit 3 can capture an image of the object to be inspected.

[0019] The lighting module 15 includes an LED (light-emitting diode) 151 as a light-emitting body that illuminates an imaging area including an inspection object, and an LED driver 152 that controls the LED 151. The timing of light emission, the duration of light emission, and the amount of light emitted by the LED 151 can be arbitrarily set by the LED driver 152. In the example of FIG. 2, the LED 151 is provided integrally with the imaging unit 3, but may be provided as an external lighting unit separate from the imaging unit 3. Although not shown, the lighting module 15 includes a reflector that reflects the light emitted from the LED 151, a lens through which the light emitted from the LED 151 passes, and the like. In the lighting module 15, the illumination range of the LED 151 is set so that the light emitted from the LED 151 is illuminated on the inspection object and the surrounding area of ​​the inspection object. Instead of the LED 151, another light-emitting body may be used. The LED driver 152 switches the LED 151 on and off and adjusts the illumination time according to an illumination control signal from the control unit 13A of the control unit 2, and adjusts the light amount of the LED 151, and the like.

[0020] (Configuration of control unit 2) The control unit 2 includes a main board 13, a connector board 16, a communication board 17, a power supply board 18, and a storage device 19. The main board 13 is equipped with an FPGA 131 and a DSP 132 that constitute a control unit 13A, and a memory 133. The FPGA 131, the DSP 132, and the memory 133 may be configured as an integrated unit.

[0021] The control unit 13A centrally controls the operation of each board and module connected to the main board 13. For example, the control unit 13A outputs an illumination control signal to the LED driver 152 of the illumination module 15 to control the turning on / off of the LED 151. The control unit 13A also outputs an imaging control signal to the imaging board 142 of the camera module 14 to control the CMOS sensor 143. This imaging control signal allows the control unit 13A to capture various images using the imaging unit 3. When the imaging unit 3 finishes capturing an image, the control unit 13A acquires an image (image signal) output from the imaging unit 3. In this embodiment, the image signal is input to the FPGA 131, processed by the FPGA 131 and the DSP 132, and stored in the memory 133. The control unit 13A also outputs an AF control signal to control an AF motor driver 181 (described later) of the power supply board 18.

[0022] The connector board 16 has a power supply interface 161, and receives power from the outside via a power connector (not shown) provided in the power supply interface 161.

[0023] The power supply board 18 distributes the power received by the connector board 16 to each board, module, etc. Specifically, the power supply board 18 distributes power to the lighting module 15, the camera module 14, the main board 13, and the communication board 17. The power supply board 18 includes an AF motor driver 181. The AF motor driver 181 supplies driving power to the AF motor 141 of the camera module 14 to realize an autofocus function. The AF motor driver 181 adjusts the power supplied to the AF motor 141 in response to an AF control signal from the control unit 13A.

[0024] The communication board 17 outputs image data about the inspection object, a user interface, and the like output from the control unit 13A to the display device 4, the personal computer 5, an external control device (not shown), and the like.

[0025] Furthermore, the communication board 17 accepts various operations input by the user from the touch panel 41 of the display device 4 or the keyboard 51 of the personal computer 5. The communication form between the communication board 17 and other devices may be wired or wireless, and either form of communication can be realized by a well-known communication module.

[0026] The storage device 19 is, for example, a storage device such as a hard disk drive. The storage device 19 stores a program file 100 and a setting file (software) for enabling various processes described later to be executed by hardware, learning images, verification images, classification results, and the like. For example, the program file 100 and the setting file may be stored in a storage medium 90 such as an optical disk, and the program file 80 and the setting file stored in the storage medium 90 may be installed in the control unit 2.

[0027] The display device 4 has a touch panel 41. The touch panel 41 of the display device 4 displays images transmitted from the control unit 2, a user interface screen, and the like. The touch panel 41 is, for example, a well-known touch-type operation panel equipped with a pressure sensor, and detects a touch operation by a user and outputs the detected operation to the communication board 17.

[0028] The personal computer 5 has a keyboard 51, a display panel such as a liquid crystal panel, a mouse, etc. The personal computer 5 accepts various operations by the user using operation devices such as the keyboard 51 and the mouse, and transmits them to the control unit 2. In addition, the personal computer 5 displays images transmitted from the control unit 2, user interface screens, etc.

[0029] Next, the functional configuration of the image inspection device 1 will be described.

[0030] FIG. 3 is a functional configuration diagram of an image inspection device according to an embodiment.

[0031] The image inspection device 1 (mainly, the control unit 2) includes an image receiving unit 21, a learning unit 22, a calculation unit 23, a classification unit 24, a display processing unit 25, and an instruction receiving unit 26. Each of these functional units (21 to 26) is configured, for example, by the control unit 13A executing a program file 100. At least one of the functional units may be configured only by hardware.

[0032] The image receiving unit 21 receives a learning image associated with a correct answer class for learning the machine learning model 23A of the calculation unit 23, and a plurality of verification images associated with a correct answer class for evaluating the accuracy of the machine learning model 23A. As a method for receiving images such as a learning image and a verification image, the name of a group of images that are grouped and stored in advance in the storage device 19 or the like, or the image name of each image may be received. The image receiving unit 21 may also receive a designation of an image as a learning image. The verification images may include the learning image.

[0033] The learning unit 22 is an example of a learning unit and a re-learning unit, and uses the training images accepted by the image accepting unit 21 to perform machine learning for the machine learning model 23A.

[0034] The calculation unit 23 has a machine learning model 23A. The machine learning model 23A calculates and outputs the degree of agreement of each of a plurality of classes that are candidates for classification for an input image to be processed (target image). The machine learning model 23A can be realized by a known model. The calculation unit 23 passes the degree of agreement of each of the plurality of classes calculated by the machine learning model 23A to the classification unit 24.

[0035] The classification unit 24 classifies the target image based on the degree of matching of each class received from the calculation unit 23. Specifically, the classification unit 24 classifies the target image into the class with the highest degree of matching.

[0036] The display processing unit 25 displays a user interface screen for evaluation (machine learning model evaluation screen 200 (see FIG. 5)) based on the images of the multiple verification images and the degree of match of each class calculated for the multiple verification images.

[0037] The instruction receiving unit 26 receives various instructions from the user via a displayed screen or the like, and notifies the display processing unit 25. For example, the instruction receiving unit 26 receives a designation of a class (designated class) to be the target of the separability graph displayed by the display processing unit 25, and notifies the display processing unit 25.

[0038] Next, the processing operation of the image inspection device 1 will be described.

[0039] FIG. 4 is a flowchart of a machine learning model setting process according to an embodiment.

[0040] The machine learning model setting process is executed, for example, before the image inspection device 1 starts operating to inspect a new inspection target.

[0041] First, the image receiving unit 21 receives one or more training images associated with correct answer classes for learning the machine learning model 23A of the calculation unit 23, and a plurality of verification images associated with correct answer classes for evaluating the accuracy of the machine learning model 23A (S11).

[0042] Next, the learning unit 22 executes machine learning for the machine learning model 23A using the received learning image (S12). Specifically, when the learning image is input to the machine learning model 23A, machine learning is performed so that the classification result of the classification unit 24 according to the output from the machine learning model 23A becomes the correct class. As the machine learning method, an existing method can be used.

[0043] Next, the learning unit 22 updates the machine learning model 23A to a model obtained as a result of the machine learning (S13).

[0044] Next, the calculation unit 23 and the classification unit 24 execute a classification process for each verification image (S14). Specifically, the calculation unit 23 calculates the degree of agreement for each class for each of the multiple verification images using the machine learning model 23A updated in step S13, and the classification unit 24 classifies each verification image into a class (inferred class) to which the verification image is inferred to belong, based on the degree of agreement for each class calculated for each verification image.

[0045] Next, the calculation unit 23 calculates a degree of separation by subtracting the maximum degree of matching among the classes other than the correct class from the minimum degree of matching among the correct class in the verification images based on the degree of matching of each class of each verification image in the verification image group for each verification image group (verification image group) having a common (identical) correct class (S15). Note that if the calculation result is negative, the degree of separation is set to 0.

[0046] Next, the display processing unit 25 performs a process of displaying a separability graph for the verification image group of the specified class (designated class), and performs a process of displaying the machine learning model evaluation screen 200 (S16). Here, when step S16 is executed for the first time, the designated class may be a predetermined class, for example, a class with the smallest degree of separation.

[0047] Here, the process of displaying a separation graph by the display processing unit 25 will be described using the machine learning model evaluation screen 200.

[0048] FIG. 5 is a diagram showing the configuration of a machine learning model evaluation screen according to one embodiment.

[0049] The machine learning model evaluation screen 200 is a screen displayed by the display processing unit 25, and includes a verification image display area 201 and a learning result display area 220.

[0050] In the verification image display area 201, a plurality of image areas 210 are displayed, which display each image used as a verification image. In addition, in the image area 210, a correct class component 212 indicating the correct class for the image in this area and an inferred class component 213 indicating the inferred class classified in the classification process for the image in this area are displayed in a distinguishable manner. The correct class component 212 and the inferred class component 213 are displayed in a manner (e.g., color, pattern, etc.) corresponding to the class. Therefore, when the correct class and the inferred class match, that is, when a normal judgment is made, the correct class component 212 and the inferred class component 213 are displayed in the same manner. On the other hand, when the correct class and the inferred class do not match, that is, when a wrong judgment is made, the correct class component 212 and the inferred class component 213 are displayed in a different manner. Therefore, the user can easily understand whether the image has been normally judged by referring to the manner of the correct class component 212 and the inferred class component 213. In this embodiment, in the image area 210 of an image in which the correct class and the inferred class do not match, a display component 214 indicating that the determination is incorrect is displayed. In addition, in the image area 210, if the corresponding image is a learning image, a learning image mark 211 is displayed. In the verification image display area 201, it is also possible to specify that the corresponding image be added as a learning image (instruction to add a learning image) by performing a predetermined operation on the image area 210.

[0051] The learning result display area 220 displays a confusion matrix display area 221 and a cumulative histogram display area 222. The confusion matrix display area 221 displays a confusion matrix showing the correspondence between the correct classes and the inferred classes for all accepted verification images. In the confusion matrix, for example, each class is arranged on the vertical axis as a correct class, and each class is arranged on the horizontal axis as an inferred class, and the number of corresponding verification images is stored in each section corresponding to the correct class and the inferred class. The display processing unit 25 creates and displays a confusion matrix based on the correct class associated with the verification image and the classification result (inferred class) by the classification unit 24.

[0052] The cumulative histogram display area 222 displays a specified class 223 indicating the correct class currently being displayed, a separation degree table 224 showing the separation degree in the verification image group for each correct class, and a separation degree graph (evaluation graph) 225 including a cumulative histogram of the degree of match for each class in the verification image group corresponding to the specified class being displayed.

[0053] In the separation degree table 224, the correct class and the separation degree for the correct class are displayed in association with each other. In this embodiment, when the user performs an operation of pressing the linked character 224A indicating the correct class, the instruction receiving unit 26 receives an instruction to change the designated class by setting the corresponding correct class as the designated class. In response to this, the display processing unit 25 creates a separation degree graph for the designated class and displays it in the cumulative histogram display area 222. The display processing unit 25 displays on the display device 3 the separation between the statistics of the degree of agreement of the predetermined correct class and the statistics of the degrees of agreement of other classes (e.g., A and C) that are different from the correct class, among the degrees of agreement of each class calculated for a group of verification images associated with a predetermined correct class (e.g., B) among a plurality of classes (e.g., A, B, C). Displaying the separation not only includes displaying the numerical values ​​shown in the separation table 224, but also including displaying the distribution of the degrees of agreement of the correct class in the evaluation graph shown in FIG. 6 in a manner that distinguishes it from the distribution of the degrees of agreement of other classes. In this embodiment, the separation degree based on the "minimum value" of the degree of agreement of the predetermined correct class and the "maximum value" of the degree of agreement of other classes that are classes different from the correct class is displayed on the display device 3. If the statistics of the degree of agreement of the predetermined correct class are equal to or less than the statistics of the degrees of agreement of other classes, the separation degree may be set to 0. When the degree of separation is calculated as a numerical value, the calculation process is executed by the display processing unit 25 or the calculation unit 23. In addition, the degree of match of each class may be converted into a corrected degree of match or a normalized degree of match, which will be described later, and the degree of separation may be displayed based on the statistics of each degree of match after the conversion. In addition, not only the minimum and maximum values ​​but also representative values ​​(average, median, mode) can be used as the statistics, and the degree of separation between the representative value of the agreement of the correct class and the representative value of the agreement of other classes may be displayed on the display device 3. In addition, when the degree of dispersion (variance, standard deviation) of the agreement of the correct class and the degree of dispersion of the agreement of other classes are equal to or less than a predetermined threshold, the degree of separation may be calculated to be higher. In an example in which an inappropriate verification image is included in terms of image quality or the like and an outlier occurs in the agreement of the correct class, the degree of separation is zero according to the comparison result using the minimum and maximum values, but if the degree of separation is based on the representative value, evaluation with reduced influence of the outlier is possible.

[0054] In the separability graph 225, the horizontal axis (first axis) indicates the degree of matching, and the vertical axis (second axis) indicates a value corresponding to the cumulative number of verification images (the cumulative number or a value corresponding to the cumulative number). The separability graph 225 includes a cumulative histogram (cumulative graph) of a verification image corresponding to the degree of matching of each class for a verification image group associated with a specified class. The cumulative histogram for the correct answer class is configured to accumulate the number of each image in the right direction (first direction) in which the degree of matching increases, and the cumulative histogram for classes other than the correct answer class (other classes) is configured to accumulate the number of each image in the left direction (second direction) in which the degree of matching decreases. In the separability graph 225, the difference in position when the left end of the cumulative histogram for the correct answer class is to the right of the right end of the cumulative histogram for the other classes indicates the degree of separation. Note that when the left end of the cumulative histogram for the correct answer class is not to the right of the right end of the cumulative histogram for the other classes, separation is not possible, and the degree of separation is 0.

[0055] A separability graph (an example of a first evaluation graph) 225 shown in FIG. 5 is a display example of a verification image group (first verification image group) in which the designated class (correct class) is B (an example of the first class), where the solid line is a cumulative histogram of the verification image for the degree of match (first degree of match) of the correct class B, the dotted line is a cumulative histogram of the verification image for the degree of match (second degree of match) of A (an example of the second class), and the dashed line is a cumulative histogram of the verification image for the degree of match of C. In the separability graph 225, the left end of the cumulative histogram for the correct class is not to the right of the right end of the cumulative histogram for the class of A, which indicates that the separability is 0. Since the separability is 0, it can be evaluated that the classification accuracy is not high for the verification image in which the correct class is B.

[0056] Next, a separation degree graph that is displayed when a linked character of class A in separation degree table 224 is pressed, and a separation degree graph that is displayed when a linked character of class B is pressed will be described.

[0057] Fig. 6 is a diagram illustrating another example of a separation degree graph according to an embodiment of the present invention, in which (A) is a separation degree graph displayed when a linked character of class A is pressed, and (B) is a separation degree graph displayed when a linked character of class C is pressed.

[0058] The separation degree graph 226 (an example of a second evaluation graph) shown in FIG. 6(A) is a separation degree graph for a verification image group (an example of a second verification image group) whose correct class is A, in which the dotted line is a cumulative histogram for the degree of agreement of the correct class A, the solid line is a cumulative histogram for the degree of agreement of B, and the dashed line is a cumulative histogram for the degree of agreement of C. In the separation degree graph 226, the left end of the cumulative histogram for the correct class A is to the right of the right end of the cumulative histograms for the classes B and C, and the difference between them is the separation degree. Here, the difference between them is 0.5 when the overall degree of agreement is 1, so the separation degree is 0.5. Since the separation degree is 0.5, it can be evaluated that the verification images whose correct class is A have been classified with relatively high accuracy, and in particular, the stability of the performance of the machine learning model is relatively high.

[0059] The separation degree graph 227 shown in FIG. 6(B) is a separation degree graph for a verification image group (an example of a second verification image group) whose correct class is C, in which the dashed line is a cumulative histogram for the degree of agreement of the correct class C, the solid line is a cumulative histogram for the degree of agreement of B, and the dotted line is a cumulative histogram for the degree of agreement of A. In the separation degree graph 227, the left end of the cumulative histogram for the correct class C is to the right of the right end of the cumulative histograms for the classes A and B, and the difference between them is the separation degree. Here, the difference between them is 0.9 when the overall degree of agreement is 1, so the separation degree is 0.9. Since the separation degree is 0.9, it can be evaluated that the verification image whose correct class is C can be classified with very high accuracy, and in particular, the stability of the performance of the machine learning model is very high. In other words, it can be evaluated that the stability of the performance of the machine learning model for the verification image whose correct class is C is higher than the stability of the performance of the machine learning model for the verification image whose correct class is A.

[0060] Returning to the description of FIG. 4, after the machine learning model evaluation screen 200 is displayed in step S16, the instruction receiving unit 26 determines whether or not an instruction to change the designated class has been received (S17).

[0061] As a result, when an instruction to change the designated class is received, specifically, when linked character 224A or the like on the machine learning model evaluation screen 200 is pressed (S17: Yes), the instruction receiving unit 26 notifies the display processing unit 25 of the instruction to change the designated class, and proceeds to step S16. As a result, the display processing unit 25 creates a separation degree graph corresponding to the designated class, and displays it on the machine learning model evaluation screen 200 (S16).

[0062] On the other hand, if the instruction to change the designated class has not been received (S17: No), the instruction receiving unit 26 determines whether an instruction to add a learning image (learning image addition instruction) has been received (S18).

[0063] As a result, when the instruction to add a learning image is received (S18: Yes), the instruction receiving unit 26 adds the image targeted by the instruction to the learning images (S19), and the process proceeds to step S12. As a result, the machine learning model 23A is re-learned using a group of learning images including the newly added image, and the new machine learning model 23A can be evaluated. Then, using the machine learning model 23A updated in step S13, the updated matching degree of each class is calculated, and the classification unit 24 reclassifies each verification image into a class (inferred class) to which the verification image is inferred to belong based on the updated matching degree of each class calculated for each verification image (S14). The display processing unit 25 updates the separation degree (including the evaluation graph) based on the distribution of the updated matching degree of each class, and displays it on the display device 3 (S15 to S16).

[0064] On the other hand, if the instruction to add a learning image has not been received (S18: No), the instruction receiving unit 26 determines whether or not an instruction to end the setting process has been received (S20).

[0065] As a result, if an instruction to end the setting process has not been received (S20: No), the instruction receiving unit 26 advances the process to step S17. On the other hand, if an instruction to end the setting process has been received (S20: Yes), the instruction receiving unit 26 notifies the calculation unit 23 that an instruction to end the setting process has been received, and the calculation unit 23 registers the current machine learning model 23A as a machine learning model (operation model) to be used during operation (S21), and ends the setting process. This enables the image inspection device 1 to perform appropriate inspection using the newly registered operation model.

[0066] According to the above-described machine learning model setting process, the trained machine learning model 23A can be appropriately evaluated based on the degree of separation, etc. In addition, it becomes possible to create a machine learning model 23A with higher classification accuracy by adding training images and retraining the machine learning model 23A.

[0067] Next, another example of the verification image display area will be described.

[0068] FIG. 7 is another example of a verification image display area according to an embodiment.

[0069] In the machine learning model evaluation screen 200 shown in Fig. 5, the display processing unit 25 is configured to display a verification image display area 201 including a plurality of image-by-image areas 210 for all images used as verification images. In contrast, the display processing unit 25 may be configured to preferentially display a verification image associated with a designated class among the verification images, as shown in a verification image display area 230. Here, preferential display includes displaying only the verification image associated with the designated class, or displaying verification images not associated with the designated class, but displaying the verification images associated with the designated class side by side from the head position (e.g., the upper left position) of the verification image display area.

[0070] Such a verification image display area 230 allows the user to easily recognize the verification image associated with the specified class, and makes it easier for the user to understand how the verification image associated with the specified class is classified.

[0071] Next, an image inspection device according to a modified example will be described.

[0072] The image inspection device according to the modified example is a device in which additional processing is performed in the calculation unit 23. The calculation unit 23 according to the modified example is an example of a corrected coincidence calculation unit.

[0073] The calculation unit 23 calculates the degree of match of each class for a plurality of verification images using the machine learning model 23A. The calculation unit 23 manages the degree of match calculated for each verification image in, for example, a working degree of match table 300 (see FIG. 8).

[0074] FIG. 8 is a configuration diagram of a matching degree table according to an embodiment.

[0075] The matching degree table 300 stores an entry for each verification image. The entry of the matching degree table 300 includes fields for image name 301, correct class 302, first place class and matching degree 303, second place class and matching degree 304, and third place class and matching degree 305.

[0076] Image name 301 stores the image name corresponding to the verification image corresponding to the entry. Correct class 302 stores the correct class associated with the verification image corresponding to the entry. First place class and degree of agreement 303 stores the maximum (first place) class (first place class) among the degrees of agreement of each class calculated for the verification image corresponding to the entry, and the degree of agreement. Second place class and degree of agreement 304 stores the second place class (second place class) among the degrees of agreement of each class calculated for the verification image corresponding to the entry, and the degree of agreement. Third place class and degree of agreement 305 stores the third place class (third place class) among the degrees of agreement of each class calculated for the verification image corresponding to the entry, and the degree of agreement.

[0077] In the above embodiment, the degree of separation is calculated using the degree of agreement calculated by the machine learning model 23A (the degree of agreement registered in the degree of agreement table 300), and the degree of separation graph is displayed.

[0078] Here, for example, as shown in the matching table 300 shown in FIG. 8, the first-ranked class of the verification image of the second entry is B and the matching degree is calculated as 54, the first-ranked class of the verification image of the third entry is B and the matching degree is calculated as 97, and the second-ranked class is A and the matching degree is calculated as 67. In this case, the first-ranked class of the verification image of the second entry and the verification image of the third entry is B, and the same class as the correct class B is appropriately classified as the inferred class. When the separation degree is calculated in this case, the matching degree of the second-ranked class of the verification image of the third entry is smaller than the matching degree of the first-ranked class of the verification image of the second entry, so the separation degree becomes 0. Therefore, even if the images are appropriately classified, the separation degree may become 0.

[0079] Therefore, the calculation unit 23 according to the modified example further determines a reference value for correcting the degree of matching for each verification image based on the maximum degree of matching for the verification image (first-place degree of matching) and other degrees of matching (other degrees of matching), corrects each degree of matching based on the reference value, and calculates the corrected degree of matching for each class. The calculation unit 23 manages the corrected degrees of matching, for example, in a corrected degree of matching table 310 (see FIG. 9). Here, the reference value may be a value for an internal or external division point between the first-place degree of matching and the other degrees of matching. For example, the other degree of matching may be the second-place degree of matching (second-place degree of matching), and in that case, for example, the value (for example, the average value) of the internal division point between the first-place degree of matching and the second-place degree of matching may be used as the reference point.

[0080] Fig. 9 is a diagram showing the configuration of a corrected matching degree table according to a modified example. A corrected matching degree table 310 in Fig. 9 shows an example in which the calculation unit 23 calculates a corrected matching degree for each verification image in the matching degree table 300 shown in Fig. 8 by using an average value of the first-ranked matching degree and the second-ranked matching degree as a reference value.

[0081] The corrected matching degree table 310 stores an entry for each verification image. The entry of the corrected matching degree table 310 includes fields for image name 311, correct class 312, Top 313, Second 314, Center 315, first class and corrected matching degree 316, second class and corrected matching degree 317, and third class and corrected matching degree 318.

[0082] Image name 311 stores the image name corresponding to the verification image corresponding to the entry. Correct class 312 stores the correct class associated with the verification image corresponding to the entry. Top 313 stores the first-place class matching degree (first-place matching degree) for the verification image for the entry. Second 314 stores the second-place class matching degree (second-place matching degree) for the verification image for the entry. Center 315 stores a reference value for the verification image for the entry. In the example of FIG. 9, the reference value is the average value of the first-place matching degree and the second-place matching degree.

[0083] The first-place class and corrected matching degree 316 stores the first-place class for the verification image corresponding to the entry and its corrected matching degree. In the example of Fig. 9, the corrected matching degree is a value obtained by subtracting a reference value from the first-place matching degree. The second-place class and corrected matching degree 317 stores the second-place class for the verification image corresponding to the entry and its corrected matching degree. The third-place class and corrected matching degree 318 stores the third-place class for the verification image corresponding to the entry and its corrected matching degree.

[0084] For example, when processing is performed on the verification image (image name is Image 103) corresponding to the third entry in Figure 8, as shown in the third entry of the corrected matching degree table 310, "97" is stored in Top 313, "67" is stored in Second 314, "82" is stored in Center 315 because (97 + 67) / 2 = 82, "15" is stored as the corrected matching degree because 97 - 82 = 15 in the first class and corrected matching degree 316, "-15" is stored as the corrected matching degree because 67 - 82 = -15 in the second class and corrected matching degree 317, and "-73" is stored as the corrected matching degree because 9 - 82 = -73 in the third class and corrected matching degree 318.

[0085] The calculation unit 23 may calculate the separation degree using a corrected matching degree instead of the matching degree. For example, in a case where the separation degree is 0 when the matching degree shown in Fig. 8 is used, that is, for the verification image of the second entry and the verification image of the third entry, when the corrected matching degree is used, the separation degree can be calculated as 11-(-15)=26, and it becomes possible to appropriately evaluate that they are separated.

[0086] Here, the display processing unit 25 may perform the process of displaying the separation graph using a corrected degree of coincidence instead of the degree of coincidence.

[0087] Furthermore, the calculation unit 23 may calculate a degree of matching normalized for the corrected degree of matching (normalized degree of matching). In this case, the calculation unit 23 identifies the maximum and minimum values ​​of the corrected degree of matching in all the verification images, and normalizes the maximum and minimum values ​​of the corrected degree of matching so that they fall within a predetermined range (for example, 0-100). Here, the calculation unit 23 in this case is an example of a normalized degree of matching calculation unit. The calculation unit 23 manages the degree of matching normalized for each verification image (normalized degree of matching) in, for example, a working normalized degree of matching table 320 (see FIG. 10).

[0088] Fig. 10 is a diagram showing the configuration of a normalized matching degree table according to a modified example. A normalized matching degree table 320 in Fig. 10 shows an example in which the calculation unit 23 normalizes each verification image in the corrected matching degree table 310 shown in Fig. 9.

[0089] The normalized matching score table 320 stores an entry for each verification image. The entry of the normalized matching score table 320 includes fields for an image name 321, a correct class 322, a first-ranked class and normalized matching score 323, a second-ranked class and normalized matching score 324, and a third-ranked class and normalized matching score 325.

[0090] Image name 321 stores the image name corresponding to the verification image corresponding to the entry. Correct class 322 stores the correct class associated with the verification image corresponding to the entry. First place class and normalized matching degree 323 stores the first place class for the verification image corresponding to the entry and its normalized matching degree. Second place class and normalized matching degree 324 stores the second place class for the verification image corresponding to the entry and its normalized matching degree. Third place class and normalized matching degree 325 stores the third place class for the verification image corresponding to the entry and its normalized matching degree.

[0091] For example, when processing is performed on the verification image (image name is Image 103) corresponding to the third entry in Figure 9, as shown in the third entry of normalized matching table 320, the first class and normalized matching degree 323 stores the normalized matching degree of "73", the second class and normalized matching degree 324 stores the normalized matching degree of "48", and the third class and normalized matching degree 325 stores the normalized matching degree of "0".

[0092] The calculation unit 23 may calculate the separation degree by using the normalized coincidence degree instead of the coincidence degree. Also, the display processing unit 25 may perform separation degree graph display processing by using the normalized coincidence degree instead of the coincidence degree.

[0093] The present invention is not limited to the above-described embodiment and modified examples, and can be appropriately modified and implemented without departing from the spirit of the present invention.

[0094] For example, in the above embodiment, the separation graph includes a distribution graph of the degree of similarity for each class, but the present invention is not limited to this. For example, the separation graph may include a distribution graph of the correct class and a distribution graph that brings together the other classes.

[0095] The present invention also includes, for example, a classification evaluation support device, a classification evaluation support method, and a classification evaluation support program according to Supplementary Notes 1 to 15 below.

[0096] (Appendix 1) A classification evaluation support device that supports evaluation of accuracy of a machine learning model used in a classification process for classifying a predetermined image into which of a plurality of classes the image belongs, a calculation unit that calculates a degree of correspondence between each of a plurality of verification images associated with a correct class by using the machine learning model; and a classification unit that classifies each of the verification images into which of the classes the verification images belong based on the degree of coincidence of each of the classes calculated for each of the verification images; a display processing unit that displays, on a predetermined display device, a degree of separation between a statistical amount of the degree of match of the correct class and a statistical amount of the degree of match of another class that is a class different from the correct class, among the degrees of match of each class calculated for a group of verification images associated with a predetermined correct class; A classification evaluation support device comprising:

[0097] (Appendix 2) the statistics of the degree of agreement of the correct class is a minimum value of the degree of agreement of the correct class, and the statistics of the degree of agreement of the other classes is a maximum value of the degree of agreement of the other classes, The display processing unit displays a separation degree between a minimum value of the degree of agreement of the correct class and a maximum value of the degree of agreement of the other classes on the display device. 2. A classification evaluation support device as described in appendix 1.

[0098] (Appendix 3) When a selection of a verification image group associated with the predetermined correct class is selected, the display processing unit generates an evaluation graph that displays a distribution of the degree of agreement of the predetermined correct class and a distribution of the degree of agreement of the other classes in a distinguished manner, and causes the display device to display the evaluation graph. 3. A classification evaluation support device according to claim 1 or 2.

[0099] (Appendix 4) The display processing unit creates the evaluation graph for each correct class corresponding to each of the plurality of classes, and displays the evaluation graph for each correct class. 4. A classification evaluation support device as described in appendix 3.

[0100] (Appendix 5) The multiple classes include three or more classes, and the other classes include two or more classes, The display processing unit causes the evaluation graph to include a distribution of the degree of coincidence for each class included in the other classes. 5. The classification evaluation support device according to claim 3 or 4.

[0101] (Appendix 6) the plurality of classes includes a first class and a second class, the calculation unit calculates, for each of the verification images, a first degree of match of the first class and a second degree of match of the second class; the display processing unit generates a first evaluation graph for a first verification image group whose correct class is the first class, in which a distribution of the first degree of agreement of each verification image in the first verification image group is displayed in a manner distinguished from a distribution of the second degree of agreement of each verification image in the first verification image group, and generates a second evaluation graph for a second verification image group whose correct class is the second class, in which a distribution of the first degree of agreement of each verification image in the second verification image group is displayed in a manner distinguished from a distribution of the second degree of agreement of each verification image in the second verification image group, Displaying at least one of the first evaluation graph and the second evaluation graph on the display device. 6. A classification evaluation support device according to any one of claims 1 to 5.

[0102] (Appendix 7) When the display processing unit causes the display device to display the first evaluation graph for the first verification image group with priority over the second evaluation graph for the second verification image group, the display device displays the first verification image group with priority over the second verification image group, and displays the first class, which is the correct class, and an inferred class into which each verification image of the first verification image group is classified, in a distinguishable manner. 7. A classification evaluation support device as described in appendix 6.

[0103] (Appendix 8) a corrected matching degree calculation unit that determines a reference value for correcting the matching degree of the verification image based on a maximum matching degree among the matching degrees of each class with respect to the verification image and other matching degrees for each of the verification images, and calculates a corrected matching degree by correcting the matching degree of each class with respect to the verification image based on the reference value; The display processing unit creates the evaluation graph by using the corrected degree of coincidence instead of the degree of coincidence. 8. A classification evaluation support device according to any one of appendix 3 to appendix 7.

[0104] (Appendix 9) The corrected matching degree calculation unit determines, as the reference value, a point that divides the maximum matching degree and the second maximum matching degree of the verification image internally or externally. 9. A classification evaluation support device as described in appendix 8.

[0105] (Appendix 10) The corrected matching degree calculation unit determines an average value of the maximum matching degree and the second maximum matching degree of the verification image as the reference value. 10. A classification evaluation support device as described in appendix 9.

[0106] (Appendix 11) a normalized matching degree calculation unit that normalizes the corrected matching degrees of each of the verification images so that a maximum corrected matching degree and a minimum corrected matching degree in all of the verification images are within a predetermined range, 11. The classification evaluation support device according to claim 8, wherein the display processing unit creates the evaluation graph by using the normalized degree of coincidence instead of the degree of coincidence.

[0107] (Appendix 12) The evaluation graph has a degree of match on a predetermined first axis and information corresponding to the number of verification images on a second axis intersecting the first axis, and is a cumulative graph in which, for the degree of match of the correct class, information corresponding to the number of verification images is accumulated in a first direction of the first axis, and, for the degree of match of the other classes, information corresponding to the number of verification images is accumulated in a second direction opposite to the first direction. 12. A classification evaluation support device according to any one of appendix 3 to appendix 11.

[0108] (Appendix 13) a reception unit that receives designation of an image to be newly used as a learning image from among the plurality of verification images that are identifiably displayed on the display device by the display processing unit; A learning unit that learns the machine learning model using learning images including the received image, The learning unit is configured to be capable of training the machine learning model using a verification image that has been misclassified into the other class among the verification image group classified by the classification unit, The calculation unit calculates an updated matching degree for each of the classes of each of the verification images by using the machine learning model trained by the learning unit, The classification unit reclassifies each of the verification images into one of the classes based on the updated matching degree of each of the classes; The display processing unit causes the display device to display the degree of separation by using the updated degree of match instead of the degree of match. 13. A classification evaluation support device according to any one of appendix 1 to appendix 12.

[0109] (Appendix 14) A classification evaluation support method using a classification evaluation support device that supports evaluation of accuracy of a machine learning model used in a classification process for classifying a predetermined image into one of multiple classes, comprising: Using the machine learning model, a degree of match of each of the multiple classes is calculated for each of the multiple verification images associated with a correct class; classifying each of the verification images into one of the classes based on the degree of match of each of the classes calculated for each of the verification images; Among the degrees of agreement of each class calculated for the verification image group associated with a predetermined correct class, a degree of separation between the statistics of the degree of agreement of the correct class and the statistics of the degrees of agreement of other classes that are classes different from the correct class is displayed on a predetermined display device. A classification evaluation support method.

[0110] (Appendix 15) A classification evaluation support program executed by a computer to support evaluation of accuracy of a machine learning model used in a classification process for classifying a predetermined image into one of a plurality of classes, The computer, a calculation unit that calculates a degree of correspondence between each of a plurality of verification images associated with a correct class by using the machine learning model; and a classification unit that classifies each of the verification images into one of the classes based on the degree of match of each of the classes calculated for each of the verification images; and a display processing unit that displays, on a predetermined display device, a degree of separation between a statistical amount of the degree of match of the correct class and a statistical amount of the degree of match of another class that is a class different from the correct class, among the degrees of match of each class calculated for a group of verification images associated with a predetermined correct class. Classification evaluation support program. [Explanation of symbols]

[0111] 1...image inspection device, 2...control unit, 3...imaging unit, 4...display device, 5...personal computer, 13A...control unit, 21...image reception unit, 22...learning unit, 23...calculation unit, 23A...machine learning model, 24...classification unit, 25...display processing unit, 26...instruction reception unit

Claims

1. A classification evaluation support device that supports evaluation of accuracy of a machine learning model used in a classification process for classifying a predetermined image into one of a plurality of classes, a calculation unit that calculates a degree of correspondence between each of a plurality of verification images associated with a correct class by using the machine learning model; and a classification unit that classifies each of the verification images into one of the classes based on the degree of match of each of the classes calculated for each of the verification images; a display processing unit that displays, on a predetermined display device, a degree of separation between a statistical amount of the degree of match of the correct class and a statistical amount of the degree of match of another class that is a class different from the correct class, among the degrees of match of each class calculated for a group of verification images associated with a predetermined correct class; A classification evaluation support device comprising:

2. the statistics of the degree of agreement of the correct class is a minimum value of the degree of agreement of the correct class, and the statistics of the degree of agreement of the other classes is a maximum value of the degree of agreement of the other classes, The display processing unit displays the degree of separation between the minimum degree of agreement of the correct class and the maximum degree of agreement of the other classes on the display device. The classification evaluation support device according to claim 1 .

3. When a verification image group associated with the predetermined correct class is selected, the display processing unit generates an evaluation graph that displays a distribution of the degree of agreement of the predetermined correct class and a distribution of the degree of agreement of the other classes in a distinguished manner, and causes the display device to display the evaluation graph. The classification evaluation support device according to claim 1 .

4. The display processing unit creates the evaluation graph for each correct class corresponding to each of the plurality of classes, and displays the evaluation graph for each correct class. The classification evaluation support device according to claim 3.

5. The multiple classes include three or more classes, and the other classes include two or more classes, The display processing unit causes the evaluation graph to include a distribution of the degree of coincidence for each class included in the other classes. The classification evaluation support device according to claim 3.

6. the plurality of classes includes a first class and a second class, the calculation unit calculates, for each of the verification images, a first degree of match of the first class and a second degree of match of the second class; the display processing unit generates a first evaluation graph for a first verification image group whose correct class is the first class, in which a distribution of the first degree of agreement of each verification image in the first verification image group is displayed in a manner distinguished from a distribution of the second degree of agreement of each verification image in the first verification image group, and generates a second evaluation graph for a second verification image group whose correct class is the second class, in which a distribution of the first degree of agreement of each verification image in the second verification image group is displayed in a manner distinguished from a distribution of the second degree of agreement of each verification image in the second verification image group, Displaying at least one of the first evaluation graph and the second evaluation graph on the display device. The classification evaluation support device according to claim 1 .

7. When the display processing unit causes the display device to display the first evaluation graph related to the first verification image group preferentially over the second evaluation graph related to the second verification image group, the display device displays the first verification image group preferentially over the second verification image group and displays the inference class classified by the classification unit of each verification image of the first verification image group. The classification evaluation support device according to claim 6.

8. a corrected matching degree calculation unit that determines a reference value for correcting the matching degree of the verification image based on a maximum matching degree among the matching degrees of each class with respect to the verification image and other matching degrees for each of the verification images, and calculates a corrected matching degree by correcting the matching degree of each class with respect to the verification image based on the reference value; The display processing unit causes the display device to display the separation degree by using the corrected coincidence degree instead of the coincidence degree. The classification evaluation support device according to claim 1 .

9. The corrected matching degree calculation unit determines, as the reference value, a point that divides the maximum matching degree and the second maximum matching degree of the verification image internally or externally. The classification evaluation support device according to claim 8.

10. The corrected matching degree calculation unit determines an average value of the maximum matching degree and the second maximum matching degree of the verification image as the reference value. The classification evaluation support device according to claim 9.

11. a normalized matching degree calculation unit that normalizes the corrected matching degrees of each of the verification images so that a maximum corrected matching degree and a minimum corrected matching degree in all of the verification images are within a predetermined range, 9. The classification evaluation support device according to claim 8, wherein the display processing unit causes the display device to display the degree of separation by using the normalized degree of coincidence instead of the degree of coincidence.

12. The evaluation graph is a cumulative graph in which the degree of match is plotted on a predetermined first axis, information corresponding to the number of verification images is plotted on a second axis intersecting the first axis, and the degree of match of the correct class is a cumulative graph in which information corresponding to the number of verification images is accumulated in a first direction of the first axis, and the degree of match of the other classes is a cumulative graph in which information corresponding to the number of verification images is accumulated in a second direction opposite to the first direction. The classification evaluation support device according to claim 3.

13. a reception unit that receives designation of an image to be newly used as a learning image from among the plurality of verification images that are identifiably displayed on the display device by the display processing unit; A learning unit that learns the machine learning model using learning images including the received image, The learning unit is configured to be able to train the machine learning model using a verification image that has been misclassified into the other class among the verification image group classified by the classification unit, The calculation unit calculates an updated matching degree for each of the classes of each of the verification images by using the machine learning model trained by the learning unit, The classification unit reclassifies each of the verification images into one of the classes based on the updated matching degree of each of the classes; The display processing unit causes the display device to display the degree of separation by using the updated degree of match instead of the degree of match. The classification evaluation support device according to claim 1 .

14. A classification evaluation support method using a classification evaluation support device that supports evaluation of accuracy of a machine learning model used in a classification process for classifying a predetermined image into one of multiple classes, comprising: Using the machine learning model, a degree of match of each of the multiple classes is calculated for each of the multiple verification images associated with a correct class; classifying each of the verification images into one of the classes based on the degree of match of each of the classes calculated for each of the verification images; Among the degrees of agreement of each class calculated for the verification image group associated with a predetermined correct class, a degree of separation between the statistics of the degree of agreement of the correct class and the statistics of the degrees of agreement of other classes that are classes different from the correct class is displayed on a predetermined display device. A classification evaluation support method.

15. A classification evaluation support program executed by a computer to support evaluation of accuracy of a machine learning model used in a classification process for classifying a predetermined image into one of a plurality of classes, The computer, a calculation unit that calculates a degree of correspondence between each of a plurality of verification images associated with a correct class by using the machine learning model; and a classification unit that classifies each of the verification images into one of the classes based on the degree of match of each of the classes calculated for each of the verification images; and a display processing unit that displays, on a predetermined display device, a degree of separation between a statistical amount of the degree of match of the correct class and a statistical amount of the degree of match of another class that is a class different from the correct class, among the degrees of match of each class calculated for a group of verification images associated with a predetermined correct class. Classification evaluation support program.

Images