Learning model evaluation support device and learning model evaluation support program

Abstract

To provide a learning model evaluation support device and a learning model evaluation support program which allow for easily evaluating a characteristic of a learning model.SOLUTION: A learning model evaluation support device 100 includes a model information display unit 110, a model selection unit 120, and an output display unit 150. The model information display unit 110 selectively displays a plurality of learning models which are stored in a prescribed storage area and share an interface, on a display device 260. The model selection unit 120 selects at least one of the plurality of learning models displayed on the display device 260. The output display unit 150 displays an output result from at least one learning model on the display device 260.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present invention relates to a learning model evaluation support device and a learning model evaluation support program. [Background technology]

[0002] In the field of AI (Artificial Intelligence), a learning model such as a neural network is subjected to machine learning of a learning data set to create a trained learning model. For example, Patent Document 1 describes a learning device that generates a learning data set and creates a learning model by learning the relationship between a substrate image and the presence or absence of defects through machine learning using the learning data set. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-047104 Summary of the Invention [Problem to be solved by the invention]

[0004] The characteristics of a learning model vary depending on the algorithm used to create the learning model. Therefore, multiple learning models may be created using multiple different algorithms. In this case, a user evaluates the characteristics of each learning model and selects and uses the learning model with the characteristics most suitable for the purpose from the multiple learning models created. Therefore, it is desirable to develop a device that can easily evaluate the characteristics of a learning model.

[0005] An object of the present invention is to provide a learning model evaluation support device and a learning model evaluation support program that can easily evaluate the characteristics of a learning model. [Means for solving the problem]

[0006] A learning model evaluation support device according to one aspect of the present invention comprises a model information display unit that displays multiple learning models stored in a predetermined memory area and having the same interface on a display device in a selectable manner, a model selection unit that selects at least one learning model from the multiple learning models displayed on the display device, and an output display unit that displays the output results from the at least one learning model on the display device.

[0007] A learning model evaluation support program according to another aspect of the present invention causes a computer to execute a model information display process that displays multiple learning models stored in a specified memory area and with the same interface in a selectable manner on a display device, a model selection process that selects at least one learning model from the multiple learning models displayed on the display device, and an output display process that displays the output results from the at least one learning model on the display device. [Effects of the Invention]

[0008] According to the present invention, the characteristics of a learning model can be easily evaluated. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing a learning model evaluation support system including a learning model evaluation support device according to one embodiment of the present invention. [Figure 2] FIG. 10 is a diagram for explaining a specific example of a classification task. [Figure 3] FIG. 10 is a diagram illustrating an example of a confusion matrix stored in a storage device. [Figure 4] FIG. 10 is a diagram illustrating an example of a menu screen. [Figure 5] FIG. 10 is a diagram illustrating an example of a queue screen. [Figure 6] FIG. 10 is a diagram illustrating an example of a learning curve screen. [Figure 7] FIG. 10 is a diagram illustrating an example of a learning result screen. [Figure 8] FIG. 10 is a diagram illustrating an example of a details screen. [Figure 9]FIG. 10 is a diagram showing an example of an expanded details screen. [Figure 10] FIG. 10 is a diagram showing an example of an expanded details screen. [Figure 11] FIG. 10 is a diagram showing an example of an expanded details screen. [Figure 12] FIG. 10 is a diagram showing an example of an expanded details screen. [Figure 13] FIG. 10 is a diagram illustrating an example of a weight adjustment screen. [Figure 14] FIG. 10 is a diagram illustrating an example of a test evaluation screen. [Figure 15] FIG. 10 is a diagram showing an example of an arrangement screen. [Figure 16] FIG. 10 is a diagram illustrating an example of a superimposed screen. [Figure 17] 2 is a block diagram showing the functional configuration of the learning model evaluation support device of FIG. 1. [Figure 18] 10 is a flowchart showing a detailed screen display process. [Figure 19] 10 is a flowchart showing a probability calculation process. [Figure 20] 10 is a flowchart illustrating a weight adjustment process. [Figure 21] 10 is a flowchart showing a test evaluation display process. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1. Learning model evaluation support system A learning model evaluation support device and a learning model evaluation support program according to one embodiment of the present invention will be described below with reference to the drawings. Figure 1 is a diagram showing a learning model evaluation support system including a learning model evaluation support device according to one embodiment of the present invention. As shown in Figure 1, the learning model evaluation support system 200 includes a CPU 210 (central processing unit), a RAM 220 (random access memory), a ROM (read only memory) 230, a storage device 240, an operation unit 250, a display device 260, an input / output I / F (interface) 270, and a bus 280.

[0011] The CPU 210, RAM 220, ROM 230, storage device 240, operation unit 250, display device 260, and input / output I / F 270 are connected to a bus 280. The CPU 210, RAM 220, and ROM 230 constitute a learning model evaluation support device 100. The learning model evaluation support device 100 may be realized by a computer including the CPU 210, RAM 220, and ROM 230.

[0012] The CPU 210 executes the learning model evaluation support program to perform the learning model evaluation support process described below. The RAM 220 is, for example, a volatile memory, and is used as a work area for the CPU 210 and temporarily stores various data. The ROM 230 is, for example, a non-volatile memory, and stores computer programs such as system programs.

[0013] The storage device 240 includes a storage medium such as a hard disk, optical disk, magnetic disk, or memory card, and stores a learning model evaluation support program in advance. The storage device 240 also stores various information. The information stored in the storage device 240 will be described later. A recording medium 290 such as a CD (compact disc)-ROM can be detachably attached to the storage device 240. Therefore, the learning model evaluation support program may be provided in a form stored on the recording medium 290, and installed in the storage device 240 by connecting the recording medium 290 to the storage device 240.

[0014] The operation unit 250 is an input device such as a keyboard or a mouse. The operation unit 250 is operated by a user to perform various designations or selections. The display device 260 includes, for example, a liquid crystal display. The display device 260 displays various screens including a GUI (graphical user interface). The operation unit 250 and the display device 260 may be integrally configured as a touch panel device.

[0015] The input / output I / F 270 is connected to a network and connects the CPU 210 to other computers connected to the network. When the input / output I / F 270 is connected to a communication network, a learning model evaluation support program distributed from a server computer connected to the communication network may be installed in the storage device 240. The learning model evaluation support program may be installed in the ROM 230.

[0016] 2.Storage device The storage device 240 stores a plurality of learning models, a plurality of detailed information, and a plurality of confusion matrices. The plurality of detailed information is information used when creating the plurality of learning models. Therefore, each learning model is stored in association with the detailed information used when creating the learning model. Each of the plurality of confusion matrices is one of the output results of the plurality of learning models. Therefore, each learning model is stored in association with the confusion matrix of the learning model. Note that the plurality of learning models, the plurality of detailed information, and the plurality of confusion matrices may be stored in a server computer or a database computer instead of the storage device 240.

[0017] Multiple learning models stored in the storage device 240 or the like have the same interface. The interface is a general term for the shape and effect of the input and output of a learning model. Specifically, when a learning model has an interface, it means that the learning model accepts one or more combinations of number strings for each input and output, has one or more functions for processing the number strings, and clearly indicates the format of each input and output and the method of instructing the functions. In other words, multiple learning models with the same interface have the same input and output formats, and the same type of function and the same method of instructing the functions.

[0018] In this example, each learning model performs a classification task of classifying an image to be classified into one of multiple classes. Specifically, each learning model identifies an object included in the image to be classified and classifies the identified object into one of multiple classes. Here, the multiple learning models are created using different detailed information. Therefore, the characteristics of the multiple learning models differ from one another. The detailed information includes multiple items of information. The multiple items of information are, for example, information about the training data, information about the augmentation method, and information about the learning network or hyperparameters.

[0019] The learning data is image data to which label information, which is a target variable in supervised learning, is assigned, and is stored in the storage device 240 of Fig. 1 or the like. The label information is information for identifying objects included in an image represented by the image data. For example, when a learning model is trained on a classification task of classifying objects included in an image represented by image data into one of dogs, cats, birds, and deer, the label information is a value indicating one of dogs, cats, birds, and deer.

[0020] Creating each learning model requires a large amount of training data. Therefore, image data used as training data may be duplicated by padding. Padding techniques include enlargement, reduction, inversion, rotation, translation, blackening, and homography transformation. Enlargement is a process of generating image data in which the object portion of the image data is enlarged. Reduction is a process of generating image data in which the object portion of the image data is reduced.

[0021] Inversion processing is processing that generates image data in which the orientation of an object in the image data is inverted. The orientation of the object includes the up-down direction or the left-right direction. Rotation processing is processing that generates image data in which an object in the image data is rotated. The rotation angle may be a predetermined angle or an angle specified by the user. Translation processing is processing that generates image data in which an object in the image data is translated in parallel. The number of pixels or direction of translation may be a predetermined number of pixels or a direction, or may be a pixel or a direction specified by the user.

[0022] The blackening process is a process for generating image data by converting pixel values ​​of unnecessary parts of the image data other than the object into values ​​indicating black. The homography conversion process is a process for generating image data by converting an object in the image data into another object using a homography matrix.

[0023] The training network is a deep learning network used to generate a training model. Specifically, the training network includes an EfficientNet, a Convolutional Neural Network (CNN), or a Simple Word-Embedding-based Methods Neural Network (SWEM NN).

[0024] Hyperparameters are parameters used to control the behavior of various algorithms defined in a learning model. Hyperparameters include, for example, the number of learning rounds (number of epochs), patch size, optimization function, or learning rate. Note that patch size is a value that determines the maximum amount of learning data learned by a learning model at one time during machine learning. Hyperparameters may also include the number of layers of a neural network, the number of neurons per layer, kernel size, etc.

[0025] 3. Learning Model The learning model classifies objects contained in the target image into one of multiple classes. Specifically, the predicted probability that the object contained in the target image falls into each of the multiple classes is output as the objective variable. The object is classified into the class with the highest predicted probability among the multiple classes.

[0026] Here, even if the predicted probability for a particular class is not the highest, if the predicted probability for that class is relatively high, it may be desirable to classify the object into that class. Therefore, a weighting factor is set for each class. This can increase the sensitivity for identifying a particular class. Note that the multiple weighting factors corresponding to the multiple classes are set so that their sum total is 1.

[0027] The predicted probability for each class is corrected by multiplying the predicted probability output for that class by the weighting coefficient set for that class. The predicted probability for each class is then updated so that the sum of the multiple predicted probabilities corresponding to the multiple classes becomes 1. In this way, if the predicted probability for a particular class is relatively high, the object can be classified into that class.

[0028] FIG. 2 is a diagram illustrating a specific example of a classification task. As shown in FIG. 2, the learning model in this example classifies objects included in an image to be classified into one of dog, cat, bird, and deer. Therefore, a predicted probability that an object included in an image to be classified corresponds to each of dog, cat, bird, and deer is calculated. In the examples shown in the upper and center left columns of FIG. 2, the predicted probabilities for dog, cat, bird, and deer are 0.5, 0.2, 0.2, and 0.1, respectively. Therefore, the object is classified as dog, which has the highest predicted probability.

[0029] In this example, even if the predicted probability for cat is not the highest, if the predicted probability for cat is relatively high, it is assumed that the object is classified as a cat. In this case, the weighting coefficient for cat is set to be larger than the weighting coefficients for other classes. In the example shown in the lower left column of Figure 2, the weighting coefficients for dog, cat, bird, and deer are 0.1, 0.7, 0.1, and 0.1, respectively. Therefore, as shown in the center column of Figure 2, the predicted probabilities for dog, cat, bird, and deer are corrected to 0.05, 0.14, 0.02, and 0.01, respectively.

[0030] Then, the predicted probabilities for each of dog, cat, bird, and deer are updated so that the sum of the multiple predicted probabilities corresponding to dog, cat, bird, and deer is 1. Specifically, as shown in the right column of Figure 2, the predicted probabilities for dog, cat, bird, and deer are updated to 0.23, 0.64, 0.09, and 0.04, respectively. In this case, the predicted probability for cat is the highest. This allows the object to be classified as a cat.

[0031] In this way, the predicted probability x for class i after the update i x' is calculated by the following formula (1), where n is the number of classes (4 in this example). i is the predicted probability output for class i without setting the weighting coefficients. i is the weighting factor set for class i. The weighting factor ω set for class i i can be changed from the weight adjustment screen displayed on the display device 260 in Fig. 1. Details will be described later.

[0032]

number

[0033] 4.Confusion matrix The confusion matrix is ​​one of the output results of the learning model, and is generated using a portion of the learning data (hereinafter referred to as evaluation data) when the learning model is created. FIG. 3 is a diagram showing an example of a confusion matrix stored in the storage device 240. As shown in FIG. 3, the confusion matrix includes a plurality of squares arranged in the column direction (vertical direction) and row direction (horizontal direction). The components of the confusion matrix are written in each square.

[0034] The column components correspond to the label information assigned to the evaluation data. The row components correspond to the classification results performed by the learning model on the images indicated by the evaluation data (hereinafter referred to as evaluation images). Therefore, if the number of classes is n, the confusion matrix has n x n components. The value of the component written in each square indicates the number of evaluation images to which the label information of the corresponding column number belongs that are classified into the class of the corresponding row number.

[0035] In the example of Figure 3, 12 evaluation images are used to generate a confusion matrix. Of the 12 evaluation images, three evaluation images have the label information of dogs, three other evaluation images have the label information of cats, three other evaluation images have the label information of birds, and the remaining three evaluation images have the label information of deer.

[0036] As a classification result, for three evaluation images that belong to the label information of dog, all three evaluation images are classified as dog. For three evaluation images that belong to the label information of cat, all three evaluation images are classified as cat. For three evaluation images that belong to the label information of deer, all three evaluation images are classified as deer. On the other hand, for three evaluation images that belong to the label information of bird, one evaluation image is classified as dog and two evaluation images are classified as birds.

[0037] In other words, the confusion matrix in Figure 3 shows that the training data correctly classified 11 of the 12 evaluation images, and incorrectly classified one evaluation image. In this way, the confusion matrix visually indicates the superiority or inferiority of a training model. Essentially, a training model in which the values ​​of the diagonal elements arranged from the upper left to the lower right of the confusion matrix are not 0, and the values ​​of the other elements are 0, is considered to be able to perform classification tasks with high accuracy.

[0038] Note that the value of each component of the confusion matrix in Fig. 3 is the number (absolute number) of evaluation images to which the label information of the corresponding column number belongs that are classified into the class of the corresponding row number, but the embodiment is not limited to this. The value of each component of the confusion matrix may also be the proportion of evaluation images to which the label information of the corresponding column number belongs that are classified into the class of the corresponding row number, i.e., a relative value. For example, the value of each component of the confusion matrix may be a value obtained by dividing the above classified number by the sum of the components in the corresponding row direction or the sum of the components in the corresponding column direction.

[0039] 5.Display device Various screens such as a menu screen are displayed on the display device 260 of FIG. 1. A user can use the operation unit 250 of FIG. 1 to perform various operations on the screen displayed on the display device 260. FIG. 4 is a diagram showing an example of a menu screen. As shown in FIG. 4, the menu screen 10 is provided with an annotation area 11, a learning area 12, an evaluation area 13, and a setting area 14. In addition, a cue screen call button 15 is displayed on the menu screen 10. When the cue screen call button 15 is operated, a cue screen is displayed on the display device 260.

[0040] The annotation area 11 is operated when generating learning data (including evaluation data). The learning area 12 is operated when creating a learning model. The evaluation area 13 is operated when evaluating the learning model. The setting area 14 is operated when making predetermined settings for the learning model. Detailed explanations of the annotation area 11, learning area 12, and setting area 14 will be omitted.

[0041] A learning curve button 13a, a learning result button 13b, and a test evaluation button 13c are displayed in the evaluation area 13. When the learning curve button 13a is operated, a learning curve screen is displayed on the display device 260. When the learning result button 13b is operated, a learning result screen is displayed on the display device 260. When the test evaluation button 13c is operated, a test evaluation screen is displayed on the display device 260.

[0042] 5 is a diagram showing an example of a queue screen. As described above, the queue screen 20 is displayed on the display device 260 by operating the queue screen call button 15 on the menu screen 10 of FIG. 4. As shown in FIG. 5, the queue screen 20 is provided with a model display area 21 and a detailed information display area 22. In addition, a back button 23 is displayed on the queue screen 20. By operating the back button 23, the queue screen 20 displayed on the display device 260 returns to the menu screen 10.

[0043] A list of the most recently created learning models is displayed in the model display area 21. The user can select the desired learning model displayed in the model display area 21 by operating the operation unit 250. A screen showing detailed information associated with the learning model selected in the model display area 21 is displayed in the detailed information display area 22. The appearance of the screen displayed in the detailed information display area 22 is the same as the appearance of the detailed screen described below.

[0044] Fig. 6 is a diagram showing an example of a learning curve screen. As described above, the learning curve screen 30 is displayed on the display device 260 by operating the learning curve button 13a on the menu screen 10 of Fig. 4. As shown in Fig. 6, the learning curve screen 30 is provided with a model display area 31 and a learning curve display area 32.

[0045] The model display area 31 displays one or more learning models selected from a plurality of learning models stored in the storage device 240 or the like in Fig. 1. In addition, a detail screen call button 31a is displayed corresponding to each learning model. When any of the detail screen call buttons 31a is operated, a detail screen (described later) for the learning model corresponding to the operated detail screen call button 31a is displayed.

[0046] Furthermore, a model addition dialog call button 31b is displayed in the model display area 31. By operating the model addition dialog call button 31b, a model addition dialog screen (not shown) is displayed on the display device 260. On the model addition dialog screen, the user can select a desired learning model from among multiple learning models stored in the storage device 240, etc. The selected learning model is added to the model display area 31 together with a detail screen call button 31a corresponding to the learning model.

[0047] The user can select one or more learning models to be evaluated from the learning models displayed in the model display area 31. The learning curve display area 32 displays the learning curves for the one or more learning models selected as the evaluation targets. The learning curve is a curve with the learning period on the horizontal axis and the evaluation index on the vertical axis. The user can select either or both of the accuracy and loss as the evaluation index.

[0048] Fig. 7 is a diagram showing an example of a learning result screen. As described above, the learning result screen 40 is displayed on the display device 260 by operating the learning result button 13b on the menu screen 10 of Fig. 4. As shown in Fig. 7, the learning result screen 40 is provided with a model display area 41 and a learning result display area 42.

[0049] Similar to the model display area 31 in FIG. 6, the model display area 41 displays one or more learning models selected from a plurality of learning models stored in the storage device 240 or the like in FIG. 1. Also, a detail screen call button 41a is displayed corresponding to each learning model. Furthermore, the model display area 41 displays a model addition dialog call button 41b. The uses of the detail screen call button 41a and the model addition dialog call button 41b are similar to the uses of the detail screen call button 31a and the model addition dialog call button 31b in FIG. 6, respectively.

[0050] The user can select a desired learning model from the learning models displayed in the model display area 41. A confusion matrix associated with the selected learning model is displayed in the learning result display area 42. Each square of the confusion matrix is ​​displayed decorated with a style (in this example, shades of color) assigned to the range of component size described in that square. The user can also select a desired square of the confusion matrix. When any square is selected, a thumbnail image of the evaluation image corresponding to that square is displayed in the learning result display area 42. A weight adjustment button 42a is also displayed in the learning result display area 42. When the weight adjustment button 42a is operated, a weight adjustment screen, which will be described later, is displayed.

[0051] 6.Details screen As described above, the details screen is displayed on the display device 260 by operating the details screen call button 31a in FIG. 6, the details screen call button 41a in FIG. 7, or the details screen call button 511 in FIG. 14 described later. Note that a button for displaying the details screen is provided on all screens that are displayed by operating the evaluation area 13 in FIG. 4. FIG. 8 is a diagram showing an example of the details screen. As shown in FIG. 8, the details screen 300 is provided with a plurality of information selection fields 310. Each information selection field 310 is associated with one of the detailed information items. Furthermore, an expand button 311 is displayed in each information selection field 310.

[0052] In the example of FIG. 8, four information selection columns 310 are arranged in a vertical line. Data information is associated with the top information selection column 310. Padded information is associated with the second information selection column 310 from the top. Learning network information is associated with the third information selection column 310 from the top. Hyperparameter information is associated with the bottom information selection column 310. When the expansion button 311 of any information selection column 310 is operated, the display of detailed information associated with that information selection column 310 is expanded downward.

[0053] 9 to 12 are diagrams showing examples of expanded detail screens. As shown in FIGS. 9 to 12, when the expand button 311 of any of the information selection fields 310 is operated, the display of the expand button 311 of that information selection field 310 changes to a collapse button 312. In addition, one or more detailed information display fields 320 are displayed below that information selection field 310. In each detailed information display field 320, detailed information associated with the information selection field 310 is displayed. When the collapse button 312 is operated, the detailed information display field 320 is collapsed, and the detail screen 300 returns to the state shown in FIG. 8.

[0054] In the examples of FIGS. 9 and 10, the expand button 311 of the top information selection column 310 is operated. In this case, data information is displayed in the detailed information display column 320. In the example of FIG. 11, the expand button 311 of the second information selection column 310 from the top is operated. In this case, padded information is displayed in the detailed information display column 320. In the example of FIG. 12, the expand button 311 of the third information selection column 310 from the top and the expand button 311 of the bottom information selection column 310 are operated. In this case, learning network information is displayed in the detailed information display column 320 below the third information selection column 310 from the top. In addition, hyperparameter information is displayed in the detailed information display column 320 below the bottom information selection column 310.

[0055] 9 or 11, when further information can be displayed in the detailed information display field 320, a details button 321 is displayed in the detailed information display field 320. When the details button 321 is operated, the display of the details button 321 changes to a simple button 322, as shown in FIG. 10 or 11. Furthermore, the display of the further information is added to the detailed information display field 320. When the simple button 322 is operated, the display of the information added to the detailed information display field 320 is collapsed, and the display of the simple button 322 returns to the details button 321.

[0056] 7.Weight adjustment screen As described above, when weight adjustment button 42a in Fig. 7 is operated, a weight adjustment screen is displayed on display device 260. Fig. 13 is a diagram showing an example of the weight adjustment screen. As shown in Fig. 13, weight adjustment screen 400 is provided with a weight coefficient adjustment slider 410, a weight coefficient adjustment table 420, and a classification result display area 430. In addition, weight adjustment screen 400 displays an OK button 440, a close button 450, and a reset button 460.

[0057] The weight coefficient adjustment slider 410 is a first GUI that accepts adjustment of the weight coefficient, and includes a bar 411 and multiple sliders 412. The bar 411 extends in one direction (the left-right direction on the paper in the example of FIG. 13). The multiple sliders 412 are arranged on the bar 411. Specifically, the number of sliders 412 is (n-1). Here, n is the number of classes, which is 3 or more. In this example, the evaluation images are classified into one of airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Therefore, n is 10, and the number of weight coefficient adjustment sliders 410 is 9.

[0058] The bar 411 is divided into n regions by (n-1) sliders 412. The n regions of the bar 411 correspond to the n classes, respectively. The length of each region of the bar 411 in one direction indicates the weighting coefficient of the corresponding class. Each slider 412 is movable in two directions parallel to one direction (i.e., leftward and rightward on the paper in the example of FIG. 13). Therefore, the user can adjust the weighting coefficient of a desired class by moving a desired slider 412 in a direction parallel to one direction using the operation unit 250 of FIG. 1. When any slider 412 is moved, the two weighting coefficients set for the two classes corresponding to the two regions of the bar divided by that slider 412 are adjusted according to the amount of movement of that slider 412.

[0059] Weight coefficient adjustment table 420 is a second GUI that accepts adjustment of weight coefficients. Weight coefficient adjustment table 420 lists multiple weight coefficients corresponding to multiple classes. A user can use operation unit 250 to rewrite the weight coefficient of a desired class listed in weight coefficient adjustment table 420. When the value of the weight coefficient of any class in weight coefficient adjustment table 420 is rewritten, the weight coefficient set for that class is changed to the rewritten value. Furthermore, based on the rewritten weight coefficient value, the weight coefficients set for all other classes are changed evenly and reflected in weight coefficient adjustment table 420.

[0060] That is, in the weight coefficient adjustment table 420, the value of the weight coefficient of class t is ω t ', the weight coefficient for class t is ω t On the other hand, the weighting coefficient of class i other than class t is calculated by the following formula (2): i ', where ω i is the weighting factor set for class i before the weighting factor is adjusted. The weighting factor for class i written in weighting factor adjustment table 420 is rewritten to the value calculated by equation (2).

[0061]

number

[0062] When an operation is performed on either the weight coefficient adjustment slider 410 or the weight coefficient adjustment table 420, the operation is reflected in the other. Therefore, when any slider 412 in the weight coefficient adjustment slider 410 is moved, at least two weight coefficients in the weight coefficient adjustment table 420 are rewritten to reflect the movement. Similarly, when any weight coefficient is rewritten in the weight coefficient adjustment table 420, each slider 412 in the weight coefficient adjustment slider 410 moves to reflect the rewriting.

[0063] When the weight coefficient adjustment slider 410 or the weight coefficient adjustment table 420 is operated to adjust the weight coefficient, the predicted probability for each class is updated using the adjusted weight coefficient and equation (1). That is, the predicted probability that the image to be classified belongs to each class is calculated based on the changed weight coefficient. Furthermore, the predicted probability for each class is updated so that the sum of the predicted probabilities calculated for all classes becomes 1.

[0064] The classification result display area 430 displays the predicted probability for each class of any evaluation image according to the learning model. When the weight coefficient adjustment slider 410 or the weight coefficient adjustment table 420 is operated, the predicted probability for each class of the evaluation image displayed in the classification result display area 430 changes in real time to reflect the operation. Therefore, the user can adjust the weight coefficient by operating the weight coefficient adjustment slider 410 or the weight coefficient adjustment table 420 while visually checking the predicted probability displayed in the classification result display area 430, so as to obtain the desired output result.

[0065] The adjusted weight coefficients are set by operating the OK button 440. The weight adjustment screen 400 is closed by operating the close button 450. The weight coefficients for each class are returned to their initial values ​​by operating the reset button 460. In this case, the display of the weight coefficient adjustment slider 410 and the weight coefficient adjustment table 420 also returns to their initial states.

[0066] 8. Test evaluation screen When the test evaluation button 13c on the menu screen 10 of FIG. 4 is operated, a model / data selection screen (not shown) is displayed on the display device 260. By performing a predetermined operation on the model / data selection screen, the user can select one or more learning models to be evaluated from a plurality of learning models stored in the storage device 240 or the like. In addition, by performing a predetermined operation on the model / data selection screen, the user can select one or more groups of evaluation data to be used for evaluating the learning model from the evaluation data stored in the storage device 240 or the like. When a learning model and evaluation data are selected on the model / data selection screen, a test evaluation screen is displayed on the display device 260.

[0067] Fig. 14 is a diagram showing an example of a test evaluation screen. As shown in Fig. 14, a test evaluation screen 500 is provided with a model data display area 510 and a test evaluation display area 520. Furthermore, a display method pull-down menu 530 is displayed in the test evaluation display area 520.

[0068] The model data display area 510 displays one or more learning models selected on the model data selection screen, as well as one or more groups of selected evaluation data. In addition, a detail screen call button 511 is displayed corresponding to each learning model. When any of the detail screen call buttons 511 is operated, the detail screen 300 of FIG. 8 is displayed for the learning model corresponding to the operated detail screen call button 511. Note that by performing a predetermined operation in the model data display area 510, it is possible to add or delete the displayed learning models or evaluation data.

[0069] The user can select one or more desired learning models from the learning models displayed in the model data display area 510. The user can also select one desired group of evaluation data from the evaluation data displayed in the model data display area 510. When one or more learning models and one group of evaluation data are selected in the model data display area 510, a screen showing a confusion matrix when the selected evaluation data (evaluation images) are classified using the selected learning models is displayed in the test evaluation display area 520.

[0070] As in Fig. 7, in the confusion matrix displayed in the test evaluation display area 520, each square is decorated with a pattern (in this example, shades of color) assigned to the range of component size described in that square. Here, in the example of Fig. 14, one learning model is selected in the model data display area 510. Therefore, a confusion matrix screen 521 showing the confusion matrix of that one learning model is displayed in the test evaluation display area 520.

[0071] On the other hand, when two or more learning models are selected in the model data display area 510, the user can operate the display method pull-down menu 530 to display an array screen or a superimposed screen, which will be described later, in the test evaluation display area 520 instead of the confusion matrix screen 521. Therefore, by operating the display method pull-down menu 530, the user can switch between the array screen and the superimposed screen and display them in the test evaluation display area 520.

[0072] Fig. 15 is a diagram showing an example of an array screen. As shown in Fig. 15, two or more confusion matrices, each representing two or more learning models selected in the model data display area 510, are displayed side by side on the array screen 522. By visually checking the two or more confusion matrices displayed on the array screen 522, the user can easily evaluate the relative merits of the two or more learning models.

[0073] 16 is a diagram showing an example of a superimposed screen. As shown in FIG. 16, one superimposed confusion matrix is ​​displayed on the superimposed screen 523. When the superimposed screen 523 is displayed in the test evaluation display area 520, a calculated value pull-down menu 540 is also displayed in the test evaluation display area 520. The superimposed confusion matrix is ​​a confusion matrix in which two or more confusion matrices representing two or more learning models selected in the model data display area 510 are superimposed. Specifically, each element of the superimposed confusion matrix is ​​the calculated value of the corresponding element of the two or more confusion matrices.

[0074] The user can select a calculated value for calculating each component of the superimposed confusion matrix by operating the calculated value pull-down menu 540. The calculated value includes various calculated values ​​such as the mean value or the variance value. Furthermore, when the number of selected learning models is two, the calculated value further includes the difference. In the example of FIG. 16, the difference is selected as the calculated value. By visually checking the confusion matrix displayed on the superimposed screen 523, the user can easily evaluate the superiority or inferiority of two or more learning models. In particular, when the difference is selected as the calculated value, the superiority or inferiority of two learning models can be more easily evaluated.

[0075] 9. Learning Model Evaluation Support Device Fig. 17 is a block diagram showing the functional configuration of the learning model evaluation support device 100 in Fig. 1. As shown in Fig. 17, the learning model evaluation support device 100 includes, as functional units, a model information display unit 110, a model selection unit 120, a probability calculation unit 130, a confusion matrix calculation unit 140, and an output display unit 150. The functional units of the learning model evaluation support device 100 are realized by the CPU 210 in Fig. 1 executing a learning model evaluation support program. Some or all of the functional units of the learning model evaluation support device 100 may be realized by hardware such as electronic circuits.

[0076] The model information display unit 110 displays a plurality of learning models stored in the storage device 240 or the like on the display device 260 in a selectable manner. In this example, a model data selection screen is displayed on the display device 260 by operating the test evaluation button 13c on the menu screen 10 in Fig. 4. Alternatively, a model addition dialog screen is displayed on the display device 260 by operating the model addition dialog call button 31b on the learning curve screen 30 in Fig. 6 or the model addition dialog call button 41b on the learning result screen 40 in Fig. 7. A plurality of learning models are displayed in a selectable manner on these screens.

[0077] The model selection unit 120 selects at least one learning model from the multiple learning models on the screen displayed on the display device 260 according to the user's designation. The user can designate a desired learning model by operating the operation unit 250 while visually checking the multiple learning models displayed on the screen. The learning model selected by the model selection unit 120 is displayed in the model display area 31 of the learning curve screen 30 in FIG. 6, the model display area 41 in FIG. 7, or the model data display area 510 in FIG. 14, etc.

[0078] The model information display unit 110 can also selectably display the evaluation data stored in the storage device 240 or the like on the display device 260. In this example, the evaluation data is selectably displayed on a model data selection screen. The model selection unit 120 selects one or more groups of evaluation data from the evaluation data on the model data selection screen displayed on the display device 260 in accordance with the user's designation. The user can specify one or more desired groups of evaluation data by operating the operation unit 250 while visually checking the evaluation data displayed on the model data selection screen. The evaluation data selected by the model selection unit 120 is displayed in the model data display area.

[0079] Furthermore, the model selection unit 120 selects a learning model to be subjected to weighting coefficient adjustment, a learning model to be subjected to detailed information display, or a learning model to be subjected to evaluation, in accordance with a user's specification. The user can specify a desired learning model from among the learning models displayed in the model display area 31, the model display area 41, the model data display area 510, or the like. Furthermore, when evaluating a learning model, the model selection unit 120 selects one group of evaluation data to be used for evaluating the learning model, in accordance with a user's specification. The user can specify a desired group of learning models from one or more groups of evaluation data displayed in the model data display area 510.

[0080] The probability calculation unit 130 changes the weighting coefficient set for one of the classes for the learning model selected by the model selection unit 120 in accordance with a user specification. The user can specify the class for which the weighting coefficient is to be changed and the value of the weighting coefficient by operating the operation unit 250 to move one of the sliders 412 of the weight coefficient adjustment sliders 410 displayed on the weight adjustment screen 400 of Fig. 13. When one of the sliders 412 is moved in one direction, the weighting coefficients set for the two classes corresponding to the two regions of the bar separated by that slider 412 are changed according to the amount of movement of that slider 412.

[0081] Alternatively, the user can specify the class for which the weight coefficient is to be changed and the value of the weight coefficient by operating the operation unit 250 to rewrite the weight coefficient of one of the classes in the weight coefficient adjustment table 420 displayed on the weight adjustment screen 400. When the value of the weight coefficient of one of the classes is rewritten, the weight coefficient set for that class is changed to the rewritten value. Furthermore, based on the rewritten weight coefficient value, the weight coefficients set for all other classes are changed to the value given by equation (2). The probability calculation unit 130 updates the predicted probability based on the changed weight coefficient and equation (1).

[0082] 16 is specified by the user, the confusion matrix calculation unit 140 calculates a superimposed confusion matrix to be displayed on the superimposed screen 523. The user can specify that the superimposed screen 523 be displayed by specifying to the model selection unit 120 that two or more learning models be selected, and then operating the display method pull-down menu 530 of the superimposed screen 523 using the operation unit 250.

[0083] The user can also specify a calculation value by operating calculation value pull-down menu 540 on superimposed screen 523 using operation unit 250. A superimposed confusion matrix is ​​calculated by calculating each component of the confusion matrix corresponding to each of two or more learning models selected by model selection unit 120. When the number of learning models selected by model selection unit 120 is two, the calculation value includes a difference.

[0084] The output display unit 150 displays the output result of at least one learning model selected by the model selection unit 120 on the display device 260. In this example, as described above, the menu screen 10 of FIG. 10 is displayed on the display device 260. The user operates the menu screen 10 and subsequent screens using the operation unit 250. As a result, the queue screen 20 of FIG. 5, the learning curve screen 30 of FIG. 6, the learning result screen 40 of FIG. 7, the test evaluation screen 500 of FIG. 14, or the like is displayed on the display device 260.

[0085] As described above, the learning model selected by the model selection unit 120 is displayed in the model display area 31 of the learning curve screen 30, the model display area 41 of the learning result screen 40, or the model data display area 510 of the test evaluation screen 500. Furthermore, the model data display area 510 of the test evaluation screen 500 displays the evaluation data selected by the model selection unit 120.

[0086] 13 is displayed as an output result of at least one learning model. The weight coefficient adjustment slider 410 and weight coefficient adjustment table 420 on the weight adjustment screen 400 display the weight coefficient changed for any class of the learning model by the probability calculation unit 130. The classification result display area 430 on the weight adjustment screen 400 displays the classification result updated by the probability calculation unit 130.

[0087] Furthermore, on the test evaluation screen 500, an arrangement screen 522 and a superimposed screen 523 are switchably displayed in the test evaluation display area 520. The arrangement screen 522 displays two or more confusion matrices respectively associated with two or more learning models selected by the model selection unit 120. The superimposed screen 523 displays the superimposed confusion matrix calculated by the confusion matrix calculation unit 140.

[0088] Furthermore, all screens displayed by operating the evaluation area 13 on the menu screen 10 as the output result of at least one learning model are provided with a detail screen call button. For example, the learning curve screen 30 is provided with a detail screen call button 31a. The learning result screen 40 is provided with a detail screen call button 41a. The test evaluation screen 500 is provided with a detail screen call button 511. When these detail screen call buttons 31a, 41a, and 511 are operated, the detail screen 300 of FIG. 8 is displayed on the display device 260 for the learning model selected by the model selection unit 120.

[0089] When any of the information items is selected on the details screen 300, the display of the information for the selected item is expanded. The user can select a desired item on the details screen 300 using the operation unit 250. This allows the user to easily check the details of the desired item that was used when creating the learning model.

[0090] 10. Learning model evaluation support processing The learning model evaluation support process is executed by CPU 210 in Figure 1 as the CPU executes the learning model evaluation support program, and includes a detailed screen display process, a probability calculation process, a weight adjustment process, and a test evaluation display process. Each of the detailed screen display process, the probability calculation process, the weight coefficient adjustment process, and the test evaluation display process will be described below.

[0091] Fig. 18 is a flowchart showing the detail screen display process. As shown in Fig. 18, in the detail screen display process, the output display unit 150 determines whether or not the detail screen call button has been operated (step S1). As described above, the detail screen call button is provided on all screens that are displayed by operating the evaluation area 13 in Fig. 4. Furthermore, operating the detail screen call button includes right-clicking the detail screen call button with a mouse.

[0092] If the detail screen call button is not operated, the output display unit 150 waits until the detail screen call button is operated. If the detail screen call button is operated, the output display unit 150 causes the display device 260 to display the detail screen 300 of Fig. 8 (step S2). This ends the detail screen display process.

[0093] 19 is a flowchart showing the probability calculation process. As shown in FIG. 19, in the probability calculation process, the probability calculation unit 130 obtains the predicted probability of each class, which is the output result of the selected learning model (step S11). Next, the probability calculation unit 130 multiplies the predicted probability of the class obtained in step S11 by the weighting coefficient set for each class (step S12). Here, if the weighting coefficient of any class is changed by the weight adjustment process described later, the class is multiplied by the changed weighting coefficient.

[0094] Next, the probability calculation unit 130 normalizes the predicted probability of each class so that the sum of the predicted probabilities of all classes calculated in step S12 becomes 1 (step S13). Subsequently, the probability calculation unit 130 outputs the classification result by classifying the object into the class with the maximum predicted probability among the predicted probabilities calculated in step S13 (step S14). Thereafter, the probability calculation unit 130 returns to step S12, and steps S12 to S14 are repeated.

[0095] 20 is a flowchart showing the weight adjustment process. As shown in FIG. 20, in the weight adjustment process, the output display unit 150 displays the weight adjustment screen 400 on the display device 260 (step S21). Next, the probability calculation unit 130 determines whether the weight coefficient adjustment slider 410 has been operated (step S22). If the weight coefficient adjustment slider 410 has not been operated, the probability calculation unit 130 proceeds to step S25.

[0096] When the weight coefficient adjustment slider 410 is operated, the probability calculation unit 130 changes the weight coefficient of one of the classes based on the amount of movement of one of the sliders 412 of the weight coefficient adjustment slider 410 (step S23). In this case, the predicted probability of one of the classes is updated in the probability calculation process. Subsequently, the output display unit 150 updates the weight adjustment screen 400 (step S24). As a result, the display of the weight coefficient adjustment table 420 and the classification result display area 430 is updated.

[0097] If the weight coefficient adjustment slider 410 is not operated in step S22, the probability calculation unit 130 determines whether the weight coefficient adjustment table 420 has been operated (step S25). If the weight coefficient adjustment table 420 has not been operated, the probability calculation unit 130 returns to step S22.

[0098] When the weight coefficient adjustment table 420 is operated, the probability calculation unit 130 changes the weight coefficient of each class based on the amount of change in the value of the weight coefficient rewritten in the weight coefficient adjustment table 420 (step S26). In this case, the predicted probability of each class is updated in the probability calculation process. Subsequently, the output display unit 150 updates the weight adjustment screen 400 (step S27). As a result, the display of the weight coefficient adjustment slider 410 and the classification result display area 430 is updated.

[0099] Thereafter, the probability calculation unit 130 determines whether or not to determine a weighting coefficient (step S28). Specifically, if the OK button 440 on the weight adjustment screen 400 is operated, it is determined that the weighting coefficient is to be determined. If the weighting coefficient is not to be determined, the probability calculation unit 130 returns to step S22. Steps S22 to S28 are repeated until the weighting coefficient is determined. If the weighting coefficient is determined, the weight adjustment process ends.

[0100] 21 is a flowchart showing the test evaluation display process. As shown in FIG. 21, in the test evaluation display process, the output display unit 150 displays the test evaluation screen 500 on the display device 260 (step S31). Next, the confusion matrix calculation unit 140 determines whether or not a learning model has been selected by the model selection unit 120 (step S32). If a learning model has not been selected, the confusion matrix calculation unit 140 waits until a learning model is selected.

[0101] If a learning model is selected, the confusion matrix calculation unit 140 determines whether the number of selected learning models is two (step S33). If the number of learning models is two, the confusion matrix calculation unit 140 determines whether "Difference" is selected from the calculation value pull-down menu 540 (step S34). If "Difference" is selected, the confusion matrix calculation unit 140 calculates a superimposed confusion matrix using the difference between the two confusion matrices corresponding to the two selected learning models (step S35), and proceeds to step S41.

[0102] If the number of selected learning models is not two in step S33, the confusion matrix calculation unit 140 determines whether the number of selected learning models is three or more (step S36). If a difference is not selected in step S34, or if the number of selected learning models is three or more in step S36, the confusion matrix calculation unit 140 determines whether the average value is selected from the calculation value pull-down menu 540 (step S37). If the average value is selected, the confusion matrix calculation unit 140 calculates a superimposed confusion matrix using the average values ​​of two or more confusion matrices corresponding to the two or more selected learning models, respectively (step S38), and proceeds to step S41.

[0103] If the mean value is not selected in step S37, the confusion matrix calculation unit 140 determines whether or not a variance value is selected from the calculation value pull-down menu 540 (step S39). If a variance value is selected, the confusion matrix calculation unit 140 calculates a superimposed confusion matrix using the variance values ​​of two or more confusion matrices corresponding to the two or more selected learning models (step S40), and then proceeds to step S41.

[0104] In step S41, the output display unit 150 displays the superimposed screen 523 showing the superimposed confusion matrix calculated in step S35, step S38, or step S40 in the test evaluation display area 520 (step S41). After that, the output display unit 150 returns to step S32.

[0105] On the other hand, if the number of learning models selected in step S36 is less than three or if no variance value is selected in step S39, the output display unit 150 displays an array screen 522 showing a confusion matrix in which two or more confusion matrices corresponding to the two or more selected learning models are arranged in the test evaluation display area 520 (step S42). Thereafter, the output display unit 150 returns to step S32. This allows the array screen 522 and the superimposed screen 523 to be switched between display.

[0106] 11.Effect In the learning model evaluation support device 100 according to this embodiment, the output results of at least one learning model selected from a plurality of learning models with the same interface are displayed on the display device 260. Therefore, the user can easily recognize the characteristics of the learning model by visually checking the output results displayed on the display device 260. This makes it easy to evaluate the characteristics of the learning model.

[0107] Two or more learning models are selected from the plurality of learning models by the model information display unit 110. An arrangement screen 522 that displays two or more confusion matrices that respectively indicate the output results of two or more learning models side by side, and a superimposition screen 523 that displays a superimposed confusion matrix based on the output results of two or more learning models are switchably displayed on the display device 260 by the output display unit 150. In this case, the merits or demerits of two or more learning models can be easily evaluated.

[0108] Each component of the superimposed confusion matrix displayed on the superimposed screen 523 is the calculated value of the corresponding component of two or more confusion matrices. In this case, the confusion matrix displayed on the superimposed screen 523 can be easily calculated. When two learning models are selected from multiple learning models by the model information display unit 110, each component of the superimposed confusion matrix displayed on the superimposed screen 523 may be the difference between the corresponding components of the two confusion matrices corresponding to the two learning models, respectively. In this case, the superiority or inferiority of the two learning models can be more easily evaluated.

[0109] On the array screen 522 or the superimposed screen 523, each element of the confusion matrix is ​​displayed decorated with a style assigned by the output display unit 150 to the range of the size of the element. In this case, the superiority or inferiority of the learning model can be evaluated more intuitively. In this example, the style is a shade of color. In this case, each element of the confusion matrix can be easily decorated according to its size.

[0110] Each learning model performs a classification task of classifying an image to be classified into one of multiple classes. A weight adjustment screen 400 may be displayed on the display device 260 by the output display unit 150, which accepts adjustment of weighting factors set for multiple classes in order to adjust the output result of at least one learning model. In this case, the sensitivity for identifying a specific class in the classification task can be increased.

[0111] Specifically, the probability calculation unit 130 calculates the predicted probability that the image to be classified belongs to each class based on the weight coefficient set for each class. Here, the weight adjustment screen 400 has a weight coefficient adjustment slider 410 and a weight coefficient adjustment table 420. In the weight coefficient adjustment slider 410, a plurality of sliders 412 are arranged along a bar 411 extending in one direction. Each slider 412 is movable in a direction parallel to the one direction. The bar 411 is divided by the plurality of sliders into a plurality of regions corresponding to a plurality of classes, respectively.

[0112] When any of the weight coefficient adjustment sliders 410 is moved in a direction parallel to one direction, the weight coefficients set for the two classes corresponding to the two regions of the bar 411 partitioned by that slider are changed by the probability calculation unit 130 in accordance with the amount of movement of that slider. In this case, the weight coefficient set for any class can be easily changed without changing the weight coefficients set for some of the classes.

[0113] Furthermore, weight coefficient adjustment table 420 lists a plurality of weight coefficients corresponding to a plurality of classes, respectively. When the value of the weight coefficient for any of the listed classes is rewritten in weight coefficient adjustment table 420, probability calculation unit 130 changes the weight coefficient set for that class to the value rewritten in weight coefficient adjustment table 420. Furthermore, based on the rewritten weight coefficient value, probability calculation unit 130 changes the weight coefficients set for all other classes. In this case, it is possible to easily change the weight coefficient set for any class while maintaining the relative relationships between the weight coefficients set for the other classes.

[0114] The probability calculation unit 130 calculates the predicted probability that the image to be classified belongs to each class based on the changed weighting coefficients. The probability calculation unit 130 also updates the predicted probability for each class so that the sum of the predicted probabilities calculated for all classes becomes a constant value. In this case, the predicted probability for each class can be easily updated based on the changed weighting coefficients.

[0115] In this example, the number of classes is 3 or more. In this case, the user can increase the sensitivity for identifying a specific class in a classification task of 3 or more classes in a similar manner to adjusting the threshold in a classification task of 2 classes.

[0116] Furthermore, as an output result of at least one learning model, the output display unit 150 displays on the display device 260 a screen including a detail screen call button for displaying a detail screen 300 containing detailed information used when creating the learning model. In this case, the detail screen 300 is displayed by operating the detail screen call button. Therefore, by visually checking the detail screen 300, the user can easily recognize the detailed information used when creating the learning model. This makes it easy to evaluate the characteristics of the learning model.

[0117] The detailed information includes multiple items of information. When one of the multiple items of information is selected on the details screen 300, the display of the information for the selected item is expanded by the output display unit 150. In this case, it is possible to expand and display information for any item of the detailed information. This improves the readability of the information for the required item.

[0118] Any of the multiple items of information includes information about the training data, information about a method for augmenting the image data to be used as training data, information about the training network, or information about hyperparameters. In this case, the user can easily identify desired information from the information about the training data, information about augmenting the image data to be used as training data, information about the training network, or information about hyperparameters.

[0119] 12. Other Embodiments (1) In the above embodiment, various screens including the details screen 300, the weight adjustment screen 400, and the test evaluation screen 500 are displayed on the display device 260, but the embodiment is not limited to this. Any one of the details screen 300, the weight adjustment screen 400, and the test evaluation screen 500 may be displayed on the display device 260, and the other screens may not be displayed on the display device 260.

[0120] Furthermore, if the details screen 300 is not displayed, the details screen call button may not be displayed on any screen. If the weight adjustment screen 400 is not displayed, the probability calculation unit 130 may not change the weight coefficient set for each class. If the test evaluation screen 500 is not displayed, the learning model evaluation support device 100 may not include the output display unit 150.

[0121] (2) In the above embodiment, each element of the confusion matrix is ​​displayed decorated with shades of colors assigned to a range of sizes, but the embodiment is not limited to this. Each element of the confusion matrix may be displayed decorated in other ways. Alternatively, each element of the confusion matrix may be displayed without decoration.

[0122] (3) In the above embodiment, the weight adjustment screen 400 includes both the weight coefficient adjustment slider 410 and the weight coefficient adjustment table 420, but the embodiment is not limited to this. The weight adjustment screen 400 may include either the weight coefficient adjustment slider 410 or the weight coefficient adjustment table 420, but not the other.

[0123] (4) In the above embodiment, the number of classes is three or more, but the embodiment is not limited to this. The number of classes may be two.

[0124] 13. Summary of the embodiment (Item 1) A learning model evaluation support device according to one aspect of the present invention comprises: a model information display unit that displays a plurality of learning models stored in a predetermined storage area and having the same interface on a display device in a selectable manner; a model selection unit that selects at least one learning model from the plurality of learning models displayed on the display device; and an output display unit that displays an output result from the at least one learning model on the display device.

[0125] According to this learning model evaluation support device, the output results of at least one learning model selected from multiple learning models with the same interface are displayed on a display device. Therefore, a user can easily recognize the characteristics of the learning model by visually checking the output results displayed on the display device. This makes it easy to evaluate the characteristics of the learning model.

[0126] (2) In the learning model evaluation support device described in (1), the model selection unit selects two or more learning models from the plurality of learning models; The output display unit may switchably display on the display device an arrangement screen that displays two or more confusion matrices that respectively indicate the output results of the two or more learning models, and an overlapping screen that displays a confusion matrix based on the output results of the two or more learning models.

[0127] In this case, the relative merits of two or more learning models can be easily evaluated.

[0128] (Item 3) In the learning model evaluation support device described in item 2, Each element of the confusion matrix displayed on the superimposed screen may be a calculated value of the corresponding element of the two or more confusion matrices.

[0129] In this case, the confusion matrix to be displayed on the superimposed screen can be easily calculated.

[0130] (4) In the learning model evaluation support device described in (3), the model selection unit selects two learning models from the plurality of learning models; Each element of the confusion matrix displayed on the superimposed screen may be a difference between the corresponding elements of two confusion matrices corresponding to the two learning models, respectively.

[0131] In this case, it is easier to evaluate the relative merits of the two learning models.

[0132] (Item 5) In the learning model evaluation support device according to any one of items 2 to 4, The output display unit may display, on the array screen or the superimposed screen, each element of the confusion matrix decorated in a manner assigned to a size range.

[0133] In this case, the superiority or inferiority of the learning model can be evaluated more intuitively.

[0134] (Item 6) In the learning model evaluation support device described in item 5, The aspect may be color.

[0135] In this case, each element of the confusion matrix can be easily decorated according to its size.

[0136] (7) In the learning model evaluation support device according to any one of paragraphs 1 to 6, Each learning model performs a classification task to classify an image to be classified into one of multiple classes. The output display unit may display a weight adjustment screen on the display device that accepts adjustment of weight coefficients set for the plurality of classes in order to adjust the output result by the at least one learning model.

[0137] In this case, the sensitivity for identifying a particular class in a classification task can be increased.

[0138] (8) The learning model evaluation support device according to the seventh aspect of the present invention a probability calculation unit that calculates a predicted probability that an image to be classified corresponds to each class based on a weighting coefficient set for each class; the weight adjustment screen includes a first GUI (Graphical User Interface) having a bar extending in one direction and a plurality of sliders arranged along the bar and movable in a direction parallel to the one direction, the bar being partitioned by the plurality of sliders into a plurality of regions corresponding to the plurality of classes, When any of the sliders in the first GUI moves in a direction parallel to the one direction, the probability calculation unit may change the weighting coefficients set for two classes corresponding to two areas of the bar partitioned by the slider in accordance with the amount of movement of the slider.

[0139] In this case, the weighting coefficients set for any class can be easily changed without changing the weighting coefficients set for some classes.

[0140] (Item 9) The learning model evaluation support device according to item 7 or 8, a probability calculation unit that calculates a predicted probability that an image to be classified corresponds to each class based on a weighting coefficient set for each class; the weight adjustment screen includes a second GUI on which a plurality of weight coefficients corresponding to the plurality of classes are displayed, When the value of the weighting coefficient for any of the classes listed in the second GUI is rewritten, the probability calculation unit may change the weighting coefficient set for that class to the value rewritten in the second GUI, and may also change the weighting coefficients set for all other classes based on the rewritten weighting coefficient value.

[0141] In this case, the weighting factor set for any class can be easily changed while maintaining the relative relationship of the weighting factors set for other classes.

[0142] (10) In the learning model evaluation support device according to claim 8 or 9, The probability calculation unit may calculate a predicted probability that the image to be classified corresponds to each class based on the changed weighting coefficient, and may update the predicted probability for each class so that the sum of the predicted probabilities calculated for all classes becomes a constant value.

[0143] In this case, the predicted probabilities for each class can be easily updated based on the modified weighting factors.

[0144] (Item 11) In the learning model evaluation support device according to any one of items 7 to 10, The number of the plurality of classes may be three or more.

[0145] In this case, the user can increase the sensitivity for identifying a particular class in a classification task of three or more classes in a similar manner to adjusting the threshold in a classification task of two classes.

[0146] (12) In the learning model evaluation support device according to any one of paragraphs 1 to 11, The output display unit may display on the display device a screen including a detail screen call button for displaying a detail screen containing detailed information used when creating the learning model as the output result of the at least one learning model.

[0147] In this case, the details screen is displayed by operating the details screen call button. Therefore, by visually checking the details screen, the user can easily recognize the detailed information used when creating the learning model. This makes it easy to evaluate the characteristics of the learning model.

[0148] (13) In the learning model evaluation support device according to the 12th paragraph, The detailed information includes a plurality of items of information, When any one of the plurality of items of information is selected on the details screen, the output display unit may expand a display of the information of the selected item.

[0149] In this case, it is possible to expand and display any desired item of detailed information, thereby improving the readability of the required information.

[0150] (14) In the learning model evaluation support device according to the 13th paragraph, Any of the multiple items of information may include information about training data, information about a method for augmenting image data that will become the training data, information about a training network, or information about hyperparameters.

[0151] In this case, the user can easily recognize the desired information, among information about the training data, information about the method of padding the image data that will be used as training data, information about the training network, or information about hyperparameters.

[0152] (15) A learning model evaluation support program according to another aspect of the present invention comprises: a model information display process for displaying, on a display device, a plurality of learning models stored in a predetermined storage area and having the same interface in a selectable manner; a model selection process for selecting at least one learning model from the plurality of learning models displayed on the display device; and an output display process for displaying an output result of the at least one learning model on the display device.

[0153] According to this learning model evaluation support program, the output results of at least one learning model selected from multiple learning models with the same interface are displayed on a display device. Therefore, a user can easily recognize the characteristics of the learning model by visually checking the output results displayed on the display device. This makes it easy to evaluate the characteristics of the learning model.

[0154] 14. Correspondence between each element of the claims and each part of the embodiment Below, examples of correspondence between each element of the claims and each element of the embodiments will be described, but the present invention is not limited to the following examples. Various other elements having the configuration or function described in the claims can also be used as each element of the claims.

[0155] In the above-described embodiment, the storage device 240 is an example of a storage area, the display device 260 is an example of a display device, the model information display unit 110 is an example of a model information display unit, and the model selection unit 120 is an example of a model selection unit. The output display unit 150 is an example of an output display unit, the learning model evaluation support device 100 is an example of a learning model evaluation support device, the array screen 522 is an example of an array screen, and the superimposed screen 523 is an example of a superimposed screen.

[0156] Weight adjustment screen 400 is an example of a weight adjustment screen, probability calculation unit 130 is an example of a probability calculation unit, bar 411 is an example of a bar, slider 412 is an example of a slider, weight coefficient adjustment slider 410 is an example of a first GUI, weight coefficient adjustment table 420 is an example of a second GUI, details screen 300 is an example of a details screen, and details screen call buttons 31a, 41a, and 511 are examples of details screen call buttons. [Explanation of symbols]

[0157] 10...Menu screen, 11...Annotation area, 12...Learning area, 13...Evaluation area, 13a...Learning curve button, 13b...Learning result button, 13c...Test evaluation button, 14...Setting area, 15...Queue screen call button, 20...Queue screen, 21, 31, 41...Model display area, 22...Detailed information display area, 23...Back button, 30...Learning curve screen, 31a, 41a, 511...Detailed screen call button, 31b...Model addition dialog call button, 32...Learning curve display area, 40...Learning result screen, 42...Learning result display area, 42a...Weight adjustment button, 100...Learning model evaluation support device, 110...Model information display unit, 120...Model selection unit, 130...Probability calculation unit, 140...Confusion matrix calculation unit, 150...Output display unit, 200...Learning model evaluation support system, 2 10...CPU, 220...RAM, 230...ROM, 240...storage device, 250...operation unit, 260...display device, 270...input / output I / F, 280...bus, 290...recording medium, 300...detail screen, 310...information selection field, 311...expand button, 312...collapse button, 320...detailed information display field, 321...detail button, 322...simple button, 400...weight adjustment screen, 410...weight coefficient adjustment slider, 420...weight coefficient adjustment table, 430...classification result display area, 440...OK button, 450...close button, 460...reset button, 500...test evaluation screen, 510...model data display area, 520...test evaluation display area, 521...confusion matrix screen, 522...arrangement screen, 523...overlay screen, 530...display method pull-down menu, 540...calculated value pull-down menu

Claims

1. a model information display unit that displays a plurality of learning models stored in a predetermined storage area and having the same interface on a display device in a selectable manner; a model selection unit that selects at least one learning model from the plurality of learning models displayed on the display device; A learning model evaluation support device comprising an output display unit that displays the output results from the at least one learning model on the display device.

2. the model selection unit selects two or more learning models from the plurality of learning models; The learning model evaluation support device of claim 1, wherein the output display unit switchably displays on the display device an arrangement screen that displays two or more confusion matrices that respectively indicate the output results of the two or more learning models, and an overlapping screen that displays a confusion matrix based on the output results of the two or more learning models.

3. 3. The learning model evaluation support device according to claim 2, wherein each element of the confusion matrix displayed on the superimposed screen is a calculated value of the corresponding element of the two or more confusion matrices.

4. the model selection unit selects two learning models from the plurality of learning models; 4. The learning model evaluation support device according to claim 3, wherein each element of the confusion matrix displayed on the superimposed screen is a difference value between the corresponding element of two confusion matrices corresponding to the two learning models, respectively.

5. The learning model evaluation support device according to any one of claims 2 to 4, wherein the output display unit displays each component of the confusion matrix on the array screen or the superimposed screen in a decorated manner according to a mode assigned to a size range.

6. 6. The learning model evaluation support device according to claim 5, wherein the aspect is color.

7. Each learning model performs a classification task to classify an image to be classified into one of multiple classes. A learning model evaluation support device described in any one of claims 1 to 4, wherein the output display unit displays a weight adjustment screen on the display device that accepts adjustments of weighting coefficients set for the multiple classes in order to adjust the output results from the at least one learning model.

8. a probability calculation unit that calculates a predicted probability that an image to be classified corresponds to each class based on a weighting coefficient set for each class; the weight adjustment screen includes a first GUI (Graphical User Interface) having a bar extending in one direction and a plurality of sliders arranged along the bar and movable in a direction parallel to the one direction, the bar being partitioned by the plurality of sliders into a plurality of regions corresponding to the plurality of classes, 8. The learning model evaluation support device of claim 7, wherein when any slider in the first GUI is moved in a direction parallel to the one direction, the probability calculation unit changes the weighting coefficients set for two classes corresponding to two areas of the bar separated by the slider in accordance with the amount of movement of the slider.

9. a probability calculation unit that calculates a predicted probability that an image to be classified corresponds to each class based on a weighting coefficient set for each class; the weight adjustment screen includes a second GUI on which a plurality of weight coefficients corresponding to the plurality of classes are described, The learning model evaluation support device of claim 7, wherein when the value of the weighting coefficient for any of the classes listed in the second GUI is rewritten, the probability calculation unit changes the weighting coefficient set for that class to the value rewritten in the second GUI, and changes the weighting coefficients set for all other classes based on the rewritten weighting coefficient value.

10. 9. The learning model evaluation support device according to claim 8, wherein the probability calculation unit calculates a predicted probability that the image to be classified belongs to each class based on the changed weighting coefficients, and updates the predicted probability for each class so that the sum of the predicted probabilities calculated for all classes becomes a constant value.

11. 8. The learning model evaluation support device according to claim 7, wherein the number of the plurality of classes is three or more.

12. A learning model evaluation support device as described in any one of claims 1 to 4, wherein the output display unit displays on the display device a screen including a details screen call button for displaying a details screen containing detailed information used when creating the learning model as the output result of the at least one learning model.

13. The detailed information includes a plurality of items of information, 13. The learning model evaluation support device according to claim 12, wherein when any one of the plurality of items of information is selected on the details screen, the output display unit expands the display of the information of the selected item.

14. A learning model evaluation support device as described in claim 13, wherein any of the multiple items of information includes information about training data, information about a method for augmenting image data that becomes the training data, information about a training network, or information about hyperparameters.

15. a model information display process for displaying, on a display device, a plurality of learning models stored in a predetermined storage area and having the same interface in a selectable manner; a model selection process for selecting at least one learning model from the plurality of learning models displayed on the display device; A learning model evaluation support program that causes a computer to execute an output display process that displays the output results from the at least one learning model on the display device.

Images