Information processing device, method, and program

Abstract

To support appropriate visualization of quality evaluation.SOLUTION: An information processing device includes a first acquisition unit, a second acquisition unit, an analysis unit, and a generation unit. The first acquisition unit obtains quantitative information relating to a machine learning model. The second acquisition unit obtains user input information that is input from a user. The analysis unit selects, on the basis of the quantitative information and the user input information, an item to be displayed and a visualizing method from display information definitions that include both items relating to evaluation of the machine learning model and visualizing methods of the evaluation. The generation unit generates, on the basis of a selection result of the analysis unit, display information that includes at least partial content of the quantitative information and the user input information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] In AI development using machine learning models, it is necessary to properly visualize the required quality information according to the behavior of trained models in the test phase and trained models in operation, and the results of qualitative evaluation by users. Quality information on machine learning models is extremely complex compared to conventional software, and a wide variety of information exists. Therefore, if all information is displayed, it is difficult for users to properly understand the necessary information. In addition, it is difficult for non-experts with advanced knowledge to scrutinize the necessary information. Therefore, conventional methods for visualizing quality information of machine learning models include displaying predetermined fixed items, but the visualized content cannot be dynamically changed depending on the behavior of the machine learning model in operation and the results of the user's qualitative evaluation, which makes it difficult to grasp the exact situation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 033338 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been made to solve the above-mentioned problems, and has an object to provide an information processing device, method, and program that can support the visualization of appropriate quality evaluation. [Means for solving the problem]

[0005] The information processing device according to the present embodiment includes a first acquisition unit, a second acquisition unit, an analysis unit, and a generation unit. The first acquisition unit acquires quantitative information related to a machine learning model. The second acquisition unit acquires user input information input by a user. The analysis unit selects an item to be displayed and a visualization method from a display information definition including an item related to an evaluation of the machine learning model and a visualization method of the evaluation, based on the quantitative information and the user input information. The generation unit generates display information including at least a part of the content of the quantitative information and the user input information, based on a selection result of the analysis unit. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing an information processing apparatus according to a first embodiment. [Diagram 2] 5 is a flowchart showing an example of the operation of the information processing device according to the first embodiment. [Diagram 3] A table showing the relationship between user attributes and development phases. [Figure 4] FIG. 2 is a view showing a first example of display information according to the first embodiment. [Diagram 5] FIG. 4 is a view showing a second example of display information according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing an information processing apparatus according to a second embodiment. [Figure 7] 10 is a flowchart showing an example of the operation of the information processing device according to the second embodiment. [Figure 8] 13A to 13C are diagrams showing display examples of display information and supplemental information according to the second embodiment. [Figure 9] FIG. 11 is a block diagram showing an information processing apparatus according to a third embodiment. [Figure 10] 13 is a flowchart showing an example of the operation of the information processing device according to the third embodiment. [Figure 11] 13 is a flowchart showing an example of the operation of an information processing device when updating a checklist according to the third embodiment. [Figure 12] FIG. 13 is a diagram showing an example of correction of a checklist when a predetermined criterion is satisfied according to the third embodiment. [Figure 13]FIG. 13 is a diagram showing an example of how to correct a checklist when a predetermined standard is not satisfied according to the third embodiment. [Figure 14] FIG. 13 is a block diagram showing an information processing apparatus according to a fourth embodiment. [Figure 15] 13 is a flowchart showing an example of the operation of the information processing device according to the fourth embodiment. [Figure 16] FIG. 13 is a block diagram showing an information processing apparatus according to a fifth embodiment. [Figure 17] 13 is a flowchart showing an example of the operation of the information processing device according to the fifth embodiment. [Figure 18] FIG. 13 is a view showing an example of displaying a comparison result according to the fifth embodiment. [Figure 19] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing device according to first to fifth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an information processing device, method, and program according to the present embodiment will be described in detail with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and duplicated descriptions will be omitted as appropriate.

[0008] (First embodiment) The information processing apparatus according to the first embodiment will be described with reference to the block diagram of FIG. The information processing device 10 according to the first embodiment includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, and a display control unit .

[0009] The storage unit 101 includes a machine learning model before training, a learned model which is a machine learning model that has been trained, training data, a display information definition, and the like. In the following, the model before training, the model under training, and the trained model are collectively referred to as a machine learning model. The machine learning model may be a machine learning model of any format as long as it has an architecture used in general machine learning. The display information definition includes items related to the evaluation of the machine learning model and a method for visualizing the evaluation. Specifically, the display information definition includes a plurality of items related to the evaluation of the machine learning model, a graph format for displaying the items, layout information, character information, and the like.

[0010] The quantitative information acquisition unit 102 acquires quantitative information related to the model. The quantitative information is a quantitative value for evaluating a trained model, such as the accuracy rate, precision rate, recall rate, and AUC (Area under the ROC curve) of the trained model. The quantitative information is not limited to numerical values, but also includes designs in which the quantitative values ​​are visualized and shaped as a graph or table.

[0011] The user input acquisition unit 103 acquires user input information input by a user. The user input information is, for example, information manually input into a checklist.

[0012] The analysis unit 104 performs an analysis based on the quantitative information and the user-input information, and selects items to be displayed and a visualization method from the display information definition.

[0013] The display information generating unit 105 generates display information including at least a part of the quantitative information and the user input information based on the selection result of the analyzing unit 104. The display control unit 106 displays the display information on, for example, a display.

[0014] Next, a first operation example of the information processing device 10 according to the first embodiment will be described with reference to the flowchart of FIG.

[0015] In step SA1, the quantitative information acquisition unit 102 acquires quantitative information. Specifically, the quantitative information acquisition unit 102 may acquire outputs such as the accuracy rate and the matching rate of the trained model generated by, for example, a machine learning development platform.

[0016] In step SA2, the user input acquisition unit 103 acquires user input. Specifically, a checklist of questions created using an existing spreadsheet software may be prepared in advance, and the user may check questions in the checklist or enter information in a free description field to acquire user input. Information given in response to questions from the system by manual or voice input may also be acquired as user input. In addition, columns for questions and categories may be provided in the checklist, and the user may be asked to check or enter information, so that the information entered by the user may include information about the categories.

[0017] In step SA3, the analysis unit 104 acquires the display information definition stored in the storage unit 101, for example.

[0018] In step SA4, the analysis unit 104 analyzes the quantitative information acquired in step SA1 and the user input acquired in step SA2, and selects an item to be displayed from the display information definition acquired in step SA3. For example, the selection criteria may be such that the item to be displayed is selected based on the importance of the information. Specifically, the user is asked to enter a goal and a borderline that should not be exceeded, and the entered goal or borderline is acquired as a user input. The analysis unit 104 analyzes the quantitative information based on the user input. For example, if a correct answer rate of 90% is entered as the goal, it is determined whether the correct answer rate in the quantitative information has reached 90%. If the correct answer rate does not reach 90%, the item may be selected and modified so that the importance is increased and the item is emphasized in the display information.

[0019] Alternatively, if the numerical value obtained from the quantitative information for a certain item differs or is inconsistent with the numerical value obtained by user input for that item, there is a strong need to inform the user, so the item can be selected and modified so that the importance of that item is set higher than other items.

[0020] The importance of each item may be initially set in advance. For example, the importance of each question according to user attributes may be defined in a hidden column of a tabular checklist created by a spreadsheet software, and may be set as the initial setting in the analysis process. In other words, when a checklist is filled out, the importance of the items in the display information definition is analyzed based on the initial value of the importance of the user attribute corresponding to the checklist, and an item is selected and modified from among the multiple items in the display information definition.

[0021] In step SA5, the display information generating unit 105 generates display information including evaluation information related to the quantitative information and information input by the user based on the selection result of step SA4. Examples of the display information may be any format commonly used by the user, such as HTML (Hyper Text Markup Language), Markdown, Word (registered trademark), PowerPoint (registered trademark), or PDF file.

[0022] In step SA6, the display control unit 106 displays the display information on a display etc. This makes it possible to display display information that compiles information that is highly important to a specific user.

[0023] Next, the relationship between the user attributes, which are an index for setting the importance level, and the development phase will be described with reference to the table in FIG. The table shown in Fig. 3 shows items that are considered important in combination with the type of user attribute and the development phase. In other words, it becomes a criterion for which user and in which development phase the importance of the items to be displayed is set to be high. Specifically, if the user attribute is "engineer" and the development phase is "verification", the analysis unit 104 sets the importance of the items related to the "feasibility" of the model higher than the other items, and the display information generation unit 105 generates display information including the items related to the "feasibility".

[0024] Instead of considering both the user attribute and the development phase, either the user attribute or the development phase may be considered. That is, the importance of an item may be determined for each user attribute such as "customer" or "sales," or the importance of an item may be determined for each development phase such as "PoC" or "development." Furthermore, the method of determining the importance may be defined in a personalized manner for each individual, such as "User A" and "User B."

[0025] Next, a first example of display information according to the first embodiment is shown in FIG. Figure 4 shows an example of information displayed for users without deep expertise. This example shows "Dataset Information" about the dataset, "Model Information" about the machine learning model, and "Additional Information" about the training data.

[0026] In the data set information, only information necessary as an item to be displayed, that is, for example, information of high importance whose importance is equal to or greater than a threshold, is displayed, and unnecessary information of low importance whose importance is less than the threshold is not displayed. In addition, information whose importance is equal to or greater than a threshold is highlighted by text modification such as bolding or coloring in blue or red. Note that the highlighting may be displayed in any manner such as blinking or highlighting, as long as it can be distinguished from other items.

[0027] The model information includes, for example, the block structure, the model architecture such as the number of parameters, the inference results by the model, and the intention and use of the model obtained from user input information. The additional information includes, for example, the label distribution of the training data.

[0028] 4 combines information based on user input information 41 and information based on quantitative information 42. This allows the display of qualitative evaluations by users in addition to quantitative evaluations of machine learning models, making it possible to visualize optimal AI quality.

[0029] Of course, the display information is not limited to the example in Fig. 4, and the layout, the method of visualization, etc. may be arranged in any manner. Also, the display information may be displayed in a part of the screen that is easily noticeable.

[0030] Next, a second example of the display information according to the first embodiment is shown in FIG. FIG. 5 is an example of display information created assuming a user with deep specialized knowledge. Here, compared to FIG. 4, user input information 51 related to the data set information is different. Information with a degree of importance equal to or greater than a threshold is highlighted, here displayed in bold. When the analysis unit 104 analyzes that the user who inputted information into a checklist or the like is an "engineer" with specialized knowledge, it is possible to present useful information to the user who is an "engineer," such as the number of training data items and visualization using t-SNE.

[0031] According to the first embodiment described above, quantitative information and user input information are analyzed, and necessary information is selected and modified from the display information definition to generate display information. This makes it possible to present information necessary for quality evaluation of a trained model while covering all necessary information according to the type of user and development phase, thereby assisting in the visualization of appropriate quality evaluation.

[0032] Second embodiment The second embodiment differs from the above-described embodiments in that information that is not displayed as display information is displayed as supplemental information.

[0033] An information processing device 10 according to the second embodiment will be described with reference to the block diagram of FIG. The information processing device 10 according to the second embodiment includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, a display control unit 106, and a supplemental information generation unit 201.

[0034] The supplemental information generating unit 201 uses the display information generated by the display information generating unit 105 to collect items that are not included in the display information, and generates supplemental information.

[0035] Next, the operation of the information processing device 10 according to the second embodiment will be described with reference to the flowchart of FIG. Steps SA1 to SA5 are the same processes as those in FIG.

[0036] In step SB1, the supplemental information generating unit 201 generates supplemental information for supplementing information that is no longer displayed due to the modification of the display information definition in step SA5. The supplemental information may be generated, for example, by summarizing (for example, listing) information on items that have been determined to be hidden due to the modification of the display information definition. In step SB2, the display control unit 106 displays the display information and the supplemental information on a display or the like.

[0037] A display example of the display information and supplemental information according to the second embodiment will be described with reference to FIG. 8, supplemental information 81 is displayed in addition to display information 80. Here, an example is assumed in which display information 80 and supplemental information 81 are displayed side by side, but supplemental information 81 may be displayed on the screen after a user instructs it to be displayed. That is, for example, a button for displaying supplemental information may be provided in the display information area, and supplemental information 81 may be displayed when the user clicks the button, or when a function is assigned to a specific button on the keyboard and the specific button is pressed.

[0038] According to the second embodiment described above, by selecting an item from the display information definition, information that is no longer displayed is displayed together as supplementary information. This supports the visualization of appropriate quality evaluation, and even if the information desired by the user becomes hidden by the analysis by the analysis unit, the user can check the necessary information by the supplementary information.

[0039] Third embodiment The third embodiment differs from the above-described embodiments in that, if there is an item to be corrected in the user-input information, feedback is given to the user so that the user can correct it.

[0040] An information processing device 10 according to the third embodiment will be described with reference to the block diagram of FIG. The information processing device 10 includes a storage unit 101 , a quantitative information acquisition unit 102 , a user input acquisition unit 103 , an analysis unit 104 , a display information generation unit 105 , a display control unit 106 , and a user input correction unit 301 .

[0041] The user input correction unit 301 corrects the user input information based on the analysis process performed by the analysis unit 104 .

[0042] Next, an example of the operation of the information processing device 10 according to the third embodiment will be described with reference to the flowchart of FIG. Steps SA1 to SA5 are the same process.

[0043] In step SC1, the user input correction unit 301 corrects the user input information according to an analysis based on the quantitative information and the user input information. For example, assume that information is manually acquired from a user using a tabular checklist. If the accuracy rate of the trained model is recorded as "A%" in the checklist, while the accuracy rate of the machine learning model based on the quantitative information is "B%," it is considered that the user made an input error.

[0044] Therefore, the user input correction unit 301 rewrites the item of the accuracy rate in the checklist from A% to B%. When the rewrite process is executed, the rewritten item may be highlighted in a manner that makes it easy for the user to understand which part has been corrected when the user uses the checklist from the next time onwards, for example, by displaying the rewritten item in red.

[0045] In step SC2, the display control unit 106 displays the display information and the corrected user-input information on a display or the like.

[0046] The user input correction unit 301 may update the checklist in accordance with the analysis of the quantitative information and the user input information. An example of the operation of the information processing device 10 when updating the checklist will be described with reference to the flowchart of FIG. Steps SA1 to SA5 are the same process.

[0047] In step SD1, the user input correction unit 301 judges whether the analysis result by the analysis unit 104 for the user input information satisfies a predetermined criterion. For example, if "MNIST" is entered for the input item "What dataset was used?" and the information on the dataset is also "MNIST" in the quantitative information, it is considered that correct information has been entered, so it can be judged that the analysis result satisfies the predetermined criterion. If the analysis result satisfies the predetermined criterion, the process proceeds to step SD2, and if the analysis result does not satisfy the predetermined criterion, the process proceeds to step SD3.

[0048] In step SD2, the user input correction unit 301 updates the checklist related to the user input so that items that meet a predetermined criterion are distinguished. For example, in the above-mentioned "MNIST" example, a method of distinction may be to distinguish and display items that meet a predetermined criterion when inputting the checklist from the next time onwards, so that they do not require input. Alternatively, if an item that did not require input this time meets a criterion for changing to an item that requires input next time, it is displayed so that it requires input. In this way, the user input correction unit 301 may correct the necessity of user input for the items in the checklist.

[0049] In step SD3, the user input correction unit 301 updates the checklist related to the user input so that items that do not meet the predetermined criteria are clearly indicated. Then, the process proceeds to step SC2, where the display information and the corrected user input information are displayed.

[0050] Next, an example of how the checklist is modified in step SD2 when a predetermined criterion is met will be described with reference to FIG. The upper diagram in Fig. 12 shows the checklist before correction by user input correction unit 301, in which all items require input. The lower diagram in Fig. 12 shows the checklist after update (correction) by user input correction unit 301, assuming a case in which items that meet specified criteria exist. Here, items that meet the specified criteria are displayed in gray-out 1201, and are treated as not requiring input. On the other hand, it is sufficient if the list is updated so that only items that require input can be entered. The user can see at a glance which items require input.

[0051] For example, a checklist in which items to be filled in differ depending on the development phase is assumed. Specifically, in the example of FIG. 12, in the case of the development phase "PoC", three items from item 3 to item 5, "item 3: Will the dataset be updated?", "item 4: Who will perform the update work?", and "item 5: How often will it be updated?" are to be filled in. On the other hand, in the development phase "development", eight items from item 1 to item 8 are to be filled in. As shown in FIG. 12, for items in which the user input correction unit 301 has determined that necessary information has been filled in for the current development phase, the checklist is updated so that it is not necessary to fill in the items for the next development phase. This reduces the burden of consideration and input on the user.

[0052] Next, an example of how to correct the checklist in step SD3 when a predetermined standard is not met will be described with reference to FIG. The upper diagram in Fig. 13 shows the checklist before correction by user input correction unit 301, and assumes a state in which all input is mandatory, as in the upper diagram in Fig. 12. The lower diagram in Fig. 13 shows the checklist updated (after correction) by user input correction unit 301. Here, an example is shown in which there is an item that does not meet a predetermined standard, and user input correction unit 301 extracts and displays items that require input as unexecuted items in list 1301. Of course, user input correction unit 301 may highlight items that require input in the checklist state, such as by highlighting them in red, or may display a message such as "Please check as the standard is not met."

[0053] According to the third embodiment described above, the user input correction unit corrects the user information according to the analysis result by the analysis unit, thereby allowing information to be fed back to the user. Furthermore, if the user input correction unit finds an item that meets a predetermined criterion based on the analysis result by the analysis unit, the unit updates the checklist related to the user input so that the item is not required to be input when the checklist is used thereafter. On the other hand, if there is an item that does not meet the predetermined criterion, the unit updates the checklist related to the user input so that the item is clearly indicated. This makes it possible to provide the user with feedback of correct information and simplify input from the next time onwards while supporting visualization of appropriate quality evaluation, thereby reducing the user's efforts.

[0054] (Fourth embodiment) The fourth embodiment differs from the above-described embodiments in that the user is prompted to confirm information that requires confirmation by the user. The information processing device 10 according to the fourth embodiment will be described with reference to the block diagram of Fig. 14. The information processing device 10 includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, a display control unit 106, and a user confirmation unit 401.

[0055] The analysis unit 104 analyzes the quantitative information and the user input information, and determines whether or not there is any information that requires confirmation by the user. A user confirmation unit 401 presents to the user information that is determined to require confirmation by the user based on the analysis by the analysis unit 104, and obtains feedback from the user. The display information generating unit 105 generates display information including feedback from the user.

[0056] Next, the operation of the information processing device 10 according to the fourth embodiment will be described with reference to the flowchart of FIG. Steps SA1 to SA4 and step SA6 are similar processes.

[0057] In step SE1, the analysis unit 104 analyzes the information based on the quantitative information, the user input information, and the display information definition, and determines whether or not there is information that requires user confirmation. The information that requires user confirmation is, for example, t-SNE information, and it is difficult for an information processing device to determine that there is no difference between the learning data and the actual operation data based on the distribution state of t-SNE. In this way, information that is difficult to determine by the device or that can be determined by the user with higher accuracy in a shorter time than by the device is considered to be information that requires user confirmation. Note that the information that requires user confirmation may be listed in advance, and when an item included in the list is selected as an item included in the display information as a result of the analysis by the analysis unit 104, it may be determined that there is information that requires user confirmation.

[0058] If the display information includes items that require user confirmation, the process proceeds to step SE2. If the display information does not include items that require user confirmation, the process proceeds to step SA5.

[0059] In step SE2, the user confirmation unit 401 displays information requiring user confirmation on a display or the like via the display control unit 106. The method of confirmation for the user may involve illustrating the relevant information and accepting a Yes / No input as to whether there are any contradictions in the contents of the illustration or the contents of the description such as the sentence explaining the contents of the illustration.

[0060] In step SE3, the user confirmation unit 401 acquires feedback from the user regarding the information that requires user confirmation. Specifically, if the information is in a Yes / No format as described above, it is sufficient to accept input of "Yes" or "No" from the user. Furthermore, the user confirmation unit 401 may acquire feedback in a free description format in addition to the input of "Yes" or "No" from the user.

[0061] In step SA5, if the user confirmation unit 401 receives an input of "Yes", it is assumed that there is no problem with the content of the item, and the display information generation unit 105 generates display information using the content of the item as is. On the other hand, if the user confirmation unit 401 receives an input of "No", it is sufficient to generate display information that displays correction information after feedback, or that highlights the information so that it is clear that there is a problem with the information.

[0062] According to the fourth embodiment described above, it is determined whether or not there is information that requires user confirmation as an item of display information based on the quantitative information and user input information, and if there is information that requires user confirmation, feedback from the user on the information is obtained. After that, display information reflecting the content of the feedback is generated. This makes it possible to provide an appropriate model quality evaluation that reflects feedback from users while helping to visualize the appropriate quality evaluation.

[0063] Fifth embodiment The fifth embodiment differs from the above-described embodiments in that the generated display information is compared with past information. An information processing device 10 according to the fifth embodiment will be described with reference to the block diagram of FIG. The information processing device 10 includes a storage unit 101 , a quantitative information acquisition unit 102 , a user input acquisition unit 103 , an analysis unit 104 , a display information generation unit 105 , a display control unit 106 , and an information comparison unit 501 .

[0064] The storage unit 101 stores a plurality of pieces of past information 505, which is display information generated in the past. The past information 505 may be stored in the storage unit 101 every time the display information generating unit 105 generates display information. The information comparison unit 501 compares with past information 505 stored in the storage unit 101 and generates a comparison result.

[0065] Next, the operation of the information processing device 10 according to the fifth embodiment will be described with reference to the flowchart of FIG. Steps SA1 to SA5 are the same process.

[0066] In step SF1, the information comparison unit 501 compares the display information generated in step SA5 with the past information 505 stored in the storage unit 101. For example, based on a search condition input by a user, past information whose items match those of the display information generated in step SA5 is searched for. Note that the information comparison unit 501 may set default search conditions according to the type of data, the task of the machine learning model, and the like, and extract past information whose items match those of the generated display information according to the default search conditions. Alternatively, past information whose similarity of items or similarity of contents between the display information and past information is equal to or greater than a threshold may be extracted, regardless of whether it is based on the search conditions. In step SF2, the display control unit 106 displays the comparison result (search result) obtained in step SF1 together with the display information on a display or the like.

[0067] Next, an example of a comparison result according to the fifth embodiment will be described with reference to FIG. 18 is a conceptual diagram showing an example of a method for generating a comparison result. The information comparison unit 501 searches for past information 1802 using one or more items of the generated display information 1801 as a search condition. Specifically, it is assumed here that a model type "model A" is trained using a data type "data C" in the development phase "development" and display information 1801 ("development / data C / model A") is generated. When the search conditions are the development phase "development" and the model type "model A", the information comparison unit 501 extracts "development / data A / model A" and "development / data B / model A" as the corresponding past information 1802.

[0068] Finally, display information 1801 "Development / Data C / Model A" and two pieces of past information 1802 "Development / Data A / Model A" and "Development / Data B / Model A" are displayed side by side for comparison as a comparison result 1803. In the comparison result 1803, common parts or different parts between the display information 1801 and the past information 1802 may be highlighted.

[0069] When the past information 1802 is stored in the storage unit 101, tags related to items or keywords may be linked to the past information 1802 and stored. For example, the development phase (development, PoC, etc.), model performance (correct rate, F value, etc.), used data, used algorithm, etc. are linked to the display information as tags. In the subsequent analysis, past information to be compared may be searched and extracted based on the tags. Note that which tag to use for comparison may be arbitrarily determined by the user, or the information comparison unit 501 may automatically select the tag based on the past comparison results and the items of the display information to be generated as the comparison source.

[0070] According to the fifth embodiment described above, display information generated in the past is stored in a storage unit as past information, and an information comparison unit searches for past information having items in common with the display information, and displays the display information and the past information side by side as a comparison result. This helps visualize appropriate quality assessment while efficiently supporting user judgment based on the displayed information and past information, such as the items the user wants to compare, such as how model performance differs due to differences in training data.

[0071] Next, an example of a hardware configuration of the information processing device 10 according to the above-described first to fifth embodiments is shown in a block diagram in FIG. The information processing device 10 includes a CPU (Central Processing Unit) 1901, a RAM (Random Access Memory) 1902, a ROM (Read Only Memory) 1903, a storage 1904, a display device 1905, an input device 1906, and a communication device 1907, each of which is connected by a bus.

[0072] The CPU 1901 is a processor that executes arithmetic processing and control processing according to a program. The CPU 1901 executes the processing of each part of the information processing device 10 described above in cooperation with programs stored in the ROM 1903 and the storage 1904, etc., using a predetermined area of ​​the RAM 1902 as a working area. Each process of the information processing device 10 may be executed by one processor, or may be executed in a distributed manner by multiple processors.

[0073] The RAM 1902 is a memory such as a Synchronous Dynamic Random Access Memory (SDRAM), and functions as a work area for the CPU 1901. The ROM 1903 is a memory that stores programs and various information in a non-rewritable manner.

[0074] The storage 1904 is a device that writes and reads data to and from a magnetic recording medium such as a hard disk drive (HDD), a semiconductor storage medium such as a flash memory, a magnetically recordable storage medium such as a HDD, an optically recordable storage medium, etc. The storage 1904 writes and reads data to and from the storage medium in response to control from the CPU 1901.

[0075] The display device 1905 is a display device such as an LCD (Liquid Crystal Display), etc. The display device 1905 displays various information based on a display signal from the CPU 1901.

[0076] The input device 1906 is an input device such as a mouse, a keyboard, etc. The input device 1906 receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 1901.

[0077] The communication device 1907 communicates with external devices via a network under the control of the CPU 1901 .

[0078] The instructions shown in the processing procedure shown in the above-mentioned embodiment can be executed based on a program, which is software. A general-purpose computer system can store this program in advance and obtain the same effect as the effect of the control operation of the above-mentioned information processing device by reading this program. The instructions described in the above-mentioned embodiment are recorded as a program that can be executed by a computer on a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray (registered trademark) Disc, etc.), a semiconductor memory, or a recording medium similar thereto. The recording medium may be in any storage format as long as it is readable by a computer or an embedded system. The computer can realize the same operation as the control of the information processing device of the above-mentioned embodiment by reading the program from this recording medium and executing the instructions described in the program based on this program with a CPU. Of course, when the computer acquires or reads the program, it may acquire or read it through a network. In addition, an OS (operating system), database management software, network, or other MW (middleware) running on a computer may execute some of the processes required to realize this embodiment based on instructions from a program installed on the computer or embedded system from a recording medium. Furthermore, the recording medium in this embodiment is not limited to a medium independent of a computer or an embedded system, but also includes a recording medium that stores or temporarily stores a program downloaded via a LAN, the Internet, or the like. Furthermore, the number of recording media is not limited to one, and a case where the processing in this embodiment is executed from a plurality of media is also included in the recording media in this embodiment, and the media may have any configuration.

[0079] The computer or embedded system in this embodiment is for executing each process in this embodiment based on a program stored in a recording medium, and may be configured as any one of a device such as a personal computer or a microcomputer, or a system in which multiple devices are connected to a network. In addition, the computer in this embodiment is not limited to a personal computer but also includes an arithmetic processing device, a microcomputer, etc. included in information processing equipment, and is a general term for equipment or devices that can realize the functions in this embodiment by a program.

[0080] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0081] 10...information processing device, 41,51...user input information, 42...quantitative information, 80,1801...display information, 81...supplementary information, 101...storage unit, 102...quantitative information acquisition unit, 103...user input acquisition unit, 104...analysis unit, 105...display information generation unit, 106...display control unit, 107...training unit, 201...supplementary information generation unit, 301...user input correction unit, 401...user confirmation unit, 501...information comparison unit, 505,1802...past information, 1201...grayed out, 1301...list, 1901...CPU, 1902...RAM, 1903...ROM, 1904...storage, 1905...display device, 1906...input device, 1907...communication device.

Claims

1. A first acquisition unit that acquires quantitative information regarding the machine learning model; A second acquisition unit that acquires user input information input by a user; an analysis unit that selects an item to be displayed and a visualization method from a display information definition including an item related to an evaluation of the machine learning model and a visualization method of the evaluation based on the quantitative information and the user input information; a first generation unit that generates display information including at least a part of the quantitative information and the user input information based on a selection result of the analysis unit; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the analysis unit sets the importance of items included in the display information definition according to attribute information of the user and a development phase in which the machine learning model is used when the quantitative information is obtained.

3. The information processing apparatus according to claim 1 , further comprising a second generating unit configured to generate items not included in the display information as supplemental information.

4. a correction unit that corrects the user input information when the user input information needs to be corrected as a result of an analysis based on the quantitative information and the user input information; The information processing apparatus according to claim 1 , further comprising: a display control unit that displays the corrected user input information to the user.

5. The information processing device of claim 1, further comprising a correction unit that, when it is determined that the results of the analysis based on the quantitative information and the user input information satisfy a predetermined criterion, corrects whether or not user input is required for the items on the checklist depending on the phase in which the checklist for obtaining the user input information will next be used.

6. The information processing device of claim 1, further comprising a correction unit that, when a relevant item is determined to not satisfy a predetermined standard based on the results of an analysis based on the quantitative information and the user input information, extracts the relevant item from a checklist for acquiring the user input information, or highlights the relevant item in the checklist.

7. A user confirmation unit is further provided, The analysis unit determines whether or not there is information that requires confirmation by the user based on the quantitative information and the user input information, The information processing apparatus according to claim 1 , wherein, when there is information that requires confirmation by the user, the user confirmation unit presents the information to the user and obtains feedback from the user.

8. a comparison unit that extracts past information that matches a search condition based on the newly generated display information from a plurality of past information that is display information generated in the past; The information processing apparatus according to claim 1 , further comprising: a display control unit that displays the extracted past information and the display information as a comparison result to the user.

9. The first acquisition unit acquires quantitative information regarding the machine learning model, The second acquisition unit acquires user input information input by a user, an analysis unit, based on the quantitative information and the user input information, selecting an item to be displayed and a visualization method from a display information definition including an item related to an evaluation of the machine learning model and a visualization method of the evaluation; An information processing method, wherein a first generation unit generates display information including at least a portion of the quantitative information and the user input information based on a selection result of the analysis unit.

10. Computer, A first acquisition means for acquiring quantitative information regarding the machine learning model; A second acquisition means for acquiring user input information input by a user; An analysis means for selecting an item to be displayed and a visualization method from a display information definition including an item related to the evaluation of the machine learning model and a visualization method of the evaluation based on the quantitative information and the user input information; an information processing program for causing the computer to function as a first generating means for generating display information including the quantitative information and at least a portion of the user input information based on a selection result of the analyzing means;

Images