Information processing device, method, and program

Abstract

To support appropriate visualization of quality evaluation.SOLUTION: An information processing device includes a first acquisition unit, a second acquisition unit, an analysis unit, and a generation unit. The first acquisition unit obtains quantitative information relating to a machine learning model. The second acquisition unit obtains user input information that is input from a user. The analysis unit selects, on the basis of the quantitative information and the user input information, an item to be displayed and a visualizing method from display information definitions that include both items relating to evaluation of the machine learning model and visualizing methods of the evaluation. The generation unit generates, on the basis of a selection result of the analysis unit, display information that includes at least partial content of the quantitative information and the user input information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0005] , , ,

[0004] , ,

[0001] Embodiments of the present invention relate to an information processing apparatus, method, and program.

Background Art

[0002] In AI development using a machine learning model, it is necessary to appropriately visualize the required quality information according to the behavior of the learned model in the test stage and the learned model during operation, and the qualitative evaluation results of the user. The quality information regarding the machine learning model is very complex compared to conventional software, and there is a wide variety of information. Therefore, if all the information is displayed, it is difficult for the user to appropriately understand the necessary information. Also, it is difficult for non-experts with advanced knowledge to scrutinize the necessary information. Therefore, as a conventional method for visualizing the quality information of a machine learning model, there is a method of displaying fixed items determined in advance, but according to the behavior of the machine learning model during operation and the results of the user's qualitative evaluation, the visualization content cannot be changed dynamically, and there is a problem that it is difficult to grasp the accurate situation.

Prior Art Documents

Patent Documents

[0003] ​​​​​​​​​​​​​​​​​​​​​The information processing device according to this embodiment includes a first acquisition unit, a second acquisition unit, an analysis unit, and a generation unit. The first acquisition unit acquires quantitative information relating to a machine learning model. The second acquisition unit acquires user input information entered by the user. The analysis unit selects items to be displayed and a visualization method from a display information definition that includes items relating to the evaluation of the machine learning model and a method for visualizing the evaluation, based on the quantitative information and the user input information. The generation unit generates display information that includes at least a portion of the quantitative information and the user input information, based on the selection result of the analysis unit. [Brief explanation of the drawing]

[0006] [Figure 1] A block diagram showing an information processing device according to the first embodiment. [Figure 2] A flowchart showing an example of the operation of the information processing device according to the first embodiment. [Figure 3] A table showing the relationship between user attributes and development phase. [Figure 4] A figure showing a first example of display information according to the first embodiment. [Figure 5] A figure showing a second example of display information according to the first embodiment. [Figure 6] A block diagram showing an information processing device according to the second embodiment. [Figure 7] A flowchart showing an example of the operation of the information processing device according to the second embodiment. [Figure 8] A figure showing an example of the display information and supplementary information according to the second embodiment. [Figure 9] A block diagram showing an information processing device according to the third embodiment. [Figure 10] A flowchart showing an example of the operation of the information processing device according to the third embodiment. [Figure 11] A flowchart showing an example of the operation of the information processing device when updating the checklist according to the third embodiment. [Figure 12] A diagram showing an example of a modified checklist when the predetermined criteria according to the third embodiment are met. [Figure 13]A diagram showing an example of a modified checklist in the case where the predetermined criteria according to the third embodiment are not met. [Figure 14] A block diagram showing an information processing device according to the fourth embodiment. [Figure 15] A flowchart showing an example of the operation of the information processing device according to the fourth embodiment. [Figure 16] A block diagram showing an information processing device according to the fifth embodiment. [Figure 17] A flowchart showing an example of the operation of the information processing device according to the fifth embodiment. [Figure 18] A figure showing an example of displaying the comparison results according to the fifth embodiment. [Figure 19] A diagram showing an example of the hardware configuration of an information processing device according to the first to fifth embodiments. [Modes for carrying out the invention]

[0007] The information processing apparatus, method, and program according to this embodiment will be described in detail below with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and redundant explanations will be omitted as appropriate.

[0008] (First Embodiment) The information processing device according to the first embodiment will be described with reference to the block diagram in Figure 1. The information processing device 10 according to the first embodiment includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, and a display control unit 106.

[0009] The storage unit 101 includes a machine learning model before training, a learned model which is a trained machine learning model, training data, display information definitions, and the like. Hereinafter, the model before training, the model during training, and the trained model are collectively referred to as a machine learning model. The machine learning model may be any form of machine learning model as long as it is an architecture used in general machine learning. The display information definition includes items related to the evaluation of the machine learning model and a method for visualizing the evaluation. Specifically, the display information definition includes a plurality of items related to the evaluation of the machine learning model, a graph format for displaying the items, layout information, character information, and the like.

[0010] The quantitative information acquisition unit 102 acquires quantitative information related to the model. The quantitative information is, for example, quantitative values for evaluating a learned model such as the accuracy rate, precision rate, recall rate, and AUC (Area under the ROC curve) of the learned model. Note that the quantitative information is not limited to numerical values and includes designs obtained by visualizing and formatting the quantitative values as graphs or tables.

[0011] The user input acquisition unit 103 acquires user input information input from the user. The user input information is, for example, information manually input to a checklist.

[0012] The analysis unit 104 analyzes based on the quantitative information and the user input information, and selects items to be displayed and a visualization method from the display information definition.

[0013] The display information generation unit 105 generates display information including at least part of the content of the quantitative information and the user input information based on the selection result of the analysis unit 104. The display control unit 106 displays the display information on, for example, a display.

[0014] Next, a first operation example of the information processing apparatus 10 according to the first embodiment will be described with reference to the flowchart of FIG. 2.

[0015] In step SA1, the quantitative information acquisition unit 102 acquires quantitative information. Specifically, the quantitative information acquisition unit 102 should acquire outputs such as the accuracy and precision of a trained model, which are generated by a machine learning development platform, for example.

[0016] In step SA2, the user input acquisition unit 103 acquires user input. Specifically, a checklist of questions created in an existing spreadsheet program can be prepared in advance, and user input can be acquired when the user checks the questions on the checklist or enters information in the free-text field. Information answered by manual or voice input to system-side questions should also be acquired as user input. In addition, the checklist can be divided into columns for questions and categories, and the user can be asked to check the boxes or enter information to include category information in the information provided by the user.

[0017] In step SA3, the analysis unit 104 acquires, for example, the display information definition to be stored in the storage unit 101.

[0018] In step SA4, the analysis unit 104 analyzes the quantitative information obtained in step SA1 and the user input obtained in step SA2, and selects items to be displayed from the display information definition obtained in step SA3. As a criterion for selection, for example, items to be displayed should be selected based on the importance of the information. Specifically, the user is asked to fill in their achievement target and the borderline that must not be exceeded, and the entered achievement target or borderline is obtained as user input. Based on this user input, the analysis unit 104 analyzes the quantitative information. For example, if a correct answer rate of 90% is entered as the achievement target, the analysis unit 104 determines whether the correct answer rate has reached 90% in the quantitative information. If the correct answer rate has not reached 90%, the item should be selected and modified so that its importance is increased and it is emphasized in the display information.

[0019] Alternatively, if the numerical value obtained from quantitative information for a particular item differs from or is inconsistent with the numerical value obtained from user input for that item, there is a greater need to inform the user. Therefore, the importance of that item should be set higher compared to other items, and the item should be selected and modified accordingly.

[0020] The importance level of each item may be set in advance. For example, the importance level for each user attribute can be defined in a hidden column of a tabular checklist created with spreadsheet software, and this can be used as the initial setting for the analysis process. In other words, when an item is filled out on the checklist, the importance level of the items in the display information definition is analyzed based on the initial importance level of the user attribute corresponding to that checklist item, and the user can then select and modify items from among the multiple items in the display information definition.

[0021] In step SA5, the display information generation unit 105 generates display information, including evaluation information related to quantitative information and user input information, based on the selection result in step SA4. Examples of display information include HTML (Hyper Text Markup Language), Markdown, Word® files, PowerPoint® files, PDF files, etc., as long as the user creates them in a common format.

[0022] In step SA6, the display control unit 106 displays the information on a display or the like. This allows for the display of information that is highly important to a specific user in a consolidated manner.

[0023] Next, we will explain the relationship between user attributes, which serve as an indicator for setting importance, and the development phase, referring to the table in Figure 3. The table shown in Figure 3 indicates the items that are considered important in combinations of user attribute types and development phases. In other words, it serves as a criterion for determining which items should be displayed and which development phases for which users should have a higher importance. Specifically, if the user attribute is "engineer" and the development phase is "verification," the analysis unit 104 should set the importance of the item related to the "feasibility" of the model higher than other items, and the display information generation unit 105 should generate display information that includes the item related to "feasibility."

[0024] Furthermore, instead of considering both user attributes and development phases, it is acceptable to consider either user attributes or development phases alone. In other words, the importance of items can be determined for each user attribute, such as "customer" or "sales," or for each development phase, such as "PoC" or "development." Furthermore, the method for determining importance can be defined in a personalized way, such as "User A" and "User B".

[0025] Next, Figure 4 shows a first example of the display information according to the first embodiment. Figure 4 shows an example of display information created for users with limited expertise. This example displays "Dataset Information" about the dataset, "Model Information" about the machine learning model, and "Additional Information" about the training data.

[0026] The dataset information will only display necessary items, such as high-importance information with an importance level above a threshold, while unnecessary, low-importance information with an importance level below the threshold will not be displayed. Furthermore, items with an importance level above the threshold will be highlighted using text formatting, such as bolding or changing the color to blue or red. Any display method is acceptable for highlighting, such as blinking or highlighting, as long as it is distinguishable from other items.

[0027] Model information includes, for example, the model architecture such as the block structure and number of parameters, the inference results from the model, and information about the model's intent and purpose derived from user input. Additional information includes, for example, the label distribution of the training data.

[0028] Here, the displayed information shown in Figure 4 combines information based on user input information 41 and information based on quantitative information 42. This allows for the display of both quantitative evaluation of the machine learning model and qualitative evaluation by the user, enabling the visualization of optimal AI quality.

[0029] Of course, the displayed information is not limited to the example in Figure 4; the layout, visualization methods, and other aspects can be arranged in any way. Furthermore, the displayed information may be shown in a more prominent location on the screen.

[0030] Next, Figure 5 shows a second example of the display information according to the first embodiment. Figure 5 shows an example of display information created assuming a user with deep expertise. Here, the user input information 51 regarding the dataset information differs from that in Figure 4. Information with an importance level above a threshold is highlighted, and in this case, it is displayed in bold. If the analysis unit 104 analyzes that the user who entered information into the checklist, etc., is an "engineer" with expertise, it can present useful information for the "engineer" user, such as the number of training data and whether it is visualized using t-SNE.

[0031] According to the first embodiment described above, by analyzing quantitative information and user input information, and selecting and modifying the necessary information from the display information definition to display the information, it is possible to present the information necessary for quality evaluation of the trained model while comprehensively covering the information required according to the type of user and development phase, thereby supporting the visualization of appropriate quality evaluation.

[0032] (Second Embodiment) The second embodiment differs from the above-described embodiment in that it displays information that is not shown as display information as supplementary information.

[0033] The information processing device 10 according to the second embodiment will be described with reference to the block diagram in Figure 6. The information processing device 10 according to the second embodiment includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, a display control unit 106, and a supplementary information generation unit 201.

[0034] The supplementary information generation unit 201 uses the display information generated by the display information generation unit 105 to consolidate items not included in the display information and generate supplementary information.

[0035] Next, the operation of the information processing device 10 according to the second embodiment will be explained with reference to the flowchart in Figure 7. Steps SA1 through SA5 are the same as those shown in Figure 2.

[0036] In step SB1, the supplementary information generation unit 201 generates supplementary information to compensate for the information that is no longer displayed due to the modification of the display information definition in step SA5. The supplementary information can be generated, for example, by compiling (e.g., creating a list of) the information of the items that have been decided to be hidden due to the modification of the display information definition. In step SB2, the display control unit 106 displays the display information and supplementary information on a display or the like.

[0037] An example of the display information and supplementary information according to the second embodiment will be explained with reference to Figure 8. In the example shown in Figure 8, supplementary information 81 is displayed in addition to the display information 80. Here, we assume an example where the display information 80 and supplementary information 81 are displayed side by side, but the supplementary information 81 may be displayed on the screen only after a user prompts it to be displayed. That is, for example, a button for displaying the supplementary information may be provided in the area of ​​the display information, and the supplementary information 81 may be displayed when the user clicks this button, or when a specific button on the keyboard is assigned a function to display the supplementary information 81 and that specific button is pressed.

[0038] According to the second embodiment described above, by selecting items from the display information definition, information that is no longer displayed is grouped together and displayed as supplementary information. This supports the visualization of appropriate quality evaluation, and even if the information requested by the user is not displayed due to analysis by the analysis unit, the user can still find the necessary information through the supplementary information.

[0039] (Third embodiment) In the third embodiment, the difference from the above-described embodiments is that if there are items in the user input information that need to be corrected, feedback is provided to the user so that the user can make the corrections.

[0040] The information processing device 10 according to the third embodiment will be described with reference to the block diagram in Figure 9. The information processing device 10 includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, a display control unit 106, and a user input correction unit 301.

[0041] The user input correction unit 301 corrects the user input information based on the analysis processing performed by the analysis unit 104.

[0042] Next, an example of the operation of the information processing device 10 according to the third embodiment will be explained with reference to the flowchart in Figure 10. Steps SA1 through SA5 involve the same process.

[0043] In step SC1, the user input correction unit 301 corrects the user input information according to the analysis based on quantitative information and user input information. For example, consider a case where information is obtained manually from the user using a tabular checklist. If the checklist shows the accuracy rate of the trained model as "A%", but the accuracy rate of the machine learning model based on quantitative information is "B%", it is considered that the user made an input error.

[0044] Therefore, the user input correction unit 301 rewrites the correct answer rate item in the checklist from A% to B%. When the rewriting process is performed, the rewritten item may be highlighted in red in the checklist, or in some other way, to make it easier for the user to understand what has been corrected the next time they use it.

[0045] In step SC2, the display control unit 106 displays the display information and the modified user input information on a display or the like.

[0046] The user input correction unit 301 may update the checklist in accordance with the analysis of quantitative information and user input information. An example of the operation of the information processing device 10 when updating the checklist will be explained with reference to the flowchart in Figure 11. Steps SA1 through SA5 involve the same process.

[0047] In step SD1, the user input correction unit 301 determines whether the analysis results from the analysis unit 104 meet predetermined criteria for the user input information. For example, if "MNIST" is entered for the input item "What dataset was used?", and the quantitative information also shows that the dataset information is "MNIST", then it is considered that the correct information has been entered, and the analysis results will be determined to meet the predetermined criteria. If the analysis results meet the predetermined criteria, the process proceeds to step SD2; otherwise, the process proceeds to step SD3.

[0048] In step SD2, the user input modification unit 301 updates the user input checklist so that items that meet predetermined criteria are distinguished. One method of distinction is, for example, in the "MNIST" example mentioned above, to distinguish and display items that meet the predetermined criteria so that they do not need to be entered in subsequent checklist entries. Alternatively, if an item that did not need to be entered this time meets the criteria to become an item that requires input next time, it will be distinguished and displayed as an item that requires input. In this way, the user input modification unit 301 only needs to modify whether user input is required for each item in the checklist.

[0049] In step SD3, the user input correction unit 301 updates the user input checklist so that items that do not meet the predetermined criteria are clearly indicated. Then, the process proceeds to step SC2, where the displayed information and the corrected user input information are displayed.

[0050] Next, an example of modifying the checklist when the predetermined criteria are met in Step SD2 will be explained with reference to Figure 12. The upper part of Figure 12 shows the checklist before correction by the user input correction unit 301, where all fields are required. The lower part of Figure 12 shows the checklist after update (correction) by the user input correction unit 301, assuming that there are items that meet the predetermined criteria. Here, items that meet the predetermined criteria are displayed in gray 1201 and treated as not requiring input. On the other hand, the checklist should be updated so that only items that require input are editable. The user can see at a glance which items need to be filled in.

[0051] For example, consider a checklist with different fields to fill in depending on the development phase. Specifically, in the example shown in Figure 12, for the "PoC" development phase, the fields to fill in are three items from 3 to 5: "Item 3: Will the dataset be updated?", "Item 4: Who will perform the update work?", and "Item 5: How often will it be updated?". On the other hand, for the "Development" development phase, the fields to fill in are eight items from 1 to 8. As shown in Figure 12, the user input correction unit 301 determines that the necessary information has been filled in for the current development phase, and updates the checklist so that it does not need to be filled in for the next development phase. This reduces the burden on the user in terms of review and input.

[0052] Next, we will explain an example of modifying the checklist in Step SD3 when the predetermined criteria are not met, with reference to Figure 13. The upper part of Figure 13 shows the checklist before correction by the user input correction unit 301, and, as with the upper part of Figure 12, assumes that all inputs are mandatory. The lower part of Figure 13 shows the checklist after updating (correction) by the user input correction unit 301. Here, there is an example where there are items that do not meet the predetermined criteria, and the user input correction unit 301 extracts and displays the items that need to be entered as List 1301, which are listed as uncompleted items. Of course, the user input correction unit 301 may also highlight the items that need to be entered in the checklist, making them red, or it may display a message such as "Please check as the criteria are not met."

[0053] According to the third embodiment described above, the user input correction unit can provide information feedback to the user by correcting user information according to the analysis results of the analysis unit. Furthermore, if the user input correction unit finds items that meet predetermined criteria based on the analysis results of the analysis unit, it updates the user input checklist so that those items do not need to be entered when the checklist is used in the future. On the other hand, if there are items that do not meet the predetermined criteria, the user input checklist is updated to clearly indicate those items. This supports the visualization of appropriate quality evaluation, provides the user with accurate information feedback, and simplifies subsequent input, thereby reducing the effort required from the user.

[0054] (Fourth Embodiment) In the fourth embodiment, the user is prompted to confirm information that requires user verification, which is a key difference from the embodiments described above. The information processing device 10 according to the fourth embodiment will be described with reference to the block diagram in Figure 14. The information processing device 10 includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, a display control unit 106, and a user confirmation unit 401.

[0055] The analysis unit 104 analyzes quantitative information and user input information to determine whether or not there is any information that requires user confirmation. Based on the analysis performed by the analysis unit 104, the user verification unit 401 presents the user with information that it has determined requires user verification and obtains feedback from the user. The display information generation unit 105 generates display information, including user feedback.

[0056] Next, the operation of the information processing device 10 according to the fourth embodiment will be explained with reference to the flowchart in Figure 15. Steps SA1 through SA4 and SA6 involve the same process.

[0057] In step SE1, the analysis unit 104 analyzes the information based on quantitative information, user input information, and display information definitions, and determines whether or not there is information that requires user confirmation. Information that requires user confirmation is, for example, t-SNE information, and it is difficult for the information processing device to determine that "there is no difference between the training data and the actual operational data" based on the distribution of t-SNE. Information that is difficult for the device to determine, or that can be determined by the user more quickly and accurately than by the device, is considered information that requires user confirmation. The items of information that require user confirmation are listed in advance, and if the analysis unit 104 selects an item from this list as an item to be included in the display information as a result of its analysis, it will be determined that there is information that requires user confirmation.

[0058] If the displayed information includes items that require user confirmation, proceed to step SE2. If the displayed information does not include items that require user confirmation, proceed to step SA5.

[0059] In step SE2, the user verification unit 401 displays information that requires user verification on a display or the like via the display control unit 106. For user verification, the relevant information can be illustrated, and the user can be asked to input a Yes / No question regarding whether there are any inconsistencies in the content of the illustration or the text explaining the content of the illustration.

[0060] In step SE3, the user verification unit 401 obtains user feedback regarding information that requires user verification. Specifically, if the information is in a Yes / No format as described above, it only needs to accept "Yes" or "No" input from the user. In addition to "Yes" or "No" input, the user verification unit 401 may also obtain feedback in a free-text format.

[0061] In step SA5, if the user confirmation unit 401 receives the input "Yes," the display information generation unit 105 should generate display information using the content of the item as is, assuming that there is no problem with the item's content. On the other hand, if the user confirmation unit 401 receives the input "No," the unit should generate display information that shows the correction information after the feedback, or display information that highlights the problem with the relevant information so that it can be understood.

[0062] According to the fourth embodiment described above, based on quantitative information and user input information, it is determined whether or not there is information that requires user confirmation as an item of display information. If there is information that requires user confirmation, feedback from the user regarding that information is obtained. Subsequently, display information that reflects the content of the feedback is generated. This enables the visualization of appropriate quality evaluations while providing appropriate model quality evaluations that reflect user feedback.

[0063] (Fifth embodiment) The fifth embodiment differs from the above-described embodiments in that it compares the generated display information with past information. The information processing device 10 according to the fifth embodiment will be described with reference to the block diagram in Figure 16. The information processing device 10 includes a storage unit 101, a quantitative information acquisition unit 102, a user input acquisition unit 103, an analysis unit 104, a display information generation unit 105, a display control unit 106, and an information comparison unit 501.

[0064] The storage unit 101 stores multiple past information 505, which are display information generated in the past. The past information 505 can be stored in the storage unit 101 whenever the display information generation unit 105 generates display information. The information comparison unit 501 compares the current information with the past information 505 stored in the storage unit 101 and generates a comparison result.

[0065] Next, the operation of the information processing device 10 according to the fifth embodiment will be explained with reference to the flowchart in Figure 17. Steps SA1 through SA5 involve the same process.

[0066] In step SF1, the information comparison unit 501 compares the display information generated in step SA5 with the historical information 505 stored in the storage unit 101. For example, based on search conditions entered by the user, it searches for historical information whose items match the display information generated in step SA5. Alternatively, the information comparison unit 501 may set default search conditions according to the type of data, the task of the machine learning model, etc., and extract historical information whose items match the generated display information according to these default search conditions. Or, not limited to based on search conditions, it may extract historical information where the similarity of items or content between the display information and the historical information is above a threshold. In step SF2, the display control unit 106 displays the comparison results (search results) obtained in step SF1 along with the display information on a display or the like.

[0067] Next, an example of the comparison results according to the fifth embodiment will be explained with reference to Figure 18. Figure 18 is a conceptual diagram showing an example of a method for generating comparison results. The information comparison unit 501 searches for past information 1802 using one or more items of the generated display information 1801 as search conditions. Specifically, let's assume that in the development phase "Development," model type "Model A" is trained using data type "Data C," and display information 1801 ("Development / Data C / Model A") is generated. When the search conditions are development phase "Development" and model type "Model A," the information comparison unit 501 extracts "Development / Data A / Model A" and "Development / Data B / Model A" as the corresponding past information 1802.

[0068] Finally, as comparison result 1803, display information 1801 "Development / Data C / Model A" and the two historical information items 1802 "Development / Data A / Model A" and "Development / Data B / Model A" are displayed side by side in a comparable manner. In comparison result 1803, common or different parts between display information 1801 and historical information 1802 may be highlighted.

[0069] Furthermore, when historical information 1802 is stored in the storage unit 101, tags related to items or keywords may be linked to the historical information 1802 and stored accordingly. For example, development phase (development, PoC, etc.), model performance (accuracy rate, F-score, etc.), data used, algorithm used, etc., can be linked to the display information as tags. In subsequent analysis, historical information to be compared can be searched and extracted based on the tags. The user may arbitrarily decide which tags to use for comparison, or the information comparison unit 501 may automatically select tags based on past comparison results and the items of the generated display information used as the comparison source.

[0070] According to the fifth embodiment described above, previously generated display information is stored in the storage unit as past information, the information comparison unit searches for past information that has items in common with the display information, and displays the display information and the past information side by side as a comparison result. This allows for the visualization of appropriate quality assessments while efficiently supporting user decision-making based on displayed information and historical data, such as how differences in training data affect model performance.

[0071] Next, an example of the hardware configuration of the information processing device 10 according to the first to fifth embodiments described above is shown in the block diagram of Figure 19. The information processing device 10 includes a CPU (Central Processing Unit) 1901, RAM (Random Access Memory) 1902, ROM (Read Only Memory) 1903, storage 1904, display device 1905, input device 1906, and communication device 1907, all of which are connected by a bus.

[0072] The CPU 1901 is a processor that performs arithmetic and control processing according to a program. The CPU 1901 uses a predetermined area of ​​the RAM 1902 as a working area and, in cooperation with programs stored in the ROM 1903 and storage 1904, executes the processing of each part of the information processing device 10 described above. Note that each processing of the information processing device 10 may be executed by a single processor or may be executed in a distributed manner by multiple processors.

[0073] RAM1902 is a type of memory such as SDRAM (Synchronous Dynamic Random Access Memory). RAM1902 functions as a workspace for CPU1901. ROM1903 is a memory that stores programs and various information in a non-rewritable format.

[0074] Storage 1904 is a device that writes and reads data from magnetic recording media such as HDDs (Hard Disk Drives), semiconductor storage media such as flash memory, or magnetically recordable storage media such as HDDs, or optically recordable storage media. Storage 1904 writes and reads data from the storage media in response to control from CPU 1901.

[0075] The display device 1905 is a display device such as an LCD (Liquid Crystal Display). The display device 1905 displays various information based on display signals from the CPU 1901.

[0076] Input device 1906 is an input device such as a mouse and keyboard. Input device 1906 receives information input by the user as an instruction signal and outputs the instruction signal to CPU 1901.

[0077] The communication device 1907 communicates with external devices via a network in response to control from the CPU 1901.

[0078] The instructions shown in the processing procedure described in the above-described embodiment can be executed based on a software program. A general-purpose computer system can store this program in advance and, by reading this program, can obtain effects similar to those of the control operation of the information processing device described above. The instructions described in the above-described embodiment are recorded as a program that can be executed by a computer on a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray® Disc, etc.), semiconductor memory, or similar recording medium. Any storage format is acceptable as long as it is a recording medium that can be read by a computer or embedded system. The computer can read the program from this recording medium and, based on this program, have the CPU execute the instructions described in the program, thereby achieving operations similar to the control of the information processing device in the above-described embodiment. Of course, when the computer acquires or reads the program, it may do so via a network. Furthermore, an operating system (OS) running on a computer, a database management software, a network, or other middleware (MW) operating on a computer, based on instructions from a program installed on a computer or embedded system from a recording medium, may execute some of the processes necessary to realize this embodiment. Furthermore, the recording medium in this embodiment is not limited to a medium independent of the computer or embedded system, but also includes a recording medium that stores or temporarily stores a program that has been downloaded via a LAN, the Internet, or the like. Furthermore, the recording medium is not limited to one; even when the processing in this embodiment is performed from multiple media, these are also included as recording media in this embodiment, and the configuration of the media may be any configuration.

[0079] In this embodiment, the computer or embedded system is used to execute each process in this embodiment based on a program stored on a recording medium, and may be configured as any of the following: a single device such as a personal computer or microcontroller, or a system in which multiple devices are connected via a network. Furthermore, the term "computer" in this embodiment is not limited to personal computers, but also includes arithmetic processing units, microcontrollers, and the like included in information processing equipment, and refers collectively to any equipment or device capable of realizing the functions of this embodiment through a program.

[0080] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0081] 10...Information processing device, 41, 51...User input information, 42...Quantitative information, 80, 1801...Display information, 81...Supplementary information, 101...Storage unit, 102...Quantitative information acquisition unit, 103...User input acquisition unit, 104...Analysis unit, 105...Display information generation unit, 106...Display control unit, 107...Training unit, 201...Supplementary information generation unit, 301...User input correction unit, 401...User confirmation unit, 501...Information comparison unit, 505, 1802...Past information, 1201...Grayed out, 1301...List, 1901...CPU, 1902...RAM, 1903...ROM, 1904...Storage, 1905...Display device, 1906...Input device, 1907...Communication device.

Claims

1. A first acquisition unit that acquires quantitative information about the machine learning model, A second acquisition unit that acquires user input information entered by the user, Based on the quantitative information and the user input information, an analysis unit selects items to be displayed and a visualization method from a display information definition that includes items related to the evaluation of the machine learning model and a method for visualizing the evaluation. A first generation unit generates display information that includes at least a portion of the quantitative information and user input information based on the selection results of the analysis unit, It is equipped with, The analysis unit is an information processing device that sets the importance of items included in the display information definition according to the user's attribute information and the development phase in which the machine learning model is used when the quantitative information is obtained.

2. The information processing apparatus according to claim 1, further comprising a second generation unit that generates items not included in the aforementioned display information as supplementary information.

3. If, as a result of the analysis based on the quantitative information and the user input information, the user input information needs to be corrected, the correction unit corrects the user input information. The information processing apparatus according to claim 1, further comprising: a display control unit that displays the modified user input information to the user.

4. The information processing apparatus according to claim 1, further comprising a modification unit that modifies whether user input is required for items of a checklist for acquiring user input information, depending on the next phase in which the checklist for acquiring user input information is used, if the result of the analysis based on the quantitative information and the user input information is determined to meet a predetermined standard.

5. The information processing apparatus according to claim 1, further comprising a modification unit that, if there are items that are determined not to meet a predetermined standard based on the results of an analysis of the quantitative information and the user input information, extracts the relevant items from a checklist for acquiring the user input information, or highlights the relevant items in the checklist.

6. It is further equipped with a user verification unit, The analysis unit determines, based on the quantitative information and the user input information, whether or not there is information that requires confirmation by the user. The information processing apparatus according to claim 1, wherein the user verification unit, if there is information that requires verification by the user, presents such information to the user and obtains feedback from the user.

7. A comparison unit that extracts past information that matches the search criteria based on newly generated display information from multiple past information that is display information generated in the past, An information processing apparatus according to claim 1, further comprising: a display control unit that displays the extracted past information and the display information to the user as a comparison result;

8. The first acquisition unit acquires quantitative information related to the machine learning model. The second acquisition unit acquires user input information entered by the user, The analysis unit selects items to be displayed and visualization methods from a display information definition that includes items related to the evaluation of the machine learning model and methods for visualizing the evaluation, based on the quantitative information and the user input information. The first generation unit generates display information that includes at least a portion of the quantitative information and the user input information based on the selection result of the analysis unit. The analysis unit is an information processing method that sets the importance of items included in the display information definition according to the user's attribute information and the development phase in which the machine learning model is used when the quantitative information is obtained.

9. Computers, A first acquisition means for obtaining quantitative information about a machine learning model, A second acquisition means for acquiring user input information entered by the user, An analysis means that, based on the quantitative information and the user input information, selects items to be displayed and a visualization method from a display information definition that includes items related to the evaluation of the machine learning model and a method for visualizing the evaluation. An information processing program to function as a first generation means that generates display information including at least a portion of the content of the quantitative information and the user input information based on the selection result of the analysis means, The analysis means is an information processing program that sets the importance of items included in the display information definition according to the user's attribute information and the development phase in which the machine learning model is used when the quantitative information is obtained.

Images