Computer system and model evaluation method

Abstract

To support evaluation of a model.SOLUTION: A computer system is connected so as to be accessible to model management information for managing model data, risk assessment management information for managing risk assessment data, and evaluation method management information for managing evaluation method data, and generates association between the model data, the risk assessment data, and the evaluation method data constituting a template as relation data. When receiving an evaluation request, the computer system refers to the model management information to retrieve the model data of a model to be evaluated, retrieves relation data associated with the model data, generates a template based on the relation data, stores an evaluation result based on an evaluation method corresponding to evaluation method data associated with the relation data in association with the relation data, and generates a report based on the template and the evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for supporting model evaluation. [Background technology]

[0002] In managing models used in AI systems, it is important to evaluate them from various perspectives. For this purpose, the technology described in Patent Document 1 is known.

[0003] Patent Document 1 describes generating information regarding the evaluation of AI based on a template that defines evaluation items. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Pat. No. 1,126,3188 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional technology, the evaluation viewpoint and the evaluation method are preset. However, the evaluation viewpoint and the evaluation method change. For example, in the development stage of a model, the evaluation viewpoint and the evaluation method change frequently. Therefore, it is necessary to generate a template according to the above-mentioned changes. When generating a template manually, the generation cost becomes an issue. In addition, it is necessary to manage the template and the evaluation result in association with each other, and the management cost becomes an issue.

[0006] An object of the present invention is to provide a model evaluation technique that reduces the costs of generating templates and managing evaluation results. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, a computer system includes a computer having a processor, a storage device connected to the processor, and a network interface connected to the processor, and is connected to model management information for managing model data including items related to a model, risk assessment management information for managing risk assessment data including items related to an evaluation viewpoint of the model, and evaluation method management information for managing evaluation method data including items related to an evaluation method, the computer system constitutes a template defining the evaluation content of a model, generates associations between the model data, the risk assessment data, and the evaluation method data as relationship data and registers the associations in the template management information, and when an evaluation request including information about a model to be evaluated is received, searches for the model data of the model to be evaluated by referring to the model management information, searches for the relationship data to which the searched model data is associated, generates the template based on the searched relationship data, stores an evaluation result based on the evaluation method corresponding to the evaluation method data associated with the searched relationship data in association with the relationship data, and generates a report based on the generated template and the evaluation result. Effect of the Invention

[0008] According to the present invention, it is possible to realize a model evaluation technique that reduces the costs of generating templates and managing evaluation results. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a system configuration according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a computer that configures the model evaluation support system of the first embodiment. [Diagram 3] FIG. 11 is a diagram illustrating an example of model data stored in model management information according to the first embodiment. [Figure 4A] 1 is a diagram showing an example of risk assessment data stored in risk assessment management information according to the first embodiment; FIG. [Figure 4B] 1 is a diagram showing an example of risk assessment data stored in risk assessment management information according to the first embodiment; FIG. [Figure 4C] 1 is a diagram showing an example of risk assessment data stored in risk assessment management information according to the first embodiment; FIG. [Figure 5A] FIG. 11 is a diagram showing an example of evaluation method data stored in the evaluation method management information according to the first embodiment. [Figure 5B] FIG. 11 is a diagram showing an example of evaluation method data stored in the evaluation method management information according to the first embodiment. [Figure 5C] FIG. 11 is a diagram showing an example of evaluation method data stored in the evaluation method management information according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of template management information according to the first embodiment. [Figure 7] FIG. 13 is a diagram illustrating an image of a template management method according to the first embodiment. [Figure 8] FIG. 11 is a diagram showing an example of evaluation results stored in evaluation result management information according to the first embodiment. [Figure 9] 11 is a flowchart illustrating an example of a template registration process executed by the model evaluation support system according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of a screen presented by the model evaluation support system of the first embodiment. [Figure 11] 11 is a flowchart illustrating an example of an evaluation report generation process executed by the model evaluation support system of the first embodiment. [Figure 12] 11 is a flowchart illustrating another example of the evaluation report generation process executed by the model evaluation support system according to the first embodiment. [Figure 13] 11 is a flowchart illustrating an example of a comparative evaluation report generation process executed by the model evaluation support system of the first embodiment. [Figure 14]11 is a flowchart illustrating an example of an evaluation viewpoint recommendation process executed by the model evaluation support system of the first embodiment. [Figure 15] 11 is a flowchart illustrating an example of an evaluation method recommendation process executed by the model evaluation support system of the first embodiment. [Figure 16] 11 is a flowchart illustrating an example of an improvement method recommendation process executed by the model evaluation support system of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not to be interpreted as being limited to the description of the embodiment shown below. It will be easily understood by those skilled in the art that the specific configuration can be changed without departing from the concept or purpose of the present invention.

[0011] In the configurations of the invention described below, the same or similar configurations or functions are given the same reference numerals, and duplicated explanations are omitted.

[0012] In this specification, the terms "first," "second," "third," and the like are used to identify components and do not necessarily limit the number or order. EXAMPLES

[0013] Fig. 1 is a diagram illustrating an example of the configuration of a system according to the first embodiment. Fig. 2 is a diagram illustrating the hardware configuration of a computer that constitutes a model evaluation support system 100 according to the first embodiment.

[0014] The system is composed of a model evaluation support system 100, a plurality of model management systems 101, and a plurality of user terminals 102. The model evaluation support system 100 and the user terminals 102 are connected via a network not shown. The model evaluation support system 100 and the model management system 101 are connected via a network not shown. The network may be a WAN (Wide Area Network) or a LAN (Local Area Network), and the connection method may be either wired or wireless. Note that the present invention is not limited to the number of model management systems 101 and user terminals 102.

[0015] The model management system 101 is a system operated by the provider or developer of the model. The model management system 101 includes a system that manages models, data used to generate the models (learning data), programs used to generate the models, data used to verify the models (verification data), programs used to verify the models, and verification results. The model management system 101 also includes a system that performs business operations using models.

[0016] The user terminal 102 is a terminal operated by a user who uses the model. The terminal has a processor, a memory, a network interface, an input device, and an output device, which are not shown. The input device is a keyboard, a mouse, a touch panel, etc. The output device is a display, etc.

[0017] The model evaluation support system 100 is a system that supports the generation of templates and the generation of evaluation reports of models based on the templates, etc. A template is data that defines the evaluation content of a model, and includes information on the model to be evaluated, the evaluation viewpoint, the evaluation method, the evaluation criteria, the evaluation index, etc.

[0018] The model evaluation support system 100 is configured, for example, with a computer 200 as shown in Fig. 2. The computer 200 has a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204. The hardware elements are connected to each other via a bus. The model evaluation support system 100 may also have an input device and an output device.

[0019] The processor 201 executes a program stored in the main memory device 202. The processor 201 executes processing according to the program, thereby operating as a functional unit (module) that realizes a specific function. In the following explanation, when the processing is explained with the functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit.

[0020] The primary storage device 202 is a memory or the like, and stores the programs executed by the processor 201 and information used by the programs. The primary storage device 202 is also used as a work area used temporarily by the programs. The secondary storage device 203 is a hard disk drive (HDD) or solid state drive (Solid State Drive) or the like, and stores data permanently. The network interface 204 is an interface for communicating with external devices via a network.

[0021] The model evaluation support system 100 holds model management information 130, risk assessment management information 131, evaluation method management information 132, evaluation result management information 133, and template management information 134. The model evaluation support system 100 also has a data management unit 120, a report generation unit 121, and a recommendation unit 122.

[0022] Model management information 130 is information for managing model data, which is data related to models. Risk assessment management information 131 is information for managing risk assessment data, which is data related to evaluation viewpoints. Evaluation method management information 132 is information for managing evaluation method data, which is data related to evaluation methods. Evaluation result management information 133 is information for managing evaluation result data, which is data including evaluation results.

[0023] The model management information 130 , the risk assessment management information 131 , and the evaluation method management information 132 may be held by the model management system 101 .

[0024] The template management information 134 is information for managing relationship data indicating the relationship between the model data, risk assessment data, and evaluation method data that constitute a template. Here, a template is data that defines the evaluation viewpoint and evaluation method of a model.

[0025] The data management unit 120 manages input and output of data. The report generation unit 121 generates a report regarding the evaluation of the model using a template. The recommendation unit 122 makes a recommendation using the template.

[0026] FIG. 3 is a diagram illustrating an example of model data stored in the model management information 130 according to the first embodiment.

[0027] The model data 300 includes items such as ID, version, release date, purpose, input, output, learning data, and improvement method. Note that the above items are merely examples and are not limited to these. For example, learning parameters may be included as an item.

[0028] "ID" is an item for managing identification information of the model data 300. "Version" is an item for managing the version of the model, which is an example of identification information of the model. "Release date" is an item for managing the release date of the model. "Purpose" is an item for managing the purpose of the model. "Input" is an item for managing data input to the model. "Output" is an item for managing the output of the model. "Learning data" is an item for managing information related to the learning data used to train the model. In "Learning data", for example, the type and number of learning data are managed. "Improvement method" is an item for managing a method for improving the model. "Improvement method" may be left blank.

[0029] 4A, 4B, and 4C are diagrams showing an example of risk assessment data stored in the risk assessment management information 131 according to the first embodiment.

[0030] The risk assessment data 400-1, 400-2, and 400-3 each include, as items, an ID, a risk, and a countermeasure. Note that the above-mentioned items are merely examples and are not limiting.

[0031] "ID" is an item for managing identification information of risk assessment data. "Risk" is an item for managing information related to evaluation perspectives. "Countermeasures" is an item for managing information related to countermeasures in the case where the evaluation results of the evaluation perspectives do not satisfy the specified conditions.

[0032] 5A, 5B, and 5C are diagrams showing an example of the evaluation method data stored in the evaluation method management information 132 according to the first embodiment.

[0033] The evaluation method data 500-1, 500-2, and 500-3 include, as items, an ID, an evaluation target, an evaluation method, a criterion, and a level of importance. Note that the above-mentioned items are merely examples and are not limited to these.

[0034] "ID" is an item that manages identification information of the evaluation method data 500. "Evaluation target" is an item that manages information related to the evaluation target. "Evaluation method" is an item that stores information related to the evaluation method. "Judgment criteria" is an item that manages information related to the criteria for judging pass / fail in an evaluation based on the evaluation method. "Importance" is an item that manages the importance of the evaluation method.

[0035] Fig. 6 is a diagram illustrating an example of the template management information 134 according to the first embodiment. Fig. 7 is a diagram illustrating an image of a template management method according to the first embodiment.

[0036] The template management information 134 stores entries each consisting of a template ID 601, a model ID 602, a risk assessment ID 603, an evaluation method ID 604, and a registration date 605. One entry corresponds to one relationship data. One template is generated from one relationship data. Note that the fields included in the entry are merely examples and are not limited to these.

[0037] The template ID 601 is a field for storing identification information of relationship data. Since there is a one-to-one relationship between relationship data and a template, the identification information of the relationship data is used as the identification information of the template.

[0038] The model ID 602 ​​is a field for storing identification information of the model data 300 corresponding to the model to be evaluated. The risk assessment ID 603 is a field for storing identification information of the risk assessment data 400 corresponding to the evaluation viewpoint of the model. The evaluation method ID 604 is a field for storing identification information of the evaluation method data 500 corresponding to the evaluation method for evaluating the evaluation viewpoint. The registration date 605 is a field for storing the registration date of the relationship data.

[0039] Relation data is information for managing connection relationships as shown in Fig. 7. In this embodiment, since there is no need to manage the templates themselves, the amount of data can be reduced. In addition, by setting the connection relationships of each data, it becomes easy to generate and update templates, so the cost of generating templates can be reduced.

[0040] FIG. 8 is a diagram showing an example of the evaluation result stored in the evaluation result management information 133 according to the first embodiment.

[0041] The evaluation result data 800 includes items such as an ID, a template ID, an evaluation method ID, an evaluation date, and an evaluation result. Note that the above-mentioned items are merely examples and are not limiting.

[0042] "ID" is an item for managing identification information of evaluation result data 800. "Template ID" is an item for managing identification information of the template (relationship data) used in the evaluation. "Evaluation method ID" is an item for managing identification information of evaluation method data 500. "Evaluation date" is an item for managing the date and time when the evaluation was performed. "Evaluation result" is an item for managing the evaluation result.

[0043] By including "template ID", it is possible to know which template the evaluation result is based on. Also, by including "assessment method ID", it is possible to know which evaluation method of the template the evaluation result is based on.

[0044] The model evaluation support system 100 accepts registration of model data 300, risk assessment data 400, and evaluation method data 500 from the model management system 101. In this embodiment, evaluation viewpoints and evaluation methods can be added and changed at any time.

[0045] Fig. 9 is a flowchart illustrating an example of a template registration process executed by the model evaluation support system 100 according to the first embodiment. Fig. 10 is a diagram illustrating an example of a screen presented by the model evaluation support system 100 according to the first embodiment.

[0046] The model evaluation support system 100 executes the process described below when receiving a template registration request from the model management system 101 or the user terminal 102. Note that the template registration process can be executed at any timing.

[0047] The data management unit 120 presents a screen 1000 as shown in FIG. 10 (step S101), and accepts user input (step S102).

[0048] The screen 1000 includes a model selection area 1001 , a risk assessment selection area 1002 , an assessment method selection area 1003 , and a registration button 1004 .

[0049] The model selection area 1001 is an area for selecting a model to which the template is applied, and includes a selection column 1011 and a display button 1012.

[0050] The selection field 1011 is a field for selecting a model. Identification information of the model data 300 is displayed in a pull-down format in the selection field 1011. The display button 1012 is an operation button for displaying the model data 300 selected in the selection field 1011. When the display button 1012 is operated, the data management unit 120 obtains and displays the selected model data 300 from the model management information 130. The model data 300 may be displayed within the screen 1000, or may be displayed as a separate screen.

[0051] The risk assessment selection area 1002 is an area for selecting the assessment perspective to be included in the template, that is, the risk assessment data 400, and includes a selection column 1021, a display button 1022, and an add button 1023.

[0052] The selection field 1021 is a field for selecting an evaluation viewpoint. Identification information of the risk assessment data 400 is displayed in a pull-down format in the selection field 1021. The display button 1022 is an operation button for displaying the risk assessment data 400 selected in the selection field 1021. When the display button 1022 is operated, the data management unit 120 obtains and displays the selected risk assessment data 400 from the risk assessment management information 131. The risk assessment data 400 may be displayed within the screen 1000, or may be displayed as a separate screen.

[0053] The add button 1023 is an operation button for adding a selection field 1021 for setting a new evaluation viewpoint and a display button 1022. When the add button 1023 is operated, the data management unit 120 adds the selection field 1021 and the display button 1022 to the risk assessment selection area 1002.

[0054] The evaluation method selection area 1003 is an area for selecting the evaluation methods to be included in the template, i.e., the evaluation method data 500, and includes selection boxes 1030. In the evaluation method selection area 1003, the same number of selection boxes 1030 as the number of evaluation perspectives selected in the risk assessment selection area 1002 are displayed.

[0055] The selection box 1030 is an area for selecting an evaluation method for one evaluation aspect, and includes a selection field 1031 , a display button 1032 , and an add button 1033 .

[0056] The selection field 1031 is a field for selecting an evaluation method. Identification information of the evaluation method data 500 is displayed in a pull-down format in the selection field 1031. The display button 1032 is an operation button for displaying the evaluation method data 500 selected in the selection field 1031. When the display button 1032 is operated, the data management unit 120 obtains and displays the evaluation method data 500 selected from the evaluation method management information 132. The evaluation method data 500 may be displayed within the screen 1000, or may be displayed as a separate screen.

[0057] The add button 1033 is an operation button for adding a selection field 1031 for setting a new evaluation method and a display button 1032. When the add button 1033 is operated, the data management unit 120 adds the selection field 1031 and the display button 1032 to the selection box 1030.

[0058] A registration button 1004 is an operation button for instructing to register a template. When the registration button 1004 is operated, a user input including information on each data selected in the model selection area 1001, the risk assessment selection area 1002, and the evaluation method selection area 1003 is transmitted to the model evaluation support system 100.

[0059] The data management unit 120 updates the template management information 134 based on the user input (step S103).

[0060] Specifically, the data management unit 120 adds an entry to the template management information 134, and sets identification information in the template ID 601 of the added entry. The data management unit 120 sets identification information of the model data 300 selected in the model selection area 1001 in the model ID 602 ​​of the added entry. The data management unit 120 adds rows to the risk assessment ID 603 of the added entry, the same number of rows as the risk assessment data 400 selected in the risk assessment selection area 1002, and sets identification information of the selected risk assessment data 400 in each row. The data management unit 120 adds rows to the rows of the risk assessment data 400, the same number of rows as the evaluation method data 500 selected in the selection box 1030, and sets identification information of the selected evaluation method data 500 in each row. The current date and time is set in the registration date 605.

[0061] FIG. 11 is a flowchart illustrating an example of the evaluation report generation process executed by the model evaluation support system 100 according to the first embodiment.

[0062] The model evaluation support system 100 receives an evaluation request including information for identifying a model to be evaluated (step S201). The information for identifying the model is, for example, identification information of the model data 300 or the version of the model.

[0063] The report generating unit 121 identifies a model to be evaluated based on the information included in the evaluation request (step S202).

[0064] Specifically, the report generating unit 121 refers to the model management information 130 and searches for the model data 300 that includes item values ​​that correspond to the information included in the evaluation request.

[0065] The report generating unit 121 generates a template to be applied to the identified model based on the template management information 134 (step S203).

[0066] Specifically, the report generation unit 121 searches for an entry in which the identification information of the model data 300 identified in step S202 is stored in the model ID 602. The report generation unit 121 acquires data from each of the model management information 130, the risk assessment management information 131, and the evaluation method management information 132 based on the searched entry, and generates a template using the acquired data.

[0067] The report generation unit 121 transmits an evaluation request including the model data 300 and the evaluation method data 500 to the model management system 101 that performs model evaluation (step S204). The model management system 101 that receives the request executes model evaluation processing based on the model data 300 and the evaluation method data 500, and transmits identification information of the evaluation method data 500 and the evaluation result to the model evaluation support system 100.

[0068] The evaluation process involves calculation of an evaluation index, and judgment based on the evaluation index, etc. The judgment may include text indicating whether or not the evaluation criteria are met.

[0069] When the report generation unit 121 receives the evaluation result from the model management system 101, it instructs the data management unit 120 to register the evaluation result data 800. The instruction includes the identification information of the template, the identification information of the evaluation method data 500, and the evaluation result. The data management unit 120 generates the evaluation result data 800 including the evaluation result, and registers it in the evaluation result management information 133 (step S205). For example, the current date and time is set in the "evaluation date" of the evaluation result data 800. The data management unit 120 outputs the evaluation result data 800 to the report generation unit 121.

[0070] The process of step S205 allows the templates and the evaluation results to be managed in association with each other. In this embodiment, the templates themselves are not stored, so that the data volume can be reduced.

[0071] After completing the evaluations corresponding to all the evaluation method data 500, the report generation unit 121 generates and outputs an evaluation report based on the template and the evaluation result data 800 (step S206). The evaluation report can be generated, for example, by embedding the evaluation results in the template.

[0072] FIG. 12 is a flowchart illustrating another example of the evaluation report generation process executed by the model evaluation support system 100 according to the first embodiment.

[0073] After the processes from step S201 to step S203 are executed, the report generation unit 121 calls the data management unit 120. The data management unit 120 presents a template (step S251) and provides an interface for correcting the template. The user can perform either a template correction operation or an evaluation continuation operation via the interface.

[0074] When the data management unit 120 receives a user operation via the interface (step S252), it determines whether or not the user operation is a template correction operation (step S253).

[0075] If the accepted user operation is a template correction operation, the data management unit 120 updates the template management information 134 based on the correction operation (step S254), and then returns to step S203.

[0076] If the accepted user operation is an operation to continue evaluation, the data management unit 120 calls the report generation unit 121. The report generation unit 121 executes the processes from step S204 to step S206.

[0077] In this way, templates can be generated interactively.

[0078] FIG. 13 is a flowchart illustrating an example of a comparative evaluation report generation process executed by the model evaluation support system 100 according to the first embodiment.

[0079] The model evaluation support system 100 receives a comparison evaluation request including identification information of two models to be compared (step S301).

[0080] The report generating unit 121 identifies two models based on the information included in the comparison evaluation request (step S302). The method of identifying the models is the same as that in step S202.

[0081] The report generating unit 121 selects a reference model from the two models (step S303). For example, the latest model is selected as the reference model. Note that the comparison evaluation request may include information on the model to be selected as the reference model. In the following, a model that is not the reference model will be referred to as a comparison model.

[0082] The report generating unit 121 generates a template to be applied to the reference model based on the template management information 134 (step S304). The process of step S304 is similar to the process of step S203.

[0083] The report generation unit 121 instructs the data management unit 120 to register a template of the comparison model. The instruction includes identification information of the template of the reference model and identification information of the model data 300 of the comparison model. The data management unit 120 updates the template management information 134 based on the instruction (step S305). The data management unit 120 outputs the identification information of the newly registered template to the report generation unit 121.

[0084] Specifically, the data management unit 120 adds an entry to the template management information 134, and sets the identification information of the template in the template ID 601 of the added entry. The data management unit 120 also sets the identification information of the model data 300 of the comparison model in the model ID. Furthermore, the data management unit 120 copies the contents of the reference model template to the risk assessment ID and evaluation method ID 604 of the added entry. The current date and time is set in the registration date 605.

[0085] The report generating unit 121 transmits an evaluation request including the model data 300 of the reference model and the evaluation method data 500 to the model management system 101 that evaluates the model (step S306).

[0086] When the report generation unit 121 receives the evaluation result from the model management system 101, it instructs the data management unit 120 to register the evaluation result data 800. The data management unit 120 generates evaluation result data 800 including the evaluation result, and registers it in the evaluation result management information 133 (step S307). The data management unit 120 outputs the evaluation result data 800 to the report generation unit 121.

[0087] The report generating unit 121 transmits an evaluation request including the model data 300 of the comparison model and the evaluation method data 500 to the model management system 101 that evaluates the model (step S308).

[0088] When the report generation unit 121 receives the evaluation result from the model management system 101, it instructs the data management unit 120 to register the evaluation result data 800. The data management unit 120 generates evaluation result data 800 including the evaluation result, and registers it in the evaluation result management information 133 (step S309). The data management unit 120 outputs the evaluation result data 800 to the report generation unit 121.

[0089] The report generation unit 121 generates and outputs a comparative evaluation report based on the templates of the reference model and the comparative model and the evaluation result data 800 (step S310). The comparative evaluation report includes evaluation indices of each model, etc. The comparative evaluation report includes information on changes in the evaluation indices, for example. The comparative evaluation report may also include comments on differences in the evaluation indices, etc.

[0090] FIG. 14 is a flowchart illustrating an example of an evaluation viewpoint recommendation process executed by the model evaluation support system 100 according to the first embodiment.

[0091] The model evaluation support system 100 receives an evaluation viewpoint recommendation request (step S401). The request includes information on the characteristics of the model. The information on the characteristics of the model is, for example, the use of the model.

[0092] The recommendation unit 122 refers to the model management information 130 and searches for the model data 300 of a model having the specified characteristics (step S402).

[0093] The recommendation unit 122 refers to the template management information 134 based on the identification information of the retrieved model data 300, and retrieves related relationship data (step S403).

[0094] The recommendation unit 122 generates and outputs recommendation information based on the risk assessment data 400 associated by the relation data (step S404). For example, the recommendation information may display the risk assessment data 400 itself.

[0095] In addition, there are cases where multiple relationship data are searched. In this case, the recommendation unit 122 may generate recommendation information based on the risk assessment data 400 of each relationship data. In addition, the recommendation unit 122 may analyze the appearance frequency of the risk assessment data 400 and generate recommendation information based on the risk assessment data 400 that appears frequently.

[0096] By referring to the recommendation information, the user can understand the evaluation viewpoints to focus on when evaluating the model.

[0097] FIG. 15 is a flowchart illustrating an example of an evaluation method recommendation process executed by the model evaluation support system 100 according to the first embodiment.

[0098] The model evaluation support system 100 receives an evaluation method recommendation request (step S501). The request includes information on evaluation viewpoints.

[0099] The recommendation unit 122 refers to the risk assessment management information 131 and searches for risk assessment data 400 corresponding to the specified evaluation viewpoint (step S502).

[0100] The recommendation unit 122 refers to the template management information 134 based on the identification information of the searched risk assessment data 400, and searches for related relationship data (step S503).

[0101] The recommendation unit 122 generates and outputs recommendation information based on the evaluation method data 500 associated by the relation data (step S504).

[0102] In addition, there are cases where multiple relationship data are searched. In this case, the recommendation unit 122 may generate recommendation information based on the evaluation method data 500 of each relationship data. In addition, the recommendation unit 122 may analyze the appearance frequency of the evaluation method data 500 and generate recommendation information based on the evaluation method data 500 that appears frequently.

[0103] By referring to the recommendation information, the user can understand the evaluation method to be adopted in evaluating the model.

[0104] FIG. 16 is a flowchart illustrating an example of an improvement method recommendation process executed by the model evaluation support system 100 according to the first embodiment.

[0105] The model evaluation support system 100 receives an improvement method recommendation request (step S601). The request includes information on the item to be improved. The information on the item to be improved is, for example, the name of the evaluation item and the target value of the evaluation item.

[0106] The recommendation unit 122 refers to the evaluation result management information 133 and searches for the evaluation result data 800 including the evaluation result of the item to be improved (step S602).

[0107] The recommendation unit 122 identifies related relationship data based on the evaluation result data 800 (step S603).

[0108] The recommendation unit 122 acquires the model data 300 from the model management information 130 based on the identified relationship data (step S604).

[0109] The recommendation unit 122 generates and outputs information on the improvement technique included in the model data 300 as recommendation information (step S605).

[0110] By referring to the recommendation information, users can understand methods for improving their models.

[0111] According to the present invention, it is possible to reduce the cost of generating templates and the cost of managing evaluation results. Furthermore, it is possible to present information on evaluation viewpoints, evaluation methods, and improvement methods using templates. This makes it possible to support the generation of templates and the learning of models.

[0112] The model evaluation support system 100 may have a function for performing model learning, model evaluation, etc. In this case, the model itself may be stored in the model management information 130. Furthermore, the model evaluation support system 100 holds learning data and validation data.

[0113] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment describes the configuration in detail to easily explain the present invention, and the present invention is not necessarily limited to the configuration including all the described configurations. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0114] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0115] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, Java (registered trademark), etc.

[0116] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0117] In the above-mentioned embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all the control lines and information lines are shown in the product. All the components may be connected to each other. [Explanation of symbols]

[0118] 100 Model evaluation support system 101 Model Management System 102 User terminal 120 Data Management Department 121 Report Generation Unit 122 Recommendation Department 130 Model Management Information 131 Risk Assessment Management Information 132 Evaluation method management information 133 Evaluation result management information 134 Template Management Information 200 calculator 201 Processor 202 Main storage 203 Secondary storage device 204 Network Interface 300 Model Data 400 Risk Assessment Data 500 Evaluation Method Data 800 Evaluation result data 1000 screens

Claims

1. 1. A computer system comprising: a computer having a processor, a storage device connected to the processor, and a network interface connected to the processor; connected in an accessible manner to model management information for managing model data including items related to models, risk assessment management information for managing risk assessment data including items related to evaluation viewpoints of models, and evaluation method management information for managing evaluation method data including items related to evaluation methods; generating relationship data that associates the model data, the risk assessment data, and the evaluation method data, which constitute a template that defines the evaluation content of the model, and registering the relationship data in template management information; When an evaluation request including information on a model to be evaluated is received, the model management information is referenced to search for the model data of the model to be evaluated; retrieving the relationship data to which the retrieved model data is related; generating the template based on the retrieved relationship data; storing an evaluation result based on the evaluation method corresponding to the evaluation technique data associated with the retrieved relationship data in association with the relationship data; A computer system that generates a report based on the generated template and the evaluation results.

2. 2. The computer system of claim 1, when a comparison and evaluation request including information for identifying a first model and a second model to be compared is received, searching for first relationship data associated with the model data of the first model by referring to the template management information; generating second relationship data of the second model based on the first relationship data, and registering the second relationship data in the template management information; storing an evaluation result of the evaluation method of the first model corresponding to the evaluation method data included in the first relation data in association with the first relation data; storing an evaluation result of the evaluation method of the second model corresponding to the evaluation method data included in the second relation data in association with the second relation data; A computer system characterized in that a comparison is made between the first model and the second model based on the evaluation results of the first model and the evaluation results of the second model, and a report is generated regarding the results of the comparison.

3. 2. The computer system of claim 1, When an evaluation viewpoint recommendation request including information regarding characteristics of a model is received, the model management information is referenced to search for the model data of the model having the specified characteristics; referring to the template management information to search for the relationship data to which the searched model data is related; A computer system which generates recommendation information for displaying the evaluation method data associated with the searched relationship data.

4. 2. The computer system of claim 1, When a request for recommending an evaluation method including information regarding an evaluation viewpoint is received, the risk assessment management information is referred to, and the risk assessment data corresponding to the specified evaluation viewpoint is searched for; referring to the template management information to search for the relationship data to which the searched risk assessment data is associated; A computer system which generates recommendation information for displaying the evaluation method data associated with the searched relationship data.

5. 2. The computer system of claim 1, The model data includes information regarding a method for improving the model as an item, The computer system comprises: When an improvement technique recommendation request including information on an evaluation technique is received, the evaluation technique management information is referenced to search for the evaluation technique data corresponding to the specified evaluation technique; referring to the template management information to search for the relationship data to which the searched evaluation method data is associated; obtaining the model data associated with the searched relation data from the model management information; A computer system that generates recommendation information for displaying information regarding a method for improving the model that is included in the acquired model data.

6. 2. The computer system of claim 1, Presenting the generated template to a user; A computer system providing an interface for modifying the relation data.

7. A method for evaluating a model executed by a computer system, comprising: The computer system comprises: a computer having a processor, a storage device connected to the processor, and a network interface connected to the processor; The system is connected to model management information for managing model data including items related to models, risk assessment management information for managing risk assessment data including items related to evaluation viewpoints of models, and evaluation method management information for managing evaluation method data including items related to evaluation methods, so as to be accessible to the information; The method for evaluating the model includes: a step of generating, by the computer, associations between the model data, the risk assessment data, and the evaluation method data constituting a template defining the evaluation content of a model as relationship data, and registering the associations in template management information; when the computer receives an evaluation request including information about a model to be evaluated, searching for the model data of the model to be evaluated by referring to the model management information; a step of searching, by the computer, the relation data to which the searched model data is related; generating the template based on the retrieved relation data by the computer; a step of storing, by the computer, an evaluation result based on the evaluation method corresponding to the evaluation technique data associated with the retrieved relationship data, in association with the relationship data; and generating a report based on the generated template and the evaluation results by the computer.

8. A method for evaluating a model according to claim 7, comprising the steps of: when a comparison and evaluation request including information for identifying a first model and a second model to be compared is received, the computer refers to the template management information to search for first relationship data to which the model data of the first model is associated; generating second relationship data of the second model based on the first relationship data, and registering the second relationship data in the template management information by the computer; a step of storing, by the computer, an evaluation result of the evaluation method of the first model corresponding to the evaluation method data included in the first relation data in association with the first relation data; storing, by the computer, an evaluation result of the evaluation method of the second model corresponding to the evaluation method data included in the second relation data in association with the second relation data; a step of the computer comparing the first model and the second model based on the evaluation result of the first model and the evaluation result of the second model, and generating a report on the result of the comparison.

9. A method for evaluating a model according to claim 7, comprising the steps of: When receiving an evaluation viewpoint recommendation request including information regarding characteristics of a model, the computer refers to the model management information to search for the model data of the model having the specified characteristics; a step of searching, by the computer, for the relationship data to which the searched model data is related, by referring to the template management information; a step of generating recommendation information by the computer for displaying the evaluation method data associated with the searched relation data.

10. A method for evaluating a model according to claim 7, comprising the steps of: When receiving an evaluation method recommendation request including information on an evaluation viewpoint, the computer refers to the risk assessment management information and searches for the risk assessment data corresponding to the specified evaluation viewpoint; a step of searching, by the computer, for the relationship data to which the searched risk assessment data is associated, by referring to the template management information; a step of generating recommendation information by the computer for displaying the evaluation method data associated with the searched relation data.

11. A method for evaluating a model according to claim 7, comprising the steps of: The model data includes information regarding a method for improving the model as an item, The method for evaluating the model includes: When an improvement technique recommendation request including information on an evaluation technique is received, the computer refers to the evaluation technique management information and searches for the evaluation technique data corresponding to the specified evaluation technique; a step of the computer referring to the template management information to search for the relationship data to which the searched evaluation method data is associated; The computer acquires the model data associated with the searched relation data from the model management information; A method for evaluating a model, comprising: a step of generating recommendation information by the computer for displaying information regarding a method for improving the model contained in the acquired model data.

12. A method for evaluating a model according to claim 7, comprising the steps of: a step of the computer presenting the generated template to a user; and a step in which the computer provides an interface for modifying the relation data.

Images