Evaluation system, information processing system, evaluation method, and program

Abstract

An evaluation system according to the present disclosure comprises a parameter acquisition means that acquires parameters of a plurality of local models of learning participants for inferring a specific event, an integration means that integrates the acquired parameters of the plurality of local models, an inference means that executes inference using the integrated model obtained by integrating the parameters of the plurality of local models, an evaluation means that evaluates a degree of contribution of each local model on the basis of a result of the inference, and an output means that outputs the degree of contribution.

Description

Technical Field

[0007]

[0001] The present disclosure relates to an evaluation system, an information processing system, an evaluation method, and a recording medium.

Background Art

[0002] Regarding an AI (Artificial Intelligence) model for solving organizational problems, there is an effort to share the AI model in each organization by associating only the parameters of the AI model without exposing the data held by each organization to the outside.

[0003] For example, Patent Document 1 discloses updating the learned model held by each learning device by applying the result of integrating each learned model collected from each of a plurality of learning devices to each learning device.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when integrating each learned model learned by each learning device as in the invention described in Patent Document 1, there are differences in the contribution to the updated integrated model. However, there is no method for appropriately evaluating learning participants with a high contribution. For this reason, for example, the reward for learning participants cannot be appropriately calculated, and each organization is reluctant to participate in federated learning.

[0006] An example of the object of the present disclosure is to provide an evaluation system capable of appropriately evaluating learning participants.

Means for Solving the Problems

[0007] An evaluation system in one aspect of this disclosure includes: parameter acquisition means for acquiring parameters of a plurality of local models of learning participants for inferring a specific event; integration means for integrating the acquired parameters of the plurality of local models; inference means for performing inference using the integrated model obtained by integrating the parameters of the plurality of local models; evaluation means for evaluating the contribution of the local models based on the inference results; and output means for outputting the contribution.

[0008] An evaluation system in one aspect of this disclosure comprises: an inference means that performs inference about a specific event based on an integrated model in which the parameters of multiple models are integrated by federated learning using secure computation; an evaluation means that evaluates the contribution of local models based on the inference results; and an output means that outputs the contribution.

[0009] An information processing system in one aspect of the present disclosure is an information processing system having a plurality of learning participant servers and the evaluation system described above, wherein each learning participant server includes a model storage means for storing a trained model for inferring about a specific event, an input / output means for inputting parameters of the stored model in an anonymized format, parameters updated by federated learning using secure computation, a restoration unit for restoring the input parameters, and a participant inference means for updating the model by applying the restored parameters to the stored model and inferring about a specific event.

[0010] An evaluation method in one aspect of this disclosure involves a computer obtaining parameters from multiple local models of learning participants for inferring a specific event, integrating the obtained parameters from the multiple local models, performing inference using the integrated model obtained by integrating the parameters from the multiple local models, evaluating the contribution of the local models based on the inference results, and outputting the contribution.

[0011] A recording medium in one aspect of this disclosure stores a program that causes a computer to perform the following actions for local models of multiple learning participants for inferring a specific event: obtaining parameters of multiple local models, integrating the obtained parameters of the multiple local models, performing inference using the integrated model obtained by integrating the parameters of the multiple local models, evaluating the contribution of the local models based on the inference results, and outputting the contribution. [Effects of the Invention]

[0012] One example of the effects of this disclosure is the provision of an evaluation system that enables appropriate assessment of learning participants. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a block diagram showing the configuration of the information processing system in the first embodiment. [Figure 2] Figure 2 shows the hardware configuration in which the evaluation system in the first embodiment is implemented using a computer device and its peripheral devices. [Figure 3] Figure 3 is a diagram illustrating the inference accuracy threshold in the first embodiment. [Figure 4] Figure 4 is a flowchart showing the calculation operation in the first embodiment. [Figure 5] Figure 5 is a block diagram showing the configuration of an information processing system in a modified example of the first embodiment. [Figure 6] Figure 6 shows an example of the types of missing data presented by the presentation unit in the first embodiment. [Figure 7] Figure 7 is a flowchart showing the information processing operation in a modified example of the first embodiment. [Figure 8] Figure 8 is a block diagram showing the configuration of the information processing system in the second embodiment. [Figure 9] Figure 9 is a flowchart showing the operation of information processing in the second embodiment. [Embodiments of the Invention]

[0014] Next, embodiments will be described in detail with reference to the drawings.

[0015] [First Embodiment] FIG. 1 is a block diagram showing the configuration of an information processing system 10 in the first embodiment. The information processing system 10 in the first embodiment is a system for calculating the reward of each learning participant when a combined model obtained by integrating a plurality of local models for inferring a specific event held by each learning participant by collaborative learning is generated. Note that the collaborative learning may be performed multiple times to generate a combined model that satisfies predetermined conditions. Examples of learning participants include organizations such as local governments or companies.

[0016] [[ID=1,6]]Referring to FIG. 1, the information processing system

[10] includes an evaluation system 100 and a plurality of learning participant servers 200 (200a, 200b). The evaluation system 100 outputs an inference result related to an event by inputting an explanatory variable value into the combined model. The specific event is, for example, a matter that can be expressed by an arbitrary model (mathematical formula) using explanatory variables corresponding to factors affecting the event. The model of the present embodiment is a model that learns the factors and occurrence conditions of an event from past case data, inputs an explanatory variable value, and outputs an inference result of improvement conditions for solving the factors of the event. Examples of the inference target include measures to encourage behavioral changes in employees for the promotion of employee health and productivity improvement, measures to encourage behavioral changes in citizens for the reduction of medical expenses for health insurance in local governments, or measures to promote the utilization of public facilities such as libraries or gymnasiums. However, the events inferred by the evaluation system 100 are not limited to these.

[0017] It should be noted that there seems to be a small error in the original text where "information processing system

[10] " is written as "information processing system 10" in some parts. I have corrected it in the translation for better understanding. If this is not allowed, please let me know.The evaluation system 100 includes a parameter acquisition unit 101, an integration unit 102, an inference unit 103, an evaluation unit 104, a calculation unit 105, and an output unit 106. However, the calculation unit 105 is an optional component. Each of the learning participant servers 200 includes a model generation unit 201 (201a, 201b) that generates a model for inferring a specific event, and an input / output unit 202 (202a, 202b) that performs input / output of parameters with the evaluation system 100. In this embodiment, there are two learning participant servers 200, but it is not limited to this. The plurality of learning participant servers 200 are provided in number corresponding to the number of learning participants who participate in learning. Hereinafter, the evaluation system 100, which is an essential configuration of this embodiment, will be described in detail.

[0018] FIG. 2 is a diagram showing an example of a hardware configuration in which the evaluation system 100 in the first embodiment of the present disclosure is realized by a computer device 500 including a processor. As shown in FIG. 2, the evaluation system 100 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a memory such as a RAM (Random Access Memory) 503, a storage device 505 such as a hard disk for storing a program 504, a communication I / F (Interface) 508 for network connection, and an input / output interface 511 for inputting / outputting data. In the first embodiment, the parameter information received from each learning participant server 200 is input to the evaluation system 100 via the communication I / F 508.

[0019] The CPU 501 operates an operating system to control the entire evaluation system 100 according to the first embodiment of the present invention. Further, the CPU 501 reads programs and data from a recording medium 506 mounted on, for example, a drive device 507 into the memory. Further, the CPU 501 functions as the parameter acquisition unit 101, the integration unit 102, the inference unit 103, the evaluation unit 104, the calculation unit 105, the output unit 106, and a part thereof in the first embodiment, and executes the processes or instructions in the flowchart shown in FIG. 4 described later based on the program.

[0020] The recording medium 506 is, for example, an optical disc, a flexible disc, a magneto-optical disc, an external hard disk, or a semiconductor memory. Some of the recording media of the storage device are non-volatile storage devices, on which the program is recorded. The program may also be downloaded from an external computer (not shown) connected to a communication network.

[0021] The input device 509 is implemented, for example, by a mouse, keyboard, or built-in key buttons, and is used for input operations. The input device 509 is not limited to a mouse, keyboard, or built-in key buttons; it may also be a touch panel, for example. The output device 510 is implemented, for example, by a display, and is used to confirm the output.

[0022] As described above, the first embodiment shown in Figure 1 is realized by the computer hardware shown in Figure 2. However, the means of realizing each part of the evaluation system 100 in Figure 1 are not limited to the configuration described above. The evaluation system 100 may also be realized by a single physically coupled device, or by two or more physically separated devices connected by wired or wireless connections. For example, the input device 509 and the output device 510 may be connected to the computer device 500 via a network. Furthermore, the evaluation system 100 in the first embodiment shown in Figure 1 can also be configured using cloud computing or the like.

[0023] The parameter acquisition unit 101 acquires the parameters of a trained model from each of the multiple learning participant servers 200, triggered, for example, by an operation for performing associative learning. The model is a model trained by machine learning in each learning participant to output, for example, inference results regarding a specific event. The machine learning model includes, but is not limited to, decision tree models, linear regression models, logistic regression models, and neural network models.

[0024] The integration unit 102 integrates the parameters of each model from multiple models. Known methods can be used for parameter integration; for example, the weights of the parameters corresponding to each model can be changed during integration depending on the characteristics of each model. The integration unit 102 then applies the parameters obtained in this way to the models and stores them in the storage device 505.

[0025] The parameter acquisition unit 101 may acquire parameters from each learning participant server 200 in an anonymized format. In this case, the integration unit 102 integrates the anonymized parameters of multiple local models using secure computation. In this embodiment, integrating the parameters of multiple anonymized local models using secure computation means that the evaluation system 100 performs machine learning (federated learning) with the evaluation system distributed across each learning participant server 200, integrates the parameters of the trained models using secure computation, and generates a new integrated model. The parameters acquired by the parameter acquisition unit 101 from each learning participant server 200 are anonymized versions of the parameters before the federated learning. In this embodiment, if federated learning is repeated, the parameters acquired by the parameter acquisition unit 101 will be anonymized versions of the parameters before the most recent federated learning.

[0026] In this embodiment, concealment is synonymous with encryption. Secure computation is computation performed while the data remains concealed, and the evaluation system 100, which acquires and integrates the parameters of the local model, cannot access the concealed raw data. As a method of secure computation, special encryption corresponding to specific processing, such as homomorphic encryption, a trusted execution environment that processes in an isolated state on hardware, or multi-party computation that performs computation while the data is secretly shared among multiple servers (secret sharing computation) can be used.

[0027] The following is an example of a specific method for secure computation in multi-party computation. For example, confidential data a, which is a parameter obtained from an arbitrary learning participant server, is secretly distributed into distributed values ​​x1, y1, ... and each of x1, y1, ... is sent to a different server by a different administrator. Similarly, confidential data b, which is a parameter obtained from another learning participant server, is secretly distributed into distributed values ​​x2, y2, ... and each of x2, y2, ... is sent to a different server by a different administrator. Then, while confidential data a and confidential data b remain secretly distributed, the computation proceeds while they communicate with each other. Finally, the distributed values ​​u, v, ... of the output, which are the computation results from each server, are collected and restored to obtain the computation result F(a,b). This computation result becomes the integrated parameter of each model. For this reason, when using multi-party computation as a secure computation method, the integration unit 102 is equipped with multiple servers. Multi-party computation eliminates the need for cryptographic key management and isolated environments, and the computation process is faster. The integration unit 102 restores the model parameters obtained in this manner and stores the model to which the restored parameters have been applied in the storage device 505.

[0028] The inference unit 103 is a means for performing inference based on an integrated model that combines the parameters of multiple local models. The inference unit 103 inputs explanatory variable values ​​into the integrated model stored in the memory device 505 and performs inference. If the parameters of the integrated model are confidential, the inference unit 103 may perform inference using secure computation, or it may decrypt the parameters of the integrated model before performing inference. The inference unit 103 outputs the inference results to the evaluation unit 104.

[0029] The evaluation unit 104 is a means for evaluating the contribution of local models based on the inference results obtained. In this embodiment, the evaluation unit 104 evaluates the contribution of each local model, for example, based on the improvement in the inference accuracy of the integrated model due to the integration of each local model. The evaluation unit 104 determines the contribution of each local model based on the difference between the inference accuracy of the integrated model when the parameters of the local models are integrated and the inference accuracy when the parameters of the local models are not integrated. In this embodiment, inference accuracy refers to the accuracy rate of improvement conditions for the factors of the events output by the model. In other words, inference accuracy indicates the extent to which the event has improved as a result of taking measures to satisfy the improvement conditions output by the model.

[0030] Here, we will specifically explain how to evaluate the contribution of each organization's local model when an integrated model is generated by integrating the local models of learning participants A to C. For example, when calculating the contribution of learning participant A's local model, the evaluation unit 104 calculates the inference accuracy of the integrated model when the local models of learning participants B and C are integrated. Then, the evaluation unit 104 compares the inference accuracy of the integrated model when the local models of learning participants B and C are integrated with the inference accuracy of the integrated model when the parameters of all local models of learning participants A to C are integrated, and evaluates the contribution based on the rate of improvement in inference accuracy.

[0031] Similarly, when evaluating the contribution of participant B's local model, the evaluation unit 104 evaluates the inference accuracy of the integrated model when the parameters of participant A's and participant C's local models are combined. Then, the evaluation unit 104 compares the inference accuracy of the integrated model when participant B's and participant C's local models are combined with the inference accuracy of the integrated model when the parameters of all participant A-C's local models are combined, and evaluates the contribution based on the rate of improvement in inference accuracy. The evaluation unit 104 evaluates the contribution of participant C in the same way. However, the method for evaluating the contribution of each local model is not limited to this.

[0032] Furthermore, the evaluation unit 104 may add the contribution of the local model if a learning participant joins associative learning before the inference accuracy of the integrated model reaches a predetermined threshold. That is, the evaluation unit 104 may add the contribution of the local model if the parameter acquisition unit 101 acquires the parameters of the local model before the inference accuracy of the integrated model reaches a predetermined threshold. Also, if the inference accuracy of the integrated model decreases after associative learning, the evaluation unit 104 may evaluate the contribution as negative and not integrate the parameters of the local model of the learning participant who joined associative learning at that time.

[0033] Figure 3 illustrates the threshold for the inference accuracy of the integrated model. As shown in Figure 3, the inference accuracy gradually increases according to the amount of model training, but once the inference accuracy reaches a predetermined threshold, the increase in inference accuracy slows down. The evaluation unit 104 may add contribution points to learners who participated in associative learning up to the threshold where the degree of increase in inference accuracy is large. In the example in Figure 3, it is described that contribution points are added (discount on model usage fees) to learners who participated in associative learning up to the threshold. Alternatively, the evaluation unit 104 may set a threshold after associative learning, such as setting the threshold at 50% of the final inference accuracy of the integrated model, based on the final inference accuracy after associative learning is sufficiently completed.

[0034] The calculation unit 105 is a means for calculating rewards for learning participants who provided parameters for local models based on the contribution of the local models. In the example of an integrated model that combines the parameters of each local model of learning participants A to C described above, the calculation unit 105 sets the ratio of the calculated contributions as the ratio of rewards distributed to each of the businesses A to C. As a method of returning rewards to each business, the usage fee for the integrated model may be discounted. Furthermore, the calculation unit 105 may make the usage fee for the integrated model during the inference stage free for learning participants of local models that have contributed above a certain level.

[0035] In this embodiment, the evaluation unit 104 may evaluate the contribution of the local model based on the timing of the learning participant's participation, rather than the inference accuracy. That is, the evaluation unit 104 may evaluate the contribution of the local model based on the timing of when the parameter acquisition unit 101 acquired the parameters of the local model. The evaluation unit 104 may also evaluate the contribution based on the order of the parameters acquired from the learning participant or the timing of their acquisition.

[0036] The output unit 106 is a means for outputting the evaluated contribution. The output unit 106 may, for example, output the contribution of each local model to an output device 510 such as a display, or it may transmit information about the contribution to each learning participant. The output unit 106 may also output information about rewards.

[0037] The operation of the evaluation system 100, configured as described above, will be explained with reference to the flowchart in Figure 4.

[0038] Figure 4 is a flowchart illustrating the operation of the evaluation system 100 in the first embodiment. Note that the processing shown in this flowchart may be executed based on the program control by the processor described above. Furthermore, the series of processes shown in this flowchart do not have to be performed consecutively; for example, steps S101 to S104 and steps S105 to S108 in Figure 4 may be performed at different times. Also, the calculation of the learning participant's reward in step S107 may be skipped. In this case, in step S108, the output unit 106 outputs the contribution score.

[0039] As shown in Figure 4, first, the parameter acquisition unit 101 acquires parameters of a trained local model for inferring a specific event from each learning participant server 200 (step S101). If the evaluation system 100 computes in secret, that is, if the parameters of the multiple local models acquired by the parameter acquisition unit 101 are in secret format (S102; YES), the integration unit 102 integrates the parameters of the multiple local models in secret using secure computation (step S103). On the other hand, if the evaluation system 100 computes without secret, that is, if the parameters of the multiple local models acquired by the parameter acquisition unit 101 are not in secret format (S102; NO), the integration unit 102 integrates the parameters of the multiple local models without using secure computation (step S104).

[0040] The inference unit 103 performs inference using the integrated model (step S105), and then the evaluation unit 104 evaluates the contribution of the local model to the integrated model (step S106). The calculation unit 105 calculates the reward for the learning participant based on the contribution (step S107). The output unit 106 outputs the calculated reward (step S108). With this, the evaluation system 100 completes its calculation operation.

[0041] In this embodiment, the evaluation system 100 has an evaluation unit 104 that evaluates the contribution of the local model to the integrated model. This allows for appropriate evaluation of the learning participants. The calculation unit 105 then calculates the learning participants' rewards based on their contributions. This allows for appropriate setting of rewards for learning participants based on the local model's contribution to the integrated model. Furthermore, in the evaluation system 100, if a learning participant joins associative learning before the inference accuracy of the integrated model reaches a predetermined threshold, the evaluation unit 104 adds to the contribution of the local model. This allows for proper evaluation of the contribution of learning participants who join associative learning at an early stage, thus encouraging participation from each organization even in the initial stages of associative learning.

[0042] Furthermore, in the evaluation system 100, if the parameters of multiple local models acquired by the parameter acquisition unit 101 are in a confidential format, the integration unit 102 integrates the confidential parameters of the multiple local models using secure computation. This allows the integrated model to be used while keeping the parameters of each model confidential.

[0043] [Modified example of the first embodiment] Next, a modified example of the first embodiment of this disclosure will be described in detail with reference to the drawings. To the extent that the description of this embodiment remains clear, any content that overlaps with the above description will be omitted. Figure 5 is a block diagram showing the configuration of the information processing system in a modified example of the first embodiment. As shown in Figure 5, the evaluation system 110 in the information processing system 11 includes a parameter acquisition unit 111, an integration unit 112, an inference unit 113, a specific unit 114, a presentation unit 115, an evaluation unit 116, a calculation unit 117, and an output unit 118. That is, it differs from the evaluation system 100 in the first embodiment in that it includes a specific unit 114 and a presentation unit 115.

[0044] In this embodiment, the inference unit 113 takes explanatory variable values ​​for each type as input and uses an integrated model to infer events. The types of training data are classified according to attributes such as age and gender, or personal data such as behavioral history and family structure. The inference unit 113 also calculates the inference accuracy for each type of integrated model and outputs it to the identification unit 114.

[0045] The identification unit 114 is a means for identifying the type of missing training data based on at least one of the type of training data, events, and inference accuracy used to train the integrated model. The identification unit 114 identifies the type of data whose inference accuracy is below a predetermined level based on the various inference accuracy levels input from the inference unit 113. The identification unit 114 may also identify the type of data for which there is insufficient training data for the integrated model. In this case, if the amount of training data from past failure cases is below a predetermined level, the identification unit 114 may identify the failure case data as the missing training data.

[0046] The presentation unit 115 is a means for presenting information for recruiting learning participants. The presentation unit 115 presents information for recruiting learning participants, such as information on the reward for participating in federative learning or information on the current status of the integrated model. The presentation unit 115 may present this information on a Web (World Wide Web) page for recruiting participants for federative learning, or it may send it to organizations that it would like to participate in federative learning. In addition, as shown in Figure 3, the presentation unit 115 may present information on the reward while also presenting the current inference accuracy of the integrated model and the threshold for inference accuracy.

[0047] The presentation unit 115 may present the type of training data identified by the identification unit 114. Figure 6 shows an example of the presentation of the types of missing data presented by the presentation unit 115. In the example in Figure 6, it is described that the contribution is added when the missing data and the parameters of the local model trained using each of the missing data are provided.

[0048] The evaluation unit 116 adds to the contribution score when a learning participant provides parameters for a local model learned using missing data, in addition to the contribution score evaluation method of the evaluation unit 104 in the first embodiment. Note that the calculation unit 117 and output unit 118 in this embodiment operate the same as the calculation unit 105 and output unit 106 in the first embodiment, so their explanation is omitted here.

[0049] Figure 7 is a flowchart illustrating the operation of the evaluation system 110 in a modified example of the first embodiment. This flowchart assumes that the missing types of training data are presented before the integration of parameters (steps S114-S115). This flowchart may also be executed based on program control by the processor described above, similar to the first embodiment.

[0050] As shown in Figure 7, first, the inference unit 113 performs inference for each type using the integrated model (step S111). Next, the identification unit 114 identifies the training data for the missing types (step S112). The presentation unit 115 presents the types of training data identified by the identification unit 114 (step S113).

[0051] Next, the parameter acquisition unit 111 acquires parameters of a trained local model for inferring a specific event from each learning participant server 210 (step S114). Then, the integration unit 112 integrates the parameters of the multiple local models (step S115).

[0052] The inference unit 113 performs inference using the integrated model (step S116), and then the evaluation unit 116 evaluates the contribution of the local model (step S117). If the learning participant provides parameters for the local model learned using the missing data (S118; YES), the evaluation unit 116 adds the learning participant's contribution (step S119). Then, the calculation unit 117 calculates the learning participant's reward based on the contribution (step S120). On the other hand, if the learning participant does not provide parameters for the local model learned using the missing data (S118; NO), the evaluation unit 116 does not add the contribution, and the calculation unit 117 calculates the learning participant's reward based on the contribution (step S120). With this, the evaluation system 110 completes its calculation operation.

[0053] In the evaluation system 110, the presentation unit 115 presents the type of training data identified by the identification unit 114, and the evaluation unit 116 adds the contribution of the learning participant if the learning participant provides parameters for the local model trained using the missing data. This allows for an appropriate evaluation of the contribution of learning participants who contribute more to the integrated model, in proportion to their contribution.

[0054] [Second Embodiment] Next, a second embodiment of this disclosure will be described in detail with reference to the drawings. To the extent that this description of the embodiment does not become unclear, any content that overlaps with the above description will be omitted. In the second embodiment, the information processing system 12 is used to provide each learning participant server with a model updated by federated learning using secure computation. These updated models are used, for example, by each learning participant to infer specific events. Each component in each embodiment of this disclosure can be implemented not only in hardware, similar to the computer device shown in Figure 2, but also in a computer device based on program control.

[0055] <Evaluation System> Figure 8 is a block diagram showing the configuration of an information processing system 12 equipped with an evaluation system 120 according to the second embodiment of this disclosure. Referring to Figure 8, the evaluation system 120 and the learning participant servers 220 (220a, 220b) according to the second embodiment will be described, focusing on the differences from the information processing system 10 according to the first embodiment. The evaluation system 120 includes a parameter acquisition unit 121, an integration unit 122, a parameter transmission unit 123, an inference unit 124, an evaluation unit 125, a calculation unit 126, and an output unit 127.

[0056] The parameter acquisition unit 121 acquires the parameters of each participant's trained model from the participant server 220 via the communication interface 508. Next, the integration unit 122 integrates the received confidential model parameters using secure computation and outputs the integrated confidential model parameters to the parameter transmission unit 123 in a confidential format. The parameter transmission unit 123 transmits the integrated parameters to each participant server 220 via the input / output unit 223. Furthermore, if the participant server 220 performs model training again and updates the parameters after transmitting the parameters to the participant server 220, the evaluation system 120 may acquire the updated parameters again. Note that the operation of the inference unit 124, evaluation unit 125, calculation unit 126, and output unit 127 is the same as the operation of the corresponding components in the first embodiment, so a description is omitted here.

[0057] <Learner participant server> In the second embodiment, the multiple learning participant servers 220 (220a, 220b) include a model generation unit 221 (221a, 221b), an anonymization unit 222 (222a, 222b), an input / output unit 223 (223a, 223b), a restoration unit 224 (224a, 224b), a model storage unit 225 (225a, 225b), and a participant inference unit 226 (226a, 226b). The model storage unit 225 stores the model generated by the model generation unit 221.

[0058] The learning participant server 220 updates the model stored in the model storage unit 225 to a model to which the parameters received from the evaluation system 120 have been applied. Specifically, the input / output unit 223 receives the parameters in an anonymized format and outputs them to the restoration unit 224. The restoration unit 224 then restores the parameters and replaces them with the parameters of the model stored in the model storage unit 225. Next, the participant inference unit 226 performs inference using the updated model. To improve the accuracy of the inference results by the participant inference unit 226, the learning participant server 220 may perform learning again based on additionally obtained training data and send further updated parameters to the evaluation system 120. The accuracy of the integrated model can be further improved by repeating the parameter updates by learning in each learning participant server 220 and the integration of parameters in the evaluation system 120, for example, until predetermined conditions are met. These predetermined conditions are stored, for example, in the memory device 505.

[0059] The operation of the information processing system 12, configured as described above, will be explained with reference to the flowchart in Figure 9.

[0060] Figure 9 is a flowchart illustrating the overview of the operation of the information processing system 12 in the second embodiment. Note that the processing shown in this flowchart may be executed based on the program control by the processor described above. The calculation of rewards in steps S212 to S215 by the evaluation system 120 may be performed each time the associative learning process in steps S201 to S210 is executed, or it may be performed when the integrated model satisfies predetermined conditions.

[0061] As shown in Figure 9, first, in the learning participant server 220, the model generation unit 221 generates a model locally using the data held by the learning participant (step S201). Next, the concealment unit 222 conceals the model parameters (step S202), and the input / output unit 223 outputs the concealed parameters to the evaluation system 120 (step S203). Next, the parameter acquisition unit 121 of the evaluation system 120 acquires the concealed parameters (step S204). Next, the integration unit 122 integrates the parameters of the multiple concealed local models using secure computation (step S205). Next, the parameter transmission unit 123 outputs the parameters of the model integrated by the integration unit 122 in a concealed format to each of the learning participant servers 220 (step S206).

[0062] Next, each learning participant server 220 obtains the integrated parameters in an anonymized format through the input / output unit 223 (step S207). Next, the restoration unit 224 restores the parameters in an anonymized format (step S208). Next, the learning participant server 220 updates the model stored in the model memory unit 225 by applying the restored parameters (step S209). Next, the learning participant server 220 determines whether the predetermined conditions are met (step S210). If the predetermined conditions are met (step S210; YES), the participant inference unit 226 performs inference using the updated model (step S211). If the predetermined conditions are not met, the learning participant server 220 returns to step S201 (step S210; NO) and performs the flow again.

[0063] Meanwhile, in the evaluation system 120, the inference unit 124 performs inference using the integrated model (step S212), and then the evaluation unit 125 evaluates the contribution of the local model (step S213). The calculation unit 126 calculates the reward for the learning participant based on the contribution (step S214). The output unit 127 outputs the calculated reward (step S215). With this, the information processing system 12 completes its information processing operation.

[0064] In a second embodiment of this disclosure, the participant inference unit 226 of each learning participant server 220 performs inference on a specific event using a model to which the parameters of the integrated model by the integration unit 122 have been applied. This enables each learning participant server 220 to output more accurate inference results.

[0065] Although the present invention has been described above with reference to the embodiments described, the present invention is not limited to the above embodiments. Various modifications to the configuration and details of the present invention can be made that will be understood by those skilled in the art within the scope of the present invention.

[0066] For example, although multiple actions are described sequentially in flowchart format, the order in which they are described does not limit the order in which the actions must be performed. Therefore, when implementing each embodiment, the order of the multiple actions can be changed as long as it does not impair the content.

[0067] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0068] (Note 1) A parameter acquisition means for obtaining the parameters of multiple local models of learning participants used to infer a specific event, Integration means for integrating the parameters of the multiple local models obtained, An inference means that performs the inference using an integrated model which combines the parameters of the multiple local models, An evaluation means for evaluating the contribution of the local model based on the results of the above inference, An evaluation system comprising an output means for outputting the aforementioned contribution level.

[0069] (Note 2) The system further comprises a calculation means for calculating a reward for the learning participant who provided the parameters for the local model based on the aforementioned contribution, The output means is the evaluation system described in Appendix 1, which outputs the reward.

[0070] (Note 3) The parameter acquisition means acquires the parameters of the multiple local models in an anonymized format, The integration means is an evaluation system according to Appendix 1 or Appendix 2, which integrates the parameters of the plurality of local models using secure computation.

[0071] (Note 4) An inference means that performs inference about a specific event based on an integrated model in which the parameters of multiple local models are integrated through federative learning using secure computation, Based on the results of the above reasoning, an evaluation means for evaluating the contribution of the local model, An evaluation system comprising an output means for outputting the aforementioned contribution level.

[0072] (Note 5) The evaluation means is an evaluation system according to any one of the appendices 1 to 4, which evaluates the contribution of the local models based on the change in the inference accuracy of the integrated model due to the integration of the parameters of the multiple local models.

[0073] (Note 6) The evaluation means is an evaluation system according to any one of the appendices 1 to 4, which evaluates the contribution of the local model based on the timing of when the multiple learning participants participated in the federative learning.

[0074] (Note 7) The evaluation means is the evaluation system described in Appendix 5 or Appendix 6, which adds the contribution of the local model if the multiple learning participants join federated learning before the inference accuracy of the integrated model reaches a predetermined threshold.

[0075] (Note 8) An evaluation system described in any of the appendices 1 to 7, which provides information for recruiting learning participants and further includes means for doing so.

[0076] (Note 9) The aforementioned presentation means is an evaluation system as described in Appendix 8, which presents the current inference accuracy of the integrated model and the threshold for inference accuracy.

[0077] (Note 10) The system further includes a means for identifying the types of missing training data based on at least one of the types of training data used to train the integrated model, the content of the events, and the inference accuracy. The presentation means is an evaluation system as described in Appendix 8, which presents the type of the identified learning data.

[0078] (Note 11) The evaluation means is the evaluation system described in Appendix 10, which adds up the contribution of the local model generated based on the missing training data.

[0079] (Note 12) An information processing system having multiple learning participant servers and an evaluation system described in any of the appendices 1 to 11, Each of the aforementioned learning participant servers includes a model storage means for storing a trained model for inferring about a specific event, An input / output means for inputting parameters of the stored model in a secure format, which have been updated by federative learning using secure computation. A restoration means for restoring the input parameters, An information processing system comprising participant inference means for updating the model by applying the restored parameters to the stored model and for inferring about the specific event.

[0080] (Note 13) Computers For local models of multiple learning participants used to infer a specific event, obtain the parameters of those multiple local models, The parameters of the multiple local models obtained are integrated, Using the integrated model which combines the parameters of the aforementioned multiple local models, the inference is performed. Based on the results of the above inference, we evaluate the contribution of the local model, An evaluation method for outputting the aforementioned contribution level.

[0081] (Note 14) For local models of multiple learning participants used to infer a specific event, obtain the parameters of those multiple local models, The parameters of the multiple local models obtained are integrated, Using the integrated model which combines the parameters of the aforementioned multiple local models, the inference is performed. Based on the results of the above inference, we evaluate the contribution of the local model, A recording medium that stores a program that causes a computer to perform the action of outputting the aforementioned contribution level. [Explanation of symbols]

[0082] 10, 11, 12 Information Processing Systems 100, 110, 120 rating system 101, 111, 121 Parameter acquisition section 102, 112, 122 Integration Department 103, 113, 124 Reasoning part 104, 116, 125 Evaluation Department 105, 117, 126 Calculation section 106, 118, 127 Output section 114 Specific section 115 Presentation section 123 Parameter transmission unit 200, 210, 220 Learning Participant Servers 201, 211, 221 Model generation unit 202, 212, 223 input / output section 222 Confidentiality Section 224 Restoration Section 225 Model Memory Unit 226 Participant Reasoning Department

Claims

1. A parameter acquisition means for obtaining the parameters of multiple local models of learning participants used to infer a specific event, Integration means for integrating the parameters of the multiple local models obtained, An inference means that performs the inference using an integrated model which combines the parameters of the multiple local models, An evaluation means for evaluating the contribution of the local model based on the results of the above inference, The system includes an output means for outputting the aforementioned contribution level, The evaluation means evaluates the contribution of the local models based on the change in the inference accuracy of the integrated model due to the integration of the parameters of the multiple local models, The evaluation means is an evaluation system that adds up the contribution of the local models if the multiple learning participants join federated learning before the inference accuracy of the integrated model reaches a predetermined threshold.

2. A parameter acquisition means for acquiring the parameters of multiple local models of multiple learning participants for inferring a specific event, Integration means for integrating the parameters of the multiple local models obtained, An inference means that performs the inference using an integrated model which combines the parameters of the multiple local models, An evaluation means for evaluating the contribution of the local model based on the results of the above inference, The system includes an output means for outputting the aforementioned contribution level, To provide information for recruiting learning participants, and to further enhance the means of presentation, The system further includes a means for identifying the types of missing training data based on at least one of the types of training data used to train the integrated model, the content of the events, and the inference accuracy. The aforementioned presentation means is an evaluation system that presents the type of the identified learning data.

3. The system further comprises a calculation means for calculating a reward for the learning participant who provided the parameters for the local model based on the aforementioned contribution, The evaluation system according to claim 1, wherein the output means outputs the reward.

4. The claim further comprises a calculation means for calculating a reward to a learning participant who provided the parameters of the local model based on the contribution, The evaluation system according to claim 2, wherein the output means outputs the reward.

5. The parameter acquisition means acquires the parameters of the multiple local models in an anonymized format, The evaluation system according to claim 1, wherein the integration means integrates the parameters of the plurality of local models using secure computation.

6. The parameter acquisition means acquires the parameters of the plurality of local models in an anonymized format, The evaluation system according to claim 2, wherein the integration means integrates the parameters of the plurality of local models using secure computation.

7. The evaluation system according to claim 1 or 2, wherein the evaluation means evaluates the contribution of the local model based on the timing of when the plurality of learning participants participated in the federative learning.

8. The evaluation system according to claim 2, wherein the presentation means presents the current inference accuracy of the integrated model and a threshold for the inference accuracy.

9. The evaluation system according to claim 2, wherein the evaluation means adds the contribution of the local model generated based on the missing training data.

10. An information processing system having a plurality of learning participant servers and an evaluation system according to any one of claims 1 to 6, 8, or 9, Each of the aforementioned learning participant servers includes a model storage means for storing a trained model for inferring about a specific event, An input / output means for inputting parameters of the stored model in a secure format, which have been updated by federative learning using secure computation. A restoration means for restoring the input parameters, An information processing system comprising participant inference means for updating the model by applying the restored parameters to the stored model and for inferring about the specific event.

11. Computers For local models of multiple learning participants used to infer a specific event, obtain the parameters of those multiple local models, The parameters of the multiple local models obtained are integrated, Using the integrated model which combines the parameters of the aforementioned multiple local models, the inference is performed. Based on the results of the above inference, we evaluate the contribution of the local model, An evaluation method for outputting the aforementioned contribution level, In evaluating the contribution, the contribution of the local models is evaluated based on the change in the inference accuracy of the integrated model due to the integration of the parameters of the multiple local models, and further, An evaluation method that adds the contribution of the local model if the multiple learning participants join federated learning before the inference accuracy of the integrated model reaches a predetermined threshold.

12. A computer, For local models of multiple learning participants used to infer a specific event, obtain the parameters of those multiple local models, The parameters of the multiple local models obtained are integrated, Using the integrated model which combines the parameters of the aforementioned multiple local models, the inference is performed. Based on the results of the above inference, we evaluate the contribution of the local model, Output the aforementioned contribution level, and further, Present information to recruit participants for the learning program. Based on at least one of the following: the type of training data used to train the integrated model, the content of the events, and the inference accuracy, the types of training data that are missing are identified. An evaluation method for presenting information for recruiting learning participants, which includes presenting the type of learning data identified.

13. For local models of multiple learning participants used to infer a specific event, obtain the parameters of those multiple local models, The parameters of the multiple local models obtained are integrated, Using the integrated model which combines the parameters of the aforementioned multiple local models, the inference is performed. Based on the results of the above inference, we evaluate the contribution of the local model, A program that causes a computer to output the aforementioned contribution level, In evaluating the contribution, the contribution of the local models is evaluated based on the change in the inference accuracy of the integrated model due to the integration of the parameters of the multiple local models, and further, A program that causes a computer to perform the following actions: if the multiple learning participants join federated learning before the inference accuracy of the integrated model reaches a predetermined threshold, add up the contributions of the local models.

14. For local models of multiple learning participants for inferring a specific event, obtain the parameters of the multiple local models, The parameters of the multiple local models obtained are integrated, Using the integrated model which combines the parameters of the aforementioned multiple local models, the inference is performed. Based on the results of the above inference, we evaluate the contribution of the local model, Output the aforementioned contribution level, and further, Present information to recruit participants for the learning program. Based on at least one of the following: the type of training data used to train the integrated model, the content of the events, and the inference accuracy, the types of training data that are missing are identified. A program that causes a computer to display the type of learning data identified when presenting information for recruiting learning participants.

Images