Evaluation system, information processing system, evaluation method, and program

Abstract

An evaluation system according to the present disclosure comprises a parameter acquisition means that acquires parameters of a plurality of local models of learning participants for inferring a specific event, an integration means that integrates the acquired parameters of the plurality of local models, an inference means that executes inference using the integrated model obtained by integrating the parameters of the plurality of local models, an evaluation means that evaluates a degree of contribution of each local model on the basis of a result of the inference, and an output means that outputs the degree of contribution.

Description

Evaluation system, information processing system, evaluation method, and recording medium

[0001] The present disclosure relates to an evaluation system, an information processing system, an evaluation method, and a recording medium.

[0002] Regarding AI (Artificial Intelligence) models for solving organizational problems, there is an initiative to share AI models among organizations by combining only the parameters of the AI ​​models without releasing the data held by each organization to the outside.

[0003] For example, Patent Document 1 discloses updating the trained models held by each learning device by applying the results of integrating trained models collected from each of multiple learning devices to each learning device.

[0004] Japanese Patent Application Laid-Open No. 2020-115311

[0005] However, when integrating trained models trained by individual learning devices, as in the invention described in Patent Document 1, there are differences in the degree of contribution to the updated integrated model. However, there is no method for properly evaluating learning participants with high levels of contribution. For this reason, for example, it is not possible to properly calculate rewards for learning participants, and organizations are reluctant to participate in federated learning.

[0006] An example of an objective of the present disclosure is to provide an evaluation system that can appropriately evaluate learning participants.

[0007] An evaluation system in one aspect of the present disclosure includes a parameter acquisition means for acquiring parameters of multiple local models of multiple learning participants for inferring a specific phenomenon, an integration means for integrating the acquired parameters of the multiple local models, an inference means for performing inference using an integrated model that integrates the parameters of the multiple local models, an evaluation means for evaluating the contribution of the local models based on the results of the inference, and an output means for outputting the contribution.

[0008] An evaluation system in one aspect of the present disclosure includes an inference means for performing inference regarding a specific event based on an integrated model in which parameters of multiple models are integrated through federated learning using secure computing, an evaluation means for evaluating the contribution of a local model based on the results of the inference, and an output means for outputting the contribution.

[0009] An information processing system in one aspect of the present disclosure is an information processing system having a plurality of learning participant servers and the evaluation system described above, wherein each learning participant server is equipped with a model storage means for storing a trained model for inferring about a specific event, an input / output means for inputting parameters of the stored model that have been updated by federated learning using secure computation in a confidential format, a restoration unit for restoring the input parameters, and a participant inference means for applying the restored parameters to the stored model to update the model and make inference about a specific event.

[0010] In one aspect of the present disclosure, an evaluation method involves a computer acquiring parameters of multiple local models of multiple learning participants for inferring a specific phenomenon, integrating the acquired parameters of the multiple local models, performing inference using an integrated model that integrates the parameters of the multiple local models, evaluating the contribution of the local models based on the results of the inference, and outputting the contribution.

[0011] In one aspect of the present disclosure, a recording medium stores a program that causes a computer to acquire parameters of multiple local models of multiple learning participants for inferring a specific phenomenon, integrate the acquired parameters of the multiple local models, perform inference using an integrated model that integrates the parameters of the multiple local models, evaluate the contribution of the local models based on the results of the inference, and output the contribution.

[0012] One example of the effect of the present disclosure is that it can provide an evaluation system that can appropriately evaluate learning participants.

[0013] FIG. 1 is a block diagram showing the configuration of an information processing system in a first embodiment. FIG. 2 is a diagram showing a hardware configuration in which an evaluation system in the first embodiment is realized by a computer device and its peripheral devices. FIG. 3 is a diagram for explaining a threshold value of inference accuracy in the first embodiment. FIG. 4 is a flowchart showing the calculation operation in the first embodiment. FIG. 5 is a block diagram showing the configuration of an information processing system in a modified example of the first embodiment. FIG. 6 is a diagram showing an example of presentation of types of missing data presented by a presentation unit in the first embodiment. FIG. 7 is a flowchart showing the information processing operation in a modified example of the first embodiment. FIG. 8 is a block diagram showing the configuration of an information processing system in a second embodiment. FIG. 9 is a flowchart showing the information processing operation in the second embodiment.

[0014] Next, an embodiment will be described in detail with reference to the drawings.

[0015] [First Embodiment] Fig. 1 is a block diagram showing the configuration of an information processing system 10 according to the first embodiment. The information processing system 10 according to the first embodiment is a system for calculating rewards for each learning participant when an integrated model is generated by integrating multiple local models for inferring specific phenomena held by each learning participant through federated learning. Note that the federated learning may be performed multiple times to generate an integrated model that satisfies predetermined conditions. Examples of learning participants include organizations such as local governments and companies.

[0016] Referring to FIG. 1 , the information processing system 10 includes an evaluation system 100 and multiple learning participant servers 200 (200a, 200b). The evaluation system 100 inputs explanatory variable values ​​into an integrated model to output inference results related to an event. A specific event is, for example, a matter that can be expressed in any format (mathematical formula) using explanatory variables corresponding to factors that influence the event. The model in this embodiment learns the causes and occurrence conditions of an event from past case data, inputs explanatory variable values, and outputs inference results for improvement conditions to resolve the causes of the event. Examples of inference targets include measures to encourage behavioral change among employees to improve employee health and productivity, measures to encourage behavioral change among citizens to reduce health insurance costs in local governments, and measures to promote the use of public facilities such as libraries and gymnasiums. However, the events inferred by the evaluation system 100 are not limited to these.

[0017] The evaluation system 100 includes a parameter acquisition unit 101, an integration unit 102, an inference unit 103, an evaluation unit 104, a calculation unit 105, and an output unit 106. However, the calculation unit 105 is an optional component. Each of the learning participant servers 200 includes a model generation unit 201 (201a, 201b) that generates a model for inferring a specific event, and an input / output unit 202 (202a, 202b) that inputs and outputs parameters to and from the evaluation system 100. In this embodiment, there are two learning participant servers 200, but this is not limited to this. The number of learning participant servers 200 is the same as the number of learning participants participating in the learning. The evaluation system 100, which is an essential component of this embodiment, will be described in detail below.

[0018] 2 is a diagram illustrating an example of a hardware configuration of the evaluation system 100 according to the first embodiment of the present disclosure, implemented by a computer device 500 including a processor. As shown in FIG. 2, the evaluation system 100 includes a central processing unit (CPU) 501, memories such as a read-only memory (ROM) 502 and a random access memory (RAM) 503, a storage device 505 such as a hard disk for storing a program 504, a communication interface (I / F) 508 for network connection, and an input / output interface 511 for inputting and outputting data. In the first embodiment, parameter information received from each learning participant server 200 is input to the evaluation system 100 via the communication I / F 508.

[0019] The CPU 501 runs an operating system to control the entire evaluation system 100 according to the first embodiment of the present invention. The CPU 501 also reads programs and data into memory from a recording medium 506 attached to, for example, a drive device 507. The CPU 501 also functions as the parameter acquisition unit 101, integration unit 102, inference unit 103, evaluation unit 104, calculation unit 105, output unit 106, or parts of these units in the first embodiment, and executes processing or commands in the flowchart shown in FIG. 4, which will be described later, based on the program.

[0020] The recording medium 506 is, for example, an optical disk, a flexible disk, a magneto-optical disk, an external hard disk, or a semiconductor memory. A part of the recording medium in the storage device is a non-volatile storage device, and the program is recorded therein. The program may also be downloaded from an external computer (not shown) connected to a communication network.

[0021] The input device 509 is realized by, for example, a mouse, a keyboard, built-in key buttons, etc., and is used for input operations. The input device 509 is not limited to a mouse, a keyboard, or built-in key buttons, and may be, for example, a touch panel. The output device 510 is realized by, for example, a display, and is used to check output.

[0022] As described above, the first embodiment shown in FIG. 1 is realized by the computer hardware shown in FIG. 2. However, the means for realizing each unit included in the evaluation system 100 of FIG. 1 are not limited to the configuration described above. Furthermore, the evaluation system 100 may be realized by a single physically coupled device, or by two or more physically separated devices connected by wire or wirelessly. For example, the input device 509 and the output device 510 may be connected to the computer device 500 via a network. Furthermore, the evaluation system 100 of the first embodiment shown in FIG. 1 can also be configured using cloud computing or the like.

[0023] The parameter acquisition unit 101 acquires trained model parameters in each of the multiple learning participant servers 200, for example, triggered by an operation for performing federated learning. The model is a model trained by machine learning in each learning participant to output, for example, an inference result regarding a specific event. The machine learning model includes, but is not limited to, a decision tree model, a linear regression model, a logistic regression model, a neural network model, etc.

[0024] The integrating unit 102 integrates the parameters of each model of the multiple models. A known method can be used as a parameter integration method. For example, when integrating the parameters, the weight of the parameters corresponding to each model can be changed depending on the characteristics of each model. The integrating unit 102 applies the parameters obtained in this way to the model and stores them in the storage device 505.

[0025] The parameter acquisition unit 101 may acquire parameters from each learning participant server 200 in a confidential format. In this case, the integration unit 102 integrates the confidential parameters of multiple local models using secure computation. In this embodiment, integrating the confidential parameters of multiple local models using secure computation means that the evaluation system 100 performs machine learning (federated learning) in a distributed state among each learning participant server 200, and integrates the parameters of the trained models using secure computation to generate a new integrated model. The parameters acquired by the parameter acquisition unit 101 from each learning participant server 200 are concealed differences from the parameters before the federated learning. Note that in this embodiment, when federated learning is repeatedly performed, the parameters acquired by the parameter acquisition unit 101 are concealed differences from the parameters before the immediately preceding federated learning.

[0026] In this embodiment, concealment is synonymous with encryption. Secure computation is a method of performing computation while keeping data confidential, and the evaluation system 100, which acquires and integrates parameters of the local model, cannot access the concealed raw data. Secure computation methods that can be used include special encryption corresponding to specific processing such as homomorphic encryption, a trusted execution environment in which processing is performed in an isolated state on hardware, and multi-party computation in which computation processing is performed while keeping secrets shared among multiple servers (secret sharing computation).

[0027] The following is an example of a specific method for secure computation using multi-party computation. For example, concealed data a, a parameter acquired from an arbitrary learning participant server, is secretly shared into shares x1, y1, ..., and x1, y1, ... are transmitted to different servers by an administrator. Furthermore, concealed data b, a parameter acquired from another learning participant server, is secretly shared into shares x2, y2, ..., and x2, y2, ... are transmitted to different servers by an administrator. Next, the concealed data a and the concealed data b communicate with each other while remaining secretly shared, and computation proceeds. Finally, the output shares u, v, ..., which are the computation results of each server, are collected and restored, resulting in the computation result F(a, b). This computation result is a parameter obtained by integrating the parameters of each model. For this reason, when multi-party computation is used as a secure computation method, the integrating unit 102 includes multiple servers. Multi-party computation eliminates the need for encryption key management or an isolated environment, and enables faster computation. The integration unit 102 restores the model parameters thus obtained, and stores the model to which the restored parameters have been applied in the storage device 505 .

[0028] The inference unit 103 is a means for executing inference based on an integrated model obtained by integrating parameters of multiple local models. The inference unit 103 executes inference by inputting explanatory variable values ​​into the integrated model stored in the storage device 505. Note that, if the parameters of the integrated model are kept confidential, the inference unit 103 may execute inference using secure computation, or may execute inference after decrypting the parameters of the integrated model. The inference unit 103 outputs the inference result to the evaluation unit 104.

[0029] The evaluation unit 104 is a means for evaluating the contribution of local models based on the inference results. In this embodiment, the evaluation unit 104 evaluates the contribution of each local model based on, for example, an improvement in the inference accuracy of the integrated model due to the integration of each local model. The evaluation unit 104 determines the contribution of each local model based on the difference between the inference accuracy of the integrated model when the parameters of the local models are integrated and the inference accuracy when the parameters of the local models are not integrated. In this embodiment, the inference accuracy refers to the accuracy rate of the improvement conditions for the causes of the event output by the model. In other words, the inference accuracy indicates the extent to which the event has improved as a result of taking measures to satisfy the improvement conditions output by the model.

[0030] Here, a specific description will be given of a method for evaluating the contribution of each organization's local model when an integrated model is generated by integrating the local models of learning participants A to C. For example, when calculating the contribution of learning participant A's local model, the evaluation unit 104 calculates the inference accuracy of the integrated model when the local models of learning participants B and C are integrated. Then, the evaluation unit 104 compares the inference accuracy of the integrated model when the local models of learning participants B and C are integrated with the inference accuracy of the integrated model when the parameters of all of learning participants A to C's local models are integrated, and evaluates the contribution based on the improvement rate of the inference accuracy.

[0031] Similarly, when evaluating the contribution of the local model of learner B, the evaluation unit 104 evaluates the inference accuracy of the integrated model when the parameters of the local models of learner A and learner C are integrated. Next, the evaluation unit 104 compares the inference accuracy of the integrated model when the local models of learner B and learner C are integrated with the inference accuracy of the integrated model when the parameters of all the local models of learners A to C are integrated, and evaluates the contribution based on the rate of improvement in inference accuracy. The evaluation unit 104 similarly evaluates the contribution of learner C. However, the method of evaluating the contribution of each local model is not limited to this.

[0032] Furthermore, the evaluation unit 104 may add the contribution of the local model if a learning participant participates in the federated learning before the inference accuracy of the integrated model reaches a predetermined threshold. That is, the evaluation unit 104 may add the contribution of the local model if the parameter acquisition unit 101 acquires the parameters of the local model before the inference accuracy of the integrated model reaches a predetermined threshold. Furthermore, if the inference accuracy of the integrated model decreases after the federated learning, the evaluation unit 104 may evaluate the contribution as negative and not integrate the parameters of the local model of the learning participant who participated in the federated learning at that time.

[0033] FIG. 3 is a diagram illustrating the threshold for the inference accuracy of the integrated model. As shown in FIG. 3, the inference accuracy gradually increases according to the amount of model learning. However, once the inference accuracy reaches a predetermined threshold, the increase in inference accuracy slows down. The evaluation unit 104 may add a contribution level to each learning participant who participated in the federated learning until the threshold is reached, at which point the inference accuracy increases significantly. The example of FIG. 3 describes adding a contribution level (a discount on the model usage fee) to each learning participant who participated in the federated learning before the threshold is reached. Furthermore, the evaluation unit 104 may set a threshold after federated learning, such as setting the threshold at 50% of the inference accuracy of the final integrated model, based on the final inference accuracy at which federated learning is fully completed.

[0034] The calculation unit 105 is a means for calculating a reward for a learning participant who provided parameters of a local model based on the contribution of the local model. In the example of the integrated model in which the parameters of the local models of learning participants A to C are integrated as described above, the calculation unit 105 sets the ratio of the calculated contributions as the ratio of rewards to be allocated to each of businesses A to C. As a method of returning rewards to each business, a discount on the usage fee for the integrated model may be used. Furthermore, the calculation unit 105 may free the usage fee for the integrated model in the inference stage for learning participants of the local model who have provided a predetermined or higher contribution.

[0035] In this embodiment, the evaluation unit 104 may evaluate the contribution of the local model based on the time when the learning participant participated, rather than on the inference accuracy. That is, the evaluation unit 104 may evaluate the contribution of the local model based on the time when the parameter acquisition unit 101 acquired the parameters of the local model. The evaluation unit 104 may evaluate the contribution based on the order in which the parameters were acquired from the learning participant or the timing at which they were acquired.

[0036] The output unit 106 is a means for outputting the evaluated contribution. The output unit 106 may output the contribution of each local model to an output device 510 such as a display, or may transmit information regarding the contribution to each learning participant. The output unit 106 may also output information regarding rewards.

[0037] The operation of the evaluation system 100 configured as above will be described with reference to the flowchart of FIG.

[0038] FIG. 4 is a flowchart showing an outline of the operation of the evaluation system 100 in the first embodiment. The processing according to this flowchart may be executed based on program control by the processor described above. The series of processes according to this flowchart do not have to be performed consecutively. For example, steps S101 to S104 and steps S105 to S108 in FIG. 4 may be performed at different times. The process of calculating the reward for the learning participant in step S107 may be skipped. In this case, in step S108, the output unit 106 outputs the degree of contribution.

[0039] 4 , first, the parameter acquisition unit 101 acquires parameters of trained local models for inferring a specific event from each learning participant server 200 (step S101). If the evaluation system 100 performs calculations with anonymity, i.e., if the parameters of the multiple local models acquired by the parameter acquisition unit 101 are in anonymized format (S102; YES), the integration unit 102 integrates the anonymized parameters of the multiple local models by secure computation (step S103). On the other hand, if the evaluation system 100 performs calculations without anonymity, i.e., if the parameters of the multiple local models acquired by the parameter acquisition unit 101 are not in anonymized format (S102; NO), the integration unit 102 integrates the parameters of the multiple local models without using secure computation (step S104).

[0040] The inference unit 103 performs inference using the integrated model (step S105), and then the evaluation unit 104 evaluates the contribution of the local model to the integrated model (step S106). The calculation unit 105 calculates a reward for each learning participant based on the contribution (step S107). The output unit 106 outputs the calculated reward (step S108). This completes the calculation operation of the evaluation system 100.

[0041] In this embodiment, in the evaluation system 100, the evaluation unit 104 evaluates the contribution of the local model to the integrated model. Thus, the evaluation of the learning participants can be performed appropriately. Furthermore, the calculation unit 105 calculates the reward for the learning participant based on the contribution. Thus, the reward for the learning participant can be appropriately set based on the contribution of the local model to the integrated model. Furthermore, in the evaluation system 100, the evaluation unit 104 adds the contribution of the local model if the learning participant participates in the federated learning before the inference accuracy of the integrated model reaches a predetermined threshold. This allows the contribution of learning participants who participate in the federated learning at an early stage to be appropriately evaluated, thereby encouraging each organization to participate even in the early stages of the federated learning.

[0042] Furthermore, in the evaluation system 100, when the parameters of the multiple local models acquired by the parameter acquisition unit 101 are in a confidential format, the integration unit 102 integrates the confidential parameters of the multiple local models using secure computation. This makes it possible to use the integrated model while keeping the parameters of each model confidential.

[0043] [Modification of First Embodiment] Next, a modification of the first embodiment of the present disclosure will be described in detail with reference to the drawings. Below, to the extent that the description of this embodiment is not unclear, description of content that overlaps with the above description will be omitted. FIG. 5 is a block diagram showing the configuration of an information processing system according to the modification of the first embodiment. As shown in FIG. 5, the evaluation system 110 in the information processing system 11 includes a parameter acquisition unit 111, an integration unit 112, an inference unit 113, an identification unit 114, a presentation unit 115, an evaluation unit 116, a calculation unit 117, and an output unit 118. That is, the evaluation system 110 differs from the evaluation system 100 according to the first embodiment at least in that it includes the identification unit 114 and the presentation unit 115.

[0044] In this embodiment, the inference unit 113 inputs explanatory variable values ​​for each type and infers an event using the integrated model. The types of learning data are types classified according to attributes such as age and gender, or personal data such as behavioral history and family structure. The inference unit 113 also calculates the inference accuracy for each type of the integrated model and outputs the results to the identification unit 114.

[0045] The identifying unit 114 is a means for identifying the type of missing learning data based on at least one of the type, event, and inference accuracy of the learning data learned by the integrated model. The identifying unit 114 identifies a type whose inference accuracy is equal to or lower than a predetermined level based on the inference accuracy for each type input from the inference unit 113. The identifying unit 114 may also identify a type for which there is little learning data for the integrated model. In this case, if the amount of learning of past failure case data is equal to or lower than a predetermined level, the identifying unit 114 may identify failure case data as the missing learning data.

[0046] The presentation unit 115 is a means for presenting information for recruiting learning participants. The presentation unit 115 presents, as information for recruiting learning participants, information on rewards for participating in the federated learning or information on the current status of the integrated model. The presentation unit 115 may present this information on a web (World Wide Web) page for recruiting participants in the federated learning, or may send this information to organizations that wish to participate in the federated learning. Furthermore, as shown in FIG. 3 , the presentation unit 115 may present information on rewards while presenting the current inference accuracy of the integrated model and a threshold for the inference accuracy.

[0047] The presenting unit 115 may present the types of learning data identified by the identifying unit 114. Fig. 6 is a diagram showing an example of presentation of the types of missing data presented by the presenting unit 115. The example of Fig. 6 describes that a contribution level is added when missing data and parameters of a local model trained using each missing data are provided.

[0048] In addition to the method of evaluating the degree of contribution of the evaluation unit 104 in the first embodiment, the evaluation unit 116 adds a degree of contribution when a learning participant provides parameters of a local model trained using missing data. Note that the calculation unit 117 and the output unit 118 in this embodiment operate in the same way as the calculation unit 105 and the output unit 106 in the first embodiment, and therefore a description thereof will be omitted here.

[0049] 7 is a flowchart showing an outline of the operation of the evaluation system 110 according to a modification of the first embodiment. The process according to this flowchart is based on the premise that missing types of learning data are presented before parameter integration (steps S114 to S115). Note that the process according to this flowchart may be executed based on program control by the processor described above, as in the first embodiment.

[0050] 7 , first, the inference unit 113 performs inference for each type using the integrated model (step S111). Next, the identification unit 114 identifies the missing types of training data (step S112). The presentation unit 115 presents the types of training data identified by the identification unit 114 (step S113).

[0051] Next, the parameter acquisition unit 111 acquires the parameters of the trained local models for inferring a specific event from each learning participant server 210 (step S114). Next, the integration unit 112 integrates the parameters of the multiple local models (step S115).

[0052] The inference unit 113 performs inference using the integrated model (step S116), and then the evaluation unit 116 evaluates the contribution of the local model (step S117). If the learning participant provides parameters of a local model trained using insufficient data (S118; YES), the evaluation unit 116 adds up the learning participant's contribution (step S119). Next, the calculation unit 117 calculates the learning participant's reward based on the contribution (step S120). On the other hand, if the learning participant does not provide parameters of a local model trained using insufficient data (S118; NO), the evaluation unit 116 does not add up the contribution, and the calculation unit 117 calculates the learning participant's reward based on the contribution (step S120). This completes the calculation operation of the evaluation system 110.

[0053] In the evaluation system 110, the presenting unit 115 presents the type of learning data identified by the identifying unit 114, and the evaluating unit 116 adds up the contribution of a learning participant when the learning participant provides parameters of a local model trained using missing data. This allows learning participants who contribute more to the integrated model to be appropriately evaluated for a contribution commensurate with their contribution.

[0054] Second Embodiment Next, a second embodiment of the present disclosure will be described in detail with reference to the drawings. Below, to the extent that the description of this embodiment is not unclear, descriptions of content that overlaps with the above description will be omitted. In the second embodiment, the information processing system 12 is used to provide each learning participant server with an updated model through federated learning using secure computation. These updated models are used, for example, by each learning participant to infer specific events. The functions of each component in each embodiment of the present disclosure can be realized not only by hardware, as with the computer device shown in FIG. 2, but also by a computer device based on program control.

[0055] <Evaluation System> Fig. 8 is a block diagram showing the configuration of an information processing system 12 including an evaluation system 120 according to a second embodiment of the present disclosure. The evaluation system 120 and learning participant servers 220 (220a, 220b) according to the second embodiment will be described, focusing on differences from the information processing system 10 according to the first embodiment, with reference to Fig. 8. The evaluation system 120 includes a parameter acquisition unit 121, an integration unit 122, a parameter transmission unit 123, an inference unit 124, an evaluation unit 125, a calculation unit 126, and an output unit 127.

[0056] The parameter acquisition unit 121 acquires parameters of the trained model of each learning participant from the learning participant server 220 via the communication I / F 508. Next, the integration unit 122 integrates the received parameters of the concealed models using secure computation and outputs the integrated parameters of the concealed model to the parameter transmission unit 123 in a concealed format. The parameter transmission unit 123 transmits the integrated parameters to each learning participant server 220 via the input / output unit 223. Furthermore, after transmitting the parameters to the learning participant server 220, if the learning participant server 220 retrains the model and updates the parameters, the evaluation system 120 may acquire the updated parameters again. Note that the operations of the inference unit 124, evaluation unit 125, calculation unit 126, and output unit 127 are similar to the operations of the corresponding components in the first embodiment, and therefore will not be described here.

[0057] <Learning participant server> In the second embodiment, the multiple learning participant servers 220 (220a, 220b) include a model generation unit 221 (221a, 221b), a concealment unit 222 (222a, 222b), an input / output unit 223 (223a, 223b), a restoration unit 224 (224a, 224b), a model storage unit 225 (225a, 225b), and a participant inference unit 226 (226a, 226b). The model storage unit 225 stores a model generated by the model generation unit 221.

[0058] The learning participant server 220 updates the model stored in the model storage unit 225 to a model that applies the parameters received from the evaluation system 120. Specifically, the input / output unit 223 receives the parameters in a confidential format and outputs them to the restoration unit 224. The restoration unit 224 then restores the parameters and replaces them with the parameters of the model stored in the model storage unit 225. The participant inference unit 226 then performs inference using the updated model. To improve the accuracy of the inference results by the participant inference unit 226, the learning participant server 220 may perform re-learning based on additionally acquired learning data and transmit the updated parameters to the evaluation system 120. The accuracy of the integrated model can be further improved by repeating the parameter update through learning in each learning participant server 220 and the parameter integration in the evaluation system 120, for example, until a predetermined condition is met. The predetermined condition is stored, for example, in the storage device 505.

[0059] The operation of the information processing system 12 configured as above will be described with reference to the flowchart of FIG.

[0060] 9 is a flowchart showing an outline of the operation of the information processing system 12 according to the second embodiment. The processing according to this flowchart may be executed based on program control by the processor described above. The reward calculation by the evaluation system 120 in steps S212 to S215 may be executed each time the associative learning processing in steps S201 to S210 is executed, or may be executed when the integrated model satisfies predetermined conditions.

[0061] As shown in FIG. 9 , first, in the learning participant server 220, the model generation unit 221 locally generates a model using data held by the learning participant (step S201). Next, the anonymization unit 222 anonymizes the model parameters (step S202), and the input / output unit 223 outputs the anonymized parameters to the evaluation system 120 (step S203). Next, the parameter acquisition unit 121 of the evaluation system 120 acquires the anonymized parameters (step S204). Next, the integration unit 122 integrates the anonymized parameters of the multiple local models using secure computation (step S205). Next, the parameter transmission unit 123 outputs the model parameters integrated by the integration unit 122 to each learning participant server 220 in anonymized form (step S206).

[0062] Next, each learning participant server 220 acquires the integrated parameters in a confidential format through the input / output unit 223 (step S207). Next, the restoration unit 224 restores the parameters in a confidential format (step S208). Next, the learning participant server 220 updates the model stored in the model storage unit 225 by applying the restored parameters (step S209). Next, the learning participant server 220 determines whether a predetermined condition is met (step S210). If the predetermined condition is met (step S210; YES), the participant inference unit 226 performs inference using the updated model (step S211). If the predetermined condition is not met, the learning participant server 220 returns to step S201 (step S210; NO) and executes the flow again.

[0063] Meanwhile, in the evaluation system 120, the inference unit 124 performs inference using the integrated model (step S212), and then the evaluation unit 125 evaluates the contribution of the local model (step S213). The calculation unit 126 calculates the reward for the learning participant based on the contribution (step S214). The output unit 127 outputs the calculated reward (step S215). This completes the information processing operation of the information processing system 12.

[0064] In the second embodiment of the present disclosure, the participant inference unit 226 of each learning participant server 220 performs inference on a specific event using a model to which parameters of the model integrated by the integration unit 122 are applied. This allows each learning participant server 220 to output more accurate inference results.

[0065] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0066] For example, although multiple operations are described in a sequential order in the form of a flowchart, the order of description does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations can be changed within the scope that does not affect the content.

[0067] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0068] (Supplementary Note 1) An evaluation system comprising: a parameter acquisition means for acquiring parameters of a plurality of local models of a plurality of learning participants for inferring a specific phenomenon; an integration means for integrating the acquired parameters of the plurality of local models; an inference means for executing the inference using an integrated model obtained by integrating the parameters of the plurality of local models; an evaluation means for evaluating the contribution of the local models based on the result of the inference; and an output means for outputting the contribution.

[0069] (Supplementary Note 2) The evaluation system according to Supplementary Note 1, further comprising a calculation means for calculating a reward for the learning participant who provided the parameters of the local model based on the degree of contribution, and the output means outputs the reward.

[0070] (Supplementary Note 3) The evaluation system according to Supplementary Note 1 or Supplementary Note 2, wherein the parameter acquisition means acquires parameters of the plurality of local models in a confidential format, and the integration means integrates the parameters of the plurality of local models using secure computation.

[0071] (Supplementary Note 4) An evaluation system comprising: an inference means for performing inference regarding a specific event based on an integrated model in which parameters of multiple local models are integrated through federated learning using secure computation; an evaluation means for evaluating the contribution of the local model based on the result of the inference; and an output means for outputting the contribution.

[0072] (Supplementary Note 5) The evaluation system according to any one of Supplementary Notes 1 to 4, wherein the evaluation means evaluates the contribution of the local models based on a change in inference accuracy of the integrated model due to integration of parameters of the plurality of local models.

[0073] (Supplementary Note 6) The evaluation system according to any one of Supplementary Notes 1 to 4, wherein the evaluation means evaluates the degree of contribution of the local model based on the time when the plurality of learning participants participated in the federated learning.

[0074] (Supplementary Note 7) The evaluation system according to Supplementary Note 5 or Supplementary Note 6, wherein the evaluation means adds the contribution of the local model when the plurality of learning participants participate in the federated learning before the inference accuracy of the integrated model reaches a predetermined threshold.

[0075] (Supplementary Note 8) The evaluation system according to any one of Supplementary Notes 1 to 7, further comprising a presentation means for presenting information for recruiting study participants.

[0076] (Supplementary Note 9) The evaluation system according to Supplementary Note 8, wherein the presenting means presents the current inference accuracy of the integrated model and a threshold value for the inference accuracy.

[0077] (Supplementary Note 10) The evaluation system according to Supplementary Note 8, further comprising: a specifying means for specifying a type of missing learning data based on at least one of the type of learning data trained into the integrated model, the content of the event, and the inference accuracy; and the presenting means for presenting the specified type of learning data.

[0078] (Supplementary Note 11) The evaluation system according to Supplementary Note 10, wherein the evaluation means adds a contribution of a local model generated based on the missing training data.

[0079] (Supplementary Note 12) An information processing system having a plurality of learning participant servers and an evaluation system according to any one of Supplements 1 to 11, wherein each of the learning participant servers comprises: a model storage means for storing a trained model for inferring about a specific event; an input / output means for inputting, in a confidential format, parameters of the stored model that have been updated by federated learning using secure computation; a restoration means for restoring the input parameters; and a participant inference means for applying the restored parameters to the stored model to update the model and make an inference about the specific event.

[0080] (Supplementary Note 13) An evaluation method in which a computer acquires parameters of local models of multiple learning participants for inferring a specific phenomenon, integrates the acquired parameters of the local models, performs the inference using an integrated model obtained by integrating the parameters of the local models, evaluates the contribution of the local models based on the results of the inference, and outputs the contribution.

[0081] (Appendix 14) A recording medium storing a program that causes a computer to execute the following steps: acquire parameters of multiple local models of multiple learning participants for inferring a specific phenomenon; integrate the acquired parameters of the multiple local models; perform the inference using an integrated model that integrates the parameters of the multiple local models; evaluate the contribution of the local models based on the results of the inference; and output the contribution.

[0082] 10, 11, 12 Information processing system 100, 110, 120 Evaluation system 101, 111, 121 Parameter acquisition unit 102, 112, 122 Integration unit 103, 113, 124 Inference unit 104, 116, 125 Evaluation unit 105, 117, 126 Calculation unit 106, 118, 127 Output unit 114 Identification unit 115 Presentation unit 123 Parameter transmission unit 200, 210, 220 Learning participant server 201, 211, 221 Model generation unit 202, 212, 223 Input / output unit 222 Concealment unit 224 Restoration unit 225 Model storage unit 226 Participant inference unit

Claims

1. A parameter acquisition means for acquiring parameters of a plurality of local models of a plurality of learning participants for inferring a specific event; An integration means for integrating the obtained parameters of the plurality of local models; an inference means for executing the inference by using an integrated model obtained by integrating parameters of the plurality of local models; An evaluation means for evaluating a contribution of the local model based on a result of the inference; An evaluation system comprising: an output means for outputting the degree of contribution.

2. A calculation means for calculating a reward for a learning participant who has provided a parameter of the local model based on the degree of contribution, The evaluation system according to claim 1 , wherein the output means outputs the reward.

3. the parameter acquisition means acquires parameters of the plurality of local models in a concealed format; The evaluation system according to claim 1 , wherein the integrating means integrates parameters of the plurality of local models using secure computation.

4. An inference means for executing an inference regarding a specific event based on an integrated model in which parameters of a plurality of local models are integrated by federated learning using secure computation; An evaluation means for evaluating a contribution of a local model based on a result of the inference; An evaluation system comprising: an output means for outputting the degree of contribution.

5. The evaluation system according to claim 1 , wherein the evaluation means evaluates the contribution of the local models based on a change in inference accuracy of the integrated model caused by integration of parameters of the plurality of local models.

6. The evaluation system according to claim 1 , wherein the evaluation means evaluates the degree of contribution of the local model based on a time when the plurality of learning participants participated in the federated learning.

7. The evaluation system according to claim 5 , wherein the evaluation means adds a contribution of the local model when the plurality of learning participants participate in the federated learning before the inference accuracy of the integrated model reaches a predetermined threshold.

8. The evaluation system according to claim 1 , further comprising a presentation means for presenting information for recruiting study participants.

9. The evaluation system according to claim 8 , wherein the presenting means presents a current inference accuracy of the integrated model and a threshold value of the inference accuracy.

10. The method further includes a specification unit that specifies a type of insufficient learning data based on at least one of the type of learning data trained in the integrated model, the content of the event, and the inference accuracy, The evaluation system according to claim 8 , wherein the presenting means presents the identified type of the learning data.

11. The evaluation system according to claim 10 , wherein the evaluation means adds a contribution of a local model generated based on the missing training data.

12. An information processing system having a plurality of learning participant servers and the evaluation system according to any one of claims 1 to 11, Each of the learning participant servers has a model storage means for storing a trained model for making inferences regarding a specific event; an input / output means for inputting, in a concealed format, parameters updated by federated learning using secure computation for the stored parameters of the model; A restoring means for restoring the input parameters; and a participant reasoning means for applying the restored parameters to the stored model to update the model and to make a reasoning regarding the particular event.

13. The computer obtaining parameters of a plurality of local models of a plurality of learning participants for inferring a particular event; Integrating the obtained parameters of the plurality of local models; performing the inference using an integrated model that integrates parameters of the plurality of local models; Evaluating the contribution of the local model based on the result of the inference; The evaluation method further comprises outputting the degree of contribution.

14. obtaining parameters of a plurality of local models of a plurality of learning participants for inferring a particular event; Integrating the obtained parameters of the plurality of local models; performing the inference using an integrated model that integrates parameters of the plurality of local models; Evaluating the contribution of the local model based on the result of the inference; A program for causing a computer to execute the steps of: