Recompense presentation device, recompense presentation method, and recompense presentation program

Abstract

This recompense presentation device can assist in decision making by a user by comprising: a model acquisition unit that acquires, from providers of models, a plurality of models for executing a prescribed task; an evaluation unit that uses some or all of the plurality of models to evaluate the degree of contribution of each model to an operation model dynamically configured during an operation; and a recompense presentation unit that presents, to the providers of the models, recompense determined in accordance with the degree of contribution of each model.

Description

Reward presentation device, reward presentation method, and reward presentation program

[0001] The present disclosure relates to techniques for offering rewards.

[0002] Techniques for presenting rewards are known. For example, Patent Literature 1 discloses a technique for presenting rewards to multiple analysts who analyze information according to a user's purpose of use. In this technique, rewards are determined based on the order in which the models generated by each analyst are provided to the user as analysis results and the performance of each model.

[0003] Japanese Patent Application Publication No. 2019-46177

[0004] Here, there is an operation mode in which an operation model is dynamically configured during operation using some or all of the models provided by each of multiple providers. In such an operation mode, the usage status indicating the situation in which each model is used during operation may be variable. However, in the technology described in Patent Document 1, the performance of each model referenced to determine the compensation for each analyst is fixedly determined based on the data used to generate the model. Therefore, the technology described in Patent Document 1 has a problem in that the compensation for each of the multiple providers cannot reflect the variable usage status of each model.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and one exemplary purpose thereof is to provide a technology that, when an operational model is dynamically constructed during operation using some or all of the models provided by each of multiple providers, reflects the fluctuating usage status of each model in the compensation paid to each of multiple providers.

[0006] A reward presentation device according to an exemplary aspect of the present disclosure includes a model acquisition means for acquiring a plurality of models that perform specified tasks from the providers of each model, an evaluation means for evaluating the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the plurality of models, and a reward presentation means for presenting a reward determined according to the contribution of each model to the provider of the model.

[0007] A reward presentation method according to an exemplary aspect of the present disclosure includes a model acquisition process in which at least one processor acquires multiple models that perform a specified task from the provider of each model; an evaluation process in which the at least one processor evaluates the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the multiple models; and a reward presentation process in which the at least one processor presents a reward determined according to the contribution of each model to the provider of the model.

[0008] A reward presentation program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as a reward presentation device, and causes the computer to function as a model acquisition means that acquires multiple models that perform specified tasks from the providers of each model, an evaluation means that evaluates the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the multiple models, and a reward presentation means that presents a reward determined according to the contribution of each model to the provider of the model.

[0009] According to one exemplary aspect of the present disclosure, when an operational model is dynamically constructed during operation using some or all of the models provided by each of multiple providers, one exemplary effect is provided in which a technology can be provided that reflects the fluctuating usage status of each model in the compensation paid to each of the multiple providers.

[0010] FIG. 1 is a block diagram showing the configuration of a reward presentation device according to the present disclosure. FIG. 2 is a flow diagram showing the flow of a reward presentation method according to the present disclosure. FIG. 3 is a block diagram showing an overview of the configuration of a reward presentation system according to the present disclosure. FIG. 4 is a block diagram showing the functional configurations of a reward presentation device and a model providing device according to the present disclosure. FIG. 5 is a diagram showing an example of a model pool table according to the present disclosure. FIG. 6 is a diagram showing an example of an evaluation table according to the present disclosure. FIG. 7 is a diagram showing an example of a usage table according to the present disclosure. FIG. 8 is a flow diagram showing the flow of a reward presentation method according to the present disclosure. FIG. 9 is a sequence diagram explaining a specific example of a reward presentation method according to the present disclosure. FIG. 10 is a block diagram showing the configuration of a reward presentation device according to the present disclosure. FIG. 11 is a block diagram showing an overview of the configuration of a reward presentation system according to the present disclosure. FIG. 12 is a block diagram showing the configuration of a reward presentation device according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Reward Presentation Device 1) The configuration of the reward presentation device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the reward presentation device 1. As shown in FIG. 1, the reward presentation device 1 includes a model acquisition unit 11, an evaluation unit 12, and a reward presentation unit 13. The model acquisition unit 11 acquires multiple models that execute predetermined tasks from the providers of each model. The evaluation unit 12 evaluates the contribution of each model in an operation model that is dynamically constructed during operation using some or all of the multiple models. The reward presentation unit 13 presents a reward determined according to the contribution of each model to the provider of the model.

[0014] (Effects of the Reward Presentation Device 1) As described above, the reward presentation device 1 employs a configuration including the above-described model acquisition unit 11, evaluation unit 12, and reward presentation unit 13. Therefore, the reward presentation device 1 has the effect of being able to reflect the fluctuating usage status of each model in the reward for each of the multiple providers when an operational model is dynamically constructed during operation using some or all of the models provided by each of the multiple providers.

[0015] (Flow of Reward Presentation Method S1) The flow of reward presentation method S1 will be described with reference to FIG. 2. FIG. 2 is a flow diagram showing the flow of reward presentation method S1. As shown in FIG. 2, reward presentation method S1 includes a model acquisition process S11, an evaluation process S12, and a reward presentation process S13. In the model acquisition process S11, at least one processor acquires multiple models that execute a predetermined task from the provider of each model. In the evaluation process S12, at least one processor evaluates the contribution of each model in an operation model that is dynamically constructed during operation using some or all of the multiple models. In the reward presentation process S13, at least one processor presents a reward determined according to the contribution of each model to the provider of the model.

[0016] (Effects of Reward Presentation Method S1) As described above, reward presentation method S1 employs a configuration including the above-described model acquisition process S11, evaluation process S12, and reward presentation process S13. Therefore, reward presentation method S1 has the effect of being able to reflect the fluctuating usage status of each model in the reward for each of the multiple providers when an operational model is dynamically constructed during operation using some or all of the models provided by each of the multiple providers.

[0017] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0018] (Overview of the Configuration of Reward Presentation System 1A) An overview of the configuration of the reward presentation system 1A will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an overview of the configuration of the reward presentation system 1A. The reward presentation system 1A includes a reward presentation device 10A and multiple model providing devices 20_1, 20_2, .... The reward presentation device 10A is a computer associated with an manager AD. The manager AD is a user or organization that manages an operation model based on multiple model MLs. The model providing device 20_1 is a computer associated with a provider DS1. The provider DS1 is a user or organization that provides the model ML. The model providing device 20_2 is a computer associated with a provider DS2. The provider DS2 is a user or organization that provides the model ML. When there is no need to particularly distinguish between the multiple model providing devices 20_1, 20_2, ..., they will also be referred to as model providing devices 20. When there is no need to particularly distinguish between the multiple providers DS1, DS2, ..., they will also be referred to as providers DS. 3 shows two model providing devices 20, the reward providing system 1A only needs to include at least one model providing device 20. The number of model providing devices 20 may also be three or more. The reward providing device 10A is communicatively connected to a plurality of model providing devices 20 via a network NW. The network NW may include some or all of a wireless LAN (Local Area Network), a wired LAN, and a WAN (Wide Area Network), but is not limited to these.

[0019] (Functional Configuration of Reward Presentation Device 10A) The functional configuration of the reward presentation device 10A will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configurations of the reward presentation device 10A and the model providing device 20. The functional configuration of the model providing device 20 will be described later, and first the functional configuration of the reward presentation device 10A will be described.

[0020] The reward presentation device 10A includes a control unit 110, a storage unit 120, and a communication unit 130. The control unit 110 controls each unit of the reward presentation device 10A. The storage unit 120 stores various data and programs used by the control unit 110. The communication unit 130 communicates with other devices under the control of the control unit 110. When the reward presentation device 10A is configured as a computer including a processor, memory, and communication interface, the control unit 110 is realized by the processor, the storage unit 120 is realized by the memory, and the communication unit 130 is realized by the communication interface.

[0021] (Functions of the control unit 110) The control unit 110 includes a model acquisition unit 11, an evaluation unit 12, a reward presentation unit 13, and an operation model construction unit 14.

[0022] The operational model configuration unit 14 dynamically configures the operational model during operation using some or all of the multiple model MLs. Dynamic configuration refers to configuring or reconfiguring the operational model after operation has begun. An example of the configuration of the operational model will be described. For example, the operational model is configured to input input data to each model ML constituting the operational model and output a representative value of the output from each model ML. The output from each model ML is assigned a weight for that model ML. For example, the representative value may be, but is not limited to, a weighted average of the outputs from each model ML. In other words, "configuring the operational model" may mean determining the weights of each of the multiple model MLs. For example, a model ML with a weight of 0 is not included in the operational model. A model ML with a weight greater than 0 is included in the operational model. If the sum of the weights is 1, when a weight of 1 is assigned to a certain model ML, only that model ML is included in the operational model. In other words, the operational model can be configured by the model ML selected from the multiple model MLs. When reconfiguring the operational model by selecting one model ML to which a weight of 1 is assigned, the operational model configuration unit 14 is also referred to as switching the model MLs that configure the operational model. Also, when assigning weights that total 1 to two or more model MLs, the operational model configuration unit 14 is also referred to as configuring the operational model as an ensemble model. The operational model configuration unit 14 may determine the weight of each model ML based on the performance of that model ML for the operational data. Details of the "performance of the model ML for the operational data" will be described later.

[0023] The model acquisition unit 11 is configured as described in the first exemplary embodiment, and in addition thereto, is configured as follows. The model acquisition unit 11 acquires an updated model (model MLk_[i+1]) obtained by updating each model MLk_i (k = 1, 2, ..., i = 1, 2, ...) from the provider DSk of the model MLk_i. The model acquisition unit 11 also includes the acquired updated model (model MLk_[i+1]) among the targets that may constitute the operational model. For example, the model acquisition unit 11 may acquire the updated model (model MLk_i+1) by requesting the provider DSk to update the model MLk_i. Alternatively, the model acquisition unit 11 may acquire the updated model (model MLk_[i+1]) without requesting the provider DSk. The model acquisition unit 11 may also acquire the updated model (model MLk_[i+1]) by providing information for improving the performance of each model MLk_i to the provider DSk of the model MLk_i. When there is no need to particularly distinguish between models ML1_1, ML1_2, ML2_1, etc., they are also simply referred to as model MLs. Details of the model MLs will be described later. The model acquisition unit 11 may acquire one model ML from one provider DS, or may acquire multiple model MLs.

[0024] The information for improving performance may be, for example, information indicating factors of prediction errors, information indicating performance degradation conditions, etc. Factors of prediction errors are factors that cause prediction results to deviate from the true value. As a specific example, when model ML predicts product sales, an example of information indicating factors of prediction errors is information such as "day of the week," which was not included in the original learning data. Furthermore, performance degradation conditions are conditions that are estimated to be commonly met when performance degradation of the operational model is observed. As a specific example, when model ML predicts product sales, an example of a performance degradation condition is a condition that performance degradation is observed on "Monday."

[0025] The evaluation unit 12 is configured as described in the first exemplary embodiment, and in addition, is configured as follows. For example, the evaluation unit 12 may evaluate the contribution of each model ML based on the weight of the model ML used to configure the operation model. Here, the weight of each model ML may be determined based on the performance of the model ML with respect to the operation data. For example, the weight of each model ML may be determined to be a larger value the better the performance of the model ML. In other words, the evaluation unit 12 may evaluate that the better the performance of a certain model ML, the higher the contribution of the model ML. Note that the weight of each model ML may be determined based on an input operation of the manager AD.

[0026] The "performance of the model ML with respect to operational data" will be described. The "performance of the model ML with respect to operational data" is expressed by an evaluation index indicating an evaluation of output data obtained by inputting operational data into the model ML. The operational data is data different from the learning data used to generate the model ML. The evaluation index may be, for example, a numerical value indicating the effect of using the model ML (e.g., prediction accuracy, precision, etc.) or a numerical value indicating the business effect (e.g., excess inventory, unsold loss, etc.). For example, the evaluation unit 12 may calculate an evaluation index indicating the "performance of the model ML with respect to operational data" based on output data and true values ​​(or estimated true values) obtained by inputting operational data into the model ML. For example, a performance index when the model ML predicts product sales based on the weather for that day will be described. In this case, the evaluation unit 12 may calculate an evaluation index indicating the performance of the model ML based on a sales forecast value obtained by inputting the weather for that day into the model ML, and actual sales for that day (true values) or estimated sales values ​​(estimated true values).

[0027] Next, we will explain a case where the reward presentation device 10A is used in the healthcare field, such as a hospital. As an example, assume that the model ML is a model that predicts a hospital's bed occupancy rate one week from now based on explanatory variables such as the date of the month, day of the week, season, current bed occupancy rate, moving average of past bed occupancy rates, local infectious disease outbreaks, and number of patients visiting the hospital. In this case, the evaluation unit 12 may calculate an operational evaluation index based on the bed occupancy rate obtained by inputting the explanatory variables into the model ML and the actual bed occupancy rate (true value) on that day. This allows hospital staff to make decisions regarding reward determination.

[0028] Furthermore, for example, the evaluation unit 12 may evaluate the contribution of each model ML based on the performance of the operation model with respect to the operation data. For example, the evaluation unit 12 may evaluate that the better the performance of the operation model, the higher the contribution of each model ML. "Performance of the operation model with respect to the operation data" can be explained in the same way by replacing "model ML" with "operation model" in the above explanation of "performance of the model ML with respect to the operation data."

[0029] Furthermore, for example, the evaluation unit 12 may evaluate the contribution of each model ML based on the period during which the model ML is included in the operation model or the number of times an operation model including the model ML is used. The period during which a certain model ML is included in the operation model is, in other words, the usage period during which the model ML is used as an operation model. Hereinafter, the "period during which the model ML is included in the operation model" will be simply referred to as the "usage period of the model ML." Furthermore, the "number of times an operation model including the model ML is used" will be simply referred to as the "number of times the model ML is used." For example, the evaluation unit 12 may evaluate the contribution of a certain model ML as being higher the longer the usage period of the model ML and / or the more times it is used.

[0030] A specific example of the usage period of model ML will be described. For example, assume that an operation model is configured by model ML1_1 on weekdays in a certain week (in other words, the weight of model ML1_1 is 1). Also, assume that an operation model is configured by models ML1_2 and ML2_1 on weekends in the same week (in other words, the sum of the weights of models ML1_2 and ML2_1 is 1). In this case, the usage period of model ML1_1 in the week is five weekday days. Also, the usage period of models ML1_2 and ML2_1 is two weekend days, respectively.

[0031] Next, a specific example of the number of times a model ML is used will be described. For example, in the above example, assume that the operation model is used three times on weekdays and ten times on weekends. In this case, the number of times model ML1_1 is used on weekdays in that week is three. Furthermore, the number of times each of models ML1_2 and ML2_1 is used is ten times on weekends.

[0032] Furthermore, for example, the evaluation unit 12 may evaluate the contribution of each model ML based on the calculation time required for inferring the model ML. For example, the evaluation unit 12 may evaluate the contribution of a model ML as higher the shorter the calculation time required for inferring the model ML. For example, the evaluation unit 12 may evaluate the contribution of each model ML based on the amount of memory required to store the model ML. For example, the smaller the memory amount of a model ML, the less storage capacity is consumed, and therefore the evaluation unit 12 may evaluate the contribution of the model ML as higher. For example, the evaluation unit 12 may evaluate the contribution of each model ML based on the number of hyperparameters of the model ML. For example, the smaller the number of hyperparameters of a model ML, the less storage capacity is consumed, and therefore the evaluation unit 12 may evaluate the contribution of the model ML as higher. For example, the evaluation unit 12 may evaluate the contribution of each model ML based on the rarity of the model ML. For example, the evaluation unit 12 may evaluate a model ML as having a higher contribution level because the rarer the model ML, the more resistant it is to environmental changes. It should be noted that known techniques can be used as an index indicating rarity. Furthermore, for example, the evaluation unit 12 may evaluate the contribution level of each model ML according to the contribution level of each sample of the model ML. For example, the evaluation unit 12 may evaluate a model ML as having a higher contribution level the greater the cumulative value of the contribution level of each sample of the model ML.

[0033] The evaluation unit 12 may evaluate the degree of contribution of each model ML according to a combination of some or all of the above-mentioned elements.

[0034] The reward presentation unit 13 is configured as described in the first exemplary embodiment, and is also configured as follows. For example, after the operation of the operation model begins, the reward presentation unit 13 presents a reward determined according to the contribution of each model ML. Here, the contribution of each model ML may fluctuate after the operation begins. This is because the factors referenced to determine the contribution as described above (each model ML's weight, performance, usage period, number of uses, calculation time, memory amount, number of hyperparameters, rarity, contribution per sample, or other factors) may fluctuate over time. Therefore, by presenting a reward after the operation of the operation model begins, the reward presentation unit 13 can make the provider DS recognize that the variable contribution of each model ML is reflected in the reward.

[0035] The timing at which the reward presentation unit 13 determines the reward and the timing at which the reward is paid to the provider DS may or may not correspond. If they correspond, for example, the reward is paid to the provider DS each time it is determined. If they do not correspond, for example, when a specified payment condition is met, the total of the unpaid rewards determined up to that point may be paid to the provider DS. The specified payment condition may be, for example, at a specified period such as the end of the month, when the total unpaid rewards exceeds a threshold, at the end of the operation of the model ML, or the like, but is not limited to these.

[0036] (Information Stored in the Storage Unit 120) As shown in FIG. 4, the storage unit 120 stores multiple models ML1_1, ML1_2, ML2_1, ..., a model pool table, an evaluation table, and a usage table. In this example, models ML1_1 and ML1_2 are models provided by provider DS1, and model ML1_2 is an updated model obtained by updating model ML1_1. Model ML2_1 is a model provided by provider DS2. Model MLk_i indicates a model provided by provider DSk. Furthermore, model MLk_[i+1] represents an updated model obtained by updating model MLk_i. However, the number of models ML provided by each provider DS may be one or more. Furthermore, the number of models ML provided by each provider DS does not have to be the same. Although three models ML are shown in FIG. 4, the storage unit 120 stores at least one model ML. Note that before the execution of the reward presentation method S1A described below begins, no model ML may be stored. The number of model MLs may be four or more. One or more model MLs constitute a model pool. For example, an operation model is constructed using part or all of the model pool. In other words, the model pool represents objects that can constitute an operation model.

[0037] The multiple model MLs perform the same task. Each model ML is a mathematical function or process that solves the task. For example, the model ML may be a machine learning model or a simple statistical model. The model ML may also include optimization. For example, the model ML may be a model that determines an optimal order quantity based on a demand forecast. The learning algorithm for generating the model MLs is not particularly limited and may be an existing learning algorithm. For example, the learning algorithm may be a random forest, a support vector machine, a naive Bayesian model, a piecewise linear model using FAB inference (Factorized Asymptotic Bayesian Inference), or a neural network. When a model pool is composed of multiple model MLs, all of the multiple model MLs may be machine learning models, or at least one may be a machine learning model and at least one may be a non-machine learning model (e.g., a statistical model). When at least two of the multiple model MLs are machine learning models, the learning algorithms for generating the at least two model MLs may be the same or different.

[0038] The model pool table contains information about one or more model MLs included in the model pool. An example of a model pool table will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating an example of a model pool table. Each row in FIG. 5 represents a record related to one model ML and includes a model ID, a provider ID, and a registration date and time. The model ID is information for identifying the model ML. The provider ID is information for identifying the provider DS. For ease of explanation, the reference symbols for each model ML shown in FIG. 4 are referred to as model IDs, and the reference symbols for each provider DS are referred to as provider IDs. However, the model ID and provider ID are not limited to these. The registration date and time indicate the time when the model ML was registered in the model pool. In this example, the "date" is listed as the registration date and time, but the time may also be included in addition to the date.

[0039] The evaluation table contains information including evaluation indicators for each use of the operation model. An example of the evaluation table will be described with reference to FIG. 6. FIG. 6 is a diagram illustrating an example of the evaluation table. Each row in FIG. 6 represents a record recorded for each use of the operation model, and includes a usage date and time, a model output, a true value, and an evaluation indicator. The usage date and time indicates the time when the operation model was used. In this example, a "date" is entered as the usage date and time, but, for example, time may be included in addition to the date. The model output is information indicating output data from the operation model. The output data is output by inputting operation data at the time of use into the operation model. Note that the operation data input into the operation model may differ each time it is used. The true value indicates the true value corresponding to the operation data. For example, if the operation model inputs the weather on the day and outputs a sales forecast value for a product on that day, the model output may be a weighted average of the sales forecast values ​​output by inputting the operation data, "weather data at the usage date and time," into each model ML constituting the operation model. The true value is the sales amount for that day. The evaluation index is an example of an index that indicates the performance of the operation model, and in this example, is the prediction accuracy calculated from the model output and the true value, but is not limited to this.

[0040] The usage table contains information regarding the model configuration of an operation model. An example of the usage table will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating an example of the usage table. Each row in FIG. 7 represents a record that is registered each time a new operation model is configured, and includes an operation period, a model configuration, and an evaluation index. The operation period indicates the period during which the operation model is operated with the model configuration, and includes the operation start date and time and the operation end date and time. In this example, the operation start date and the operation end date and time are described as "dates," but they may also include time in addition to the date, for example. The model configuration indicates the configuration of the operation model during the operation period and includes weights assigned to each model ML. The evaluation index is an evaluation index that indicates the performance of the operation model during the operation period. For example, the evaluation index may be a representative value of the evaluation index for each usage recorded in the evaluation table during the operation period. The representative value of the evaluation index for each usage may be, but is not limited to, the average value, median, maximum value, minimum value, etc. of the evaluation index for each usage.

[0041] In the following, adding a new record to the model pool table, usage table, or evaluation table will be referred to as "registering" that record. Also, adding information to an already registered record will be referred to as "recording the information in that record."

[0042] (Functional Configuration of Model Providing Device 20) The functional configuration of the model providing device 20 will be described with reference to FIG. 4. As shown in FIG. 4, the model providing device 20 includes a control unit 210, a storage unit 220, a communication unit 230, an input unit 240, and a display unit 250. The control unit 210 controls each unit of the model providing device 20. The storage unit 220 stores various data and programs used by the control unit 210. The communication unit 230 communicates with other devices under the control of the control unit 210. The input unit 240 accepts user operations. The display unit 250 displays information under the control of the control unit 210. When the model providing device 20 is configured by a computer including a processor, a memory, and a communication interface, the control unit 210 is realized by the processor, the storage unit 220 is realized by the memory, and the communication unit 230 is realized by the communication interface. The input unit 240 is configured by a keyboard, a mouse, a touch pen, a touch pad, or the like, which is built into or connected to the computer. The display unit 250 is configured by a liquid crystal display, an organic EL (Electro Luminescence) display, or the like, which is built into or connected to the computer. The input unit 240 and the display unit 250 may be configured by an integrally formed touch panel.

[0043] (Functions of the control unit 210) The control unit 210 includes a request acquisition unit 21, a model generation unit 22, and a reward presentation unit 23. The request acquisition unit 21 acquires information requesting the provision of a model ML from the reward presentation device 10A. The requested information includes tasks to be executed by the model ML and learning data that can be used to create the model ML. For example, the request acquisition unit 21 may present the requested information to the provider DS by displaying it on the display unit 250. The model generation unit 22 generates a model ML in response to a request from the reward presentation device 10A. Specifically, the model generation unit 22 uses the learning data presented by the reward presentation device 10A to generate a model ML to execute the task presented by the reward presentation device 10A. For example, the model generation unit 22 may generate a model ML based on an operation by the provider DS. The reward presentation unit 23 presents the reward presented by the reward presentation device 10A to the provider DS. For example, the reward presenter 23 may present the reward to the provider DS by displaying it on the display unit 250 .

[0044] (Flow of reward presentation method S1A) The reward presentation method S1A executed by the reward presentation system 1A configured as above will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing the flow of reward presentation method S1A. As shown in Fig. 8, reward presentation method S1A includes steps S101 to S131.

[0045] In step S101, the model acquisition unit 11 of the reward presentation device 10A presents to each of the multiple providers DSk the tasks to be executed by the model MLk_i and training data that can be used to generate the model MLk_i. The tasks and training data presented to each of the multiple providers DSk are the same. Specifically, the model acquisition unit 11 presents the tasks and training data to each of the multiple model providing devices 20_k by transmitting the tasks and training data. After step S101 is performed for the first time, it may be performed again as described below. In this case, the model MLk_[i-1] has already been acquired. In addition to the tasks and training data, the model acquisition unit 11 may provide each provider DS with information for improving the performance of the model MLk_[i-1]. Specific examples of the information for performance improvement are as described above, and therefore will not be described in detail again.

[0046] In step S102, the model acquisition unit 11 requests the provider DSk to provide a model MLk_i using the presented task and learning data. Specifically, the model acquisition unit 11 requests the provider DSk to provide the model MLk_i by transmitting information to the model providing device 20_k instructing it to generate a model MLk_i using the presented task and learning data. Note that steps S101 to S102 may be executed again after the first execution, as described below. In this case, the model MLk_[i-1] has already been acquired. In other words, by executing steps S101 to S102 again, the model acquisition unit 11 requests the provider DSk to update the model MLk_[i-1].

[0047] In step S103, the request acquisition unit 21 of the model providing device 20_k acquires the task, the learning data, and information instructing the creation of the model MLk_i in accordance with steps S101 and S102. The request acquisition unit 21 also displays the acquired information on the display unit 250.

[0048] In step S104, the model generation unit 22 generates a model MLk_i that executes the task presented by the reward presentation device 10A, and provides the model MLk_i to the reward presentation device 10A. Specifically, the model generation unit 22 generates a model MLk_i that executes the task using the learning data presented by the reward presentation device 10A, and provides the model MLk_i by transmitting the generated model MLk_i to the reward presentation device 10A.

[0049] In step S105, the model acquisition unit 11 of the reward presentation device 10A acquires a model MLk_i from a model providing device 20_k. The models MLk_i acquired from the multiple model providing devices 20_k constitute a model pool for constructing an operation model. As described above, step S105 may be executed in response to the re-execution of steps S101 and S102. In this case, model MLk_[i-1] has already been acquired. In other words, in step S105, the model acquisition unit 11 acquires model MLk_i, which is an updated model. In this case, model MLk_i, which is an updated model, is added to the model pool as a target for constructing an operation model.

[0050] In step S106, the model acquisition unit 11 determines whether the model MLk_i acquired from the model providing device 20_k satisfies specifications. The specifications include the above-mentioned tasks. The specifications may also include a requirement that the performance of the model MLk_i on the training data meets a predetermined level.

[0051] If it is determined that the model MLk_i acquired from the model providing device 20_k does not satisfy the specifications (No in step S106), the model acquiring unit 11 repeats the process from step S102. In other words, the model acquiring unit 11 again requests the provider DSk to provide a model MLk_i that satisfies the specifications.

[0052] If it is determined that the model MLk_i acquired from the model providing device 20_k satisfies the specifications (Yes in step S106), in step S107, the model acquisition unit 11 registers a new record in the model pool table and records the model ID and registration date and time of the model MLk_i. The registration date and time may be, for example, the date and time corresponding to the time when S107 is executed. By executing step S107 for the models MLk_i acquired from each of the multiple model providing devices 20_k, multiple models ML that can be targets for configuring an operation model are registered in the model pool table.

[0053] Steps S101 to S107 constitute an example of a model acquisition process for acquiring a plurality of models MLk_i that execute a predetermined task from the provider DSk of each model MLk_i.

[0054] In step S108, the control unit 110 determines whether or not a configured operation model currently exists. If an operation model exists, the acquired model MLk_i is an updated model, and if an operation model does not exist, the acquired model MLk_i is not an updated model.

[0055] If the determination in step S108 is No, the process proceeds to step S109. In step S109, the operation model configuration unit 14 configures an operation model using some or all of the multiple models MLk_i registered in the model pool table. For example, the operation model configuration unit 14 may configure the operation model by determining a weight for each model MLk_i based on the performance of each model MLk_i with respect to the operation data.

[0056] Following step S109, the process of step S130 is executed. In step S130, the control unit 110 starts using the operation model configured in step S109. In addition, the operation model configuration unit 14 registers a new record in the usage table and records the operation start date and time and the model configuration. As the operation start date and time, for example, the date and time corresponding to the time when step S130 is executed may be recorded. As the model configuration, the weight of each model MLk_i determined in step S109 is recorded.

[0057] In step S131, the control unit 110 accepts the use of the operation model. Here, "use of the operation model" means inputting operation data into the operation model and obtaining output data. For example, in response to a request from a user of a user terminal (not shown), the control unit 110 may input operation data received from the user terminal into the operation model and obtain output data from the operation model. Furthermore, the output data from the operation model may be presented to the user. In this way, the operation model is "used."

[0058] In step S131, the evaluation unit 12 registers a new record in the evaluation table in accordance with the use of the operation model. The evaluation unit 12 also records the use date and time, model output, and true value in the record. The evaluation unit 12 also calculates the evaluation index of the operation model for the operation data based on the model output and true value, and records it in the record.

[0059] Next, the reward presentation system 1A executes the process from step S120 described below. Also, if it is determined that the result of step S108 is Yes (the operation model exists), the process from step S120 is executed.

[0060] In step S120, the evaluation unit 12 determines whether the operational model needs to be updated. This determination may be made, for example, based on whether a predetermined regeneration condition is satisfied. The regeneration condition indicates a condition for determining that the operational model should be updated. For example, the regeneration condition may be, but is not limited to, that the evaluation index of the operational model calculated for each use falls below a threshold value for a predetermined number of consecutive times, or that the average value of the evaluation index of the operational model for the most recent multiple times falls below a threshold value. Note that the regeneration condition may be satisfied when a new model ML is added to the model pool, regardless of the evaluation index of the operational model.

[0061] If the determination in step S120 is No, the process of step S131 described above is executed, and the evaluation table is updated according to the use of the operation model. If the determination in step S120 is Yes, the process of the next step S121 is executed.

[0062] In step S121, the operation model configuration unit 14 configures a new operation model using some or all of the multiple model MLs registered in the model pool table. The details of step S121 are the same as those of step S109, and therefore will not be described in detail again.

[0063] In step S122, the evaluation unit 12 determines whether the new model configuration of the operational model has improved performance compared to the current model configuration. For example, the evaluation unit 12 inputs the same operational data as the operational data used to calculate the evaluation indexes for the current model configuration, which was referenced when the evaluation unit 12 determined that the regeneration conditions were satisfied, into the operational model with the new model configuration. The evaluation unit 12 may also calculate the evaluation indexes for the new model configuration based on output data from the operational model with the new model configuration corresponding to the same operational data. The evaluation unit 12 may also determine that the performance has improved if the evaluation indexes for the new model configuration are higher than the evaluation indexes for the current model configuration.

[0064] If the answer to step S122 is No, it is highly likely that further performance improvement cannot be expected even if an operational model is constructed from the current model pool. Therefore, in order to add a new updated model to the model pool, the reward presentation system 1A repeats the process from step S101.

[0065] If the answer to step S122 is Yes, the operation of the operation model with the current model configuration will be terminated in order to start operation of the operation model with a new model configuration. Therefore, in this case, the processing from the next step S123 is executed. In step S123, the evaluation unit 12 records the operation end date and time in the record corresponding to the current model configuration of the operation model in the usage table. As the operation end date and time, for example, the date and time corresponding to the time when step S123 is executed may be recorded.

[0066] In step S124, the evaluation unit 12 records the evaluation index of the operation model for the operation period in the record corresponding to the current model configuration of the operation model in the usage table.

[0067] In step S125, the evaluation unit 12 evaluates the contribution of each model ML in the operation model of the current model configuration. For example, the evaluation unit 12 may refer to the usage table and calculate the contribution based on the weight of each model ML, the evaluation index of the operation model, and the operation period of the operation model (i.e., the usage period of each model ML). As an example, the contribution may be a value obtained by multiplying the weight of each model ML by the representative value of the evaluation index, and weighting the value according to the operation period, but is not limited to this. Note that steps S123 to S125 constitute an example of evaluation processing.

[0068] In step S126, the reward presentation unit 13 determines a reward for the provider DS of each model ML according to the degree of contribution of that model ML. In step S127, the reward presentation unit 13 presents the reward determined for each model ML to the provider DS that provided that model ML. Specifically, the reward presentation unit 13 transmits information indicating the reward determined for each model ML to the model providing device 20 from which that model ML was obtained.

[0069] In step S128, the reward presenting unit 23 of the model providing device 20 displays information indicating the reward received in step S126 on the display unit 250. Here, use of the operation model has already started in step S130. In other words, the information indicating the reward is presented after operation of the operation model has started.

[0070] Here, steps S127 to S128 constitute an example of a reward presentation process. Steps S127 to S128 do not necessarily have to be performed each time a reward is determined, as in this example, but may also be performed when a predetermined payment condition is met. In this case, the reward presentation unit 13 may record a record associating the determined reward with the provider DS in a reward table (not shown). Furthermore, when the payment condition is met, the reward presentation unit 13 may present information indicating a reward to the provider DS based on a record recorded in the reward table that includes a reward that has not yet been presented to the provider DS. Furthermore, if there are multiple records for one provider DS, the information indicating the reward may include the total and breakdown of the multiple rewards.

[0071] Following step S127, the reward presentation system 1A executes the processes of steps S130 to S131 described above, and then repeats the processes from step S120. As a result, the reward presentation system 1A continues to operate the operation model newly configured in step S121 until an update of the operation model is required, and determines the reward for each provider DS when operation with this configuration ends. This concludes the explanation of the reward presentation method S1A.

[0072] (Specific Example of Reward Presentation Method S1A) A specific example of reward presentation method S1A will be described with reference to Fig. 9. Fig. 9 is a sequence diagram illustrating a specific example of reward presentation method S1A. As shown in Fig. 9, the specific example of reward presentation method S1A includes steps M1 to M23.

[0073] Step M1 is a specific example of steps S101 to S102. In step M1, the reward providing device 10A requests each of the model providing devices 20_1 and 20_2 to provide a model ML by presenting a task and learning data.

[0074] Step M2 is a specific example of steps S103 to S107. In step M2, the model providing device 20_1 uses the training data to generate two different models ML1_1 and ML1_2 that perform a predetermined task. Then, the model providing device 20_1 provides the two models ML1_1 and ML1_2 to the reward presentation device 10A. The reward presentation device 10A determines that model ML1_1 satisfies the specifications (Yes in S106) and registers record R11 in the model pool table shown in FIG. 5. The reward presentation device 10A also records the model ID "ML1_1," the provider ID "DS1," and the registration date and time "January 10, 2023" in record R11. The reward presentation device 10A also determines that model ML1_2 also satisfies the specifications (Yes in S106) and registers record R12 in the model pool table. In addition, the reward presentation device 10A records the model ID "ML1_2", the provider ID "DS1", and the registration date and time "January 10, 2023" in the record R12.

[0075] Step M3 can be explained in the same way as step M2, except that the reference numeral for model providing device 20_1 is replaced with 20_2, ML1_1 with ML2_1, ML1_2 with ML2_2, and DS1 with DS2. As a result, records R13 and R14 are recorded in the model pool table shown in FIG.

[0076] Step M4 is a specific example of steps S108 to S109 and S130. In step M4, the reward presentation device 10A determines that there is no operation model currently in operation (No in M108), and configures an operation model by referring to the model pool table. Here, it is assumed that the reward presentation device 10A determines the weight of model ML1_1 to be 1. Note that the sum of the weights is 1, and the weights of models ML1_2, ML2_1, and ML2_2 are each 0. In other words, the reward presentation device 10A selects only model ML1_1 from the model pool. As a result, the operation model is configured using only model ML1_1. The reward presentation device 10A also registers record R31 in the usage table shown in FIG. 7. In addition, the reward presentation device 10A records in the record R31 the operation start date and time "January 10, 2023", the weight of model ML1_1 "1", the weight of model ML1_2 "0", the weight of model ML2_1 "0", and the weight of model ML2_2 "0".

[0077] Step M5 is a specific example of steps S131 and S120. In step M5, the operation model is used, and the evaluation index of the operation model is recorded in the evaluation table for each use. Specifically, the reward presentation device 10A inputs operation data into model ML1_1 that constitutes the operation model, thereby obtaining output data output from model ML1_1. The reward presentation device 10A also registers record R21 in the evaluation table shown in FIG. 6, recording the usage date and time "January 10, 2023," the model output "92," and the true value "100." The reward presentation device 10A also records the prediction accuracy "92%" calculated from the model output and true value in record R21 as an evaluation index.

[0078] The use of the operational model in step M5 is repeated until February 9, 2023, when operation of the operational model with the current model configuration will end in the next step M6. In other words, it is assumed that it is determined that updating of the operational model is not necessary during this period (No in S120). As a result, it is assumed that record set RS22 is recorded in the evaluation table as shown in FIG. 6 between January 10, 2023 and February 9, 2023.

[0079] Step M6 is a specific example of steps S120 to S123. In step M6, the reward presentation device 10A determines that the performance of the operational model has deteriorated on February 10, 2023, satisfying the regeneration condition and therefore requiring an update (Yes in step S120). Therefore, the reward presentation device 10A configures a new operational model by referencing the model pool table. Here, it is assumed that the reward presentation device 10A determines the weight of model ML1_1 to be 0.1 and the weight of model ML2_2 to be 0.8. Since the sum of the weights is 1, the weights of models ML1_2 and ML2_1 are each 0. In other words, the reward presentation device 10A selects two models ML1_1 and ML2_2 from the model pool and configures the operational model as an ensemble model using the two selected models ML.

[0080] The reward presentation device 10A also determines that the performance of the new model configuration of the operation model is improved over the performance of the current model configuration (Yes in S122), and ends the operation of the current model configuration. Therefore, the reward presentation device 10A records the operation end date and time "February 9, 2023" in record R31 of the usage table, as shown in FIG. 7.

[0081] Step M7 is a specific example of steps S124 to S126. In step M7, the reward presentation device 10A determines the reward for the provider DS1 of model ML1_1 based on the contribution of the model ML1_1 included in the current model configuration of the operation model. To this end, the reward presentation device 10A first calculates an evaluation index for the current model configuration of the operation model. Specifically, the reward presentation device 10A references the record set RS22 recorded in the usage table for the operation period of the current model configuration, "January 10, 2023 to February 9, 2023," and calculates a representative value of the evaluation index, "0.85." The reward presentation device 10A then records the evaluation index "0.85" in record R31 corresponding to the current model configuration in the usage table. The reward presentation device 10A also references record R31 corresponding to the current model configuration in the usage table to evaluate the contribution of model ML1_1. For example, the reward presentation device 10A calculates the contribution of model ML1_1 by multiplying the weight of model ML1_1, "1," by the evaluation index of the operation model, "0.85," and weighting the result according to the length of the operation period, "31 days" (i.e., the usage period of model ML1_1). Furthermore, the reward presentation device 10A determines the reward for provider DS1 based on the calculated contribution of model ML1_1. For example, the reward presentation device 10A may determine the reward using a predetermined rule for calculating the reward from the contribution.

[0082] Step M8 is a specific example of steps S127 to S128. In step M8, the reward presentation device 10A transmits information indicating the reward for provider DS1 to the model providing device 20_1. For example, the information indicating the reward may include, in addition to the reward, information indicating the contribution of model ML1_1. Furthermore, the information indicating the reward may include, in addition to the reward, information indicating the usage period of model ML1_1, the weight of model ML1_1, the evaluation index of the operational model, etc. This allows provider DS1 to recognize the reward according to the contribution of model ML1_1 that he or she provided. Note that since the current model configuration of the operational model does not include models ML2-1 and ML2-2 provided by provider DS2, no reward is presented to provider DS2.

[0083] Step M9 is a specific example of step S130. In step M9, the reward presentation device 10A starts operation of the operation model with a new model configuration. Specifically, as shown in FIG. 7, the reward presentation device 10A registers a record R32 corresponding to the new model configuration of the operation model in the usage table. In addition, the reward presentation device 10A records in record R32 the operation start date and time "February 10, 2023", the weight of model ML1_1 "0.2", the weight of model ML1_2 "0", the weight of model ML2_1 "0", and the weight of model ML2_2 "0.8".

[0084] Step M10 is explained in a similar manner to step M5. However, in step M10, operational data is input to models ML1_1 and ML2_2 that constitute the operational model. In addition, weights of 0.2 and 0.8 are assigned to the output data from models ML1_1 and ML2_2, respectively, to obtain a weighted average as output data from the operational model.

[0085] The use of the operational model in step M10 is repeated until March 7, 2023, when operation of the operational model with its current model configuration will end in step M14, which will be described later. In other words, it is assumed that it is determined that updating the operational model is not necessary during this period (No in S120). As a result, it is assumed that record set RS23 is recorded in the evaluation table from February 10, 2023 to March 7, 2023, as shown in FIG. 6.

[0086] Step M11 is a specific example of steps S120 to S122 and S101 to S102. In step M11, the reward presentation device 10A determines that the performance of the operational model has deteriorated on February 25, 2023, satisfying the regeneration condition and therefore requiring an update (Yes in step S120). Furthermore, the reward presentation device 10A determines that, although a new operational model has been configured by referencing the current model pool table, the performance of the new operational model configuration has not improved compared to the performance of the current model configuration (No in S122). Therefore, the reward presentation device 10A requests each of the model providing devices 20_1 and 20_2 to update the model ML by transmitting tasks and training data. Furthermore, the reward presentation device 10A may transmit information for improving the performance of the model ML to each of the model providing devices 20_1 and 20_2, in addition to the tasks and training data.

[0087] Steps M12 and M13 are explained in the same manner as steps M2 and M3. However, in step M12, it is assumed that provider DS1 provided one model, ML1_3, as an updated model on March 8, 2023. In addition, in step M13, it is assumed that provider DS2 provided two different models, ML2_3 and ML2_4, as updated models on March 8, 2023. As a result, records R15, R16, and R17 are registered in the model pool table, as shown in FIG. 5.

[0088] Step M14 is a specific example of steps S120 to S123. In step M14, the reward presentation device 10A determines that an update is necessary because the regeneration condition is met due to the addition of a new model ML to the model pool (Yes in step S120). Therefore, on March 8, 2023, the reward presentation device 10A configures a new operational model by referencing the model pool table. Note that, among the models ML registered in the model pool table, the newly added updated models ML1_3, ML2_3, and ML2_4 may be targeted, while the pre-update models ML1_1, ML1_2, ML2_1, and ML2_2 may not be targeted. Here, it is assumed that the reward presentation device 10A determines the weight of model ML1_3 to be 0.4, the weight of model ML2_3 to be 0.5, and the weight of model ML2_4 to be 0.1. In other words, the reward presentation device 10A selects three models ML1_3, ML2_3, and ML2_4 from the model pool, and configures an operation model as an ensemble model using the three selected models ML.

[0089] Steps M15 to M17 are explained in the same way as steps M7 to M8. However, because the current model configuration of the operation model includes models ML1_1 and ML2_2, the rewards for providers DS1 and DS2 are determined based on their contributions. As a result, the operation end date and time "March 7, 2023" and the evaluation index "0.87" are recorded in record R32 corresponding to the current model configuration in the usage table. This evaluation index "0.87" is calculated based on records RS23 in the evaluation table. The reward for provider DS1 is determined based on the contribution of model ML1_1. The contribution of model ML1_1 may be a value obtained by multiplying the weight of model ML1_1 (0.2) by the evaluation index "0.87" based on record R32 in the usage table, and weighting the value according to the length of the operation period (26 days). The reward for provider DS2 is determined based on the contribution of model ML2_2. The contribution of model ML2_2 may be a value obtained by multiplying the weight of model ML2_2 (0.8) by the evaluation index (0.87) based on record R32, and weighting the result according to the length of the operation period (26 days). This allows provider DS1 to recognize the reward according to the contribution of model ML1_1 that he / she provided. Also, provider DS2 can recognize the reward according to the contribution of model ML2_2 that he / she provided.

[0090] Steps M18 to M23 are explained in the same way as steps 9 to M17. However, there is a difference as follows: The operation period during which the operation model was operated with the current model configuration was "March 8, 2023 to March 31, 2023." Furthermore, as shown in FIG. 6, a record set RS24 is recorded in the evaluation table for this operation period. As a result, a record R33 corresponding to the current model configuration is registered in the usage table, and the operation start date and time "March 7, 2023," the weight of model ML1_3 "0.4," the weight of model ML2_3 "0.5," and the weight of model ML2_4 "0.1" are recorded. Furthermore, at the end of operation, the operation end date and time "March 31, 2023" and the evaluation index "0.88" calculated from record set RS24 are recorded.

[0091] Furthermore, the reward for provider DS1 is determined according to the contribution of model ML1_3. The contribution of model ML1_3 may be calculated by multiplying the weight of model ML1_3, "0.4," by the evaluation index, "0.88," based on record R33 of the usage table, and weighting the resulting value according to the length of the operation period, "25" days. Furthermore, the reward for provider DS2 is determined according to both the contribution of model ML2_2 and the contribution of model ML2_3. The contribution of model ML2_2 may be calculated according to record R33 of the usage table, and weighting the resulting value according to the length of the operation period, "25" days. The contribution of model ML2_3 may be calculated based on record R33 in the usage table by multiplying the weight of model ML2_3 (0.1) by the evaluation index (0.87), and then weighting the result according to the length of the operation period (25 days). This allows provider DS1 to recognize the reward according to the contribution of model ML1_3 that he / she provided. Furthermore, provider DS2 can recognize the reward according to the contribution of models ML2_3 and ML2_4 that he / she provided.

[0092] (Effects of Reward Presentation System 1A) As described above, the reward presentation system 1A employs a configuration in which the reward presentation unit 13 presents a reward after the operation of the operation model begins. Therefore, with this configuration, in addition to the effects of the reward presentation device 1, the provider DS can recognize that the contribution of the model ML, which may fluctuate in the operation model, is reflected in the reward.

[0093] Furthermore, in the reward presentation system 1A, the evaluation unit 12 is configured to evaluate the contribution of each model ML based on the weight of the model ML used to configure the operation model. Therefore, with this configuration, in addition to the effect achieved by the reward presentation device 1, the effect of being able to more highly evaluate the contribution of a model ML that has a greater influence on the output of the operation model can be obtained.

[0094] Furthermore, the reward presentation system 1A employs a configuration in which the weight of each model ML is determined based on the performance of that model ML for the operational data. Therefore, with this configuration, in addition to the effects of the reward presentation device 1, it is possible to obtain the effect of more highly evaluating the contribution of a model ML with higher performance for the operational data.

[0095] Furthermore, the reward presentation system 1A employs a configuration in which the evaluation unit 12 evaluates the contribution of each model ML based on the performance of the operation model relative to the operation data. Therefore, with this configuration, in addition to the effect achieved by the reward presentation device 1, it is possible to obtain an effect that the contribution of a model ML that contributed to improving the performance of the operation model can be highly evaluated.

[0096] Furthermore, the reward presentation system 1A employs a configuration in which the evaluation unit 12 evaluates the contribution of each model ML based on the period during which the model ML is included in the operation model or the number of times the operation model including the model ML is used. Therefore, with this configuration, in addition to the effect achieved by the reward presentation device 1, it is possible to obtain an effect of highly evaluating the contribution of a model ML that has been actually used for a long period of time or has been used many times as an operation model.

[0097] Furthermore, the reward presentation system 1A is configured to further include an operational model configuration unit 14 that dynamically configures an operational model during operation using some or all of the multiple models ML. Therefore, with this configuration, in addition to the effects of the reward presentation device 1, it is possible to obtain the effect of reducing performance degradation of the operational model compared to when the configuration of the operational model is fixed.

[0098] Furthermore, in the reward presentation system 1A, the model acquisition unit 11 acquires an updated model obtained by updating each model ML from the provider DS of that model ML, and includes the acquired updated model in the targets that can constitute the operational model. Therefore, with this configuration, in addition to the effects achieved by the reward presentation device 1, the effect of reducing performance degradation of the operational model can be obtained compared to when multiple model MLs that can constitute the operational model are fixed.

[0099] Furthermore, the reward presentation system 1A employs a configuration in which the model acquisition unit 11 acquires an updated model (MLk_[i+1]) from the provider of each model MLk_i by providing the provider of that model with information for improving the performance of that model. Therefore, this configuration has the effect of making it easier to improve the updated model (MLk_[i+1]) in addition to the effects of the reward presentation device 1.

[0100] [Variation 1] In the above-described exemplary embodiment 2, the reward presentation device 10A has been described as obtaining the updated model MLk_[i+1] by requesting the provider DS to update the model MLk_i. However, the reward presentation device 10A may also obtain the updated model MLk_[i+1] without requesting an update of the model MLk_i. The model presentation device 20_k may be configured to periodically update the model MLk_i and provide it to the reward presentation device 10A. Furthermore, the model presentation device 20_k may be configured to update the model MLk_i based on the operation of the provider DSk and provide it to the reward presentation device 10A.

[0101] According to this modification, the reward presentation device 10A can reduce the processing costs of requesting an update of the model ML, communication costs, and the processing costs of determining whether to request an update of the model ML. Furthermore, the provider DS can proactively increase the possibility of increasing rewards by providing an updated model without waiting for a request.

[0102] [Variation 2] In the above-described exemplary embodiment 2, the reward presentation device 10A can be modified to include a function for managing a model ML. Fig. 10 is a block diagram showing the configuration of a reward presentation device 10B modified in this way. As shown in Fig. 10, the reward presentation device 10B includes a model management unit 15 in the control unit 110 in addition to the same configuration as the reward presentation device 10A.

[0103] The model management unit 15 performs at least the following operations: adding a new model ML to a plurality of model MLs; or deleting any one of the plurality of model MLs.

[0104] For example, the model management unit 15 may delete a model ML that has not been used for a predetermined period of time. "Not being used for a predetermined period of time" means that the model ML has not been included in the configuration of the operation model for a predetermined period of time. In other words, the period during which the weight is determined to be 0 exceeds the predetermined period of time. Furthermore, for example, the model management unit 15 may manage the number of model MLs registered by one provider DS (or one organization) to control the model acquisition unit 11 so that the number of model MLs acquired from one provider DS does not exceed an upper limit. The upper limit may be set in advance for all provider DSs or may be set individually for each provider DS. Furthermore, the model management unit 15 may manage whether a registration fee for registering a model ML in the model pool has been received from the provider DS. Furthermore, the model management unit 15 may have a function for inspecting the specifications of the model ML in order to register only model MLs that meet predetermined specifications in the model pool. Furthermore, the model management unit 15 may add a new model ML or delete any of the multiple model MLs based on an operation by the administrator AD.

[0105] According to this modification, it is possible to prevent the number of model MLs registered in the model pool from becoming too large, and also to reduce the number of low-quality model MLs registered by providers DS.

[0106] [Variation 3] In the above-described exemplary embodiment 2, the reward presentation device 10A can be modified to include a function for searching for a model ML. Fig. 11 is a block diagram showing the configuration of a reward presentation device 10C modified in this way. As shown in Fig. 11, the reward presentation device 10C includes a model search unit 16 in the control unit 110 in addition to the same configuration as the reward presentation device 10A.

[0107] The model search unit 16 searches for a model ML that matches the conditions from among a plurality of model MLs. The operation model configuration unit 14 configures an operation model based on the search results by the model search unit 16.

[0108] For example, the model search unit 16 may be configured to search for a model ML whose performance satisfies a predetermined condition among multiple model MLs registered in the model pool. In this case, the predetermined condition may be a condition related to a time series of performance (e.g., a condition indicating that the degree of performance degradation is smaller than a threshold, etc.). The time series of performance may be, for example, an evaluation index of each model ML for the operational data one day ago, two days ago, ..., n days ago. Furthermore, the predetermined condition may be a condition related to the recent performance of each model ML (e.g., a condition indicating that the recent performance is higher than a threshold, etc.). The recent performance may be, for example, the average, maximum, or minimum value of the evaluation index for the most recent week. However, the predetermined condition related to performance is not limited to the above example.

[0109] By using the search results of the model search unit 16, the operation model configuration unit 14 can configure the operation model by preferentially using the model ML with higher performance. As a result, it is possible to include in the configuration a model ML that is more suitable for improving the performance of the operation model, and to offer appropriate rewards to the provider DS of that model ML.

[0110] Furthermore, for example, the model search unit 16 may be configured to be able to search for model MLs whose usage history satisfies a predetermined condition among the multiple model MLs registered in the model pool. In this case, the predetermined condition may be that the last usage date is after a predetermined time point, that the usage frequency in the most recent predetermined period is equal to or greater than a threshold, etc. However, the predetermined condition regarding the usage history is not limited to the above example.

[0111] By using the search results of the model search unit 16, the operation model configuration unit 14 can configure the operation model by preferentially using the model ML that is frequently used. As a result, it is possible to include in the configuration a model ML that is more suitable for improving the performance of the operation model, and to present an appropriate reward to the provider DS of that model ML.

[0112] Furthermore, for example, the model search unit 16 may be configured to search for a model ML whose accumulated contribution degree satisfies a predetermined condition among the multiple model MLs registered in the model pool. In this case, the predetermined condition may be that the accumulated contribution degree is equal to or greater than a threshold. However, the predetermined condition regarding the accumulated contribution degree is not limited to the above example.

[0113] By using the search results of the model search unit 16, the operation model configuration unit 14 can configure the operation model by preferentially using model MLs that have a high degree of contribution and are frequently used, thereby improving the performance of the operation model. As a result, it is possible to include in the configuration a model ML that is more suitable for improving the performance of the operation model, and to present an appropriate reward to the provider DS of that model ML.

[0114] Furthermore, for example, the model search unit 16 may be configured to search for model MLs whose registration date and time satisfy a predetermined condition among the multiple model MLs registered in the model pool. In this case, the predetermined condition may be that the registration date and time is after a predetermined point in time. However, the predetermined condition regarding the registration date and time is not limited to the above example.

[0115] Here, a model ML with a more recent registration date and time is expected to have higher performance. Therefore, by using the search results from the model search unit 16, the operation model configuration unit 14 can configure the operation model by preferentially using such newer model MLs, thereby improving the performance of the operation model. As a result, a model ML that is more suitable for improving the performance of the operation model can be included in the configuration, and an appropriate reward can be presented to the provider DS of that model ML.

[0116] [Variation 4] In the above-described exemplary embodiment 2, the reward presentation system 1A can be modified to present a usage fee to users of the operation model. Figure 12 is a block diagram showing the configuration of such a modified reward presentation system 1D. As shown in Figure 12, the reward presentation system 1D is configured almost the same as the reward presentation system 1A, but includes a reward presentation device 10D instead of the reward presentation device 10A, and also includes a user terminal 30.

[0117] The usage terminal 30 is a computer associated with the user U. The user U is a user or organization that uses the operation model. The usage terminal 30 performs processing for using the operation model based on the operation of the user U. For example, the usage terminal 30 transmits operation data to the reward presentation device 10D and displays output data received from the reward presentation device 10D on a display device.

[0118] Fig. 13 is a block diagram showing the configuration of a reward presentation device 10D. As shown in Fig. 13, the reward presentation device 10D includes a usage fee presentation unit 17 in the control unit 110 in addition to the same configuration as the reward presentation device 10A.

[0119] When the control unit 110 receives operation data from the user terminal 30, it inputs the operation data into the operation model. In addition, the control unit 110 transmits output data output from the operation model to the user terminal 30.

[0120] The usage fee presentation unit 17 presents the usage fee of the operation model to a user U who uses the operation model. Specifically, the usage fee presentation unit 17 presents the usage fee to the user U by transmitting information indicating the usage fee to the usage terminal 30.

[0121] The timing at which the usage fee presentation unit 17 presents the usage fee is not particularly limited, but for example, the reward presentation device 10D may be configured to accept receipt of operation data from the usage terminal 30 once it has confirmed that the payment of the presented usage fee has been completed. Furthermore, for example, after accepting receipt of the operation data from the usage terminal 30, the reward presentation device 10D may be configured to present output data from the operation model corresponding to the operation data to the user U once it has confirmed that the payment of the presented usage fee has been completed. This makes it possible to procure funds for the reward for the provider DS from the user U.

[0122] [Example of implementation using software] Some or all of the functions of the reward presentation device 1, 10A, 10B, 10C, 10D, the model providing device 20, and the user terminal 30 (hereinafter also referred to as "each of the above devices") may be implemented using hardware such as an integrated circuit (IC chip), or may be implemented using software.

[0123] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 14. Figure 14 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0124] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0125] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0126] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0127] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0128] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0129] (Appendix A1) A reward presentation device comprising: a model acquisition means for acquiring a plurality of models that perform a specified task from a provider of each model; an evaluation means for evaluating the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the plurality of models; and a reward presentation means for presenting a reward determined according to the contribution of each model to the provider of the model.

[0130] (Supplementary Note A2) The reward presentation device according to Supplementary Note A1, wherein the reward presentation means presents the reward after operation of the operation model begins.

[0131] (Supplementary Note A3) The reward presentation device according to Supplementary Note A1 or A2, wherein the evaluation means evaluates the contribution of each model based on a weight of the model used to configure the operation model.

[0132] (Supplementary Note A4) The reward presentation device according to Supplementary Note A3, wherein the weight of each model is determined based on the performance of the model with respect to operational data.

[0133] (Supplementary Note A5) The reward presentation device according to any one of Supplementary Notes A1 to A4, wherein the evaluation means evaluates the contribution of each of the models based on performance of the operation model with respect to operation data.

[0134] (Appendix A6) The reward presentation device according to any one of Appendices A1 to A5, wherein the evaluation means evaluates the contribution of each model based on a period during which the model is included in the operational model or a number of times an operational model including the model is used.

[0135] (Supplementary Note A7) The reward presentation device according to any one of Supplementary Notes A1 to A6, further comprising an operation model configuration means that dynamically configures the operation model during operation using some or all of the plurality of models.

[0136] (Supplementary Note A8) The reward presentation device according to any one of Supplementary Notes A1 to A7, further comprising a model management means that at least adds a new model to the plurality of models or deletes any model from the plurality of models.

[0137] (Supplementary Note A9) The reward presentation device according to any one of Supplementary Notes A1 to A8, further comprising a usage fee presentation means for presenting a usage fee for the operation model to a user who uses the operation model.

[0138] (Appendix A10) The reward presentation device described in any one of Appendices A1 to A9, wherein the model acquisition means acquires an updated model obtained by updating each model from the provider of the model, and includes the acquired updated model in the targets that can constitute the operational model.

[0139] (Supplementary Note A11) The reward presentation device according to Supplementary Note A10, wherein the model acquisition means acquires the updated model from a provider of the model by providing information for improving performance of the model to the provider of the model.

[0140] (Appendix A12) The reward presentation device according to any one of Appendices A1 to A11, further comprising a model search means for searching for a model that matches a condition from among the plurality of models, and the operational model is constructed based on a search result of the model search means.

[0141] (Supplementary Note A13) The reward presentation device according to any one of Supplementary Note A1 to Supplementary Note A12, wherein at least one of the plurality of models is a machine learning model.

[0142] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0143] (Appendix B1) A reward presentation method including: a model acquisition process in which at least one processor acquires multiple models that perform a specified task from providers of each model; an evaluation process in which the at least one processor evaluates the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the multiple models; and a reward presentation process in which the at least one processor presents a reward determined according to the contribution of each model to the provider of the model.

[0144] (Supplementary Note B2) The reward presentation method according to Supplementary Note B1, wherein in the reward presentation process, the at least one processor presents the reward after operation of the operation model begins.

[0145] (Supplementary Note B3) The reward presentation method according to Supplementary Note B1 or B2, wherein in the evaluation process, the at least one processor evaluates the contribution of each model based on a weight of the model used to configure the operation model.

[0146] (Supplementary Note B4) The reward presentation method according to Supplementary Note B3, wherein the weight of each model is determined based on the performance of the model on operational data.

[0147] (Supplementary Note B5) The reward presentation method according to any one of Supplementary Notes B1 to B4, wherein in the evaluation process, the at least one processor evaluates the contribution of each of the models based on performance of the operation model with respect to operation data.

[0148] (Appendix B6) The reward presentation method according to any one of Appendices B1 to B5, wherein in the evaluation process, the at least one processor evaluates the contribution of each model based on the period during which the model is included in the operational model or the number of times the operational model including the model is used.

[0149] (Supplementary Note B7) The reward presentation method described in any one of Supplementary Notes B1 to B6, wherein the at least one processor further includes an operational model configuration process that dynamically configures the operational model during operation using some or all of the multiple models.

[0150] (Appendix B8) The reward presentation method according to any one of Appendices B1 to B7, further comprising a model management process in which the at least one processor performs at least the following: adding a new model to the plurality of models; or deleting any model from the plurality of models.

[0151] (Appendix B9) The reward presentation method described in any one of Appendices B1 to B8, wherein the at least one processor further includes a usage fee presentation process that presents a usage fee for the operation model to a user who uses the operation model.

[0152] (Appendix B10) The reward presentation method described in any one of Appendices B1 to B9, wherein in the model acquisition process, the at least one processor acquires an updated model obtained by updating each model from the provider of the model, and includes the acquired updated model in the targets that may constitute the operational model.

[0153] (Supplementary Note B11) The reward presentation method according to Supplementary Note B10, wherein in the model acquisition process, the at least one processor acquires the updated model from a provider of the model by providing information for improving performance of the model to the provider of the model.

[0154] (Appendix B12) The reward presentation method described in any one of Appendices B1 to B11, further comprising a model search process in which the at least one processor searches for a model that matches a condition among the plurality of models, and the operational model is constructed based on a search result of the model search process.

[0155] (Supplementary Note B13) The reward presentation device according to any one of Supplementary Note B1 to Supplementary Note B12, wherein at least one of the plurality of models is a machine learning model.

[0156] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0157] (Appendix C1) A reward presentation program that causes a computer to function as a reward presentation device, causing the computer to function as: a model acquisition means that acquires multiple models that perform specified tasks from providers of each model; an evaluation means that evaluates the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the multiple models; and a reward presentation means that presents a reward determined according to the contribution of each model to the provider of the model.

[0158] (Supplementary Note C2) The reward presentation program according to Supplementary Note C1, wherein the reward presentation means presents the reward after the operation of the operation model begins.

[0159] (Supplementary Note C3) The reward presentation program according to Supplementary Note C1 or C2, wherein the evaluation means evaluates the contribution of each model based on a weight of the model used to configure the operation model.

[0160] (Supplementary Note C4) The reward presentation program according to Supplementary Note C3, wherein the weight of each model is determined based on the performance of the model with respect to the operational data.

[0161] (Supplementary Note C5) The reward presentation program according to any one of Supplementary Notes C1 to C4, wherein the evaluation means evaluates the contribution of each of the models based on performance of the operation model with respect to operation data.

[0162] (Appendix C6) The reward presentation program according to any one of Appendices C1 to C5, wherein the evaluation means evaluates the contribution of each model based on the period during which the model is included in the operational model or the number of times the operational model including the model is used.

[0163] (Appendix C7) The reward presentation program described in any one of Appendices C1 to C6, further causing the computer to function as an operational model configuration means that dynamically configures the operational model during operation using some or all of the multiple models.

[0164] (Appendix C8) The reward presentation program according to any one of Appendices C1 to C7, further causing the computer to function as a model management means that at least adds a new model to the plurality of models or deletes any model from the plurality of models.

[0165] (Appendix C9) The reward presentation program according to any one of Appendices C1 to C8, further causing the computer to function as a usage fee presentation means for presenting a usage fee for the operation model to a user who uses the operation model.

[0166] (Appendix C10) The reward presentation program described in any one of Appendices C1 to C9, wherein the model acquisition means acquires an updated model obtained by updating each model from the provider of the model, and includes the acquired updated model in the targets that can constitute the operational model.

[0167] (Supplementary Note C11) The reward presentation program according to Supplementary Note C10, wherein the model acquisition means acquires the updated model from a provider of the model by providing information for improving performance of the model to the provider of the model.

[0168] (Appendix C12) The reward presentation program described in any one of Appendices C1 to C11, further causing the computer to function as a model search means for searching for a model that meets conditions among the plurality of models, and the operational model is constructed based on the search results of the model search means.

[0169] (Supplementary Note C13) The reward presentation device according to any one of Supplementary Note C1 to Supplementary Note C12, wherein at least one of the plurality of models is a machine learning model.

[0170] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0171] (Appendix D1) A reward presentation device comprising at least one processor, the at least one processor performing the following: a model acquisition process for acquiring a plurality of models that perform a predetermined task from a provider of each model; an evaluation process for evaluating the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the plurality of models; and a reward presentation process for presenting a reward determined according to the contribution of each model to the provider of the model.

[0172] The reward presentation device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0173] (Supplementary Note D2) The reward presentation device according to Supplementary Note D1, wherein in the reward presentation process, the at least one processor presents the reward after operation of the operation model begins.

[0174] (Supplementary Note D3) The reward presentation device according to Supplementary Note D1 or D2, wherein in the evaluation process, the at least one processor evaluates the contribution of each model based on a weight of the model used to configure the operation model.

[0175] (Supplementary Note D4) The reward presentation device according to Supplementary Note D3, wherein the weight of each model is determined based on the performance of the model with respect to operational data.

[0176] (Supplementary Note D5) The reward presentation device according to any one of Supplementary Notes D1 to D4, wherein in the evaluation process, the at least one processor evaluates the contribution of each of the models based on performance of the operation model with respect to operation data.

[0177] (Appendix D6) The reward presentation device described in any one of Appendices D1 to D5, wherein in the evaluation process, the at least one processor evaluates the contribution of each model based on a period during which the model is included in the operational model or a number of times an operational model including the model is used.

[0178] (Appendix D7) The reward presentation device described in any one of Appendices D1 to D6, wherein the at least one processor further executes an operational model configuration process that dynamically configures the operational model during operation using some or all of the multiple models.

[0179] (Appendix D8) The reward presentation device described in any one of Appendices D1 to D7, wherein the at least one processor further executes a model management process that at least adds a new model to the plurality of models or deletes any model from the plurality of models.

[0180] (Appendix D9) The reward presentation device described in any one of Appendices D1 to D8, wherein the at least one processor further executes a usage fee presentation process that presents a usage fee for the operation model to a user who uses the operation model.

[0181] (Appendix D10) The reward presentation device described in any one of Appendices D1 to D9, wherein in the model acquisition process, the at least one processor acquires an updated model that updates each model from the provider of the model, and includes the acquired updated model in the targets that can constitute the operational model.

[0182] (Supplementary Note D11) The reward presentation device described in Supplementary Note D10, wherein in the model acquisition process, the at least one processor acquires the updated model from a provider of the model by providing information for improving performance of the model to the provider of the model.

[0183] (Appendix D12) The reward presentation device described in any one of Appendices D1 to D11, wherein the at least one processor further executes a model search process to search for a model that meets a condition among the plurality of models, and the operational model is constructed based on the search results of the model search process.

[0184] (Supplementary Note D13) The reward presentation device according to any one of Supplementary Note D1 to Supplementary Note D12, wherein at least one of the plurality of models is a machine learning model.

[0185] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0186] (Appendix E1) A non-transitory recording medium having recorded thereon a reward presentation program that causes a computer to function as a reward presentation device, the reward presentation program causing the computer to execute: a model acquisition process that acquires multiple models that perform specified tasks from the providers of each model; an evaluation process that evaluates the contribution of each model in an operational model that is dynamically constructed during operation using some or all of the multiple models; and a reward presentation process that presents a reward determined according to the contribution of each model to the provider of the model.

[0187] 1A, 1D Reward presentation system 1, 10A, 10B, 10C, 10D Reward presentation device 20, 20_1, 20_2, 20_k Model providing device 30 User terminal 110, 210 Control unit 120, 220 Storage unit 130, 230 Communication unit 240 Input unit 250 Display unit 11 Model acquisition unit 12 Evaluation unit 13 Reward presentation unit 14 Operation model configuration unit 15 Model management unit 16 Model search unit 17 Usage fee presentation unit 21 Request acquisition unit 22 Model generation unit 23 Reward presentation unit C1 Processor C2 Memory

Claims

1. A model acquisition means for obtaining multiple models that perform a predetermined task from the provider of each model, An evaluation means for evaluating the contribution of each model in an operational model dynamically configured during operation using some or all of the aforementioned multiple models, A reward offering means that presents a reward determined according to the contribution of each model to the provider of that model, A reward presentation device equipped with the following features.

2. The reward presentation device according to claim 1, wherein the reward presentation means presents the reward after the start of operation of the operational model.

3. The evaluation means evaluates the contribution of each model based on the weight of the model used to construct the operational model. The reward presentation device according to claim 1.

4. The evaluation means evaluates the contribution of each model based on the performance of the operational model with respect to operational data. The reward presentation device according to claim 1.

5. The evaluation means evaluates the contribution of each model based on the period during which the model is included in the operational model or the number of times the operational model including the model is used. The reward presentation device according to claim 1.

6. The operation model configuration means further comprises a means for dynamically configuring the operation model during operation using some or all of the plurality of models. The reward presentation device according to claim 1.

7. The system further comprises model management means for adding a new model to the plurality of models, or deleting any of the plurality of models. The reward presentation device according to claim 1.

8. The system further comprises a means for presenting a usage fee for the operational model to a user who uses the operational model, The reward presentation device according to claim 1.

9. A model acquisition process in which at least one processor acquires a plurality of models that perform a predetermined task from the provider of each model, The at least one processor performs an evaluation process that evaluates the contribution of each model in an operational model dynamically configured during operation using some or all of the multiple models, The at least one processor performs a reward presentation process in which it presents a reward determined according to the contribution of each model to the provider of the model, A method of offering rewards that includes this.

10. A program that causes a computer to function as a reward presentation device, The aforementioned computer, A model acquisition means for obtaining multiple models that perform a predetermined task from the provider of each model. An evaluation means for evaluating the contribution of each model in an operational model dynamically configured during operation using some or all of the aforementioned multiple models, A reward offering means that presents a reward determined according to the contribution of each model to the provider of the model, A reward-based program that functions as such.