Reward presentation device, reward presentation method, and reward presentation program

Abstract

This reward presentation device can assist in a user's decision making by being provided with: a model acquisition unit that acquires, from a provider, a model for executing a prescribed task and having been generated using training data; an evaluation unit that evaluates the performance during operation of the model with respect to operation data that is different from the training data; and a reward presentation unit that presents, to the provider, the reward determined in accordance with the performance during operation.

Description

Reward presentation device, reward presentation method, and reward presentation program

[0001] The present disclosure relates to techniques for offering rewards.

[0002] Techniques for presenting rewards are known. For example, Patent Literature 1 discloses a technique for presenting rewards to multiple analysts who analyze information according to a user's purpose of use. In this technique, rewards are determined based on the order in which the models generated by each analyst are provided to the user as analysis results and the performance of each model.

[0003] Japanese Patent Application Publication No. 2019-46177

[0004] However, the performance of a model may be evaluated lower after the start of operation than before the start of operation. This is because data different from the data used to generate the model is applied, or factors not anticipated when the model was generated affect the model's output. In the technology described in Patent Document 1, the performance of the model referenced to determine the reward is calculated based on the data used to generate the model. Therefore, there is a problem in that the reward for the model provider does not reflect changes in the model's performance after the start of operation.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and one exemplary purpose thereof is to provide a technology that allows compensation to model providers to reflect changes in the performance of a model after it goes into operation.

[0006] A reward presentation device according to an exemplary aspect of the present disclosure includes a model acquisition means for acquiring from a provider a model that is generated using training data and that performs a specified task, an evaluation means for evaluating the operational performance of the model for operational data that differs from the training data, and a reward presentation means for presenting to the provider a reward determined according to the operational performance.

[0007] A reward presentation method according to an exemplary aspect of the present disclosure includes a model acquisition process in which at least one processor acquires from a provider a model that performs a predetermined task and is generated using training data; an evaluation process in which the at least one processor evaluates the operational performance of the model on operational data that is different from the training data; and a reward presentation process in which the at least one processor presents to the provider a reward determined according to the operational performance.

[0008] A reward presentation program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as a reward presentation device, and causes the computer to function as a model acquisition means that acquires from a provider a model that executes a specified task and is generated using training data, an evaluation means that evaluates the operational performance of the model on operational data that differs from the training data, and a reward presentation means that presents to the provider a reward determined according to the operational performance.

[0009] According to one exemplary aspect of the present disclosure, an exemplary effect is provided in which a technology can be provided that allows the compensation paid to a model provider to reflect changes in the performance of the model after it goes into operation.

[0010] FIG. 1 is a block diagram showing the configuration of a reward presentation device according to the present disclosure. FIG. 2 is a flow diagram showing the flow of a reward presentation method according to the present disclosure. FIG. 3 is a block diagram showing an overview of the configuration of a reward presentation system according to the present disclosure. FIG. 4 is a block diagram showing the functional configurations of a reward presentation device and a model providing device according to the present disclosure. FIG. 5 is a diagram showing an example of a model pool table according to the present disclosure. FIG. 6 is a diagram showing an example of an evaluation table according to the present disclosure. FIG. 7 is a diagram showing an example of a reward table according to the present disclosure. FIG. 8 is a flow diagram showing the flow of a reward presentation method according to the present disclosure. FIG. 9 is a sequence diagram explaining a specific example of a reward presentation method according to the present disclosure. FIG. 10 is a block diagram showing an example of the hardware configuration of a computer functioning as each device according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Reward Presentation Device 1) The configuration of the reward presentation device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the reward presentation device 1. As shown in FIG. 1, the reward presentation device 1 includes a model acquisition unit 11, an evaluation unit 12, and a reward presentation unit 13. The model acquisition unit 11 is an example of a configuration that realizes model acquisition means. The evaluation unit 12 is an example of a configuration that realizes evaluation means. The reward presentation unit 13 is an example of a configuration that realizes reward presentation means. The model acquisition unit 11 acquires from a provider a model that executes a predetermined task and is generated using training data. The evaluation unit 12 evaluates the operational performance of the model for operational data that differs from the training data. The reward presentation unit 13 presents the provider with a reward determined according to the operational performance. Here, operational performance refers to the accuracy of output data from a model corresponding to operational data. The operational performance may change depending on the operational data even if the model remains unchanged.

[0014] (Effects of the Reward Presentation Device 1) As described above, the reward presentation device 1 employs a configuration including the above-described model acquisition unit 11, evaluation unit 12, and reward presentation unit 13. Therefore, the reward presentation device 1 has the effect of being able to reflect changes in the performance of a model after operation begins in the reward paid to a model provider.

[0015] (Flow of reward presentation method S1) The flow of reward presentation method S1 will be described with reference to FIG. 2. FIG. 2 is a flow diagram showing the flow of reward presentation method S1. As shown in FIG. 2, reward presentation method S1 includes a model acquisition process S11, an evaluation process S12, and a reward presentation process S13. In the model acquisition process S11, at least one processor acquires from a provider a model that executes a predetermined task and that has been generated using training data. In the evaluation process S12, at least one processor evaluates the operational performance of the model for operational data that differs from the training data. In the reward presentation process S13, at least one processor presents to the provider a reward determined according to the operational performance.

[0016] (Effects of Reward Presentation Method S1) As described above, reward presentation method S1 employs a configuration including the above-described model acquisition process S11, evaluation process S12, and reward presentation process S13. Therefore, reward presentation method S1 has the effect of being able to reflect changes in model performance after operation begins in the reward for the model provider.

[0017] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0018] (Overview of the configuration of reward presentation system 1A) An overview of the configuration of reward presentation system 1A will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an overview of the configuration of reward presentation system 1A. The reward presentation system 1A includes a reward presentation device 10 and a model providing device 20. The reward presentation device 10 is a computer associated with an administrator AD. The administrator AD is a user or organization that manages a model ML. The model providing device 20 is a computer associated with a provider DS. The provider DS is a user or organization that provides a model ML. The reward presentation device 10 is communicatively connected to the model providing device 20 via a network NW. The network NW may include some or all of a wireless LAN (Local Area Network), a wired LAN, and a WAN (Wide Area Network), but is not limited to these.

[0019] (Functional configuration of reward presentation device 10) The functional configuration of the reward presentation device 10 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configurations of the reward presentation device 10 and the model providing device 20. The functional configuration of the model providing device 20 will be described later, and first the functional configuration of the reward presentation device 10 will be described.

[0020] The reward presentation device 10 includes a control unit 110, a storage unit 120, and a communication unit 130. The control unit 110 controls each unit of the reward presentation device 10. The storage unit 120 stores various data and programs used by the control unit 110. The communication unit 130 communicates with other devices under the control of the control unit 110. When the reward presentation device 10 is configured as a computer including a processor, memory, and communication interface, the control unit 110 is realized by the processor, the storage unit 120 is realized by the memory, and the communication unit 130 is realized by the communication interface.

[0021] (Functions of the control unit 110) The control unit 110 includes a model acquisition unit 11, an evaluation unit 12, and a reward presentation unit 13. Each of these functional blocks is configured as described in the first exemplary embodiment, and in addition, is configured as follows.

[0022] After the operation of model MLi (i = 1, 2, ...) begins, the model acquisition unit 11 acquires an updated model (model MLi+1) obtained by updating model MLi from the provider. For example, the model acquisition unit 11 may acquire the updated model (model MLi+1) by requesting the provider to update model MLi. For example, the model acquisition unit 11 may acquire the updated model (model MLi+1) by providing the provider with information for improving the performance of model MLi. When there is no need to particularly distinguish between models ML1, ML2, ..., they are also simply referred to as model ML. Details of model ML will be described later.

[0023] The information for improving performance may be, for example, information indicating factors of prediction errors, information indicating performance degradation conditions, etc. Factors of prediction errors are factors that cause prediction results to deviate from the true value. As a specific example, when the model ML predicts product sales, an example of information indicating factors of prediction errors is information on the "day of the week," which was not included in the original learning data. Furthermore, the performance degradation conditions are conditions that are estimated to be commonly satisfied when a performance degradation of the model ML is observed. As a specific example, when the model ML predicts product sales, an example of a performance degradation condition is a condition that performance degradation is observed on "Monday." Furthermore, the model acquisition unit 11 may acquire a model ML from each of multiple providers DS. Furthermore, the model acquisition unit 11 may acquire multiple model MLs from a single provider DS.

[0024] The evaluation unit 12 may evaluate the operational performance by calculating an index (hereinafter referred to as an operational evaluation index) that indicates the performance of the model ML for the operational data. The operational evaluation index may be, for example, a numerical value that indicates the effect of using the model ML (e.g., prediction accuracy, precision, etc.) or a numerical value that indicates the business effect (e.g., excess inventory, unsold loss, etc.). For example, the evaluation unit 12 may calculate the operational evaluation index based on output data and true values ​​obtained by inputting operational data into the model ML. However, if true values ​​cannot be obtained, the evaluation unit 12 may evaluate the operational performance based on an estimate of the true value corresponding to the operational data.

[0025] For example, the following describes the operational evaluation index when the model ML predicts product sales based on the weather for that day. In this case, the evaluation unit 12 may calculate the operational evaluation index based on the sales forecast value obtained by inputting the weather for that day into the model ML and the actual sales for that day (true value). On the other hand, the following describes the operational evaluation index when the model ML predicts the quality of fruit in autumn based on the weather in early summer. In this case, the true value of fruit quality cannot be obtained until autumn, but the reward for the provider DS of the model ML needs to be determined before autumn. In such a case, the evaluation unit 12 calculates an estimated value of the quality (true value) of fruit in autumn. The evaluation unit 12 may also calculate the operational evaluation index based on the quality forecast value and the estimated value of fruit quality in autumn obtained by inputting operational data into the model ML. Note that known techniques can be used to calculate the estimated true value.

[0026] Next, a case will be described in which the reward presentation device 10 is used in a healthcare field such as a hospital. As an example, assume that the model ML is a model that predicts a hospital's bed occupancy rate one week from now based on explanatory variables such as the date of the month, day of the week, season, current bed occupancy rate, moving average of past bed occupancy rates, local infectious disease outbreaks, and number of patients visiting the hospital. In this case, the evaluation unit 12 may calculate an operational evaluation index based on the bed occupancy rate obtained by inputting the explanatory variables into the model ML and the actual bed occupancy rate (true value) on that day. This allows hospital staff to make decisions regarding reward determination.

[0027] After the model ML begins operation, the reward presentation unit 13 presents a reward determined according to the performance during operation. Here, the performance during operation tends to deteriorate over time. One example of the cause of such deterioration is when the characteristics of the data during operation gradually diverge (for example, as the seasons change) from the characteristics of the training data used to generate the model ML (for example, images taken in winter). Furthermore, the performance during operation may also deteriorate relative to the performance at the time of generation, regardless of the passage of time. One example of the cause of such deterioration is when the characteristics of the data during operation referenced during use (for example, images taken in a dark place) differ from the characteristics of the training data used to generate the model ML (for example, images taken in a bright place).

[0028] Furthermore, the reward presentation unit 13 may determine a reward each time a predetermined period has elapsed since the start of operation of the model ML, based on the operational performance evaluated during the most recent predetermined period. For example, the reward presentation unit 13 may determine a reward based on a representative value of the operational evaluation index evaluated for each use of the model ML (e.g., every five uses if the model ML has been used five times) during the most recent predetermined period (e.g., the most recent 10 days).

[0029] Furthermore, when a deviation condition indicating that the most recently evaluated operational performance deviates from operational performance evaluated in the past is satisfied, the reward presentation unit 13 may determine a reward based on at least the most recently evaluated operational performance. For example, the deviation condition may be a condition that the degree of decline of the most recently evaluated operational evaluation index is equal to or greater than a threshold value compared to the operational evaluation index referenced at the time of the previous reward determination. Furthermore, the deviation condition may be a condition that the degree of decline of the operational evaluation index evaluated at the time of the current use from the operational evaluation index evaluated at the time of the previous use is equal to or greater than a threshold value for a predetermined number of consecutive times. However, specific examples of the deviation condition are not limited to these.

[0030] Furthermore, the reward presenter 13 may determine the reward according to the degree of improvement in the operational performance of the updated model (model MLi+1) relative to the operational performance of the model MLi. For example, the degree of improvement may be the improvement range or improvement rate of the operational evaluation index of the updated model (model MLi+1) relative to the operational evaluation index of the model MLi. Furthermore, the target for comparison with the operational performance of the updated model (model MLi+1) may be the last evaluated operational performance of the model MLi before the update.

[0031] Furthermore, the reward presentation unit 13 may determine the reward according to the lead time from when an update of the model MLi is requested until the updated model (model MLi+1) is acquired. Furthermore, when multiple models MLs have been acquired, the reward presentation unit 13 may present a reward determined according to the performance during operation to the provider DS of a model ML selected from the multiple models MLs according to the performance during operation.

[0032] The timing at which the reward presentation unit 13 determines the reward and the timing at which the reward is paid to the provider DS may or may not correspond. If they correspond, for example, the reward is paid to the provider DS each time it is determined. If they do not correspond, for example, when a specified payment condition is met, the total of the unpaid rewards determined up to that point may be paid to the provider DS. The specified payment condition may be, for example, at a specified period such as the end of the month, when the total unpaid rewards exceeds a threshold, at the end of the operation of the model ML, or the like, but is not limited to these.

[0033] (Information Stored in the Storage Unit 120) As shown in FIG. 4, the storage unit 120 stores models ML1, ML2, ..., a model pool table, an evaluation table, and a reward table. In this example, model ML1 is the model first provided by the provider DS in response to a request from the manager AD, and model ML2 is an updated model obtained by updating model ML1. Similarly, in the case of models ML3 and onward, model MLi+1 is an updated model obtained by updating model ML. As mentioned above, when there is no need to particularly distinguish between models ML1, ML2, ..., each will also be referred to as model ML. Furthermore, the updated model MLi+1 may also be referred to as the updated model MLi+1, etc., or simply as the "updated model." Although FIG. 4 shows two models ML, the number of models ML stored in the storage unit 120 is not limited to two. Furthermore, before the start of execution of a reward presentation method S1A (described later), no model ML may be stored in the storage unit 120. One or more model MLs constitute a model pool. For example, one of the model MLs constituting the model pool (e.g., the latest model ML) is set as the model ML in operation.

[0034] A model ML is a mathematical function or process that solves a task. For example, the model ML may be a machine learning model or a simple statistical model. The model ML may also include optimization. For example, the model ML may be a model that determines an optimal order quantity based on a demand forecast. The learning algorithm for generating the model ML is not particularly limited and may be an existing learning algorithm. For example, the learning algorithm may be a random forest, a support vector machine, a naive Bayesian model, a piecewise linear model using FAB (Factorized Asymptotic Bayesian Inference), or a neural network. When a model pool is composed of multiple model MLs, all of the multiple model MLs may be machine learning models, or at least one may be a machine learning model and at least one may be a non-machine learning model (e.g., a statistical model). When at least two of the multiple model MLs are machine learning models, the learning algorithms for generating the at least two model MLs may be the same or different.

[0035] The model pool table stores information about one or more model MLs included in a model pool. An example of a model pool table will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating an example of a model pool table. Each row in FIG. 5 represents a record related to one model ML and includes a model ID, a request date and time, an operation start date and time, and an operation end date and time. The model ID is information for identifying the model ML. For ease of explanation, the reference symbols for each model ML shown in FIG. 4 are referred to as model IDs, but model IDs are not limited to these. The request date and time indicates the time when the administrator AD requests the provider DS to provide the model ML. The operation start date and time indicates the time when use of the model ML began. The operation end date and time indicates the time when use of the model ML ended. In this example, the request date and time, operation start date and time, and operation end date and time each contain a "date," but the time may also be included in addition to the date.

[0036] The evaluation table is information containing evaluation results of the operational performance of a model ML in operation. An example of an evaluation table will be described with reference to FIG. 6. FIG. 6 is a diagram illustrating an example of the evaluation table. Each row in FIG. 6 represents a record recorded for each use of the model ML in operation, and includes the date and time of use, the model output, a true value, and an operational evaluation index. The date and time of use indicates the time when the model ML was used. In this example, the date and time of use is recorded as the "date," but for example, the time may be included in addition to the date. The model output is information indicating the output data from the model ML. The output data is output by inputting operational data at the time of use into the model ML. Note that the operational data input into the model ML may differ each time it is used. The true value indicates the true value corresponding to the operational data. For example, if a model ML inputs the weather for the day and outputs a sales forecast value for a product for that day, the model output is the sales forecast value output by inputting the "weather data for the date and time of use," which is operational data, into the model ML. The true value is the sales amount for that day. The operational evaluation index is an example of an index indicating operational performance, and in this example is prediction accuracy calculated from the model output and true value, but is not limited to this.

[0037] The reward table is information including rewards determined according to the operational performance of a model ML in operation. An example of the reward table will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the reward table. Each row shown in FIG. 7 indicates a record recorded each time a reward is determined, and includes a reference period, a reference index, a reward, and a type. The reference period indicates the period or time point referenced to determine the reward. The reference index is an index referenced to determine the reward, and is a representative value of an operational evaluation index, an improvement range of an operational evaluation index, or a lead time. The reward indicates the determined reward. The type indicates the type of the determined reward. In this example, the type is one of a performance reward, an improvement reward, and a lead time reward. The performance reward is a reward determined according to a representative value of an operational evaluation index in the reference period. The improvement reward is a reward determined according to the degree of improvement in operational performance in the reference period. The lead time reward is a reward determined according to the lead time from when the provision of the model ML is requested to when it is provided.

[0038] Furthermore, in this example, a "date" is recorded as the reference period, but for example, time may be included in addition to the date. Furthermore, when a representative value of an operational evaluation index is recorded as a reference index, the representative value may be, but is not limited to, the average value, maximum value, minimum value, median value, etc. of one or more operational evaluation indexes obtained during that period. Furthermore, when an improvement range of an operational evaluation index is recorded as a reference index, the improvement range may be, but is not limited to, the difference between two operational evaluation indexes or a normalized value of the difference. Furthermore, when a lead time is recorded as a reference index, the lead time may be, but is not limited to, the number of days or a normalized value of the number of days. The unit of compensation may be any currency unit.

[0039] In the following, adding a new record to the model pool table, evaluation table, or reward table will be referred to as "registering" the record. Also, adding information to an already registered record will be referred to as recording the information in the record.

[0040] (Functional Configuration of Model Providing Device 20) The functional configuration of the model providing device 20 will be described with reference to FIG. 4. As shown in FIG. 4, the model providing device 20 includes a control unit 210, a storage unit 220, a communication unit 230, an input unit 240, and a display unit 250. The control unit 210 controls each unit of the model providing device 20. The storage unit 220 stores various data and programs used by the control unit 210. The communication unit 230 communicates with other devices under the control of the control unit 210. The input unit 240 accepts user operations. The display unit 250 displays information under the control of the control unit 210. When the model providing device 20 is configured by a computer including a processor, a memory, and a communication interface, the control unit 110 is realized by the processor, the storage unit 120 is realized by the memory, and the communication unit 130 is realized by the communication interface. The input unit 240 is configured by a keyboard, a mouse, a touch pen, a touch pad, or the like, which is built into or connected to the computer. The display unit 250 is configured by a liquid crystal display, an organic EL (Electro Luminescence) display, or the like, which is built into or connected to the computer. The input unit 240 and the display unit 250 may be configured by an integrally formed touch panel.

[0041] (Functions of the control unit 210) The control unit 210 includes a request acquisition unit 21, a model generation unit 22, and a reward presentation unit 23. The request acquisition unit 21 acquires information requesting the provision of a model ML from the reward presentation device 10. The requested information includes tasks to be executed by the model ML and learning data that can be used to create the model ML. For example, the request acquisition unit 21 may present the requested information to the provider DS by displaying it on the display unit 250. The model generation unit 22 generates a model ML in response to a request from the reward presentation device 10. Specifically, the model generation unit 22 uses the learning data presented by the reward presentation device 10 to generate a model ML so as to execute a task presented by the reward presentation device 10. For example, the model generation unit 22 may generate a model ML based on an operation by the provider DS. The reward presentation unit 23 presents the reward presented by the reward presentation device 10 to the provider DS. For example, the reward presenter 23 may present the reward to the provider DS by displaying it on the display unit 250 .

[0042] (Flow of reward presentation method S1A) The reward presentation method S1A executed by the reward presentation system 1A configured as above will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing the flow of the reward presentation method S1A. As shown in Fig. 8, the reward presentation method S1A includes steps S101 to S121.

[0043] In step S101, the model acquisition unit 11 of the reward presentation device 10 registers a new record in the model pool table and records the request date and time when the provider DS is requested to provide the model MLi. The request date and time may be the date and time when step S101 is executed.

[0044] In step S102, the model acquisition unit 11 presents the provider DS with the task to be executed by the model MLi and the training data that can be used to generate the model MLi. Specifically, the model acquisition unit 11 presents the task and the training data to the provider DS by transmitting the task and the training data to the model providing device 20. Note that after step S102 is executed for the first time, it may be executed again as described below. In this case, the model MLi-1 has already been acquired. In addition to the task and the training data, the model acquisition unit 11 may also provide the provider DS with information for improving the performance of the model MLi-1. Specific examples of the information for performance improvement are as described above, and therefore detailed description will not be repeated.

[0045] In step S103, the model acquisition unit 11 requests the provider DS to provide a model MLi using the presented task and learning data. Specifically, the model acquisition unit 11 requests the provider DS to provide a model ML by transmitting information to the model providing device 20 instructing the model providing device 20 to generate a model MLi using the presented task and learning data. Note that steps S101 to S103 may be executed again after the first execution, as described below. In this case, a model MLi-1 has already been acquired. In other words, by executing steps S101 to S103 again, the model acquisition unit 11 requests the provider DS to update the model MLi-1.

[0046] In step S104, the request acquisition unit 21 of the model providing device 20 acquires the task, the learning data, and information instructing the creation of the model MLi in accordance with steps S102 and S103. The request acquisition unit 21 also displays the acquired information on the display unit 250.

[0047] In step S105, the model generation unit 22 generates a model MLi that executes the task presented by the reward presentation device 10, and provides the model MLi to the reward presentation device 10. Specifically, the model generation unit 22 generates a model MLi that executes the task using the learning data presented by the reward presentation device 10, and provides the model MLi by transmitting the generated model MLi to the reward presentation device 10.

[0048] In step S106, the model acquisition unit 11 of the reward presentation device 10 acquires the model MLi from the model providing device 20. As mentioned above, step S106 may be executed in response to the re-execution of steps S101 to S103. In this case, the model MLi-1 has already been acquired. In other words, in step S106, the model acquisition unit 11 acquires the model MLi, which is an updated model.

[0049] In step S107, the model acquisition unit 11 determines whether the model MLi acquired from the model providing device 20 satisfies specifications. The specifications include the above-mentioned tasks. The specifications may also include a requirement that the performance of the model MLi on the training data meets a predetermined level.

[0050] If it is determined that the model MLi acquired from the model providing device 20 does not satisfy the specifications (No in step S107), the model acquiring unit 11 repeats the process from step S103. In other words, the model acquiring unit 11 again requests the provider DS to provide a model MLi that satisfies the specifications.

[0051] If it is determined that the model MLi acquired from the model providing device 20 satisfies the specifications (Yes in step S107), the model acquisition unit 11 records the model ID and operation start date and time of the model MLi in the record of the model pool table registered in step S101. As the operation start date and time, for example, the date and time corresponding to the time when step S108 is executed may be recorded.

[0052] Steps S101 to S108 constitute an example of a model acquisition process for acquiring, from a provider DS, a model MLi that has been generated using learning data to execute a predetermined task.

[0053] In step S109, the control unit 110 determines whether the model MLi acquired in step S106 is an updated model. In other words, the control unit 110 may determine whether a model MLi-1 already in operation exists. In this case, if a model MLi-1 in operation exists, the acquired model MLi is an updated model, and if a model MLi-1 in operation does not exist, the acquired model MLi is not an updated model.

[0054] If the answer to step S109 is Yes, the use of the model MLi-1 that is already in operation will be terminated. In this case, the processing from the next step S110 onwards will be executed. If the answer to step S109 is No, the processing from steps S110 to S111 will not be executed, and the processing from the next step S112 onwards will be executed.

[0055] In step S110, the control unit 110 records the operation end date and time in the record corresponding to the model MLi-1 that is already in operation in the model pool table. The operation end date and time may be, for example, the date and time corresponding to the time when step S110 is executed.

[0056] Step S111 constitutes an example of a reward presentation process. In step S111, the reward presentation unit 13 determines a reward based on the lead time from the request for updating the model MLi-1 to the acquisition of the updated model MLi. The reward presentation unit 13 also registers a new record including the determined reward in the reward table. For example, the reference period recorded in the record may be the period from the request date and time for the model MLi to the start date and time for operation of the model MLi. The length of the reference period may be recorded as the lead time. The reward may be determined linearly or nonlinearly depending on the lead time. The rule or function for calculating the reward from the lead time may be determined in advance.

[0057] In step S112, the control unit 110 starts operation of the latest model MLi as the model MLi currently in operation. If the answer is Yes in step S109, the latest model MLi is the model MLi as an updated model. That is, operation of the model MLi as the updated model is started. Also, if the answer is No in step S109, the processing of step S112 is executed. In this case, the latest model MLi is the model MLi that has been provided for the first time in correspondence with the above-mentioned task and learning data. That is, operation of the model MLi that has been provided for the first time is started.

[0058] Step S113 constitutes an example of evaluation processing. In step S113, the evaluation unit 12 registers a new record in the evaluation table in accordance with the use of the model MLi in operation. The evaluation unit 12 also calculates an evaluation index during operation and records the evaluation index during operation, the use date and time, the model output, and the true value in the record.

[0059] Here, "using the model MLi in operation" means inputting operational data different from the learning data into the model MLi and obtaining output data. For example, in response to a request from a user of a terminal (not shown), the control unit 110 may input operational data received from the terminal into the model MLi and obtain output data from the model MLi. Furthermore, the output data from the model MLi may be presented to the user. In this way, the model MLi is "used."

[0060] When a model MLi is used in this manner, the evaluation unit 12 records the date and time when the model MLi was used as the usage date and time in one new record for each model MLi. The evaluation unit 12 also records output data from the model MLi corresponding to the operational data as the model output in the record. The evaluation unit 12 also acquires and records a true value corresponding to the operational data in the record. As described above, the evaluation unit 12 may also record an estimate of the true value corresponding to the operational data in the record. The evaluation unit 12 also records an operational performance evaluation index (in this example, prediction accuracy) calculated from the model output and the true value as the operational evaluation index in the record.

[0061] In step S114, the control unit 110 determines whether a regeneration condition is met. The regeneration condition indicates a condition for determining that the model MLi should be updated. For example, the regeneration condition may be, but is not limited to, that the operational evaluation index calculated for each use falls below a threshold value for a predetermined number of consecutive times, or that the average value of the operational evaluation index indicating the operational performance of the most recent multiple times falls below a threshold value.

[0062] If the answer to step S114 is Yes, the control unit 110 repeats the process from step S101. As a result, a model MLi+1 acquired by a new request is acquired, and a lead time remuneration according to the lead time from the new request to acquisition is determined. If the answer to step S114 is No, the process from step S115 onwards is executed.

[0063] In step S115, the control unit 110 determines whether (Condition 1) a predetermined period has elapsed and / or whether (Condition 2) a deviation condition has been met. Specifically, for (Condition 1), it is determined whether a predetermined period has elapsed since the previous remuneration was determined. However, if a remuneration has never been determined, it is determined whether a predetermined period has elapsed since the start date and time of operation of the model MLi. It should be noted that if operation of the model MLi has ended, it is determined that the predetermined period has elapsed.

[0064] The deviation condition in (Condition 2) is a condition indicating that the most recently calculated operational evaluation index deviates from the operational evaluation index calculated earlier than the operational evaluation index. Specific examples of the deviation condition have been described above, and therefore detailed explanations will not be repeated. If both (Condition 1) and (Condition 2) are determined to be No, the processing from step S113 is repeated. As a result, operational evaluation indexes are continuously recorded according to the use of the model MLi in operation. If at least one of (Condition 1) and (Condition 2) is determined to be Yes, the processing of the next step S116 is executed.

[0065] Step S116 constitutes an example of a reward presentation process. In step S116, the reward presentation unit 13 determines a performance reward based on the representative value of the operational evaluation index during the reference period. If condition 1 is satisfied in step S115, the reference period is the most recent predetermined period. If condition 2 is satisfied in step S115, the reference period is the time when the operational evaluation index was most recently calculated. For example, the reward presentation unit 13 registers a new record in the evaluation table and records the reference period. The reward presentation unit 13 also calculates a representative value of the operational evaluation index evaluated for each use of the model MLi in operation during the reference period and records the calculated representative value as the reference index in the record. The reward presentation unit 13 also records a reward calculated according to the representative value in the record. The reward presentation unit 13 also records "performance reward" as the type in the record. The reward may be determined linearly or nonlinearly according to the representative value of the operational evaluation index. The rule or function for calculating the reward from the representative value of the performance indicator is determined in advance. Furthermore, such rule or function may be determined independently of the rule or function for calculating the reward according to the lead time.

[0066] In step S117, the control unit 110 determines whether the performance-based reward determined in step S116 is the initial reward corresponding to the model MLi as an updated model. For example, when the model MLi in operation is an updated model, the control unit 110 may determine whether the start date and time of the reference period recorded in step S116 matches the operation start date and time of the model ML in operation recorded in step S108. If the start date and time of the reference period matches the operation start date and time, the performance-based reward is the initial reward. Furthermore, if the start date and time of the reference period does not match the operation start date and time, the performance-based reward is not the initial reward.

[0067] If the answer to step S117 is Yes, an improvement reward needs to be determined. In this case, the process proceeds to the next step S118 and subsequent steps. If the answer to step S117 is No, the process proceeds to the next step S119 and subsequent steps without executing step S118.

[0068] Step S118 constitutes an example of a reward presentation process. In step S118, the reward presentation unit 13 calculates the improvement in the operational performance of the updated model MLi relative to the pre-update model MLi-1, and determines a reward according to the calculated improvement. For example, the reward presentation unit 13 registers a new record in the reward table and records the same reference period as in step S116 as the reference period. The reward presentation unit 13 also records the calculated improvement as a reference index in the record. The reward presentation unit 13 also records the reward calculated according to the improvement. The reward presentation unit 13 also records "improvement reward" as the type in the record.

[0069] To calculate the improvement range, for example, the reward presentation unit 13 refers to the reward table as follows. Specifically, the reward presentation unit 13 calculates the difference between (1) the representative value of the first operational evaluation index corresponding to the updated model MLi and (2) the representative value of the last operational evaluation index corresponding to the pre-update model MLi-1 as the improvement range of the operational evaluation index. More specifically, the representative value of (1) is the representative value of the operational evaluation index referenced to calculate the first performance reward corresponding to the updated model MLi, and is recorded, for example, in the record registered in the reward table in step S116. The representative value of (2) is the representative value of the operational evaluation index referenced to calculate the last performance reward corresponding to the pre-update model MLi-1, and is recorded, for example, in the record indicating the performance reward immediately preceding the record registered in the reward table in step S116. However, the details of the calculation process of the improvement range are not limited to the above-described process.

[0070] In step S119, the control unit 110 determines whether or not predetermined payment conditions are met. Specific examples of payment conditions have been described above, and detailed description will not be repeated. If the determination in step S119 is No, the processing from step S113 onwards is repeated. As a result, the operation evaluation index continues to be recorded according to the use of the model MLi in operation, and the performance reward and / or improvement reward are recorded. If the determination in step S119 is Yes, the processing from the next step S120 onwards is executed.

[0071] Step S120 constitutes an example of a reward presentation process. In step S120, the reward presentation unit 13 presents a reward to the provider DS. Specifically, the reward presentation unit 13 presents information indicating rewards recorded in the reward table that have not yet been presented to the provider DS to the model providing device 20, thereby presenting the information to the provider DS. For example, when the payment conditions are met for the first time, the reward presentation unit 13 presents information indicating the rewards contained in all records recorded in the reward table to the provider DS. Furthermore, when the payment conditions are met for the second or subsequent time, the reward presentation unit 13 presents information indicating the rewards contained in each record registered in the reward table since the previous reward presentation to the provider DS. When there are multiple applicable records, the information indicating the reward may include the total and breakdown of the multiple rewards.

[0072] In step S121, the reward presenting unit 23 of the model providing device 20 displays information indicating the reward received in step S120 on the display unit 250. Here, operation of the model MLi has already started in step S112. In other words, the information indicating the reward is presented after operation of the model ML has started.

[0073] Thereafter, the reward presentation system 1A repeats the process from step S113. As a result, the operation evaluation index is continuously recorded according to the use of the model MLi in operation, and the performance reward and / or improvement reward is recorded. This concludes the explanation of the reward presentation method S1A.

[0074] (Specific Example of Reward Presentation Method S1A) A specific example of reward presentation method S1A will be described with reference to Fig. 9. Fig. 9 is a sequence diagram illustrating a specific example of reward presentation method S1A. As shown in Fig. 9, the specific example of reward presentation method S1A includes steps M1 to M15.

[0075] Step M1 is a specific example of steps S101 to S103. In step M1, the reward presentation device 10 requests the model providing device 20 to provide model ML1 by presenting a task and learning data to the model providing device 20 on January 5, 2023. The reward presentation device 10 also registers record R11 in the model pool table shown in FIG. 5 and records the requested date and time "January 5, 2023."

[0076] Step M2 is a specific example of steps S104 to S108. In step M2, the model providing device 20 generates a model ML1 that executes a task using the learning data, and provides the model ML1 to the reward presentation device 10 on January 10, 2023. The reward presentation device 10 determines that the model ML1 satisfies the specifications (Yes in S107), and records the model ID "ML1" and the operation start date and time "January 10, 2023" in record R11 of the model pool table shown in FIG. 5 .

[0077] Step M3 is an example of steps S109 and S112. Furthermore, since the acquired model ML1 is not an updated model (No in S109), the reward presentation device 10 starts using the model ML1 (S112). In other words, the model currently in use is the model ML1.

[0078] Step M4 is a specific example of steps S113 to S115. Here, it is assumed that model ML1 was first used on January 10, 2023. As a result, the reward presentation device 10 registers record R21 in the evaluation table as shown in FIG. 6, recording the usage date and time "January 10, 2023," model output "92," and true value "100." The reward presentation device 10 also records the prediction accuracy "92%" calculated from the model output and true value as the performance value evaluation index in the record. It is also assumed that model ML1 was used multiple times (S113) without the regeneration condition being met (No in S114), the predetermined period not having elapsed, and the deviation condition not being satisfied (No in S115). As a result, it is assumed that record set RS22 was registered in the evaluation table as shown in FIG. 6 between January 10, 2023 and January 19, 2023.

[0079] Step M5 is a specific example of steps S115 and S116. Here, on January 19, 2023, the reward presentation device 10 determines that a predetermined period (10 days) has passed since January 10, 2023, the start date and time of the model's operation (Yes in S115). Therefore, the reward presentation device 10 calculates, as a reference index, the representative value of the operation evaluation indexes in the record set RS22 recorded during the predetermined period from January 10, 2023 to January 19, 2023. Here, the reward presentation device 10 calculates the average value of these operation evaluation indexes (90%) as the reference index. Furthermore, the reward presentation device 10 calculates a reward of 9,000 yen based on the reference index of 90%. Note that a rule is established here that the performance reward is calculated as the reference index x 100. As a result, record R31 is registered in the reward table, as shown in FIG. 7 . Record R31 records the reference period "January 10, 2023 to January 19, 2023", the reference index "90", the reward "9,000", and the type "performance reward".

[0080] By repeating steps M4 and M5, it is assumed that record set RS23 is registered in the evaluation table between January 20, 2023 and January 29, 2023, as shown in Figure 6. Furthermore, record R32 is registered in the reward table, as shown in Figure 7. Record R32 records the reference period "January 20, 2023 to January 29, 2023", the reference index "88" calculated with reference to record set RS23, reward "8800", and type "performance reward".

[0081] Step M6 is a specific example of steps S117 to S120. Here, on January 31, 2023, model ML1 is not an updated model (No in S117). Furthermore, the payment condition of being the end of the month is met (Yes in S119). Therefore, the reward presentation device 10 transmits information indicating the rewards recorded in the reward table that have not yet been presented to the provider DS to the model providing device 20. In this case, the relevant reward is the total of the rewards recorded in records R31 to R32 shown in FIG. 7, which is "17,800," so information such as "Total reward for January: 17,800 yen" is transmitted.

[0082] Steps M4 to M6 are subsequently repeated, and records R24 to R26 are registered in the evaluation table as shown in FIG.

[0083] Step M7 is a specific example of steps S114 and S101 to S103. Here, the regeneration condition is assumed to be that the prediction accuracy is 85% or less for three consecutive times, and on February 25, 2023, the reward presentation device 10 determines that the regeneration condition is met (Yes in S114). Therefore, on February 25, 2023, the reward presentation device 10 requests the model providing device 20 to update model ML1 by presenting the task, learning data, and information for performance improvement. In other words, the reward presentation device 10 requests model ML2. The reward presentation device 10 also registers record R12 in the model pool table shown in FIG. 5 and records the request date and time "February 25, 2023."

[0084] Step M8 is a specific example of steps S104 to S108. In step M8, the model providing device 20 generates model ML2 that executes a task using the learning data and information for performance improvement. Model ML2 is an updated version of model ML1. On March 8, 2023, the model providing device 20 provides model ML2 to the reward presentation device 10. The reward presentation device 10 determines that model ML2 satisfies the specifications (Yes in S107), and records the model ID "ML2" and the operation start date and time "March 8, 2023" in record R12 of the model pool table shown in FIG. 5 .

[0085] Note that until March 7, 2023, when model ML2 begins operation, steps M4 to M6 continue to be repeated for model ML1 in operation, the evaluation table and reward table are updated, and rewards are presented to the provider DS. For example, as shown in FIG. 7, record R33 is registered in the reward table. This indicates the reward determined for the last reference period before model ML1 ends operation, "March 1, 2023 to March 7, 2023."

[0086] Step M9 is a specific example of steps S109 to S111. In step M9, because the acquired model ML2 is an updated model (Yes in S109), the reward presentation device 10 records the operation end date and time "March 8, 2023" in record R11 for the currently operating model ML1 (S110). The reward presentation device 10 also starts operating model ML2. The reward presentation device 10 also calculates the lead time as 11 days, the number of days from the requested date and time to the day before the operation start date and time. The reward presentation device 10 also calculates the difference between the lead time of "11 days" and the standard lead time of "15 days," which is "4," as the reference index. The reward presentation device 10 also calculates a reward of "4,000 yen" based on the reference index of "4." Note that a rule is defined here that the lead time reward is calculated as the reference index x 1,000. As a result, record R34 is registered in the reward table, as shown in FIG. 7 . Record R34 records the reference period "March 8, 2023", the reference index "4", the remuneration "4,000", and the type "lead time remuneration".

[0087] Step M10 is an example of step S112. The reward presentation device 10 starts using the updated model ML2. In other words, the model currently in use becomes model ML2.

[0088] Step M11 is explained in the same way as step M4, except that the model in operation is updated from model ML1 to model ML2. As a result, it is assumed that record set RS27 is registered in the evaluation table as shown in FIG. 6 between March 8, 2023 and March 17, 2023.

[0089] Step M12 is explained in the same way as step M5, except that the model in operation is updated from model ML1 to model ML2. As a result, record R35 is registered in the reward table, as shown in FIG. 7. Record R35 records the reference period "March 8, 2023 to March 17, 2023," the reference index "94" calculated with reference to record set RS27, the reward "9,400," and the type "performance reward."

[0090] Step M13 is an example of steps S117 and S118. Here, model ML2 is an updated model on March 17, 2023 (Yes in S117). Therefore, the reward presentation device 10 calculates the reference index as the improvement range "12" between the first reference index "94" (record R35) corresponding to the updated model ML2 and the last reference index "82" (record R33) corresponding to the pre-update model ML1. The reward presentation device 10 also calculates a reward of "1,200" yen based on the reference index "12." Note that a rule is established here that the improvement reward is calculated as the reference index x 100. As a result, record R36 is registered in the reward table, as shown in FIG. 7 . Record R36 records the reference period "March 8, 2023," the reference index "12," the reward "1,200," and the type "improvement reward."

[0091] Step M14 will be explained in the same manner as step M11. As a result, it is assumed that a record set RS28 is registered in the evaluation table as shown in FIG. 6 between March 18, 2023 and March 26, 2023.

[0092] Step M15 is explained in the same manner as step M12. As a result, record R37 is registered in the reward table, as shown in Figure 7. Record R37 records the reference period "March 18, 2023 to March 26, 2023," the reference index "92" calculated with reference to record set RS28, reward "9,200," and type "performance reward."

[0093] Step M16 is a specific example of steps S119 to S120. On March 31, 2023, the payment condition of being the end of the month is met (Yes in S119). Therefore, the reward presentation device 10 transmits to the model providing device 20 information indicating the rewards recorded in the reward table that have not yet been presented to the provider DS. In this case, the relevant reward is the total of "32,000" yen of the rewards recorded in records R33 to R37 shown in FIG. 7, so information such as "Total reward for March: 32,000 yen" is transmitted.

[0094] Until the operation of the model ML2 is completed, steps M14 to M16 are continuously repeated for the model ML2 in operation, the evaluation table and the reward table are updated, and rewards are presented to the provider DS.

[0095] This concludes the explanation of the specific example of the reward presentation method S1A.

[0096] (Effects of Reward Presentation System 1A) As described above, the reward presentation system 1A employs a configuration in which the reward presentation unit 13 presents a reward after the start of operation of the model ML. Therefore, with this configuration, in addition to the effects of the reward presentation device 1, the provider DS can obtain an effect that the reward reflects the performance during operation.

[0097] Furthermore, the reward presentation system 1A is configured to determine the reward each time a predetermined period of time has passed since the start of operation of the model ML, based on the operational performance evaluated during the most recent predetermined period. Therefore, with this configuration, in addition to the effects achieved by the reward presentation device 1, it is possible to obtain the effect of periodically revising the reward in response to changes in operational performance.

[0098] Furthermore, in the reward presentation system 1A, the reward presentation unit 13 is configured to determine a reward based on at least the most recently evaluated operational performance when a deviation condition is satisfied indicating that the most recently evaluated operational performance deviates from the operational performance evaluated in the past. Therefore, with this configuration, in addition to the effects achieved by the reward presentation device 1, the effect of being able to revise the reward when a change in operational performance satisfies the deviation condition is obtained.

[0099] Furthermore, in the reward presentation system 1A, the model acquisition unit 11 acquires a model MLi+1, which is an updated version of the model MLi, from the provider DS after the model MLi begins operation, and the reward presentation unit 13 determines a reward according to the degree of improvement in the operational performance of the updated model (model MLi+1) relative to the operational performance of the model MLi. Therefore, with this configuration, in addition to the effects achieved by the reward presentation device 1, the effect of increasing the motivation of the provider DS to generate an updated model (model MLi+1) that achieves a greater degree of improvement is obtained.

[0100] Furthermore, the reward presentation system 1A employs a configuration in which the model acquisition unit 11 acquires an updated model (model MLi+1) by requesting the provider DS to update the model MLi, and the reward presentation unit 13 determines the reward according to the lead time from the request to update the model MLi to acquiring the updated model (model MLi+1). Therefore, with this configuration, in addition to the effects achieved by the reward presentation device 1, the effect of improving the motivation of the provider DS to shorten the lead time until generating the updated model (model MLi+1) is obtained.

[0101] Furthermore, the reward presentation system 1A employs a configuration in which the model acquisition unit 11 acquires an updated model (model MLi+1) by providing information for improving the performance of model MLi to the provider DS. Therefore, this configuration has the effect of making it easier to improve the updated model (model MLi+1) in addition to the effects of the reward presentation device 1.

[0102] Furthermore, the reward presentation system 1A employs a configuration in which the evaluation unit 12 evaluates operational performance based on an estimate of the true value corresponding to the operational data. Therefore, with this configuration, in addition to the effects achieved by the reward presentation device 1, it is possible to obtain an effect of being able to evaluate operational performance and determine a reward even when the true value cannot be easily or immediately obtained.

[0103] [Variation 1] In the above-described exemplary embodiment 2, the reward presentation device 10 has been described as obtaining an updated model MLi+1 by requesting the provider DS to update the model MLi. However, the reward presentation device 10 may also obtain an updated model MLi+1 without requesting an update of the model MLi. The model presentation device 20 may be configured to periodically update the model MLi and provide it to the reward presentation device 10. The model presentation device 20 may also be configured to update the model MLi based on the operation of the provider DS and provide it to the reward presentation device 10. In this case, the process of recording the request date and time and the process of determining the lead time reward are omitted. Other points are described in the same manner as in exemplary embodiment 2.

[0104] This modification can reduce the processing cost of requesting an update of the model MLi, the communication cost, and the processing cost of determining whether to request an update of the model MLi in the reward presentation device 10. Furthermore, the provider DS can proactively increase the reduced reward by updating the model MLi and providing the model ML2 without waiting for a request.

[0105] [Variation 2] In the above-described exemplary embodiment 2, the model acquisition unit 11 may acquire a model ML from each of multiple provider DSs. In this case, the reward presentation unit 13 may present a reward determined according to the performance of the model ML during operation to the provider DS of a model ML selected from the acquired multiple model MLs. In this variation, each record in the model pool table shown in FIG. 4 may record a provider ID that identifies the provider DS. Furthermore, the evaluation table shown in FIG. 5 and the reward table shown in FIG. 6 may be recorded for each provider DS. The model ML selected from the multiple model MLs may be, for example, the model ML with the highest performance during operation among the multiple model MLs, or the model ML with the highest performance at the time of generation. Furthermore, each time a user needs to use a model ML, one of the multiple model MLs may be selected. Alternatively, the same model ML may be used until the regeneration condition is satisfied, and a new model ML may be selected from the multiple updated model MLs in response to a request to update the model ML.

[0106] According to this modification, multiple providers DS compete to generate a model ML with higher performance during operation, which has the advantage of making it possible to obtain a model ML with higher performance during operation.

[0107] [Variation 3] In the above-described exemplary embodiment 2, the reward determined by the reward presentation unit 13 may be set to one or both of a maximum value and a minimum value. For example, a maximum value may be set to a reward based on the performance during operation at the start of operation. This is because, in general, the performance of the model ML declines over time after the start of operation, and therefore the reward at the start of operation can be considered to be the highest level. However, the maximum value is not limited to the above-described example. Furthermore, the reward determined most recently when the regeneration condition is satisfied may be set to the minimum value. This is because the reward determined immediately before the regeneration condition is satisfied can be considered to be the lowest level. However, the minimum value is not limited to the above-described example. Furthermore, a maximum value, a minimum value, or both may be set to the total reward presented when a specified payment condition is satisfied.

[0108] In this modified example, if a minimum value of the reward is set, the provider DS of the model ML has the advantage that the minimum amount of reward is guaranteed. This allows the provider DS to maintain motivation to provide the model ML. In addition, in this modified example, if a maximum value of the reward is set, the manager AD of the model ML has the advantage that the reward to be paid will not be too high.

[0109] [Variation 4] In the above-described exemplary embodiment 2, the reward determined by the reward presentation unit 13 may be determined so that the difference between the current reward and the previously determined reward falls within a predetermined range. For example, an upper limit on the difference between the performance reward may be set. Furthermore, the reward presented when a predetermined payment condition is met may be determined so that the difference between the current reward and the reward presented when the previous payment condition was met falls within a predetermined range.

[0110] This modification has the advantage that the rewards obtained by the provider DS of the model ML do not fluctuate significantly, which helps maintain the provider DS's motivation to provide the model ML.

[0111] [Example of implementation using software] Some or all of the functions of the reward presentation device 1, the reward presentation device 10, and the model provision device 20 (hereinafter also referred to as "each of the above devices") may be implemented using hardware such as an integrated circuit (IC chip), or may be implemented using software.

[0112] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Figure 10 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0113] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0114] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0115] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0116] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0117] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0118] (Appendix A1) A reward presentation device comprising: a model acquisition means for acquiring from a provider a model that executes a predetermined task and is generated using training data; an evaluation means for evaluating the operational performance of the model for operational data different from the training data; and a reward presentation means for presenting to the provider a reward determined according to the operational performance.

[0119] (Supplementary Note A2) The reward presentation device according to Supplementary Note A1, wherein the reward presentation means presents the reward after operation of the model begins.

[0120] (Appendix A3) The reward presentation device described in Appendix A1 or A2, wherein the reward presentation means determines the reward every time a predetermined period has elapsed since the start of operation of the model, based on the operation performance evaluated during the most recent predetermined period.

[0121] (Appendix A4) The reward presentation device described in any one of Appendices A1 to A3, wherein the reward presentation means determines the reward based on at least the most recently evaluated operational performance when a deviation condition is satisfied indicating that the most recently evaluated operational performance deviates from an operational performance evaluated earlier than the currently evaluated operational performance.

[0122] (Appendix A5) The reward presentation device described in any one of Appendices A1 to A4, wherein the model acquisition means acquires an updated model obtained by updating the model from the provider after operation of the model begins, and the reward presentation means determines the reward depending on the degree of improvement in operational performance of the updated model relative to the operational performance of the model.

[0123] (Appendix A6) The reward presentation device described in Appendix A5, wherein the model acquisition means acquires the updated model by requesting the provider to update the model, and the reward presentation means determines the reward depending on a lead time from the request for the model update to acquisition of the updated model.

[0124] (Supplementary Note A7) The reward presentation device according to Supplementary Note A5 or A6, wherein the model acquisition means acquires the updated model by providing information for improving performance of the model to the provider.

[0125] (Supplementary Note A8) The reward presentation device according to any one of Supplementary Notes A1 to A7, wherein the evaluation means evaluates the operational performance based on an estimate of a true value corresponding to the operational data.

[0126] (Appendix A9) The reward presentation device described in any one of Appendices A1 to A8, wherein the model acquisition means acquires a model from each of a plurality of providers, and the reward presentation means presents a reward to the provider of a model selected from the plurality of models, the reward being determined according to the performance of the model during operation.

[0127] (Supplementary Note A10) The reward presentation device according to any one of Supplementary Notes A1 to A9, wherein the reward determined by the reward presentation means has a maximum value and / or a minimum value.

[0128] (Supplementary Note A11) The reward presentation device according to any one of Supplementary Notes A1 to A10, wherein the reward presentation means determines the reward such that a difference between the reward determined previously and the reward determined now is within a predetermined range.

[0129] (Supplementary Note A12) The reward presentation device according to any one of Supplementary Note A1 to Supplementary Note A11, wherein at least one of the plurality of models is a machine learning model.

[0130] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0131] (Appendix B1) A reward presentation method including: a model acquisition process in which at least one processor acquires from a provider a model that executes a predetermined task and is generated using training data; an evaluation process in which the at least one processor evaluates the operational performance of the model for operational data different from the training data; and a reward presentation process in which the at least one processor presents to the provider a reward determined according to the operational performance.

[0132] (Supplementary Note B2) The reward presentation method according to Supplementary Note B1, wherein in the reward presentation process, the at least one processor presents the reward after operation of the model begins.

[0133] (Appendix B3) The reward presentation method described in Appendix B1 or B2, wherein in the reward presentation process, the at least one processor determines the reward every time a predetermined period has elapsed since the model began operation, based on the operational performance evaluated during the most recent predetermined period.

[0134] (Appendix B4) The reward presentation method described in any one of Appendices B1 to B3, wherein in the reward presentation process, the at least one processor determines the reward based on at least the most recently evaluated operational performance when a deviation condition is satisfied indicating that the most recently evaluated operational performance deviates from an operational performance evaluated earlier than the currently evaluated operational performance.

[0135] (Appendix B5) The reward presentation method described in any one of Appendices B1 to B4, wherein in the model acquisition process, the at least one processor acquires an updated model obtained by updating the model from the provider after the model has started to be used, and in the reward presentation process, the at least one processor determines the reward depending on the degree of improvement in the operational performance of the updated model relative to the operational performance of the model.

[0136] (Appendix B6) The reward presentation method described in Appendix B5, wherein, in the model acquisition process, the at least one processor acquires the updated model by requesting the provider to update the model, and in the reward presentation process, the at least one processor determines the reward depending on a lead time from the request to update the model to acquiring the updated model.

[0137] (Supplementary Note B7) The reward presentation method according to Supplementary Note B5 or B6, wherein in the model acquisition process, the at least one processor acquires the updated model by providing information for improving performance of the model to the provider.

[0138] (Supplementary Note B8) The reward presentation method according to any one of Supplementary Notes B1 to B7, wherein in the evaluation process, the at least one processor evaluates the operational performance based on an estimate of a true value corresponding to the operational data.

[0139] (Appendix B9) The reward presentation method described in any one of Appendices B1 to B8, wherein in the model acquisition process, the at least one processor acquires a model from each of a plurality of providers, and in the reward presentation process, the at least one processor presents a reward to the provider of a model selected from the plurality of models, the reward being determined according to the model's performance during operation.

[0140] (Supplementary Note B10) The reward presentation method according to any one of Supplementary Notes B1 to B9, wherein the reward determined by the reward presentation process has a maximum value and / or a minimum value set.

[0141] (Supplementary Note B11) The reward presentation method according to any one of Supplementary Notes B1 to B10, wherein the reward determined by the reward presentation process is determined so that a difference between the reward determined previously and the reward is within a predetermined range.

[0142] (Supplementary Note B12) The reward presentation device according to any one of Supplementary Note B1 to Supplementary Note B11, wherein at least one of the plurality of models is a machine learning model.

[0143] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0144] (Appendix C1) A reward presentation program that causes a computer to function as a reward presentation device, causing the computer to function as: a model acquisition means that acquires from a provider a model that executes a specified task and is generated using training data; an evaluation means that evaluates the operational performance of the model on operational data different from the training data; and a reward presentation means that presents to the provider a reward determined according to the operational performance.

[0145] (Supplementary Note C2) The reward presentation program according to Supplementary Note C1, wherein the reward presentation means presents the reward after operation of the model begins.

[0146] (Appendix C3) The reward presentation program according to appendix C1 or C2, wherein the reward presentation means determines the reward each time a predetermined period has elapsed since the start of operation of the model, based on the performance during operation evaluated during the most recent predetermined period.

[0147] (Appendix C4) The reward presentation program described in any one of Appendices C1 to C3, wherein the reward presentation means determines the reward based on at least the most recently evaluated operational performance when a deviation condition is satisfied indicating that the most recently evaluated operational performance deviates from an operational performance evaluated in the past.

[0148] (Appendix C5) The reward presentation program described in any one of Appendices C1 to C4, wherein the model acquisition means acquires an updated model obtained by updating the model from the provider after operation of the model begins, and the reward presentation means determines the reward depending on the degree of improvement in operational performance of the updated model relative to the operational performance of the model.

[0149] (Appendix C6) The reward presentation program described in Appendix C5, wherein the model acquisition means acquires the updated model by requesting the provider to update the model, and the reward presentation means determines the reward depending on a lead time from the request for the model update to acquisition of the updated model.

[0150] (Supplementary Note C7) The reward presentation program according to Supplementary Note C5 or C6, wherein the model acquisition means acquires the updated model by providing information for improving performance of the model to the provider.

[0151] (Supplementary Note C8) The reward presentation program according to any one of Supplementary Notes C1 to C7, wherein the evaluation means evaluates the operational performance based on an estimate of a true value corresponding to the operational data.

[0152] (Appendix C9) The reward presentation program described in any one of Appendices C1 to C8, wherein the model acquisition means acquires a model from each of a plurality of providers, and the reward presentation means presents a reward to the provider of a model selected from the plurality of models, the reward being determined according to the model's performance during operation.

[0153] (Supplementary Note C10) The reward presentation program according to any one of Supplementary Note C1 to C9, wherein the reward determined by the reward presentation means has a maximum value and / or a minimum value.

[0154] (Supplementary Note C11) The reward presentation program according to any one of Supplementary Note C1 to C10, wherein the reward determined by the reward presentation means is determined so that a difference from a previously determined reward falls within a predetermined range.

[0155] (Supplementary Note C12) The reward presentation device according to any one of Supplementary Note C1 to Supplementary Note C11, wherein at least one of the plurality of models is a machine learning model.

[0156] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0157] (Appendix D1) A reward presentation device comprising at least one processor, the at least one processor performing: a model acquisition process for acquiring from a provider a model that is generated using training data and that executes a predetermined task; an evaluation process for evaluating the operational performance of the model on operational data that is different from the training data; and a reward presentation process for presenting to the provider a reward determined according to the operational performance.

[0158] The reward presentation device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0159] (Supplementary Note D2) The reward presentation device according to Supplementary Note D1, wherein in the reward presentation process, the at least one processor presents the reward after operation of the model begins.

[0160] (Appendix D3) The reward presentation device described in Appendix D1 or D2, wherein in the reward presentation process, the at least one processor determines the reward every time a predetermined period has passed since the model began operation, based on the operational performance evaluated during the most recent predetermined period.

[0161] (Appendix D4) The reward presentation device described in any one of Appendices D1 to D3, wherein in the reward presentation process, the at least one processor determines the reward based on at least the most recently evaluated operational performance when a deviation condition indicating that the most recently evaluated operational performance deviates from an operational performance evaluated earlier than the most recently evaluated operational performance is satisfied.

[0162] (Appendix D5) The reward presentation device described in any one of Appendices D1 to D4, wherein in the model acquisition process, the at least one processor acquires an updated model obtained by updating the model from the provider after the model has started to be used, and in the reward presentation process, the at least one processor determines the reward depending on the degree of improvement in the operational performance of the updated model relative to the operational performance of the model.

[0163] (Appendix D6) The reward presentation device described in Appendix D5, wherein in the model acquisition process, the at least one processor acquires the updated model by requesting the provider to update the model, and in the reward presentation process, the at least one processor determines the reward depending on the lead time from the request to update the model to acquiring the updated model.

[0164] (Supplementary Note D7) The reward presentation device according to Supplementary Note D5 or D6, wherein in the model acquisition process, the at least one processor acquires the updated model by providing information for improving performance of the model to the provider.

[0165] (Appendix D8) The reward presentation device according to any one of Appendices D1 to D7, wherein in the evaluation process, the at least one processor evaluates the operational performance based on an estimate of a true value corresponding to the operational data.

[0166] (Appendix D9) The reward presentation device described in any one of Appendices D1 to D8, wherein in the model acquisition process, the at least one processor acquires a model from each of a plurality of providers, and in the reward presentation process, the at least one processor presents a reward to the provider of a model selected from the plurality of models, the reward being determined according to the model's performance during operation.

[0167] (Supplementary Note D10) The reward presentation device according to any one of Supplementary Notes D1 to D9, wherein the reward determined by the reward presentation process has a maximum value and / or a minimum value set.

[0168] (Supplementary Note D11) The reward presentation device according to any one of Supplementary Notes D1 to D10, wherein the reward determined by the reward presentation process is determined so that a difference from a previously determined reward falls within a predetermined range.

[0169] (Supplementary Note D12) The reward presentation device according to any one of Supplementary Note D1 to Supplementary Note D11, wherein at least one of the plurality of models is a machine learning model.

[0170] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the appendices below, and various modifications are possible within the scope of the claims.

[0171] (Appendix E1) A non-transitory recording medium having recorded thereon a reward presentation program that causes a computer to function as a reward presentation device, the reward presentation program causing the computer to execute the following: a model acquisition process that acquires from a provider a model that is generated using training data and that executes a specified task; an evaluation process that evaluates the operational performance of the model on operational data that is different from the training data; and a reward presentation process that presents to the provider a reward determined according to the operational performance.

[0172] REFERENCE SIGNS LIST 1, 10, 23 Reward presentation device 1A Reward presentation system 11 Model acquisition unit 12 Evaluation unit 13 Reward presentation unit 20 Model providing device 21 Requirement acquisition unit 22 Model generation unit 110, 210 Control unit 120, 220 Storage unit 130, 230 Communication unit 240 Input unit 250 Display unit C1 Processor C2 Memory

Claims

1. A model acquisition means for obtaining a model from a provider that performs a predetermined task, which is generated using training data, An evaluation means for evaluating the operational performance of the model on operational data different from the training data, A reward presentation means that presents the reward determined according to the operational performance to the provider, A reward presentation device equipped with the following features.

2. The reward offering means presents the reward after the start of operation of the model, as described in claim 1.

3. The reward offering means determines the reward based on the operational performance evaluated during the most recent predetermined period, after a predetermined period has elapsed since the start of operation of the model. The reward presentation device according to claim 1 or 2.

4. The reward offering means determines the reward based at least on the most recently evaluated operational performance when a deviation condition is met indicating that the most recently evaluated operational performance deviates from the operational performance evaluated in the past. The reward presentation device according to claim 1 or 2.

5. The model acquisition means, after the start of operation of the model, acquires an updated model from the provider, which is an updated version of the model. The reward offering means determines the reward according to the degree of improvement in the operational performance of the updated model relative to the operational performance of the model. The reward presentation device according to claim 1 or 2.

6. The model acquisition means acquires the updated model by requesting the provider to update the model. The reward offering means determines the reward according to the lead time from the request for updating the model to obtaining the updated model. The reward presentation device according to claim 5.

7. The model acquisition means acquires the updated model by providing the provider with information for improving the performance of the model. The reward presentation device according to claim 5.

8. The evaluation means evaluates the operational performance based on an estimated true value corresponding to the operational data. The reward presentation device according to claim 1 or 2.

9. At least one processor performs a model acquisition process, which involves obtaining a model from a provider that performs a predetermined task, generated using training data. A reward presentation method comprising: an evaluation process in which at least one processor evaluates the operational performance of the model on operational data different from the training data; and a reward presentation process in which the at least one processor presents a reward determined according to the operational performance to the provider.

10. A program that causes a computer to function as a reward presentation device, wherein the computer, A model acquisition means for obtaining a model from a provider that performs a predetermined task, which is generated using training data, An evaluation means for evaluating the operational performance of the model on operational data different from the training data, A reward presentation means that presents the reward determined according to the operational performance to the provider, A reward-based program that functions as such.