Information processing device, method, and program

Abstract

An information processing device according to one embodiment comprises: an input unit for accepting input of a total reward, which is the total budget for rewards to be given to agents under the condition that the agents obtain rewards by increasing progress in tasks done by the agents in respective periods that are divided in a continuous length of time, as well as input of a total period of the continuous length of time; and a calculation unit for calculating, on the basis of the inputted total period, a number of divisions of the total period so as to maximize the sum of progress in the total period, and calculating, on the basis of the number of divisions, the total reward, and the total period, the progress in the periods that have been divided in accordance with the number of divisions, as well as the rewards obtained by the agents for the progress in the divided periods so as to maximize the sum of the progress in each of the divided periods.

Description

Information processing device, method and program

[0001] FIELD Embodiments of the present invention relate to an information processing device, method, and program.

[0002] It is important to model human behavior when trying to achieve a certain goal, such as completing an online course or achieving a step count goal over a certain period of time. In particular, "present bias," which places a relatively high value on current gains and losses and a relatively low value on future gains and losses, is known to have a significant impact on goal achievement. Effective interventions can be achieved by implementing optimized interventions based on modeling that takes this present bias into account.

[0003] As one such modeling method, for example, Non-Patent Document 1 discloses a model based on graph theory. In this model, states that a person can take are represented as vertices, and actions that a person can take in each state are represented as edges. A cost is assigned to each edge, representing the effort required for a person to take each action. A reward is assigned to each vertex, representing the reward for reaching the corresponding state through the action. An agent, such as a user, evaluates the payoff of a sequence of actions that the agent can take in the future on this graph, i.e., candidate paths on the graph, and selects the action with the highest payoff. The payoff of this sequence of actions is calculated by weighting future costs relatively lightly and recent costs relatively heavily using a present bias, also known as quasi-geometric discounting.

[0004] Furthermore, Non-Patent Document 2 discloses a model based on the above model, in which the task is limited to "a task of raising progress, expressed as real numbers within a time limit, to a target value" (hereinafter referred to as a "numerical accumulation task"). In this model, a human state is expressed as a pair of a discrete time and the progress at that time. This model has the great advantage of being able to express human behavior in a closed form, and by utilizing this advantage, theoretical optimal solutions or algorithms for finding these optimal solutions have been derived, such as a goal optimization problem or an optimization problem of a schedule for providing rewards to an agent (sometimes referred to as a reward schedule).

[0005] Jon Kleinberg and Sigal Oren. Time-inconsistent planning: a computational problem in behavioral economics. In Proceedings of the 15th ACM Conference on Economics and Computation, pages 547‐564, 2014. Yasunori Akagi, Naoki Marumo, and Takeshi Kurashima. Analytically tractable models for decision making under present bias, 2023.

[0006] The existing algorithms described above deal only with discrete time, so it is necessary to determine the discretization width. In addition, if the discretization width is relatively coarse, there will be a relatively large number of times that cannot be handled, and if the discretization width is relatively fine, the amount of calculation will be relatively large.

[0007] The present invention has been made in light of the above circumstances, and its object is to provide an information processing device, method, and program that can optimize the schedule for rewarding agents.

[0008] An information processing device according to one aspect of the present invention comprises an input unit that accepts input of a total reward, which is the total budget of rewards to be given to an agent, and a total period of continuous time, under the condition that the agent earns reward by increasing the agent's progress on a task in each divided period of continuous time; and a calculation unit that calculates the number of divisions of the total period based on the input total period so that the sum of the progress in the total period is maximized, and calculates the progress in the divided periods and the reward to be earned by the agent from the progress in the divided periods based on the number of divisions, the total reward, and the total period so that the sum of the progress in each period divided according to the number of divisions is maximized.

[0009] An information processing method according to one aspect of the present invention is a method performed by an information processing device, and includes: an input unit of the information processing device accepting input of a total reward, which is the total budget of rewards to be given to the agent, and a total period of the continuous time, under the condition that the agent obtains reward by increasing the agent's progress on a task in each divided period of the continuous time; and a calculation unit of the information processing device calculating, based on the input total period, the number of divisions of the total period so that the sum of the progress in the total period is maximized; and calculating, based on the number of divisions, the total reward, and the total period, the progress in the divided periods and the reward obtained by the agent from the progress in the divided periods so that the sum of the progress in each period divided according to the number of divisions is maximized.

[0010] According to the present invention, it is possible to optimize the schedule for giving rewards to agents.

[0011] FIG. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a function increase / decrease table. FIG. 3 is a flowchart showing an example of the processing operation of an information processing device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of information stored in an input information storage unit. FIG. 5 is a diagram showing an example of the number of divisions stored in an optimal reward schedule storage unit. FIG. 6 is a diagram showing an example of an optimal reward schedule stored in an optimal reward schedule storage unit. FIG. 7 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention.

[0012] An embodiment of the present invention will now be described. Fig. 1 is a diagram showing an application example of an information processing device according to one embodiment of the present invention. As shown in Fig. 1, information processing device 100, which is a reward schedule optimization device according to one embodiment of the present invention, comprises an operation unit 1, an input unit 2, an input information storage unit 3, an optimal reward schedule calculation unit 4, an optimal reward schedule storage unit 5, and an output unit 6. The function of each unit will be described in detail below.

[0013] In this embodiment, the model disclosed in the above-mentioned non-patent document 2 is extended to continuous time, making it possible to examine human behavior under present bias over continuous time and to obtain an optimal reward schedule, which is an optimal reward schedule.

[0014] (A) Problem Setting In this embodiment, a task is assumed in which a real value called "progress" is increased over a certain period of time until the goal is reached. This progress value is assumed to never decrease. For example, the following tasks (A-1) and (A-2) are applicable. (A-1) A task to exercise 30 hours in one month in order to improve health. (A-2) A task to complete a graduation thesis within six months.

[0015] In the task shown in (A-1) above, the duration is one month, and the value of the task progress is the cumulative exercise time up to the point in time when the progress is being calculated. In the task shown in (A-2) above, the duration is six months, and the value of the task progress is the percentage of completion of the graduation thesis at the point in time when the progress is being calculated.

[0016] Here, (1) below is the period from the start to the completion of the task (sometimes referred to as the final time), (2) below is the goal of the task, i.e., the progress at the time of completion, and (3) below is the reward given to the agent based on the progress.

[0017]

[0018] Furthermore, the following (4) is the time index, the following (5) is the progress from the start of the task to the current time, and the agent's state is expressed as shown in the following (6).

[0019]

[0020] (t, x) ... (6)

[0021] The state of the agent at the start of the task is the initial state (0, 0). t ) to state (t+1, x t+1 ) requires a cost. This cost is expressed as shown in (8) below using a cost function shown in (7) below.

[0022]

[0023] Next, we will explain the trajectory that an agent follows as time passes. (B) Mathematical Model Here, let us assume that an agent at a current location shown in (10) below at the current time shown in (9) below aims to reach a destination shown in (11) below at the final time T. 0<t c <T … (9) x(t c )=x c ...(10) x(T)=θ ...(11)

[0024] The agent plans a trajectory that minimizes the cost, as shown in (13) below, according to the equation shown in (12) below.

[0025]

[0026] Here, γ(·) in (12) above represents a discount function. This discount function indicates how much the agent at time s discounts the reward or cost at the time shown in (14) below. c (>s) … (14)

[0027] Furthermore, C(·) in the above (12) represents cost. The cost of producing progress dy in an infinitesimal time dt is expressed as in the following (15): C(dy / dt) ... (15)

[0028] The agent calculates a trajectory that minimizes the cost, as shown in (13) above, and then moves along this trajectory. That is, within an infinitesimal time dt, it moves by the value shown in (16) below.

[0029]

[0030] As a result, the trajectory that the agent actually takes is determined by the differential equation expressed as follows (17).

[0031]

[0032] Hereinafter, the cost function is assumed to be the function expressed as in (18) below, but other forms of cost functions can also be used: C(y'(s)) = (y'(s)) α (α>1) ... (18) Furthermore, as the discount function, a hyperbolic discount function expressed in the following (19) is used, which is a relatively commonly used discount function to express the effect of present bias. In this (19), k is a present bias parameter indicating the degree of present bias related to reward.

[0033]

[0034] Furthermore, when the agent believes that the costs it will have to pay in the future are not worth the reward it will ultimately receive through progress, it will give up on achieving the target value θ and abandon the task. The condition for mid-task abandonment is that the condition shown in (21) below is satisfied in the state shown in (20) below. (t c , x c ) … (20)

[0035]

[0036] To solve the above equation (12), we can use the calculus of variations. In particular, this can be reduced to the Euler-Lagrange equation. The idea behind the calculus of variations is that to find y(s) such that the functional expressed in the following (22) takes an extreme value, we need to solve the differential equation expressed in the following (23).

[0037]

[0038] In this case, since L depends only on y', it is sufficient if the value of the partial derivative shown in the following (24) is a constant that does not depend on s.

[0039]

[0040] That is, it is sufficient to be able to solve the following equations (25) and (26): (y'(s)) α-1 γ(s, t c ) = const. … (25)

[0041]

[0042] In this case, the following equation (27) can be obtained from the above equation (25).

[0043]

[0044] Here, when the boundary condition expressed as in (28) below is taken into consideration, the following equation (29) is obtained: y(t c )=x c , y(T)=θ …(28)

[0045]

[0046] Furthermore, from the above (17), the following equation (30) is obtained.

[0047]

[0048] After transformation, the following equation (31) is obtained.

[0049]

[0050] The condition for a task to be abandoned midway is that the condition expressed in the following (32) is satisfied, and from the above (21), this can be expressed as the following (33).

[0051]

[0052] By substituting the value shown in the following (34) into the above (33), the above-mentioned condition for mid-way abandonment is expressed as the following (35). c = τ, x c = x(τ) ... (34)

[0053]

[0054] If there exists a value expressed by the following (36) that satisfies this condition, the task will be abandoned midway. 0≦τ<T ... (36) The action taken by the agent can be determined by numerically calculating the integral part of the above (35). For example, when α=2, the integral part of the above (35) is expressed as the following (37).

[0055]

[0056] From this (37), the following equation (38) is obtained.

[0057]

[0058] Therefore, the condition under which a task can be abandoned midway is expressed as follows (39).

[0059]

[0060] Next, the increase / decrease table of f(τ) will be explained. Fig. 2 is a diagram showing an example of a function increase / decrease table. Here, the increase / decrease table shown in Fig. 2 can be obtained by setting the value expressed by the following (40) and calculating the value expressed by the following (41).

[0061]

[0062] Therefore, f(x) takes the maximum value as shown in the following (42).

[0063]

[0064] Therefore, the left side of the above (39) is a value expressed as in the following (43), and takes the maximum value expressed as in the following (44).

[0065]

[0066] From the above, when the following formula (45) holds, the agent will not give up on the above task midway.

[0067]

[0068] On the other hand, when the following equation (46) holds, the agent will give up on the task midway.

[0069]

[0070] The time when this happens is τ when the following equation (47) holds:

[0071]

[0072] Here, we explain the problem of maximizing x(T) under the condition that the reward R is given. In this problem, the agent needs to obtain the reward, so the condition shown in (48) below must be satisfied.

[0073]

[0074] Since the following equation (49) holds, the maximum value of x(T) is expressed as the following equation (50): x(T) = θ (49)

[0075]

[0076] (C) Reward Schedule Optimization Problem Next, we will explain the reward schedule problem for maximizing an agent's task progress when rewards can be presented to the agent multiple times. Given the total period, which is the continuous time from the start to the completion of the task, shown in (51) below, and the total reward (sometimes called the total budget), which is the sum of rewards given to the agent from the start to the completion of the task, shown in (52) below, these can be divided into K parts according to (53) below.

[0077]

[0078] Here, the period into which the total period is divided as described above is expressed as (54) below, and the reward into which the total reward is divided as described above according to the period divided as described above is expressed as (55) below.

[0079]

[0080] In the i-th period, the agent accumulates task progress under the condition expressed as follows (56).

[0081]

[0082] The objective of this embodiment is to maximize the total sum of progress within the total period. To achieve this, in this embodiment, the optimal reward schedule expressed as in (57) below is obtained.

[0083]

[0084] Note that the rewards for each period are not allocated simultaneously, and once the reward for one period is obtained, the reward for the next period is allocated. From the above (50), the optimization problem can be expressed as the following (58).

[0085]

[0086] Regarding the objective function, the following equation (59) is derived from the Cauchy-Schwarz inequality.

[0087]

[0088] Therefore, the optimization problem expressed by (58) above is equivalent to the optimization problem expressed by (60) below.

[0089]

[0090] We will now explain how to solve this problem. Here, we will explain the problem when the above-mentioned division number K is fixed. When the equation shown in the following (61) is set, the equation shown in the following (62) holds.

[0091]

[0092] Therefore, f(x) is a convex function when the following (63) is satisfied, and is a concave function when the following (64) is satisfied.

[0093]

[0094] Under the condition expressed as in (65) below, the objective function is expressed as in (66) below.

[0095]

[0096] Here, we will explain K that minimizes this objective function. Under the condition expressed as in (67) below, the following equation (68) holds.

[0097]

[0098] Therefore, when the following (69) holds, the above g(x) takes the maximum value. At this time, the interval is approximately 2 / k.

[0099]

[0100] Moreover, a more precise calculation is also possible assuming that the above-mentioned K is an integer. The equation shown in the following (70) holds, and the equation shown in the following (71) holds when the condition shown in the following (72) is satisfied, so the condition for g(x) to take the maximum value is when the equation shown in the following (73) is satisfied.

[0101]

[0102] Once K is determined in this way, the optimal schedule shown in the following (74) can be easily determined. Specifically, it can be calculated using the following equation (75).

[0103]

[0104] (D) Operation of Each Unit Based on the explanation so far, specific processing by each unit of information processing device 100 according to this embodiment will be described. Figure 3 is a flowchart showing an example of processing operations of the information processing device according to one embodiment of the present invention. Operation unit 1 is an interface that accepts operations from outside, and in response to these operations, as process invocations, enables data to be stored or modified in input information storage unit 3 by issuing an instruction to input unit 2, enables the start of optimization of the reward schedule by issuing an instruction to optimal reward schedule calculation unit 4, and enables the output of the results of the optimized reward schedule by issuing an instruction to output unit 6.

[0105] The input unit 2 stores or modifies data in the input information storage unit 3 in accordance with commands from the operation unit 1 (S1). Fig. 4 is a diagram showing an example of information stored in the input information storage unit. The input information storage unit 3 stores the input information, which is data from the input unit 2, specifically, the total budget R of rewards, the total period T, and the current bias parameter k, in an internal storage device as shown in Fig. 4 (S2).

[0106] The optimal remuneration schedule calculation unit 4 reads the information stored in the input information storage unit 3 in accordance with instructions from the operation unit 1, calculates the optimal remuneration schedule based on the contents shown in (C) above (S3), and stores this calculated optimal remuneration schedule in a memory device within the optimal remuneration schedule storage unit 5.

[0107] Fig. 5 is a diagram showing an example of the number of divisions of an optimal remuneration schedule stored in the optimal remuneration schedule storage unit. Fig. 6 is a diagram showing an example of an optimal remuneration schedule stored in the optimal remuneration schedule storage unit. The optimal remuneration schedule storage unit 5 stores the number of divisions of the optimal remuneration schedule calculated by the optimal remuneration schedule calculation unit 4 in an internal storage device as shown in Fig. 5, and stores the i-th period T into which the total period T is divided in the optimal remuneration schedule calculated by the optimal remuneration schedule calculation unit 4. ｉ , the progress θ of the task in the i-th period into which the total period T is divided ｉ , and the reward R obtained by the agent due to progress in the i-th period into which the total period T is divided. ｉ 6 (S4). The output unit 6 reads out the optimal reward schedule stored in the optimal reward schedule storage unit 5 and outputs it to an external device, for example, a display device (S5).

[0108] In the present embodiment described above, by optimizing the schedule for rewarding an agent under the influence of present bias in continuous time rather than discrete time, it is possible to maximize the performance of a target human in a task.

[0109] 7 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. In the example shown in FIG. 7, the information processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0110] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0111] An input device 200 and an output device 300 attached to the information processing device 100 and used by a user or the like are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 200 such as a keyboard, touch panel, touchpad, or mouse, and can output output data to an output device 300 including a display device using a liquid crystal or organic electroluminescence (EL) display, etc., for display. The input device 200 and the output device 300 may be devices built into the information processing device 100, or may be input devices and output devices of other information terminals that can communicate with the information processing device 100 via a network.

[0112] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0113] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processing steps.

[0114] An information processing apparatus 100 according to an embodiment of the present invention can be configured as a data processing apparatus having the units shown in FIG. 1 as software-based processing function units.

[0115] The information storage unit used as a work memory or the like by each unit of the information processing device 100 can be configured by using the data memory 112 shown in Fig. 7. However, these configured storage areas are not essential components within the information processing device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0116] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0117] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0118] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0119] 100... Information processing device 1... Operation unit 2... Input unit 3... Input information storage unit 4... Optimal remuneration schedule calculation unit 5... Optimal remuneration schedule storage unit 6... Output unit

Claims

1. An information processing device comprising: an input unit that accepts input of a total reward, which is a total budget of rewards to be given to an agent, and a total period of continuous time, under the condition that the agent earns reward by increasing the agent's progress on a task in each divided period of continuous time; and a calculation unit that calculates the number of divisions of the total period based on the input total period so that the sum of the progress in the total period is maximized, and calculates the progress in the divided periods and the reward to be earned by the agent from the progress in the divided periods based on the number of divisions, the total reward, and the total period so that the sum of the progress in each period divided according to the number of divisions is maximized.

2. The information processing device of claim 1, wherein the input unit accepts input of a parameter indicating the degree of present bias related to the reward, and the calculation unit calculates the number of divisions based on the input total period and the parameter, and calculates the progress in the divided period based on the number of divisions, the total budget, the total period, and the parameter.

3. A method performed by an information processing device, comprising: an input unit of the information processing device accepting input of a total reward, which is the total budget of rewards to be given to the agent, and a total period of the continuous time, under the condition that the agent earns reward by increasing the agent's progress on a task in each divided period of the continuous time; and a calculation unit of the information processing device calculating, based on the input total period, the number of divisions of the total period so that the sum of the progress in the total period is maximized, and calculating, based on the number of divisions, the total reward, and the total period, the progress in the divided periods and the reward to be earned by the agent from the progress in the divided periods so that the sum of the progress in each period divided according to the number of divisions is maximized.

4. An information processing program that causes a processor to function as each part of the information processing device according to claim 1 or 2.

Images