Cost Estimation Device, Method, and Program for Behavior Model

Abstract

An aspect of the present invention pertains to a behavior model using a graph. When the cost pertaining to a behavior is estimated for each of a plurality of sides, which indicate behaviors, between a plurality of vertexes which indicate states, the behavior model acquires graph data including at least information about rewards set for the structure of the graph and the plurality of vertexes of the graph, and acquires behavior data including a plurality of behavior trajectories from the graph. In addition, the cost is shown using parameters, the parameters are estimated using a gradient method pertaining to the likelihood function on the basis of the graph data and the behavior data, and the estimated parameters are output as the estimation value of the cost.

Description

Technical Field

[0001] One aspect of the present invention relates to a cost estimation device, method, and program for estimating the cost of an action model for estimating, for example, human behavior.

Background Art

[0002] For example, it is important to model the behavior of humans who attempt to achieve a certain goal, such as weight loss through dieting or completion of a course of lessons in online classes. This is because it becomes possible to determine appropriate intervention measures for assisting the person in achieving their goal by predicting the person's future behavior and changes in behavior during intervention based on such modeling.

[0003] By the way, as one means of such modeling, for example, a model based on graph theory has been proposed in Non-Patent Document 1. In this model, the states that a human can take are represented as vertices, and the actions that a human can take in each state are represented as edges. Further, a cost is set for each edge, and this cost represents the effort, that is, the load, when taking an action. Furthermore, a reward is set for each vertex, and this represents the reward for reaching the corresponding state.

[0004] An agent evaluates its own gain for candidates for the trajectory (path on the graph) of actions that it can take in the future on this graph, and selects to take the action with the largest gain. The gain of the action trajectory is calculated by weighting so as to reduce future costs and increase recent costs by a discounting method called quasi-hyperbolic discounting.

[0005] This model has attracted attention as being able to appropriately explain human behavior including irrationality, and has also been extended to models including other biases.

Prior Art Documents

Non-Patent Documents

[0006] [Non-Patent Document 1] Jon Kleinberg and Sigal Oren, “Time-inconsistent planning: a computational problem in behavioral economics.” In Proceedings of the 15th ACM Conference on Economics and Computation, pages 547-564, 2014. [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] However, in the model described in Non-Patent Document 1, the cost of each edge of the graph is treated as given. However, in reality, it is not easy to know in advance the cost of taking an action. Therefore, the information necessary for analysis using the model is insufficient, and it is difficult to actually use the model.

[0008] This invention has been made paying attention to the above circumstances, and aims to provide a technique that enables estimation of the cost related to actions that a person can take in a behavior model using a graph. [Means for Solving the Problems]

[0009] To solve the above problems, one aspect of the cost estimation device or cost estimation method of the behavior model according to this invention, in a behavior model using a graph, when estimating the cost related to each of a plurality of edges indicating actions between a plurality of vertices indicating states, obtains graph data including at least the structure of the graph and information representing rewards set for the plurality of vertices of the graph, and obtains behavior data including a plurality of behavior trajectories in the graph. Then, the cost is expressed using a parameter and the feature vectors assigned to each side and the graph data and the behavior data By representing the number of times an edge is selected from any vertex to any vertex based on the above, the generation probability of the set of action trajectories is obtained, and the parameter that minimizes the negative log-likelihood in the generation probability of the set of action trajectories is calculated using the gradient method to obtain the parameterIt is configured to estimate and output the estimated parameter as an estimated value of the cost.

[0010] According to one aspect of the present invention, by expressing the cost related to behavior with parameters and applying the gradient method related to the likelihood function to the estimation of these parameters, for example, from behavior data representing observed human behavior, it becomes possible to estimate the parameters corresponding to the costs of each edge of the behavior model. As a result, it becomes possible to analyze actual human behavior using the behavior model.

Advantages of the Invention

[0011] That is, according to one aspect of the present invention, it is possible to provide a technique that enables the estimation of the cost of an edge that a human can take in a behavior model using a graph.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0014] [One Embodiment] (Configuration Example) FIGS. 1 and 2 are block diagrams showing an example of the hardware configuration and software configuration of a cost estimation device according to an embodiment of the present invention.

[0015] The cost estimation device ML is constituted by, for example, a server computer or a personal computer. The cost estimation device ML includes a control unit 1 using a hardware processor such as a Central Processing Unit (CPU), and a storage unit having a program storage unit 2 and a data storage unit 3 is connected to the control unit 1 via a bus 5, and an input / output interface (hereinafter, the interface is referred to as I / F) unit 4. Note that the cost estimation device ML may include a communication I / F unit that transmits and receives information data among networks or the like.

[0016] An external device EX used by an administrator or the like is connected to the input / output I / F unit 4 via a signal cable or a network. The input / output I / F unit 4 receives graph data and action data necessary for creating an action model from the external device EX, and outputs a parameter estimated as the cost of an edge on the graph by the control unit 1 to the external device EX.

[0017] The program storage unit 2 is constituted by, for example, a combination of a non-volatile memory such as a Hard Disk Drive (HDD) or a Solid State Drive (SSD) that can be written and read at any time as a storage medium, and a non-volatile memory such as a Read Only Memory (ROM). In addition to middleware such as an Operating System (OS), it stores various programs necessary for executing various control processes according to an embodiment of the present invention.

[0018] The data storage unit 3 is configured by combining, for example, as a storage medium, a non-volatile memory such as an HDD or an SSD that can be written and read at any time, and a volatile memory such as a RAM (Random Access Memory). As a storage area necessary for implementing one embodiment of the present invention, it includes a graph data storage unit 31, an action data storage unit 32, and a parameter storage unit 33.

[0019] The graph data storage unit 31 is used to store the graph data input in the external device EX.

[0020] The action data storage unit 32 is used to store the action data input in the external device EX.

[0021] The parameter storage unit 33 is used to store the parameters estimated by the control unit 1. The parameters indicate the cost of each edge of the graph that constitutes the action model.

[0022] The control unit 1 includes a data acquisition processing unit 11, a parameter estimation processing unit 12, and a parameter output processing unit 13 as processing functions according to one embodiment of the present invention.

[0023] Each of these processing units 11 to 13 is realized by causing the hardware processor of the control unit 1 to execute the application program stored in the program storage unit 2. Note that the above application program does not necessarily have to be stored in the program storage unit 2 in advance, and may be downloaded from the external device EX or other server devices when necessary and stored in the program storage unit 2.

[0024] The data acquisition processing unit 11 captures the graph data and action data input in the external device EX via the input / output I / F unit 4, and performs a process of storing the captured graph data in the graph data storage unit 31 and the action data in the action data storage unit 32 respectively.

[0025] The graph data includes, for example, information representing the structure of the graph, starting and ending points on the graph, and values indicating rewards set for each vertex on the graph. The action data includes action trajectories of a plurality of humans arbitrarily selected as learning targets. The action trajectory represents the action path of a human on the graph.

[0026] The parameter estimation processing unit 12 reads the graph data of the extended probabilistic model from the graph data storage unit 31 and reads the action model from the action data storage unit 32. Then, based on the read graph data and action data, it estimates the parameters of each edge on the graph data and stores the estimated parameters in the parameter storage unit 33.

[0027] The parameter output processing unit 13 reads the estimated parameters from the parameter storage unit 33 and outputs the read parameters from the input / output I / F unit 4 to the external device EX.

[0028] (Operation example) Next, an operation example of the cost estimation device SV configured as described above will be described.

[0029] (1) Action model using a graph First, prior to the description of the operation of the cost estimation device SV according to an embodiment, an overview of the Kleinberg and Oren model, which is the basis of the action model used in an embodiment of this invention, will be described.

[0030] Now, assume a directed acyclic graph G = (V, E), and define the starting and ending points of the graph as s and t respectively, and the cost of each edge as c: E → R. However, assume that there is at least one path from the starting point s to the ending point t. In this case, a naive agent A with a bias parameter β β moves from the starting point s to the ending point t while taking the following actions.

[0031] That is, let the set of paths from the current vertex v to the end vertex t be S(v), and if the i-th edge of the path P ∈ S(v) is represented as e i (P), then agent A β chooses

Number

Number

[0032] Furthermore, when considering the reward, if the i-th vertex of P ∈ S(v) is v i (P) (where v 0 = v), then agent A β chooses

Number

Number

Number

[0033] Note that since the model of Kleinberg and Oren is described in detail in Non-Patent Document 1, a detailed explanation here is omitted.

[0034] (2) Extension to a Probabilistic Model In one embodiment, in order to make the model using the above graph easier to handle, the above model is extended to a probabilistic model in the following procedure in advance.

[0035] That is, now agent A β has three parameters α, β, γ and A α,β,γIt is expressed as follows. Let the currently existing vertex be u, and let the set of vertices with outgoing directed edges from u be N(u). Then, for each vertex u ∈ V, the discounted value D(u) is

Number

[0036] Since the graph is a directed acyclic graph (DAG), the above discounted value D(u) is uniquely determined inductively, and its value can be obtained by the dynamic programming method described in Algorithm1.

Number

[0037] Furthermore, for each vertex u ∈ V,

Number

[0038] Agent A α,β,γ moves to the next vertex v with probability T e (u, v) at each vertex u. The probability T e (u, v) is expressed as follows.

Number

[0039] Furthermore, a new vertex v 0 is added to the graph, and edges with cost 0 are added from all vertices other than t to v 0 . When agent A α,β,γ reaches vertex t, it means that the task has been accomplished. On the other hand, when v 0If it reaches this point, it means that the task has not been achieved.

[0040] This model is consistent with the original model when α→∞ and γ = 1. That is, this model is an extension of the original model that includes the original model as a subset.

[0041] (3) Operation of the cost estimation device SV Based on the above extended action model, the cost estimation device SV according to an embodiment executes cost estimation processing as follows. FIG. 3 is a flowchart showing an example of the processing procedure and processing content of the cost estimation processing executed by the control unit 1 of the cost estimation device SV.

[0042] (3-1) Acquisition of data The control unit 1 of the cost estimation device SV monitors the input of data necessary for the action model creation process in step S10. In this state, when data is input from the external device EX, the control unit 1 of the cost estimation device SV, under the control of the data acquisition processing unit 11, acquires the data input from the external device EX via the input / output I / F unit 4 in steps S11 and S12, and stores the acquired data in the data storage unit 3.

[0043] The data input from the external device EX includes graph data and action data. As described above, the graph data includes information representing the structure of the graph, the start point s and the end point t on the graph, and values representing the rewards set for each vertex v on the graph. The data acquisition processing unit 11 stores the acquired graph data in the graph data storage unit 31 in the data storage unit 3 in step S11. FIG. 4 shows an example of the structure of the stored graph, and FIG. 5 shows an example of the rewards set for the vertices v (v 1 , v 2 , …, v 6 ) of the graph. Note that the start point s and the end point t are respectively the vertices v 1 , v 6 of the graph.

[0044] On the one hand, the behavior data includes the behavior trajectories of multiple humans. The behavior trajectory represents the behavior path on the human graph. The data acquisition processing unit 11 stores the above-mentioned behavior trajectory in the behavior data storage unit 32 in the data storage unit 3 by step S12. FIG. 6 shows an example of the stored behavior trajectory.

[0045] (3-2) Estimation of parameters Next, the control unit 1 of the cost estimation device SV estimates the cost of each edge of the graph as follows under the control of the parameter estimation processing unit 12 in step S13.

[0046] That is, when the set X of the above-mentioned behavior trajectories is given, consider the problem of estimating the costs (c uv ) (u,v) ∈E of multiple edges. Here, let the set X consist of M behavior trajectories, and the m-th behavior trajectory is represented by (u m 1 ,…,u m Hm ). Also, the cost c uv of each edge of the graph is related to the parameter θ∈R d and the feature vector ξ uv ∈R d assigned to each edge (u,v), and is expressed as

Equation

[0047] Now, in the behavior trajectory set X, let the number of times the edge going from vertex u to vertex v is selected be f uv , then this number f uv is

Equation

Number

Number

[0048] Consider the problem of minimizing this.

Number

Number

Number

[0049] After that, for any (u, v) ∈ E, we just need to find ∂Q uv / ∂θ. That is, from the above equation (2),

Number

Number

Number

[0050] Also, when there are multiple v ∈ N(u) that achieve the maximum value, we take the one with the minimum index. From equation (11), ∂D(u) / ∂θ can also be calculated by dynamic programming. The calculation method is shown below as Algorithm2.

Number

[0051] Summarizing the above, the algorithm for estimating the parameter θ can be described as Algorithm 3 below.

Equation

[0052] The computational complexity per gradient calculation is bottlenecked by the calculation of ∂L(θ) / ∂θ. Therefore, it can be performed in O(d·(|V| + |E|)).

[0053] Thus, the parameter θ can be estimated. The parameter estimation processing unit 12 stores the parameter θ estimated as described above in the parameter storage unit 33. FIG. 7 shows an example of the parameter θ stored in the parameter storage unit 33.

[0054] (3-3) Output of Parameters When the estimation process of the above parameters is completed, the control unit 1 of the cost estimation device SV reads out the parameter θ from the parameter storage unit 33 under the control of the parameter output processing unit 13 and outputs the read parameter θ from the input / output I / F unit 4 to the external device EX at step S14.

[0055] The external device EX sets costs for each corresponding edge of the separately stored action model based on the parameter θ given from the cost estimation device SV. Therefore, the external device EX can subsequently estimate human behavior using the action model with these costs set.

[0056] (Function and Effect) As described above, in one embodiment, in the behavior model using a graph, when estimating the cost of each edge of the graph, graph data including the structure of the graph of the behavior model, information specifying the vertices serving as the start point and the end point, and the rewards set for each vertex is acquired, and a plurality of observed human behavior trajectories are acquired. Then, in the parameter estimation processing unit 12, the cost is represented using parameters, and based on the graph data and the behavior data, the parameters are estimated using the gradient method regarding the likelihood function, and the result is output by the parameter output processing unit 13.

[0057] Therefore, according to one embodiment, for example, it is possible to estimate parameters corresponding to the cost of a human from behavior data representing the observed human behavior. Thereby, actual human behavior analysis using the behavior model becomes possible.

[0058] [Other Embodiments] (1) In the above-described embodiment, the case where the process of expanding the behavior model into a probabilistic model is performed in advance by another device such as the external device EX has been described as an example. However, the processing function for expanding into the probabilistic model may be provided in the cost estimation device SV. In this case, the cost estimation device SV generates an extended probabilistic model including the original model as a subset based on, for example, the graph data acquired from the external device EX.

[0059] (2) In the above-described embodiment, the case where the cost estimation device SV is provided independently of the external device EX has been described as an example. However, it is not limited thereto, and each function provided in the cost estimation device SV may be provided in the external device EX. Thereby, for example, the external device can perform all the behavior model creation processes including the cost estimation process in a batch.

[0060] (3) In addition, regarding the configuration of the cost estimation device, the processing procedure and processing content of the parameter estimation process, etc., various modifications can be made and implemented without departing from the gist of the present invention.

[0061] The embodiments of the present invention have been described in detail above. However, the foregoing description is merely illustrative of the present invention in every aspect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. That is, in practicing the present invention, a specific configuration according to the embodiment may be appropriately adopted.

[0062] In short, the present invention is not limited to the above embodiments as they are. In the implementation stage, the components can be modified and embodied without departing from the gist thereof. Also, various inventions can be formed by appropriately combining a plurality of components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

Explanation of Reference Numerals

[0063] ML... Cost Estimation Device EX... External Device 1... Control Unit 2... Program Storage Unit 3... Data Storage Unit 4... Input / Output I / F Unit 5... Bus 11... Data Acquisition Processing Unit 12... Parameter Estimation Processing Unit 13... Parameter Output Processing Unit 31... Graph Data Storage Unit 32... Behavior Data Storage Unit 33... Parameter Storage Unit

Claims

1. A cost estimation device for estimating the cost related to an action for each of a plurality of edges indicating actions between a plurality of vertices indicating states in an action model using a graph, comprising: a first acquisition processing unit that acquires graph data including at least the structure of the graph and information representing rewards set for the plurality of vertices of the graph; a second acquisition processing unit that acquires action data including a plurality of action trajectories in the graph; a parameter estimation processing unit that represents the cost using a parameter and a feature vector assigned to each edge, obtains the occurrence probability of an action trajectory set by representing the number of times an edge that can be directed from an arbitrary vertex to an arbitrary vertex is selected based on the graph data and the action data, and estimates the parameter by calculating the parameter that minimizes the negative log-likelihood in the occurrence probability of the action trajectory set using a gradient method; an output processing unit that outputs the estimated parameter as an estimated value of the cost A cost estimation device for an action model comprising the above.

2. The cost estimation device for an action model further comprises a model expansion processing unit that expands the action model into a probabilistic action model including the action model as a subset, wherein the first acquisition processing unit and the second acquisition processing unit acquire the graph data and the action data corresponding to the probabilistic action model. The cost estimation device for an action model according to Claim 1.

3. A cost estimation method for estimating the cost related to an action for each of a plurality of edges indicating actions between a plurality of vertices indicating states in an action model using a graph, comprising: a process in which a first acquisition processing unit acquires graph data including at least the structure of the graph and information representing rewards set for the plurality of vertices of the graph; a process in which a second acquisition processing unit acquires action data including a plurality of action trajectories in the graph; a process in which a parameter estimation processing unit represents the cost using a parameter and a feature vector assigned to each edge, obtains the occurrence probability of an action trajectory set by representing the number of times an edge that can be directed from an arbitrary vertex to an arbitrary vertex is selected based on the graph data and the action data, and estimates the parameter by calculating the parameter that minimizes the negative log-likelihood in the occurrence probability of the action trajectory set using a gradient method; A process in which the output processing unit outputs the estimated parameter as an estimated value of the cost A method for estimating the cost of an action model including the above.

4. A program that causes a processor included in the cost estimation device to execute the processing by each processing unit included in the cost estimation device according to any one of Claims 1 to 2.

Images