Information processing device, information processing method, information processing system, and information processing program

Abstract

This information processing device (1) comprises: a selection unit (11) which selects an action at of each round t; an execution unit (12) which executes a process P(at) of each round t; and an acquisition unit (13) which acquires a risk ft(at) of each round t. In order that the expectation value of the action at to be selected is caused to approximate an action xt that minimizes an empirical risk Φt-1(x), the selection unit (11): (1) executes a process for setting the action at to an action zt that is arbitrarily selected from an action set A at a probability of rt-1; and (2) executes a process for setting the action at to an action yt that is randomly selected from a partial set Ut of a convex hull X of the action set A at a probability of rt. The execution unit (12) executes, as the process P(at), for example, a process for controlling a robot 2 to move from an A spot to a B spot via a selected moving path as the action at.

Description

Information processing device, information processing method, information processing system, information processing program, and recording medium

[0001] The present invention relates to an information processing device, an information processing method, an information processing system, and an information processing program for solving bandit problems, and also to a recording medium on which such an information processing program is recorded.

[0002] There is a demand for technology that allows computers to replace various decision-making tasks performed by humans. For example, the problem of players selecting an action in each round is an example of a problem that can be solved by a computer. Among such problems, the problem of selecting an action in each round without considering the loss determined by that action is called a "bandit problem." In particular, the problem of selecting an action a in each round t t is expressed as a d-dimensional vector (d is any natural number greater than or equal to 1), and the loss f t (a t ) is a linear function <l t , a t >+ξ t A bandit problem expressed as follows is called a "linear bandit problem."

[0003] A method for solving the bandit problem by computer is to determine the action a in each round t. t Let f1(a1), f2(a2), ..., f t-1 (a t-1 ) t-1 Action x that minimizes (x) t In Non-Patent Document 1, the action a in each round t is selected. t As the empirical loss Φ t-1 Action x that minimizes (x) t Instead of selecting t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]}, an action y chosen randomly tHere, a method for selecting {λ1, λ2, ..., λ d} and {e1, e2, ..., e d} are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull of the action set, respectively. 2 According to the method described in Non-Patent Document 1, in an adversarial environment, that is, the loss f t (a t ) are the actions a1, a2, ..., a t-1 In an environment where the relevance depends on the relevance of the system, it is guaranteed that regrets can be kept sufficiently small.

[0004] Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. "Competing in the dark: An efficient algorithm for bandit linear optimization". In Proceedings of the 21st Annual Conference on Learning Theory (COLT), pages 263-274, 2008.

[0005] However, the environment in which the bandit problem should be solved is not limited to an adversarial environment. For example, there is a stochastic environment, that is, a loss f t (a t ) are the actions a1, a2, ..., a t-1 There are cases where a bandit problem is solved in an environment determined according to a probability distribution that does not depend on . Alternatively, there are cases where a bandit problem is solved in an environment where it is unclear whether it is hostile or stochastic. In such cases, the method described in Non-Patent Document 1 does not guarantee that regret can be kept sufficiently small.

[0006] One aspect of the present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize an information processing device, information processing method, or information processing system that solves bandit problems and can keep regrets sufficiently small in both hostile environments and stochastic environments.

[0007] An information processing device according to one aspect of the present invention is t and a selection means for selecting a process P(a t ) as the action a t and a loss f t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring means for acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t and the selection means is a selecting action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t is a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ Run it with:

[0008] In an information processing method according to one aspect of the present invention, an action a t and a selection process for selecting P(a t ) as the action a t and the loss f t (a t) as the process P(a t ) losses resulting from said action a t and an acquisition process for acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t In the selection process, the action of selecting a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t is a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ It is executed in

[0009] An information processing system according to one aspect of the present invention includes an information processing device and another device, wherein the information processing device determines an action a in each round t. t and a selection means for selecting a process P(a t ) as a control for the other device, t and a control unit for controlling the loss f t (a t ) as a loss resulting from the control, t and acquiring means for acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t and the selection means is a selecting action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a tis an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t Let us consider a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ Run it with:

[0010] According to one aspect of the present invention, regrets can be kept sufficiently small in both hostile and stochastic environments.

[0011] 5 is a block diagram showing the configuration of an information processing device according to a first exemplary embodiment of the present invention. FIG. 6 is a flow chart showing the flow of an information processing method according to the first exemplary embodiment of the present invention. FIG. 7 is a flow chart showing a specific example of a selection process included in the information processing method shown in FIG. 2. FIG. 8 is a schematic diagram showing the configuration of an information processing system according to a second exemplary embodiment of the present invention. FIG. 9 is a schematic diagram showing the configuration of an information processing system according to a third exemplary embodiment of the present invention. FIG. 10 is a plan view showing a screen displayed on an administrator terminal in the information processing systems shown in FIGS. 4 and 5. FIG. 11 is a block diagram showing the configuration of a computer that functions as the information processing device shown in FIG.

[0012] [Linear bandit problem] Action a in each round t ∈ [T] t Consider a problem in which a player selects an action set A from the set of actions A. In such a problem, the action a t The action a t The loss f of that round t is determined depending on t (a t A problem in which players make a choice without referring to the action a in each round is called a "bandit problem." t is expressed as a d-dimensional vector (d is any natural number greater than or equal to 1), and the loss f t (a t ) is a linear function <l t , a t >+ξ t A bandit problem expressed as follows is called a "linear bandit problem." tis a d-dimensional vector and is called the "loss vector". t is a scalar.

[0013] R defined by the following formula (1) T The algorithm for solving the banded problem, i.e., the action a in each round t, t The superiority of the algorithm for selecting this Regret R T Regret R T The smaller the algorithm can be, the better the algorithm.

[0014] Loss f for each round t t (a t ) are the actions a1, a2, ..., a t-1 The loss f follows a probability distribution that does not depend on t (a t The environment in which the loss f t (a t ) are the actions a1, a2, ..., a t-1 The information processing device 1 according to the embodiment exemplified below can reduce regret R in both the stochastic environment and the hostile environment. T action a in each round t so that t The following is selected.

[0015] In the following description, X represents the convex hull of the action set A. ψ represents the barrier function of the convex hull X, ∇ψ(x) represents the gradient of the barrier function ψ in x∈X, and D(x, y) represents the Bergman divergence of the barrier function ψ in x, y∈X. ∇ 2 ψ(x) denotes the Hessian matrix of the barrier function ψ for x∈X, and ∇ -2 ψ(x) is the Hessian matrix ∇ 2 represents the inverse matrix of ψ(x).

[0016] The barrier function ψ of the convex hull X refers to a function whose value ψ(x) diverges to infinity as x∈X approaches the boundary of the convex hull X. Among the barrier functions ψ, a barrier function ψ that satisfies the following equation (2) for θ>0 is called a "θ-self-concordant barrier function." θ is called a self-concordance parameter. Here, Dψ(x) represents the first derivative of the barrier function at x∈X, and D 2 ψ(x) represents the second derivative of the barrier function at x∈X, and D 3 ψ(x) denotes the third derivative of the barrier function at xεX.

[0017] Also, the norm ||| x is ||h|| x = (h T ∇ 2 ψ(x)h) 1 / 2 is defined by the norm ||·|| x * is ||h|| x * = (h T ∇ -2 ψ(x)h) 1 / 2 Also, W r (x) is W r (x)={y∈A|||y−x|| x ≦r} and is called a Dikin ellipsoid. A (x, z) is π A (x, z)=inf{r>0|z+r -1 (x−z)∈A} and is called the Minkowski function. λ(x, ψ) is the Newton decrement of the barrier function ψ at x∈A, and λ(x, φ) = ||∇ψ(x)|| x * is defined as:

[0018] In addition, the set U(α, z, E) = {z + α(x - z) | x ∈ E} defined for an arbitrary element z of the convex hull X, an arbitrary subset E of the convex hull X, and an arbitrary scalar α ≥ 1 is called an "upscaled set obtained by expanding the set E by a factor of α for z." In the embodiment exemplified below, an action z arbitrarily selected from the action set A ist For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) is used, where x t is the empirical loss Φ in the convex hull X t-1 (x) is the action that minimizes {λ1, λ2, ..., λ d} and {e1, e2, ..., e d} are the Hessian matrices ∇ of the barrier function ψ, respectively. 2 are the eigenvalue and eigenvector sequences of ψ. t is the upscale set U t (1 / r t , z t , E t ) is set to be the largest upscale set contained in the convex hull.

[0019] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described with reference to the drawings. This exemplary embodiment is a base for the exemplary embodiments described below.

[0020] (Configuration of Information Processing Apparatus) The configuration of an information processing apparatus 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing apparatus 1.

[0021] As shown in FIG. 1 , the information processing device 1 includes a selection unit 11 , an execution unit 12 , and an acquisition unit 13 .

[0022] The selection unit 11 selects the action a t In each round t, the action a t is assumed to be represented by a d-dimensional numeric vector.

[0023] The execution unit 12 executes the process P(at ) where, the process P(a t ) is the action a selected by the selection unit 11 in that round t. t In addition, the process P(a t ) is the action a in that round t. t A case in which the robot 2 is controlled as another device will be described in an exemplary embodiment 2 to be described later, and a case in which the user terminal 5 is controlled as another device will be described in an exemplary embodiment 3 to be described later.

[0024] The acquisition unit 13 acquires the loss f t (a t ) for each round t. t (a t ) is the process P(a t ) is the loss resulting from the action a selected by the selection unit 11 in that round t. t In this embodiment, the loss f t (a t ) is a linear function <l t , a t >+ξ t or a linear function <l t , a t >+ξ t Assume that it is approximated by

[0025] The selection unit 11 selects an action a t The expected value of the losses f1(a1), f2(a2), ..., f t-1 (a t-1 ) t-1 Action x that minimizes (x) t (1) Action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ Execute at -1, (2) Action a t Let us consider a subset U of the convex hull X of the action set A. ｔA randomly selected action y t The process of setting ｔ Run it with:

[0026] In particular, in this embodiment, the above-mentioned subset U t As, action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) is used, where x t is the empirical loss Φ in the convex hull X t-1 (x) is the action that minimizes {λ1, λ2, ..., λ d} and {e1, e2, ..., e d} are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalue and eigenvector sequences of ψ. t is the upscale set U t (1 / r t , z t , E t ) is determined to be the maximum upscale set contained in the convex hull. In this case, the action a selected by the selection unit 11 is t The expected value of is the empirical loss Φ t-1 Action x that minimizes (x) t It can be mathematically proven that

[0027] (Flow of Information Processing Method) The flow of the information processing method S1 performed by the information processing device 1 will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of the information processing method S1.

[0028] 2, the information processing method S1 includes a selection process S11, an execution process S12, and an acquisition process S13. The selection process S11, the execution process S12, and the acquisition process S13 are executed in each round t∈[T].

[0029] The selection process S11 is a process of selecting an action a t As mentioned above, the action a t is represented by a d-dimensional numeric vector. In this exemplary embodiment, the selection process S11 is performed by the selection unit 11.

[0030] The execution process S12 is a process P(a t As described above, the process P(a t ) is the action a selected by the selection unit 11 in that round t. t In this exemplary embodiment, the execution process S12 is executed by the execution unit 12. Note that the process P(a t ) is the action a in that round t. t A case in which the robot 2 is controlled as another device will be described in an exemplary embodiment 2 to be described later, and a case in which the user terminal 5 is controlled as another device will be described in an exemplary embodiment 3 to be described later.

[0031] The acquisition process S13 calculates the loss f t (a t As mentioned above, the loss f(a t ) is the process P(a t ) is the loss resulting from the action a selected in the selection process S11 of that round t. t As mentioned above, the loss f t (a t ) is a linear function <l t , a t >+ξ t or a linear function <l t , at >+ξ t In this exemplary embodiment, the acquisition process S13 is performed by the acquisition unit 13.

[0032] In the selection process S11, the action a to be selected is t The expected value of the losses f1(a1), f2(a2), ..., f t-1 (a t-1 ) t-1 Action x that minimizes (x) t (1) Action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ -1, and (2) action a t Let us consider a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ It is executed in

[0033] In particular, in this embodiment, the above-mentioned subset U t As, action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) is used, where x t is the empirical loss Φ in the convex hull X t-1 (x) is the action that minimizes {λ1, λ2, ..., λ d} and {e1, e2, ..., e d} are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalue and eigenvector sequences of ψ. t is the upscale set Ut (1 / r t , z t , E t ) is determined to be the maximum upscale set contained in the convex hull. In this case, the action a selected by the selection unit 11 is t The expected value of is the empirical loss Φ t-1 Action x that minimizes (x) t It can be mathematically proven that the above-mentioned selection process S11 is consistent with the above-mentioned. A specific example of this selection process S11 will be described later with reference to different drawings.

[0034] In addition, in the selection process S11, the action a t In preparation for selecting t (a t ) in the current round t. t (a t ) is the action a of the current round t in the selection process S11. t In the execution process S12, the process P(a t ) and in the acquisition process S13, the loss f t (a t ) is obtained.

[0035] (Specific Example of Selection Process) A specific example of the selection process S11 will be described with reference to Fig. 3. Fig. 3 is a flow diagram showing the flow of the selection process S11 according to this specific example. Note that in the selection process S11 according to this specific example, the empirical loss Φ t-1 As a parameter for defining (x), s (In the formula, ^ is written above l)∈R d , m t ∈R d , β t >0 is used. Here, ^l s is the loss vector l s (s=1, 2, ..., t-1). Also, m t is the loss vector l t represents the optimistic prediction of t represents the learning rate.

[0036] As shown in FIG. 3, the selection process S11 includes a predicted value setting step S101, a learning rate setting step S102, a reference behavior selection step S103, a set creation step S104, a candidate behavior selection step S105, a probability setting step S106, a flag setting step S107, a behavior selection step S108, an unbiased estimator setting step S109, and a predicted value update step S110.

[0037] The predicted value setting step S101 is a step executed only in the first round (t=1), and the other steps S102 to S110 are steps executed in each round (t=1, 2, ..., T). Of the other steps S102 to S110, steps S102 to S108 are steps executed by the execution unit 12 to execute the process P(a t ), and steps S109 to S110 are steps performed by the selection unit 11 before the acquisition unit 13 executes the loss f t (a t ) is obtained.

[0038] The predicted value setting step S101 is a step of setting a predicted value m1. In the predicted value setting step S101, the selection unit 11 sets the predicted value m1 to m1=0.

[0039] The learning rate setting step S102 is a learning rate β t In the learning rate setting step S102, the selection unit 11 sets the learning rate β t When t=1, β t = 16d 2 If t>1, the flag b is set according to the following formulas (a1) and (a2). s The definition of will be explained later.

[0040] The reference action selection step S103 is a step of selecting a reference action x t In the reference action selection step S103, the selection unit 11 selects an element x of the convex hull X, which is an experiential loss Φ t-1The element x that minimizes (x) is called the reference action x. t Select as.

[0041] In the set creation step S104, a set E t In the set creation step S104, the selection unit 11 creates a set of reference actions x t and the Hessian matrix ∇ of the θ-self-concordant barrier function ψ 2 The eigenvalues ​​of ψ {λ1, λ2, ..., λ d} and the eigenvectors {e1, e2, ..., e d} and E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} t Create a.

[0042] The candidate action selection step S105 selects candidate actions z t In the candidate action selection step S105, the selection unit 11 selects an arbitrary element a in the action set A as a candidate action z t Select as.

[0043] The probability setting step S106 sets the probability r t In the probability setting step S106, the selection unit 11 sets the probability r t is set according to the following formula (a4). t is the set E t Candidate action z t The upscaled set U(1 / r,z t , E t ) coincides with the smallest positive number r contained in the convex hull X.

[0044] In the flag setting step S107, flag b t In the flag setting step S107, the selection unit 11 sets the flag b t bt = 1 or b t = 0. Here, flag b t A b t The probability of setting t and flag b t A b t The probability of setting = 0 is 1-r t is.

[0045] The action selection step S108 is to select the action a t In the action selection step S108, the selection unit 11 selects the flag b t Ga b t If z = 0, then the candidate action z t Let action a in round t be t The selection unit 11 also selects the flag b t Ga b t = 1, then i randomly selected from [d] t and ε chosen randomly from {-1, +1} t and action a in round t t a t =z t +(1 / r t ) (x t +ε t λ it -1 / 2 e it -z t ) is set according to the t = z t +(1 / r t ) (x t +ε t λ it -1 / 2 e it -z t ) according to the action a in round t t Setting U(1 / r t , z t , E t ) element y t Let action a in round t be t is equivalent to selecting

[0046] The unbiased estimator setting step S109 is performed by setting the loss vector lt An unbiased estimator of ^l s In the unbiased estimator setting step S109, the selection unit 11 sets the unbiased estimator ^l t is set according to the following formula (a5): where the loss f t (a t ) is the action a t The execution unit 12 selects the process P(a t ) is executed, the acquisition unit 13 acquires it.

[0047] The predicted value update step S110 is performed by updating the loss vector l t The predicted value m t In the predicted value update step S110, the selector 11 updates the predicted value t is set according to the following equation (a6).

[0048] The above-mentioned selection process S11 can be embodied as the algorithm shown in Table 1 below.

[0049] The inventors perform the action a in each round t according to the algorithm shown in Table 1 above. t By selecting T Specifically, we have succeeded in proving that (1) in any environment, regret R T can be evaluated as the following formula (b1), and (2) in any stochastic environment, regret R T can be evaluated as the following equation (b2), and (3)ξ t In a hostile environment where = 0, the regret R T can be evaluated as the following equation (b3).

[0050] where σ 2 is defined by the following formula (b4), and Δ min is a * ∈argmin a∈AIt is defined by the following formula (b5) using <l*, a>.

[0051] Furthermore, the inventors further propose that (4) for any action a∈A and any round t∈[T], t (a) If ≧0, then regret R T can be evaluated as the following equation (b6).

[0052] The above evaluation is based on the loss f t (a t ) is a linear function <l t , a>+ξ t However, more broadly, the loss f of round t t (a t ) is a linear function <l t , a>+ξ t It is clear that the same evaluation holds when the loss f t (a t ) is a linear function <l t , a>+ξ t The aspect expressed by is the loss f t (a t ) is a linear function <l t , a>+ξ t That is, the loss f of round t is t (a t ) is a linear function <l t , a>+ξ t The loss f of round t is approximated by t (a t ) is a linear function <l t , a>+ξ t This does not imply that the expression is not expressed by

[0053] In addition, the above evaluation is based on the action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t More broadly, the action a t The expected value of the empirical loss Φt-1 Action x that minimizes (x) t It is clear that the same evaluation holds when the action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t The behavior that is consistent with the above is the behavior a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t That is, the action a in each round t is t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t The action a in each round t is approximated by t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t does not imply that it does not match.

[0054] Second Exemplary Embodiment A second exemplary embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing the configuration of an information processing system 100A according to the second exemplary embodiment.

[0055] As shown in FIG. 4, the information processing system 100A includes, in addition to the information processing device 1, a robot 2 configured to be able to communicate with the information processing device 1.

[0056] In this exemplary embodiment, the selection unit 11 of the information processing device 1 selects the action a t A path for robot 2 to travel from point A to point B is selected as follows. The path for robot 2 to travel from point A to point B is expressed by a d-dimensional vector, where the i-th component is 1 when robot 2 travels along path Ri, and is 0 when robot 2 does not travel along path Ri. Here, d is the number of paths that can constitute the path from point A to point B (d=8 in the example shown).

[0057] In this exemplary embodiment, the execution unit 12 of the information processing device 1 executes the process P(a t ) and the action a in that round t tThe execution unit 12 of the information processing device 1 executes a process to control the robot 2 so that it moves from point A to point B along the selected movement route. However, paths R1 to Rd are also used by other robots 3 and 4. Therefore, the execution unit 12 of the information processing device 1 controls the robot 2 so that it does not collide with the other robots 3 and 4.

[0058] In this embodiment, the acquisition unit 13 of the information processing device 1 acquires the loss f t (a t ) and the action a in that round t t The time required for the robot 2 to move from point A to point B along the selected movement route is acquired from the robot 2.

[0059] Loss f for each round t t (a t ) (i.e., the movement time of robot 2) depends on the actions of other robots 3 and 4. Therefore, the action a t (i.e., the path of the robot 2) cannot be known before the action a t The problem of selecting a path (i.e., the movement path of the robot 2) is a bandit problem.

[0060] In particular, if the actions of the other robots 3 and 4 are determined according to a certain probability distribution, the action a t The problem of selecting the path of robot 2 (i.e., the path of robot 2) is a bandit problem in a stochastic environment. On the other hand, if the actions of other robots 3 and 4 are the same as the actions a1, a2, ..., a1 up to the previous round t-1, t-1 (i.e., the path of the robot 2 up to now), the action a t The problem of selecting a path (i.e., the future movement path of the robot 2) becomes a bandit problem in a hostile environment.

[0061] In response to this, the selection unit 11 of the information processing device 1 selects the action a in each round t according to the algorithm shown in Table 1 above. t (i.e., the path of movement of the robot 2). Therefore, even if the environment is stochastic or hostile, regret R TThis provides a practical effect that the robot 2 can be controlled so as to keep the value of the rotational speed of the robot 2 sufficiently small.

[0062] The information processing device 1 uses the loss f acquired in the acquisition process S13 t (a t ) may be displayed on the display of the administrator terminal as a "cumulative loss" graph. An example of a screen I1 including a "cumulative loss" graph displayed on the display of the administrator terminal is shown in FIG. 6. This provides a practical effect in that the administrator can easily grasp the change in cumulative loss for each round.

[0063] In addition, the information processing device 1 uses the unbiased estimator ^l calculated in the selection process S11. t or predicted value m t The graph of the above may be displayed on the display of the administrator terminal as a graph of "estimated loss." An example of a screen I2 including the graph of "estimated loss" displayed on the display of the administrator terminal is shown in Figure 6. This has the practical effect of allowing the administrator to easily grasp the changes in estimated loss for each round.

[0064] Third Exemplary Embodiment A third exemplary embodiment of the present invention will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing the configuration of an information processing system 100B according to the second exemplary embodiment.

[0065] As shown in FIG. 5, the information processing system 100B includes, in addition to the information processing device 1, a user terminal 5 configured to be able to communicate with the information processing device 1.

[0066] In this exemplary embodiment, the selection unit 11 of the information processing device 1 selects the action a t A d-dimensional feature vector is selected as

[0067] In this exemplary embodiment, the execution unit 12 of the information processing device 1 executes the process P(a t ) and the action a in that round t t The advertisement generating unit 100 generates an advertisement according to the selected feature vector, and controls the user terminal 100 to display the generated advertisement on a display.

[0068] In this embodiment, the acquisition unit 13 of the information processing device 1 acquires the loss f t (a t ) and the action a in that round t t When the user clicks on an advertisement corresponding to the selected feature vector, a loss value of "0" is obtained from the user terminal 5, and the action a t If the user does not click on the advertisement corresponding to the feature vector selected as the user, a loss value of "1" is acquired from the user terminal 5.

[0069] Loss f for each round t t (a t ) depends on the user's actions, so the action a t (i.e., the feature vector of the advertisement to be displayed) cannot be known before selecting the action a t The problem of selecting the feature vector of the advertisement to be displayed is a bandit problem.

[0070] In particular, if a user's click behavior is determined according to a certain probability distribution, the action a t The problem of selecting the feature vector of the advertisement to be displayed is a bandit problem in a stochastic environment. On the other hand, if the user's click behavior is the same as the actions a1, a2, ..., a t-1 (i.e., the feature vector of the advertisements displayed so far), the behavior a t The problem of selecting the feature vector of the advertisement to be displayed becomes a bandit problem in an adversarial environment.

[0071] In response to this, the selection unit 11 of the information processing device 1 selects the action a in each round t according to the algorithm shown in Table 1 above. t (i.e., the feature vector of the advertisement to be displayed). Therefore, whether the environment is stochastic or hostile, the regret R T This provides a practical effect that the advertisement can be displayed on the user terminal 5 so as to keep the number of times the advertisement is displayed sufficiently small.

[0072] The information processing device 1 uses the loss f acquired in the acquisition process S13 t (a t ) may be displayed on the display of the administrator terminal as a "cumulative loss" graph. An example of a screen I1 including a "cumulative loss" graph displayed on the display of the administrator terminal is shown in FIG. 6. This provides a practical effect in that the administrator can easily grasp the change in cumulative loss for each round.

[0073] In addition, the information processing device 1 uses the unbiased estimator ^l calculated in the selection process S11. t or predicted value m t The graph of the above may be displayed on the display of the administrator terminal as a graph of "estimated loss." An example of a screen I2 including the graph of "estimated loss" displayed on the display of the administrator terminal is shown in Figure 6. This has the practical effect of allowing the administrator to easily grasp the changes in estimated loss for each round.

[0074] [Example of Implementation by Software] Some or all of the functions of the information processing device 1 may be implemented by hardware such as an integrated circuit (IC chip) or by software. In the latter case, the functions of each part of the information processing device 1 are implemented by, for example, a computer that executes instructions of a software program.

[0075] An example of such a computer (hereinafter referred to as computer C) is shown in Fig. 7. As shown in Fig. 7, computer C includes at least one processor C1 and at least one memory C2. Memory C2 stores an information processing program P for causing computer C to operate as information processing device 1. In computer C, processor C1 reads and executes program P from memory C2, thereby realizing the functions of each part of information processing device 1.

[0076] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0077] The computer C may further include a RAM (Random Access Memory) for expanding the information processing program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input devices such as a keyboard and a mouse, and / or output devices such as a display and a printer.

[0078] Furthermore, the information processing program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the information processing program P via such a recording medium M. Furthermore, the information processing program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the information processing program P via such a transmission medium.

[0079] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0080] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the aspects described below as an addition.

[0081] (Appendix 1) Action a in each round t t A selection means for selecting a process P(a t ) as the action a t and an execution means for executing a process according to the loss f t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring means for acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t and the selection means is a selection action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t is a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ An information processing device that runs on

[0082] (Supplementary Note 2) The subset U t is the action z t For the set E t = {xt +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and the λ1, λ2, ..., λ d and the e1, e2, ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, and t is the upscaled set U t (1 / r t , z t , E t ) is set to be the largest upscale set contained in the convex hull X.

[0083] (Appendix 3) The experience loss Φ t-1 The information processing device according to Supplementary Note 1 or 2, wherein (x) is defined by the following formula (c1):

[0084] Here, ^l s is an unbiased estimator of the loss vector at each round s∈[t−1], and m t is the predicted value of the loss vector at round t, and β t is the learning rate at round t, and ψ is the θ-self-concordant barrier function for the convex hull X.

[0085] (Appendix 4) The loss f t (a t ) is a linear function <l t , a>+ξ t and the selection means is a selection action a t The expected value of the action x t (1) the action a t The action zt The process of setting the probability r ｔ (2) the action a t The action y t The process of setting ｔ 4. The information processing device according to claim 1, wherein the information processing device executes the above-mentioned steps.

[0086] (Appendix 5) Action a in each round t t A selection process for selecting P(a t ) as the action a t The execution process executes the process according to the loss f t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring a loss according to the action a. t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t In the selection process, the selection action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t is a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ An information processing method performed by

[0087] (Supplementary Note 6) The subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i-1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and the λ1, λ2, ..., λ d and the e1, e2, ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, and t is the upscaled set U t (1 / r t , z t , E t 6. The information processing method according to claim 5, wherein the convex hull X is set to be the largest upscale set contained in the convex hull X.

[0088] (Supplementary Note 7) The information processing device and another device are included, and the information processing device performs an action a in each round t. t A selection means for selecting a process P(a t ) as a control for the other device, t and a control unit for controlling the loss f of each round t. t (a t ) as a loss resulting from the control, t and acquiring means for acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t and the selection means is a selection action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action at Let us consider a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ An information processing system that runs on

[0089] (Supplementary Note 8) The subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and the λ1, λ2, ..., λ d and the e1, e2, ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, and t is the upscaled set U t (1 / r t , z t , E t 8. The information processing system of claim 7, wherein the convex hull X is set to be the largest upscale set contained in the convex hull X.

[0090] (Supplementary Note 9) An information processing program for causing a computer to operate as the information processing device according to any one of Supplementary Notes 1 to 4, the information processing program causing the computer to function as the selection means, the execution means, and the acquisition means.

[0091] (Supplementary Note 10) A computer-readable recording medium on which the information processing program according to Supplementary Note 9 is recorded.

[0092] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.

[0093] (Supplementary Note 11) At least one processor is provided, and the processor performs an action a in each round t. t A selection process for selecting P(a t ) as the action a t The execution process executes the process according to the loss f t (a t ) as the process P(a t ) losses resulting from said action a t and performing an acquisition process to acquire a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t In the selection process, the selection action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t is a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ An information processing device executed by

[0094] (Supplementary Note 12) The information processing device described in Supplementary Note 11 may further include a memory, and the memory may store an information processing program for causing the processor to execute the selection process, the execution process, and the acquisition process. The information processing program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0095] REFERENCE SIGNS LIST 1 Information processing device 11 Selection unit 12 Execution unit 13 Acquisition unit S1 Information processing method S11 Selection process S12 Execution process S13 Acquisition process 100A Information processing system 2 Robot (another device) 5 User terminal (another device)

Claims

1. Action a in each round t t a selection means for selecting Processing P(a t ) as the action a t an execution means for executing a process according to the Loss f for each round t t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring means for acquiring losses according to the The action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t is approximated by The selection means selects an action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t A subset U of the convex hull X of the action set A ｔ A randomly selected action y t The process of setting ｔ Run it with Information processing device.

2. Said subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and Said λ 1 , λ 2 , …,λ d and the above e 1 , e 2 , ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t , z t , E t ) is set to be the largest upscaled set contained in the convex hull X. The information processing device according to claim 1 .

3. the experienced loss Φ t-1 The information processing device according to claim 1 , wherein (x) is defined by the following formula (c1): [Equation 1] Here, ^l s (in the formula, ^ on l) is an unbiased estimator of the loss vector in each round s∈[t−1], and m t is the predicted value of the loss vector at round t, and β t is the learning rate at round t, and ψ is the θ-self-concordant barrier function for the convex hull X.

4. The loss f t (a t ) is a linear function <l t , a>+ξ t is expressed by The selection means selects an action a t The expected value of the action x t (1) the action a t The action z t The process of setting the probability r ｔ (2) the action a t The action y t The process of setting ｔ Run it with The information processing device according to claim 1 .

5. Action a in each round t t a selection process for selecting Processing P(a t ) as the action a t an execution process that executes a process according to the Loss f for each round t t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring a loss according to the The action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t is approximated by In the selection process, the action to be selected is t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t A subset U of the convex hull X of the action set A ｔ A randomly selected action y t The process of setting r ｔ An information processing method performed by

6. Said subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and Said λ 1 , λ 2 , …,λ d and the above e 1 , e 2 , ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t , z t , E t ) is set to be the largest upscaled set contained in the convex hull X. The information processing method according to claim 5 .

7. The information processing method according to claim 5, wherein the empirical loss Φ t-1 (x) is defined by the following formula (c1): [Equation 2] where ^l s (^ over l in the formula) is an unbiased estimator of the loss vector at each round s∈[t−1], m t is the predicted value of the loss vector at round t, β t is the learning rate at round t, and ψ is the θ-self-concordant barrier function for the convex hull X.

8. an information processing device and another device, The information processing device includes: Action a in each round t t a selection means for selecting Processing P(a t ) as a control for the other device, t an execution means for executing control according to the Loss f for each round t t (a t ) as a loss resulting from the control, t and acquiring means for acquiring losses according to the The action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t is approximated by The selection means selects an action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t Let us consider a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ Run it with Information processing system.

9. Said subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and Said λ 1 , λ 2 , …,λ d and the above e 1 , e 2 , ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t , z t , E t ) is set to be the largest upscaled set contained in the convex hull X. The information processing system according to claim 8 .

10. 5. An information processing program for causing a computer to operate as the information processing device according to any one of claims 1 to 4, the information processing program causing the computer to function as the selection means, the execution means, and the acquisition means.

Images