Information processing device, information processing method, information processing system, and information processing program

Abstract

This information processing device (1) comprises: a selection unit (11) which selects an action at of each round t; an execution unit (12) which executes a process P(at) of each round t; and an acquisition unit (13) which acquires a risk ft(at) of each round t. In order that the expectation value of the action at to be selected is caused to approximate an action xt that minimizes an empirical risk Φt-1(x), the selection unit (11): (1) executes a process for setting the action at to an action zt that is arbitrarily selected from an action set A at a probability of rt-1; and (2) executes a process for setting the action at to an action yt that is randomly selected from a partial set Ut of a convex hull X of the action set A at a probability of rt. The execution unit (12) executes, as the process P(at), for example, a process for controlling a robot 2 to move from an A spot to a B spot via a selected moving path as the action at.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing system, and an information processing program for solving bandit problems, and also to a recording medium on which such an information processing program is recorded. [Background technology]

[0002] There is a demand for technology that can replace various human decision-making tasks with computers. For example, the problem of players choosing an action in each round is an example of a problem that can be solved by a computer. Among such problems, the problem of choosing an action in each round without considering the loss that is determined by that action is called a "bandit problem." In particular, the problem of choosing an action a in each round t is called a "bandit problem." t is expressed as a d-dimensional vector (d is any natural number greater than or equal to 1), and the loss f t (a t ) is a linear function <l t ,a t >+ξ t A bandit problem expressed as follows is called a "linear bandit problem."

[0003] A computer-based method for solving bandit problems is to determine the action a in each round t. t Let f1(a1), f2(a2), …, f be the losses up to the previous round t-1. t-1 (a t-1 ) t-1 Action x that minimizes (x) t In Non-Patent Document 1, the action a in each round t is selected. t As the empirical loss Φ t-1 Action x that minimizes (x) t Instead of selecting t ={x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i|i∈[d]}, an action y chosen randomly t Here, a method for selecting {λ1,λ2,…,λ d} and {e1,e2,…,e d} are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull of the action set, respectively. 2 According to the method described in Non-Patent Document 1, in an adversarial environment, that is, the loss f t (a t ) is the action a1, a2,…, a up to the previous round t-1. t-1 In an environment where the relevance depends on the relevance of the system, it is guaranteed that regrets can be kept sufficiently small. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. "Competing in the dark: An efficient algorithm for bandit linear optimization". In Proceedings of the 21st Annual Conference on Learning Theory (COLT), pages 263-274, 2008. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the environment in which the bandit problem should be solved is not limited to an adversarial environment. For example, there is a stochastic environment, i.e., the loss f t (a t ) is the action a1, a2,…, a up to the previous round t-1. t-1There are cases where a bandit problem is solved in an environment determined according to a probability distribution that does not depend on . Alternatively, there are cases where a bandit problem is solved in an environment where it is unclear whether it is hostile or stochastic. In such cases, the method described in Non-Patent Document 1 does not guarantee that regret can be kept sufficiently small.

[0006] One aspect of the present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize an information processing device, information processing method, or information processing system that solves bandit problems and can keep regrets sufficiently small in both hostile environments and stochastic environments. [Means for solving the problem]

[0007] An information processing device according to an aspect of the present invention is t and the process P(a t ) as the action a t and the loss f t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring means for acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t ,a>+ξ t and the selection means is a selecting action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t (2) execute the action a t Let us consider a subset U of the convex hull X of the action set A. t A randomly selected action from yt The process of setting t Run it with:

[0008] In an information processing method according to one aspect of the present invention, an action a t and the selection process for each round t, P(a t ) as the action a t The execution process executes the process according to the loss f t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t ,a>+ξ t In the selection process, the action to be selected is a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t (2) the action a t Let us consider a subset U of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t It is executed in

[0009] An information processing system according to one aspect of the present invention includes an information processing device and another device, and the information processing device determines an action a in each round t. t and the process P(a t ) as a control for the other device, and the action a t The execution method to execute control according to the loss f t (a t ) as a result of the control, the loss tand acquiring means for acquiring a loss according to the action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t ,a>+ξ t and the selection means is a selecting action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t (2) execute the action a t Let U be a subset of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t Run it with: [Effects of the Invention]

[0010] According to one aspect of the present invention, regrets can be kept sufficiently small in both hostile and stochastic environments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Figure 2] 1 is a flowchart showing the flow of an information processing method according to a first exemplary embodiment of the present invention. [Figure 3] 3 is a flowchart showing a specific example of a selection process included in the information processing method shown in FIG. 2. FIG. [Figure 4] FIG. 10 is a schematic diagram showing a configuration of an information processing system according to a second exemplary embodiment of the present invention. [Figure 5] FIG. 10 is a schematic diagram showing a configuration of an information processing system according to a third exemplary embodiment of the present invention. [Figure 6] 6 is a plan view showing a screen displayed on an administrator terminal in the information processing system shown in FIGS. 4 and 5. FIG. [Figure 7] 2 is a block diagram showing the configuration of a computer that functions as the information processing device shown in FIG. 1. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] [Linear bandit problem] Action a in each round t∈[T] t Consider a problem in which a player selects an action set A from the set of actions A. In such a problem, the action a t The action a t The loss f in round t is determined by t (a t ) without referring to the action a in each round is called a "bandit problem." t is expressed as a d-dimensional vector (d is any natural number greater than or equal to 1), and the loss f t (a t ) is a linear function <l t ,a t >+ξ t A bandit problem expressed as follows is called a "linear bandit problem." Here, l t is a d-dimensional vector called the "loss vector". t is a scalar.

[0013] R defined by the following formula (1) T The algorithm for solving the bandit problem, i.e., the action a in each round t, t The superiority of the algorithm for selecting this Regret R T It is evaluated using Regret R T The smaller the algorithm can be, the better the algorithm.

[0014]

number

[0015] In the following explanation, X represents the convex hull of the action set A. Also, ψ represents the barrier function of the convex hull X, ∇ψ(x) represents the gradient of the barrier function ψ at x∈X, and D(x,y) represents the Bergman divergence of the barrier function ψ at x,y∈X. Also, ∇ 2 ψ(x) denotes the Hessian of the barrier function ψ for x∈X, and ∇ -2 ψ(x) is the Hessian matrix ∇ 2 Represents the inverse matrix of ψ(x).

[0016] The barrier function ψ of the convex hull X is a function whose value ψ(x) diverges to infinity as x∈X approaches the boundary of the convex hull X. Among the barrier functions ψ, a barrier function ψ that satisfies the following equation (2) for θ>0 is called a "θ-self-concordant barrier function." θ is called the self-concordance parameter. Here, Dψ(x) represents the first derivative of the barrier function at x∈X, and D 2 ψ(x) represents the second derivative of the barrier function at x∈X, and D 3 ψ(x) denotes the third derivative of the barrier function at x∈X.

[0017]

number

[0018] In addition, the set U(α, z, E) = {z + α(xz)|x∈E} defined for any element z of the convex hull X, any subset E of the convex hull X, and any scalar α≧1 is called an "upscaled set obtained by expanding the set E by α times for z." In the embodiment exemplified below, an action z arbitrarily selected from the action set A is t For the set E t ={x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t ,z t ,E t ) is used, where x t is the empirical loss Φ in the convex hull X t-1 (x) is the action that minimizes {λ1,λ2,…,λ d} and {e1,e2,…,ed} are the Hessian matrices ∇ of the barrier function ψ, respectively. 2 The sequence of eigenvalues ​​and eigenvectors of ψ. t is the upscaled set U t (1 / r t ,z t ,E t ) is the maximum up that is contained in the convex hull scale It is set to be a set.

[0019] Exemplary Embodiment 1 A first exemplary embodiment of the present invention will be described with reference to the drawings. This exemplary embodiment is a base for the exemplary embodiments described below.

[0020] (Configuration of information processing device) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1.

[0021] As shown in FIG. 1, the information processing device 1 includes a selection unit 11, an execution unit 12, and an acquisition unit 13.

[0022] The selection unit 11 selects the action a t In each round t, the action a t is assumed to be represented by a d-dimensional numeric vector.

[0023] The execution unit 12 executes the process P(a t ) where, the process P(a t ) is the action a selected by the selection unit 11 in that round t. t The process P(a t ) is the action a in round t. t Examples of the other device include a process of controlling another device depending on the robot 2. A case in which the robot 2 is controlled as the other device will be described in an exemplary embodiment 2, which will be described later, and a case in which the user terminal 5 is controlled as the other device will be described in an exemplary embodiment 3, which will be described later.

[0024] The acquisition unit 13 acquires the loss f t (a t ) for each round t. t (a t ) is the process P(a t ) is the loss resulting from the action a selected by the selection unit 11 in that round t. t In this embodiment, the loss f t (a t ) is a linear function <l t ,a t >+ξ t or a linear function <l t ,a t >+ξ t Assume that it is approximated by

[0025] The selection unit 11 selects an action a t The expected value of the losses f1(a1), f2(a2), …, f t-1 (a t-1 ) t-1 Action x that minimizes (x) t (1) Action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t -1 executes action a t Let U be a subset of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t Run it with:

[0026] In particular, in this embodiment, the above-mentioned subset U t As, act z t For the set E t ={x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λi -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t ,z t ,E t ) where x t is the empirical loss Φ in the convex hull X t-1 (x) is the action that minimizes {λ1,λ2,…,λ d} and {e1,e2,…,e d} are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 The sequence of eigenvalues ​​and eigenvectors of ψ. t is the upscaled set U t (1 / r t ,z t ,E t ) is the maximum up that is contained in the convex hull scale In this case, the action a selected by the selection unit 11 is t The expected value of is the empirical loss Φ t-1 Action x that minimizes (x) t It can be mathematically proven that

[0027] (Flow of information processing method) The flow of the information processing method S1 performed by the information processing device 1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1.

[0028] 2, the information processing method S1 includes a selection process S11, an execution process S12, and an acquisition process S13. The selection process S11, the execution process S12, and the acquisition process S13 are executed in each round t∈[T].

[0029] The selection process S11 selects the action a t As mentioned above, the action a in each round t is tis assumed to be represented by a d-dimensional numeric vector. In this exemplary embodiment, the selection process S11 is performed by the selection unit 11.

[0030] The execution process S12 is a process P(a t As mentioned above, the process P(a t ) is the action a selected by the selection unit 11 in that round t. t In this exemplary embodiment, the execution process S12 is executed by the execution unit 12. Note that the process P(a t ) is the action a in round t. t Examples of the other device include a process of controlling another device depending on the robot 2. A case in which the robot 2 is controlled as the other device will be described in an exemplary embodiment 2, which will be described later, and a case in which the user terminal 5 is controlled as the other device will be described in an exemplary embodiment 3, which will be described later.

[0031] The acquisition process S13 calculates the loss f t (a t ) is obtained. As mentioned above, the loss ft(a t ) is the process P(a t ) is the loss resulting from the action a selected in the selection process S11 of that round t. t As mentioned above, the loss f t (a t ) is a linear function <l t ,a t >+ξ t or a linear function <l t ,a t >+ξ t In this exemplary embodiment, the acquisition process S13 is performed by the acquisition unit 13.

[0032] In the selection process S11, the action a to be selected is tThe expected value of the losses f1(a1), f2(a2), …, f t-1 (a t-1 ) t-1 Action x that minimizes (x) t (1) Action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t -1, and (2) action a t Let U be a subset of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t It is executed in

[0033] In particular, in this embodiment, the above-mentioned subset U t As, act z t For the set E t ={x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t ,z t ,E t ) where x t is the empirical loss Φ in the convex hull X t-1 (x) is the action that minimizes {λ1,λ2,…,λ d} and {e1,e2,…,e d} are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 The sequence of eigenvalues ​​and eigenvectors of ψ. t is the upscaled set U t (1 / r t ,z t ,E t ) is the maximum up that is contained in the convex hull scale In this case, the action a selected by the selection unit 11 ist The expected value of is the empirical loss Φ t-1 Action x that minimizes (x) t It can be mathematically proven that the above-mentioned selection process S11 is consistent with the above-mentioned. A specific example of this selection process S11 will be described later with reference to different drawings.

[0034] In addition, in the selection process S11, the action a t In preparation for selecting t (a t ) in the current round t. t (a t ) is the step that requires reference to the action a of the current round t in the selection process S11. t In the execution process S12, the process P(a t ) and in the acquisition process S13, the loss f t (a t ) is obtained.

[0035] (Example of selection process) A specific example of the selection process S11 will be described with reference to Fig. 3. Fig. 3 is a flow diagram showing the flow of the selection process S11 according to this specific example. In the selection process S11 according to this specific example, the empirical loss Φ t-1 ^l as a parameter to define (x) s (In the formula, ^)∈R above l d ,m t ∈R d ,β t >0 is used. Here, ^l s is the loss vector l s represents an unbiased estimator of (s=1,2,…,t-1). Also, m t is the loss vector l t represents the optimistic prediction of t represents the learning rate.

[0036] As shown in FIG. 3, the selection process S11 includes a predicted value setting step S101, a learning rate setting step S102, a reference action selection step S103, a set creation step S104, a candidate action selection step S105, a probability setting step S106, a flag setting step S107, an action selection step S108, an unbiased estimator setting step S109, and a predicted value update step S110.

[0037] The predicted value setting step S101 is a step executed only in the first round (t=1), and the other steps S102 to S110 are steps executed in each round (t=1, 2, ..., T). Of the other steps S102 to S110, steps S102 to S108 are steps executed by the execution unit 12 in the process P(a t ), and steps S109 to S110 are steps that the selection unit 11 executes before executing the loss f t (a t ) is obtained.

[0038] The predicted value setting step S101 is a step of setting a predicted value m1. In the predicted value setting step S101, the selection unit 11 sets the predicted value m1 to m1=0.

[0039] The learning rate setting step S102 sets the learning rate β t In the learning rate setting step S102, the selection unit 11 sets the learning rate β t For t=1, β t =16d 2 If t>1, the flag b is set according to the following formulas (a1) and (a2). s The definition of will be explained later.

[0040]

number

[0041]

number

[0042] The candidate action selection step S105 selects candidate actions z t In the candidate action selection step S105, the selection unit 11 selects an arbitrary element a in the action set A as a candidate action z t Select as.

[0043] The probability setting step S106 sets the probability r t In the probability setting step S106, the selection unit 11 sets the probability r t is set according to the following formula (a4). The probability r t is the set E t Candidate Action z t The upscaled set U(1 / r,z t ,E t) is the smallest positive number r that is contained in the convex hull X.

[0044]

number

[0045] The action selection step S108 is to select the action a t In the action selection step S108, the selection unit 11 selects the flag b t Ga b t = 0, then candidate action z t , the action of round t t The selection unit 11 also selects the flag b t Ga b t If =1, then a randomly selected i from [d] t and ε chosen randomly from {-1,+1} t Using and, action a of round t t a t =z t +(1 / r t )(x t +ε t λ it -1 / 2 e it -z t ) and set it according to the t =z t +(1 / r t )(x t +ε t λ it -1 / 2 e it -z t) according to the action a in round t t Setting the randomly selected upscale set U(1 / r t ,z t ,E t ) element y t , the action of round t t is equivalent to selecting

[0046] The unbiased estimator setting step S109 is performed by setting the loss vector l t An unbiased estimator of ^l s In the unbiased estimator setting step S109, the selection unit 11 sets the unbiased estimator ^l t is set according to the following equation (a5). Here, the loss f t (a t ) is the action a in round t that the selection unit 11 t The execution unit 12 selects the process P(a t ) is executed, the acquisition unit 13 acquires it.

[0047]

number

[0048]

number

[0049] [Table 1] The inventors determine the action a in each round t according to the algorithm shown in Table 1 above. tBy selecting T Specifically, we have succeeded in proving that (1) in any environment, the regret R T can be evaluated as the following formula (b1), and (2) in any stochastic environment, regret R T can be evaluated as the following equation (b2), and (3)ξ t In a hostile environment where = 0, the regret R T We have successfully proven that it can be evaluated as the following equation (b3).

[0050]

number

[0051]

number

[0052]

number

[0053] In addition, the above evaluation is based on the action a in each round t. t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t More broadly, the action a in each round t t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t It is clear that the same evaluation holds when the action a of each round t is approximated by t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t The behavior that matches with the action a in each round t t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t That is, the action a in each round t is t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t The action a in each round t is approximated by t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) tdoes not imply that it does not match.

[0054] Exemplary Embodiment 2 A second exemplary embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing the configuration of an information processing system 100A according to the second exemplary embodiment.

[0055] As shown in FIG. 4, the information processing system 100A includes, in addition to the information processing device 1, a robot 2 configured to be able to communicate with the information processing device 1.

[0056] In this exemplary embodiment, the selection unit 11 of the information processing device 1 selects the action a t The path of movement of robot 2 from point A to point B is selected as follows. The path of movement of robot 2 from point A to point B is expressed by a d-dimensional vector, where the i-th component is 1 when robot 2 passes through path Ri, and is 0 when robot 2 does not pass through path Ri. Here, d is the number of paths that can constitute the path from point A to point B (d=8 in the example shown).

[0057] In this exemplary embodiment, the execution unit 12 of the information processing device 1 executes the process P(a t ) and the action a in that round t t The execution unit 12 of the information processing device 1 executes a process to control the robot 2 so that it moves from point A to point B along the selected movement route. However, the paths R1 to Rd are also used by other robots 3 and 4. For this reason, the execution unit 12 of the information processing device 1 controls the robot 2 so that it does not collide with the other robots 3 and 4.

[0058] In this embodiment, the acquisition unit 13 of the information processing device 1 calculates the loss f t (a t ) and the action a in that round t t The time required for the robot 2 to move from point A to point B along the selected movement route is acquired from the robot 2.

[0059] Loss f for each round tt (a t ) (i.e., the movement time of robot 2) depends on the actions of other robots 3 and 4. Therefore, the action a t (i.e., the path of robot 2) cannot be known before the action a t The problem of selecting a path (i.e., the path of movement of robot 2) becomes a bandit problem.

[0060] In particular, if the actions of other robots 3 and 4 are determined according to a certain probability distribution, then the action a t The problem of selecting the path of robot 2 (i.e., the path of robot 2) is a bandit problem in a stochastic environment. On the other hand, the actions of other robots 3 and 4 are the same as the actions a1, a2, ..., a t-1 (i.e., the path of robot 2 so far), then the action a in each round t t The problem of selecting a path (i.e., the path that Robot 2 will take) becomes a bandit problem in a hostile environment.

[0061] In response to this, the selection unit 11 of the information processing device 1 selects the action a in each round t according to the algorithm shown in Table 1 above. t (i.e., the path of robot 2). Therefore, even if the environment is stochastic or hostile, regret R T This provides a practical effect that the robot 2 can be controlled so as to keep the value of the rotational speed of the robot 2 sufficiently small.

[0062] The information processing device 1 uses the loss f acquired in the acquisition process S13. t (a t ) may be displayed on the display of the administrator terminal as a "cumulative loss" graph. An example of screen I1 containing a "cumulative loss" graph, displayed on the display of the administrator terminal, is shown in Figure 6. This has the practical effect of allowing the administrator to easily grasp the changes in cumulative loss for each round.

[0063] In addition, the information processing device 1 uses the unbiased estimator ^l calculated in the selection process S11. t or predicted value m t The graph of the above may be displayed on the display of the administrator terminal as a graph of "estimated loss." An example of screen I2 containing the "estimated loss" graph, displayed on the display of the administrator terminal, is shown in Figure 6. This has the practical effect of allowing the administrator to easily grasp the changes in estimated loss for each round.

[0064] Exemplary Embodiment 3 A third exemplary embodiment of the present invention will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing the configuration of an information processing system 100B according to the second exemplary embodiment.

[0065] As shown in FIG. 5, the information processing system 100B includes, in addition to the information processing device 1, a user terminal 5 configured to be able to communicate with the information processing device 1.

[0066] In this exemplary embodiment, the selection unit 11 of the information processing device 1 selects the action a t A d-dimensional feature vector is selected as

[0067] In this exemplary embodiment, the execution unit 12 of the information processing device 1 executes the process P(a t ) and the action a in that round t t The advertisement generating unit 100 generates an advertisement according to the selected feature vector, and controls the user terminal 100 to display the generated advertisement on a display.

[0068] In this embodiment, the acquisition unit 13 of the information processing device 1 calculates the loss f t (a t ) and the action a in that round t t If the user clicks on an advertisement corresponding to the feature vector selected as tIf the user does not click on the advertisement corresponding to the feature vector selected as the user, a loss value of "1" is acquired from the user terminal 5.

[0069] Loss f for each round t t (a t ) depends on the user's actions, so the action a t (i.e., the feature vector of the ad to be displayed) cannot be known before selecting it. Therefore, the action a t The problem of selecting the feature vector of the advertisement to display is a bandit problem.

[0070] In particular, if a user's click behavior is determined according to a certain probability distribution, then the action a in each round t t The problem of selecting the feature vector of the advertisement to display is a bandit problem in a stochastic environment. On the other hand, the user's click behavior is determined by the actions a1, a2, ..., a t-1 (i.e., the feature vector of the ads displayed so far), then the action a in each round t t The problem of selecting the feature vector of the advertisement to be displayed is a bandit problem in an adversarial environment.

[0071] In response to this, the selection unit 11 of the information processing device 1 selects the action a in each round t according to the algorithm shown in Table 1 above. t (i.e., the feature vector of the ad to be displayed). Therefore, whether the environment is stochastic or adversarial, the regret R T This provides a practical effect that the advertisement can be displayed on the user terminal 5 so as to keep the number of times the advertisement is displayed sufficiently small.

[0072] The information processing device 1 uses the loss f acquired in the acquisition process S13. t (a t) may be displayed on the display of the administrator terminal as a "cumulative loss" graph. An example of screen I1 containing a "cumulative loss" graph, displayed on the display of the administrator terminal, is shown in Figure 6. This has the practical effect of allowing the administrator to easily grasp the changes in cumulative loss for each round.

[0073] In addition, the information processing device 1 uses the unbiased estimator ^l calculated in the selection process S11. t or predicted value m t The graph of the above may be displayed on the display of the administrator terminal as a graph of "estimated loss." An example of screen I2 containing the "estimated loss" graph, displayed on the display of the administrator terminal, is shown in Figure 6. This has the practical effect of allowing the administrator to easily grasp the changes in estimated loss for each round.

[0074] [Software implementation example] Some or all of the functions of the information processing device 1 may be realized by hardware such as an integrated circuit (IC chip) or by software. In the latter case, the functions of each part of the information processing device 1 are realized by, for example, a computer that executes instructions of a program, which is software.

[0075] An example of such a computer (hereinafter referred to as computer C) is shown in Fig. 7. As shown in Fig. 7, computer C includes at least one processor C1 and at least one memory C2. Memory C2 stores an information processing program P for causing computer C to operate as information processing device 1. In computer C, processor C1 reads and executes program P from memory C2, thereby realizing the functions of each part of information processing device 1.

[0076] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0077] The computer C may further include a RAM (Random Access Memory) for expanding the information processing program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input devices such as a keyboard and a mouse, and / or output devices such as a display and a printer.

[0078] Furthermore, the information processing program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the information processing program P via such a recording medium M. Furthermore, the information processing program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the information processing program P via such a transmission medium.

[0079] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0080] [Appendix 2] Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the aspects described below as supplementary notes.

[0081] (Appendix 1) Action a for each round t t a selection means for selecting The process P(a t ) as the action a t an execution means for executing a process according to the Loss f for each round t t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring means for acquiring losses according to the The above action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t ,a>+ξ t is approximated by The selection means selects an action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t (2) execute the action a t Let us consider a subset U of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t Run it with Information processing device.

[0082] (Appendix 2) Said subset U t is the action z t For the set E t ={x t +λ i -1 / 2 e i |i∈[ d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t ,z t ,E t ) and The λ1, λ2, ..., λ d and the e1, e2, ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t ,z t ,E t ) is the maximum up that is included in the convex hull X scale It is set to be a set 2. The information processing device according to claim 1.

[0083] (Appendix 3) the experienced loss Φ t-1 (x) is defined by the following formula (c1): 3. The information processing device according to claim 1 or 2.

[0084]

number

[0085] (Appendix 4) The loss f t (a t ) is a linear function <l t ,a>+ξ t is expressed by The selection means selects an action a t The expected value of the action x t (1) Action a t The action z t The process of setting the probability r t (2) execute the action a t The action y t The process of setting t Run it with 4. The information processing device according to any one of appendices 1 to 3.

[0086] (Appendix 5) Action a for each round t t a selection process for selecting The process P(a t ) as the action a t an execution process that executes a process according to the Loss f for each round t t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring a loss according to the The above action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t ,a>+ξ t is approximated by In the selection process, the action to be selected is t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a tLet z be an action selected arbitrarily from the action set A. t The process of setting t (2) the action a t Let us consider a subset U of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t It is executed in Information processing methods.

[0087] (Appendix 6) Said subset U t is the action z t For the set E t ={x t +λ i -1 / 2 e i |i∈[ d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t ,z t ,E t ) and The λ1, λ2, ..., λ d and the e1, e2, ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t ,z t ,E t ) is the maximum up that is included in the convex hull X scale It is set to be a set 1. The information processing method described in Appendix 5.

[0088] (Appendix 7) an information processing device and another device, The information processing device includes: Action a for each round t ta selection means for selecting The process P(a t ) as a control for the other device, and the action a t an execution means for executing control according to the Loss f for each round t t (a t ) as a result of the control, the loss t and acquiring means for acquiring losses according to the The above action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t ,a>+ξ t is approximated by The selection means selects an action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t (2) execute the action a t Let U be a subset of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t Run it with Information processing system.

[0089] (Appendix 8) Said subset U t is the action z t For the set E t ={x t +λ i -1 / 2 e i |i∈[ d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t ,z t ,Et ) and The λ1, λ2, ..., λ d and the e1, e2, ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t ,z t ,E t ) is the maximum up that is included in the convex hull X scale It is set to be a set 8. The information processing system of claim 7.

[0090] (Appendix 9) An information processing program for causing a computer to operate as the information processing device according to any one of Supplementary Notes 1 to 4, the information processing program causing the computer to function as the selection means, the execution means, and the acquisition means.

[0091] (Appendix 10) A computer-readable recording medium on which the information processing program according to Supplementary Note 9 is recorded.

[0092] [Appendix 3] Some or all of the above-described embodiments can also be expressed as follows.

[0093] (Appendix 11) at least one processor; The processor: Action a for each round t t a selection process for selecting The process P(a t ) as the action a t an execution process that executes a process according to the Loss f for each round t t (a t ) as the process P(a t) losses resulting from said action a t and performing an acquisition process to acquire a loss according to The above action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t ,a>+ξ t is approximated by In the selection process, the action to be selected is t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t Let z be an action selected arbitrarily from the action set A. t The process of setting t (2) the action a t Let us consider a subset U of the convex hull X of the action set A. t A randomly selected action from y t The process of setting t It is executed in Information processing device.

[0094] (Appendix 12) The information processing device described in Supplementary Note 11 may further include a memory, and the memory may store an information processing program for causing the processor to execute the selection process, the execution process, and the acquisition process. The information processing program may be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0095] 1. Information processing equipment 11 Selection section 12 Executive Department 13 Acquisition Department S1 Information processing method S11 Selection process S12 Execution process S13 Acquisition process 100A Information Processing System 2. Robots (other devices) 5 User terminals (other devices)

Claims

1. Action a in each round t t a selection means for selecting Processing P(a t ) as the action a t an execution means for executing a process according to the Loss f for each round t t (a t ) as the process P(a t ) losses resulting from said action a t and acquiring means for acquiring losses according to the The action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t is approximated by The selection means selects an action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t A subset U of the convex hull X of the action set A ｔ A randomly selected action y t The process of setting ｔ Run it with Information processing device.

2. Said subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and Said λ 1 , λ 2 , …,λ d and the above e 1 , e 2 , ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t , z t , E t ) is set to be the largest upscaled set contained in the convex hull X. The information processing device according to claim 1 .

3. the experienced loss Φ t-1 The information processing device according to claim 1 , wherein (x) is defined by the following formula (c1): [Equation 1] Here, ^l s (in the formula, ^ on l) is an unbiased estimator of the loss vector in each round s∈[t−1], and m t is the predicted value of the loss vector at round t, and β t is the learning rate at round t, and ψ is the θ-self-concordant barrier function for the convex hull X.

4. The loss f t (a t ) is a linear function <l t , a>+ξ t is expressed by The selection means selects an action a t The expected value of the action x t (1) the action a t The action z t The process of setting the probability r ｔ (2) the action a t The action y t The process of setting ｔ Run it with The information processing device according to claim 1 .

5. A computer comprising: Action a in each round t t a selection process for selecting Processing P(a t ) as the action a t an execution process that executes a process according to the Loss f for each round t t (a t ) as the process P(a t ) losses resulting from said action a t and performing an acquisition process to acquire a loss according to The action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t is approximated by In the selection process, the action to be selected is t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) the action a t A subset U of the convex hull X of the action set A ｔ A randomly selected action y t The process of setting ｔ An information processing method performed in

6. Said subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and Said λ 1 , λ 2 , …,λ d and the above e 1 , e 2 , ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t , z t , E t ) is set to be the largest upscaled set contained in the convex hull X. The information processing method according to claim 5 .

7. the experienced loss Φ t-1 7. The information processing method according to claim 5, wherein (x) is defined by the following formula (c1): [Equation 2] Here, ^l s (in the formula, ^ on l) is an unbiased estimator of the loss vector in each round s∈[t−1], and m t is the predicted value of the loss vector at round t, and β t is the learning rate at round t, and ψ is the θ-self-concordant barrier function for the convex hull X.

8. an information processing device and another device, The information processing device includes: Action a in each round t t a selection means for selecting Processing P(a t ) as a control for the other device, t an execution means for executing control according to the Loss f for each round t t (a t ) as a loss resulting from the control, t and acquiring means for acquiring losses according to the The action a t is expressed by a d-dimensional vector, and the loss f t (a t ) is a linear function <l t , a>+ξ t is approximated by The selection means selects an action a t The expected value of the empirical loss Φ t-1 Action x that minimizes (x) t (1) the action a t is an action z arbitrarily selected from the action set A. t The process of setting ｔ (2) execute the action a t Let us consider a subset U of the convex hull X of the action set A. ｔ A randomly selected action y t The process of setting ｔ Run it with Information processing system.

9. Said subset U t is the action z t For the set E t = {x t +λ i -1 / 2 e i |i∈[d]}∪{x t -λ i -1 / 2 e i |i∈[d]} as 1 / r t Upscaled set U t (1 / r t , z t , E t ) and Said λ 1 , λ 2 , …,λ d and the above e 1 , e 2 , ..., e d are the Hessian matrices ∇ of the θ-self-concordant barrier function ψ for the convex hull X, respectively. 2 are the eigenvalues ​​and eigenvectors of ψ, The r t is the upscaled set U t (1 / r t , z t , E t ) is set to be the largest upscaled set contained in the convex hull X. The information processing system according to claim 8 .

10. 5. An information processing program for causing a computer to operate as the information processing device according to any one of claims 1 to 4, the information processing program causing the computer to function as the selection means, the execution means, and the acquisition means.

Images