Information processing device, information processing method, and program

Abstract

In order to generate a more suitable reward function, an information processing device (1) is provided with: an acquisition unit (11) that acquires reference data; and a determination unit (12) that determines a reward function including weighting factors and feature quantity parameters by inverse reinforcement learning that uses the reference data and includes the feature quantity parameters as operation targets.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In reinforcement learning (RL), a machine learning technique, a reward function is used to evaluate the value of various actions. Inverse reinforcement learning (IRL) is known as a method for generating this reward function.

[0003] Non-Patent Document 1 describes Maximum Entropy-IRL (ME-IRL), which is one type of inverse reinforcement learning. In ME-IRL, the maximum entropy principle is used to specify the distribution of trajectories, and the reward function is learned by approximating the true distribution (i.e., maximum likelihood estimation).

[0004] Furthermore, Non-Patent Document 2 describes GCL (Guided Cost Learning), which is one of the inverse reinforcement learning methods that is an improvement over maximum entropy inverse reinforcement learning. The method described in Non-Patent Document 2 updates the weights of the reward function using importance sampling. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] BD Ziebart, A. Maas, JA Bagnell, and AK Dey, "Maximum entropy inverse reinforcement learning", In AAAI, AAAI'08, 2008. [Non-patent document 2] Chelsea Finn, Sergey Levine, Pieter Abbeel, "Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization", Proceedings of The 33rd International Conference on Machine Learning, PMLR 48, pp.49-58, 2016. Summary of the Invention [Problem to be solved by the invention]

[0006] However, both of the techniques described in Non-Patent Document 1 and Non-Patent Document 2 have room for improvement in terms of generating an appropriate reward function.

[0007] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of a purpose thereof is to provide a technology that can generate a more appropriate reward function. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present invention includes an acquisition means for acquiring reference data, and a determination means for determining a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target.

[0009] An information processing device according to one aspect of the present invention includes: an acquisition means for acquiring target data; a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in an operation target; and a generation means for generating output data corresponding to the target data by solving an optimization problem using the target data acquired by the acquisition means.

[0010] An information processing method according to one aspect of the present invention is an information processing method by an information processing device, and includes acquiring reference data and determining a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target.

[0011] An information processing method according to one aspect of the present invention is an information processing method by an information processing device, which includes acquiring target data; and generating output data corresponding to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning including the feature parameter as an operation target, and the target data acquired in the acquiring step.

[0012] A program according to one aspect of the present invention is a program that causes a computer to function as an information processing device, and functions as an acquisition means for acquiring reference data, and a determination means for determining a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using the reference data, in which the feature parameter is included in an operation target.

[0013] A program according to one aspect of the present invention causes a computer to function as an information processing device, and functions as: an acquisition means for acquiring target data; and a generation means for generating output data corresponding to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in an operation target; and the target data acquired by the acquisition means. [Effects of the Invention]

[0014] According to one aspect of the present invention, a more appropriate reward function can be generated. [Brief explanation of the drawings]

[0015] [Figure 1]1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Figure 2] 1 is a flowchart showing the flow of an information processing method according to a first exemplary embodiment of the present invention. [Figure 3] 1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Figure 4] 1 is a flowchart showing the flow of an information processing method according to a first exemplary embodiment of the present invention. [Figure 5] FIG. 10 is a block diagram showing the configuration of an information processing device according to a second exemplary embodiment of the present invention. [Figure 6] FIG. 10 is a flowchart showing the flow of an information processing method according to a second exemplary embodiment of the present invention. [Figure 7] 10A and 10B are diagrams illustrating an example of a display generated by a display control unit in the second exemplary embodiment of the present invention. [Figure 8] FIG. 10 is a block diagram showing the configuration of an information processing device according to a third exemplary embodiment of the present invention. [Figure 9] 10A and 10B are diagrams illustrating examples of displays generated by a display control unit in the third exemplary embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing a second display example of the information processing device according to the third exemplary embodiment of the present invention. [Figure 11] FIG. 10 is a diagram illustrating an application example of an information processing device according to a third exemplary embodiment of the present invention. [Figure 12] FIG. 1 is a diagram illustrating an example of a computer that realizes an information processing device according to each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Exemplary Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0017] (Overview of information processing device 1) The information processing device 1 according to this exemplary embodiment is a device that determines a reward function including weight coefficients and feature amount parameters by inverse reinforcement learning using reference data.

[0018] Here, the reference data refers to data referenced by inverse reinforcement learning, and includes, as an example, a set of state data and action data. For example, the reference data may include state data that represents the state of a certain system, and action data that represents the action taken by a specific expert in that state. As an example, the reference data τ may be defined as {τ1, τ2, . . . τ N}(where τ N =(s1, a1), (s2, a2), ...(s N , a N )), where N is any natural number. Also, s i (i=1 to N) represents the state data indicating the state of the above system, and a i (i=1 to N) represents the behavioral data selected in the state indicated by the state data. Thus, the reference data may include, for example, one or more pairs of state data and behavioral data. In addition, some or all of the data included in the reference data are also called explanatory variables, which are arguments of the reward function.

[0019] The behavioral data is not limited to data showing the behavior of a specific expert, and the behavioral data a i The data may be data indicating an action taken by an entity that performs an action associated with the action, for example, data indicating an action taken by a robot.

[0020] In the following description, unless there is any confusion, state data may be simply referred to as state, and action data may be simply referred to as action.

[0021] In this exemplary embodiment, inverse reinforcement learning refers to learning for determining a reward function. In the inverse reinforcement learning according to this exemplary embodiment, the reward function is determined by referencing reference data and updating feature parameters included in the reward function as the operation target. In addition, in the inverse reinforcement learning according to this exemplary embodiment, the reference data may be referenced and the weight coefficient included in the reward function as the operation target.

[0022] Here, the reward function is, for example, a function for evaluating the value of each of various actions. The reward function includes, as parameters, a weight coefficient and a feature parameter. For example, the weight coefficient is a weight multiplied by each of one or more feature parameters included in the reward function. For example, the feature parameter is a parameter that characterizes one or more feature parameters included in the reward function.

[0023] A simple example of the reward function Reward is as follows:

number

[0024]

number

[0025]

number

[0026] Furthermore, in this exemplary embodiment, the reward function is the inverse of the cost function, as shown in Equation 1. Therefore, there is a relationship in which the smaller the cost, the larger the reward.

[0027] Furthermore, as shown in Equation 2, the cost function includes one or more cost terms each including a feature expressed using an explanatory variable and a weighting coefficient representing the weight of the feature, and at least one of the one or more cost terms includes the feature parameter that characterizes the cost term together with the explanatory variable.

[0028] Also, in Eq.

number

[0029]

number

[0030] In the above reward function, the information processing device 1 uses the reference data τ to calculate, for example, the feature parameter

number

[0031] (Configuration of information processing device 1) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this exemplary embodiment.

[0032] 1, the information processing device 1 includes an acquisition unit 11 and a determination unit 12. In this exemplary embodiment, the acquisition unit 11 and the determination unit 12 are components that respectively realize an acquisition means and a determination means.

[0033] The acquisition unit 11 acquires reference data and supplies the acquired reference data to the determination unit 12.

[0034] The determination unit 12 determines a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using reference data, in which the feature parameter is included in the operation target.

[0035] As described above, the information processing device 1 according to this exemplary embodiment includes an acquisition unit 11 that acquires reference data and a determination unit 12 that determines a reward function including weight coefficients and feature parameters by inverse reinforcement learning using the reference data, in which the feature parameters are included in the operation target. Therefore, according to the information processing device 1 according to this exemplary embodiment, since the operation target includes feature parameters for determining the feature, it is possible to use the results of a prediction model, etc., as the feature. Therefore, according to the information processing device 1 according to this exemplary embodiment, it is possible to generate a more appropriate reward function.

[0036] (Flow of information processing method S1) The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1 according to this exemplary embodiment.

[0037] (Step S11) In step S11, the acquisition unit 11 acquires reference data and supplies the acquired reference data to the determination unit 12.

[0038] (Step S12) In step S12, the determination unit 12 determines a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using the reference data supplied from the acquisition unit 11, in which the feature parameter is included in the operation target.

[0039] As described above, in the information processing method S1 according to this exemplary embodiment, in step S11, the acquisition unit 11 acquires reference data, and in step S12, the determination unit 12 determines a reward function including weight coefficients and feature parameters by inverse reinforcement learning using the reference data supplied from the acquisition unit 11 and including the feature parameters as operation targets. Therefore, the information processing method S1 according to this exemplary embodiment achieves the same effects as the information processing device 1.

[0040] (Overview of information processing device 2) The information processing device 2 according to this exemplary embodiment is a device that generates output data corresponding to target data by solving an optimization problem using the target data and a reward function determined by inverse reinforcement learning. Here, the reward function and inverse reinforcement learning are as described above.

[0041] In addition, in this exemplary embodiment, the target data includes at least a portion of state data that indicates the state of a certain system, and action data that indicates the action taken by a specific expert in that state.

[0042] Here, solving an optimization problem refers to maximizing a reward function by using target data as input and manipulating target data.

[0043] As an example, the target data TD is expressed as {s1, s2, . . . s N}, and the data MD to be operated on can be represented by {a1, a2, a N}, where s i (i=1~N) represents the state data that indicates the state of the system, and a i(i=1 to N) represent selectable behavior data in the state indicated by the state data TD. In this example, the information processing device 2 according to this exemplary embodiment maximizes a reward function having the target data TD and the operation target data MD as explanatory variables by manipulating the operation target data MD. In other words, the information processing device 2 solves an optimization problem using the target data as input, and generates operation target data that maximizes the reward function as output data.

[0044] (Overview of information processing device 2) The configuration of the information processing device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 2 according to this exemplary embodiment.

[0045] 3, the information processing device 2 includes an acquisition unit 11 and a generation unit 22. In this exemplary embodiment, the acquisition unit 11 and the generation unit 22 are configured to respectively realize an acquisition means and a generation means.

[0046] The acquisition unit 11 acquires target data. The acquisition unit 11 supplies the acquired target data to the generation unit 22.

[0047] The generation unit 22 generates output data according to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning that includes the feature parameter as an operation target, and the target data acquired by the acquisition unit 11.

[0048] Here, as an example of the reward function determined by inverse reinforcement learning, the reward function determined by the determination unit 12 included in the information processing device 1 described above can be used.

[0049] As described above, the information processing device 2 according to this exemplary embodiment employs a configuration including an acquisition unit 11 that acquires target data, a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in the operation target, and a generation unit 22 that generates output data corresponding to the target data by solving an optimization problem using the target data acquired by the acquisition unit 11.

[0050] Therefore, according to the information processing device 2 of this exemplary embodiment, the optimization problem is solved using a reward function determined by inverse reinforcement learning that includes feature parameters as operation targets, and therefore output data that maximizes a more appropriate reward function can be generated.

[0051] (Flow of information processing method S2) The flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S2 according to this exemplary embodiment.

[0052] (Step S21) In step S21, the acquisition unit 11 acquires target data.

[0053] (Step S22) In step S22, the generation unit 22 generates output data corresponding to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in the operation target, and the target data acquired by the acquisition unit 11.

[0054] Here, as an example of the reward function determined by inverse reinforcement learning, the reward function determined in step S12 included in the above-described information processing method S1 can be used.

[0055] As described above, in the information processing method S2 according to this exemplary embodiment, in step S21, the acquisition unit 11 acquires target data, and in step S22, the generation unit 22 generates output data according to the target data by solving an optimization problem using a reward function that includes a weighting coefficient and a feature parameter and that is determined by inverse reinforcement learning that includes the feature parameter as an operation target, and the target data acquired by the acquisition unit 11. Therefore, the information processing method S2 according to this exemplary embodiment achieves the same effects as the information processing device 2.

[0056] Exemplary Embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and their description will be omitted as appropriate.

[0057] (Overview of information processing device 3) The information processing device 3 according to this exemplary embodiment is a device that determines a reward function including a weighting factor WF and a feature parameter FP by inverse reinforcement learning using reference data RD. The information processing device 3 also displays information corresponding to at least one of the determined weighting factor WF, feature parameter FP, and reward function.

[0058] The reference data, inverse reinforcement learning, reward function, weighting coefficients, and feature parameters are as described above.

[0059] (Configuration of information processing device 3) The configuration of the information processing device 3 according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 3 according to this exemplary embodiment.

[0060] As shown in FIG. 5, the information processing device 3 includes a storage unit 31, an input unit 32, an output unit 33, a communication unit , and a control unit .

[0061] The storage unit 31 is a memory that stores various data referenced by the control unit 35, which will be described later. Examples of data stored in the storage unit 31 include reference data RD, weighting coefficients WF, and feature parameters FP. An example of the reference data RD may be stored decision-making history data (trajectory) of an expert received by the input unit 32, which will be described later. The storage unit 31 may also store feature candidates for the reward function used by the determination unit 12 for learning. However, the feature candidates do not necessarily have to be feature values ​​used in the reward function.

[0062] The storage unit 31 may also store a mathematical optimization solver for implementing the processing by the determination unit 12. The content of the mathematical optimization solver is arbitrary and may be determined depending on the environment or device on which it is executed.

[0063] The input unit 32 accepts various types of data to be input to the information processing device 3. For example, the input unit 32 may accept input of the decision-making history data (specifically, pairs of states and actions) of the expert described above. The input unit 32 may also accept input of initial states and constraint conditions used when an inverse reinforcement learning device (described later) performs inverse reinforcement learning.

[0064] The input unit 32 is configured to include input devices such as a keyboard, a mouse, and a touch panel, for example. The input unit 32 may also function as an interface that acquires data from other connected devices. In this configuration, the input unit 32 supplies the data acquired from the other devices to the control unit 35, which will be described later.

[0065] The output unit 33 is configured to output the calculation results of the information processing device 3. As an example, the output unit 33 includes a display panel (display unit) and displays the calculation results on the display panel. The output unit 33 may also function as an interface that outputs data to other connected devices. In this configuration, the output unit 33 outputs data supplied from the control unit 35, which will be described later, to other connected devices.

[0066] The communication unit 34 is a communication module that communicates with other devices via a network (not shown). For example, the communication unit 34 outputs data supplied from a control unit 35 (described later) to other devices via the network, and acquires data output from other devices via the network and supplies the data to the control unit 35.

[0067] The specific configuration of the network does not limit this embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0068] In this exemplary embodiment, the calculation results are displayed via at least one of the output unit 33 and the communication unit 34.

[0069] (Control unit 35) The control unit 35 controls each unit included in the information processing device 3. As an example, the control unit 35 stores data acquired from the input unit 32 or the communication unit 34 in the storage unit 31, and supplies data stored in the storage unit 31 to the output unit 33 or the communication unit 34.

[0070] 5, the control unit 35 also functions as an acquisition unit 11, a determination unit 12, and a display control unit 13. The acquisition unit 11, the determination unit 12, and the display control unit 13 are configured to realize an acquisition means, a determination means, and a first display means, respectively, in this exemplary embodiment.

[0071] The acquiring unit 11 acquires the reference data RD via the input unit 32 or the communication unit 34. The acquiring unit 11 stores the acquired reference data RD in the storage unit 31.

[0072] The determination unit 12 acquires the reference data RD stored in the storage unit 31 and determines a reward function including the weighting factor WF and the feature parameter FP by inverse reinforcement learning using the reference data RD, in which the feature parameter FP is included in the operation target. The determination unit 12 stores the determined feature parameter FP in the storage unit 31.

[0073] Furthermore, the weighting factor WF included in at least one of one or more cost terms may be included in the manipulation target in the inverse reinforcement learning by the determination unit 12. The determination unit 12 stores the manipulated weighting factor WF in the storage unit 31.

[0074] An example of the process executed by the determination unit 12 will be described later.

[0075] The display control unit 13 displays, via the output unit 33, information corresponding to at least one of the weighting factor WF, the feature parameter FP, and the reward function.

[0076] An example of the processing executed by the display control unit 13 will be described later.

[0077] <Explanation of problem setting and methodology> To facilitate understanding, the problem setting and the method of maximum entropy inverse reinforcement learning (ME-IRL) according to this exemplary embodiment will be described below. In maximum entropy inverse reinforcement learning (ME-IRL), the following problem setting is assumed: N}(where τ i =((s1,a1),(s2,a2),…,(s N ,a N ))) is set to estimate a single reward function R(s, a) = θ f(s, a). ME-IRL can reproduce the decision-making of an expert by estimating θ.

[0078] Here, θ is a weighting coefficient vector whose components are weighting coefficients WF. Also, f(s, a) is a feature vector that may include multiple terms corresponding to each feature. Also, the total number of weighting coefficients WF included in the weighting coefficient vector θ is determined according to the number of components of the feature vector f(s, a).

[0079] Next, the ME-IRL method will be explained. In ME-IRL, the trajectory τ is expressed by the following formula A1, and the distribution of the trajectory p θ The probability model representing (τ) is expressed by the following formula A2. T f τ represents the reward function (see Equation A3), and Z represents the sum of the rewards for all trajectories (see Equation A4).

number

number

number

[0080]

number

number

[0081]

number

[0082] Furthermore, simply updating the weighting coefficient vector θ as in Equation A5 poses a problem in that there is room for improvement in terms of generating an appropriate reward function.

[0083] In the maximum entropy inverse reinforcement learning according to this exemplary embodiment, at least one of the multiple terms in the feature vector f(s, a) includes a feature parameter FP that characterizes the term. Furthermore, in the maximum entropy inverse reinforcement learning according to this exemplary embodiment, not only the above-mentioned θ but also the feature parameter FP is estimated. Therefore, the maximum entropy inverse reinforcement learning according to this exemplary embodiment should be referred to as improved maximum entropy inverse reinforcement learning. However, to avoid complicating the name, hereinafter, this "improved maximum entropy inverse reinforcement learning" will also be simply referred to as "maximum entropy inverse reinforcement learning (ME-IRL)."

[0084] (An example of processing executed by the determination unit 12) Next, an example of the process executed by the determination unit 12 will be described.

[0085] The determination unit 12 sets the feature values ​​of the reward function from reference data including states and actions. As an example, the determination unit 12 may be configured to set the feature values ​​of the reward function so that the tangent gradient is finite across the entire function, so that Wasserstein distance can be used as a distance measure between distributions in the inverse reinforcement learning process. Furthermore, the determination unit 12 may set the feature values ​​of the reward function so that, for example, the Lipschitz continuity condition is satisfied.

[0086] For example, f τ Let be the feature vector of the trajectory τ. The reward function θ T f τ If is linear, then the map F:τ→fτ If is Lipschitz continuous, then θ T f τ is also Lipschitz continuous. Therefore, the determination unit 12 may set the feature amount so that the reward function becomes a linear function.

[0087] For example, Equation 4 shown below has an infinity gradient at a0, and therefore can be said to be an inappropriate reward function for the present disclosure.

[0088]

number

[0089] The determination unit 12 may be configured to initialize the weighting factor WF. The method by which the determination unit 12 initializes the weighting factor WF is not particularly limited, and the weighting factor WF may be initialized based on any method predetermined according to a user or the like.

[0090] Furthermore, the determination unit 12 derives a trajectory τ^ (τ^ is a superscript of τ) that minimizes the distance between the probability distribution of the reference data RD and the probability distribution of the optimal solution determined based on the optimized parameters (of the reward function). Specifically, the determination unit 12 uses the Wasserstein distance as a distance measure between the distributions, and estimates the expert's trajectory τ^ by performing mathematical optimization to minimize the Wasserstein distance.

[0091] The Wasserstein distance is defined by the following formula 5. That is, the Wasserstein distance represents the distance between the probability distribution of the expert's trajectory and the probability distribution of the trajectory determined based on the parameters of the reward function. Note that, due to the constraints of the Wasserstein distance, the reward function θ T f τmust be a function that satisfies the Lipschitz continuity condition. On the other hand, in this exemplary embodiment, the determination unit 12 sets the feature amount of the reward function so as to satisfy the Lipschitz continuity condition, and therefore, it becomes possible to use the Wasserstein distance as exemplified below.

[0092]

number

[0093] Furthermore, the determination unit 12 updates the reward function parameter θ and the feature parameter FP based on the estimated expert's trajectory τ^ so as to maximize the distance measure between the distributions. Here, in maximum entropy inverse reinforcement learning (i.e., ME-IRL), the trajectory τ is assumed to follow a Boltzmann distribution according to the maximum entropy principle. Therefore, similar to ME-IRL, the determination unit 12 updates the reward function parameter θ and the feature parameter FP based on the estimated expert's trajectory τ^ so as to maximize the log-likelihood of the Boltzmann distribution derived from the maximum entropy principle.

[0094] As mentioned above, the second term in Equation A6 is the sum of the rewards for all trajectories. ME-IRL assumes that the value of this second term can be calculated exactly. However, in reality, there is a problem in that it is difficult to calculate the sum of the rewards for all trajectories.

[0095] Therefore, the determination unit 12 sets a lower limit of the log-likelihood expressed using the reward function, and updates the operation target (parameter θ, feature parameter FP) so as to maximize the lower limit of the log-likelihood expressed using the reward function.

number

number

[0096]

number

[0097] Furthermore, the determination unit 12 may use a formula obtained by transforming the derived formula for the lower limit of the logarithmic likelihood of ME-IRL into a formula that subtracts an entropy regularization term from the Wasserstein distance. The formula for the lower limit of the logarithmic likelihood of ME-IRL is decomposed into the Wasserstein distance and the entropy regularization term, as shown in the following formula 7.

[0098]

number

[0099] The second parenthetical expression in Equation 7 represents an entropy regularization term that contributes to increasing the logarithmic likelihood of the Boltzmann distribution derived from the maximum entropy principle. Specifically, in the entropy regularization term exemplified in Equation 7 (i.e., the second parenthetical expression in Equation 7), the first term represents the maximum reward value for the current parameter θ, and the second term represents the average reward value for the current parameter θ.

[0100] In this way, the inverse reinforcement learning by the determination unit 12 includes an update process that updates the operation target (parameter θ, feature parameter FP) so as to maximize the lower limit of the log-likelihood expressed using the reward function. As shown in Equation 7, the lower limit of the log-likelihood in the update process is expressed using the Wasserstein distance, which represents the distance between the reference probability distribution and the probability distribution expressed using the reward function, and a regularization term, which represents the difference between the maximum value of the reward function and the average value of the reward function.

[0101] Here, we explain why the second term in the second bracket in Equation 7 functions as an entropy regularization term. To maximize the lower bound of the log-likelihood of ME-IRL, the value of the second term needs to be small, which corresponds to reducing the difference between the maximum reward value and the average value. A smaller difference between the maximum reward value and the average value indicates a smaller trajectory variability.

[0102] In other words, a smaller difference between the maximum reward value and the average value means an increase in entropy, so entropy regularization works and contributes to maximizing entropy. This contributes to maximizing the log-likelihood of the Boltzmann distribution, and as a result, contributes to resolving uncertainty in inverse reinforcement learning.

[0103] The determination unit 12 updates the parameter θ and the feature parameter FP by the gradient ascent method, for example, by fixing the estimated trajectory τ̂ based on the above-mentioned formula 7. However, there is a risk that the normal gradient ascent method will not converge. In the entropy regularization term, the feature parameter (f τθmax ) is the feature of other trajectories (f τ(n) ) (i.e., the difference between the two is never 0). Therefore, in a normal gradient ascent method, the log-likelihood oscillates and does not converge, making it unstable and difficult to properly determine convergence (see Equation 8 below for updating the parameter θ).

[0104]

number

[0105] Specifically, the determination unit 12 assigns a damping coefficient β t The determination unit 12, for example, differentiates the above-mentioned formula 7 with respect to θ, and defines formula 9 shown below as an example, in which a damping coefficient is set in the part corresponding to the entropy regularization term out of the part corresponding to the term indicating the Wasserstein distance (i.e., the part contributing to the process of increasing the Wasserstein distance) and the part corresponding to the entropy regularization term.

[0106]

number

[0107]

number

[0108] Furthermore, since the Wasserstein distance has a weaker phase than the log-likelihood, which is the KL divergence, it is possible to bring the Wasserstein distance closer to 0 by bringing the log-likelihood closer to 0. Therefore, the determination unit 12 may update the parameter θ and the feature amount parameter FP without attenuating the portion corresponding to the entropy regularization term in the initial stage of the update, and update the parameter θ and the feature amount parameter FP so as to reduce the influence of the portion corresponding to the entropy regularization term when the log-likelihood starts to oscillate.

[0109] Specifically, the determination unit 12 determines the attenuation coefficient β t = 1 to update the parameter θ and the feature parameter FP. Then, the determination unit 12 updates the attenuation coefficient β t = 0, the parameter θ and the feature amount parameter FP may be updated while eliminating the influence of the part corresponding to the entropy regularization term.

[0110] The determination unit 12 may determine that the log likelihood has started to oscillate, for example, when the moving average of the log likelihood has become constant. Specifically, the determination unit 12 may determine that the log likelihood has started to oscillate when the change in the moving average in the time window (from the current value to several points in the past) of the "lower limit of the log likelihood" is small (for example, 1e -3 The moving average may be considered to have stabilized when

[0111] In addition, the determination unit 12 suddenly changes the attenuation coefficient β tInstead of setting β = 0, the oscillation coefficient may be first changed as shown in the above formula 10. After the change, the decision unit 12 changes the damping coefficient β t = 0. The method for determining the timing at which vibration begins is the same as the method described above.

[0112] Furthermore, after changing the oscillation coefficient as shown in the above-described formula 10, the determination unit 12 may change the method of updating the parameter θ and the feature quantity parameter FP at the timing when the log-likelihood starts to oscillate further. Specifically, the determination unit 12 may update the parameter θ and the feature quantity parameter FP using a momentum method as shown in the following formula 11. The values ​​of γ1 and α in formula 11 are determined in advance. For example, γ1 may be set to 0.9 and α to 0.001.

number

[0113]

number

[0114] An example of a process in which the decision unit 12 determines that the lower limit of the log-likelihood has converged is a configuration in which the decision unit 12 determines that the distance measure between distributions has converged when the absolute value of the lower limit of the log-likelihood becomes smaller than a predetermined threshold.

[0115] If the determination unit 12 determines that the distance measure between the distributions has not converged, it continues the trajectory estimation process and the update process of the parameter θ and the feature parameter FP. On the other hand, if the determination unit 12 determines that the distance measure between the distributions has converged, it ends the trajectory estimation process and the update process of the parameter θ and the feature parameter FP.

[0116] <More specific processing examples> The following describes a more specific example of the processing by the determination unit 12. In the following example, a case is taken as an example where the reward function (Reward) and the cost function (Cost) are given by the following Equations 12 and 13.

[0117]

number

[0118]

number

number

[0119] In the above setting, the determination unit 12 derives the lower limit of the log-likelihood shown in the above-mentioned formula 9, and updates the operation target of the reward function. For example, when the reward function is the following formula 12, the determination unit 12 updates the operation target of the reward function by setting the weighting coefficient λ1, the weighting coefficient λ2, and the feature parameter

number

number

number

number

[0120]

number

[0121] In this way, the determination unit 12 updates the reward function parameter θ and the feature parameter FP so as to maximize the log-likelihood of the Boltzmann distribution derived from the principle of maximum entropy. In this updating process, inverse reinforcement learning is performed on not only the weight coefficients but also the feature parameters as update targets, so a more appropriate reward function can be generated.

[0122] (Flow of information processing method S3) Next, the flow of the information processing method S3 according to this exemplary embodiment will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing the flow of the information processing method S3 according to this exemplary embodiment.

[0123] (Step S31) In step S31, the acquisition unit 11 acquires reference data RD via the input unit 32 or the communication unit 34. The acquisition unit 11 stores the acquired reference data RD in the storage unit 31. The reference data RD has been described above, so a description thereof will be omitted here.

[0124] (Step S32) In step S32, the determination unit 12 initializes the weight coefficients and feature parameters, which are the operation targets in inverse reinforcement learning, among the parameters included in the reward function. As an example, the determination unit 12 may initialize the weight coefficients and feature parameters, which are the operation targets in inverse reinforcement learning, using initial values ​​stored in the storage unit 31.

[0125] (Step S33) In step S33, the determination unit 12 performs mathematical optimization to minimize the Wasserstein distance. As an example, the determination unit 12 estimates a trajectory that minimizes the Wasserstein distance, which represents the distance between the probability distribution of the expert's trajectory and the probability distribution of a trajectory determined based on the parameters of the reward function.

[0126] (Step S34) In step S34, the determination unit 12 updates the reward function parameter θ and the feature parameter FP so as to maximize the log likelihood of the Boltzmann distribution derived from the principle of maximum entropy. This update process has been described above with specific examples, so a detailed description will be omitted here.

[0127] (Step S35) In step S35, the decision unit 12 determines whether the lower limit of the log-likelihood has converged. If it is determined that the lower limit of the log-likelihood has converged (YES in S35), the process proceeds to step S36; otherwise (NO in S35), the process returns to step S33.

[0128] (Step S36) In step S35, the determination unit 12 determines whether the lower bound of the log-likelihood has converged. If it is determined that the lower bound of the log-likelihood has converged, in step S36, the determination unit 12 outputs the reward function.

[0129] The parameters (weighting coefficient WF and feature parameter FP) included in the reward function output by the determination unit 12 are stored in the storage unit 31, for example.

[0130] (Display example) Next, a display example by the information processing device 3 according to this exemplary embodiment will be described with reference to FIG. 7. As described above, the output unit 33 may be configured to include a display panel (display unit) and display various information on the display panel. Here, the information displayed on the display panel may include information corresponding to at least any of the weighting factor WF, the feature parameter FP, and the reward function. Here, the display content displayed by the output unit 33 is generated by the display control unit 13, for example.

[0131] FIG. 7 is a diagram showing an example of a display generated by the display control unit 13. As shown in FIG. 7, a display screen may be generated that shows the relationship between the number of steps and the values ​​of at least some of the parameters of the operation target (weighting coefficient WF and feature amount parameter FP). In other words, a display screen may be generated that shows the change in the parameter value of the operation target in accordance with an increase in the number of steps in the update process. In the example shown in FIG. 7, the number of steps, weighting coefficient λ1, and feature amount parameter FP are displayed.

number

[0132] As described above, the information processing device 3 according to this exemplary embodiment can appropriately present to the user whether inverse reinforcement learning is being performed appropriately by displaying information corresponding to at least one of the weighting factor WF, the feature parameter FP, and the reward function.

[0133] Exemplary Embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first and second exemplary embodiments are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0134] (Configuration of information processing device 4) The configuration of an information processing device 4 according to this exemplary embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of an information processing device 3 according to this exemplary embodiment.

[0135] 8, the information processing device 4 includes a control unit 45 instead of the control unit 35 included in the information processing device 3. The control unit 45 includes a generation unit 14 in addition to the components included in the control unit 35.

[0136] 8, the information processing device 4 includes a storage unit 41 instead of the storage unit 31 included in the information processing device 3. In addition to the various pieces of information stored in the storage unit 31, the storage unit 41 stores target data TD.

[0137] Furthermore, the acquisition unit 11 included in the information processing device 4 further acquires target data TD in addition to the various data acquired by the acquisition unit 11 according to exemplary embodiment 2. The acquired target data TD is stored in the storage unit 41 described above, for example.

[0138] Here, in this exemplary embodiment, the target data TD includes at least a portion of state data that indicates the state of a certain system, and action data that indicates the action taken by a specific expert in that state.

[0139] As an example, the target data TD is expressed as {s1, s2, . . . s N}, and the data MD to be operated on can be represented by {a1, a2, a N}, where s i (i=1~N) represents the state data that indicates the state of the system, and a i (i=1 to N) represents selectable action data in the state indicated by the state data TD.

[0140] (Generation unit 14) The generation unit 14 included in the information processing device 4 maximizes a reward function having as explanatory variables the target data TD and the operation target data MD acquired by the acquisition unit 11, by manipulating the operation target data MD. In other words, the information processing device 4 solves an optimization problem using the target data as input, and generates operation target data that maximizes the reward function as output data.

[0141] In other words, the generation unit 14 generates output data according to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in the operation target, and the target data acquired by the acquisition unit 11.

[0142] Here, the reward function determined by the determiner 12 through the process described in the second exemplary embodiment can be used as the reward function.

[0143] As described above, the information processing device 4 according to this exemplary embodiment employs a configuration including an acquisition unit 11 that acquires target data, a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in the operation target, and a generation unit 14 that generates output data corresponding to the target data by solving an optimization problem using the target data acquired by the acquisition unit 11.

[0144] Therefore, according to the information processing device 4 of this exemplary embodiment, the optimization problem is solved using a reward function determined by inverse reinforcement learning that includes feature parameters as operation targets, and therefore output data that maximizes a more appropriate reward function can be generated.

[0145] (Display example 1) Next, a first display example by the information processing device 4 according to this exemplary embodiment will be described with reference to Fig. 9. As described above, the output unit 33 may be configured to include a display panel (display unit) and display various types of information on the display panel. In this exemplary embodiment, the information displayed on the display panel may include at least some of the data included in the output data generated by the generation unit 14.

[0146] 9 is a diagram showing a display example generated by the display control unit 13. In the example shown in FIG. The reward function is given by Equation 12 described in Example 2, Weighting coefficients λ1, λ2, λ3, and feature parameters

number

[0147] As shown in FIG. 9, the display screen generated by the display control unit 13 includes the values ​​of explanatory variables x1 and x2 contained in the target data TD and the value of the data x3 to be operated (i.e., the value recommended to the user) determined by the generation unit 14 based on these values.

[0148] As described above, the information processing device 4 according to this exemplary embodiment can suitably present the solution to the optimization problem to the user by displaying the output data generated by the generating unit 14.

[0149] (Display example 2) Next, a second display example by the information processing device 4 according to this exemplary embodiment will be described with reference to Fig. 10. In this example, the acquisition unit 11 accepts input of at least one of explanatory variables, weighting coefficients, and feature parameters from a user via the input unit 32. Then, as shown in the upper part of Fig. 10, the display control unit 13 may be configured to display at least one of the values ​​of explanatory variables, weighting coefficients, and feature parameters input by the user in a manner that allows comparison with at least one of the values ​​of explanatory variables, weighting coefficients, and feature parameters obtained by inverse reinforcement learning using reference data TD of one or more experts.

[0150] The display control unit 13 may also be configured to generate a GUI (Graphical User Interface) including operation objects that can be operated by the user, and display the GUI on the output unit 33. Such a GUI is shown in the lower left of Fig. 10. By sliding a bar included in the GUI, it is possible to change the value of at least one of the explanatory variables, weighting coefficients, and feature parameters corresponding to the bar.

[0151] Furthermore, the display control unit 13 may rank at least any one of the explanatory variables, the weighting coefficients, and the feature parameters, and display the variables together with the rankings.

[0152] <Application example> An application example of the information processing device 4 according to this exemplary embodiment will be described below with reference to FIG.

[0153] In this application example, the information processing device 4 generates an operation plan related to a water distribution plan for a water infrastructure. The water infrastructure according to this exemplary embodiment includes, for example, a plurality of bases such as reservoirs, distribution reservoirs, water intake facilities, water purification plants, water supply stations, and demand points. The operation plan includes, for example, information indicating the operation patterns of pumps at each base.

[0154] (Acquisition part 11) The acquisition unit 11 acquires the target data TD and the reference data RD. As an example, the acquisition unit 11 acquires the target data TD and the reference data RD from another device via the communication unit 34. As another example, the acquisition unit 11 may acquire the target data TD and the reference data RD input via the input unit 32. As an example, the acquisition unit 11 may acquire the target data TD and the reference data RD by reading the target data TD and the reference data RD from the storage unit 41 or an externally connected storage device. Details of the target data TD and the reference data RD according to this example will be described later.

[0155] (Decision Unit 12) The determination unit 12 determines a reward function to be used in an optimization problem for generating an operation plan OP for a target water distribution plan by inverse reinforcement learning with reference to the reference data RD. As described above, the inverse reinforcement learning of the reward function includes an update process in which the weighting factor WF and the feature parameter FP are manipulated.

[0156] (Generation unit 14) The generation unit 14 generates an operation plan OP for a target water distribution plan by solving an optimization problem using a reward function determined by inverse reinforcement learning using reference data RD for a reference water distribution plan and target data TD acquired by the acquisition unit 11. The process of generating the operation plan OP executed by the generation unit 14 will be described later.

[0157] (Storage unit 41) The storage unit 41 stores the target data TD and reference data RD acquired by the acquisition unit 11. The storage unit 41 also stores the operation plan OP generated by the generation unit 14. The storage unit 41 also stores the reward function and constraints LC determined by the determination unit 12. Here, storing the reward function in the storage unit 41 means that the parameters that determine the reward function are stored in the storage unit 41.

[0158] (Target data TD) The target data TD is data used by the generation unit 14 to generate the operation plan OP. The target data TD includes information indicating the state of the target water infrastructure. As an example, the target data TD includes information regarding at least one of pumps, water distribution networks, water pipelines, and demand points in the target water infrastructure.

[0159] Specifically, the target data TD includes, as an example, at least one of the following data (i) to (x) for the water infrastructure that is the target of the operation plan. However, the data included in the target data TD is not limited to these, and may include other data.

[0160] (i) Power consumption at each location, (ii) demand forecast margin, (iii) reservoir margin, (iv) water distribution loss, (v) number of operating personnel at each location, (vi) electricity charges at each location, (vii) voltage at each location, (viii) water level at each location, (ix) water pressure at each location, (x) water volume at each location.

[0161] (i) The power consumption at each base indicates the power consumption at each base, such as the water purification plant and water supply station. (ii) The demand forecast margin indicates the extent to which supply exceeds demand. (iii) The distribution reservoir margin indicates the extent to which the designed water storage volume at the distribution reservoir exceeds the actual storage volume. (iv) The water distribution loss indicates the extent to which water cannot be distributed to each demand point. (v) The number of operating personnel indicates the number of operating personnel at each base.

[0162] (Reference data RD) The reference data RD is data used by the determination unit 12 when determining the reward function. The reference data RD includes information representing the state of the reference water infrastructure. Here, the reference water infrastructure may be the same as or different from the water infrastructure for which the operation plan is generated. More specifically, the reference data RD includes, for example, information regarding at least one of pumps, a water distribution network, water pipelines, and demand points in the reference water infrastructure. Furthermore, the reference data RD includes, for example, information regarding at least one of pump operation patterns and personnel in the reference water infrastructure. Each item included in the reference data RD may be treated as state data or as behavior data.

[0163] Specifically, the reference data RD includes, as an example, at least one of the following data (i) to (x) in the reference water infrastructure. However, the data included in the reference data RD is not limited to these, and may include other data.

[0164] (i) Power consumption at each location, (ii) demand forecast margin, (iii) reservoir margin, (iv) water distribution loss, (v) number of operating personnel at each location, (vi) electricity charges at each location, (vii) voltage at each location, (viii) water level at each location, (ix) water pressure at each location, (x) water volume at each location.

[0165] Furthermore, the reference data RD includes, for example, data indicating an operation plan prepared by an expert for the reference water infrastructure. More specifically, the reference data RD includes, for example, data represented by variables controlled based on operation rules, such as valve opening / closing, water intake, and pump thresholds. Such data can also be said to represent the decision-making history (the expert's intentions) of the expert or other expert who prepared the reference operation plan.

[0166] (Operational Plan OP) The operational plan OP includes, for example, information about the operation patterns of pumps in the target water infrastructure, and also includes, for example, information about personnel involved in the target water infrastructure.

[0167] (reward function) The reward function includes cost terms that include variables corresponding to the items included in the reference data RD. The general theory of reward functions is as described in the above exemplary embodiment.

[0168] (Constraint condition LC) The constraints LC are constraints on the optimization problem to be solved by the generation unit 14. The constraints LC include, for example, the following (i) to (iv). Note that the constraints LC are not limited to these and may include other conditions.

[0169] (i) The water volume of the reservoir / distribution reservoir is greater than or equal to threshold value X and less than threshold value Y.

[0170] (ii) Supply exceeds demand by at least X%.

[0171] (iii) Water is being distributed to all demand points.

[0172] (iv) Do not use routes that are under construction.

[0173] <Processing Executed by the Determining Unit 12> The determination unit 12 determines a reward function to be used for an optimization problem for generating an operation plan for a target water distribution plan by inverse reinforcement learning with reference to the reference data RD. As an example, the determination unit 12 determines a weight coefficient for a cost term included in the reward function and a feature parameter that characterizes the cost term by inverse reinforcement learning using the state data and behavior data included in the reference data RD. An example of inverse reinforcement learning by the determination unit 12 is as described above.

[0174] Furthermore, the determination unit 12 outputs the determined reward function. The determination unit 12 may output the reward function by writing it to the storage unit 41 or an external storage device, or may output it to the output unit 33.

[0175] <Processing Executed by the Generation Unit 14> The generation unit 14 generates an operation plan OP for a target water distribution plan by solving an optimization problem using a reward function and target data TD under constraint conditions LC. In this exemplary embodiment, the generation unit 14 generates an operation plan OP for a target water distribution plan by solving an optimization problem using a reward function, in which the target data TD acquired by the acquisition unit 11 is used as a fixed variable and variables other than the fixed variables among the variables included in each cost term in the reward function are used as manipulated variables.

[0176] The generation unit 14 also outputs the generated operation plan OP. The generation unit 14 may output the operation plan OP by writing it to the storage unit 41 or an external storage device, or may output it to the output unit 33.

[0177] <Optimization problem settings> FIG. 11 is a diagram illustrating a specific example of setting an optimization problem according to this example. The operation plan OP must be determined taking into consideration various perspectives, such as how much margin to leave in the forecast demand, how much to reduce power consumption, and how much to consider the water level of the water reservoir. Setting the weighting of these perspectives is difficult because the degree of importance given to each perspective varies depending on the operator of the water infrastructure, and is not uniformly determined. For example, there may be cases where local government A, which is the generator of a certain operation plan, places emphasis on the perspective of power consumption, while local government B places emphasis on the water level of the water reservoir.

[0178] In this exemplary embodiment, the generator 14 solves an optimization problem using the target data TD and a reward function in which the weight coefficients and feature parameters of each cost term are determined by inverse reinforcement learning with reference to the reference data RD under the constraints LC. Here, the weight coefficients and feature parameters of each cost term included in the reward function are determined by inverse reinforcement learning with reference to the reference data RD, and therefore are values ​​that reflect the behavioral data included in the reference data RD, i.e., values ​​that reflect the intentions of the expert or other person who generated the reference operation plan. By solving the optimization problem using a reward function including such weight coefficients and feature parameters, it is possible to generate an operation plan that reflects the intentions of the expert or other person who generated the reference operation plan.

[0179] For example, in the example of Figure 11, the weight coefficients α1 to α6 and feature parameters included in the reward function used to generate the operation plan OP of municipality A are values ​​that reflect the intentions of the experts who generated the reference operation plan used to determine the reward function. Also, the weight coefficients α1 to α6 and feature parameters included in the reward function used to generate the operation plan OP of municipality B are values ​​that reflect the intentions of the experts who generated the reference operation plan used to determine the reward function. By comparing the weight coefficients and feature parameters of municipality A with the weight coefficients and feature parameters of municipality B, it becomes easier to understand what perspectives each municipality emphasizes.

[0180] Also, for example, the determination unit 12 can determine a reward function by referring to reference data RD including an operation plan created by an expert a1 in the local government A, and the generation unit 14 can generate a future operation plan OP using the reward function determined by the determination unit 12 and the target data TD of the local government A. In this case, the generation unit 14 can generate a future operation plan OP for the local government A that reflects the intentions of the expert a1.

[0181] Furthermore, according to this exemplary embodiment, the intentions of the creator of an operation plan for one municipality can be reflected in the operation plans of other municipalities. For example, the determiner 12 can determine a reward function by referring to reference data RD including an operation plan created by an expert a1 in municipality A, and the generator 14 can generate a future operation plan OP using the reward function determined by the determiner 12 and the target data TD of municipality B. In this case, the generator 14 can generate an operation plan OP for municipality B that reflects the intentions of the expert a1.

[0182] [Software implementation example] Some or all of the functions of the information processing devices 1, 2, 3, and 4 may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0183] In the latter case, the information processing devices 1, 2, 3, and 4 are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 12. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1, 2, 3, and 4. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1, 2, 3, and 4.

[0184] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0185] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0186] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0187] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0188] [Appendix 2] Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the following described aspects.

[0189] (Appendix 1) An information processing device comprising: an acquisition means for acquiring reference data; and a determination means for determining a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target.

[0190] (Appendix 2) The information processing device described in Appendix 1, wherein the reward function includes one or more cost terms including a feature expressed using an explanatory variable and the weight coefficient representing a weight of the feature, and at least one of the one or more cost terms includes the feature parameter that characterizes the cost term together with the explanatory variable.

[0191] (Appendix 3) 3. The information processing device according to claim 2, wherein the weight coefficient included in at least one of the one or more cost terms is included in an operation target in the inverse reinforcement learning by the determining means.

[0192] (Appendix 4) An information processing device described in any of Appendices 1 to 3, wherein the inverse reinforcement learning by the determination means includes an update process that updates the operation target so as to maximize a lower limit of the logarithmic likelihood expressed using the reward function.

[0193] (Appendix 5) 5. The information processing device according to claim 4, wherein the lower limit of the logarithmic likelihood is expressed using a Wasserstein distance that represents the distance between a reference probability distribution and a probability distribution expressed using the reward function, and a regularization term that represents the difference between the maximum value of the reward function and the average value of the reward function.

[0194] (Appendix 6) The information processing device according to claim 5, wherein the lower limit of the logarithmic likelihood is a decay coefficient multiplied by the regularization term, and includes a decay coefficient that decays the contribution of the regularization term the more the update process is repeated.

[0195] (Appendix 7) 7. The information processing device according to any one of appendices 1 to 6, further comprising a first display means for displaying information corresponding to at least one of the weight coefficients, the feature parameters, and the reward function.

[0196] (Appendix 8) The information processing device of any of Appendices 1 to 7, wherein the acquisition means further acquires target data, and the information processing device further comprises a generation means for generating output data corresponding to the target data by solving an optimization problem using the reward function determined by the determination means and the target data acquired by the acquisition means.

[0197] (Appendix 9) 9. The information processing device according to claim 8, further comprising a second display means for displaying the output data.

[0198] (Appendix 10) An information processing device comprising: an acquisition means for acquiring target data; and a generation means for generating output data corresponding to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in an operation target; and the target data acquired by the acquisition means.

[0199] (Appendix 11) An information processing method by an information processing device, comprising: acquiring reference data; and determining a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target.

[0200] (Appendix 12) An information processing method by an information processing device, comprising: acquiring target data; and generating output data corresponding to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning including the feature parameter as an operation target, and the target data acquired in the acquiring step.

[0201] (Appendix 13) A program that causes a computer to function as an information processing device, the program functioning as: an acquisition means that acquires reference data; and a determination means that determines a reward function including a weighting coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target.

[0202] (Appendix 14) A program that causes a computer to function as an information processing device, the program functioning as: an acquisition means that acquires target data; and a generation means that generates output data corresponding to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning that includes the feature parameter as an operation target, and the target data acquired by the acquisition means.

[0203] [Appendix 3] Some or all of the above-described embodiments can also be expressed as follows.

[0204] An information processing device comprising at least one processor, the processor executing an acquisition process of acquiring reference data, and a determination process of determining a reward function including a weight coefficient and a feature parameter by inverse reinforcement learning using the reference data, the reward function including the feature parameter as an operation target.

[0205] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process and the determination process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0206] An information processing device comprising at least one processor that executes an acquisition process to acquire target data, and a generation process to generate output data corresponding to the target data by solving an optimization problem using a reward function including a weighting coefficient and a feature parameter, the reward function being determined by inverse reinforcement learning in which the feature parameter is included in an operation target, and the target data acquired in the acquisition process.

[0207] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process and the generation process. The program may also be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0208] 1, 2, 3, 4... Information processing device 11 ... Acquisition unit (acquisition means) 12. Determination unit (determination means) 13 Display control unit (display means) 22,14...Generation unit (generation means)

Claims

1. An acquisition means for acquiring reference data; a determination means for determining a reward function including a weight coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target; Equipped with the inverse reinforcement learning by the determination means includes an update process of updating the operation target so as to maximize a lower limit of a logarithmic likelihood expressed using the reward function; The lower limit of the log-likelihood is expressed using a Wasserstein distance that represents the distance between a reference probability distribution and a probability distribution expressed using the reward function, and a regularization term that represents the difference between the maximum value of the reward function and the average value of the reward function. Information processing device.

2. An acquisition means for acquiring reference data; a determination means for determining a reward function including a weight coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target; Equipped with the reward function includes one or more cost terms including a feature expressed using an explanatory variable and the weighting coefficient representing a weight of the feature; at least one of the one or more cost terms includes the explanatory variable and the feature parameter that characterizes the cost term; the inverse reinforcement learning by the determination means includes an update process of updating the operation target so as to maximize a lower limit of a logarithmic likelihood expressed using the reward function; The lower limit of the log-likelihood is expressed using a Wasserstein distance that represents the distance between a reference probability distribution and a probability distribution expressed using the reward function, and a regularization term that represents the difference between the maximum value of the reward function and the average value of the reward function. Information processing device.

3. the lower limit of the logarithmic likelihood is a decay coefficient by which the regularization term is multiplied, and includes a decay coefficient that decays the contribution of the regularization term as the update process is repeated.

3. The information processing device according to claim 1 or 2.

4. An information processing method by an information processing device, Obtain reference data, determining a reward function including a weight coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target; the inverse reinforcement learning includes an update process of updating the operation target so as to maximize a lower bound of a logarithmic likelihood expressed using the reward function; The lower limit of the log-likelihood is expressed using a Wasserstein distance that represents the distance between a reference probability distribution and a probability distribution expressed using the reward function, and a regularization term that represents the difference between the maximum value of the reward function and the average value of the reward function. Information processing methods.

5. An information processing method by an information processing device, Obtain reference data, determining a reward function including a weight coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target; the reward function includes one or more cost terms including a feature expressed using an explanatory variable and the weighting coefficient representing a weight of the feature; at least one of the one or more cost terms includes the explanatory variable and the feature parameter that characterizes the cost term; the inverse reinforcement learning includes an update process of updating the operation target so as to maximize a lower bound of a logarithmic likelihood expressed using the reward function; The lower limit of the log-likelihood is expressed using a Wasserstein distance that represents the distance between a reference probability distribution and a probability distribution expressed using the reward function, and a regularization term that represents the difference between the maximum value of the reward function and the average value of the reward function. Information processing methods.

6. A program that causes a computer to function as an information processing device, An acquisition means for acquiring reference data; a determination means for determining a reward function including a weight coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target; It functions as the inverse reinforcement learning by the determination means includes an update process of updating the operation target so as to maximize a lower limit of a logarithmic likelihood expressed using the reward function; The lower limit of the log-likelihood is expressed using a Wasserstein distance that represents the distance between a reference probability distribution and a probability distribution expressed using the reward function, and a regularization term that represents the difference between the maximum value of the reward function and the average value of the reward function. program.

7. A program that causes a computer to function as an information processing device, An acquisition means for acquiring reference data; a determination means for determining a reward function including a weight coefficient and a feature parameter by inverse reinforcement learning using the reference data, the inverse reinforcement learning including the feature parameter as an operation target; It functions as the reward function includes one or more cost terms including a feature expressed using an explanatory variable and the weighting coefficient representing a weight of the feature; at least one of the one or more cost terms includes the explanatory variable and the feature parameter that characterizes the cost term; the inverse reinforcement learning by the determination means includes an update process of updating the operation target so as to maximize a lower limit of a logarithmic likelihood expressed using the reward function; The lower limit of the log-likelihood is expressed using a Wasserstein distance that represents the distance between a reference probability distribution and a probability distribution expressed using the reward function, and a regularization term that represents the difference between the maximum value of the reward function and the average value of the reward function. program.

Images