Information processing device, information processing method, and program

Abstract

This information processing device that performs constrained reinforcement learning comprises: an input unit that inputs the state and an action of an agent; a reward calculation unit that calculates a reward on the basis of the state and the action of the agent; a determination unit that determines, on the basis of the state and the action of the agent, whether or not a constraint condition is satisfied; and a learning unit that learns a plan so as to maximize the reward under a condition in which the probability of violating the constraint condition is less than or equal to a prescribed allowable probability.

Description

Information processing device, information processing method, and program

[0001] This disclosure relates to constrained reinforcement learning.

[0002] Constrained reinforcement learning is a method of reinforcement learning that learns strategies to maximize rewards while imposing behavioral constraints. It is useful in ensuring safety and generating diverse behaviors. However, it has been pointed out that actually designing reward and cost functions is difficult, and as a solution, methods that learn reward and cost functions from human feedback (user feedback) are considered promising (Non-Patent Document 1, etc.).

[0003] Reinforcement learning based on user feedback suffers from the problem of time-consuming and laborious labeling. Existing research has focused on how constraints are learned from what kind of feedback, and how environment exploration is performed while satisfying constraints using feedback, but has not attempted to reduce the cost of labeling.

[0004] Furthermore, in constrained reinforcement learning, it is common practice to learn a policy under the condition that no constraint violations occur. However, there are cases where constraint violations are acceptable if they occur below a certain probability. Constrained learning under such conditions has not been performed conventionally.

[0005] Glazier, Arie, et al. "Making human-like trade-offs in constrained environments by learning from demonstrations." arXiv preprint arXiv:2109.11018 (2021).<URL: https: / / arxiv.org / abs / 2109.11018>

[0006] This disclosure aims to provide a technology that enables constrained reinforcement learning based on user feedback more favorably than conventional methods.

[0007] One aspect of the present disclosure is an information processing device for performing constrained reinforcement learning, comprising: an input unit for inputting the state and actions of an agent; a reward calculation unit for calculating a reward based on the state and actions of the agent; a determination unit for determining whether or not constraint conditions are satisfied based on the state and actions of the agent; and a learning unit for learning a policy to maximize the reward under the condition that the probability of violating the constraint conditions is less than or equal to a predetermined allowable probability.

[0008] According to this embodiment, the probability of a constraint violation occurring becomes less than or equal to the acceptable probability, and it becomes possible to learn a policy that maximizes the reward.

[0009] One aspect of the present disclosure is an information processing device comprising: a trajectory data acquisition unit that acquires a plurality of trajectory data representing the state and actions of an agent; a feedback acquisition unit that acquires user feedback for at least a portion of the plurality of trajectory data; an estimation unit that includes a machine learning model for estimating the cost of the trajectory data and estimates whether or not the trajectory data satisfies constraints; a loss calculation unit that calculates a loss based on the estimation result by the estimation unit and the user feedback; and a learning unit that learns the machine learning model based on the loss.

[0010] Thus, in this embodiment, we focus on the fact that constraint learning becomes a two-class classification problem based on user feedback, and by training a classifier, we can obtain an estimate of the cost function.

[0011] In one embodiment, the machine learning model may be configured to estimate multiple costs using multiple cost functions. The loss calculation unit may also be configured to calculate the cost based on the softmax of the multiple costs and the loss function of the user feedback if the user feedback indicates that the constraints are not satisfied, and to calculate the cost based on multiple loss functions of each of the multiple costs and the user feedback otherwise.

[0012] In one aspect, the learning unit learns the machine learning model by learning with a weak teacher. By using learning with a weak teacher, constraints can be learned at a low feedback cost.

[0013] In one aspect, the information processing apparatus includes a machine learning model learned by the learning unit, and includes a constraint satisfaction determination unit that determines whether or not the trajectory of the agent's action satisfies a constraint, a reward calculation unit that calculates a reward for the agent's action, and a reinforcement learning unit that learns the policy of the agent so as to maximize the reward while satisfying the constraint. By performing reinforcement learning using the constraints obtained by learning in this way, an agent can be made to learn appropriate actions that satisfy the constraint conditions.

[0014] The present disclosure can be regarded as an information processing apparatus including at least a part of the above means. The present disclosure can also be regarded as a method including at least a part of the above processing, or a program for realizing such a method and a recording medium on which the program is non-temporarily recorded. Further, the present disclosure includes an apparatus or method in which each of the above means and processing is combined with each other as much as possible.

[0015] According to the present disclosure, constrained reinforcement learning can be more suitably performed based on user feedback.

[0016] FIG. showing a configuration example of a constraint learning apparatus according to the present disclosure. A flowchart showing the flow of constraint learning processing in the present disclosure. FIG. showing a configuration example of a constraint satisfaction discriminator in the present disclosure. FIG. schematically explaining reinforcement learning in the present disclosure. FIG. showing a configuration example of a reinforcement learning apparatus according to the present disclosure. Pseudo code showing a reinforcement learning algorithm according to the present disclosure. FIG. showing the hardware configuration of a constraint learning apparatus and a reinforcement learning apparatus according to the present disclosure.

[0017] <Overview> In the finite-time MDP (Markov Decision Process) with constraints according to the present embodiment, the following problem is solved. Specifically, the cumulative expected reward sum v １ π (P １ ) when following the policy π is maximized, and the policy π is a cost function c for each constraint ｎThe sum exceeds the threshold θ ｎ under the condition that the probability is δ or less.

[0018] c ｎ is a function called the cost function, which is a mapping from the state-action pair space S×A to [-1,1]. Here, the subscript n indicates the n-th constraint. Also, let δ∈[0,1) represent the allowable failure probability, which is given.

[0019] It is difficult to actually design the constraints or the cost function. For example, specialized knowledge is required to represent the constraints as code, and it is difficult for general users to change the behavior of the agent according to their desires. Also, there are constraints that are difficult to represent as code in the first place. For example, the constraint of "behaving in a human-like manner" cannot be represented as code. Furthermore, it is difficult to pre-grasp the actions that the agent should not perform, and it may be noticed that a certain action should be made a constraint violation after observing the actual action. For such problems where it is difficult to design the constraints or the cost function, a method of learning the constraints based on user feedback has been proposed.

[0020] Also, the conventional method can only solve the case where the allowable constraint violation probability δ is zero. By setting δ>0, the optimal policy can be obtained under the condition of allowing constraint violations within the range of probability δ.

[0021] <Constraint Learning>. In this aspect, a method of learning constraints based on user feedback is proposed by reducing the labor of user feedback or labeling of learning data.

[0022] FIG. 1 is a block diagram showing the functional configuration of the constraint learning device 100. As shown in FIG. 1, the constraint learning device 100 includes a trajectory data acquisition unit 101, a trajectory data presentation unit 102, a feedback acquisition unit 103, a constraint satisfaction discriminator 104, a loss calculation unit 106, and a learning unit 107. The constraint satisfaction discriminator 104 includes a cost estimation model 105 for estimating the cost.

[0023] FIG. 2 is a flowchart showing the flow of the constraint learning process.

[0024] In step S1, the trajectory data acquisition unit 101 acquires trajectory data τ consisting of the state s ｉ and action a ｉ (i = 1,..., H. H is the episode length), and the trajectory data presentation unit 102 presents the acquired trajectory data or the movement of the agent corresponding to the trajectory data to the user. In step S2, the feedback acquisition unit 103 acquires the user's feedback on the presented trajectory data. Here, the user feedback y is set to y = 1 when the user determines that it is a constraint violation, and y = -1 when the user determines that the constraint is satisfied. Note that user feedback is not required for all trajectory data, and it is sufficient that user feedback is given for some of the trajectory data. Also, here we are presenting the trajectory data to the user and acquiring feedback, but this process can be omitted if trajectory data with pre-labeled user feedback is available. The trajectory data may be real data acquired based on actions in the real world, or data generated by simulation or a generative model. The combination of trajectory data and user feedback is the training data in constraint learning.

[0025] In step S3, the value of the decision function f(τ) is calculated by the forward propagation process of the constraint satisfaction discriminator 104. The constraint satisfaction discriminator 104 is a two-class discriminator that estimates whether the trajectory data satisfies the constraint, and its decision function f(τ) is represented by the following equation.

[0026] Here, the subscript m represents the m-th constraint, ĉ ｍ (the hat symbol ^ should originally be written above the character, but when described in text format, for convenience of notation, it is shown next to the character. The same applies hereinafter) is an estimated value of the cost function, θ̂ ｍ represents an estimated value of the threshold. The function g is a strictly increasing function that satisfies g(0) = 0. The function g is typically a hyperbolic tangent (tanh) for example, but is not particularly limited as long as the above conditions are satisfied, and may be ln(0.5 + 0.5e x ) or an identity function. fｍ (τ) takes a value corresponding to the difference between the cumulative sum of the cost estimates for each action and the threshold, and the cumulative cost is equal to the threshold θ^ ｍ It takes a positive value if it exceeds a certain value, and a negative value if it does not exceed that value. f(τ) is f ｍ Since it takes the sign of the maximum value of (τ) (+1 or -1), it becomes +1 if any one of the constraints is violated, and -1 if all constraints are satisfied.

[0027] Estimated cost function c^ ｍ and threshold θ^ ｍ The estimated value is calculated by the cost estimation model 105, which is comprised of a neural network (NN).

[0028] Figure 3 shows the details of the constraint satisfaction discriminator 104. State s ｉ , action a ｉ and state s ｉ＋１ The values ​​(i=1, ..., H-1) are input to the cost estimation model 105, and the estimated cost function c^ is obtained through forward propagation. ｍ (s ｉ , a ｉ ,s ｉ＋１ ) obtains. The cost estimation model 105 estimates two cost functions here, but it may estimate three or more cost functions. The constraint satisfaction discriminator 104 obtains c^ ｍ The sum and threshold θ^ ｍ Calculate the difference between the two values ​​and output the sign of the largest of the m differences.

[0029] In step S4, the loss calculation unit 106 calculates the loss L using different methods depending on whether or not there is user feedback indicating that the trajectory data does not satisfy the constraints. If the trajectory data does not satisfy the constraints (S4-YES), the process proceeds to step S5, where the loss calculation unit 106 calculates the loss L based on the difference between f(τ) and label y. Otherwise (S4-NO), the process proceeds to step S6, where the loss calculation unit 106 calculates the loss L based on the difference between each of the fm(τ) values ​​and label y.

[0030] Specifically, the loss calculation unit 106 calculates the loss L according to the following formula in one embodiment, based on the output of the constraint satisfaction discriminator 104 and user feedback.

[0031] Here, function l is a margin-based loss function, and various loss functions can be used, such as linear loss, squared loss, hinge loss, logistic loss, and sigmoid loss.

[0032] In this example, if the user feedback is y = 1, i.e., the constraint is not satisfied, the loss L is the value obtained by giving the loss function l the product of the softmax of multiple costs fm and the user feedback y. On the other hand, if the user feedback is otherwise, multiple costs f ｍ The loss L is the average of the values ​​obtained by applying the product of user feedback y to the loss function l. Here, the loss function l is a margin-based loss function similar to the one described above.

[0033] The softmax function is a function that smoothly approximates the max function. The softmax function is not limited to the function shown above; any convex, invertible, and monotonically increasing function will suffice. For example, the following function can also be used. By using such a function, the following learning process can be applied.

[0034] In another embodiment, the loss L is calculated according to the following formula. In this embodiment, the loss L is determined by the same formula regardless of the value of label y, so the branching process in step S4 is unnecessary.

[0035] In step S7, the learning unit 107 trains (updates) the neural network 105 so that the empirical risk R is minimized. The empirical loss R is expressed by the following formula, where D is the dataset set. The learning unit 107 trains the neural network 105 by performing backpropagation so that the empirical loss R is minimized.

[0036] The learning unit 107 trains the neural network 105 using a weakly supervised learning algorithm. Weakly supervised learning is an algorithm that enables learning even when only a portion of the training data has correct labels, or when the training data has incorrect correct labels. Examples of weakly supervised learning include PU (Positive-Unlabeled) learning, where only some positive examples (in this example, samples that do not satisfy the constraints) are labeled; PNU (Positive-Negative-Unlabeled) learning, where only some positive examples and some negative examples are labeled; and UU (Unlabeled-Unlabeled) learning, which uses two sets of unlabeled datasets with different class prior probabilities.

[0037] PU learning is an effective learning method when reliable positive example data can be obtained, or when labeling negative examples is costly or defining negative examples is difficult. For example, it can be used to determine whether driving data constitutes a traffic violation. Insurance companies and other organizations possess large amounts of driving data for traffic violations, and unlabeled data is easily obtainable. Also, when determining whether a robot's actions satisfy constraints, trajectory data from when a human operates the robot can be used as positive examples, and data from when an agent that has not learned the constraints operates the robot can be used as unlabeled data when learning the constraints. This makes it possible to find constraint conditions from human robot operation. PNU learning is basically the same as PU learning, but it can be used when negative example data is also available.

[0038] UU learning is an effective learning method when datasets with different probabilities of constraint violations are available. For example, since the probability of committing a violation differs between experts and novice users, constraint learning can be performed using a dataset of trajectories from experts and a dataset of trajectories from novice users. Alternatively, in social media, it can be used to identify inappropriate comments by using the text posts of users who frequently make inappropriate comments and users who rarely make inappropriate comments as training data.

[0039] For example, in PNU learning, the positive example dataset D Ｐand unlabeled dataset D Ｕ The risk R is calculated using this method. ＰＵ and negative example dataset D Ｎ and unlabeled dataset D Ｕ The risk R is calculated using this method. ＮＵ And, positive example dataset D Ｐ and negative example dataset D Ｎ The risk R is calculated using this method. ＰＮと、 The risk function R, which combines these functions. ＰＮＵ The learning process is performed to minimize each risk R. ＰＮＵ , R ＰＵ , R ＮＵ , R ＰＮ It can be defined as follows. PU learning or NU learning is the same as PNU learning.

[0040] Here, π Ｐ η is the probability that a positive example exists within the entire dataset being classified, and η is a hyperparameter in the range [-1, 1].

[0041] In UU learning, the class prior probabilities differ in the unlabeled dataset D. Ｕ and D Ｕ For each of these, the risk is calculated considering the prior probability of whether the sample is positive or negative, and these are combined to determine the final risk. The learning process is then performed to minimize this risk. The specific risk can be defined as follows:

[0042] Here, θ and θ' are from the dataset D Ｕ and D Ｕ This is the prior probability that a positive example will appear in '. Also, π Ｐ This represents the probability that a positive example exists within the entire dataset being classified. Note that dataset D Ｕ and D Ｕ In both cases, the features within the class are the same; in other words, we assume that both datasets follow the same distribution when the labels are fixed.

[0043] As described above, according to this embodiment, by formulating the constraint learning problem as a binary classification problem, it becomes possible to learn constraints by training the classifier, and by adopting a weakly supervised learning method, it becomes possible to learn constraints using datasets with few labeled data or datasets with incorrect labels. In other words, the effort required for labeling can be reduced in constraint learning.

[0044] <Reinforcement Learning> The following section explains reinforcement learning using learned constraints or cost functions.

[0045] Figure 4 is a schematic diagram illustrating reinforcement learning. Agent 401 includes an actor 402 and a critic 403. Actor 402 observes the current state s and decides on the next action a according to policy 404. Policy 404 is a function that represents the probability distribution of which action to take. In reinforcement learning, actor 402 actively interacts with the environment 406, performing exploration and utilization. Critic 403 evaluates actor 402's action a. Specifically, it obtains the reward r and constraint satisfaction f for the action, and improves policy 404 by considering the cumulative reward expectation using the action value function 405.

[0046] Figure 5 is a block diagram showing the functional configuration of a reinforcement learning device 500 that performs reinforcement learning. As shown in Figure 5, the reinforcement learning device 500 includes a trajectory data acquisition unit 501, a reward calculation unit 502, a constraint satisfaction determination unit 503, a policy update unit 504, and a policy 505.

[0047] The trajectory data acquisition unit 501 acquires sequence data consisting of the agent's state s and action a. The reward calculation unit 502 calculates the reward for each action of the agent included in the trajectory data based on a given reward function. The constraint satisfaction determination unit 503 determines whether the agent's entire action satisfies the constraints using the constraints learned using the constraint learning process described above. The constraint satisfaction determination unit 503 may determine constraint satisfaction based on the identification result of the constraint satisfaction classifier 104 shown in Figure 1, or it may determine constraint satisfaction based on the cost value obtained from the cost estimation model 105. The policy update unit 504 updates the policy 505 so as to maximize the cumulative expected reward under the constraint that the probability of satisfying the constraints is δ or greater.

[0048] In this embodiment, as shown in Equation 1, cost c ｎ The sum of the threshold θ ｎ The constraint is that the probability of violating any of the following constraints is less than or equal to the acceptable probability δ. Here, we propose a novel algorithm that allows learning even when δ > 0.

[0049] First, the constrained finite-time MDP problem in Equation 1 can be formulated as a constrained optimization problem using Lagrangian functions as follows.

[0050] This problem exhibits strong duality, making the primal-dual method applicable.

[0051] Policies are often represented using parametric models, so the policy class {π ｗ | w∈R Ｄ Let's assume that}, where w is the parameter vector. Then, we optimize equation 9 above using gradient ascending with respect to the policy parameters.

[0052] Herein lies the theorem: the probability of a constraint violation P with respect to the policy parameter w. πｗ The gradient ∇ ｗ P πｗ However, the probability P of the constraint violation ｈ πｗ and policy πｗ，ｈ This is equal to the cumulative expected value of the logarithmic derivative with respect to the policy parameter.

[0053] Therefore, in this embodiment, a constrained finite-time MDP that allows constraint violations with probability δ (>0) is solved by the procedure shown in Figure 6. In this way, the policy gradient method, based on the cumulative expected value of the product of the constraint violation probability and the policy gradient, enables the learning of a policy that maximizes the reward under the condition that the probability of constraint violation is less than or equal to δ.

[0054] <Hardware Configuration> Figure 7 shows the hardware configuration of an information processing device (computer) 200 that functions as a constraint learning device 100 or a reinforcement learning device 500. As shown in Figure 7, the information processing device 200 according to this embodiment is a computer (information processing device) in which a control unit 201, a storage unit 202, an input device 205, an output device 206, a communication interface 207, and a drive 208 are electrically connected.

[0055] The control unit 201 includes hardware processors such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory), and is configured to perform information processing based on programs and various data. The control unit 201 (CPU) is an example of processor resources.

[0056] The storage unit 202 is an example of a memory resource and is composed of, for example, a hard disk drive, a solid-state drive, etc. In this embodiment, the storage unit 202 stores various information such as a constraint learning program 203 and a reinforcement learning program 204.

[0057] The constraint learning program 203 is a program that executes information processing to learn constraints based on user feedback, and causes the information processing device 200 to function as a constraint learning device 100. The reinforcement learning program 204 is a program that executes reinforcement learning processing of the agent's policy, and causes the information processing device 200 to function as a reinforcement learning device 500.

[0058] The input device 205 is a device for inputting data, such as a mouse, keyboard, keypad, or touch panel device. The output device 206 is a device for outputting data, such as a display or speaker. The user can operate the information processing device 200 by using the input device 205 and the output device 206. The input device 205 and the output device 206 may be integrated into a single unit, such as a touch panel display.

[0059] The communication interface 207 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc., and is an interface for wired or wireless communication over a network. The information processing device 200 can communicate data with external devices and other computers via the communication interface 207.

[0060] Drive 208 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading various information such as programs stored in the storage medium 209. At least one of the constraint learning program 203 and the reinforcement learning program 204 may be stored in the storage medium 209.

[0061] The storage medium 209 is a medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical means so that a computer or other device or machine can read the stored information such as programs. The information processing device 200 may obtain at least one of the constraint learning program 203 and the reinforcement learning program 204 from this storage medium 209.

[0062] In Figure 7, a disk-type storage medium such as a CD or DVD is shown as an example of a storage medium 209. However, the type of storage medium 209 is not limited to disk type and may be other types. Examples of storage media other than disk type include semiconductor memory such as flash memory. The type of drive 208 may be appropriately selected according to the type of storage medium 209.

[0063] Regarding the specific hardware configuration of the information processing device 200, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 201 may include multiple hardware processors. The hardware processors may consist of a microprocessor, FPGA (field-programmable gate array), DSP (digital signal processor), etc. The storage unit 202 may consist of RAM and ROM included in the control unit 201. At least one of the input device 205, output device 206, communication interface 207, and drive 208 may be omitted. The information processing device 200 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. Furthermore, the information processing device 200 may be an information processing device designed specifically for the services provided, as well as a general-purpose server device, a general-purpose PC (personal computer), an industrial PC, etc.

[0064] <Other Embodiments> The embodiments described above are merely examples, and this disclosure may be modified as appropriate without departing from its essence.

[0065] The constrained reinforcement learning according to the above embodiment can be applied to the following situations. For example, it can be used by ordinary users to teach robots constraints in the field. In vision-based reinforcement learning, it can be used to teach constraints that are difficult to express in code, such as moving in a way that avoids collisions with people in the path. Furthermore, it can be used to teach pet robots which heartbeats are acceptable and which are not, based on feedback from humans.

[0066] <Note> 1. An information processing device that performs constrained reinforcement learning, comprising: an input unit (501) for inputting the state and actions of an agent; a reward calculation unit (502) for calculating a reward based on the state and actions of the agent; a determination unit (503) for determining whether or not the constraints are satisfied based on the state and actions of the agent; and a learning unit (504) for learning a policy to maximize the reward under the condition that the probability of violating the constraints is less than or equal to a predetermined allowable probability.

[0067] 2. Information processing device (100, 200) comprising: a trajectory data acquisition unit (101) that acquires a plurality of trajectory data representing the state and actions of an agent; a feedback acquisition unit (103) that acquires user feedback for at least a portion of the plurality of trajectory data; an estimation unit (104) that includes a machine learning model (105) for estimating the cost of the trajectory data and estimates whether the trajectory data satisfies constraints; a loss calculation unit (106) that calculates a loss based on the estimation result by the estimation unit and the user feedback; and a learning unit (107) that learns the machine learning model based on the loss.

[0068] 100: Constraint learning device 101: Trajectory data acquisition unit 102: Trajectory data presentation unit 103: Feedback acquisition unit 104: Constraint satisfaction discriminator 105: Cost estimation model 106: Loss calculation unit 107: Learning unit 500: Reinforcement learning device 501: Trajectory data acquisition unit 502: Reward calculation unit 503: Constraint satisfaction determination 504: Policy update unit

Claims

1. An information processing device for performing constrained reinforcement learning, comprising: an input unit for inputting the state and actions of an agent; a reward calculation unit for calculating a reward based on the state and actions of the agent; a determination unit for determining whether or not constraint conditions are satisfied based on the state and actions of the agent; and a learning unit for learning a policy to maximize the reward under the condition that the probability of violating the constraint conditions is less than or equal to a predetermined allowable probability.

2. The information processing apparatus according to claim 1, characterized in that the allowable probability is greater than zero.

3. The information processing device according to claim 1, characterized in that the learning unit learns the policy using a policy gradient method based on the cumulative expected value of the product of the constraint violation probability and the policy gradient.

4. The information processing apparatus according to claim 1, characterized in that the constraints consist of two or more conditions, and the learning unit learns a policy to maximize the reward under the condition that the probability of violating any of the constraints is less than or equal to a predetermined allowable probability.

5. An information processing device comprising: a trajectory data acquisition unit that acquires a plurality of trajectory data representing the state and actions of an agent; a feedback acquisition unit that acquires user feedback for at least a portion of the plurality of trajectory data; an estimation unit that includes a machine learning model for estimating the cost of the trajectory data and estimates whether the trajectory data satisfies constraints; a loss calculation unit that calculates a loss based on the estimation result by the estimation unit and the user feedback; and a learning unit that learns the machine learning model based on the loss.

6. The information processing apparatus according to claim 5, characterized in that the machine learning model estimates multiple costs using multiple cost functions.

7. The information processing apparatus according to claim 6, wherein the loss calculation unit calculates the cost based on the softmax of the plurality of costs and the loss function of the user feedback when the user feedback indicates that the constraint is not satisfied, and otherwise calculates the cost based on a plurality of loss functions of each of the plurality of costs and the user feedback.

8. The information processing apparatus according to claim 5, characterized in that the machine learning model is a neural network model.

9. The information processing apparatus according to claim 5, characterized in that the learning unit learns the machine learning model by weakly supervised learning.

10. The information processing method according to claim 5, further comprising: a constraint satisfaction determination unit that includes a machine learning model learned by the learning unit and determines whether the trajectory of the agent's actions satisfies the constraints; a reward calculation unit that calculates a reward for the agent's actions; and a reinforcement learning unit that learns the agent's policy to maximize the reward while satisfying the constraints.

11. An information processing method performed by a computer, comprising: an input step of inputting the state and actions of an agent; a reward calculation step of calculating a reward based on the state and actions of the agent; a determination step of determining whether or not constraints are satisfied based on the state and actions of the agent; and a learning step of learning a policy to maximize the reward under the condition that the probability of violating the constraints is less than or equal to a predetermined allowable probability.

12. The information processing method according to claim 11, characterized in that the allowable probability is greater than zero.

13. The information processing method according to claim 11, characterized in that the learning step learns the policy using a policy gradient method based on the cumulative expected value of the product of the constraint violation probability and the policy gradient.

14. The information processing method according to claim 11, characterized in that the constraints consist of two or more conditions, and in the learning step, a policy is learned to maximize the reward under the condition that the probability of violating any of the constraints is less than or equal to a predetermined allowable probability.

15. An information processing method performed by a computer, characterized by performing: a trajectory data acquisition step of acquiring a plurality of trajectory data representing the state and actions of an agent; a feedback acquisition step of acquiring user feedback for at least a portion of the plurality of trajectory data; an estimation step of including a machine learning model for estimating the cost of the trajectory data and estimating whether the trajectory data satisfies constraints; a loss calculation step of calculating a loss based on the estimation result in the estimation step and the user feedback; and a learning step of learning the machine learning model based on the loss.

16. The information processing method according to claim 15, characterized in that the estimation step involves estimating multiple costs using multiple cost functions with the machine learning model.

17. The information processing method according to 16, characterized in that, in the loss calculation step, if the user feedback indicates that the constraint is not satisfied, the cost is calculated based on the difference between the softmax of the plurality of costs and the user feedback, and otherwise the cost is calculated based on the plurality of differences between each of the plurality of costs and the user feedback.

18. The information processing method according to claim 15, characterized in that the machine learning model is a neural network model.

19. The information processing method according to claim 15, characterized in that the machine learning model is trained by weakly supervised learning in the learning step.

20. The information processing method according to 15, further comprising: a constraint satisfaction determination step of determining whether the trajectory of the agent's actions satisfies the constraints using the machine learning model learned in the learning step; a reward calculation step of calculating a reward for the agent's actions; and a reinforcement learning step of learning the agent's policy to maximize the reward while satisfying the constraints.

21. A program for causing a computer to perform the method described in any one of claims 11 to 20.

Images