Information processing device, information processing method, and program

Abstract

This provides a technology that enables policy learning such that constraint violations are less than or equal to the acceptable probability. [Solution] An information processing device for performing constrained reinforcement learning, comprising: an input unit for inputting the state and actions of an agent; a reward calculation unit for calculating a reward based on the state and actions of the agent; a determination unit for determining whether or not constraint conditions are satisfied based on the state and actions of the agent; and a learning unit for learning a policy to maximize the reward under the condition that the probability of violating the constraint conditions is less than or equal to a predetermined allowable probability.

Description

[Technical Field]

[0001] This disclosure relates to constrained reinforcement learning. [Background technology]

[0002] Constrained reinforcement learning is a method of reinforcement learning that learns strategies to maximize rewards while imposing behavioral constraints. It is useful in ensuring safety and generating diverse behaviors. However, it has been pointed out that actually designing reward and cost functions is difficult, and as a solution, methods that learn reward and cost functions from human feedback (user feedback) are considered promising (Non-Patent Document 1, etc.).

[0003] Reinforcement learning based on user feedback suffers from the problem of time-consuming and laborious labeling. Existing research has focused on how constraints are learned from what kind of feedback, and how environment exploration is performed while satisfying constraints using feedback, but has not attempted to reduce the cost of labeling.

[0004] Furthermore, in constrained reinforcement learning, it is common practice to learn a policy under the condition that no constraint violations occur. However, there are cases where constraint violations are acceptable if they occur below a certain probability. Constrained learning under such conditions has not been performed conventionally. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Glazier, Arie, et al. "Making human-like trade-offs in constrained environments by learning from demonstrations." arXiv preprint arXiv:2109.11018 (2021). <URL: https: / / arxiv.org / abs / 2109.11018>

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present disclosure aims to provide a technique that enables reinforcement learning with constraints more preferably based on user feedback than in the past.

Means for Solving the Problems

[0007] One aspect of the present disclosure is an information processing apparatus that performs reinforcement learning with constraints, comprising: an input unit that inputs the state and actions of an agent; a reward calculation unit that calculates a reward based on the state and actions of the agent; a determination unit that determines whether or not the constraint conditions are satisfied based on the state and actions of the agent; and a learning unit that learns a policy so as to maximize the reward under the condition that the probability of violating the constraint conditions is equal to or less than a predetermined allowable probability.

[0008] According to this aspect, it is possible to learn a policy such that the probability of a constraint violation is equal to or less than the allowable probability and the reward is maximized.

[0009] One aspect of this disclosure is an information processing device comprising: a trajectory data acquisition unit that acquires a plurality of trajectory data representing the state and actions of an agent; a feedback acquisition unit that acquires user feedback on at least a portion of the plurality of trajectory data; an estimation unit that includes a machine learning model for estimating the cost of the trajectory data and estimates whether the trajectory data satisfies constraints; a loss calculation unit that calculates a loss based on the estimation result by the estimation unit and the user feedback; and a learning unit that learns the machine learning model based on the loss. ru.

[0010] Thus, in this embodiment, we focus on the fact that constraint learning becomes a two-class classification problem based on user feedback, and by training a classifier, we can obtain an estimate of the cost function.

[0011] In one embodiment, the machine learning model may be configured to estimate multiple costs using multiple cost functions. The loss calculation unit may also be configured to calculate the cost based on the softmax of the multiple costs and the loss function of the user feedback if the user feedback indicates that the constraints are not satisfied, and to calculate the cost based on multiple loss functions of each of the multiple costs and the user feedback otherwise.

[0012] In one embodiment, the learning unit learns the machine learning model using weakly supervised learning. By using weakly supervised learning, constraints can be learned with a low feedback cost.

[0013] In one embodiment, the information processing device includes a machine learning model learned by the learning unit, and further includes a constraint satisfaction determination unit that determines whether the trajectory of the agent's actions satisfies the constraints, a reward calculation unit that calculates a reward for the agent's actions, and a reinforcement learning unit that learns the agent's policy to maximize the reward while satisfying the constraints. By performing reinforcement learning using the constraints obtained through learning in this way, the agent can be made to learn appropriate actions that satisfy the constraint conditions.

[0014] This disclosure can be understood as an information processing apparatus comprising at least a part of the above means. This disclosure can also be understood as a method comprising at least a part of the above processing, or as a program for implementing such a method or a recording medium on which such program is non-temporarily recorded. Furthermore, this disclosure includes apparatuses or methods in which each of the above means and processing is combined with each other as much as possible. [Effects of the Invention]

[0015] According to this disclosure, constrained reinforcement learning can be more effectively performed based on user feedback. [Brief explanation of the drawing]

[0016] [Figure 1] A diagram showing an example configuration of the constraint learning device related to this disclosure. [Figure 2] A flowchart illustrating the constraint learning process in this disclosure. [Figure 3] A diagram showing an example of the configuration of the constraint satisfaction discriminator in this disclosure. [Figure 4] A diagram illustrating reinforcement learning as described in this disclosure. [Figure 5] A diagram showing an example configuration of the reinforcement learning device related to this disclosure. [Figure 6] Pseudocode illustrating the reinforcement learning algorithm related to this disclosure. [Figure 7] A diagram showing the hardware configuration of the constraint learning device and reinforcement learning device related to this disclosure. [Modes for carrying out the invention]

[0017] <Overview> In the constrained finite-time MDP (Markov Decision Process) according to this embodiment, the following applies: Solve the following problem. Specifically, find the cumulative expected reward sum v1 when following policy π. π The policy π that maximizes (P1) is defined by the cost function c for each constraint. n The sum of the threshold θ n We find the value under the condition that the probability of exceeding a certain value is less than or equal to δ.

number

[0018] c n This function is called the cost function and is a mapping from the state-action pair space S×A to [-1,1]. Here, the index n indicates the nth constraint. Also, δ∈[0,1) represents the allowed failure probability and is given.

[0019] Designing constraints or cost functions is inherently difficult. For example, expressing constraints in code requires specialized knowledge, making it difficult for average users to modify agent behavior to meet their needs. Furthermore, some constraints are simply difficult to express in code. For instance, the constraint "behave like a human" cannot be expressed in code. Additionally, it is difficult to anticipate actions that should not be performed by the agent, and sometimes it is only after observing actual behavior that we realize those actions should be considered constraint violations. To address this challenge of designing constraints or cost functions, a method has been proposed that learns constraints based on user feedback.

[0020] Furthermore, conventional methods can only solve the problem when the acceptable constraint violation probability δ is zero. By setting δ > 0, it is possible to find the optimal policy under the condition that constraint violations are acceptable within the range of probability δ.

[0021] <Constraint Learning> In this embodiment, we propose a method for learning constraints based on user feedback, thereby reducing the effort required for user feedback or labeling training data.

[0022] Figure 1 is a block diagram showing the functional configuration of the constraint learning device 100. As shown in Figure 1, the constraint learning device 100 includes a trajectory data acquisition unit 101, a trajectory data presentation unit 102, a feedback acquisition unit 103, a constraint satisfaction discriminator 104, a loss calculation unit 106, and a learning unit 107. The constraint satisfaction discriminator 104 includes a cost estimation model 105 for estimating costs.

[0023] Figure 2 is a flowchart showing the flow of the constraint learning process.

[0024] In step S1, the trajectory data acquisition unit 101 determines the agent's state s i and action a i The trajectory data presentation unit 102 acquires trajectory data τ consisting of (i=1,...,H, where H is the episode length) and presents the acquired trajectory data or the movement of the agent corresponding to the trajectory data to the user. In step S2, the feedback acquisition unit 103 acquires the user's feedback on the presented trajectory data. Here, the user feedback y is set to y=1 if the user determines that there is a constraint violation, and to y=-1 if the user determines that the constraint is satisfied. Note that user feedback is not required for all trajectory data; it is sufficient to provide user feedback for some of the trajectory data. Also, although trajectory data is presented to the user and feedback is acquired here, this process can be omitted if trajectory data with pre-labeled user feedback is available. The trajectory data may be real data acquired based on real-world actions, or it may be data generated by simulation or generative models. The combination of trajectory data and user feedback is the training data in constraint learning.

[0025] In step S3, the value of the decision function f(τ) is calculated by the forward propagation process of the constraint satisfaction discriminator 104. The constraint satisfaction discriminator 104 is a two-class discriminator that estimates whether the trajectory data satisfies the constraints, and its decision function f(τ) is represented by the following equation. [Number]

[0026] Here, the subscript m represents the m-th constraint, and ĉ m (the caret symbol ^ should originally be written above the character, but for convenience of notation in text form, it is shown next to the character. The same applies hereinafter) is the estimated value of the cost function, and θ̂ m represents the estimated value of the threshold. The function g is a strictly increasing function that satisfies g(0) = 0. The function g is typically, for example, the hyperbolic tangent (tanh), but is not particularly limited as long as it satisfies the above conditions, and may be ln(0.5 + 0.5e x ) or the identity function. f m (τ) takes a value according to the difference between the cumulative sum of the cost estimation values for each action and the threshold, and takes a positive value when the cumulative cost exceeds the threshold θ̂ m and a negative value when it does not exceed. f(τ) takes the sign (+1 or -1) of the maximum value of f m (τ), so it becomes +1 when violating any one of any of the constraints and -1 when satisfying all the constraints.

[0027] The estimated value ĉ m of the cost function and the estimated value θ̂ m of the threshold are estimated by the cost estimation model 105. Here, the cost estimation model 105 is composed of a neural network (NN).

[0028] Figure 3 is a diagram showing the details of the constraint satisfaction discriminator 104. The state s i , action a i and state s i+1 (i = 1,..., H - 1) are input to the cost estimation model 105, and the estimated value ĉ m (s i,a i ,s i+1 ) obtains. The cost estimation model 105 estimates two cost functions here, but it may estimate three or more cost functions. The constraint satisfaction discriminator 104 obtains c^ m The sum and threshold θ^ m Calculate the difference between the two values ​​and output the sign of the largest of the m differences.

[0029] In step S4, the loss calculation unit 106 calculates the loss L using different methods depending on whether or not there is user feedback indicating that the trajectory data does not satisfy the constraints. If the trajectory data does not satisfy the constraints (S4-YES), the process proceeds to step S5, where the loss calculation unit 106 calculates the loss L based on the difference between f(τ) and label y. Otherwise (S4-NO), the process proceeds to step S6, where the loss calculation unit 106 calculates the loss L based on the difference between each of fm(τ) and label y.

[0030] Specifically, the loss calculation unit 106 calculates the loss L according to the following formula in one embodiment, based on the output of the constraint satisfaction discriminator 104 and user feedback.

number

[0031] Here, function l is a margin-based loss function, and various loss functions can be used, such as linear loss, squared loss, hinge loss, logistic loss, and sigmoid loss.

[0032] In this example, if the user feedback is y=1, i.e., the constraint is not satisfied, then multiple The loss L is obtained by applying the product of the softmax of the number of costs fm and the user feedback y to the loss function l. On the other hand, if the user feedback is otherwise, the loss L is obtained by multiplying the costs f m The loss L is the average of the values ​​obtained by applying the product of user feedback y to the loss function l. Here, the loss function l is a margin-based loss function similar to the one described above.

[0033] The softmax function is a function that smoothly approximates the max function. The softmax function is not limited to the function shown above; any convex, invertible, and monotonically increasing function will suffice. For example, the following function can also be used. By using such a function, the following learning process can be applied.

number

[0034] In another embodiment, the loss L is calculated according to the following formula. In this embodiment, the loss L can be obtained by the same formula regardless of the value of label y, so the branching process in step S4 is unnecessary.

number

[0035] In step S7, the learning unit 107 trains (updates) the neural network 105 so that the empirical risk R is minimized. The empirical loss R is expressed by the following formula, where D is the dataset set. The learning unit 107 trains the neural network 105 by performing backpropagation so that the empirical loss R is minimized.

number

[0036] The learning unit 107 trains the neural network 105 using a weakly supervised learning algorithm. Weakly supervised learning is an algorithm that enables learning even when only some of the training data have correct labels, or when some of the training data has incorrect correct labels. Weakly supervised learning includes PU (Positive-Unlabeled) learning, where only some positive examples (in this example, samples that do not satisfy the constraints) are labeled, PNU (Positive-Negative-Unlabeled) learning, where only some positive examples and some negative examples are labeled, and One example is UU (Unlabeled-Unlabeled) learning, which uses two sets of unlabeled datasets with different prior probabilities.

[0037] PU learning is an effective learning method when reliable positive example data can be obtained, or when labeling negative examples is costly or defining negative examples is difficult. For example, it can be used to determine whether driving data constitutes a traffic violation. Insurance companies and other organizations possess large amounts of driving data for traffic violations, and unlabeled data is easily obtainable. It can also be used to determine whether a robot's actions satisfy constraints. Trajectory data from when a human operates the robot can be used as positive examples, and data from when an agent that has not learned the constraints operates the robot can be used as unlabeled data when learning the constraints. This allows constraint conditions to be found from human robot operation. PNU learning is basically the same as PU learning, but it can be used when negative example data is also available.

[0038] UU learning is an effective learning method when datasets with different probabilities of constraint violations are available. For example, since the probability of committing a violation differs between experts and novice users, constraint learning can be performed using datasets of trajectories from experts and novice users. Alternatively, it can be used to identify inappropriate comments on social media by using the text posts of users who frequently make inappropriate comments and users who rarely make inappropriate comments as training data.

[0039] For example, in PNU training, the positive example dataset D P and unlabeled dataset D U The risk R is calculated using the method described above. PU and negative example dataset D N and unlabeled dataset D U The risk R is calculated using the method described above. NU And, positive example dataset D P and negative example dataset D N The risk R is calculated using the method described above. PNと、 The risk function R, which combines these functions. PNUThe learning process is performed to minimize each risk R. PNU ,R PU ,R NU ,R PN This can be defined as follows: PU learning or N U-learning is similar to PNU-learning.

number

[0040] Here, π P η is the probability that a positive example exists within the entire dataset being classified, and η is a hyperparameter in the range [-1,1].

[0041] In UU learning, the class prior probabilities differ across unlabeled datasets D U and D U For each of these, the risk is calculated considering the prior probability of whether the sample is positive or negative, and these are combined to determine the final risk. The learning process is then performed to minimize this risk. The specific risk can be defined as follows:

number

[0042] Here, θ and θ' are from the dataset D U and D U This is the prior probability that a positive example will appear in '. Also, π P This represents the probability that a positive example exists within the entire dataset being classified. Note that dataset D U and D U In both cases, the features within the class are the same; in other words, we assume that both datasets follow the same distribution when the labels are fixed.

[0043] As described above, according to this embodiment, by formulating the constraint learning problem as a binary classification problem, it becomes possible to learn constraints by training the classifier, and by adopting a weakly supervised learning method, it becomes possible to learn constraints using datasets with few labeled data or datasets with incorrect labels. In other words, the effort required for labeling can be reduced in constraint learning.

[0044] <Reinforcement Learning> The following section explains reinforcement learning using learned constraints or cost functions.

[0045] Figure 4 is a schematic diagram illustrating reinforcement learning. Agent 401 includes actor 402 and critic 403. Actor 402 observes the current state s and decides on the next action a according to policy 404. Policy 404 is a function that represents the probability distribution of which action to take. In reinforcement learning, actor 402 actively interacts with the environment 406, performing exploration and utilization. Critic 403 evaluates actor 402's action a. Specifically, it obtains the reward r and constraint satisfaction f for the action, and improves policy 404 by considering the cumulative reward expectation using the action-value function 405.

[0046] Figure 5 is a block diagram showing the functional configuration of a reinforcement learning device 500 that performs reinforcement learning. As shown in Figure 5, the reinforcement learning device 500 includes a trajectory data acquisition unit 501, a reward calculation unit 502, a constraint satisfaction determination unit 503, a policy update unit 504, and a policy 505.

[0047] The trajectory data acquisition unit 501 acquires sequential data consisting of the agent's state s and action a. The reward calculation unit 502 calculates the reward for each action of the agent included in the trajectory data based on a given reward function. The constraint satisfaction determination unit 503 determines whether the agent's entire action satisfies the constraints using the constraints learned using the constraint learning process described above. The constraint satisfaction determination unit 503 may determine constraint satisfaction based on the identification result of the constraint satisfaction classifier 104 shown in Figure 1, or it may determine constraint satisfaction based on the cost value obtained from the cost estimation model 105. The policy update unit 504 updates the policy 505 so as to maximize the cumulative expected reward, under the constraint that the probability of satisfying the constraints is δ or greater.

[0048] In this embodiment, as shown in Equation 1, cost c n The sum of the threshold θ n The constraint is that the probability of violating any of the following constraints is less than or equal to the acceptable probability δ. Here, we propose a novel algorithm that allows learning even when δ > 0.

[0049] First, the constrained finite-time MDP problem in Equation 1 can be formulated as a constrained optimization problem using Lagrangian functions as follows.

number

[0050] This problem exhibits strong duality, therefore the primal-dual method (primary dual) The law is applicable.

[0051] Policies are often represented using parametric models, so the policy class {π w |w∈R D Let's assume that}, where w is the parameter vector. Then, we optimize equation 9 above using gradient rafting with respect to the policy parameters.

[0052] Herein lies the theorem: the probability of a constraint violation P with respect to the policy parameter w. πw The gradient ∇ w P πw However, the probability P of the constraint violation h πw and policy π w,h This is equal to the cumulative expected value of the logarithmic derivative with respect to the policy parameter.

number

[0053] Therefore, in this embodiment, a constrained finite-time MDP that allows constraint violations with probability δ (>0) is solved by the procedure shown in Figure 6. In this way, the policy gradient method, based on the cumulative expected value of the product of the constraint violation probability and the policy gradient, enables the learning of a policy that maximizes the reward under the condition that the probability of constraint violation is less than or equal to δ.

[0054] <Hardware Configuration> Figure 7 shows the hardware configuration of an information processing device (computer) 200 that functions as a constraint learning device 100 or a reinforcement learning device 500. As shown in Figure 7, the information processing device 200 according to this embodiment is a computer (information processing device) in which a control unit 201, a storage unit 202, an input device 205, an output device 206, a communication interface 207, and a drive 208 are electrically connected.

[0055] The control unit 201 is a hardware processor, the CPU (Central Processing Unit) It includes RAM (Random Access Memory), ROM (Read Only Memory), etc., and is configured to perform information processing based on programs and various data. The control unit 201 (CPU) is an example of processor resources.

[0056] The memory unit 202 is an example of a memory resource and is composed of, for example, a hard disk drive, a solid-state drive, etc. In this embodiment, the memory unit 202 stores various information such as a constraint learning program 203 and a reinforcement learning program 204.

[0057] The constraint learning program 203 is a program that performs information processing to learn constraints based on user feedback, and causes the information processing device 200 to function as the constraint learning device 100. The reinforcement learning program 204 is a program that performs reinforcement learning processing of the agent's policy, and causes the information processing device 200 to function as the reinforcement learning device 500.

[0058] The input device 205 is a device for inputting data, such as a mouse, keyboard, keypad, or touch panel. The output device 206 is a device for outputting data, such as a display or speaker. The user can operate the control device 110 by using the input device 205 and the output device 206. The input device 205 and the output device 206 may be integrated into a single unit, such as a touch panel display.

[0059] The communication interface 207 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc., and is an interface for wired or wireless communication over a network. The control device 110 can communicate data with the mobile robot 120, LLM 130, and other computers via the communication interface 207.

[0060] Drive 208 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading various information such as programs stored on the storage medium 209. At least one of the above constraint learning program 203 and reinforcement learning program 204 is stored on the storage medium It may be stored in 209.

[0061] The storage medium 209 is a medium that stores information such as stored programs by electrical, magnetic, optical, mechanical, or chemical means so that a computer or other device or machine can read the stored information such as programs. The control device 110 may obtain at least one of the constraint learning program 203 and the reinforcement learning program 204 from this storage medium 209.

[0062] In Figure 7, a disk-type storage medium such as a CD or DVD is shown as an example of a storage medium 209. However, the type of storage medium 209 is not limited to disk type; it may be a non-disk type. Examples of non-disk type storage media include semiconductor memory such as flash memory. The type of drive 208 may be appropriately selected according to the type of storage medium 209.

[0063] Regarding the specific hardware configuration of the control device 110, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 201 may include multiple hardware processors. The hardware processors may be microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), etc. The configuration may be as follows. The memory unit 202 may be composed of RAM and ROM included in the control unit 201. At least one of the input device 205, output device 206, communication interface 207, and drive 208 may be omitted. The control device 110 may be composed of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. Furthermore, the control device 110 may be an information processing device designed specifically for the services provided, as well as a general-purpose server device, a general-purpose PC (Personal Computer), an industrial PC, etc.

[0064] <Other Embodiments> The embodiments described above are merely examples, and this disclosure may be modified as appropriate without departing from its essence.

[0065] The constrained reinforcement learning according to the above embodiment can be applied to the following situations. For example, it can be used by ordinary users to teach robots constraints in the field. In vision-based reinforcement learning, it can be used to teach constraints that are difficult to express in code, such as moving in a way that avoids collisions with people in the path. Furthermore, it can be used to teach pet robots which heartbeats are acceptable and which are not, based on feedback from humans.

[0066] <Note> 1. An information processing device that performs constrained reinforcement learning, An input section (501) for inputting the agent's status and actions, A reward calculation unit (502) calculates a reward based on the state and actions of the agent, A determination unit (503) determines whether or not the constraint conditions are satisfied based on the agent state and actions, A learning unit (504) learns a policy to maximize the reward under the condition that the probability of violating the aforementioned constraints is less than or equal to a predetermined allowable probability, An information processing device equipped with the following features.

[0067] 2. Trajectory data acquisition unit (101) that acquires multiple trajectory data representing the state and actions of the agent, A feedback acquisition unit (103) that acquires user feedback on at least a portion of the aforementioned plurality of trajectory data, Includes a machine learning model (105) for estimating the cost of the trajectory data, and an estimation unit (104) for estimating whether or not the trajectory data satisfies the constraints, A loss calculation unit (106) calculates the loss based on the estimation result from the estimation unit and the user feedback, Based on the loss, a learning unit (107) learns the machine learning model, An information processing device (100,200) equipped with the following. [Explanation of Symbols]

[0068] 100: Constraint learning device 101: Trajectory data acquisition unit 102: Trajectory data presentation unit 103: Feedback acquisition unit 104: Constraint satisfaction discriminator 105: Cost estimation model 106: Loss calculation unit 107: Learning unit 500: Reinforcement learning device 501: Trajectory data acquisition unit 502: Reward calculation unit 503: Constraint satisfaction determination 504: Policy update section

Claims

1. An information processing device that performs constrained reinforcement learning, An input section for inputting the agent's status and actions, A reward calculation unit that calculates a reward based on the state and actions of the agent, A determination unit that determines whether or not the constraint conditions are satisfied based on the state and actions of the agent, A learning unit learns a policy to maximize the reward under the condition that the probability of violating the aforementioned constraints is less than or equal to a predetermined acceptable probability, An information processing device equipped with the following features.

2. The aforementioned acceptable probability is greater than zero. The information processing apparatus according to feature 1.

3. The learning unit learns the policy using a policy gradient method based on the cumulative expected value of the product of the constraint violation probability and the policy gradient. The information processing apparatus according to feature 1.

4. The aforementioned constraints consist of two or more conditions, The learning unit learns a policy to maximize the reward under the condition that the probability of violating any of the constraints is less than or equal to a predetermined acceptable probability. The information processing apparatus according to claim 1.

5. A trajectory data acquisition unit acquires multiple trajectory data representing the state and actions of an agent, A feedback acquisition unit that acquires user feedback on at least a portion of the aforementioned plurality of trajectory data, Includes a machine learning model for estimating the cost of the trajectory data, and an estimation unit for estimating whether the trajectory data satisfies the constraints, A loss calculation unit calculates the loss based on the estimation results from the estimation unit and the user feedback, A learning unit that learns the machine learning model based on the loss, An information processing device equipped with the following features.

6. The aforementioned machine learning model estimates multiple costs using multiple cost functions. The information processing apparatus according to feature 5.

7. The loss calculation unit calculates the cost based on the softmax of the multiple costs and the loss function of the user feedback if the user feedback indicates that the constraint is not satisfied, and otherwise calculates the cost based on the multiple loss functions of each of the multiple costs and the user feedback. The information processing apparatus according to feature 6.

8. The aforementioned machine learning model is a neural network model. The information processing apparatus according to feature 5.

9. The learning unit learns the machine learning model through weakly supervised learning. The information processing apparatus according to feature 5.

10. The machine learning model learned by the learning unit, and the trajectory of the agent's actions. A constraint satisfaction determination unit that determines whether or not the constraint is satisfied, A reward calculation unit that calculates a reward for the actions of the agent, A reinforcement learning unit that learns the agent's policy to maximize rewards while satisfying constraints, The information processing method according to claim 5, further comprising the features described above.

11. A method of information processing performed by a computer, An input step to enter the agent's status and actions, A reward calculation step that calculates the reward based on the state and actions of the agent, A determination step that determines whether or not the constraints are satisfied based on the state and actions of the agent, A learning step in which a policy is learned to maximize the reward under the condition that the probability of violating the aforementioned constraints is less than or equal to a predetermined acceptable probability, Information processing methods, including those mentioned above.

12. The aforementioned acceptable probability is greater than zero. The information processing method according to feature 11.

13. The learning step involves learning the policy using a policy gradient method based on the cumulative expected value of the product of the constraint violation probability and the policy gradient. The information processing method according to feature 11.

14. The aforementioned constraints consist of two or more conditions, In the learning step, a policy is learned to maximize the reward under the condition that the probability of violating any of the constraints is less than or equal to a predetermined acceptable probability. The information processing method according to feature 11.

15. A method of information processing performed by a computer, A trajectory data acquisition step that acquires multiple trajectory data representing the state and actions of the agent, A feedback acquisition step to obtain user feedback on at least a portion of the aforementioned plurality of trajectory data, Includes a machine learning model for estimating the cost of the trajectory data, and an estimation step for estimating whether the trajectory data satisfies the constraints, A loss calculation step which calculates the loss based on the estimation results in the estimation step and the user feedback, A learning step in which the machine learning model is trained based on the loss, An information processing method characterized by performing the following.

16. In the estimation step described above, the machine learning model estimates multiple costs using multiple cost functions. The information processing method according to feature 15.

17. In the loss calculation step, if the user feedback indicates that the constraint is not satisfied, the cost is calculated based on the difference between the softmax of the multiple costs and the user feedback; otherwise, the cost is calculated based on the multiple differences between each of the multiple costs and the user feedback. The information processing method according to feature 16.

18. The aforementioned machine learning model is a neural network model. The information processing method according to feature 15.

19. In the aforementioned learning step, the machine learning model is trained using weakly supervised learning. The information processing method according to feature 15.

20. A constraint satisfaction determination step in which the machine learning model learned in the learning step determines whether the trajectory of the agent's actions satisfies the constraints, A reward calculation step for calculating the reward for the agent's actions, A reinforcement learning step that learns the agent's policy to maximize rewards while satisfying constraints, The information processing method according to claim 15, further characterized by performing the above.

21. A program for causing a computer to perform the method described in any one of claims 11 to 20.

Images