Reinforcement learning processing system, reinforcement learning processing method, and program

Abstract

To realize a reinforcement learning processing method which realizes reinforcement learning of a strategy suited for a difference in a potential state.SOLUTION: A reinforcement learning processing system 1000 performs strategy processing (processing performed by a strategy π(a|s, z)), by using a latent variable (the latent variable in which a difference in environment (potential difference) and a difference in tasks (potential difference) are reflected), which are acquired by encoding context data including data (data about environment) which can express the difference in the environment, and the data (data about the tasks) which can express the difference in the tasks (for example, a difference of objects (targets)). Thereby, the strategy π(a|s, z) which can be easily applied even when there are the difference in the environment (the potential difference) and the difference in the tasks (the potential difference) can be acquired.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the application of policies in reinforcement learning, and in particular to a technique for applying policies in multitask reinforcement learning (a machine learning (reinforcement learning) method that solves multiple problems (tasks) with a single model). [Background technology]

[0002] In recent years, multitask reinforcement learning, a method of reinforcement learning that solves multiple tasks with a single model, has been attracting attention. In particular, meta-reinforcement learning, a method that attempts to solve the problems of multitask reinforcement learning by updating the parameters of a shared policy from among various policies for different tasks, has attracted attention (for example, see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson, "Fast Context Adaptation via Meta-Learning," in Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. Long Beach, California, USA: PMLR, June, 9-15, 2019, pp. 7693-7702. Summary of the Invention [Problem to be solved by the invention]

[0004] However, most meta-reinforcement learning methods require learning using a large-scale policy network (e.g., a network for acquiring policies realized using a neural network) to accommodate each task. Therefore, in the above-mentioned meta-reinforcement learning methods, a learning process using a large-scale policy network must be performed to acquire a set of policies (multiple policies) to accommodate multiple tasks (a large-scale policy network model must be trained). For example, in soccer, when a robot is made to head a soccer ball, if the heading point is slightly shifted or the friction coefficient of the soccer ball changes, the direction in which the soccer ball headed by the robot flies will change significantly even if the actions made by the robot are similar. In such a case (when the controlled object (in this case, the robot) is made to perform a similar action when there is a slight difference in the environment or task as described above, and the results are significantly different), in order to control a robot using meta-learning and have it learn to head a soccer ball so that it flies into an arbitrary goal, it would be necessary to prepare tasks with infinitely different goals (purposes), prepare a policy network to achieve the prepared tasks (an infinite number of tasks), and perform reinforcement learning for robot control, which is impossible to achieve.

[0005] In view of the above problems, the present invention aims to realize a reinforcement learning processing system, a reinforcement learning processing method, and a program that acquire a policy for appropriately controlling a control target even when there is a subtle change in a potential situation, and realize reinforcement learning that enables the control target to be appropriately controlled by the acquired policy. In other words, the present invention aims to realize a reinforcement learning processing system, a reinforcement learning processing method, and a program that realize reinforcement learning of a policy that matches differences in potential situations. [Means for solving the problem]

[0006] In order to solve the above problems, a first invention is a reinforcement learning processing system for performing reinforcement learning processing in which a control object placed in a predetermined environment is made to perform an action based on behavior data, and the control object is made to learn to perform a predetermined action based on state data of a state transitioned to by the action, the behavior data, and reward data determined based on the behavior data and the state data, the system comprising a context acquisition unit, an encoder, and a policy processing unit.

[0007] The context acquisition unit acquires context data including (1) data capable of expressing differences in environments, and (2) data capable of expressing differences in tasks to be executed by a control target.

[0008] The encoder performs an encoding process on the context data to obtain latent variables.

[0009] The policy processing unit executes a policy process to acquire behavior data based on the latent variables and state data acquired from the environment.

[0010] In this reinforcement learning processing system, policy processing (processing using policy π(a|s,z)) is performed using latent variables (latent variables reflecting environmental differences (potential differences) and task differences (potential differences)) obtained by encoding context data including (1) data that can represent environmental differences (data related to the environment) and (2) data that can represent task differences (e.g., differences in objectives (goals)). Therefore, it is possible to obtain a policy π(a|s,z) that is easy to apply even when there are environmental differences (potential differences) or task differences (potential differences).

[0011] Therefore, in this reinforcement learning processing system, a policy is acquired that enables the control object to be appropriately controlled even when there are subtle changes in the underlying situation, and the control object can be appropriately controlled using the acquired policy.

[0012] An example of "data that can represent differences in the environment" is the coefficient of friction between the surface of a robot (control target (agent)) and a soccer ball in a task in which the robot heads the soccer ball and reaches a specified position.

[0013] Furthermore, examples of "data capable of expressing differences in tasks" include, in the case of the above task, the throwing position and target position of the soccer ball.

[0014] A second invention is the first invention, further comprising a first trajectory data acquisition unit and an evaluation update processing unit.

[0015] The first trajectory data acquisition unit causes the control target to execute an action according to the action data, and acquires time-series data of state data of a state transitioned to by the action, the action data, and reward data determined based on the action data and the state data.

[0016] The evaluation update processing unit performs evaluation update processing for the reinforcement learning processing.

[0017] The encoder has an encoder network model that is a model that can be trained by updating the parameter η.

[0018] The policy processing unit has a policy network model that is a model that can be learned by updating a parameter ψ.

[0019] The evaluation update processing unit has a Q network model, which is a model that can learn by updating the parameter φ, and performs a parameter update process that updates the parameter φ of the Q network model, the parameter η of the encoder network model, and the parameter ψ of the policy network model by evaluating the time series data acquired by the first trajectory data acquisition unit.

[0020] The encoder updates the encoder network model with the parameter η updated by the evaluation update processing unit, and the policy processing unit updates the policy network model with the parameter ψ updated by the evaluation update processing unit.

[0021] This makes it possible to realize a reinforcement learning processing system using a model that can learn by updating parameters.

[0022] A third invention is the second invention, wherein the context acquisition unit acquires context data from one of a plurality of set domains by changing, in an environment, settings that affect the environment and / or settings that affect a task to be executed by the controlled object.

[0023] The encoder acquires latent variables by performing encoding processing on the context data, and the policy processor acquires behavior data based on the latent variables and state data acquired from the domain.

[0024] The first trajectory data acquisition unit causes the control target to execute an action according to the action data, and acquires time-series data of state data of a state transitioned to by the action, the action data, and reward data determined based on the action data and the state data.

[0025] The evaluation update processing unit executes a parameter update process based on the time-series data.

[0026] The context acquisition unit, the encoder, the policy processing unit, the first trajectory data acquisition unit, and the evaluation update processing unit execute a policy generation process that is realized by repeatedly executing the above process for all of the multiple domains that have been set.

[0027] The evaluation update processing unit treats the parameters ψ of the policy network model obtained after the policy generation process is completed as multi-domain compatible parameters that correspond to multiple domains, and obtains the policy corresponding to the multi-domain compatible parameters as a multi-domain compatible policy.

[0028] As a result, in this reinforcement learning processing system, it is possible to change the environment (domain) (change the context data) and obtain a policy that is compatible with multiple domains (a multi-domain compatible policy).

[0029] A fourth aspect of the present invention is the third aspect of the present invention, further comprising an optimal policy acquisition unit that selects an optimal policy.

[0030] In this reinforcement learning processing system, a plurality of multi-domain policies are generated by executing the policy generation process a plurality of times.

[0031] Then, the optimal policy acquisition unit acquires the average value of the KL loss, which is the loss when a reinforcement learning process is executed for each of the multiple multi-domain compatible policies generated, derived from the KL divergence between the cumulative reward and the prior distribution of the latent variables and the conditional distribution based on the context data of the latent variables, and executes an optimal policy selection process to select the multi-domain compatible policy with the maximum average KL loss as the optimal policy.

[0032] As a result, this reinforcement learning processing system can select (obtain) the multi-domain policy with the maximum average KL loss as the optimal policy. The policy with the maximum average KL loss (maximum policy) is guaranteed to have low average loss for any domain, so when applied to an unknown environment (unknown domain), simply by adjusting the latent variables, the learning process will proceed smoothly and there is a high possibility of realizing highly accurate optimization processing.

[0033] A fifth invention is a reinforcement learning processing system for performing reinforcement learning processing in which a control object placed in a predetermined environment is made to perform an action based on behavior data, and the control object is made to learn to perform a predetermined action based on state data of a state transitioned to by the action, the behavior data, and reward data determined based on the behavior data and the state data, the system comprising a second trajectory data acquisition unit, a latent variable update processing unit, and a second policy processing unit.

[0034] The second trajectory data acquisition unit causes the control target to execute an action according to the action data, and acquires time-series data of state data of a state transitioned to by the action, the action data, and reward data determined based on the action data and the state data.

[0035] The latent variable update processing unit performs an update process of the latent variables based on the time series data acquired by the second trajectory data acquisition unit.

[0036] The second policy processing unit is set with the optimal policy obtained by the reinforcement learning processing system of the fourth invention executing an optimal policy selection process, and executes policy processing to obtain behavioral data based on the latent variables and state data obtained from the environment.

[0037] Then, the latent variable update processing unit performs a process of optimizing the latent variables so as to maximize the profit.

[0038] In this reinforcement learning processing system, when the optimal policy obtained by the optimal policy selection processing is applied to an unknown environment (unknown domain) that is not the learning target, a processing is performed to optimize the latent variables so as to maximize profits (for example, TPE Bayesian optimization processing or Bayesian optimization processing using a Gaussian process, etc.). Then, in this reinforcement learning processing system, the latent variables obtained by the optimization processing of the latent variables are obtained as optimal latent variables, and processing is performed using the optimal latent variables and the optimal policy, thereby making it possible to appropriately control the controlled object (agent) even in an unknown domain (unknown environment) (even in an environment with different potential situations).

[0039] In other words, this reinforcement learning processing system acquires a strategy for appropriately controlling the control object even when there are subtle changes in the underlying situation, and reinforcement learning can be realized that enables appropriate control of the control object using the acquired strategy.

[0040] A sixth invention is a reinforcement learning processing method for performing reinforcement learning processing in which a control object placed in a predetermined environment is made to perform an action based on behavior data, and the control object is made to learn to perform a predetermined action based on state data of a state transitioned to by the action, the behavior data, and reward data determined based on the behavior data and the state data, the method comprising: a context acquisition step, an encoder step, and a policy processing step.

[0041] The context acquisition step acquires context data including (1) data capable of expressing differences in environments, and (2) data capable of expressing differences in tasks to be executed by the controlled object.

[0042] The encoder step performs encoding processing on the context data to obtain latent variables.

[0043] The policy processing step executes a policy process for acquiring behavior data based on the latent variables and state data acquired from the environment.

[0044] This makes it possible to realize a reinforcement learning processing method that has the same effects as the first aspect of the invention.

[0045] A seventh invention is a reinforcement learning processing method for performing reinforcement learning processing in which a control object placed in a predetermined environment is made to perform an action based on behavior data, and the control object is made to learn to perform a predetermined action based on state data of a state transitioned to by the action, the behavior data, and reward data determined based on the behavior data and the state data, the method comprising: a second trajectory data acquisition step; a latent variable update processing step; and a second policy processing step.

[0046] The second trajectory data acquisition step involves having the control target execute an action based on the action data, and acquiring time-series data of state data of a state transitioned to by the action, the action data, and reward data determined based on the action data and the state data.

[0047] The latent variable update processing step performs an update process of the latent variables based on the time series data acquired in the second trajectory data acquisition step.

[0048] The second policy processing step executes a policy process to obtain behavioral data based on latent variables and state data obtained from the environment, using the optimal policy obtained by the reinforcement learning processing system of the fourth invention executing an optimal policy selection process.

[0049] Then, the latent variable update processing step performs processing to optimize the latent variables so as to maximize profits.

[0050] This makes it possible to realize a reinforcement learning processing method that has the same effects as the fifth aspect of the invention.

[0051] An eighth aspect of the present invention is a program for causing a computer to execute the reinforcement learning processing method according to the sixth or seventh aspect of the present invention.

[0052] This makes it possible to realize a program for causing a computer to execute a reinforcement learning processing method that has the same effects as those of the sixth or seventh aspect of the present invention. Effect of the Invention

[0053] According to the present invention, it is possible to obtain a measure for appropriately controlling a control object even when there is a subtle change in the underlying situation, and to realize a reinforcement learning processing system, a reinforcement learning processing method, and a program that realizes reinforcement learning that enables appropriate control of the control object using the obtained measure. [Brief description of the drawings]

[0054] [Figure 1] FIG. 1 is a schematic configuration diagram of a reinforcement learning processing system 1000 according to a first embodiment. [Diagram 2] FIG. 1 is a schematic configuration diagram of an optimal policy acquisition device 100 of a reinforcement learning processing system 1000 according to a first embodiment. [Diagram 3] FIG. 1 is a schematic configuration diagram of an optimal policy adaptation device 100a of a reinforcement learning processing system 1000 according to a first embodiment. [Figure 4] 3 is a flowchart of a policy generation process executed in the reinforcement learning processing system 1000 according to the first embodiment. [Diagram 5] 3 is a flowchart of a policy selection process executed in the reinforcement learning processing system 1000 according to the first embodiment. [Figure 6] 3 is a flowchart of a policy selection process executed in the reinforcement learning processing system 1000 according to the first embodiment. [Figure 7] 4 is a flowchart of a policy adaptation process executed in the reinforcement learning processing system 1000 according to the first embodiment. [Figure 8] FIG. 13 is a diagram showing the results of comparison between the present invention and the prior art. [Figure 9] FIG. 13 is a diagram showing the results of a comparison between the present invention and the prior art (distribution diagram of revenue values). [Figure 10] A diagram showing the CPU bus configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] [First embodiment] The first embodiment will be described below with reference to the drawings.

[0056] <1.1: Configuration of reinforcement learning processing system> FIG. 1 is a schematic configuration diagram of a reinforcement learning processing system 1000 according to the first embodiment.

[0057] FIG. 2 is a schematic configuration diagram of an optimal policy acquisition device 100 of the reinforcement learning processing system 1000 according to the first embodiment.

[0058] FIG. 3 is a schematic configuration diagram of an optimal policy adaptation device 100a of the reinforcement learning processing system 1000 according to the first embodiment.

[0059] The reinforcement learning processing system 1000 is a system for performing reinforcement learning (multitask reinforcement learning), and includes an optimal policy acquisition device 100 and an optimal policy adaptation device 100a, as shown in FIG. 1. The reinforcement learning processing system 1000 is a system for performing learning processing so as to maximize an accumulated reward by observing a state that changes as an agent (for example, a control object Rbt1 in FIG. 1) acts in a predetermined environment, as shown in FIG. 1. For ease of explanation, in the reinforcement learning processing system 1000, as shown in FIG. 1, the agent of reinforcement learning is the control object Rbt1, and reinforcement learning (multitask reinforcement learning) is performed for processing in which the control object Rbt performs a heading motion to make a soccer ball thrown from a start position Pos_start reach a target position Pos_goal. It is assumed that a plurality of situations (multi-domains) can be realized in an environment by changing the start position Pos_start at which the soccer ball is thrown and / or the target position Pos_goal to which the soccer ball is to be reached.

[0060] (1.1.1: Configuration of the optimal policy acquisition device 100) The optimal policy acquisition device 100 is a device for acquiring an optimal policy (optimal policy) for each of a plurality of tasks by reinforcement learning (multitask reinforcement learning). As shown in Fig. 2, the optimal policy acquisition device 100 includes a state acquisition unit 1, a context acquisition unit 2, an encoder 3, a policy processing unit 4, a control target driving unit 5, a first trajectory data acquisition unit 6, a data storage unit DB1, an evaluation update processing unit 7, an optimal policy acquisition unit 8, and a policy candidate data storage unit DB2.

[0061] The state acquisition unit 1 acquires state acquisition data Ds from an environment (for example, the j-th domain (the environment in which a predetermined setting is performed)) via, for example, a predetermined sensor or the like (not shown). tThen, the status acquisition unit 1 acquires the status acquisition data Ds t From state data t The state data s t to the policy processor 4. The state acquisition data acquired at the time step t is called state acquisition data Ds t and the state data acquired at time step t is denoted as state data s t It is written as follows.

[0062] The context acquisition unit 2 acquires context acquisition data Dc from an environment (for example, the j-th domain (the environment in which a predetermined setting is performed)) via, for example, a predetermined sensor or the like (not shown). j Then, the context acquisition unit 2 acquires the context acquisition data Dc j From the context variable data j Retrieve the retrieved context variable data c j Data including data D1c j The context acquisition data acquired in the jth (j: natural number) domain (environment in a situation where a predetermined setting is performed) is referred to as context acquisition data Dc j Also, the context acquisition data Dc j It is preferable that the model includes (1) data capable of expressing differences in environments, and (2) data capable of expressing differences in tasks (eg, differences in objectives (goals)).

[0063] Encoder 3 uses the context variable data c j Encode the latent variable z j It is an encoder that acquires the above-mentioned data, and is realized, for example, by using a trainable model (for example, a neural network model) (referred to as an "encoder network model"). For ease of explanation, the parameter of the trainable model of the encoder 3 is set as a parameter η.

[0064] The encoder 3 receives the data D1c output from the context acquisition unit 2. j and the data Dq output from the evaluation update processing unit 7i (data for updating the parameter η (described later)) is input to the encoder 3. j The context variable data c contained in j Then, we perform the encoding process on the latent variable z j Then, the encoder 3 obtains the latent variable z j The data including the data Dz j The parameters of the learnable model of the encoder 3 are output to the policy processor 4 as the data Dq i It is updated based on (data for updating the parameter η).

[0065] The policy processor 4 uses the state data s t and the latent variable z j Then, act according to the policy t The policy processing unit 4 is a functional unit that performs a process (policy processing) to acquire (select) the policy. The policy processing unit 4 is realized, for example, by using a learnable model (for example, a neural network model) (referred to as a "policy network model"). For ease of explanation, the parameter of the learnable model of the policy processing unit is referred to as a parameter ψ.

[0066] The policy processor 4 receives the state data s output from the state acquisition unit 1. t and the data Dz output from encoder 3 j and the data Dπ output from the evaluation update processing unit 7 i (data for updating the parameter ψ (described later)) is input to the policy processor 4. t And, Data Dz j The latent variable z contained in j For each, a policy process (corresponding to π(a|s,z)) is performed, and the action data a at time step t is t For example, the policy processor 4 acquires state data s t And, Data Dz j The latent variable z contained in j and are input to a trainable model for policy processing (parameters: ψ), and the output from the trainable model for policy processing is the action data a at time step t. tThen, the policy processing unit 4 acquires the acquired action data a t to the controlled object driving unit 5. The parameters of the trainable model of the policy processing unit 4 are the data Dπ i It is updated based on (data for updating the parameter ψ).

[0067] The control target driving unit 5 receives the action data a output from the policy processing unit 4. t Enter the behavior data a t Data for controlling the control target Rbt1 (agent) (for example, torque data for driving each joint of the control target Rbt1) trq t Then, the controlled object drive unit 5 acquires the data trq t Based on this, the control target Rbt1 (agent) is controlled (for example, data trq t Based on the above, the actuators of each joint of the control object Rbt1 are driven to control the control object Rbt1 (agent) according to the action data a t (The person in question must take an action equivalent to the above.)

[0068] The first trajectory data acquisition unit 6 acquires context data c (c in the case of the jth domain) used by the optimal policy acquisition device 100. j ), the latent variable z (for the jth domain, z j ), state data s (for time step t, s t ), and behavioral data a (for time step t, a t The first trajectory data acquisition unit 6 applies a reward function r(s, a, c) (∈R (R: a set of real numbers)) to the state data s, the action data a, and the context data c, and obtains a reward r (for time step t, r t ) to get the

[0069] The first trajectory data acquisition unit 6 acquires (1) context data c, (2) latent variables z, and (3) state data s at time step t. t , Behavioral data a t , and reward data r tThe first trajectory data acquisition unit 6 acquires the data (time-series data) and stores it in the data storage unit DB1 as data D0_τ. If the data (time-series data) acquired by the first trajectory data acquisition unit 6 and stored in the data storage unit DB1 is trajectory data τ, the trajectory data τ can be expressed as, for example, τ={c,z,s0,a0,r0,s1,a1,r1,...,s T ,a T ,r T} It is.

[0070] The data storage unit DB1 is a functional unit that stores predetermined data in a predetermined memory area. The data storage unit DB1 reads data from the predetermined memory area and / or writes data to the predetermined memory area in accordance with a data read command and / or a data write command from the first trajectory data acquisition unit 6 and the evaluation update processing unit 7.

[0071] The evaluation update processing unit 7 is a functional unit that performs evaluation update processing based on trajectory data τ, and includes a Q-value evaluation unit 71 and an update processing unit 72. The evaluation update processing unit 7 reads out trajectory data τ stored in the data storage unit DB1 (inputs data D1_τ including trajectory data τ), and performs evaluation update processing based on the read trajectory data τ. Furthermore, when the evaluation update processing unit 7 determines that the parameters of the policy network model of the policy processing unit 4 have been optimized, it determines that a policy π for multiple domains that have been the learning target has been acquired, and inputs data on the policy π (this data is referred to as policy π k The data including the data Dπ k and outputs it to the optimal policy acquisition unit 8.

[0072] The Q-value evaluation unit 71 is a functional unit that performs processing to obtain a Q-value (action value) based on the trajectory data τ, and is realized, for example, by using a learnable model (for example, a neural network model) (referred to as a "Q-value network model"). For ease of explanation, the parameter of the learnable model of the Q-value evaluation unit 71 is set as a parameter φ.

[0073] In addition, the Q value evaluation unit 71 calculates a predicted value V of the state value from the acquired Q value. - The Q-value evaluation unit 71 performs a process of acquiring the Q-value Q(s, a, c; φ) acquired by the Q-value network model, the reward r(s, a, c) acquired from the trajectory data τ, and the predicted value V - Based on (s, c), the parameter φ of the Q-value network model that minimizes the Bellman residual is obtained. Then, the Q-value evaluation unit 71 updates the parameter of the Q-value network model based on the obtained parameter φ.

[0074] The update processing unit 72 acquires the parameters ψ of the policy network model and the parameters η of the encoder network model based on the trajectory data τ so as to maximize the state value taking into account the KL divergence between the encoder distribution of the encoder 3 and the prior distribution. Then, the update processing unit 72 converts data including the acquired parameters ψ (update parameters) into data Dπ i to the policy processor 4, and data including the acquired parameter η (update parameter) is output as data Dq i and output it to encoder 3.

[0075] The optimal policy acquisition unit 8 receives the data Dπ k And, policy π k The reward r obtained when the parameters of t , Strategy π t (a|s,z), and the encoder distribution q(z|c) (policy π k The optimal policy acquisition unit 8 receives data D_sel including the data of the optimal parameter η of the encoder network model of the encoder 3 acquired when the optimal policy π k The reward r obtained when the parameters of t , Strategy π t(a|s,z), the encoder distribution q(z|c), and the context data (context data c belonging to the context set C) are used to calculate the KL loss (KL loss calculation process), and the policy π k About KL loss k Then, the optimal policy acquisition unit 8 acquires the policy π k and the obtained policy π k About KL loss k Data paired with {π k, loss k} in the policy candidate data storage unit DB2. The optimal policy acquisition unit 8 stores K pieces of data {π k, loss k} (k: natural number, 1≦k≦K), the K pieces of data {π k, loss k} from the policy candidate data storage unit DB2, and the KL loss loss k The policy with the smallest π k The optimal policy π * Then, the optimal policy acquisition unit 8 selects (acquires) the acquired optimal policy π * The data including the data Dπ * The output is as follows.

[0076] The policy candidate data storage unit DB2 is a functional unit that stores predetermined data in a predetermined memory area. The policy candidate data storage unit DB2 reads data from the predetermined memory area and / or writes data to the predetermined memory area in accordance with a data read command and / or a data write command from the optimal policy acquisition unit 8.

[0077] (1.1.2: Configuration of the optimal policy adaptation device 100a) The optimal policy adaptation device 100a is a device for appropriately performing reinforcement learning processing for an unknown domain (unknown environment) by applying the optimal policy acquired by the optimal policy acquisition device 100. As shown in FIG. 3, the optimal policy adaptation device 100a includes a state acquisition unit 1a, a policy processing unit 4a, a control target driving unit 5a, a second trajectory data acquisition unit 10, and a latent variable update processing unit 11.

[0078] The state acquisition unit 1 a has the same configuration and functions as the state acquisition unit 1 of the optimal policy acquisition device 100 .

[0079] The policy processing unit 4a has the same configuration and function as the policy processing unit 4 of the optimal policy acquisition device 100. The policy network model of the policy processing unit 4a includes the optimal policy π * Parameter (optimal policy π * The optimal parameters set when * The policy processor 4a receives the state data s output from the state acquisition unit 1a. t and the latent variable z output from the latent variable update processing unit 11. The policy processing unit 4a receives the state data s t and the latent variable z and the optimal parameter ψ * A policy network model in which the optimal policy π * (a|s,z)) and the output data of the policy network model is converted into action data a t Then, the policy processing unit 4a acquires the acquired action data a t is output to the controlled object drive unit 5a.

[0080] The control target driving unit 5a has the same configuration and function as the control target driving unit 5 of the optimal measure acquisition device 100, and the action data a output from the measure processing unit 4a t Enter the behavior data a t Data for controlling the control target Rbt1 (agent) (for example, torque data for driving each joint of the control target Rbt1) trq t Then, the control target drive unit 5a acquires the data trq t Based on this, the control target Rbt1 (agent) is controlled (for example, data trq tBased on the above, the actuators of each joint of the control object Rbt1 are driven to control the control object Rbt1 (agent) according to the action data a t (The person in question must take an action equivalent to the above.)

[0081] The second trajectory data acquisition unit 10 acquires latent variables z and state data s (for time step t, s t ), and behavioral data a (for time step t, a t The second trajectory data acquisition unit 10 processes the state data s and the action data a using a reward function to obtain the reward r (for time step t, r t ) in time series. Then, the second trajectory data acquisition unit 10 outputs the data acquired in time series to the latent variable update processing unit 11 as data D2_τ(={z, r0, r1, r2,}).

[0082] The latent variable update processing unit 11 calculates the data D2_τ output from the second trajectory data acquisition unit 10 and the optimal policy π * The reward r obtained when the parameter of t , and policy π t The latent variable update processing unit 11 receives the data D2_τ and the reward r t , and policy π t Using (a|s,z), a process of updating the latent variable z (latent variable update process) is performed to obtain the updated latent variable z. The latent variable update processing unit 11 has, for example, an optimization model (parameter-updatable model) (for example, a Bayesian optimization model or a TPE Bayesian optimization model (TPE: tree-structured Parzen estimator)), and obtains the updated latent variable z by the optimization model. Then, the latent variable update processing unit 11 outputs the updated latent variable z to the policy processing unit 4a.

[0083] In addition, the latent variable update processing unit 11 generates history data {z k ,J(zk )}(z k : the kth acquired latent variable, J(): profit value) are stored in the history data storage unit DB3.

[0084] The history data storage unit DB3 is a functional unit that stores predetermined data in a predetermined memory area. The history data storage unit DB3 reads data from the predetermined memory area and / or writes data to the predetermined memory area in accordance with a data read command and / or a data write command from the latent variable update processing unit 11.

[0085] <1.2: Operation of the reinforcement learning processing system> The operation of the reinforcement learning processing system 1000 configured as above will be described below with reference to the drawings.

[0086] FIG. 4 is a flowchart of the policy generation process executed in the reinforcement learning processing system 1000 according to the first embodiment.

[0087] 5 and 6 are flowcharts of the policy selection process executed in the reinforcement learning processing system 1000 according to the first embodiment.

[0088] FIG. 7 is a flowchart of the policy adaptation process executed in the reinforcement learning processing system 1000 according to the first embodiment.

[0089] For ease of explanation, the operation of the reinforcement learning processing system 1000 will be explained for the case (one example) where multitask reinforcement learning is performed in the environment shown in Fig. 1. In Fig. 1, the following assumptions are assumed to hold. (1) It is assumed that the control object Rbt1 performs a heading motion for the soccer ball, which has been thrown from the starting position Pos_start, and performs an action for the purpose of making the soccer ball arrive at the goal position Pos_goal. (2) The start position Pos_start and / or the goal position Pos_goal can be set to any position. (3) The coefficient of friction between the soccer ball and the part of the control object Rbt1 that is being headed can be set to any value.

[0090] By changing the configurable values ​​of (1) to (3) above, it is possible to set multiple tasks.

[0091] (1.2.1: Contextual Markov Decision Process (cMDP)) First, as prior knowledge, we will explain the contextual Markov decision process (cMDP).

[0092] We, the inventors, have defined a contextual Markov decision process (cMDP) to formulate multi-task reinforcement learning in situations where the task objective and environment potentially change. Below, we explain a method for performing multi-task reinforcement learning using a contextual Markov decision process (cMDP).

[0093] <<Solving MDPs by minimizing KL divergence>> To search for an optimal policy in an entropy maximization reinforcement learning framework, the cumulative reward is considered as follows:

number

number

number

number

[0094] <<Solution of cMDP>> Next, we explain the solution method for cMDP. Specifically, we explain a variational inference-based RL method for solving cMDP.

[0095] A contextual Markov decision process (cMDP) has a state space S, an action space A, a context space C, a reward function r(s,a,c)∈R (R: real number), a state transition probability p(s'|s,a,c)∈R, and an initial distribution of states and contexts p0=p(s0)p(c)∈R. The state transition probability and the reward function are conditioned by a context c (context vector c) contained in the context space C. In a cMDP, to obtain an optimal policy, we need to learn a policy π(a|s,z) that depends on the state s as well as the latent variable z. Here, the latent variable z is a context-related feature vector sampled from the encoder distribution q(z|c). That is, the encoder converts the current (current time step) context variables into the latent variable z.

[0096] By processing according to the following formulas, the cMDP policy and encoder can be trained (the training process of the cMDP policy and encoder can be realized).

number

number

number

number

[0097] (1.2.2: Specific operation of the reinforcement learning processing system) Next, specific operations of the reinforcement learning processing system 1000 will be described with reference to the flowcharts of Figures 4 to 7. The specific operations of the reinforcement learning processing system 1000 will be described separately for (A) policy generation processing, (B) policy selection processing, and (C) policy adaptation processing.

[0098] <A: Policy generation process> First, there will be described the policy generation process executed by the reinforcement learning processing system 1000. The policy generation process is executed by the optimal policy acquisition device 100 of the reinforcement learning processing system 1000.

[0099] The environment is the jth domain Domain j The context variable c is the jth domain j The context variable c is assumed to be set to data equivalent to the following: Context variables: (1) Friction coefficient, (2) horizontal position of goal position Pos_goal, (3) vertical position of goal position Pos_goal, (4) horizontal position of soccer ball throwing position Pos_start, (5) vertical position of soccer ball throwing position Pos_start Furthermore, the observable data includes, for example, the following data: Observable data: (1) Center of gravity of the control object Rbt1, (2) Head position of the control object Rbt1, (3) Leg position of the control object Rbt1, (4) Angle of each joint of the control object Rbt1, (5) Angular velocity of each joint of the control object Rbt1, (6) Position of the soccer ball, (7) Speed ​​of the soccer ball, (8) Behavior data one time step before (Step S1): In step S1, an initialization process is executed. Specifically, the following process is executed.

[0100] A set D of trajectory data stored in the data storage unit DB1 of the optimal policy acquisition device 100 is set to an empty set. In addition, the encoder network model q(z|c;η) of the encoder 3 of the optimal policy acquisition device 100, the policy network model π(a|s,z;ψ) of the policy processing unit 4, and the Q-value network model Q(s,a,c;φ) of the Q-value evaluation unit 71 are initialized (each of the parameters η, ψ, and φ is set to an initial parameter).

[0101] (Step S2): In step S2, a loop process (loop 1 process) is started. The loop 1 process is repeatedly executed a number of times corresponding to one epoch of the training data (multitask reinforcement learning is possible by changing the context variable data c sampled from the context space C each time the loop 1 process is executed).

[0102] (Step S3): In step S3, a sample process is performed on the context variable data c (c~p(c)) and the latent variable z (z~q(z|c; η)) (the domain corresponding to the context variable data c sampled from the context space C is the target of the learning process).

[0103] Specifically, the context acquisition unit 2 of the optimal policy acquisition device 100 acquires the jth domain jContext acquisition data Dc is obtained from an environment in a state (domain corresponding to context variable data c sampled from context space C) via a predetermined sensor or the like (not shown). j Then, the context acquisition unit 2 acquires the context acquisition data Dc j From the context variable data j Retrieve the retrieved context variable data c j Data including data D1c j The context variable data c j It is assumed that the data includes (1) data that can represent differences in the environment, and (2) data that can represent differences in the tasks (e.g., differences in objectives (goals)).

[0104] The encoder 3 receives the data D1c output from the context acquisition unit 2. j Then, we perform the encoding process on the latent variable z j Specifically, data D1c is obtained. j The context variable data c contained in j is input to the encoder network model of the encoder 3, and the latent variable z j The latent variable z obtained in this way is j is a set of context variable data c that includes (1) data that can represent differences in the environment and (2) data that can represent differences in the task (e.g., differences in the purpose (goal)). j Since the data is obtained by encoding the above, it is data (latent variables) that can represent (1) differences in the environment and (2) differences in the task (e.g., differences in purpose (goals)).

[0105] Then, Encoder 3 extracts the latent variable z j The data including the data Dz j and outputs it to the policy processing unit 4.

[0106] (Step S4): In step S4, loop processing (loop 2 processing) is started. Note that the loop 2 processing is repeatedly executed for time steps t=0 to t=T, for example.

[0107] (Step S5): In step S5, a data collection process is executed. Specifically, the following process is executed.

[0108] The state acquisition unit 1 acquires the environment (the jth domain j ) from the state acquisition data Ds at time step t via a predetermined sensor or the like (not shown). t Then, the status acquisition unit 1 acquires the status acquisition data Ds t state data s at time step t t The state data s t is output to the policy processing unit 4.

[0109] The policy processor 4 receives the state data s output from the state acquisition unit 1. t and the data Dz output from encoder 3 j The latent variable z contained in j For each, a policy process (corresponding to π(a|s,z)) is performed, and the action data a at time step t is t For example, the policy processor 4 acquires state data s t And, Data Dz j The latent variable z contained in j and are input to a trainable model for policy processing (parameters: ψ), and the output from the trainable model for policy processing is the action data a at time step t. t Then, the policy processing unit 4 acquires the acquired action data a t is output to the controlled object drive unit 5.

[0110] The control target driving unit 5 receives the action data a output from the policy processing unit 4. t Enter the behavior data a t Data for controlling the control target Rbt1 (agent) (for example, torque data for driving each joint of the control target Rbt1) trqt Then, the controlled object drive unit 5 acquires the data trq t Based on this, the control target Rbt1 (agent) is controlled (for example, data trq t Based on the above, the actuators of each joint of the control object Rbt1 are driven to control the control object Rbt1 (agent) according to the action data a t (The person in question must take an action equivalent to the above.)

[0111] The first trajectory data acquisition unit 6 acquires context data c (c in the case of the jth domain) used by the optimal policy acquisition device 100. j ), the latent variable z (for the jth domain, z j ), state data s (for time step t, s t ), and behavioral data a (for time step t, a t The first trajectory data acquisition unit 6 applies a reward function r(s, a, c) (∈R (R: a set of real numbers)) to the state data s, the action data a, and the context data c, and obtains a reward r (for time step t, r t ) to get the

[0112] The first trajectory data acquisition unit 6 acquires (1) context data c (=c j ), (2) state data s at time step t t , Behavioral data a t , and reward data r t , and (3) state data s at time step t t+1 and adds the acquired data to the data set D (stores it in the data storage unit DB1). D←D∪{(s t ,a t ,r(s t ,a t ,c),s t+1 ,c)} Then, the trajectory data is added to the data set D. Note that the state data s t+1 The control object Rbt1 is given the action a tAt time step t+1 after the execution of the above, the state acquisition unit 1 acquires the state data and inputs it to the first trajectory data acquisition unit 6.

[0113] (Step S6): In step S6, the end of the loop process (loop 2 process) is determined. That is, if the time step t≦T, the process returns to step S4, the time step is advanced by one time step, and the process of step S5 is repeatedly executed. On the other hand, if the time step t>T, the process proceeds to step S7.

[0114] (Step S7): In step S7, a parameter update process is executed. Specifically, the following process is executed.

[0115] The evaluation update processing unit 7 reads out a data set D of trajectory data stored in the data storage unit DB1.

[0116] The Q-value evaluation unit 71 inputs state data s, action data a, and context data c included in the trajectory data included in the data set D into a Q-value network model, and obtains a Q-value Q(a,s,c;φ) output from the Q-value network model. Furthermore, the Q-value evaluation unit 71 calculates the state value V of the state s' to which the transition is made after the action a is performed. - (s', c) is obtained, and the process corresponding to the following formula is executed to update the parameter φ of the Q network model.

number

[0117] Further, the update processing unit 72 executes a process corresponding to the following formula using a data set D of trajectory data to update a parameter ψ of the policy network model and a parameter η of the encoder network model.

number

[0118] The updated parameters ψ of the policy network model and the updated parameters η of the encoder network model acquired by the above process are output from the evaluation update processing unit 7 to the policy processing unit 4 and the encoder 3. Then, the policy processing unit 4 sets (updates) the updated parameters ψ of the policy network model output from the evaluation update processing unit 7 to the parameters of the policy network model. Also, the encoder 3 sets (updates) the updated parameters η of the encoder network model output from the evaluation update processing unit 7 to the parameters of the policy network model.

[0119] (Step S8): In step S8, an end judgment process of the loop process (loop 1 process) is executed. That is, if the end judgment process judges that the end judgment condition is satisfied, the process proceeds to step S9. On the other hand, if the end judgment process judges that the end judgment condition is not satisfied, the process returns to step S2, and the processes of steps S3 to S7 are repeated. In this embodiment, it is judged that the end judgment condition is satisfied when the process is executed for the number of times corresponding to one epoch of the training data (as an example).

[0120] (Step S9): In step S9, a policy acquisition process is executed. That is, when the loop 1 process is completed, it can be determined that a policy π(z|s, z) having a predetermined accuracy has been acquired for a plurality of domains (domains of several epochs). Then, the optimal policy acquisition device 100, for example, converts the policy π(z|s, z) acquired as described above into a policy π kStore and store as (z|s,z) (k: natural number).

[0121] <B: Policy selection process> Next, a description will be given of the policy selection process executed by the reinforcement learning processing system 1000. The policy selection process is executed by the optimal policy acquisition device 100 of the reinforcement learning processing system 1000.

[0122] (Step S21): In step S21, an initialization process is executed. Specifically, the data {π k, loss k} is set to the empty set.

[0123] (Step S22): In step S22, a domain setting process is executed. Specifically, the context data c is randomly sampled from the context space C by num c pieces(num c : natural number). And this extracted num c The domain corresponding to each piece of context data c is the target of the policy selection process.

[0124] (Step S23): In step S23, a loop process (loop 3 process) is started. Note that the loop 3 process is repeatedly executed a predetermined number of times (for example, K times (K: natural number)).

[0125] (Step S24): In step S24, a policy generation process is executed. The policy generation process is executed by steps S241 to S249 as shown in the flowchart of Fig. 6, and steps S241 to S249 of the policy generation process (the process of step S24) are the same as steps S1 to S9 of the policy generation process described above, respectively.

[0126] In the policy generation process of step S24, the num sampled in step S22 is c Domains (num c The process is executed for the context data c) in step S24. That is, in the policy generation process in step S24, the loop 1A process is executed for num c The loop is executed 1A times, and in each loop 1A process, the num sampled in step S22 is c One piece of context data c included in this context data is sampled (processing of step S243), and processing is executed.

[0127] The policy obtained in the kth (k: natural number, 1≦k≦K) loop 3 process is called policy π k It is written as follows.

[0128] (Step S25): In step S25, a KL loss calculation process is executed. Specifically, the following process is executed.

[0129] The optimal policy acquisition unit 8 receives the data Dπ k And, policy π k The reward r obtained when the parameters of t , Strategy π t (a|s,z) and data D_sel including the encoder distribution q(z|c) are input. The optimal policy acquisition unit 8 receives the policy π k The parameter of the policy network model of the policy processor 4 is set in response to the parameter of the policy π k The parameters of the policy network model are set in the policy network model. k The optimal parameter η of the encoder network model of the encoder 3 obtained when is obtained is set in the encoder network model of the encoder 3. In response to this, the encoder 3 sets the strategy π k is obtained, the optimal parameter η of the encoder network model of the encoder 3 is set in the encoder network model.

[0130] In this state, the optimal policy acquisition device 100 samples num c Domains (num c For each context data c), the control target Rbt1 (agent) is given a policy π k The KL loss lossk is obtained by making the robot act based on the above formula, acquiring trajectory data, and executing the process equivalent to the formula below.

number

[0131] (Step S26): In step S26, a process for determining whether or not the loop process (loop 3 process) has ended is executed. That is, if the number of times that the loop 3 process has been executed is less than K, the process returns to step S4, k is incremented by +1, and the processes of steps S24 to S25 are executed repeatedly. On the other hand, if the number of times that the loop 3 process has been executed reaches K, the process proceeds to step S27.

[0132] (Step S27): In step S27, the optimal policy acquisition unit 8 calculates K pieces of data (K: natural number) {π k, loss k} (k: natural number, 1≦k≦K), the K pieces of data {π k, loss k} from the policy candidate data storage unit DB2, and the KL loss loss k The policy with the smallest π k The optimal policy π *Then, the optimal policy acquisition unit 8 selects (acquires) the acquired optimal policy π * The data including the data Dπ * The output is as follows.

[0133] <C: Policy Adaptation Processing> Next, the policy adaptation process executed in the reinforcement learning processing system 1000 will be described. The policy adaptation process is executed by the optimal policy adaptation device 100a of the reinforcement learning processing system 1000. For ease of explanation, in this embodiment, a case (one example) where the process is executed in the optimal policy adaptation device 100a using the TPE Bayesian optimization method will be described. In addition, when the policy adaptation process is executed, it is assumed that the environment is set to a domain other than the domain targeted by the multitask reinforcement learning in the optimal policy acquisition device 100 (i.e., an unknown domain (Domain_unknown)).

[0134] (Step S31): In step S31, an initialization process is executed. Specifically, the following process is executed.

[0135] The policy processing unit 4a of the optimal policy adaptation device 100a selects the optimal policy π * (Data Dπ * The optimal policy π contained in * ) parameters (optimum parameters) are set in the policy network model of the policy processor 4a.

[0136] The optimal policy adaptation device 100a acquires a history data set D, which is a set of history data (data stored in the history data storage unit DB3) used in the optimal policy adaptation. h Set to the empty set.

[0137] Moreover, the optimal policy adaptation device 100a sets an initial value of the latent variable z (for example, sets z=0).

[0138] (Step S32): In step S32, a loop process (loop 4 process) is started. The loop 4 process is, for example, a process in which a variable k=1 to a variable k=k max (k max : natural number), the variable k is incremented by +1 each time.

[0139] (Step S33): In step S33, a prediction process is performed using the TPE Bayesian optimization method. Specifically, the latent variable update processing unit 11 retrieves the history data set D h Read (acquire) the historical data {z k , J(z k )}, choose the latent variable z that maximizes the profit value J(). k and obtain the latent variable z k The latent variable update processing unit 11 updates the acquired latent variable z k (TPE predicted value) is output to the strategy processor 4a.

[0140] (Step S34): In step S34, a revenue calculation process is executed. Specifically, the following process is executed.

[0141] The optimal policy adaptation device 100a performs processing corresponding to the following formula to obtain the profit value J(z).

number

[0142] (Step S35): In step S35, the TPE model update process is executed. Specifically, the following process is executed.

[0143] The latent variable update processing unit 11 updates the profit value J(z) (=J(z k ) and the latent variable z used when obtaining the revenue value k Data paired with {z k ,J(z k )} into a historical data set D h That is, the latent variable update processing unit 11 adds D h ←D h ∪{z k ,J(z k )} (The latent variable update processing unit 11 executes a process equivalent to the data {z k ,J(z k )} is written to the history data storage unit DB3).

[0144] Then, the latent variable update processing unit 11 calculates the TPE model p(z|J)(p(z|J):D h = {(z, J)}}) is updated. In other words, the latent variable update processing unit 11 updates the data {z k ,J(z k )} is added to the historical data set D hUsing this, we make it possible to perform prediction processing using the TPE Bayesian optimization method (update the TPE model).

[0145] As the TPE Bayesian optimization method, the method disclosed in the following document may be adopted. (Reference A): J. Bergstra, R. Bardenet, Y. Bengio, and B. K´egl, "Algorithms for Hyper-Parameter Optimization," in Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and KQ Weinberger, Eds., vol. 24. Granada, Spain: Curran Associates, Inc., Dec, 12-17, 2011, pp. 2546-2554. (Step S36): In step S36, the end determination process of the loop process (loop 4 process) is executed. <k max If so, the process returns to step S32, the variable k is incremented by +1, and the processes of steps S33 to S35 are repeated. max If so, the process proceeds to step S37.

[0146] (Step S37): In step S37, an optimal policy acquisition process is executed. Specifically, the latent variable update processing unit 11 acquires the history data set D h Read the history data set D h Among them, the revenue value J(z k ) is maximized when the latent variable z k Let z be the optimal latent variable * That is, the latent variable update processing unit 11 executes the process corresponding to the following formula to obtain the optimal latent variable z * Get the.

number

[0147] <Summary> As described above, in the reinforcement learning processing system 1000, a policy for appropriately controlling a control target even when there is a subtle change in a potential situation is acquired, and reinforcement learning that allows the control target to be appropriately controlled by the acquired policy can be realized. That is, in the reinforcement learning processing system 1000, a policy process (processing by a policy π(a|s,z)) is performed using a latent variable z (a latent variable z reflecting an environmental difference (latent difference) or a task difference (latent difference)) acquired by encoding context data c including (1) data that can express an environmental difference (data related to the environment) and (2) data that can express a task difference (for example, a difference in a purpose (goal)) (data related to the task), so that a policy π(a|s,z) that is easy to apply even when there is an environmental difference (latent difference) or a task difference (latent difference) can be acquired. And, in the reinforcement learning processing system 1000, since the environment (domain) is changed and multiple policies are acquired by the above process, multiple policies adapted to various environments (domains) can be acquired. Then, in the reinforcement learning processing system 1000, the policy obtained as described above (policy π k ) among them, the policy that maximizes the average KL loss when applied to multiple environments (multiple domains) (when multiple context data c are applied) is called the optimal policy π * Then, in the reinforcement learning processing system 1000, the optimal policy π *When applying to an unknown environment (unknown domain) that is not the learning target, a process (for example, TPE Bayesian optimization process) is performed to optimize the latent variable z so as to maximize the profit. Then, in the reinforcement learning processing system 1000, the latent variable obtained by the optimization process of the latent variable z is called the optimal latent variable z * and the optimal latent variable z * and the optimal policy π * By processing using and, it is possible to appropriately control the control target Rbt1 (agent) even in an unknown domain (unknown environment) (even in an environment with potential differences in situations).

[0148] In this way, the reinforcement learning processing system 1000 can acquire a policy for appropriately controlling the control object even when there is a subtle change in the underlying situation, and can realize reinforcement learning that enables appropriate control of the control object using the acquired policy.

[0149] Here, the results of a comparative experiment between the present invention and the prior art will be presented.

[0150] In the comparative experiment, the control target Rbt1 (agent) is a single-legged robot (monopod), and the task is to have the single-legged robot head a soccer ball thrown from a given position and have it reach another given position (goal position). Note that the task is based on an unknown environment (domain) that is not the learning target.

[0151] FIG. 8 shows the results of a comparison between the application of the present invention and the application of a conventional technique (technique for determining a policy without using the latent variable z) to this task.

[0152] FIG. 8(a) is a diagram showing the control object and target position (shown by a large sphere) and the ideal trajectory position of a soccer ball (shown by a small sphere) when the present invention is applied, at intervals of 0.2 s for the period from time t=1.0 s to t=2.0 s.

[0153] FIG. 8(b) shows the control object and target position (shown by a large sphere) and the ideal trajectory position of the soccer ball (shown by a small sphere) when applying conventional technology, every 0.2 s for the period from time t=1.0 s to t=2.0 s.

[0154] FIG. 9 shows the distribution of profit values ​​when the present invention is applied and the distribution of profit values ​​when the conventional technology is applied.

[0155] The reward function r was calculated using the following formula:

number

[0156] [Other embodiments] In the above embodiment, the reinforcement learning processing system 1000 has been described as having a configuration including the optimal policy acquisition device 100 and the optimal policy adaptation device 100a. However, the present invention is not limited to this. For example, the optimal policy acquisition device 100 and the optimal policy adaptation device 100a may be configured to share common functional units, etc.

[0157] In the above embodiment, a case has been described in which the policy generation process is executed while the policy selection process is being executed in the reinforcement learning processing system 1000. However, this is not limited to this. For example, the policy selection process may be executed first to obtain multiple policy candidates, and then the KL loss calculation process may be performed for the policy candidates already obtained in the policy generation process.

[0158] Furthermore, in the reinforcement learning processing system 1000 of the above embodiment, each block may be individually implemented as a single chip using a semiconductor device such as an LSI, or some or all of the blocks may be included in a single chip.

[0159] Although we refer to it as LSI here, it may also be called IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0160] The method of integration is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. It is also possible to use a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI.

[0161] In addition, a part or all of the processing of each functional block in each of the above embodiments may be realized by a program. Then, a part or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. Also, the programs for performing each processing are stored in a storage device such as a hard disk or a ROM, and are executed in the ROM or by being read into the RAM.

[0162] In addition, each process in the above embodiment may be realized by hardware, or may be realized by software (including the case where it is realized together with an OS (operating system), middleware, or a predetermined library). Furthermore, it may be realized by a mixed process of software and hardware.

[0163] Furthermore, for example, when each functional unit of the above embodiment (including the modified examples) is realized by software, each functional unit may be realized by software processing using the hardware configuration shown in FIG. 10 (for example, a hardware configuration in which a CPU (which may be a GPU), ROM, RAM, input units, output units, etc. are connected via a bus).

[0164] Furthermore, when each functional unit of the above embodiment (variation example) is realized by software, the software may be realized using a single computer having the hardware configuration shown in FIG. 10, or may be realized by distributed processing using multiple computers.

[0165] In addition, the execution order of the processing method in the above embodiment is not necessarily limited to the description of the above embodiment, and the execution order can be changed without departing from the scope of the invention. In addition, in the processing method in the above embodiment, some steps may be executed in parallel with other steps without departing from the scope of the invention.

[0166] The scope of the present invention includes a computer program for causing a computer to execute the above-mentioned method and a computer-readable recording medium having the program recorded thereon. Here, examples of the computer-readable recording medium include a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a large-capacity DVD, a next-generation DVD, and a semiconductor memory.

[0167] The computer program is not limited to one recorded on the recording medium, but may be one transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, or the like.

[0168] In the above embodiment (including the modified examples), "the same" is a concept including being roughly the same. "At the same time" is a concept including being roughly the same. "The same" is a concept including being roughly the same.

[0169] In the description of this specification and the claims, "optimization" refers to bringing the system into the best state, and a parameter for "optimizing" a system (model) refers to a parameter when the value of the objective function of the system is the optimal value. The "optimum value" is a maximum value when the system is in a better state as the value of the objective function of the system increases, and is a minimum value when the system is in a better state as the value of the objective function of the system decreases. The "optimum value" may also be an extreme value. The "optimum value" may also be one that allows for a predetermined error (measurement error, quantization error, etc.), and may be a value included in a predetermined range (a range that can be considered to have converged sufficiently).

[0170] The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the gist of the invention. [Explanation of symbols]

[0171] 1000 Reinforcement Learning Processing System 100 Optimal policy acquisition device 100a Optimal policy application device 2 Context Acquisition Section 3 Encoder 4, 4a Policy Processing Unit 6 First trajectory data acquisition section 7 Evaluation update processing section 8. Optimal Policy Acquisition Section 10 Second trajectory data acquisition unit 11 Latent variable update processing section

Claims

1. A reinforcement learning processing system for performing a reinforcement learning process in which a control object placed in a predetermined environment is made to execute an action according to action data, and the control object is made to learn to execute a predetermined action based on state data of a state transitioned by the action, the action data, and reward data determined based on the action data and the state data, (1) a context acquisition unit that acquires context data including data that can represent a difference in the environment and (2) data that can represent a difference in a task to be executed by the control target; An encoder that performs an encoding process on the context data to obtain latent variables; a policy processing unit that executes a policy process to acquire behavior data based on the latent variables and state data acquired from the environment; A reinforcement learning processing system comprising:

2. a first trajectory data acquisition unit that causes the control target to execute an action according to action data, and acquires time-series data of state data of a state transitioned by the action, the action data, and reward data determined based on the action data and the state data; An evaluation update processing unit that performs evaluation update processing for the reinforcement learning processing; Further equipped with The encoder has an encoder network model that is a model that can be trained by updating a parameter η; The policy processing unit has a policy network model that is a model that can be learned by updating a parameter ψ, The evaluation update processing unit: a Q network model that is a model that can learn by updating a parameter φ, and a parameter update process is executed to update the parameter φ of the Q network model, the parameter η of the encoder network model, and the parameter ψ of the policy network model by evaluating the time-series data acquired by a first trajectory data acquisition unit; The encoder updates the encoder network model using the parameter η updated by the evaluation update processing unit; The policy processing unit updates the policy network model using the parameter ψ updated by the evaluation update processing unit. The reinforcement learning processing system according to claim 1 .

3. the context acquisition unit acquires the context data from one of a plurality of domains set by changing, in the environment, a setting that affects the environment and / or a setting that affects a task to be executed by the control target; The encoder obtains the latent variables by performing an encoding process on the context data; The policy processing unit acquires behavioral data based on the latent variables and state data acquired from the domain; and the first trajectory data acquisition unit causes the control target to execute an action according to action data, and acquires time-series data of state data of a state transitioned by the action, the action data, and reward data determined based on the action data and the state data; The evaluation update processing unit executes the parameter update processing based on the time series data, the context acquisition unit, the encoder, the policy processing unit, the first trajectory data acquisition unit, and the evaluation update processing unit execute a policy generation process that is realized by repeatedly executing the above process for all of a plurality of set domains; the evaluation update processing unit sets the parameter ψ of the policy network model acquired after the policy generation process is completed as a multi-domain corresponding parameter corresponding to the plurality of domains, and acquires a policy corresponding to the multi-domain corresponding parameter as a multi-domain corresponding policy. The reinforcement learning processing system according to claim 2 .

4. An optimal policy acquisition unit for selecting an optimal policy is further provided, Executing the policy generation process multiple times to generate multiple multi-domain policies; The optimal policy acquisition unit acquires an average value of a KL loss, which is a loss when a reinforcement learning process is executed for each of the generated multiple multi-domain policies, derived from a KL divergence between a cumulative reward and a prior distribution of a latent variable and a conditional distribution of the latent variable based on context data, and executes an optimal policy selection process to select the multi-domain policy with the maximum average value of the KL loss as an optimal policy. The reinforcement learning processing system according to claim 3 .

5. A reinforcement learning processing system for performing a reinforcement learning process in which a control object placed in a predetermined environment is made to execute an action according to action data, and the control object is made to learn to execute a predetermined action based on state data of a state transitioned by the action, the action data, and reward data determined based on the action data and the state data, a second trajectory data acquisition unit that causes the control target to execute an action according to action data, and acquires time-series data of state data of a state transitioned to by the action, the action data, and reward data determined based on the action data and the state data; A latent variable update processing unit that performs a latent variable update process based on the time series data acquired by the second trajectory data acquisition unit; a second policy processing unit that executes a policy process to acquire action data based on the latent variables and state data acquired from the environment, and in which an optimal policy acquired by executing the optimal policy selection process is set by the reinforcement learning processing system according to claim 4; Equipped with The latent variable update processing unit: Optimizing the latent variables so as to maximize profits. Reinforcement learning processing system.

6. A reinforcement learning processing method for performing a reinforcement learning process in which a control object placed in a predetermined environment is made to execute an action according to action data, and the control object is made to learn to execute a predetermined action based on state data of a state transitioned by the action, the action data, and reward data determined based on the action data and the state data, (1) a context acquisition step of acquiring context data including data capable of expressing a difference in the environment and (2) data capable of expressing a difference in a task to be executed by the control target; An encoder step of performing an encoding process on the context data to obtain latent variables; a policy processing step of executing a policy processing to acquire behavior data based on the latent variables and state data acquired from the environment; A reinforcement learning processing method comprising:

7. A reinforcement learning processing method for performing a reinforcement learning process in which a control object placed in a predetermined environment is made to execute an action according to action data, and the control object is made to learn to execute a predetermined action based on state data of a state transitioned by the action, the action data, and reward data determined based on the action data and the state data, a second trajectory data acquisition step of making the control target execute an action according to action data, and acquiring time-series data of state data of a state transitioned to by the action, the action data, and reward data determined based on the action data and the state data; a latent variable update processing step of performing a latent variable update processing based on the time series data acquired by the second trajectory data acquisition step; a second policy processing step of executing a policy process to acquire action data based on the latent variables and state data acquired from the environment, using the optimal policy acquired by executing the optimal policy selection process according to the reinforcement learning processing system according to claim 4; Equipped with The latent variable update processing step includes: Optimizing the latent variables so as to maximize profits. Reinforcement learning processing method.

8. A program for causing a computer to execute the reinforcement learning processing method according to claim 6 or 7.

Images