Learning device, learning method, and learning program

Abstract

To appropriately and efficiently execute training of learning models including hyper parameters.SOLUTION: A learning device comprises a processing unit for training an agent's learning models. The learning models include hyper parameters. The learning includes imitation learning of training the hyper parameters of the learning models so that the agent executes a predetermined behavior in a predetermined state in a predetermined environment, and a reinforcement learning of training the hyper parameters of the learning models so that a reward given to the agent in the predetermined environment becomes maximum. The processing unit executes a step of setting parameter values of the hyper parameters of the learning models, a step of executing the imitation learning by using the learning models after the setting, a step of evaluating the learning models after the imitation learning to extract the learning models after the evaluation, and a step of executing the reinforcement learning by using the leaning models after the extraction.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a multi-agent learning device, a learning method, and a learning program. [Background technology]

[0002] Conventionally, a learning device and a learning method are known that learn a policy by imitation learning based on learning data (see, for example, Patent Document 1). In imitation learning, a reward function corresponding to the functional form of the reward is learned. Specifically, in imitation learning, learning data for imitation learning that associates states with actions is used to learn a policy, which is a rule for selecting the agent's action. As a result, by using the learned policy, the agent outputs the optimal imitation action according to the state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 115904 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, a neural network is used as the learning model as a strategy. A neural network is a network in which multiple nodes are connected. Neural network parameters include weights and biases between nodes, and these parameters are adjusted through learning. Neural network parameters also include hyperparameters that are set in advance, such as the number of layers, the number of nodes, and the learning rate. Generally, hyperparameters are set based on empirical guesses, so there is a possibility that a learning model with inappropriate hyperparameters may be constructed. For this reason, it is necessary to set hyperparameters, perform reinforcement learning, and then evaluate the learning model for each combination of hyperparameters, which is time-consuming.

[0005] Therefore, an object of the present disclosure is to provide a learning device, a learning method, and a learning program that can appropriately and efficiently execute learning of a learning model including hyperparameters. [Means for solving the problem]

[0006] The learning device of the present disclosure is a learning device that includes a processing unit for learning a learning model of an agent, wherein the learning model includes hyperparameters, and the learning includes imitation learning, in which the hyperparameters of the learning model are learned so that the agent performs a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning, in which the agent learns the hyperparameters of the learning model so that a reward given to the agent in a predetermined environment is maximized, and the processing unit executes the steps of setting parameter values ​​of the hyperparameters of the learning model, performing the imitation learning using the learning model after setting, evaluating the learning model after the imitation learning and extracting the learning model after evaluation, and performing the reinforcement learning using the learning model after extraction.

[0007] The learning method of the present disclosure is a learning method for learning a learning model of an agent using a learning device, wherein the learning model includes hyperparameters, and the learning includes imitation learning, in which the hyperparameters of the learning model are learned so that the agent performs a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning, in which the agent learns the hyperparameters of the learning model so that a reward given to the agent in a predetermined environment is maximized, and the learning device is caused to perform the following steps: setting parameter values ​​of the hyperparameters of the learning model; performing the imitation learning using the learning model after setting; evaluating the learning model after the imitation learning and extracting the learning model after evaluation; and performing the reinforcement learning using the learning model after extraction.

[0008] The learning program of the present disclosure is a learning program for learning an agent's learning model using a learning device, wherein the learning model includes hyperparameters, and the learning includes imitation learning, in which the hyperparameters of the learning model are learned so that the agent performs a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning, in which the agent learns the hyperparameters of the learning model so that the reward given to the agent in a predetermined environment is maximized, and causes the learning device to execute the following steps: setting parameter values ​​of the hyperparameters of the learning model; performing the imitation learning using the learning model after setting; evaluating the learning model after the imitation learning and extracting the learning model after evaluation; and performing the reinforcement learning using the learning model after extraction. [Effects of the Invention]

[0009] According to the present disclosure, learning of a learning model including hyperparameters can be performed appropriately and efficiently. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram regarding learning using a learning model according to this embodiment. [Figure 2] FIG. 2 is a diagram schematically illustrating the learning device according to this embodiment. [Figure 3] FIG. 3 is a diagram showing a flow relating to the learning method according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical. Furthermore, the components described below can be combined as appropriate, and when there are multiple embodiments, the respective embodiments can also be combined.

[0012] [Embodiment] The learning device 10 and learning method according to this embodiment are a device and method for learning a learning model including hyperparameters. FIG. 1 is an explanatory diagram of learning using the learning model according to this embodiment. FIG. 2 is a diagram schematically illustrating the learning device according to this embodiment. FIG. 3 is a diagram showing a flow relating to the learning method according to this embodiment.

[0013] (Learning using learning models) First, learning using a learning model M will be described with reference to Fig. 1. The learning model M is installed in an agent 5 that executes an action At. The agent 5 may be, for example, a machine capable of executing an action, such as a robot, a vehicle, a ship, or an aircraft. The agent 5 executes a predetermined action At in a predetermined environment 6 using the learning model M.

[0014] As shown in FIG. 1, the learning model M is a neural network having multiple nodes. The neural network is a network in which multiple nodes are connected, has multiple layers, and each layer has multiple nodes. The parameters of the neural network include weights and biases between nodes. The parameters of the neural network also include hyperparameters such as the number of layers, the number of nodes, and a learning rate. In this embodiment, in learning the learning model M, learning is performed on the learning model M including the hyperparameters.

[0015] Next, learning using the learning model M will be explained. There are two types of learning: imitation learning and reinforcement learning. Imitation learning is supervised learning, in which the hyperparameters of the learning model M are learned so that the agent 5 executes a predetermined action At when a predetermined state St is input in a predetermined environment 6. Reinforcement learning is unsupervised learning, in which the weights and biases between nodes in the learning model M are learned so that the reward Rt given to the agent 5 in the predetermined environment 6 is maximized.

[0016] In reinforcement learning, the agent 5 acquires a state St from the environment 6 (an environment unit 22, described later), and also acquires a reward Rt from the environment 6. The agent 5 then selects an action At from the learning model M based on the acquired state St and reward Rt. When the agent 5 executes the selected action At, the state St of the agent 5 transitions to state St+1 in the environment 6. The agent 5 is also given a reward Rt+1 based on the executed action At, the state St before the transition, and the state St+1 after the transition. Then, in reinforcement learning, the above learning is repeated a predetermined number of evaluable steps so that the reward Rt given to the agent 5 is maximized.

[0017] In imitation learning, training data that associates a predetermined state St with a predetermined action At is used. The training data is, for example, data on a predetermined action At selected by an expert operating the agent 5 when the agent reaches a predetermined state St in a predetermined environment 6, and is also called expert data. In other words, the expert data is data that associates a predetermined state St with a predetermined action At. In imitation learning, when the predetermined state St of the training data is input to the learning model M of the agent 5, the weights and biases between nodes in the learning model M are learned so that the action At output from the learning model M becomes the predetermined action At of the training data.

[0018] Here, environment 6 is an environment where reward Rt is sparse. An environment where reward Rt is sparse is an environment where reward Rt is unlikely to be given for an action At performed by agent 5, and where the time between action At and when reward Rt is given is sparse (there is a long time interval). In other words, an environment where reward Rt is sparse is an environment where rewards are not given immediately for an action At performed by agent 5, but are given after a certain time period or after subsequent actions At+1 to At+n.

[0019] (Learning device) Next, the learning device 10 will be described with reference to FIG. 2. The learning device 10 performs imitation learning and reinforcement learning for an agent 5 in a virtual space environment. The learning device 10 includes an input unit 21, an environment unit 22, and a learning unit 23. The environment unit 22 and the learning unit 23 each include a processing unit for learning a learning model of the agent, and a storage unit for storing various data used in learning. The processing unit includes, for example, an integrated circuit such as a CPU (Central Processing Unit). The storage unit 12 is any storage device such as a semiconductor storage device or a magnetic storage device. The hardware configuration of the learning device 10 is not particularly limited.

[0020] The input unit 21 is an input device such as a keyboard and a mouse. The input unit 21 is connected to the environment unit 22 and the learning unit 23, and inputs teacher data used in imitation learning to the environment unit 22 and the learning unit 23. Examples of teacher data such as expert data include a rule base D1 that is predefined to take a predetermined action in a predetermined state, and human operation data D2 that is an action taken by an expert in a predetermined state.

[0021] The environment unit 22 provides the agent 5 with an environment in which rewards are sparse. Specifically, the environment unit 22 grants a reward Rt to the agent 5 and derives the state St of the agent 5 to which the action At transitions. The environment unit 22 stores various models such as a motion model Ma, an environment model Mb, and a battle model Mc. The motion model Ma, the environment model Mb, and the battle model Mc are models that use the action At performed by the agent 5 as input to calculate the state St of the agent 5, which is the output. The calculated state St is output to the learning unit 23. The environment unit 22 also stores a reward model Md for calculating rewards. The reward model Md is a model that uses the action At performed by the agent 5, the state St, and the transition destination state St+1 as input to calculate the reward Rt to be granted to the agent 5, which is the output. The calculated reward Rt is output to the learning unit 23.

[0022] The learning unit 23 executes learning of the learning model M. The learning unit 23 executes imitation learning and reinforcement learning as learning. The learning unit 23 has an imitation learning unit 31 that performs imitation learning, an imitation learning evaluation unit 32 that evaluates the learning result of the imitation learning, and a reinforcement learning unit 33 that performs reinforcement learning. The learning unit 23 also generates an imitation learning model M as a learning result obtained by imitation learning. 0~N and a database 35 that stores the reinforcement learning model R obtained as a result of reinforcement learning. 0~N and a database 36 that stores the

[0023] The imitation learning unit 31 uses teacher data such as expert data to train the learning model M while updating hyperparameters so that the agent 5 executes a predetermined action At when it is in a predetermined state St under a predetermined environment. In addition, the imitation learning unit 31 executes learning of the learning model M, and generates an imitation learning model M that is the learning result of the imitation learning. 0~NThe imitation learning model M obtained for each update of the hyperparameter is stored in the database 35. If the initial value of the hyperparameter is 0 and the update value is N, the database 35 stores the imitation learning models M0 to M1. N The learning model up to is stored.

[0024] The imitation learning evaluation unit 32 evaluates the imitation learning model M stored in the database 35. 0~N The imitation learning evaluation unit 32 evaluates the behavior At based on the training data and the imitation learning model M 0~N Specifically, the imitation learning evaluation unit 32 evaluates a predetermined state St in the teacher data by comparing it with the imitation learning model M 0~N and the imitation learning model M 0~N It is determined whether the action At output from the training data matches the predetermined action At in the training data. As an index of the degree of match, for example, KL divergence (KL distance) is used. KL divergence is calculated by comparing the probability distribution of the state St and the action At in the training data with the imitation learning model M 0~N The KL divergence is an index showing the similarity between the probability distribution of the state St and the action At in the state, and the similarity is high if the KL divergence is small, and low if the KL divergence is large. If the KL divergence is smaller than a preset threshold, the imitation learning evaluation unit 32 evaluates the similarity between the action At based on the teacher data and the imitation learning model M. 0~N On the other hand, if the KL divergence is equal to or greater than a preset threshold, the imitation learning evaluation unit 32 determines that the behavior At based on the teacher data does not match the behavior At output from the imitation learning model M, and evaluates the imitation learning model M. 0~N Assume that imitation by the

[0025] The reinforcement learning unit 33 performs learning based on the reward Rt given by the environment unit 22. The reinforcement learning unit 33 also performs learning based on the imitation learning model M that has been evaluated by the imitation learning evaluation unit 32 as having realized imitation. x Specifically, the reinforcement learning unit 33 updates the hyperparameters so as to maximize the reward Rt given to each agent 5, while learning the imitation learning model M x The reinforcement learning unit 33 executes the reinforcement learning of the imitation learning model M x By executing reinforcement learning, the reinforcement learning model R is obtained as the learning result of reinforcement learning. 0~N and the reinforcement learning model R obtained for each hyperparameter update. 0~N are stored in the database 36. If the initial value of the hyperparameter is 0, the updated value is N, and the initial step of the learning step T is 0 and the final step is S, the database 35 stores reinforcement learning models R0 to R1. N The learning models up to are stored, and each reinforcement learning model R 0~N In this case, from learning step T0 to learning step T S The reinforcement learning model R up to this point is stored.

[0026] (Learning method) Next, a learning method executed by the learning device 10 will be described with reference to Fig. 3. In the learning method, first, a step of setting parameter values ​​of hyperparameters of the learning model M in the learning device 10 is executed (step S1). In step S1, the parameter values ​​of the hyperparameters are set arbitrarily, and may be set based on the learning results of imitation learning, for example.

[0027] Next, in the learning method, a step of executing imitation learning using the learning model M after the setting is performed (step S2). In step S2, the imitation learning unit 31 of the learning device 10 executes imitation learning of the learning model M for which the parameter values ​​of the hyperparameters have been set in step S1, using the teacher data. Also, in step S2, the imitation learning unit 31 executes imitation learning of the imitation learning model M obtained by executing imitation learning.0~N are stored in the database 35.

[0028] After this, the learning method executes a step (step S3) of evaluating the imitation learning model M after imitation learning. In step S3, the imitation learning evaluation unit 32 of the learning device 10 determines whether or not a predetermined state St and a predetermined action At in the teacher data match a predetermined state St and a predetermined action At in the imitation learning model M, thereby determining whether or not imitation by the imitation learning model M has been achieved (whether or not imitation learning has been achieved). In step S3, the imitation learning evaluation unit 32 determines that imitation learning has been achieved (step S3: Yes), and thus evaluates the imitation learning model M as having achieved imitation learning. X is extracted as the reinforcement learning model R. On the other hand, if the imitation learning evaluation unit 32 determines in step S3 that imitation learning has not been achieved (step S3: No), the process returns to step S1, where the parameter values ​​of the hyperparameters are reset. In other words, in steps S1 to S3, one imitation learning model M that is evaluated as having achieved imitation learning is extracted. X is being explored and acquired.

[0029] In this embodiment, steps S1 to S3 are performed using one imitation learning model M X However, the imitation learning model M X and multiple imitation learning models M X may be extracted as a reinforcement learning model R.

[0030] After step S3, the learning method executes a step of setting parameter values ​​for hyperparameters of the reinforcement learning model R (step S4). In step S4, the parameter values ​​for the hyperparameters are set arbitrarily. Here, the types of hyperparameters set in step S1 (imitation learning) and the types of hyperparameters set in step S4 (reinforcement learning) may be different or the same, and are not particularly limited. Specifically, in step S1, the number of layers and the number of nodes in each layer are set as types of hyperparameters, and in step S4, in addition to the types of hyperparameters in step S1, a discount rate γ, a random action probability ε, etc. are further set.

[0031] Then, in the learning method, a step of executing reinforcement learning using the set (extracted) reinforcement learning model R is carried out (step S5). In step S5, the reinforcement learning unit 33 of the learning device 10 executes reinforcement learning of the reinforcement learning model R for which the parameter values ​​of the hyperparameters are set in step S4 so as to maximize the reward Rt given to the agent 5. Also, in step S5, the reinforcement learning unit 33 executes reinforcement learning of the reinforcement learning model R obtained by executing reinforcement learning. 0~N are stored in the database 36.

[0032] Next, in the learning method, the reinforcement learning unit 33 of the learning device 10 determines whether the learning step T has reached the final step S (step S6). In step S6, if the reinforcement learning unit 33 determines that the learning step T has reached the final step S (step S6: Yes), reinforcement learning is terminated, and the series of learning methods is terminated. Note that after the reinforcement learning is terminated, the trained reinforcement learning model R 0~N may be evaluated, and the reinforcement learning model R may be selected based on the evaluation result. On the other hand, if the reinforcement learning unit 33 determines in step S6 that the learning step T has not reached the final step S (step S6: No), the process proceeds to step S4, and the process is repeated until the final step S is reached.

[0033] In this way, the learning unit 23, which executes steps S1 to S6 above, functions as a processing unit for performing imitation learning and reinforcement learning on the agent 5. A learning program P for executing the above learning method is stored in the memory unit of the learning device 10.

[0034] As described above, the learning device 10, the learning method, and the learning program P described in this embodiment can be understood, for example, as follows.

[0035] The learning device 10 according to the first aspect is a learning device 10 including a processing unit for learning a learning model M of an agent 5, wherein the learning model M includes hyperparameters, and the learning includes imitation learning for learning the hyperparameters of the learning model M so that the agent 5 performs a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning for learning the hyperparameters of the learning model M so that the reward given to the agent 5 in a predetermined environment is maximized. The processing unit executes the steps of: step S1 of setting parameter values ​​of the hyperparameters of the learning model M; step S2 of performing the imitation learning using the learning model M after the setting; step S3 of evaluating the learning model M after the imitation learning and extracting the learning model M after evaluation; and step S5 of performing the reinforcement learning using the learning model M after extraction.

[0036] According to this configuration, it is possible to extract a learning model M that has a high evaluation in imitation learning, and to perform reinforcement learning on the extracted learning model M. In other words, since reinforcement learning can be performed using a learning model trained with hyperparameters that have high imitation properties, it is possible to appropriately and efficiently perform learning of a learning model that includes hyperparameters.

[0037] In a second aspect, the environment is an environment in which the reward is sparse.

[0038] With this configuration, even in an environment where rewards are sparse, it is possible to train a learning model including hyperparameters by imitation learning. Therefore, even when training based on rewards is performed in reinforcement learning, it is possible to appropriately and efficiently train a learning model including hyperparameters.

[0039] As a third aspect, in the step S3 of extracting the learning model, if the evaluation result in the step of evaluating the learning model M satisfies an evaluation index, one learning model that satisfies the evaluation index is extracted.

[0040] According to this configuration, by extracting one learning model M based on the evaluation index, there is no need to perform reinforcement learning for multiple learning models M, which makes it possible to improve the efficiency of reinforcement learning.

[0041] As a fourth aspect, in step S1 of setting the parameter values, if the evaluation result in the step of evaluating the learning model M does not satisfy the evaluation index, new parameter values ​​are set.

[0042] According to this configuration, imitation learning can be performed until hyperparameters that satisfy the evaluation index are found.

[0043] In a fifth aspect, in step S3 of extracting the learning model, an evaluation is performed based on the degree of correspondence between the specified state and specified behavior learned by the imitation learning in a specified environment and the specified state and specified behavior in the learning model in a specified environment.

[0044] According to this configuration, it is possible to evaluate whether or not the learning model has achieved imitation based on the degree of coincidence between a predetermined state and a predetermined action.

[0045] In a sixth aspect, the degree of coincidence is calculated using KL divergence.

[0046] According to this configuration, the degree of agreement between actions can be appropriately evaluated by using KL divergence (KL distance).

[0047] A learning method according to a seventh aspect is a learning method for learning a learning model M of an agent 5 using a learning device 10, wherein the learning model M includes hyperparameters, and the learning includes imitation learning in which the hyperparameters of the learning model M are learned so that the agent 5 performs a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning in which the hyperparameters of the learning model M are learned so that the reward given to the agent 5 in a predetermined environment is maximized. The learning method causes the learning device 10 to execute the following steps: step S1 of setting parameter values ​​of the hyperparameters of the learning model M; step S2 of performing the imitation learning using the learning model M after the setting; step S3 of evaluating the learning model M after the imitation learning and extracting the learning model M after evaluation; and step S5 of performing the reinforcement learning using the learning model M after extraction.

[0048] According to this configuration, it is possible to extract a learning model M that has a high evaluation in imitation learning, and to perform reinforcement learning on the extracted learning model M. In other words, since reinforcement learning can be performed using a learning model trained with hyperparameters that have high imitation properties, it is possible to appropriately and efficiently perform learning of a learning model that includes hyperparameters.

[0049] The learning program P according to the eighth aspect is a learning program P for learning a learning model M of an agent 5 using a learning device 10, wherein the learning model M includes hyperparameters, and the learning includes imitation learning in which the hyperparameters of the learning model M are learned so that the agent 5 performs a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning in which the hyperparameters of the learning model M are learned so that the reward given to the agent 5 in a predetermined environment is maximized. The learning program P causes the learning device 10 to execute the following steps: step S1 of setting parameter values ​​of the hyperparameters of the learning model M; step S2 of executing the imitation learning using the learning model M after the setting; step S3 of evaluating the learning model M after the imitation learning and extracting the learning model M after evaluation; and step S5 of executing the reinforcement learning using the learning model M after extraction.

[0050] According to this configuration, it is possible to extract a learning model M that has a high evaluation in imitation learning, and to perform reinforcement learning on the extracted learning model M. In other words, since reinforcement learning can be performed using a learning model trained with hyperparameters that have high imitation properties, it is possible to appropriately and efficiently perform learning of a learning model that includes hyperparameters. [Explanation of symbols]

[0051] 5 Agent 10 Learning Device 21 Input section 22 Environment Department 23 Learning Department 31 Imitation Learning Department 32 Imitation Learning Evaluation Department 33 Reinforcement Learning Department 35 databases 36 databases D1 Rule Base D2 Human operation data M Learning Model Ma movement model Mb Environment Model Mc Battle Model Md Reward Model P Study Program

Claims

1. A learning device including a processing unit for learning a learning model of an agent, the learning model includes hyperparameters; the learning includes imitation learning, which learns the hyperparameters of the learning model so that the agent executes a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning, which learns the hyperparameters of the learning model so that a reward given to the agent in a predetermined environment is maximized; The processing unit setting parameter values ​​of the hyperparameters of the learning model; performing the imitation learning using the learning model after the setting; evaluating the learning model after the imitation learning and extracting the learning model after evaluation; and performing the reinforcement learning using the extracted learning model.

2. The learning device according to claim 1 , wherein the environment is an environment in which the reward is sparse.

3. In the step of extracting the learning model, 3. The learning device according to claim 1, wherein, when the evaluation results of the learning models satisfy an evaluation index, one of the learning models that satisfies the evaluation index is extracted.

4. In the step of setting the parameter values, The learning device according to claim 1 , wherein if the evaluation result in the step of evaluating the learning model does not satisfy the evaluation index, new parameter values ​​are set.

5. In the step of extracting the learning model, The learning device according to claim 1, wherein evaluation is performed based on the degree of agreement between a predetermined state and a predetermined action learned by the imitation learning in a predetermined environment and a predetermined state and a predetermined action in the learning model in a predetermined environment.

6. The learning device according to claim 5 , wherein the degree of agreement is calculated using KL divergence.

7. A learning method for learning an agent learning model using a learning device, comprising: the learning model includes hyperparameters; the learning includes imitation learning, which learns the hyperparameters of the learning model so that the agent executes a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning, which learns the hyperparameters of the learning model so that a reward given to the agent in a predetermined environment is maximized; The learning device includes: setting parameter values ​​of the hyperparameters of the learning model; performing the imitation learning using the learning model after the setting; evaluating the learning model after the imitation learning and extracting the learning model after evaluation; a step of performing the reinforcement learning using the extracted learning model.

8. A learning program for learning a learning model of an agent using a learning device, the learning model includes hyperparameters; the learning includes imitation learning, which learns the hyperparameters of the learning model so that the agent executes a predetermined action in a predetermined state in a predetermined environment, and reinforcement learning, which learns the hyperparameters of the learning model so that a reward given to the agent in a predetermined environment is maximized; The learning device includes: setting parameter values ​​of the hyperparameters of the learning model; performing the imitation learning using the learning model after the setting; evaluating the learning model after the imitation learning and extracting the learning model after evaluation; and a learning program that executes the step of performing the reinforcement learning using the extracted learning model.

Images