Learning device, learning method, and learning program

Abstract

To provide a learning device, a learning method, and a learning program with which a high-performance learning model can be appropriately selected from learning models during training.SOLUTION: A learning device 10 is to train an agent's learning models. A learning unit comprises: a reinforced learning unit that trains the learning models so that a reward given to the agent in a predetermined environment becomes maximum; an evaluation index value calculation unit that calculates a first index value and a second index value of the learning models; and a model extraction unit that extracts the learning models with a predetermined number or more of learning steps as learned models. The model extraction unit selects, from the learned models, the learned models in which the first index value and the second index value each satisfy a predetermined condition as learned models to be evaluated.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, and a learning program. [Background technology]

[0002] In machine learning, there is known a technique for extracting a machine learning model with high prediction accuracy from multiple machine learning models. For example, Patent Document 1 describes a technique for automatically extracting a more accurate learning model from multiple learning models at high speed while reducing the load on computer resources. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6801149 Summary of the Invention [Problem to be solved by the invention]

[0004] In reinforcement learning, learning is unstable in problems with sparse rewards or problems with high-dimensional states, and therefore, convergence to a high-performance model does not necessarily occur as learning progresses. Problems with sparse rewards require the designer to design dense rewards based on empirical rules to promote learning. Depending on the reward, the model may learn to obtain a reward rather than what it should achieve. In this case, the learned model may converge to a model with a high reward but low performance. On the other hand, even if convergence is not achieved or convergence is achieved to an unintended model, it is possible to obtain a model with temporarily good performance during the learning process. When evaluating a model during training in detail, it is necessary to temporarily stop the training process and check its generalization performance using test data. This poses the problem of requiring enormous computational time and occupying a large amount of computer resources.

[0005] Therefore, an object of the present disclosure is to provide a learning device, a learning method, and a learning program that can appropriately select a learning model with good performance from among learning models currently being trained. [Means for solving the problem]

[0006] The learning device of the present disclosure is a learning device for learning a learning model of an agent, and includes a reinforcement learning unit that learns the learning model so as to maximize the reward given to the agent in a predetermined environment, an evaluation index value calculation unit that calculates a first index value and a second index value of the learning model, and a model extraction unit that extracts the learning model with a predetermined number of learning steps or more as a trained model, and the model extraction unit selects, from the trained models, the trained models whose first index value and second index value each satisfy predetermined conditions as the trained models to be evaluated.

[0007] The learning method disclosed herein is a learning method for learning an agent's learning model using a learning device, and includes the steps of: training the learning model so that the reward given to the agent in a predetermined environment is maximized; calculating a first index value and a second index value of the learning model; extracting the learning model with a predetermined number of learning steps or more as trained models; and selecting, from the trained models, the trained models whose first index value and second index value each satisfy predetermined conditions as the trained models to be evaluated.

[0008] The learning program disclosed herein is a learning program for learning an agent's learning model using a learning device, and causes the learning device to execute the following steps: training the learning model so that the reward given to the agent in a predetermined environment is maximized; calculating a first index value and a second index value of the learning model; extracting the learning model with a predetermined number of learning steps or more as trained models; and selecting, from the trained models, the trained models whose first index value and second index value each satisfy predetermined conditions as the trained models to be evaluated. [Effects of the Invention]

[0009] According to the present disclosure, a learning model with good performance can be appropriately selected from among learning models currently being trained. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram regarding learning using a learning model according to this embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 3] FIG. 3 is a diagram for explaining a comparative example. [Figure 4] FIG. 4 is a flowchart illustrating an example of a trained model extraction process according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating a method for selecting a trained model according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical. Furthermore, the components described below can be combined as appropriate, and when there are multiple embodiments, the respective embodiments can also be combined.

[0012] [Embodiment] The learning device 10 and learning method according to this embodiment are a device and method for learning a learning model including hyperparameters. Fig. 1 is an explanatory diagram of learning using the learning model according to this embodiment. Fig. 2 is a block diagram showing an example of the configuration of the learning device according to this embodiment.

[0013] (Learning using learning models) First, learning using a learning model M will be described with reference to Fig. 1. The learning model M is installed in an agent 2 that executes an action At. The agent 2 may be, for example, a machine capable of executing an action, such as a robot, a vehicle, a ship, or an aircraft. The agent 2 executes a predetermined action At in a predetermined environment 4 using the learning model M.

[0014] As shown in FIG. 1, the learning model M is a neural network having multiple nodes. The neural network is a network in which multiple nodes are connected, has multiple layers, and each layer has multiple nodes. Parameters of the neural network include weights and biases between nodes. Other parameters of the neural network include hyperparameters such as the number of layers, the number of nodes, and a learning rate. In this embodiment, the weights and biases between nodes of the learning model M are learned.

[0015] Next, learning using the learning model M will be explained. There are two types of learning: imitation learning and reinforcement learning. Imitation learning is supervised learning, in which the hyperparameters of the learning model M are learned so that the agent 2 executes a predetermined action At when a predetermined state St is input in a predetermined environment 4. Reinforcement learning is unsupervised learning, in which the weights and biases between nodes of the learning model M are learned so that the reward Rt given to the agent 2 in the predetermined environment 4 is maximized.

[0016] In reinforcement learning, agent 2 acquires state St from environment 4, and also acquires reward Rt from environment 4. Agent 2 then selects action At from learning model M based on the acquired state St and reward Rt. When agent 2 executes the selected action At, state St of agent 2 transitions to state St+1 in environment 4. Agent 2 is also given a reward Rt+1 based on the executed action At, the state St before the transition, and the state St+1 after the transition. Then, in reinforcement learning, the above learning is repeated a predetermined number of evaluable steps so that the reward Rt given to agent 2 is maximized.

[0017] The learning device 10 performs reinforcement learning of the behavior of an agent in a competitive environment, regardless of whether the environment is symmetric or asymmetric. In a problem for which reinforcement learning is attempted, the learning device 10 extracts in advance models during learning that are thought to have good performance using evaluation indices (accumulated reward, cumulative win rate, etc.) obtained during learning, and after learning, evaluates only the models extracted during learning.

[0018] (Comparative Example) Before describing this embodiment, a comparative example of this embodiment will be described below. Fig. 3 is a diagram for explaining the comparative example.

[0019] Figure 3 is a graph showing an example of the results of reinforcement learning, where the horizontal axis represents the number of steps and the vertical axis represents the cumulative reward.

[0020] In the technology according to the comparative example, for example, based on graph G1 as shown in Figure 3, it is estimated which step of the trained model has high performance, and trained models are extracted at regular step intervals. Then, in the technology according to the comparative example, the extracted trained model is pitted against an opponent that serves as an evaluation standard multiple times, and the performance of the trained model is evaluated, for example, by the win rate against the opponent.

[0021] In the technology of the comparative example, the trained model may converge to a "model with high earned reward but low performance." Therefore, in the evaluation of the comparative example, as shown in FIG. 3, for example, in a section within range R1 where the cumulative reward is relatively high, the training process must be temporarily stopped during training and generalization performance must be confirmed using test data. For this reason, the technology of the comparative example requires a huge amount of calculation time and occupies a large amount of computer resources. Furthermore, in the technology of the comparative example, even if a model with good performance is temporarily obtained during the training process, it is discarded without being evaluated.

[0022] (Learning device) Returning to Fig. 2, as shown in Fig. 2, the learning device 10 includes an environment unit 12, a storage unit 14, and a control unit 16.

[0023] The environment unit 12 provides an environment for executing reinforcement learning on the trained model. The environment unit 12 has an exercise model 20, a competition model 22, an environment model 24, and a reward model 26. The environment unit 12 provides an environment for executing reinforcement learning based on the exercise model 20, the competition model 22, the environment model 24, and the reward model 26. Specifically, the environment unit 12 provides rewards for the trained model and derives the state of the trained model that transitions depending on actions.

[0024] The storage unit 14 is a memory that stores various types of information. The storage unit 14 stores, for example, information such as the calculation contents of the control unit 16 and programs. The storage unit 14 includes, for example, at least one of a RAM (Random Access Memory), a main storage device such as a ROM (Read Only Memory), and an external storage device such as an HDD (Hard Disk Drive). The storage unit 14 stores a reinforcement learning model 30.

[0025] The reinforcement learning model 30 includes a plurality of trained models in reinforcement learning. The reinforcement learning model 30 stores, for example, a plurality of trained models trained for each learning step.

[0026] The control unit 16 controls the operation of each unit of the learning device 10. The control unit 16 is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing a program stored in the storage unit 14 or the like using RAM or the like as a work area. The control unit 16 may be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 16 may also be realized by a combination of hardware and software.

[0027] The control unit 16 includes a setting unit 40 and a learning unit 42.

[0028] The setting unit 40 sets various conditions for executing machine learning, such as a behavioral judgment model (state and behavior), a reward function, a deep reinforcement learning algorithm, model granularity, and hyperparameters.

[0029] The learning unit 42 executes learning of the learning model and includes a reinforcement learning unit 50, an evaluation index value calculation unit 52, and a model extraction unit 54.

[0030] The reinforcement learning unit 50 performs learning based on the reward given by the environment unit 12. Details of the reinforcement learning unit 50 will be described later.

[0031] The evaluation index value calculation unit 52 calculates an evaluation index value for evaluating the performance of the trained model. Specifically, the evaluation index value calculation unit 52 calculates two evaluation index values: a first evaluation index value for evaluating the performance of the trained model, and a second evaluation index value different from the first evaluation index value. The evaluation index value calculation unit 52 may calculate three or more evaluation index values. Details of the evaluation index value calculation unit 52 will be described later.

[0032] The model extraction unit 54 extracts a trained model that satisfies a predetermined condition from among a plurality of trained models. The model extraction unit 54 extracts the trained model based on, for example, the evaluation index value calculated by the evaluation index value calculation unit 52. Details of the model extraction unit 54 will be described later.

[0033] (Trained model extraction process) The trained model extraction process according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the trained model extraction process according to the embodiment.

[0034] The setting unit 40 sets hyperparameters for executing reinforcement learning (step S10), and then proceeds to step S12. In this embodiment, it is assumed that appropriate values ​​for the hyperparameters to be set have been considered in advance.

[0035] The reinforcement learning unit 50 executes reinforcement learning (step S12). Specifically, the reinforcement learning unit executes learning so as to maximize the reward given to the trained model, for example. Then, the process proceeds to step S14.

[0036] The evaluation index value calculation unit 52 calculates the evaluation index value (step S14). Specifically, the evaluation index value calculation unit 52 calculates the value of the cumulative reward and the value of the cumulative winning percentage as the evaluation index value. Then, the process proceeds to step S16.

[0037] The reinforcement learning unit 50 determines whether the number of steps for which reinforcement learning has been performed is equal to or greater than a predetermined number of steps (step S16). The predetermined number of steps may be set arbitrarily depending on the problem being addressed, etc. If it is determined that the number of steps is equal to or greater than the predetermined number of steps (step S16; Yes), the process proceeds to step S18. If it is determined that the number of steps is not equal to or greater than the predetermined number of steps (step S16; No), the process proceeds to step S12. That is, in this embodiment, reinforcement learning and the process of calculating the evaluation index value are repeated until the predetermined number of steps is reached.

[0038] If the determination in step S16 is Yes, the model extraction unit 54 extracts trained models (step S18). Specifically, the model extraction unit 54 extracts all trained models that have undergone reinforcement learning for a predetermined number of steps or more. Then, the process proceeds to step S20.

[0039] The reinforcement learning unit 50 determines whether the number of steps for which reinforcement learning has been performed has reached the maximum number of steps (step S20). The maximum number of steps may be set arbitrarily depending on the problem being addressed, etc. If it is determined that the number of steps has reached the maximum number of steps (step S20; Yes), the process proceeds to step S22. If it is determined that the number of steps has not reached the maximum number of steps (step S20; No), the process proceeds to step S18.

[0040] If the determination in step S20 is Yes, the model extraction unit 54 selects a trained model (step S22). Specifically, the model extraction unit 54 extracts a trained model whose cumulative reward value and cumulative winning percentage value both satisfy predetermined conditions. FIG. 5 is a diagram for explaining a method for selecting a trained model according to an embodiment. The upper diagram in FIG. 5 is a graph showing the number of learning steps on the horizontal axis and the cumulative reward on the vertical axis. The lower diagram in FIG. 5 is a graph showing the number of learning steps on the horizontal axis and the cumulative winning percentage on the vertical axis. In FIG. 5, as shown in graphs G1 and G2, range R11 indicates a range in which the slope of the cumulative winning percentage with respect to the number of learning steps is positive and the derivative value is equal to or greater than a predetermined value. As shown in graphs G1 and G2, range R12 indicates a range in which the slope of the cumulative winning percentage with respect to the number of learning steps is positive and the derivative value is equal to or greater than a predetermined value, and indicates a range in which the cumulative reward value is equal to or greater than a predetermined value. In this case, the model extraction unit 54 selects a trained model in range R12 as the trained model to be evaluated. For example, the model extraction unit 54 selects a trained model in range R12 as the trained model to be evaluated. The model extraction unit 54 discards a trained model for which, for example, both the cumulative reward value and the cumulative winning percentage value do not satisfy predetermined conditions. Then, the process proceeds to step S24. The model extraction unit 54 may store the selected trained model in the storage unit 14.

[0041] The learning unit 42 evaluates the selected trained model (step S24). Then, the processing in FIG. 4 ends. In this embodiment, model extraction can be performed without checking generalization performance during learning, which makes it possible to reduce the evaluation time during learning. Furthermore, in this embodiment, candidate models are extracted and saved in advance, which makes it possible to reduce the evaluation time after learning.

[0042] The learning device, learning method, and learning program described in this embodiment can be understood, for example, as follows.

[0043] A first embodiment of the learning device 10 is a learning device for training a learning model of an agent. The learning device 10 includes a reinforcement learning unit 50 that trains the learning model so as to maximize the reward given to the agent in a predetermined environment, an evaluation index value calculation unit 52 that calculates a first index value and a second index value of the learning model, and a model extraction unit that extracts, as trained models, learning models with a predetermined number of learning steps or more. The model extraction unit 54 selects, from among the learned models, learning models whose first index value and second index value each satisfy predetermined conditions as the learning models to be evaluated. This allows the learning device of the first embodiment to appropriately select learning models with good performance from among learning models currently being trained. Furthermore, the learning device of the first embodiment can perform model extraction without checking generalization performance during training, thereby reducing evaluation time during training. Furthermore, the learning device of the first embodiment can reduce evaluation time after learning by extracting and saving candidate models in advance.

[0044] In the learning device of the second aspect, the evaluation index value calculation unit 52 calculates the cumulative winning percentage value and cumulative reward value of the learning model. As a result, the learning device of the second aspect can use the cumulative winning percentage value and cumulative reward value of the learning model as index values ​​for evaluating the performance of the learning model.

[0045] In the learning device of the third aspect, the model extraction unit 54 selects a trained model whose cumulative reward value is equal to or greater than a predetermined value as a trained model to be evaluated. This allows the learning device of the third aspect to more appropriately select a trained model with good performance from among the trained models currently being trained.

[0046] In the learning device of the fourth aspect, the model extraction unit 54 selects the trained models to be evaluated from those in which the slope of the cumulative winning percentage value relative to the number of learning steps is positive and the differential value is equal to or greater than a predetermined value. This allows the learning device of the fifth aspect to more appropriately select trained models with good performance from among the trained models currently being trained.

[0047] The fifth aspect of the learning device is a learning method for using the learning device to learn a learning model of an agent, and includes the steps of: training the learning model so that the reward given to the agent in a predetermined environment is maximized; calculating a first index value and a second index value of the learning model; extracting learning models with a predetermined number of learning steps or more as trained models; and selecting, from the trained models, trained models whose first index value and second index value each satisfy predetermined conditions as trained models to be evaluated.

[0048] The learning program of the sixth aspect is a learning program for using a learning device to learn a learning model of an agent, and causes the learning device to execute the following steps: learning the learning model so that the reward given to the agent in a predetermined environment is maximized; calculating a first index value and a second index value of the learning model; extracting the learning model with a predetermined number of learning steps or more as a trained model; and selecting from the trained models those trained models whose first index value and second index value each satisfy predetermined conditions as trained models to be evaluated.

[0049] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0050] 10 Learning Device 12 Environment Department 14 Storage section 16 Control Unit 20 Exercise Model 22 Battle Model 24 Environmental Model 26 Remuneration Model 30 Reinforcement Learning Models 40 Setting section 42 Learning Department 50 Reinforcement Learning Department 52 Evaluation index value calculation unit 54 Model Extraction Unit

Claims

1. A learning device for learning a learning model of an agent, comprising: a reinforcement learning unit that trains the learning model so that a reward given to the agent in a predetermined environment is maximized; an evaluation index value calculation unit that calculates an accumulated winning rate value and an accumulated reward value of the learning model; a model extraction unit that extracts the learning model having a predetermined number of learning steps or more as a learned model, The model extraction unit selects, from the trained models, a trained model whose accumulated winning rate value and accumulated reward value each satisfy predetermined conditions as the trained model to be evaluated.

2. The model extraction unit selects the trained model having the cumulative reward value equal to or greater than a predetermined value as the trained model to be evaluated. The learning device according to claim 1 .

3. The model extraction unit selects, as the trained model to be evaluated, the trained model in which the slope of the cumulative winning percentage value with respect to the number of learning steps is positive and the differential value is equal to or greater than a predetermined value. The learning device according to claim 1 or 2.

4. A learning method for learning an agent learning model using a learning device, comprising: A step of training the learning model so that the reward given to the agent in a predetermined environment is maximized; Calculating an accumulated winning rate value and an accumulated reward value of the learning model; extracting the learning model having a predetermined number of learning steps or more as a trained model; selecting, from the trained models, the trained models whose cumulative winning rate value and cumulative reward value each satisfy predetermined conditions as the trained models to be evaluated; including, learning methods.

5. A learning program for learning a learning model of an agent using a learning device, A step of training the learning model so that the reward given to the agent in a predetermined environment is maximized; Calculating an accumulated winning rate value and an accumulated reward value of the learning model; extracting the learning model having a predetermined number of learning steps or more as a trained model; selecting, from the trained models, the trained models whose cumulative winning rate value and cumulative reward value each satisfy predetermined conditions as the trained models to be evaluated; A learning program that causes the learning device to execute the above.

Images