Learning device, learning method, and learning program

Abstract

To efficiently perform reinforcement learning of a plurality of agents under an asymmetrical environment.SOLUTION: A learning device is provided in which reinforcement learning of an operation of an agent is performed on the basis of self-playing under a multi-agent environment, the multi-agent environment is an asymmetrical environment in which at least one of a type of an action performed by the agent, a type of a state acquired by the agent and definition of reward given to the agent is different, and an evaluation index of learning is given to each of agents. The learning device performs the steps of: performing learning of a predetermined agent of the plurality of agents using a learning model; acquiring an evaluation index in the learning model of the predetermined agent after learning; comparing the evaluation index in the predetermined agent with an evaluation index in other agent; and setting an agent having a low evaluation index as a learning target.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a multi-agent learning device, a learning method, and a learning program. [Background technology]

[0002] A system that provides a Go game service trained by deep learning as a form of reinforcement learning among multiple agents is known. In this system, learning is performed by self-play using a situation judgment model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-013750 Summary of the Invention [Problem to be solved by the invention]

[0004] In the reinforcement learning of Patent Document 1, since it is a Go game service, the learning conditions between multiple agents are the same. On the other hand, in reinforcement learning between multiple agents, the learning conditions such as rewards may differ between the multiple agents. In this case, due to the difference in learning conditions, the learning progress of a specific agent may diverge from the learning progress of other agents, which may slow down the learning progress. In addition, due to the difference in learning conditions, during the learning of a specific agent, the behavior of other agents may not contribute to the learning of the specific agent, and the learning of the specific agent may not progress. In this way, when the learning conditions between multiple agents are different, there is a possibility that the learning efficiency of reinforcement learning may decrease.

[0005] Therefore, an object of the present disclosure is to provide a learning device, a learning method, and a learning program that can efficiently execute reinforcement learning of multiple agents in an asymmetric environment. [Means for solving the problem]

[0006] The learning device of the present disclosure is a learning device that has a processing unit for performing reinforcement learning of the behavior of an agent through self-play in a multi-agent environment in which a plurality of agents exist, wherein the multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards granted to the agents is different between the agents, and each of the agents is assigned an evaluation index for learning, and the processing unit executes the steps of: performing learning of a predetermined agent among the plurality of agents using a learning model; acquiring the evaluation index in the learning model of the predetermined agent after learning; comparing the evaluation index of the predetermined agent with the evaluation index of the other agents; and setting the agent with the lower evaluation index as the learning target.

[0007] The learning method disclosed herein is a learning method for performing reinforcement learning of the behavior of an agent through self-play in a multi-agent environment in which a plurality of agents exist, the multi-agent environment being an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards granted to the agents is different among the agents, and each of the agents is assigned an evaluation index for learning, and the method executes the following steps: performing learning of a predetermined agent among the plurality of agents using a learning model; acquiring the evaluation index in the learning model for the predetermined agent after learning; comparing the evaluation index for the predetermined agent with the evaluation index for the other agents; and setting the agent with the lower evaluation index as a learning target.

[0008] The learning program of the present disclosure is a learning program to be executed by a learning device in a multi-agent environment in which a plurality of agents exist, in order to cause the agents to perform reinforcement learning of their behavior through self-play, wherein the multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards granted to the agents are different among the agents, and each of the agents is assigned an evaluation index for learning, and the learning device is caused to execute the steps of: executing learning of a predetermined agent among the plurality of agents using a learning model; acquiring the evaluation index in the learning model of the predetermined agent after learning; comparing the evaluation index for the predetermined agent with the evaluation index for the other agents; and setting the agent with the lower evaluation index as a learning target. Effect of the Invention

[0009] According to the present disclosure, reinforcement learning for multiple agents in an asymmetric environment can be efficiently performed. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a schematic diagram of a learning system including a learning device according to the present embodiment. [Diagram 2] FIG. 2 is an explanatory diagram regarding the learning method according to the present embodiment. [Diagram 3] FIG. 3 is a diagram showing a flow relating to the learning method according to the present embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a multi-agent environment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a multi-agent environment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially the same. Furthermore, the components described below can be appropriately combined, and when there are multiple embodiments, each embodiment can be combined.

[0012] [Embodiment] The learning device 10 and the learning method according to the present embodiment are devices and methods for performing reinforcement learning for each agent 5 in an environment in which a plurality of agents 5 performing actions exist, i.e., in a multi-agent environment. The agent 5 may be, for example, a machine capable of performing an action, such as a robot, a vehicle, a ship, or an aircraft.

[0013] In this embodiment, the multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by agents 5, the types of states acquired by agents 5, and the definitions of rewards granted to agents 5 differ between agents 5.

[0014] In this embodiment, the multi-agent environment is, for example, a competition environment in which a kicker agent 5 and a goalkeeper agent 5 play an FK (Free Kick) competition. In the following description, the multi-agent environment is applied to an FK competition environment, that is, an asymmetric competition environment, but is not particularly limited as long as it is an asymmetric environment. In other words, as long as it is an asymmetric environment, it may be a cooperative environment in which a plurality of agents 5 cooperate with each other.

[0015] Fig. 1 is a diagram showing a schematic diagram of a system including a learning device according to the present embodiment. Fig. 2 is an explanatory diagram of a learning method according to the present embodiment. Fig. 3 is a diagram showing a flow relating to the learning method according to the present embodiment. Fig. 4 is a diagram showing an example of a multi-agent environment. Fig. 5 is a diagram showing an example of a multi-agent environment.

[0016] (system) As shown in Fig. 1, the learning device 10 is a device for learning a learning model installed in a plurality of robots 7 provided in the system 1. The system 1 is in an asymmetric environment, and includes a plurality of robots 7 that are targets of a plurality of agents 5, and a learning device 10 for learning the movements of the plurality of robots 7.

[0017] The robots 7 include a kicker robot 7a for kickers and a keeper robot 7b for goalkeepers. In this embodiment, since the environment is a free kick match environment, two opposing robots are used, but depending on the environment, three or more agents 5 may be included.

[0018] Each robot 7 has a processing unit 11, a storage unit 12, a sensor 13, and an actuator 14. The processing unit 11 includes an integrated circuit such as a CPU (Central Processing Unit). The processing unit 11 executes motion control based on a learning model. The storage unit 12 is any storage device such as a semiconductor storage device and a magnetic storage device. The storage unit 12 stores a learning model. Specifically, the storage unit 12 of the kicker robot 7a stores a kicker model (kicker model N) which is a learning model for a kicker. Also, the storage unit 12 of the keeper robot 7b stores a keeper model (keeper model M) which is a learning model for a keeper. The sensor 13 acquires a state (St: state) of the robot 7. The sensor 13 is connected to the processing unit 11 and outputs the acquired state St to the processing unit 11. The sensor 13 is, for example, a speed sensor, an acceleration sensor, or the like. The actuator 14 is an operating unit that causes the robot 7 to execute a predetermined motion. The actuator 14 is connected to the processing unit 11, and is controlled in operation by the processing unit 11 to execute an action (At).

[0019] When the state St is input from the sensor 13, the processing unit 11 of each robot 7 selects a predetermined operation (At: action) using a learning model based on the state St, and executes operation control of the actuator 14.

[0020] The learning model stored in the memory unit 12 of each robot 7 is a model learned by a learning device 10, which will be described later.

[0021] (Learning device) The learning device 10 executes reinforcement learning of a plurality of agents 5 in a multi-agent environment that is a virtual space. In the learning device 10, the actions of the agents 5 are reinforced learned through self-play. The learning device 10 includes a plurality of agents 5, an environment unit 25, and a storage unit 23.

[0022] The agents 5 include a kicker agent 5a for the kicker and a keeper agent 5b for the keeper. Each agent 5 has a learning unit 31, a database 32, and a processing unit 33. The learning unit 31, database 32, and processing unit 33 of the kicker agent 5a may be integrated with the learning unit 31, database 32, and processing unit 33 of the keeper agent 5b, and the hardware configuration is not particularly limited.

[0023] The learning unit 31 executes learning of the learning model. The learning unit 31 executes learning based on a reward (Rt) given by the environment unit 25. Specifically, the learning unit 31 executes learning so as to maximize the reward given to each agent 5.

[0024] The database 32 is a storage device that stores the learning model after learning. The database 32 stores the learning model every time learning is performed, thereby accumulating the learning models. The kicker database 32 stores kicker models from an initial kicker model 0 to a predetermined kicker model N. The goalkeeper database 32 stores kicker models from an initial goalkeeper model 0 to a predetermined goalkeeper model M.

[0025] The processing unit 33 executes motion control based on a learning model, similar to the processing unit 11. When a state St is input from the environment unit 25 described later, the processing unit 11 selects and executes a predetermined behavior (At: action) using the learning model based on the state St.

[0026] The environment unit 20 provides a multi-agent environment for multiple agents 5. Specifically, the environment unit 20 grants rewards Rt to multiple agents 5, and derives states St of each agent 5 that transition according to actions At. The environment unit 20 also calculates evaluation indices for learning, and selects learning targets based on the evaluation indices.

[0027] The environment unit 20 has a state transition processing unit 41, a reward giving unit 42 for kickers, a reward giving unit 43 for keepers, and a learning agent determination unit 44.

[0028] The state transition processing unit 41 receives the actions At performed by the multiple agents 5 as input, and calculates the state St of each agent 5, which is the output, using a state transition function for calculating state transitions. The state transition processing unit 41 outputs the calculated state St to the learning unit 31 of each agent 5. The state transition processing unit 41 also outputs the calculated state St to reward granting units 42 and 43.

[0029] The reward granting units 42, 43 use the action At performed by each agent 5, the state St, and the transition destination state St+1 as input, and calculate the reward Rt to be granted to each agent 5 as an output, using a reward function for calculating a reward. The reward granting units 42, 43 output the calculated reward Rt to the learning unit 31 of each agent 5. The reward function for the kicker agent 5a is, for example, a reward of "+1" if the kicker agent 5a scores a goal, and a reward of "-1" if the kicker agent 5a misses the goal. The reward function for the keeper agent 5b is, for example, a reward of "-1" if the kicker agent 5a scores a goal, and a reward of "+1" if the kicker agent 5a does not score a goal.

[0030] As described above, the learning agent determination unit 44 calculates the learning evaluation index and selects a learning target based on the evaluation index. The learning evaluation index is the ELO rating, and in this embodiment, a rating indicating the strength of the kicker agent 5a and the keeper agent 5b is used. The learning evaluation index is not particularly limited to the ELO rating, and may be a Glico rating. In addition, the learning agent determination unit 44 calculates a rating for each agent 5 for each learning, and obtains the rating in association with the learning model of the agent 5. That is, the database 32 stores a rating associated with each learning model.

[0031] In addition, since the learning evaluation index is in an asymmetric environment, it is different for each agent 5. For example, the evaluation index is for the kicker in the case of the kicker agent 5a, and for the keeper agent 5b, it is for the keeper. Note that the calculation model for finding the evaluation index may be the same, but the input value input to the calculation model is for the kicker in the case of the kicker agent 5a, and is for the keeper in the case of the keeper agent 5b.

[0032] Furthermore, the learning agent determination unit 44 uses the acquired ratings to select an agent to be learned from among the multiple agents 5. Specifically, the learning agent determination unit 44 compares the rating of the kicker agent 5a with the rating of the keeper agent 5b, and selects the agent with the lower rating as the agent 5 to be learned.

[0033] The storage unit 23 is an arbitrary storage device such as a semiconductor storage device or a magnetic storage device, similar to the storage unit 12. The storage unit 23 stores a learning program P for performing the above learning and executing a learning method described later.

[0034] In such a learning device 10, each agent 5 acquires a state St from the state transition processing unit 41 of the environment unit 25 during reinforcement learning, and also acquires a reward Rt from the reward granting units 42 and 43 of the environment unit 25. Then, in each agent 5, the learning unit 31 selects an action At from the learning model based on the acquired state St and reward Rt. The learning unit 31 inputs the selected action At to the state transition processing unit 41 and the reward granting units 42 and 43 of the environment unit 25, respectively. The reward granting units 42 and 43 calculate a reward Rt based on the selected action At, the state St, and the transition destination state St+1. In addition, the state transition processing unit 41 calculates a state St+1 after the transition based on the selected action At. Then, the learning unit 31 of each agent 5 repeats the above learning for a predetermined number of steps that can be evaluated (number of evaluation steps) so that the reward Rt granted to each agent 5 is maximized, thereby executing learning of the learning model.

[0035] (How to learn) Next, a learning method executed by the learning device 10 will be described with reference to Figs. 2 and 3. In the learning method, first, learning of a predetermined agent 5a among a plurality of agents 5 is executed using a learning model (step S1). Specifically, in step S1, learning of the kicker agent 5a is executed. At this time, the rating of the kicker agent 5a and the rating of the keeper agent 5b are the same value (for example, 1500). If they have the same rating, either agent 5 may be selected. Note that an agent 5 that is not selected becomes an agent 5 not to be learned, and is treated as one element of the environment in a multi-agent environment, that is, as an agent 5 that executes an action based on a fixed learning model.

[0036] After executing step S1, the learning device 10 executes learning of the kicker model using the kicker agent 5a and the environment unit 25, and determines whether the learning step of the kicker model is an evaluation step (step S2). When the learning device 10 determines that the learning step has not progressed to the evaluation step (step S2: No), the learning device 10 repeatedly executes the learning step until the learning step becomes the evaluation step. When the learning device 10 determines in step S2 that the learning step has progressed to the evaluation step (step S2: Yes), the learning device 10 calculates a rating for the latest kicker model after learning by the learning agent determination unit 44 (step S3). In step S3, the rating of the kicker model after learning becomes, for example, 1450. After executing step S3, the learning device 10 associates the latest kicker model with the rating and stores them in the database 32 (step S4). After executing step S4, the learning device 10 determines whether the learning step executed to learn the agent 5 has become greater than the learning end step, which is the learning step that ends (step S5).

[0037] If the learning device 10 determines in step S5 that the learning step is greater than the learning end step (step S5: Yes), the learning device 10 ends a series of processes related to the learning method. On the other hand, if the learning device 10 determines in step S5 that the learning step is equal to or less than the learning end step (step S5: No), the learning device 10 proceeds to step S6.

[0038] In step S6, the learning device 10 determines by the learning agent determination unit 44 whether the rating of the latest kicker model of the kicker agent 5a is higher than the rating of the latest keeper model of the keeper agent 5b. As shown in the center diagram of Fig. 2, if the rating (1450) of the kicker model is equal to or lower than the rating (1500) of the keeper model, the learning device 10 proceeds to step S1 again and executes learning of the kicker model. On the other hand, as shown in the lower diagram of Fig. 2, if the rating (1510) of the kicker model becomes higher than the rating (1500) of the keeper model by re-learning or the like, the learning device 10 executes learning of the keeper model (step S7).

[0039] In step S7, the learning of the keeper agent 5b is executed. After executing step S7, the learning device 10 executes learning of the keeper model by the keeper agent 5b and the environment unit 25, and judges whether the learning step of the keeper model is the evaluation step or not (step S8). If the learning device 10 judges that the learning step has not progressed to the evaluation step (step S8: No), the learning device 10 repeatedly executes the learning step until the learning step becomes the evaluation step. If the learning device 10 judges in step S8 that the learning step has progressed to the evaluation step (step S8: Yes), the learning device 10 calculates a rating for the latest keeper model after learning by the learning agent judgment unit 44 (step S9). After executing step S9, the learning device 10 associates the latest keeper model with the rating and stores them in the database 32 (step S10). After executing step S10, the learning device 10 proceeds to step S5, and repeatedly executes steps S1 to S10 until the learning step becomes greater than the learning end step.

[0040] In this way, the multiple agents 5 and the environment unit 25 that execute the above steps S1 to S10 function as a processing unit for performing reinforcement learning of the actions of the agents 5 through self-play.

[0041] (Multi-agent environment) Next, a multi-agent environment will be described with reference to FIG. 4 and FIG. 5. The multi-agent environment is not limited to the FK battle environment as described above. For example, as shown in the upper diagram of FIG. 4, the environment E1 may be one in which a plurality of agents 51a, which are the invading unmanned aerial vehicles, and a plurality of agents 51b, which are the defending unmanned aerial vehicles, battle each other. In the environment E1, the actions At performed by the invading unmanned aerial vehicles and the defending unmanned aerial vehicles include actions according to the aircraft performance. In addition, the state St acquired by the invading unmanned aerial vehicles and the defending unmanned aerial vehicles includes the measurement results of the radar. Furthermore, the reward Rt of the unmanned aerial vehicles is different between the invading and defending sides. In this case, the evaluation index is a rating based on the win or loss between the invading unmanned aerial vehicle and the defending unmanned aerial vehicle. The learning model learned by the defending unmanned aerial vehicle in the multi-agent environment E1 is installed in the actual unmanned aerial vehicle, so that the unmanned aerial vehicle can execute defense based on the learned learning model.

[0042] Also, as shown in the center diagram of FIG. 4, the environment may be an environment E2 in which a plurality of agents 52a and 52b serving as a plurality of unmanned drones on the defense side and an agent 52c serving as a unmanned drone on the invading side compete against each other. The agent 52a is an unmanned surface ship, the agent 52b is an unmanned aerial vehicle, and the agent 52c is an unmanned submarine. In the environment E2, the actions At performed by the unmanned submarine on the invading side, the unmanned surface ship on the defense side, and the unmanned aerial vehicle on the defense side include different actions according to the type of the vehicle. The state St acquired by the unmanned submarine on the invading side, the unmanned surface ship on the defense side, and the unmanned aerial vehicle on the defense side includes a sonar detection result. Furthermore, the rewards Rt of the unmanned submarine, the unmanned surface ship, and the unmanned aerial vehicle are different between the invading side and the defending side. In this case, the evaluation index is a rating based on the win or loss between the unmanned submarine on the invading side and the unmanned surface ship and the unmanned aerial vehicle on the defense side. The learning models learned by the defending unmanned surface vessels and unmanned aerial vehicles in the multi-agent environment E2 are installed on actual unmanned surface vessels and unmanned aerial vehicles, allowing the unmanned surface vessels and unmanned aerial vehicles to carry out defense based on the learned learning models.

[0043] Also, as shown in the lower diagram of FIG. 4, the environment may be an environment E3 in which an agent 53a, which is a security robot, and an agent 53b, which is an intruder, exist. In the environment E3, the actions At performed by the security robot are movement and waiting at a charging position, and the action At performed by the intruder is movement. The states St acquired by the security robot include a camera image, its own position, and the positions of other security robots. The state St acquired by the intruder is its own position. The rewards Rt of the security robot are "+1" for the discovery of the intruder and "-1" for the intruder's intrusion into a specified area, and the rewards Rt of the intruder are "-1" for being discovered by the security robot and "+1" for the intruder's intrusion into a specified area. In this case, the evaluation index is a rating based on the outcome of a battle between the security robot and the intruder. The learning model learned by the security robot in the multi-agent environment E3 is installed in an actual security robot, so that the security robot can perform security based on the learned learning model.

[0044] Also, as shown in the upper diagram of FIG. 5, the environment E4 may be one in which an agent 54a, which is a predetermined game character, and an agent 54b, which is another game character, compete against each other. The actions At performed by each game character include movement and attack, and are different depending on the game character. The state St acquired by each game character includes the game screen, the position of an enemy character, and the like. The reward Rt of each game character is "+1" if the game character defeats an enemy, and "-1" if the game character is defeated by the enemy. In this case, the evaluation index is a rating based on the win or loss of each game character. The learning model learned by each game character in the multi-agent environment E4 is executed in a competitive game, so that each game character can execute an action based on the learned learning model.

[0045] Also, as shown in the lower diagram of FIG. 5, an environment E5 may be one in which an agent 55a serving as a shovel and an agent 55b serving as a dump truck work together. The actions At performed by the shovel are movement and shovel operation, and the actions At performed by the dump truck are movement and unloading of earth and sand. The states St acquired by the shovel and the dump truck are the positions of the shovel and the dump truck. The reward Rt of the shovel is "0 to +1" depending on the amount of earth and sand when the dump truck is loaded with earth and sand, and is "-1" when the dump truck collides with the dump truck. The reward Rt of the dump truck is "0 to +1" depending on the amount of earth and sand transported and the transport distance, and is "-1" when the dump truck collides with the shovel and the dump truck. In this case, the evaluation index is a rating based on the amount of earth and sand loaded on the dump truck in the case of the shovel, and a rating based on the transported earth and the transport distance in the case of the dump truck. The learning models learned by the dump trucks and excavators in the multi-agent environment E5 are installed in the actual dump trucks and excavators, allowing the dump trucks and excavators to perform cooperative tasks of transporting soil and sand based on the learned learning models.

[0046] In this embodiment, a step of comparing ratings is executed in step S7, but a difference in ratings between a plurality of agents 5 may be calculated. If the difference calculated in step S7 does not decrease even after repeatedly executing the learning step, the learning device 10 may determine that the learning progress is not progressing, and may change the learning model of the agent 5 to a learning model associated with a different rating. Specifically, when the learning device 10 determines that the learning progress is not progressing based on the calculated difference, it may change the learning model to, for example, the learning model with the highest rating.

[0047] As described above, the learning device 10, the learning method, and the learning program P described in this embodiment can be understood, for example, as follows.

[0048] A learning device 10 according to a first aspect is a learning device 10 including a processing unit (agent 5 and environment unit 25) for performing reinforcement learning of the behavior of the agent 5 through self-play in a multi-agent environment in which a plurality of agents 5 exist. The multi-agent environment is an asymmetric environment in which at least one of the types of actions At performed by the agents 5, the types of states St acquired by the agents 5, and the definition of the rewards Rt granted to the agents 5 is different among the agents 5. Each of the agents 5 is assigned an evaluation index for learning. The processing unit executes steps S1 and S7 of performing learning of a predetermined agent 5 among the plurality of agents 5 using a learning model, steps S3 and S9 of acquiring the evaluation index in the learning model of the predetermined agent 5 after learning, a step S6 of comparing the evaluation index of the predetermined agent 5 with the evaluation index of the other agents 5, and steps S1 and S7 of setting the agent 5 with the lower evaluation index as a learning target.

[0049] A learning method according to a fourth aspect is a learning method for performing reinforcement learning of the behavior of an agent 5 through self-play in a multi-agent environment in which a plurality of agents 5 exist, the multi-agent environment being an asymmetric environment in which at least one of the types of actions At performed by the agents 5, the types of states St acquired by the agents 5, and the definition of the rewards Rt granted to the agents 5 is different among the agents 5, and each of the agents 5 is assigned an evaluation index for learning. The learning method includes steps S1 and S7 of performing learning of a predetermined agent 5 among the plurality of agents 5 using a learning model, steps S3 and S9 of acquiring the evaluation index in the learning model for the predetermined agent 5 after learning, a step S6 of comparing the evaluation index for the predetermined agent 5 with the evaluation index for the other agents 5, and steps S1 and S7 of setting the agent 5 with the lower evaluation index as the learning target.

[0050] A learning program P according to a fifth aspect is a learning program P to be executed by a learning device 10 in a multi-agent environment in which a plurality of agents 5 exist, for reinforcement learning of the behavior of the agent 5 through self-play. The multi-agent environment is an asymmetric environment in which at least one of the types of actions At performed by the agents 5, the types of states St acquired by the agents 5, and the definition of the rewards Rt granted to the agents 5 is different among the agents 5, and each of the agents 5 is assigned an evaluation index for learning. The learning device 10 is caused to execute steps S1 and S7 of learning a predetermined agent 5 among the plurality of agents 5 using a learning model, steps S3 and S9 of acquiring the evaluation index in the learning model of the predetermined agent after learning, a step S6 of comparing the evaluation index for the predetermined agent with the evaluation index for the other agents, and steps S1 and S7 of setting the agent with the lower evaluation index as a learning target.

[0051] According to these configurations, an agent 5 with a low evaluation index can be given priority in learning over other agents 5. Therefore, it is possible to prevent the learning progress between the multiple agents 5 from diverging. Furthermore, by giving priority to learning to a specific agent 5 with a low evaluation index, the other agents can learn based on the specific agent 5 with advanced learning during the learning of the other agents. Therefore, the other agents can avoid learning based on an agent 5 with a low learning progress (low evaluation index), and learning in a state where learning progress is low can be reduced. This makes it possible to efficiently execute reinforcement learning of multiple agents 5 in an asymmetric environment.

[0052] In a second embodiment, the evaluation index of learning is a rating.

[0053] According to this configuration, since a rating that serves as an appropriate index can be used as an evaluation index for learning, reinforcement learning of the multiple agents 5 can be appropriately carried out.

[0054] As a third aspect, in step S6 of comparing the evaluation indexes, the difference between the evaluation index for a given agent 5 and the evaluation index for another agent 5 is calculated, and if it is determined that learning progress is not progressing based on the calculated difference, the learning model of the agent 5 determined to be not progressing in learning is changed to a learning model with a different evaluation index.

[0055] According to this configuration, even if learning does not progress because the difference in ratings between multiple agents 5 does not decrease, it is possible to continue learning by using a learning model that differs from the evaluation index before the change. [Explanation of symbols]

[0056] 1 System 5. Agent 10 Learning Device 11 Processing section 12 Storage section 13 Sensors 14 Actuator 23 Memory section 25 Environment Department 31 Learning Department 32 Database 33 Processing section 41 State transition processing section 42 Kicker Reward Unit 43 Reward section for keeper 44 Learning Agent Judgment Unit P Study Program

Claims

1. A learning device comprising a processing unit for performing reinforcement learning of an action of an agent through self-play in a multi-agent environment in which a plurality of agents exist, The multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards given to the agents are different among the agents; Each of the agents is assigned a learning evaluation index; The processing unit includes: A first step of executing learning of a predetermined agent among the plurality of agents using a learning model; a second step of acquiring the evaluation index in the learning model of the predetermined agent after learning; a third step of comparing the evaluation index for a given agent with the evaluation index for other agents; a fourth step of selecting, from among the plurality of agents, the agent having the low evaluation index as a learning target, and excluding the agent not selected as a learning target; A learning device that repeatedly executes the first step to the fourth step until the number of learning steps executed to learn the agent becomes greater than a learning end step that is a terminating learning step.

2. The learning device according to claim 1 , wherein the evaluation index of learning is a rating.

3. In the third step, a difference between the evaluation index for a given agent and the evaluation index for another agent is calculated, The learning device according to claim 1 or 2, wherein when it is determined that learning progress is not progressing based on the calculated difference, the learning model of the agent determined to be not progressing in learning is changed to the learning model having a different evaluation index.

4. A learning device comprising a processing unit for performing reinforcement learning of an agent's behavior through self-play in a multi-agent environment in which a plurality of agents exist, The multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards given to the agents are different among the agents; Each of the agents is assigned a learning evaluation index; The processing unit includes: Executing learning of a predetermined agent among the plurality of agents using a learning model; obtaining the evaluation index in the learning model of a given agent after learning; A step of comparing the evaluation index for a given agent with the evaluation index for other agents; setting the agent having a low evaluation index as a learning target; In the step of comparing the evaluation indexes, a difference between the evaluation index for a given agent and the evaluation index for another agent is calculated, When it is determined that learning progress is not progressing based on the calculated difference, the learning device changes the learning model of the agent for which it is determined that learning progress is not progressing to a learning model with a different evaluation index.

5. A learning method for performing reinforcement learning of an agent's behavior through self-play in a multi-agent environment in which a plurality of agents exist, comprising the steps of: The multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards given to the agents are different among the agents; Each of the agents is assigned a learning evaluation index; A first step of executing learning of a predetermined agent among the plurality of agents using a learning model; a second step of acquiring the evaluation index in the learning model of the predetermined agent after learning; a third step of comparing the evaluation index for a given agent with the evaluation index for other agents; a fourth step of selecting, from among the plurality of agents, the agent having the low evaluation index as a learning target, and excluding the agent not selected as a learning target; A learning method that repeatedly executes the first step to the fourth step until the number of learning steps executed to learn the agent becomes greater than a learning end step that is a terminating learning step.

6. A learning method for performing reinforcement learning of an agent's behavior through self-play in a multi-agent environment in which a plurality of agents exist, comprising: The multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards given to the agents are different among the agents; Each of the agents is assigned a learning evaluation index; Executing learning of a predetermined agent among the plurality of agents using a learning model; obtaining the evaluation index in the learning model of a given agent after learning; A step of comparing the evaluation index for a given agent with the evaluation index for other agents; setting the agent having a low evaluation index as a learning target; In the step of comparing the evaluation indexes, a difference between the evaluation index for a given agent and the evaluation index for another agent is calculated, A learning method in which, when it is determined based on the calculated difference that learning progress is not progressing, the learning model of the agent determined to be not progressing in learning is changed to a learning model with a different evaluation index.

7. A learning program to be executed by a learning device for performing reinforcement learning of an action of an agent through self-play in a multi-agent environment in which a plurality of agents exist, the learning program comprising: The multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards given to the agents are different among the agents; Each of the agents is assigned a learning evaluation index; The learning device includes: A first step of executing learning of a predetermined agent among the plurality of agents using a learning model; a second step of acquiring the evaluation index in the learning model of the predetermined agent after learning; a third step of comparing the evaluation index for a given agent with the evaluation index for other agents; a fourth step of selecting, from among the plurality of agents, the agent having the lowest evaluation index as a learning target, and excluding the agent not selected as a learning target; A learning program that repeatedly executes the first step to the fourth step until the number of learning steps executed to learn the agent becomes greater than a learning end step that is a terminating learning step.

8. A learning program to be executed by a learning device in a multi-agent environment in which a plurality of agents exist, for reinforcement learning of the actions of the agents through self-play, comprising: The multi-agent environment is an asymmetric environment in which at least one of the types of actions performed by the agents, the types of states acquired by the agents, and the definitions of rewards given to the agents are different among the agents; Each of the agents is assigned a learning evaluation index; The learning device includes: Executing learning of a predetermined agent among the plurality of agents using a learning model; obtaining the evaluation index in the learning model of a given agent after learning; A step of comparing the evaluation index for a given agent with the evaluation index for other agents; setting the agent having a low evaluation index as a learning target; In the step of comparing the evaluation indexes, a difference between the evaluation index for a given agent and the evaluation index for another agent is calculated, A learning program that, when it is determined based on the calculated difference that learning progress is not progressing, changes the learning model of the agent whose learning progress is determined to be not progressing to a learning model with a different evaluation index.

Images