A course learning method based on strategy space response to prophecy

Abstract

The application discloses a curriculum learning method based on a strategy space response oracle (PSRO), and relates to the curriculum learning field. Two agents are composed: an autonomous agent and an environment agent. The autonomous agent learns to obtain a high reward value in the environment distribution generated by the environment agent. The environment agent continuously adjusts its own strategy according to the performance of the autonomous agent strategy pool and generates a new environment to guide the autonomous agent to continuously enhance the strategy strength and universality of the autonomous agent. The two agents maintain their respective strategy pools and are trained using the strategy space response oracle paradigm, and finally converge to a Nash equilibrium point. The autonomous agent can obtain a strategy pool with stronger generalization through the application, and can achieve good performance in a large number of environments similar to the training environment.

Description

Technical Field

[0001] This invention relates to the field of curriculum learning, and more particularly to a curriculum learning method based on policy space response prophets. Background Technology

[0002] In the development of machine learning, human learning has inspired various algorithm designs. As a prominent feature of human learning, curriculum has gradually formed a sub-discipline. Curriculum learning is a training strategy that sorts the data or tasks of machine learning models by simulating the beneficial learning order in human curriculum, usually from easy to difficult [1]. Benefiting from its ease of use and the significant increase in computing resources and data sample size in recent years, curriculum learning strategies have been widely used in computer vision, natural language processing, reinforcement learning and other scenarios.

[0003] Co-evolution is a multi-agent curriculum learning method in which the curriculum is generated by the interaction of multiple agents (teacher-student model) [2] or multiple versions of the same agent (self-play) [3] in the same environment. The actions of these agents may be cooperative or competitive, or both. Implicit curriculum is generated through the interaction between these agents, and the policy strength of the agents is continuously improved.

[0004] Current research on co-evolutionary learning methods has yielded considerable results, primarily focusing on three key issues: 1) How to ensure algorithm convergence; 2) How to improve algorithm efficiency and reduce implementation complexity; and 3) How to enhance the agent's final performance.

[0005] Co-evolution involves the interaction between multiple agents, so game theory knowledge and theories are usually used to make it converge / approach the Nash equilibrium. Dennis et al. proposed a teacher-student model based on minimizing regret [4]. The regret value is the reward value of the teacher agent in the environment minus the reward value of the student agent in the environment. The environment and teacher aim to maximize the regret value, while the student aims to minimize the regret value. Through this method, the student can continuously improve its own ability and eventually converge to the Nash equilibrium. However, a major drawback of this method is that it is very dependent on the ability of the teacher agent, and it is very difficult to train a high-strength teacher agent in a sparse reward and complex environment.

[0006] Wang et al. proposed a model based on an evolutionary algorithm [5]. This model consists of several agent-environment pairs. The main process has three steps: 1) Randomly perturb the existing environment to generate a new environment and filter out environments whose reward values ​​do not meet the requirements. 2) The agent learns in its corresponding environment. 3) Parameter transfer is performed between agents to transfer the parameters of strong agents to weak agents. Although this method can achieve good performance, it consumes a lot of resources, is inefficient, requires a lot of prior knowledge about the environment, and has poor transferability.

[0007] Jiang et al. used a priority queue for hierarchical replay [6]. This method uses temporal difference in reinforcement learning as an approximation of regret value, and sets priority based on temporal difference and access frequency. Environment parameters that meet the regret value conditions are put into the replay pool. During each training session, environment parameters are selected from the replay pool according to priority with a certain probability, or randomly selected from the environment parameter space, and the corresponding regret value is updated. If the requirements are met, the parameters are put into the replay pool. This method is simple and easy to use, but it is not suitable for complex environments.

[0008] Existing methods cannot effectively address all three issues simultaneously, thus necessitating the research of new learning methods to improve algorithm efficiency and performance while ensuring convergence.

[0009] In the field of game theory, the policy space response oracle [7] is a widely used paradigm. It mainly consists of three steps: 1) evaluation of the payoff matrix. 2) calculation of the meta-policy based on the payoff matrix. 3) each agent makes the best response to the meta-policy of the other agents to generate a new policy and puts the new policy into the policy pool. The policy space response oracle is based on a population-based training method, which has high training efficiency. In addition, it has the theoretical property of converging to Nash equilibrium, both of which meet our needs. It should be noted that the agents used in the policy space response oracle are agents in the traditional sense, not environmental agents. Therefore, if it is to be applied to course learning, some additional modifications are required.

[0010] Therefore, those skilled in the art are dedicated to developing a course learning method based on policy space response prophets. This aims to improve the algorithm's efficiency and performance while ensuring convergence.

[0011] References:

[0012] [1]Bengio Y,Louradour J,Collobert R,et al.Curriculum learning[C] / / Proceedings of the 26th annual international conference on machinelearning.2009:41-48.

[0013] [2]Sainbayar Sukhbaatar,Zeming Li,Ilya Kostrikov,Gabriel Synnaeve,Arthur Szlam,and Rob Fergus.Intrinsic motivation and automatic curricula viaasymmetric self-play.In International Conference on Learning Representations(ICLR),2018.

[0014] [3]David Silver,Aja Huang,Chris J Maddison,Arthur Guez,Laurent Sifre,George Van Den Driessche,Julian Schrittwieser,Ioannis Antonoglou,VedaPanneershelvam,Marc Lanctot,et al.Mastering the game ofgo with deep neuralnetworks and tree search.Nature,529(7587):484,2016.

[0015] [4]Dennis M,Jaques N,Vinitsky E,et al.Emergent complexity and zero-shot transfer via unsupervised environment design[J].Advances in NeuralInformation Processing Systems,2020,33:13049-13061.

[0016] [5]Wang R,Lehman J,Clune J,et al.Paired open-ended trailblazer(poet):Endlessly generating increasingly complex and diverse learning environments and their solutions[J].arXiv preprint arXiv:1901.01753,2019.

[0017] [6] Jiang, M., Grefenstette, E., and Rockta schel, T. (2021b). Prioritized level replay. In The International Conference on Machine Learning.

[0018] [7] Lanctot M, Zambaldi V, Gruslys A, et al. A unified game-theoretic approach to multiagent reinforcement learning [J]. Advances in neural information processing systems, 2017, 30. Summary of the Invention

[0019] In view of the above-mentioned deficiencies of the prior art, the technical problems to be solved by the present invention are: how to ensure the convergence of the algorithm; how to improve the efficiency of the algorithm and reduce the difficulty of algorithm implementation; and how to improve the final performance of the agent.

[0020] To achieve the above objectives, this invention provides a course learning method based on policy space response oracle, including setting environmental agent policies, training environmental agents and autonomous agents using a policy space response paradigm based on an evaluation matrix, and extracting a policy pool of autonomous agents to cope with new environments during the testing phase.

[0021] Furthermore, it includes the following steps:

[0022] Step 1: Setting up the environmental agent strategy;

[0023] Step 2: Performance Matrix Evaluation;

[0024] Step 3: Meta-policy generation;

[0025] Step 4: The autonomous intelligent agent and the environmental intelligent agent interact to generate new strategies.

[0026] Furthermore, the environmental agent strategy involves generating new environmental parameters based on the current state, and then generating a new environment based on these environmental parameters for training the autonomous agent.

[0027] Furthermore, the current state includes the performance of the autonomous intelligent agent and the current environmental settings.

[0028] Furthermore, in step 2, the revenue matrix evaluation is to evaluate the strategies in the strategy pool.

[0029] Furthermore, the policy pool includes a policy pool for the environmental intelligent agent and a policy pool for the autonomous intelligent agent.

[0030] Furthermore, the autonomous intelligent agent strategy Environmental Intelligent Agent Strategies The profit value Where i represents the autonomous intelligent agent, -i represents the environmental intelligent agent, and k and j are the indices of the policies in the policy pool.

[0031] Furthermore, the use of environmental intelligent agent strategies To deal with the use of autonomous intelligent agents The return value is α(r) K -r)-r, where r is Environmental Intelligent Agent Strategies The profit value, r K for exist The new policy obtained after training for K steps in the generated environment Environmental Intelligent Agent Strategies The return value, where α is a coefficient, typically decreases as the number of training steps increases.

[0032] Furthermore, the meta-policy is the probability distribution of policies in the policy pool.

[0033] Furthermore, in step 4, an optimal coping meta-strategy approach based on an evaluation matrix rather than environmental rewards is used.

[0034] In a preferred embodiment of the present invention, the objective of this application is to develop a course learning method based on policy space response oracle to solve these three problems. This method has three characteristics: 1. It uses a population-based training method, resulting in high algorithm efficiency. 2. It modifies the reward matrix evaluation and optimal response generation steps in the policy space response oracle paradigm, enabling its extension to interactions between autonomous agents and environmental agents. 3. It uses a regret-based reward matrix evaluation, leveraging the properties of the policy response oracle paradigm to guarantee the convergence of the algorithm.

[0035] Compared with the traditional "teacher-student" model, this invention treats the environment as a special intelligent agent, which is different from the teacher intelligent agent. The latter is an advanced version of the student intelligent agent, and its structure is similar to that of the student intelligent agent, while the former is more like a task generator. The reward of the environment intelligent agent in this invention is obtained based on the performance of the main intelligent agent. It does not need to be pre-trained, which can solve the problem of "difficulty in training the teacher intelligent agent" in [4]. In addition, although the initial policy of the environment intelligent agent is a random policy, it will gradually converge with updates, thus solving the problem of difficulty in random uniform sampling of large parameter space encountered in large-scale complex environments in [6]. The method of this invention perturbs in the direction of increasing the regret value of the main intelligent agent, which greatly improves efficiency compared with the random perturbation in [5].

[0036] The present invention includes the following steps:

[0037] 1. Environmental agent strategy settings

[0038] This invention uses a tuple of a partially observable Markov decision process (POMDP).<A,O,S,T,I,R,γ> Let Θ represent an environment with defined parameters. Here, A is the action space set, O is the observation space set, S is the state space set, T is the environment state transition function, I is the information set, R is the reward function, and γ is the discount factor. The problem with unsupervised environment design (UED) is that it generates a fully specified environment distribution from an unspecified environment. The controllable parameters of the environment agent are represented by Θ, and the environment generated by the environment agent is represented by tuples. <A,O,Θ,S M ,T M ,I M ,R M ,γ> represents, where T M S×A×Θ→Δ(S). For example, in a maze game, the environmental agent can be controlled by Θ, which controls the agent's starting / ending point, the position and number of walls, and the agent's observability. The environmental agent's strategy is to generate new environmental parameters based on the current state (the agent's performance, the current environmental settings), and then generate a new environment based on these parameters for training the agent.

[0039] 2. Benefit Matrix Evaluation

[0040] Both the autonomous agent and the environmental agent maintain a policy pool. The reward matrix evaluation step involves evaluating these policies in the policy pools. This represents the payoff for both the autonomous agent using strategy k and the environment agent using strategy j. Traditional policy space response prophets typically use a predefined payoff matrix from the game or the actual reward value obtained by the agent in the environment as the payoff value, but this is not applicable to the environment agent. This invention makes some adjustments to address this issue.

[0041] For an autonomous agent, the goal is to obtain a high reward value in an environment generated by an environmental agent. This invention operates within a fully specified environment (this environment being the environmental policy). The generated pair uses a strategy The autonomous agent undergoes reinforcement learning training. When the reward value obtained by the autonomous agent from the environment reaches a threshold (e.g., when the environment reward value reaches 300 in [5], or when the agent successfully reaches the endpoint and receives a reward value of +1 in [4]) or when the number of steps taken in the current environment reaches a preset upper limit, the training of the autonomous agent in that environment is terminated. In the policy evaluation reward matrix, the autonomous agent policy Environmental Intelligent Agent Strategies The profit value

[0042] For an environmental agent, the goal is to generate environments of appropriate difficulty to help the autonomous agent learn general skills. In the policy space response-prophecy paradigm, this is typically zero-sum, ensuring eventual convergence. However, in unsupervised environment design, minimization strategies can easily go to extremes, where the environmental agent generates extremely difficult environments that the autonomous agent cannot solve, thus allowing the environmental agent to consistently gain high rewards. To address this issue, this invention adds a reward term based on the autonomous agent's regret value in the early stages of the algorithm. This coefficient decreases over time, and in the later stages, the rewards for both the environmental and autonomous agents remain zero-sum, thus guaranteeing eventual convergence.

[0043] Autonomous agent usage strategy The resulting reward value is r, and the autonomous agent uses a strategy in response to the environmental agent. The new policy obtained by training K steps in the generated environment The resulting profit value is r k So, what are the strategies used by environmental intelligent agents? To deal with the use of autonomous intelligent agents The return value is α(r) K -r)-r, where α is initially 1, and is multiplied by a decay coefficient after several rounds of policy space response prophecy iterations, which can be adjusted according to the actual situation.

[0044] 3. Meta-policy generation

[0045] The meta-policy is the "policy" of the policy. It uses the probability distribution of the policies in the policy pool. After the second step of calculation, a policy payoff matrix is ​​obtained. Then, the meta-policy σ is generated by using regret value matching based on this matrix.

[0046] Cumulative regret value update method:

[0047]

[0048] The new meta-strategy obtains regret value through normalization:

[0049]

[0050] in K is the current policy pool size.

[0051] In addition, using exploration strategies ensures thorough exploration:

[0052] σ′ i =γUniform(K+1)+(1-γ)σ i

[0053] 4. Interaction and Response between Autonomous Intelligent Agents and Environmental Intelligent Agents

[0054] In the third step, the autonomous agent and the environmental agent calculate their meta-policies, which are also known to each other. Therefore, the autonomous agent and the environmental agent will generate new policies in response to each other's meta-policies, a step also known as "foresight." The new policies will be added to the existing policy pool.

[0055] The optimal implementation method for the strategy space response oracle is as follows:

[0056]

[0057] Where β -i M represents the opponent's policy pool, M represents the current size of your own policy pool, and r represents the size of your own policy pool. i (s,a) represents the reward the agent obtains in the environment. Since the environment used does not have r... i (s,a), therefore some changes are needed.

[0058] The best solution after the change is as follows:

[0059]

[0060] The reward from the environment is replaced with the result of the reward matrix evaluation in step two. Since the reward matrix evaluation in the policy space response oracle also uses rewards from the environment and is aligned with the optimal response, the change also aligns it. From a higher-level perspective, the rewards obtained by the environmental agent and the autonomous agent are given by a higher-level environment, and this reward is used in both steps two and four and remains consistent, thus maintaining the structural integrity and consistency of the policy space response oracle.

[0061] Compared with the prior art, the present invention has the following obvious substantive features and significant advantages:

[0062] 1. Using a population-based training method, the algorithm runs efficiently.

[0063] 2. Modify the two steps of payoff matrix evaluation and optimal response generation in the policy space response oracle paradigm to extend it to the interaction between autonomous agents and environmental agents.

[0064] 3. The algorithm uses a regret-based payoff matrix for evaluation and leverages the properties of the policy response prophetic paradigm to ensure convergence.

[0065] 4. Autonomous agents can obtain a more generalizable policy pool in this way, and can achieve good performance in many environments similar to the training environment.

[0066] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of a preferred embodiment of the present invention;

[0068] Figure 2 This is an environmental schematic diagram of a preferred embodiment of the present invention;

[0069] Figure 3 This is a graph showing the performance of the agent policy pool of a preferred embodiment of the present invention on a test set. Detailed Implementation

[0070] The technical solution of the present invention will be described in detail below with reference to specific embodiments and accompanying drawings, so that its technical content is clearer and easier to understand. The present invention can be embodied in many different embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0071] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary; the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thicknesses of components are appropriately exaggerated in some places in the drawings.

[0072] This invention relates to the fields of curriculum learning, game theory, and multi-agent reinforcement learning, particularly to the field of curriculum learning based on multi-agent interaction.

[0073] like Figure 2 As shown, the robot learns to walk in the BipedalWalker environment. The autonomous agent controls two mechanical legs to maintain balance and cross obstacles to earn rewards. The environmental agent controls the steepness of the terrain and the generation of obstacles to create new environments. If the agent reaches a reward value of 300 in the current environment, or reaches the time limit (two thousand steps), the process is forcibly terminated and the environment is switched.

[0074] This implementation provides a course learning method based on policy space response prophets, involving setting environmental agent policies according to environmental characteristics, evaluating the payoff matrix based on environmental interaction data, using regret value matching to generate meta-policies, and expanding the policy pool by generating new policies using the best response; specifically, such as Figure 1 As shown, it includes the following steps:

[0075] Step S1: Set the strategy for the environmental agent based on the ground features on which the robot walks.

[0076] Step S2: Using reinforcement learning algorithms and environmental interaction data, evaluate the robot policy (autonomous agent) and the policies in the environmental agent policy pool to generate a reward matrix.

[0077] Step S3: Based on the payoff matrix, use regret value matching to generate meta-policies for the autonomous agent and the environmental agent.

[0078] Step S4: Based on the generated meta-policy, the autonomous agent and the environmental agent interact to make the best response and generate a new policy.

[0079] Step S1 includes: extracting environmental features from the BipedalWalker environment, including ground slope, obstacles, ground protrusions, and ground depressions. After feature extraction, the extracted feature format is used as the policy output format of the environmental agent, and then the policy network of the environmental agent is initialized. Specifically:

[0080] Step S101: Initialize the parameters of the policy network of the environmental agent based on the characteristics of the current environment.

[0081] Step S102: Randomly initialize the policy network of the autonomous agent, and add the policies generated by the policy network of the autonomous agent and the policies generated by the policy network of the environment agent to the corresponding policy pools.

[0082] Step S2 includes: using reinforcement learning algorithms and environmental interaction data, performing a one-to-one evaluation of the policies in the policy pools of the autonomous agent and the environmental agent, and generating a reward matrix, specifically:

[0083] Step S201: Select autonomous agent policies sequentially from the policy pool. and environmental intelligent agent strategy

[0084] Step S202: Use the environmental agent strategy Generate a specific environment so that the autonomous agent can use the strategy. Reinforcement learning training is conducted in this environment. The training is terminated when the reward value obtained by the autonomous agent from the environment reaches a threshold or when the number of steps taken by the autonomous agent in the environment reaches a preset upper limit.

[0085] Step S203: Based on the results of step S202, set the autonomous agent strategy. Environmental Intelligent Agent Strategies Profit value

[0086] Step S204: Strategies for the Autonomous Agent A new strategy is obtained by training K steps in this environment.

[0087] Step S205: Apply the new strategy in the manner of steps S202 and S203. The evaluation yields a return value r. K Configure environmental intelligent agent policies Strategies for dealing with autonomous intelligent agents The return value is α(r) K -r)-r.

[0088] Step S3 includes: generating meta-policies for the autonomous agent and the environmental agent based on the payoff matrix using regret value matching, specifically:

[0089] Step S301: Update the cumulative regret value:

[0090]

[0091] Step S302: Obtain a new meta-policy by normalizing the regret value:

[0092]

[0093] Step S303: Use a hybrid exploration strategy to ensure sufficient exploration:

[0094] σ′ i =γUniform(K+1)+(1-γ)σ i

[0095] Step S4 includes: based on the generated meta-policy, the autonomous agent and the environmental agent respectively make optimal responses to each other, generating a new policy, specifically:

[0096] Step S401: The autonomous intelligent agent responds optimally to the meta-policy of the environmental intelligent agent.

[0097]

[0098] Step S402: The environmental agent responds optimally to the meta-policy of the autonomous agent.

[0099]

[0100] The present invention (PSRO-CL) was compared with the algorithms in [4][6] and the randomly selected environment parameters (DR). Since the environment selection tendencies of each algorithm are inconsistent, only a part of the environment is put into the training set and the other part into the test set during the environment generation process. The training set and the test set will change dynamically with the increase of training steps and there is no overlap between the two. The policy pool of the current autonomous agent is extracted every 200,000 steps, and a certain number of environments are extracted from the test set for testing. The evaluation index is the average reward obtained by the agent's policy pool in each environment. The results are as follows. Figure 3 As shown in Table 1, the method proposed in this invention has significant advantages over the baseline in solving and training similar but different environments, effectively improving the generalization and diversity of the policy pool of autonomous agents.

[0101] Table 1 shows the performance of the agent policy pool on the test set.

[0102]

[0103] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A course learning method based on policy space response oracle, characterized in that, This includes setting environmental agent policies, training environmental agents and autonomous agents using a policy space response paradigm based on an evaluation matrix, and extracting a policy pool of autonomous agents to cope with new environments during the testing phase. The robot walks by controlling two mechanical legs to maintain balance and cross obstacles, including the following steps: Step S1: Set the strategy for the environmental agent based on the ground features where the robot walks; Step S2: Using reinforcement learning algorithms and environmental interaction data, evaluate the robot policy and the policies in the environmental agent policy pool to generate a reward matrix; Step S3: Based on the payoff matrix, use regret value matching to generate meta-policies for the autonomous agent and the environmental agent; Step S4: Based on the generated meta-policy, the autonomous agent and the environmental agent interact to make the best response and generate a new policy; Environmental intelligent agent strategy usage To deal with the use of autonomous intelligent agents The profit value ,in for Environmental Intelligent Agent Strategies The profit value, for exist The new policy obtained after training for K steps in the generated environment Environmental Intelligent Agent Strategies The profit value, is a coefficient.

2. The course learning method based on policy space response oracle as described in claim 1, characterized in that, The aforementioned environmental agent strategy involves generating new environmental parameters based on the current state, and then generating a new environment based on these environmental parameters for training the autonomous agent.

3. The course learning method based on policy space response oracle as described in claim 2, characterized in that, The current state includes the performance of the autonomous intelligent agent and the current environmental settings.

4. The course learning method based on policy space response oracle as described in claim 1, characterized in that, The policy pool includes the policy pool of the environmental intelligent agent and the policy pool of the autonomous intelligent agent.

5. The course learning method based on policy space response oracle as described in claim 1, characterized in that, The autonomous intelligent agent strategy Environmental Intelligent Agent Strategies The profit value ,in Represents an autonomous intelligent agent. Represents an environmental intelligent agent. This is the index of the strategy in the strategy pool.

6. The course learning method based on policy space response oracle as described in claim 1, characterized in that, The meta-policy is the probability distribution of using policies in the policy pool.

7. The course learning method based on policy space response oracle as described in claim 6, characterized in that, Step S4 uses an optimal coping meta-strategy approach based on an evaluation matrix rather than environmental rewards.

Images