Group imitation learning expert demonstration enhancement method based on orthogonal transformation

Abstract

The invention discloses a group imitation learning expert demonstration enhancement method based on orthogonal transformation, and belongs to the field of imitation learning. The method specifically comprises the steps that for a multi-agent system, the track of each agent is obtained by executing a task, and therefore expert demonstration is obtained; secondly, geometric transformation is preset through an orthogonal group, the tracks of all agents in each demonstration expert are enhanced through symmetry, and the tracks are combined with original expert demonstration; generating a respective track for each agent by using a generator; randomly sampling from the merged expert demonstration and the generated trajectory, calculating the cross entropy of the enhanced sample as symmetry loss, and merging the symmetry loss with the original loss to obtain the loss of the discriminator; and updating the discriminator and feeding back the output to the generator as a reward function, updating the strategy network parameter pi theta of each agent by using reinforcement learning, and repeating until the algorithm converges. The problem that expert demonstration data is scarce is effectively relieved.

Description

Technical Field

[0001] The invention belongs to the field of imitation learning, and in particular relates to an expert demonstration enhancement method for group imitation learning based on orthogonal transformation. Background Art

[0002] In recent years, multi-agent reinforcement learning has achieved remarkable results in many fields such as multiplayer games and robotics, and has become a powerful algorithmic tool. The effectiveness of multi-agent reinforcement learning mainly stems from the ability to optimize manually designed reward functions. However, in complex and uncertain environments, designing accurate reward functions is often a very challenging task. If the reward function is not designed properly, it may lead to poor agent behavior strategies.

[0003] To address these problems, group imitation learning has become a commonly used algorithm that can learn hidden expert strategies directly from expert demonstrations without the need to manually design complex reward functions.

[0004] Imitation learning is a method of learning the optimal decision-making model of experts from expert demonstrations. It is developed based on the theory of reinforcement learning. In the field of imitation learning, the most widely used algorithms are behavioral cloning and generative adversarial imitation learning.

[0005] The behavior cloning algorithm is a simple and intuitive imitation learning method that learns the mapping of states to actions from expert demonstrations through supervised learning. During training, the agent receives state-action pairs demonstrated by experts and then fits them using a classifier or regressor. However, since the agent does not interact with the environment, the learned strategy is usually affected by accumulated errors and its performance is limited by the quality and quantity of expert demonstrations.

[0006] Generative adversarial imitation learning is based on generative adversarial networks. Its idea is to design a generator and a discriminator. The goal of the generator is to learn from the training data and generate sample data with the same distribution as the expert demonstration, while the task of the discriminator is to determine whether the input data comes from the real training data or the data generated by the generator. The two compete with each other, and finally the discriminator and generator reach the global optimal state where the probability of discriminating samples and generating samples is equal.

[0007] However, existing imitation learning methods assume that the expert trajectory is sufficient, which may lead to the following problems:

[0008] 1. High cost of data acquisition: In certain areas, such as multi-agent robotics tasks, obtaining a sufficient amount of expert demonstration data may require a lot of time and resources, which makes it difficult to obtain sufficient data support in practical applications.

[0009] 2. Overfitting: When the amount of available expert demonstrations is limited, imitation learning algorithms tend to overfit these data, resulting in poor generalization of the model in practical applications.

[0010] 3. Insufficient sample diversity: The limited number of expert demonstrations also means that it is difficult to cover all possible environmental states or decision-making scenarios, making the model unable to handle unseen states and decisions, thus affecting overall performance.

[0011] 4. Intensified challenges in multi-agent systems: In multi-agent systems, the problem of insufficient data is even more serious. The behavior of each agent may depend on the strategies of other agents, and the lack of expert demonstrations will result in the learned strategies being unable to effectively cope with complex interactive dynamics.

[0012] Therefore, there needs to be a method to perform data augmentation on expert demonstrations to solve the problem of scarcity of expert demonstrations in group imitation learning, so that imitation learning can learn more effective strategies. Summary of the invention

[0013] In order to solve the problem of scarce expert demonstrations in group imitation learning, the present invention provides a method for enhancing expert demonstrations in group imitation learning based on orthogonal transformation. By utilizing the symmetry that is common in multi-agent systems, additional expert demonstration samples are generated, effectively increasing the number and diversity of samples; by designing symmetry loss, the model's ability to identify enhanced samples is effectively improved. The two modules jointly improve the performance of the group imitation learning algorithm and alleviate the problem of scarce expert demonstration data.

[0014] The group imitation learning expert demonstration enhancement method based on orthogonal transformation has the following specific steps:

[0015] Step 1: For a multi-agent system consisting of N agents, obtain the trajectory of each agent through the expert strategy execution task, and thus obtain the expert demonstration τ E ;

[0016] Trajectory means: the execution time of each task is a time series of length T, and the state and action of each agent can be collected at each moment;

[0017] In each task, the actions and states of all agents at T complete moments constitute an expert demonstration;

[0018] By repeatedly executing the same task multiple times to obtain expert demonstrations, the expert demonstration set is Where M is the number of expert demonstrations, which is also the number of executions; is the state set of all agents at time t in the jth expert demonstration; is the set of actions of all agents at time t in the j-th expert demonstration.

[0019] Step 2: Initially randomly set the policy network π with parameters θ for each agent θ And the discriminator network D with parameter ω ω ;

[0020] Each agent consists of a generator policy network and a discriminator network; θ i is the policy network parameter of the i-th agent; the discriminator ω i is the discriminator parameter of the i-th agent;

[0021] Step 3: Use orthogonal group preset geometric transformations to enhance the states and actions of all agents in each demonstration expert using symmetry;

[0022] Geometric transformations in Euclidean space include rotations, reflections, and their combinations; they are represented by the orthogonal group O(n), which is a group of n×n orthogonal matrices, defined as follows:

[0023]

[0024] Among them, the matrix Z is an orthogonal matrix, representing a geometric transformation; Z T represents the transposed matrix of the matrix Z, and I represents the identity matrix. Represents the set of all n×n matrices over the real number field.

[0025] Specifically:

[0026] Step 301: Select rotation and reflection transformation from the orthogonal group O(n) as geometric transformation g, and obtain corresponding rotation matrix and reflection transformation matrix;

[0027] Step 302: multiply the state and action that need to be transformed of each agent in the current demonstration expert by the rotation matrix and the reflection matrix to obtain the transformed state and action;

[0028] The details are as follows:

[0029] The state includes the state s that is equivariant to geometric transformations equ and the state s that is invariant to geometric transformations inv , the transformed state is represented by L g [s]=[L g [s equ ],s inv ].

[0030] The actions of the agent include equivariant actions aequ and unchanged action a inv , the transformed action is represented by K g [a]=[K g [a equ ],a inv ].

[0031] Step 303: Similarly, transform the states and actions of each agent in each expert demonstration to obtain an enhanced expert demonstration;

[0032] Step 4: Merge the original expert demonstration with the enhanced expert demonstration to obtain the subsequent learned expert demonstration τ′ E ;

[0033] That is: τ′ E ←τ E ∪gτ E .

[0034] Step 5: Utilize the policy network π of the generator in each agent θ Generate a trajectory for this agent, and the generated trajectory of all agents is τ G ;

[0035] Each agent's generated trajectory consists of a sequence of states and actions selected by the neural network.

[0036] Step 6: From the combined expert demonstration τ′ E and generate the trajectory τ G Random sampling is performed to obtain expert samples corresponding to the state actions of each agent and generate samples

[0037] Step 7: Use the same geometric transformation again to transform the expert samples of each agent and generate samples Do data enhancement to obtain enhanced expert samples And generate samples

[0038] Step 8: Utilize the Augmented Expert Samples And generate samples Calculate the cross entropy as the symmetry loss, take the sum of the original loss and the symmetry loss as the discriminator loss function, and use the loss function to update the discriminator.

[0039] The goal of the discriminator is to minimize the difference between the expert trajectory samples and the trajectory samples generated by the generator, and to distinguish the generated trajectory from the expert demonstration trajectory as much as possible.

[0040] The loss function of the discriminator is:

[0041] is the original loss function, and its calculation formula is:

[0042]

[0043] in, is the discriminator, s is the state of the multi-agent system, s′ is the next state of the multi-agent system, a i is the action of the ith agent.

[0044] The loss function of symmetric entropy is introduced, and its calculation formula is:

[0045]

[0046] Among them, L g [s] represents the state after the geometric transformation of state s, L g [s′] represents the state after the geometric transformation of state s′, K g [a i ] means a i Action after geometric transformation.

[0047] Step 9: Feed the output of the discriminator back to the generator as the reward function of the reinforcement learning algorithm to update the policy network π of each agent θ Parameter θ, repeat steps 5 to 9 until the algorithm converges.

[0048] The advantages of the expert demonstration enhancement method for group imitation learning based on orthogonal transformation of the present invention are:

[0049] 1) Traditional imitation learning ignores the situation of insufficient expert demonstrations. In reality, the cost of collecting expert demonstrations is high and the amount of data is often insufficient. The present invention can make full use of the small amount of existing expert demonstrations to generate enough expert demonstrations, thereby solving the problem of scarce expert demonstrations in group imitation learning.

[0050] 2) In generative adversarial imitation learning, it is crucial for the discriminator to effectively distinguish between expert samples and generated samples. The introduction of symmetry loss can help the discriminator better distinguish expert samples and generated samples after geometric transformation, thereby providing more accurate feedback information for the generator, thereby improving the imitation ability of the algorithm.

[0051] 3) This invention is an improvement on the generative adversarial imitation learning framework. It does not require any specific imitation learning algorithm assumptions and can be embedded in any generative adversarial imitation learning algorithm, so it has a wide range of applications. In addition, the data enhancement module can also be applied to any imitation learning, which can effectively solve the problem of scarce expert demonstrations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Example diagram of symmetry of a multi-agent system

[0053] Figure 2 The framework diagram of the expert demonstration enhancement method for group imitation learning based on orthogonal transformation of the present invention

[0054] Figure 3 Flow chart of the expert demonstration enhancement method for group imitation learning based on orthogonal transformation of the present invention DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0056] In the existing technology, the use of prior knowledge can effectively solve the problem of data scarcity in imitation learning. Prior knowledge can help neural networks learn more knowledge from limited data sets, thereby significantly improving the performance of the algorithm.

[0057] There are often symmetries in multi-agent systems, including equivariance and invariance: given a transformation L g :X→X and a function f:X→Y, if there exists another transformation K on the output space of function f g :Y→Y, so that K g [f(x)] = f(L g (x)), where g∈G, G is a group, then the function f is said to transform L g It is equivariant; if for any g∈G, f(L g (x)) = f(x), then the function f is called the transformation L g Has immutability.

[0058] like Figure 1 As shown in the figure, for a multi-agent chasing task, when the multi-agent system is rotated, the optimal action of the agent rotates accordingly with the rotation of the global state, which is equivariance; and the characteristics such as the distance between agents will remain unchanged regardless of whether the system is rotated or reflected, which is invariance. Inspired by this symmetry idea, the present invention proposes an expert demonstration enhancement method for group imitation learning based on orthogonal transformation, which can amplify a small number of expert samples into a sufficient number of expert samples, thereby effectively solving the problem of scarce expert data in imitation learning. At the same time, this amplification can also increase the diversity of samples, thereby improving the performance of the algorithm.

[0059] The expert demonstration enhancement method for group imitation learning based on orthogonal transformation, such as Figure 2 As shown, the framework diagram is as follows Figure 3 As shown in the figure, the main components are the data enhancement module and the symmetry loss. The specific steps are as follows:

[0060] Step 1: For a multi-agent system consisting of N agents, obtain the trajectory of each agent through the expert strategy execution task, and thus obtain the expert demonstration τ E ;

[0061] Trajectory means: the execution time of each task is a time series of length T, and the state and action of each agent can be collected at every moment; T is the round length.

[0062] In each task, the actions and states of all agents at T complete moments constitute an expert demonstration;

[0063] Perform the same task multiple times and collect a certain number of expert demonstrations. The expert demonstration set is Where M is the number of expert demonstrations, which is also the number of executions; is the state set of all agents at time t in the jth expert demonstration; is the set of actions of all agents at time t in the j-th expert demonstration.

[0064] Step 2: Initially randomly set the policy network π with parameters θ for each agent θ And the discriminator network D with parameter ω ω ;

[0065] Each agent consists of a generator policy network and a discriminator network; θ i is the policy network parameter of the i-th agent; the discriminator ω i is the discriminator parameter of the i-th agent;

[0066] Step 3: Use orthogonal group preset geometric transformations to enhance the states and actions of all agents in each demonstration expert using symmetry;

[0067] In the context of multi-robot systems, symmetries are mainly manifested by equivariance and invariance to rotations and reflections. Geometric transformations in Euclidean space include rotations, reflections, and their combinations; they are represented by the orthogonal group O(n), which is a group of n×n orthogonal matrices defined as follows:

[0068]

[0069] Among them, the matrix Z is an orthogonal matrix, representing a geometric transformation; Z T represents the transposed matrix of the matrix Z, and I represents the identity matrix. Represents the set of all n×n matrices over the real number field.

[0070] Specifically:

[0071] Step 301: Select rotation and reflection transformation from the orthogonal group O(n) as geometric transformation g, and obtain corresponding rotation matrix and reflection transformation matrix;

[0072] Step 302: multiply the state and action that need to be transformed of each agent in the current demonstration expert by the rotation matrix and the reflection matrix to obtain the transformed state and action;

[0073] The details are as follows:

[0074] In a multi-agent system, there are features that are equivariant and invariant to geometric transformations, that is, the state includes the state s that is equivariant to geometric transformations. equ and the state s that is invariant to geometric transformations inv , so the state in the environment can be expressed as s = [s equ ,s inv ]∈S. For a feature s with equivariance equ , which will be affected by the corresponding geometric transformations, such as the coordinates of the agent, which will change when the multi-agent system rotates, but its essence is unchanged, which can show new feasible states in potential real scenes. inv , such as agent distance, are not affected by geometric transformation, so these features will remain unchanged after geometric transformation. Therefore, the transformed state is represented by L g [s]=[L g [s equ ],s inv ].

[0075] After applying geometric transformation to the environment state, the action of the agent should also be transformed accordingly, which ensures that the transformation of state and action is consistent, thus accurately describing the system. Similar to the change of environment state, the action of each agent is divided into equivariant and invariant actions, represented as a equ and a inv , the overall action is expressed as a=[a equ ,a inv ]. For the equivariant action q equ , which will change accordingly with geometric transformations, such as vector actions such as speed, and for actions with invariance a inv , which is not affected by geometric transformations, such as scalar actions such as linear velocity and angular velocity, so the transformed action can be expressed as K g [a]=[K g [a equ ],a inv ].

[0076] Step 303: Similarly, each agent in each demonstration expert is transformed in state and action to obtain an enhanced expert demonstration;

[0077] Through the above transformation, a small number of expert demonstrations can be expanded into a sufficient number of expert demonstrations.

[0078] Step 4: Merge the original expert demonstration with the enhanced expert demonstration to obtain the subsequent learned expert demonstration τ′ E ;

[0079] That is: τ′ E ←τ E ∪gτ E .

[0080] Step 5: Utilize the policy network π of the generator in each agent θ Generate a trajectory for this agent, and the generated trajectory of all agents is τ G ;

[0081] The generator is a neural network that generates behavior trajectories that are as close to the expert as possible in generative adversarial imitation learning. After initializing the parameter θ, the neural network can be used to interact with the environment to generate trajectories. The trajectory includes a series of states and actions selected by the neural network. However, when it is just initialized, the trajectory generated by the generator is far from the expert trajectory.

[0082] Step 6: From the combined expert demonstration τ′ E and generate the trajectory τ G Random sampling is performed to obtain expert samples corresponding to the state actions of each agent and generate samples

[0083] Step 7: Use the same geometric transformation again to transform the expert samples of each agent and generate samples Do data enhancement to obtain enhanced expert samples And generate samples

[0084] Step 8: Utilize the Augmented Expert Samples And generate samples Calculate the cross entropy as the symmetry loss, take the sum of the original loss and the symmetry loss as the discriminator loss function, and use the loss function to update the discriminator;

[0085] The goal of the discriminator is to minimize the difference between the expert trajectory samples and the generator generated trajectory samples, and to distinguish the generated trajectory from the expert demonstration trajectory as much as possible. In generative adversarial imitation learning, cross entropy is used to measure this difference, and the network is helped to identify expert samples and generated samples by minimizing the size of entropy. The original loss is However, after using symmetry for enhancement, the discriminator also needs to distinguish the difference between the transformed expert samples and the generated samples. In order to improve the recognition ability of the discriminator, symmetric entropy is introduced as a loss. Symmetric entropy calculates the binary cross entropy between the enhanced expert samples and the generated samples;

[0086] The calculation formula is:

[0087]

[0088] in, is the discriminator, s is the state of the multi-agent system, s′ is the next state of the multi-agent system, a i is the action of the ith agent.

[0089] The calculation formula is:

[0090]

[0091] Among them, L g [s] represents the state after the geometric transformation of state s, L g [s′] represents the state after the geometric transformation of state s′, K g [a i ] means a i Action after geometric transformation.

[0092] The loss function of the final discriminator consists of two parts, namely The goal of the discriminator is to minimize this loss value and use it to update the discriminant parameters.

[0093] Step 9: Feed the output of the discriminator back to the generator as the reward function of the reinforcement learning algorithm to update each agent’s policy network π θ Parameter θ, repeat steps 5 to 9 until the algorithm converges.

[0094] The discriminator's identification results for expert samples and generated samples will be transmitted back to the generator as information. The algorithm used by the generator is a multi-agent reinforcement learning algorithm, such as MAPPO, MADDPG, etc.; the strategy network π is updated using the multi-agent reinforcement learning algorithm θ .

[0095] After the algorithm converges, the training is completed and the parameters of the policy network are determined. At this time, the agent's behavior strategy is the same as the strategy demonstrated by the expert. If the imitation learning algorithm used separates the reward function, the reward function at this time is the reward function of the expert demonstration task.

[0096] Since then, the present invention has enhanced the expert demonstration by utilizing the symmetry in the multi-agent system, and embedded the symmetry loss at the same time, which effectively solves the problem caused by the insufficient number of expert demonstrations and improves the performance of the algorithm. This method can be used in any generative adversarial imitation learning framework. In addition, the data enhancement module can also be used in any imitation learning.

Claims

1. A method for enhancing expert demonstration in group imitation learning based on orthogonal transformation, characterized in that: The specific steps are as follows: Step 1: For a multi-agent system consisting of N agents, obtain the trajectory of each agent through the expert strategy execution task, and thus obtain the expert demonstration τ E ; The set of expert demonstrations for all tasks is Where M is the number of expert demonstrations; is the state set of all agents at time t in the jth expert demonstration; is the set of actions of all agents at time t in the jth expert demonstration; Step 2: Initially randomly set the policy network π with parameters θ for each agent θ And the discriminator network D with parameter ω ω ; Each agent consists of a generator policy network and a discriminator network; Among them, the policy network θ i is the policy network parameter of the i-th agent; the discriminator ω i is the discriminator parameter of the i-th agent; Step 3: Use orthogonal group preset geometric transformations to enhance the states and actions of all agents in each demonstration expert using symmetry; Geometric transformations in Euclidean space include rotations, reflections, and their combinations; they are represented by the orthogonal group O(n), which is a group of n×n orthogonal matrices, defined as follows: Among them, the matrix Z is an orthogonal matrix, representing a geometric transformation; Z T represents the transposed matrix of the matrix Z, and I represents the identity matrix. Represents the set of all n×n matrices over the real number field; Specifically: Step 301: Select rotation and reflection transformation from the orthogonal group O(n) as geometric transformation g, and obtain corresponding rotation matrix and reflection transformation matrix; Step 302: multiply the state and action that need to be transformed of each agent in the current demonstration expert by the rotation matrix and the reflection matrix to obtain the transformed state and action; The details are as follows: The state includes the state s that is equivariant to geometric transformations equ and the state s that is invariant to geometric transformations inv , the transformed state is represented by L g [s]=[L g [s equ ],s inv ]; The actions of the agent include equivariant actions a equ and unchanged action a inv , the transformed action is represented by K g [a]=[K g [a equ ],a inv ]; Step 303: Similarly, each agent in each demonstration expert is transformed in state and action to obtain an enhanced expert demonstration; Step 4: Merge the original expert demonstration with the enhanced expert demonstration to obtain the subsequent learned expert demonstration τ′ E ; That is: t′ E ←t E ∪gτ E ; Step 5: Utilize the policy network π of the generator in each agent θ Generate a trajectory for this agent, and the generated trajectory of all agents is τ G ; Step 6: From the combined expert demonstration τ′ E and generate the trajectory τ G Random sampling is performed to obtain expert samples corresponding to the state actions of each agent and generate samples Step 7: Use the same geometric transformation again to transform the expert samples of each agent and generate samples Do data enhancement to obtain enhanced expert samples And generate samples Step 8: Utilize the Augmented Expert Samples And generate samples Calculate the cross entropy as the symmetry loss, take the sum of the original loss and the symmetry loss as the discriminator loss function, and use the loss function to update the discriminator; The loss function of the discriminator is: is the original loss function, and its calculation formula is: in, is the discriminator, s is the state of the multi-agent system, s′ is the next state of the multi-agent system, a i is the action of the ith agent; The loss function of symmetric entropy is introduced, and its calculation formula is: Among them, L g [s] represents the state after the geometric transformation of state s, L g [s′] represents the state after the geometric transformation of state s′, K g [a i ] means a i Action after geometric transformation; Step 9: Feed the output of the discriminator back to the generator as the reward function of the reinforcement learning algorithm to update each agent’s policy network π θ , and return to step 5 until the algorithm converges.

2. The method for enhancing expert demonstration in group imitation learning based on orthogonal transformation as claimed in claim 1, characterized in that: In step 1, the trajectory refers to: the execution time of each task is a time series of length T, and the state and action of each agent can be collected at each moment; In each task, the actions and states of all agents at T complete moments constitute an expert demonstration.

3. The method for enhancing expert demonstration in group imitation learning based on orthogonal transformation as claimed in claim 1, characterized in that: In step 5, the generated trajectory of each agent includes a series of states and actions selected by the neural network.

4. The method for enhancing expert demonstration in group imitation learning based on orthogonal transformation as claimed in claim 1, characterized in that: In step 8, the goal of the discriminator is to minimize the difference between the expert trajectory samples and the trajectory samples generated by the generator, and to distinguish the generated trajectory from the expert demonstration trajectory as much as possible.

5. The method for enhancing expert demonstration in group imitation learning based on orthogonal transformation as claimed in claim 1, characterized in that: The symmetry of the multi-agent system is exploited to perform data augmentation on expert demonstrations through orthogonal transformations; a symmetry loss is introduced to update the parameters of the discriminator.

Images