A mobile multi-agent knowledge transfer method based on permutation strategy network

Abstract

The application discloses a mobile multi-agent knowledge migration method based on a permutation strategy network and relates to the technical field of multi-agent reinforcement learning. The method comprises the following steps: embedding a permutation invariance strategy network and a permutation homomorphism strategy network into a super network framework, dynamically generating input layer and output layer weight matrices through the super network, and establishing a dynamic adaptation relationship between a joint state-action space and the size of an agent and environmental changes; introducing permutation matrix characteristics to realize decoupling of agent sequence independence and task target responsiveness, and optimizing strategy network parameters through a centralized training-distributed execution architecture; constructing a knowledge migration model containing permutation invariance and homomorphism constraints; and realizing efficient migration of strategies between similar domain tasks. The application solves the problem of low efficiency of knowledge migration caused by changes in the size of an agent and dimension explosion of a joint state-action space in a dynamic complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent reinforcement learning, and more particularly to a mobile multi-agent knowledge transfer method based on a permutation policy network. BACKGROUND

[0002] With the wide application of mobile multi-agent systems in complex scenarios such as industrial collaboration, intelligent navigation, and emergency rescue, the increase in the number of agents and the dynamic changes in the environment lead to an exponential expansion of the joint state-action space, making traditional multi-agent reinforcement learning (MARL) methods face the dual challenges of dimensionality disaster and inefficient knowledge transfer. In multi-agent collaborative tasks, the dynamic interaction between agents and environmental disturbances (such as obstacle movement and task target reallocation) exacerbate the non-stationarity of the system, resulting in extremely low sample efficiency for policy learning.

[0003] Traditional multi-agent knowledge transfer methods mainly rely on cross-domain feature alignment (such as partial domain adaptation PADA), but such methods cannot capture the permutation invariance (the order of agents does not affect the global collaboration goal) and permutation homomorphism (the action output needs to be adjusted with the order of target allocation) of the multi-agent system. For example, in the collaborative navigation task, the physical identity independence of agents requires the input layer to be insensitive to the order, while in the target point reallocation scenario, the action output needs to change synchronously with the order of agents. Traditional methods lack such inductive bias, easily causing policy oscillation or negative transfer, and the transfer efficiency is improved by less than 20% in scenarios where the number of agents changes.

[0004] In recent years, physical information neural networks (PINNs) have improved model interpretability by embedding physical constraints, but their application in multi-agent scenarios still has limitations. For example, the deep Lagrangian network (DeLaN) can ensure the physical consistency of the inertia matrix through mechanical principles, but it cannot model non-conservative interactions between agents (such as communication delay and collaboration conflict), resulting in a policy error rate of over 30% in dynamic task allocation. Existing improved methods (such as introducing a dissipation force network) rely on hand-designed physical models and are difficult to generalize complex interactions; while black-box models driven purely by data (such as FFNN) lack structural constraints, and the risk of policy divergence when extrapolating samples is as high as 40%.

[0005] In addition, the contradiction between the distributed execution requirements of multi-agent systems and the overhead of centralized training is increasingly prominent. Traditional centralized training methods have exponential computational complexity with the increase in the number of agents, while independent learning leads to policy collapse due to environmental non-stationarity. How to ensure global optimization of the policy while achieving efficiency and robustness of distributed execution has become a key bottleneck restricting the large-scale application of multi-agent systems.

[0006] Therefore, a mobile multi-agent knowledge transfer method based on a permutation strategy network is proposed to solve the problems in the prior art, which is an urgent problem for those skilled in the art. SUMMARY

[0007] Therefore, the present application provides a mobile multi-agent knowledge transfer method based on a permutation strategy network, aiming to solve the problem of inefficient knowledge transfer caused by the change of agent scale and the explosion of joint state-action space dimension in a dynamic complex environment.

[0008] To achieve the above purpose, the present application provides the following technical solutions:

[0009] A mobile multi-agent knowledge transfer method based on a permutation strategy network comprises the following steps:

[0010] S1, embedding a permutation invariance strategy network and a permutation homomorphism strategy network into a hypernetwork framework, generating input layer and output layer weight matrices through the hypernetwork, and establishing a dynamic adaptation relationship between the joint state-action space and the agent scale and the environment change;

[0011] S2, introducing permutation matrix characteristics to realize decoupling of agent sequence independence and task target responsiveness, and optimizing strategy network parameters through a centralized training-distributed execution architecture;

[0012] S3, constructing a knowledge transfer model containing permutation invariance and homomorphism constraints;

[0013] S4, for similar domain tasks with changes in the number of agents or dynamic adjustments of the environment, using the constructed knowledge transfer model to realize efficient transfer of strategies between similar domain tasks.

[0014] Optionally, the input layer of the permutation invariance strategy network in S1 satisfies permutation invariance, specifically:

[0015] For any permutation matrix g, the input layer outputs h in satisfies h in (g·X)=h in (X), where X is an agent observation feature vector, and the input layer weight matrix W i is generated by a hypernetwork, and the calculation formula is:

[0016]

[0017] where x i is the observation feature of a single agent, and W i is generated by a hypernetwork according to x i .

[0018] Optionally, the output layer of the permutation homomorphism strategy network in S1 satisfies permutation homomorphism, specifically:

[0019] For any permutation matrix g, the output layer action a satisfies a(g·X)=g·a(X), and the output layer weight matrix W is generated by the super network ji , the calculation formula is:

[0020]

[0021] Wherein, h hidden is the hidden layer output of the neural network, W ji is dynamically adjusted according to the input sequence.

[0022] Optionally, the super network framework in S1 is a neural network, the input is the observation feature of the agent or the task environment parameter, and the output is the weight matrix of the policy network.

[0023] The input layer and the output layer of the super network framework both adopt two fully connected networks, the dimension of the hidden layer is 64, the activation function is ReLU, and the output layer generates the weight matrix through linear transformation.

[0024] Optionally, the introduction of the permutation matrix property in S2 realizes the decoupling of the agent sequence independence and the task target responsiveness, and the specific content of the centralized training-distributed execution architecture optimization policy network parameter is:

[0025] Based on the combination of the policy network and the multi-agent deep reinforcement learning algorithm, the centralized training-distributed execution framework, the global observation optimization strategy training and the local observation independent decision execution are adopted.

[0026] The optimization objective function of the global observation optimization strategy training is:

[0027]

[0028] Wherein, r(θ) is the policy ratio, is the generalized advantage estimation, alpha is the entropy regularization coefficient, S(π θ ) is the policy entropy, is the mathematical expectation, and epsilon is the clipping parameter of PPO.

[0029] Optionally, the similar domain task migration for the change of the number of agents or the dynamic adjustment of the environment in S4 includes the scenes of the change of the number of agents or the dynamic adjustment of the environment, the input layer is insensitive to the number of agents through the permutation invariance, and the target point distribution or the obstacle change task target adjustment is adapted through the permutation homomorphism.

[0030] According to the above technical solution, compared with the prior art, the mobile multi-agent knowledge migration method based on the permutation policy network has the following beneficial effects:

[0031] (1) The application effectively solves the problems of joint state-action space dimension explosion in multi-agent reinforcement learning and inefficient knowledge transfer in dynamic environments; the permutation invariance and permutation homomorphism policy network are integrated into the super network framework, the input layer and output layer weight matrix are dynamically generated, and the dynamic adaptation relationship between the agent scale, environmental changes and policy space is established; the inductive bias based on the permutation matrix characteristics is proposed, the agent sequence independence and task target responsiveness are decoupled, and a knowledge transfer model containing physical consistency constraints is constructed;

[0032] (2) The multi-agent cooperative navigation experiment in the Gym environment shows that the application can adapt to the change of the number of agents and the dynamic obstacle scene without retraining, and realizes efficient policy transfer; compared with existing methods (such as MAPPO and PADA), the learning speed is improved by more than 50%, the asymptotic performance (final reward) is improved by 60%, and the training stability is significantly enhanced;

[0033] (3) In the task of increasing the number of agents, the threshold time of PPN is shortened by 64% (from 10e6 steps to 3.6e6 steps) compared with the baseline MAPPO, and the final reward is improved by 59.4% (from 496.11 to 790.60); in the dynamic environment task, the threshold time is shortened by 78% (from 10e6 steps to 2.2e6 steps), and the final reward is improved by 73.7% (from 410.56 to 713.14);

[0034] (4) In the untrained dynamic obstacle scene, the reward fluctuation range of PPN is reduced by 42% compared with the PADA method, the asymptotic performance (R 2 >0.95) is significantly better than the traditional method, and the policy convergence speed is improved by more than 50% under external disturbance;

[0035] (5) The application ensures that the agent policy conforms to the physical laws of multi-agent interaction (such as sequence independence and target responsiveness) through permutation invariance / homomorphism constraints, avoids the unexplainability of black box models, and improves the reliability of the model;

[0036] (6) The application provides an efficient and explainable technical solution for real-time cooperative decision-making of multi-agent systems in complex dynamic environments through theoretical modeling and experimental verification, which has wide application prospects in industrial cooperation, intelligent navigation and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the application, and those skilled in the art can obtain other drawings without creative labor on the basis of the provided drawings.

[0038] Figure 1 A flow chart of a mobile multi-agent knowledge transfer method based on a permutation strategy network is provided in the present application.

[0039] Figure 2 An output layer permutation homomorphism strategy network structure diagram is provided in the present application.

[0040] Figure 3 An input layer permutation invariance strategy network structure diagram is provided in the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] Referring to Figure 1 The present application discloses a mobile multi-agent knowledge transfer method based on a permutation strategy network, comprising the following steps:

[0043] S1, embedding a permutation invariance strategy network and a permutation homomorphism strategy network into a hypernetwork framework, generating input layer and output layer weight matrices through the hypernetwork, and establishing a dynamic adaptation relationship between a joint state-action space and the size of an agent and environmental changes;

[0044] S2, introducing permutation matrix characteristics to realize decoupling of agent sequence independence and task target responsiveness, and optimizing strategy network parameters through a centralized training-distributed execution architecture;

[0045] S3, constructing a knowledge transfer model containing permutation invariance and homomorphism constraints;

[0046] S4, for similar domain tasks with changes in the number of agents or environmental dynamics, using the constructed knowledge transfer model to realize efficient transfer of strategies between similar domain tasks.

[0047] Further, the input layer of the permutation invariance strategy network in S1 satisfies permutation invariance, specifically:

[0048] For any permutation matrix g, the input layer output h in satisfies h in (g·X)=h in (X), where X is an agent observation feature vector, the input layer weight matrix W i is generated through a hypernetwork, and the calculation formula is:

[0049]

[0050] wherein x i is the observation feature of a single agent, W i is generated by the super network according to x i .

[0051] Further, the output layer of the permutation homomorphism strategy network in S1 satisfies permutation homomorphism, specifically:

[0052] For any permutation matrix g, the output layer action a satisfies a(g·X)=g·a(X), and the output layer weight matrix W ji is generated by the super network, and the calculation formula is:

[0053]

[0054] wherein h hidden is the output of the neural network hidden layer, W ji is dynamically adjusted with the input order.

[0055] Specifically, the permutation matrix characteristic constraint: the orthogonality of the permutation matrix ((P T P=E)) and the transformation property ((PTAP)) are used to ensure the mathematical consistency of the permutation invariant strategy network (PI) and the permutation homomorphism strategy network (PE), wherein (P) is a permutation matrix, and (A) is a state-action space matrix.

[0056] Further, the super network framework in S1 is a neural network, the input is the observation feature of the agent or the task environment parameter, and the output is the weight matrix of the strategy network;

[0057] The input layer and the output layer of the super network framework both adopt a two-layer fully connected network, the hidden layer dimension is 64, and the activation function is ReLU, and the output layer generates the weight matrix through linear transformation.

[0058] Further, the permutation matrix characteristics are introduced in S2 to realize the decoupling of the agent order independence and the task target responsiveness, and the specific content of the centralized training-distributed execution architecture for optimizing the strategy network parameters is:

[0059] Based on the combination of the strategy network and the multi-agent deep reinforcement learning algorithm, a centralized training-distributed execution framework, global observation optimization strategy training, and local observation independent decision execution are adopted;

[0060] The optimization objective function of the global observation optimization strategy training is:

[0061]

[0062] wherein r(θ) is the strategy ratio, For generalized advantage estimation, alpha is an entropy regularization coefficient, S(pi θ ) is policy entropy, is the mathematical expectation, and epsilon is the clipping parameter of PPO.

[0063] Further, the similar domain task migration in S4 for the change of the number of agents or the adjustment of the environment dynamics includes the scenarios of the change of the number of agents or the adjustment of the environment dynamics, the input layer is ensured to be insensitive to the number of agents through the invariance of permutation, and the target point allocation or obstacle change task target adjustment is adapted through the homomorphism of permutation.

[0064] The experimental verification framework of the application: based on OpenAI Gym, two-dimensional grid environment is constructed, two types of tasks of increasing the number of agents (from 3 to 5) and complexifying the environment (dynamic obstacles) are set, and the average round reward and threshold time (the number of training rounds to reach the convergence reward of the baseline algorithm) are used as evaluation indexes.

[0065] Embodiment 1

[0066] To solve the problems of dimension explosion of joint state-action space and inefficient knowledge transfer in dynamic environment in multi-agent reinforcement learning, the application embeds the permutation invariance (PI) and permutation equivariance (PE) policy network into a super network framework, and proposes a similar domain knowledge transfer method based on permutation policy network (PPN).

[0067] For the input order sensitivity problem caused by the change of the number of agents, the input layer weight matrix is dynamically generated through the super network, the local observation features of the agents are mapped into globally invariant features through the super network, and the input layer is ensured to be insensitive to the increase or decrease of the number of agents. For example, in the scene of expanding from 3 agents to 5 agents, the PI network maps the local observation features of the agents into globally invariant features through the summation pooling operation h in =Σ i W i x i eliminates the order interference, wherein W i is generated by the super network according to the observation x i of a single agent.

[0068] For the action response demand caused by the task target adjustment, the PE network is designed to generate the output layer dynamic weight matrix through the super network, so that the action output and the agent order maintain the homomorphism relationship. Taking the dynamic target point allocation task as an example, the PE network ensures that when the order of the target point is replaced, the action of the agent a j is adjusted synchronously, which satisfies a(g·X)=g·a(X) (g is a permutation matrix), and maintains the consistency of system cooperation.

[0069] Experiments were conducted in the OpenAI Gym multi-agent cooperative navigation environment, setting two types of tasks: agent number expansion (from 3 to 5) and dynamic obstacles. The results show that in the agent number expansion task, the threshold time of PPN is shortened by 64% compared with the baseline MAPPO, and the final reward is increased by 59.4%; in the dynamic environment task, the threshold time is shortened by 78%, and the reward fluctuation range is reduced by 42%. Compared with the partial domain adaptation method PADA, PPN is significantly optimized in learning speed, asymptotic performance and stability, verifying the effectiveness of the permutation policy network and the dynamic adaptation mechanism of the super network.

[0070] Embodiment 2

[0071] In a multi-agent system, when the number of agents changes (such as from 3 to 5) or the input order is randomly arranged, the traditional fixed architecture network cannot maintain the consistency of feature extraction, resulting in knowledge transfer failure. Based on the PI network structure of Figure 2 , this embodiment solves the input order sensitivity problem and ensures the permutation invariance of the joint state space, including the following contents:

[0072] 1. PI network core architecture:

[0073] Super network design: as shown in Figure 2 , the super network of the PI network is a double-layer fully connected neural network, the input is the local observation (such as position, velocity, target point distance) of a single agent, the hidden layer uses ReLU activation function, and the output dimension is a weight matrix W in with a dimension of d feat xd i , where d in is the input layer dimension.

[0074] Sum-pooling operation: all agent features are fused through , where m is the number of agents. This operation ensures that the global feature h in remains unchanged when the input order is permuted, i.e. h in (g·X)=h in (X), where g is the permutation matrix.

[0075] 2. Dynamic adaptation of agent scale change:

[0076] Parameter generation mechanism: when the number of agents increases from m to m', the super network does not need to be retrained, and directly generates the corresponding W i matrix for the new agents. For example, when the number of agents is expanded from 3 to 5, the super network generates W4 and W5 for the observations x4 and x5 of the 4th and 5th agents, and keeps the dimension of h in as d in , avoiding dimension explosion.

[0077] Physical meaning: Through permutation invariance, the physical identity (such as number) of an agent does not affect global features, but only depends on its observation content (such as positional relationships), which is consistent with the isomorphism assumption of multi-agent systems.

[0078] 3. Integration with the MAPPO algorithm:

[0079] Training phase: In intensive training, the output h of the PI network is... in The Critic network, which uses MAPPO as the global feature input, optimizes the PPO loss function to ensure that the hypernetwork parameters {W} are consistent. i Co-convergence with the policy network;

[0080]

[0081] Where r(θ) is the strategy ratio, For generalized advantage estimation, α is the entropy regularization coefficient, S(π) θ ) represents the policy entropy. Let be the mathematical expectation, and ∈ be the clipping parameter of PPO.

[0082] Execution phase: The agent only needs to input its own observation x into the supernetwork. i You can then generate your own W. i It eliminates the need to share sequence information with other intelligent agents, thus reducing communication overhead.

[0083] Example 3

[0084] In multi-agent collaborative tasks, dynamic adjustments to the task objective (such as target point reassignment or agent role switching) can lead to a mismatch between the action space and the agent order. Traditional fixed-weight networks cannot synchronously adjust action outputs, resulting in policy confusion. This embodiment is based on... Figure 3 The PE network structure addresses the issue of sequential dependencies in action responses, ensuring consistent system collaboration. This includes the following:

[0085] 1. PE Network Core Architecture:

[0086] Hypernetwork design: such as Figure 3 As shown, the supernetwork input of the PE network is the hidden layer features. (Includes global collaboration information), output dimension is d action ×d hidden The weight matrix W ji , where j is the agent index and i is the action dimension index.

[0087] Generation of substitution-based isomorphic actions: through The computing action ensures that when the input order is affected by the permutation matrix g, the action output satisfies a(gX) = g a(X), for example, when the target point order is permuted from [T1, T2, T3] to [T2, T1, T3], the action sequence generated by the PE network is adjusted synchronously to [a2, a1, a3], maintaining the one-to-one correspondence between the agent and the target.

[0088] 2. Dynamic task target adaptation:

[0089] Dynamic weight matrix update: when the task target is permuted (such as reassigning the search and rescue area in emergency rescue), the super network generates a new W hidden ' matrix according to the new target order information (embedding h ji , without retraining the entire network.

[0090] Action space decoupling: divide the action into entity-dependent actions (such as target point assignment) and entity-independent actions (such as movement speed), entity-dependent actions achieve permutation invariance through PE network, entity-independent actions maintain permutation invariance through PI network, ensuring that the action logic conforms to physical constraints.

[0091] 3. Integration with MAPPO algorithm:

[0092] Training phase: in centralized training, the target point order information is taken as part of the global state, and the super network parameters are optimized through the PPO loss function, strengthening the commutative relationship between action and target order.

[0093] Execution phase: the agent directly maps to the corresponding action through W ji matrix according to the local observation of the target point index, with a decision delay of less than 10ms, meeting the real-time task adjustment requirements.

[0094] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0095] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A mobile multi-agent knowledge transfer method based on a permutation strategy network, characterized in that, The method comprises the following steps: S1, embedding the permutation invariance policy network and the permutation homomorphism policy network into a super network framework, dynamically generating input layer and output layer weight matrices through the super network, and establishing a dynamic adaptation relationship between the joint state-action space and the agent scale and environmental changes; S2, introducing permutation matrix characteristics to realize the decoupling of agent sequence independence and task target responsiveness, and optimizing policy network parameters through a centralized training-distributed execution architecture; S3, constructing a knowledge transfer model containing permutation invariance and homomorphism constraints; S4, for similar domain tasks with changes in the number of agents or dynamic adjustments of the environment, using the constructed knowledge transfer model to realize efficient transfer of policies between similar domain tasks; In S1, the super network framework is a neural network, the input is the observation features of the agent or the task environment parameters, and the output is the weight matrix of the policy network; The input layer and the output layer of the super network framework both use two-layer fully connected networks, the hidden layer dimension is 64, and the activation function is ReLU, and the output layer generates the weight matrix through linear transformation.

2. The mobile multi-agent knowledge transfer method based on permutation policy network according to claim 1, characterized in that: In S1, the input layer of the permutation invariance policy network satisfies the permutation invariance, specifically: For any permutation matrix , the input layer output satisfies , where is the agent observation feature vector, and the input layer weight matrix is generated by the hypernetwork, and the calculation formula is: wherein, is an observation feature of the single agent, generated by the super network according to .

3. The mobile multi-agent knowledge transfer method based on permutation policy network according to claim 1, characterized in that: In S1, the output layer of the permutation homomorphism policy network satisfies the permutation homomorphism, specifically: For any permutation matrix , the output layer action satisfies , the output layer weight matrix is generated by a super network, and the calculation formula is: wherein, is the neural network hidden layer output, is dynamically adjusted with input order.

4. The mobile multi-agent knowledge transfer method based on permutation policy network according to claim 1, characterized in that: In S2, the permutation matrix characteristics are introduced to realize the decoupling of agent sequence independence and task target responsiveness, and the specific content of optimizing policy network parameters through a centralized training-distributed execution architecture is: Based on the combination of policy network and multi-agent deep reinforcement learning algorithm, a centralized training-distributed execution framework, global observation optimization policy training, and local observation independent decision execution are adopted; The optimization objective function of global observation optimization policy training is: where, is the policy ratio, is the generalized advantage estimate, is the entropy regularization coefficient, is the policy entropy, is the mathematical expectation, is the clipping parameter of PPO.

5. The mobile multi-agent knowledge transfer method based on permutation policy network according to claim 1, characterized in that: In S4, the similar domain task transfer for changes in the number of agents or dynamic adjustments of the environment includes scenes with changes in the number of agents or dynamic adjustments of the environment, the input layer is insensitive to the number of agents through permutation invariance, and the target point allocation or obstacle change task target adjustment is adapted through permutation homomorphism.

Images