A heterogeneous multi-agent reinforcement learning method based on sparse parameter sharing

Abstract

The present invention relates to multi-agent collaboration technology, and specifically to a heterogeneous multi-agent reinforcement learning method based on sparse parameter sharing. It includes two parts: training multiple agent models and agent decision-making. In the process of training the multi-agent model, an over-parameterized network is initialized as a network shared by all agents, and the shared network is trained to obtain basic network parameters; based on the basic network parameters, the model representation of each agent sub-network is obtained through a binary mask. In the agent decision-making process, each agent obtains its own observation information, and inputs its own observation information into the corresponding sub-network, and obtains the output of the sub-network as a basis for decision-making. Compared with the prior art, the present invention adopts a sparse sharing mechanism and obtains the model representation of each agent sub-network through a binary mask, which can effectively reduce the negative transfer phenomenon caused by the traditional agent model parameter sharing and improve the performance of the multi-agent system.

Description

Technical Field

[0001] The present invention relates to multi-agent collaboration technology, and in particular to a heterogeneous multi-agent reinforcement learning method based on sparse parameter sharing. Background Art

[0002] In traditional multi-agent systems, heterogeneous agents typically share a common agent model, achieved by sharing the same set of model parameters. This approach effectively reduces the number of model parameters, thereby reducing the complexity and time cost of model training. However, when the correlation between agents is weak, the training of different types of agents will affect each other, reducing model performance and potentially leading to negative transfer. Summary of the Invention

[0003] The purpose of the present invention is to provide a heterogeneous multi-agent reinforcement learning method based on sparse sharing to address the problems existing in the above-mentioned prior art. By introducing a sparse sharing mechanism, the negative transfer phenomenon caused by parameter sharing of traditional agent models is effectively reduced, and the correctness of agent decision-making is improved.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A heterogeneous multi-agent reinforcement learning method based on sparse sharing, comprising:

[0006] Step 1: Train the multi-agent model

[0007] S1. Initialize an over-parameterized network as a shared network for all agents;

[0008] S2. Unify the observation vector dimensions of each type of agent. Based on the unified observation vector dimensions, use end-to-end training to train the shared network to obtain the optimal shared network parameters.

[0009] S3. Use the optimal shared network parameters as the basic network parameters, and generate sub-networks for each agent based on the basic network parameters; train each type of agent sub-network separately to obtain the action with the maximum Q value output by each agent, which is used to combine into a joint action;

[0010] Step 2: Use the trained multi-agent model to make decisions to improve the accuracy of the agent's decision-making.

[0011] Furthermore, the S1 over-parameterized network consists of a 28x64 fully connected layer, a 64x64 GRU layer, and a 64x16 fully connected layer connected in sequence.

[0012] Furthermore, the detailed operation steps of S2 are:

[0013] S2.1. Unify the dimensions of observation vectors for all types of agents;

[0014] S2.2. Input the unified dimension observation vectors of each agent into the shared network, calculate the Q value corresponding to each action of each agent, and select the action corresponding to the maximum Q value of each agent;

[0015] S2.3. Combine the actions corresponding to the maximum Q value of each agent to obtain a joint action; calculate the team Q value based on the action collection;

[0016] S2.4. The system constructs a training loss function based on the team Q value and reversely updates the shared network parameters based on the training loss function. The training loss function calculation formula is as follows:

[0017]

[0018] Where b is the size of the replay buffer sampling; o is the joint observation of the agent, and the joint observation is the collection of all the local observations of the agents o = (o 1 ,o 2 ...o n ), where o i is the local observation of agent i; u is the joint action of agents, and the joint action is the collection of all agent actions u=(u 1 ,u 2 ...u n ), where u i is the action of agent i; s is the absolute global state, which is provided by the simulation environment system of reinforcement learning and contains the state information of all agents and the environment information; θ is the parameter of the estimated network, y tot is the output of the target network; y tot is calculated as follows:

[0019] y tot =r+γmax u′ Q tot (o′,u′,s′;θ - )

[0020] where γ is the discount factor, set to 0.99, r is the team reward, o′ is the joint observation for the next step, u′ is the joint action for the next step, s′ is the next global state, and θ - are the parameters of the target network.

[0021] Furthermore, the method for unifying the dimensions of each agent in S2.1 is as follows:

[0022] Assume that the input vector dimension of the shared network is the largest dimension of all agents; fill the observation vectors of all agents that do not meet the dimension requirements with zero values ​​to align them with the largest dimension of the agents.

[0023] Furthermore, the S2.3 team Q value is calculated by a hybrid network consisting of a parameter generation network and an inference network; wherein the parameter generation network is a super network that uses the absolute global information provided by the reinforcement learning simulation environment system during training as input to generate the weight matrix and bias vector of the inference network, and the weight matrix is ​​constrained to be non-negative to satisfy the monotonicity constraint;

[0024]

[0025] Among them, Q tot is the team Q value, Q i is the Q value of agent i, is the symbol of partial derivative, A is the action space of the agent;

[0026] The inference network is a feedforward neural network whose input is the joint action and the team Q value output is calculated based on the weight matrix and bias vector.

[0027] Furthermore, the method of generating each agent sub-network in S3 includes the following sub-steps:

[0028] S3.1. Obtaining the sub-network parameters of each agent

[0029] S3.1.1. Assume that each type of agent corresponds to a set of binary masks, which are used to control the activation status of parameters. In the binary mask, 1 indicates that the parameter is activated, and 0 indicates that the parameter is not activated.

[0030] S3.1.2. Initialize the agent. After initialization, all masks of the agent are set to 1.

[0031] S3.1.3. Each agent extracts some parameters from the basic network parameters; the binary mask corresponding to each agent and its extracted parameters is set to 1, and the rest are set to 0, thereby obtaining the sub-network parameters of each agent;

[0032] S3.2. Set the parameters of the agent to the parameters of the shared network Parameters based on shared network Perform a separate, preset number of training iterations on each agent sub-network;

[0033] S3.3. After the training is completed, the parameters of each intelligent sub-network are pruned to remove invalid data.

[0034] Furthermore, the specific steps of the S3.3 pruning operation are as follows:

[0035] S3.3.1. Calculate the absolute values ​​of the agent sub-network parameters;

[0036] S3.3.2. Sort the parameters by their absolute values, with the top 50% of the parameters being classified as having the largest values ​​and the bottom 50% of the parameters being classified as having the smallest values.

[0037] S3.3.3. Keep the larger parameter value part and cut off the smaller parameter value part; then, in the corresponding mask, set the mask value corresponding to the cut parameter to 0 to ensure that these parameters no longer play a role in subsequent training and decision-making processes.

[0038] Furthermore, the detailed process of step 2 includes the following sub-steps:

[0039] (1) Obtaining observation information of each agent and unifying the dimensions of the vector. The observation information includes the state information of the agent and observation data from the environment. The state information includes position, speed, and target.

[0040] Determine the corresponding sub-network according to the type of intelligent agent;

[0041] (2) Input the acquired agent observation information into the corresponding sub-network to obtain the Q value corresponding to each action, and select the action output with the largest Q value as the basis for decision-making.

[0042] After adopting the above technical solution, the present invention has the following advantages:

[0043] (1) The present invention adopts a sparse sharing mechanism and obtains the model representation of each agent sub-network through binary masking, which can effectively reduce the negative transfer phenomenon caused by traditional agent model parameter sharing and improve the performance of the multi-agent system.

[0044] (2) The present invention can provide more refined feature representations based on the characteristics of different types of intelligent agents, thereby further improving the decision-making accuracy of the intelligent agents.

[0045] (3) The present invention calculates the team Q value through a hybrid network, which can take into account the contribution of a single intelligent agent and the collaboration of multiple intelligent agents, thereby improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of sparse sharing in an embodiment of the invention;

[0047] Figure 2 A schematic diagram of sparse sharing in an embodiment of the invention;

[0048] Figure 3 It is the overall framework diagram of the model in the embodiment of the invention;

[0049] Figure 4 is a structural diagram of a hybrid network in an embodiment of the invention; DETAILED DESCRIPTION

[0050] This paper provides a heterogeneous multi-agent reinforcement learning method based on sparse sharing, aiming to effectively reduce the negative transfer phenomenon caused by parameter sharing in traditional agent models and improve the performance of multi-agent systems. The method includes two steps: training the multi-agent model and using the trained multi-agent model to make agent decisions.

[0051] When training a multi-agent model, an over-parameterized network is first initialized as the shared network for all agents. The shared network is trained for a specified number of rounds using an end-to-end approach to obtain the base network parameters. For each agent type, a set of binary masks is assigned to control which parameters are activated. Each agent type extracts a subnetwork from the base network, selecting some parameters and setting their corresponding masks to 1 and the rest to 0. Each subnetwork is then trained individually to optimize performance for that agent type. The output of each agent serves as the input to a hybrid network, which outputs the team's Q-value. This incorporates the global state into the hybrid network input, enhancing the correlation between agents.

[0052] During agent decision-making, the agent obtains its own observation information, which includes its state information and observation data from the environment. State information includes location, speed, and target. Each type of agent selects its own subnetwork to provide optimal decision support. The agent's observation information is then input into the corresponding subnetwork, and the subnetwork's output is used as the basis for decision-making.

[0053] Example:

[0054] The following is an embodiment for illustrating the present invention in detail.

[0055] Imagine a system with two types of robots: a drone and an unmanned vehicle. The drone and the vehicle need to collaborate to find a target. The drone and the vehicle have different fields of view. Drones have a wider field of view in the air. Their movements are also different. Unlike unmanned vehicles, drones can control not only forward, backward, left, and right movements, but also altitude. Thus, the drones and the vehicle form a heterogeneous multi-agent environment.

[0056] This system includes two different types of agents: Agent A (unmanned vehicle) and Agent B (drone). The observation spaces of Agent A and Agent B are 20 and 28, respectively, and the action spaces are 16 and 10, respectively. The system contains three Agent A agents and two Agent B agents. All agents must collaborate to complete the task. The global vector is absolute global information provided by the system during training. It contains the absolute positions and states of all agents and has a dimension of 128.

[0057] See also Figure 1 The specific implementation process is as follows:

[0058] Step 1: Train the multi-agent model

[0059] S1. Initialize an over-parameterized network as a shared network for all agents. The role of the shared network is to generate individual Q values ​​based on the observation information of each agent.

[0060] The structure of the over-parameterized network is shown in Table 1: it includes a 28x64 fully connected layer, a 64x64 GRU layer, and a 64x16 fully connected layer, and the three layers are connected sequentially.

[0061] Table 1 Agent network structure

[0062] name size Fully connected layer 1 28x64 GRU layer 64x64 Fully connected layer 2 64x16

[0063] S2. Train the shared network according to the set rounds to obtain the optimal shared network parameters. The optimal shared network parameters are used as the basic network parameters.

[0064] In this embodiment, the training rounds are set to 10,000 according to the requirements, and the basic network parameters are finally obtained. The training steps of the shared network are:

[0065] S2.1. Because the observation vectors of different types of agents have different dimensions, the input vector dimensions need to be unified. In this embodiment, the input vector dimension is set to the largest dimension among all agents, and zero values ​​are padded when inputting other types of agents to achieve consistent dimensions. The observation vectors of the two agents are converted to a uniform length of 28, and insufficient zeros are padded to align the dimensions of the observation vectors of the two agents. The two aligned vectors are input into the shared network, which outputs the Q value corresponding to each action of the agent. The Q value of actions for which the agent does not exist is set to -9999999.

[0066] S2.2. Based on the output of the shared network in S2.1, select the action corresponding to the maximum Q value from each agent.

[0067] S2.3. Combine the actions corresponding to the maximum Q value of each agent to obtain a joint action, and calculate the team Q value based on the action collection. The team Q value is calculated using a hybrid network consisting of a parameter generation network and an inference network. The parameter generation network is a super network that uses the absolute global information provided by the reinforcement learning simulation environment system during training as input to generate the weight matrix and bias vector of the inference network. The weight matrix is ​​constrained to be non-negative to meet the monotonicity constraint.

[0068]

[0069] Among them, Q tot is the team Q value, Q i is the Q value of agent i, is the symbol of partial derivative, A is the action space of the agent;

[0070] The inference network is a feedforward neural network whose input is the joint action. It calculates the team Q-value output based on the weight matrix and bias vector. In this embodiment, the structure and parameters of the inference network are shown in Figure 2. The parameter generation network consists of multiple fully connected layers, the last of which uses an absolute value activation function. The parameter generation network structure and parameters are shown in Table 3.

[0071] Table 2 Inference network structure

[0072]

[0073]

[0074] Table 3 Parameter generation network structure

[0075] name size Fully connected layer 1 (used to generate W1) 128x160 Fully connected layer 2 (used to generate bias for W1) 128x32 Fully connected layer 3 (used to generate W2) 128x32 Fully connected layer 4 (used to generate W2 bias) 128x32 Fully connected layer 5 32x1

[0076] S2.4. The system constructs a training loss function based on the team Q value and reversely updates the shared network parameters based on the training loss function. The training loss function calculation formula is as follows:

[0077]

[0078] Where b is the size of the replay buffer sampling; o is the joint observation of the agent, and the joint observation is the collection of all the local observations of the agents o = (o 1 ,o 2 ...o n ), where o i is the local observation of agent i; u is the joint action of agents, and the joint action is the collection of all agent actions u=(u 1 ,u 2 ...u n ), where u iis the action of agent i; s is the absolute global state, which is provided by the simulation environment system of reinforcement learning and contains the state information of all agents and the environment information; θ is the parameter of the estimated network, y tot is the output of the target network; y tot is calculated as follows:

[0079] y tot =r+γmax u′ Q tot (o′,u′,s′;θ - )

[0080] where γ is the discount factor, set to 0.99, r is the team reward, o′ is the joint observation for the next step, u′ is the joint action for the next step, s′ is the next global state, and θ - are the parameters of the target network.

[0081] S3. Use the optimal shared network parameters as the basic network parameters, and generate sub-networks of each agent based on the basic network parameters; train each type of agent sub-network separately to obtain the action with the maximum Q value of each intelligent output, which is used to combine into a joint action; refer to Figure 2 , the generation of the sub-network includes the following sub-steps:

[0082] S3.1. Obtain the sub-network parameter representation of each agent:

[0083] S3.1.1. Assume that each type of agent corresponds to a set of binary masks, which are used to control which parameters are activated; in the binary mask, 1 indicates that the parameter is activated, and 0 indicates that the parameter is not activated.

[0084] S3.1.2. Initialize the agent. After initialization, all masks of the agent are set to 1.

[0085] S3.1.3 For each agent type, a subnetwork is extracted from the base network parameters. This involves selecting some parameters and setting the corresponding masks to 1, while the rest are set to 0, thereby obtaining the parameter representation of the subnetwork for each agent type. In this way, the subnetwork of each agent is part of the base network and may overlap with the subnetworks of other agent types.

[0086] In this embodiment, a subnet mask M is assigned to each of the two types of agents in the multi-agent model. A ,M B ∈{0,1}, the subnet mask is initialized to all 1. Assume that the model parameters of the agent subnetworks A and B can be expressed as θ A =θ ε ⊙M A ,θ B =θ ε ⊙MB , where ⊙ represents element-wise multiplication.

[0087] S3.2. For Agent A and Agent B, perform the following operations respectively:

[0088] Set the agent's parameters to those of the shared network Each agent is trained for 10,000 iterations. Figure 3 In this embodiment, each agent type corresponds to a subnetwork. During subsequent training, Agents A and B only need to update their own subnetworks. Because the parameters of each agent's subnetwork consist of both unique parameters and parameters shared by all agents, this provides a more refined feature representation for each agent while preserving shared features. Furthermore, during the update process, shared parameters are updated jointly by all agent types, achieving parameter sharing.

[0089] S3.3 After training, the parameters of each agent sub-network are pruned to remove invalid data. The specific steps of the pruning operation are as follows:

[0090] S3.3.1. Calculate the absolute values ​​of the agent sub-network parameters;

[0091] S3.3.2. Sort the parameters by their absolute values, with the top 50% of the parameters being classified as having the largest values ​​and the bottom 50% of the parameters being classified as having the smallest values.

[0092] S3.3.3. Keep the larger parameter value part and cut off the smaller parameter value part; then, in the corresponding mask, set the mask value corresponding to the cut parameter to 0 to ensure that these parameters no longer play a role in subsequent training and decision-making processes.

[0093] Step 2: Use the trained multi-agent model to make decisions. During execution, each agent only needs to make decisions by itself, achieving the effect of distributed decision-making. Specific steps:

[0094] 1. Agents A and B obtain their own observation information and convert its dimension to 28 by zero-padding.

[0095] 2. Each type of agent selects its corresponding sub-network. Agent A selects type A agents, and agent B selects type B sub-network.

[0096] 3. Input the agent's observation information into the corresponding sub-network and obtain the sub-network output as the basis for decision-making. Input the observations of Agent A and Agent B into the corresponding sub-network respectively, obtain the Q value corresponding to each action, and select the action with the largest Q value as the output.

[0097] Negative transfer refers to the phenomenon that when agents of different types or characteristics are forced to share the same set of model parameters, the model performance will be degraded or fail. For example, in a multi-agent system, if one agent is used to explore the environment and the other agent is used to perform tasks, then if they share the same set of model parameters, the exploration agent may interfere with the behavior of the task-performing agent, making it unable to complete the task. The present embodiment provides a heterogeneous multi-agent reinforcement learning method based on sparse sharing. By introducing a sparse sharing mechanism, agents of different types or characteristics can only share some relevant parameters, effectively reducing the negative transfer phenomenon and improving the correctness of agent decision-making.

Claims

1. A heterogeneous multi-agent reinforcement learning method based on sparse sharing, comprising: Step 1: Train the multi-agent model S1. Initialize an over-parameterized network as a shared network for all agents; S2. Unify the observation vector dimensions of each type of agent. Based on the unified observation vector dimensions, use end-to-end training to train the shared network to obtain the optimal shared network parameters. The detailed process includes: S2.

1. Unify the dimensions of observation vectors for all types of agents; S2.

2. Input the unified dimension observation vectors of each agent into the shared network, calculate the Q value corresponding to each action of each agent, and select the action corresponding to the maximum Q value of each agent; S2.

3. Combine the actions corresponding to the maximum Q value of each agent to obtain a joint action; calculate the team Q value based on the action collection; S2.

4. The system constructs a training loss function based on the team Q value and reversely updates the shared network parameters based on the training loss function. The training loss function calculation formula is as follows: Where b is the size of the replay buffer sampling; o is the joint observation of the agent, and the joint observation is the collection of all the local observations of the agents o = (o 1 ,o 2 ...o n ), where o i is the local observation of agent i; u is the joint action of agents, and the joint action is the collection of all agent actions u=(u 1 ,u 2 ...u n ), where u i is the action of agent i; s is the absolute global state, which is provided by the simulation environment system of reinforcement learning and contains the state information of all agents and the environment information; θ is the parameter of the estimated network, y tot is the output of the target network; y tot is calculated as follows: y tot =r+γmax u′ Q tot (o′,u′,s′;θ - ) where γ is the discount factor, set to 0.99, r is the team reward, o′ is the joint observation for the next step, u′ is the joint action for the next step, s′ is the next global state, and θ - are the parameters of the target network; S3. Use the optimal shared network parameters as the basic network parameters, and generate sub-networks for each agent based on the basic network parameters; train each type of agent sub-network separately to obtain the action with the maximum Q value output by each agent, which is used to combine into a joint action; Step 2: Use the trained multi-agent model to make decisions to improve the accuracy of the agent's decision-making.

2. The heterogeneous multi-agent reinforcement learning method based on sparse sharing according to claim 1, characterized in that: The S1 over-parameterized network consists of a 28x64 fully connected layer, a 64x64 GRU layer, and a 64x16 fully connected layer connected in sequence.

3. The heterogeneous multi-agent reinforcement learning method based on sparse sharing according to claim 1, characterized in that: The method for unifying the dimensions of each agent in S2.1 is: Assume that the input vector dimension of the shared network is the largest dimension of all agents; fill the observation vectors of all agents that do not meet the dimension requirements with zero values ​​to align them with the largest dimension of the agents.

4. The heterogeneous multi-agent reinforcement learning method based on sparse sharing according to claim 1, characterized in that: The S2.3 team Q value is calculated by a hybrid network, which is composed of a parameter generation network and an inference network; wherein the parameter generation network is a super network that uses the absolute global information provided by the reinforcement learning simulation environment system during training as input to generate the weight matrix and bias vector of the inference network, and the weight matrix is ​​constrained to be non-negative to satisfy the monotonicity constraint; Among them, Q tot is the team Q value, Q i is the Q value of agent i, is the symbol of partial derivative, A is the action space of the agent; The inference network is a feedforward neural network whose input is the joint action and the team Q value output is calculated based on the weight matrix and bias vector.

5. The heterogeneous multi-agent reinforcement learning method based on sparse sharing according to claim 1, characterized in that: The method S3 for generating each agent sub-network includes the following sub-steps: S3.

1. Obtaining the sub-network parameters of each agent S3.1.

1. Assume that each type of agent corresponds to a set of binary masks, which are used to control the activation status of parameters. In the binary mask, 1 indicates that the parameter is activated, and 0 indicates that the parameter is not activated. S3.1.

2. Initialize the agent. After initialization, all masks of the agent are set to 1. S3.1.

3. Each agent extracts some parameters from the basic network parameters; the binary mask corresponding to each agent and its extracted parameters is set to 1, and the rest are set to 0, thereby obtaining the sub-network parameters of each agent; S3.

2. Set the parameters of the agent to the parameters of the shared network Parameters based on shared network Perform a separate, preset number of training iterations on each agent sub-network; S3.

3. After the training is completed, the parameters of each intelligent sub-network are pruned to remove invalid data.

6. The heterogeneous multi-agent reinforcement learning method based on sparse sharing according to claim 5, characterized in that: The specific steps of the S3.3 pruning operation are as follows: S3.3.

1. Calculate the absolute values ​​of the agent sub-network parameters; S3.3.

2. Sort the parameters by their absolute values, with the top 50% of the parameters being classified as having the largest values ​​and the bottom 50% of the parameters being classified as having the smallest values. S3.3.

3. Keep the larger parameter value part and cut off the smaller parameter value part; then, in the corresponding mask, set the mask value corresponding to the cut parameter to 0 to ensure that these parameters no longer play a role in subsequent training and decision-making processes.

7. The heterogeneous multi-agent reinforcement learning method based on sparse sharing according to claim 1, characterized in that: The detailed process of step 2 includes the following sub-steps: (1) Obtaining observation information of each agent and unifying the dimensions of the vector. The observation information includes the state information of the agent and observation data from the environment. The state information includes position, speed, and target. Determine the corresponding sub-network according to the type of intelligent agent; (2) Input the acquired agent observation information into the corresponding sub-network to obtain the Q value corresponding to each action, and select the action output with the largest Q value as the basis for decision-making.

Images