Multi-agent reinforcement learning exploration method based on policy diversity

Abstract

The application discloses a kind of multi-agent reinforcement learning exploration methods based on strategy diversity, belong to reinforcement learning technical field.The method of the present application includes: constructing double-strategy network architecture including explorer network, user network and predictor network;Double-reference strategy with significant exploration direction difference is constructed;Experience replay samples are constructed by collecting cluster probe trajectory data;Based on explorer reference Q value and predictor predicted Q value, the local prediction error of each unmanned single body is calculated and the novelty of the environment is generated internal reward;At the same time, through three types of collaborative mechanisms of global prediction error constraint, variance regularization constraint and attention weighting constraint, the global collaborative optimization of single body local prediction error is carried out;Fusion environment external task reward and collaborative optimization after internal reward, complete the gradient update of user network and predictor network, and the optimal cluster collaborative exploration strategy is obtained by iterative training.The present application effectively improves the exploration efficiency of agent by integrating global multi-agent interaction information into single-agent local error calculation process, and weighting fusion is carried out on the local internal reward of each agent.

Description

Technical Field

[0001] This invention relates to multi-agent reinforcement learning technology, and more particularly to a multi-agent reinforcement learning exploration method based on policy diversity. Background Technology

[0002] In reinforcement learning, the trade-off between exploration and exploitation is one of the most critical issues, attracting significant attention in single-agent reinforcement learning research. A natural approach to balancing exploration and exploitation is counting-based methods, which require statistically analyzing the number of visits to all states in the environment. However, these methods do not scale well to continuous environments. Other exploration methods for single-agent reinforcement learning include, but are not limited to, maximum entropy, reward shaping, and curiosity. For example, SAC trains a policy by simultaneously maximizing expected reward and policy entropy; RND uses the prediction error between the prediction network and the fixed random initialization network to define an intrinsic reward, measuring the novelty of a sample; to better extract valuable information from states, ICM learns an inverse environment dynamic, predicting intermediate actions from the current state and the next state in a supervised manner. The prediction error of the dynamic model is then treated as an intrinsic reward for more efficient exploration. However, due to the lack of coordinated exploration, the above methods may perform poorly when directly extended to multi-agent scenarios.

[0003] Since different agents need to consider each other's information and coordinate their exploration directions when exploring the environment, multi-agent collaborative exploration is achieved. This invention aims to utilize attention mechanisms to coordinate the exploration directions of different agents, thereby effectively improving the exploration efficiency of agents. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multi-agent reinforcement learning exploration method based on policy diversity, which addresses the shortcomings of the prior art.

[0005] The technical solution adopted by this invention to solve its technical problem is: a multi-agent reinforcement learning exploration method based on policy diversity, comprising the following steps: 1) Construct a dual-strategy network architecture; the dual-strategy network architecture includes a user network for environmental interaction, an explorer network for exploration guidance, and a predictor network; Both the user network and the explorer network are configured with a pair of independent dual-Q value networks; The predictor network is a multi-layer neural network used to predict the Q-value estimated by the explorer for each agent, and thus obtain the prediction error between the predicted value function estimated by the predictor and the true value function estimated by the explorer. 2) Differentiated updates are performed on the dual Q-value networks of the explorer network. The min and max operators are used to constrain the training objectives of the two Q-value networks respectively, so that the two Q-value networks of the explorer form underestimated and overestimated numerical distributions respectively, generating a dual benchmark strategy with differentiated exploration directions. 3) Control the real-time interaction between the user network and the multi-agent environment, collect environmental interaction trajectory data, and build an experience replay pool to store interaction quadruple data; Interactive quadruple data In this context, s represents the agent's state at the current time step, a represents the action performed by the agent at the current time step, s' represents the agent's state at the next time step, and r represents the agent's state at the next time step. ext This indicates the external task reward returned by the environment; The interaction quadruple describes the time interval [time]. Each agent selects the corresponding action 'a' based on its local observations and the policy network. According to the state transition function, the environment transitions to the next time step state 's' and returns the corresponding reward value 'r'. ext ; 4) Sample batch data from the experience replay pool, calculate the local prediction error of a single agent based on the baseline Q value of the explorer network and the prediction Q value of the predictor network, and use it as an intrinsic reward to guide exploration. 5) By using a pre-set collaborative exploration mechanism, the state and action information of all agents are integrated to correct and optimize the local prediction error of a single agent, thereby obtaining the local prediction error of global collaboration. The collaborative exploration mechanism includes at least one of global prediction error constraints, variance regularization constraints, and attention weighting constraints. 6) Integrate external rewards from environmental feedback with optimized internal rewards to perform gradient updates on the user network and predictor network, and update the target network of the explorer network and user network at preset intervals. 7) Iterate through the environment interaction and network parameter update process until the model converges and outputs the optimal collaborative exploration strategy for multiple agents.

[0006] According to the above scheme, in step 2), the dual-Q value network is assumed to be two... function and The following formulas are used to update the data:

[0007]

[0008] in, This is a batch of data sampled from the experience replay pool, where r represents the external reward and γ is the discount factor. State-action pairs for the next moment after the agent interacts with the environment. The corresponding output value of the dual-Q network.

[0009] According to the above scheme, in step 4), the local prediction error of a single agent is calculated based on the baseline Q-value of the explorer network and the predicted Q-value of the predictor network, as follows: ; Where ψ represents the predictor network parameters. For the explorer target network parameters, This is a local observation of the j-th agent at the next moment.

[0010] According to the above scheme, in step 5), the single-intelligent local prediction error is corrected and optimized using global prediction error constraints, as follows: 1) Global prediction error; a global prediction network is used. Used to predict the global value function, where the input to the global prediction network is the joint state space and joint action space of all agents; 2) Calculate the error between the global prediction Q-value output by the global prediction network and the global baseline Q-value output by the explorer network to obtain the global prediction loss; By measuring the global value function estimated by the explorer The global prediction function of the global prediction network The global prediction network is trained using the global prediction loss between the two methods, and the specific formula is as follows:

[0011] in, It is the loss value of the global prediction network, and is also regarded as the global prediction error of the value function; 3) Construct a hybrid network to nonlinearly fuse the local prediction errors of all agents to obtain the global hybrid error; Use a hybrid network (Mixing Network) nonlinearly weights the local prediction errors of all individuals into a global mixed error; Global Mixed Error The learning objective is the global prediction error mentioned above, and the specific formula is as follows:

[0012] 4) Using the global prediction loss as a supervision signal, optimize the parameters of the hybrid network so that the global hybrid error matches the loss value of the global prediction network, thus making up for the limitations of local observation by a single agent.

[0013] According to the above scheme, in step 5), the variance regularization term constraint is used to correct and optimize the local prediction error of the single intelligent system, as follows: 1) Calculate the local intrinsic reward values ​​of all agents in the current batch of samples, and calculate the standard deviation of all intrinsic rewards; 2) The standard deviation is added as a regularization term to the local prediction loss of a single agent to obtain the regularized local loss; 3) The average of the regularized local losses of all agents is taken to obtain the final prediction loss, which is then backpropagated to update the predictor network. The exploration breadth of the multi-agent network is enhanced by exploring noise. The variance of the prediction errors of all agents is used as the regularization term for the local prediction error, and the specific formula is as follows:

[0014]

[0015] in, The variance represents the local prediction error. This represents the local prediction loss of a single agent after variance regularization correction. The variance-regularized total prediction loss is obtained by taking the mean of the corrected local loss for all individual unmanned units; This is the variance regularization hyperparameter.

[0016] According to the above scheme, in step 5), attention-weighted constraints are used to correct and optimize the local prediction error of single intelligence, as follows: 1) Construct an attention network, taking the global state and action information of all agents as input, and calculate the interaction attention weights of each agent through query, key-value mapping and softmax normalization; 2) Use attention weights to weight and fuse the local prediction errors of all agents to generate a weighted prediction error term that includes agent association information; 3) The weighted error term is fused with the original local prediction error to construct the attention collaborative loss function, update the predictor network, and achieve collaborative alignment of the exploration directions of multiple agents.

[0017] The weighted prediction error is used as an auxiliary loss value to update the prediction network, and the specific formula is as follows:

[0018]

[0019] in, Let the local prediction loss of the j-th agent be the result of fusing attentional collaboration information. Let be the original local prediction error of the j-th agent; This indicates that the attention mechanism calculates attention weights based on the state and actions of each agent m. These are hyperparameters used to control the degree of exploration; The averaged total prediction loss for attention collaboration among all agents; This is a weighted prediction error term that includes information about the association between agents.

[0020] According to the above scheme, in step 6), the user network update formula is as follows:

[0021] in, It is a reward scaling factor, which controls the degree of exploration by scaling the intrinsic reward value. The Q-value network parameter represents the user's network parameters. This represents the user's target network parameters. It is used to update the weights of the target network; As an external reward, For optimized intrinsic rewards; This is the discount factor.

[0022] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described scheme.

[0023] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method in the above-described scheme.

[0024] The beneficial effects of this invention are: This invention integrates global multi-agent interaction information into the local error calculation process of a single agent, and performs weighted fusion of the local intrinsic rewards of each agent, thus completely breaking the limitations of single local observation and effectively improving the exploration efficiency of agents.

[0025] This invention generates differentiated interaction weights based on the real-time state and actions of each agent, effectively balancing the relationship between independent exploration by a single agent and collaborative exploration by multiple agents, and avoiding excessive or insufficient collaboration. Attached Figure Description

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a diagram of the attention network structure according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0028] like Figure 1 As shown, a multi-agent reinforcement learning exploration method based on policy diversity includes the following steps: 1) Construct a dual-strategy network architecture suitable for unmanned cluster operations; the dual-strategy network architecture includes a user network for environmental interaction, an explorer network for exploration guidance, and a predictor network; Both the user network and the explorer network are configured with a pair of independent dual-Q value networks; The predictor network is a multi-layer neural network used to predict the Q-value estimated by the explorer for each agent, and thus obtain the prediction error between the predicted value function estimated by the predictor and the true value function estimated by the explorer. Unmanned swarms consist of a collaborative swarm of multiple drones used to perform tasks such as searching unknown areas and conducting reconnaissance across the entire domain. 2) Differentiated updates are performed on the dual Q-value networks of the explorer network. The min and max operators are used to constrain the training objectives of the two Q-value networks respectively, so that the two Q-value networks of the explorer form underestimated and overestimated numerical distributions respectively, generating a dual benchmark strategy with differentiated exploration directions. In step 2), the dual-Q value network is assumed to be two... function and The following formulas are used to update the data:

[0029]

[0030] in, This is a batch of data sampled from the experience replay pool, where r represents the external reward and γ is the discount factor. State-action pairs for the next moment after the agent interacts with the environment. The corresponding output value of the dual-Q network.

[0031] 3) Control the real-time interaction between the user network and the multi-agent environment, collect environmental interaction trajectory data, and build an experience replay pool to store interaction quadruple data; Environments include unknown environments, urban obstructed environments, or dynamic obstacle environments; Interactive quadruple data In this context, s represents the agent's state at the current time step, a represents the action performed by the agent at the current time step, s' represents the agent's state at the next time step, and r represents the agent's state at the next time step. ext This indicates the external task reward returned by the environment; The interaction quadruple describes the time interval [time]. Each agent selects the corresponding action 'a' based on its local observations and the policy network. According to the state transition function, the environment transitions to the next time step state 's' and returns the corresponding reward value 'r'. ext ; 4) Sample batch data from the experience replay pool, calculate the local prediction error of a single agent based on the baseline Q value of the explorer network and the prediction Q value of the predictor network, and use it as an intrinsic reward to guide exploration. In step 4), the local prediction error of a single agent is calculated based on the baseline Q-value of the explorer network and the predicted Q-value of the predictor network, as follows: ; Where ψ represents the predictor network parameters. For the explorer target network parameters, This is a local observation of the j-th agent at the next moment.

[0032] 5) By using a pre-set collaborative exploration mechanism, the state information and action information (policy information) of all agents are integrated to correct and optimize the local prediction error of a single agent, thereby obtaining the local prediction error of global collaboration. The collaborative exploration mechanism includes at least one of global prediction error constraints, variance regularization constraints, and attention weighting constraints. The local prediction error of a single intelligent system is corrected and optimized using global prediction error constraints, as follows: 1) Global prediction error; a global prediction network is used. Used to predict the global value function, where the input to the global prediction network is the joint state space and joint action space of all agents; 2) Calculate the error between the global prediction Q-value output by the global prediction network and the global baseline Q-value output by the explorer network to obtain the global prediction loss; By measuring the global value function estimated by the explorer The global prediction function of the global prediction network The global prediction network is trained using the global prediction loss between the two methods, and the specific formula is as follows:

[0033] in, It is the loss value of the global prediction network, and is also regarded as the global prediction error of the value function; 3) Construct a hybrid network to nonlinearly fuse the local prediction errors of all agents to obtain the global hybrid error; Use a hybrid network (Mixing Network) nonlinearly weights the local prediction errors of all individuals into a global mixed error; Global Mixed Error The learning objective is the global prediction error mentioned above, and the specific formula is as follows:

[0034] 4) Using the global prediction loss as a supervision signal, optimize the parameters of the hybrid network so that the global hybrid error matches the loss value of the global prediction network, thus making up for the limitations of local observation by a single agent.

[0035] The variance regularization constraint is used to correct and optimize the local prediction error of the single intelligent system, as follows: 1) Calculate the local intrinsic reward values ​​of all agents in the current batch of samples, and calculate the standard deviation of all intrinsic rewards; 2) The standard deviation is added as a regularization term to the local prediction loss of a single agent to obtain the regularized local loss; 3) The average of the regularized local losses of all agents is taken to obtain the final prediction loss, which is then backpropagated to update the predictor network. The exploration breadth of the multi-agent network is enhanced by exploring noise. The variance of the prediction errors of all agents is used as the regularization term for the local prediction error, and the specific formula is as follows:

[0036]

[0037] in, The variance represents the local prediction error. This represents the local prediction loss of a single agent after variance regularization correction. The variance-regularized total prediction loss is obtained by taking the mean of the corrected local loss for all individual unmanned units; This is the variance regularization hyperparameter.

[0038] Attention-weighted constraints are used to correct and optimize the local prediction error of single-intelligence systems, as follows: 1) Construct an attention network, taking the global state and action information of all agents as input, and calculate the interaction attention weights of each agent through query, key-value mapping and softmax normalization; 2) Use attention weights to weight and fuse the local prediction errors of all agents to generate a weighted prediction error term that includes agent association information; 3) The weighted error term is fused with the original local prediction error to construct the attention collaborative loss function, update the predictor network, and achieve collaborative alignment of the exploration directions of multiple agents.

[0039] The weighted prediction error is used as an auxiliary loss value to update the prediction network, and the specific formula is as follows:

[0040]

[0041] in, Let the local prediction loss of the j-th agent be the result of fusing attentional collaboration information. Let be the original local prediction error of the j-th agent; This indicates that the attention mechanism calculates attention weights based on the state and actions of each agent m. These are hyperparameters used to control the degree of exploration; The averaged total prediction loss for attention collaboration among all agents; This is a weighted prediction error term that includes information about the association between agents; Attention networks such as Figure 2 As shown.

[0042] It should be noted that in the formula... In the corresponding Attention framework and The softmax output of the network, without considering Network output.

[0043] 6) Integrate external rewards from environmental feedback with optimized internal rewards to perform gradient updates on the user network and predictor network, and update the target network of the explorer network and user network at preset intervals. In step 6), the user network update formula is as follows:

[0044] in, It is a reward scaling factor, which controls the degree of exploration by scaling the intrinsic reward value. The Q-value network parameter represents the user's network parameters. This represents the user's target network parameters. It is used to update the weights of the target network; As an external reward, For optimized intrinsic rewards; This is the discount factor.

[0045] 7) Iterate through the environment interaction and network parameter update process until the model converges, and output the optimal cooperative exploration strategy of multi-agents, including the detection status, position and action information of the optimal cooperative multi-agents in the unmanned swarm.

[0046] Applying the method of this invention to unmanned swarm operations can effectively unify the exploration direction of unmanned swarms, compensate for the local detection blind spots of single sensors, and enhance the collaborative exploration capabilities of multiple unmanned individuals. It can significantly improve the exploration efficiency and environmental coverage accuracy of unmanned swarms in complex tasks such as searching unknown areas and conducting all-domain reconnaissance, and has good environmental adaptability and engineering application value.

[0047] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multi-agent reinforcement learning exploration method based on policy diversity, characterized in that, Includes the following steps: 1) Construct a dual-strategy network architecture; the dual-strategy network architecture includes a user network for environmental interaction, an explorer network for exploration guidance, and a predictor network; Both the user network and the explorer network are configured with a pair of independent dual-Q value networks; The predictor network is a multi-layer neural network used to predict the Q-value estimated by the explorer for each agent, and thus obtain the prediction error between the predicted value function estimated by the predictor and the true value function estimated by the explorer. 2) Differentiate the update of the explorer network’s dual Q-value network so that the two Q-value networks of the explorer form underestimated and overestimated numerical distributions respectively, generating a dual-benchmark strategy with differentiated exploration directions; 3) Control the real-time interaction between the user network and the multi-agent environment, collect environmental interaction trajectory data, and build an experience replay pool to store interaction quadruple data; Interactive quadruple data In this context, s represents the agent's state at the current time step, a represents the action performed by the agent at the current time step, s' represents the agent's state at the next time step, and r represents the agent's state at the next time step. ext This indicates the external task reward returned by the environment; 4) Sample batch data from the experience replay pool, calculate the local prediction error of a single agent based on the baseline Q value of the explorer network and the prediction Q value of the predictor network, and use it as an intrinsic reward to guide exploration. 5) By using a pre-set collaborative exploration mechanism, the state and action information of all agents are integrated to correct and optimize the local prediction error of a single agent, thereby obtaining the local prediction error of global collaboration. The collaborative exploration mechanism includes at least one of global prediction error constraints, variance regularization constraints, and attention weighting constraints. 6) Integrate external rewards from environmental feedback with optimized internal rewards to perform gradient updates on the user network and predictor network, and update the target network of the explorer network and user network at preset intervals. 7) Iterate through the environment interaction and network parameter update process until the model converges and outputs the optimal collaborative exploration strategy for multiple agents.

2. The multi-agent reinforcement learning exploration method based on policy diversity according to claim 1, characterized in that, In step 2), the dual-Q value network is assumed to be two... function and The following formulas are used to update the data: in, This is a batch of data sampled from the experience replay pool, where r represents the external reward and γ is the discount factor. State-action pairs for the next moment after the agent interacts with the environment. The corresponding output value of the dual-Q network.

3. The multi-agent reinforcement learning exploration method based on policy diversity according to claim 1, characterized in that, In step 4), the local prediction error of a single agent is calculated based on the baseline Q-value of the explorer network and the predicted Q-value of the predictor network, as follows: ； Where ψ represents the predictor network parameters. For the explorer target network parameters, This is a local observation of the j-th agent at the next moment.

4. The multi-agent reinforcement learning exploration method based on policy diversity according to claim 1, characterized in that, In step 5), the single-intelligent local prediction error is corrected and optimized using global prediction error constraints, as follows: 1) Global prediction error; using a global prediction network Used to predict the global value function, where the input to the global prediction network is the joint state space and joint action space of all agents; 2) Calculate the error between the global prediction Q-value output by the global prediction network and the global baseline Q-value output by the explorer network to obtain the global prediction loss; By measuring the global value function estimated by the explorer The global prediction function of the global prediction network The global prediction network is trained using the global prediction loss between the two methods, and the specific formula is as follows: in, It is the loss value of the global prediction network, and is also regarded as the global prediction error of the value function; 3) Construct a hybrid network to nonlinearly fuse the local prediction errors of all agents to obtain the global hybrid error; Use a hybrid network The local prediction errors of all individuals are nonlinearly weighted into a global mixed error; Global Mixed Error The learning objective is the global prediction error mentioned above, and the specific formula is as follows: 4) Using the global prediction loss as a supervision signal, optimize the parameters of the hybrid network so that the global hybrid error matches the loss value of the global prediction network, thereby completing the optimization of the local observation error of the single agent.

5. The multi-agent reinforcement learning exploration method based on policy diversity according to claim 1, characterized in that, In step 5), the variance regularization term constraint is used to correct and optimize the local prediction error of the single intelligent system, as follows: 1) Calculate the local intrinsic reward values ​​of all agents in the current batch of samples, and calculate the standard deviation of all intrinsic rewards; 2) The standard deviation is added as a regularization term to the local prediction loss of a single agent to obtain the regularized local loss; 3) The average of the regularized local loss of all agents is taken to obtain the final prediction loss, which is then backpropagated to update the predictor network. The exploration breadth of the multi-agent network is enhanced by exploring noise.

6. The multi-agent reinforcement learning exploration method based on policy diversity according to claim 1, characterized in that, In step 5), attention-weighted constraints are used to correct and optimize the local prediction error of the single intelligence, as follows: 1) Construct an attention network, using the global state and action information of all agents as input, and calculate the interaction attention weights of each agent; 2) Use attention weights to weight and fuse the local prediction errors of all agents to generate a weighted prediction error term that includes agent association information; 3) The weighted error term is fused with the original local prediction error to construct the attention collaboration loss function, update the predictor network, and achieve collaborative alignment of the exploration directions of multiple agents; The weighted prediction error is used as an auxiliary loss value to update the prediction network, and the specific formula is as follows: in, Let the local prediction loss of the j-th agent be the result of fusing attentional collaboration information. Let be the original local prediction error of the j-th agent; This indicates that the attention mechanism calculates attention weights based on the state and actions of each agent m. These are hyperparameters used to control the degree of exploration; The attention-coordinated total prediction loss is averaged across all agents.

7. The multi-agent reinforcement learning exploration method based on policy diversity according to claim 1, characterized in that, In step 6), the user network update formula is as follows: in, It is a reward scaling factor, which controls the degree of exploration by scaling the intrinsic reward value. The Q-value network parameter represents the user's Q-value. This represents the user's target network parameters. It is used to update the weights of the target network; As an external reward, For optimized intrinsic rewards; This is the discount factor.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images