Multi-agent experience exploration cooperation method based on curiosity mechanism

Abstract

The invention discloses a multi-agent experience exploration cooperation method based on a curiosity mechanism. The method comprises the following steps: constructing a multi-agent decision framework; a curiosity mechanism suitable for a multi-agent decision-making framework is constructed; constructing an experience exploration component suitable for the multi-agent decision-making architecture; constructing an experience playback pool and a state playback pool; a multi-agent cooperative confrontation environment is adapted; training the obtained deep neural network formed by the multi-agent experience exploration decision architecture based on the curiosity mechanism to obtain a multi-agent experience exploration collaborative strategy model; and in the multi-agent experience exploration collaborative strategy model, solving the multi-agent collaborative confrontation strategy in the current confrontation scene. Through the above mode, the method can balance the exploration and utilization process of a multi-agent algorithm during training, alleviates the influence caused by reward sparsity in a multi-agent confrontation environment, helps the multi-agent to carry out experiential exploration, searches an optimal strategy for confrontation, and improves the learning efficiency at the same time.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent deep reinforcement learning, and specifically to a multi-agent experience exploration collaborative method based on a curiosity mechanism. Background Art

[0002] Cooperative multi-agent systems have attracted widespread attention because many real-world problems can be naturally modeled, such as traffic signal control, sensor networks, and multi-robots. Multi-agent deep reinforcement learning is a key component in solving the problem of learning optimal policies for these systems. It learns the optimal policy through trial and error in the environment and under the guidance of environmental reward signals to maximize the expected cumulative reward. Despite significant progress, multi-agent deep reinforcement learning still faces a fundamental dilemma, the trade-off between exploration and utilization, that is, how to enable agents to make good use of existing strategies and explore new behaviors in order to seek potential better strategies in complex coordination tasks that require coordination and efficient exploration.

[0003] The more effective the strategy exploration method is, the easier it is for multi-agents to find the optimal strategy in a complex environment. Currently, traditional methods include exploration methods such as ε-greedy and Boltzmann. However, these methods no longer meet the requirements of complex, high-dimensional or sparse reward environments and may lead to suboptimal learning results. Strategy exploration has made some new progress in the field of single-agent reinforcement learning, including pseudo-counting, curiosity mechanism and information gain. However, these methods often perform poorly in partially observable and non-stationary multi-agent environments and cannot be directly applied to multi-agent deep reinforcement learning algorithms. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-agent empirical exploration collaborative method based on a curiosity mechanism, aiming to solve the problem that existing multi-agents cannot meet the requirements of complex, high-dimensional or sparsely rewarded environments when searching for optimal strategies.

[0005] In order to solve the above technical problems, the present invention provides a multi-agent experience exploration collaboration method based on the curiosity mechanism. By combining the intrinsic exploration mechanism driven by curiosity, the accumulated behavioral experience of multiple agents is used to guide the learning process, so as to promote multiple agents to explore more intelligent collaborative strategies. The specific operation steps are as follows:

[0006] Step 1: Construct a multi-agent decision-making architecture; the multi-agent system includes but is not limited to SMAC platform combat units, multi-vehicle autonomous driving control systems, war game simulation confrontation systems, and other systems that have multiple independent individuals collaborating to achieve common goals;

[0007] Step 2: Build an experience exploration component suitable for multi-agent decision-making architecture;

[0008] Step 3: Build a curiosity mechanism suitable for multi-agent decision-making architecture;

[0009] Step 4: Build the experience replay pool and state replay pool;

[0010] Step 5: Adapt to the multi-agent collaborative confrontation environment;

[0011] Step 6: training the deep neural network formed by the multi-agent experience exploration decision architecture based on the curiosity mechanism obtained in steps 1 to 5 to obtain a multi-agent experience exploration collaborative strategy model based on the curiosity mechanism;

[0012] Step 7: In the multi-agent experience exploration collaborative strategy model based on the curiosity mechanism, solve the multi-agent collaborative confrontation strategy in the current confrontation scenario.

[0013] The present invention is also characterized in that:

[0014] An intelligent agent is a system that can autonomously perceive the environment, make decisions and perform actions. It has basic characteristics such as autonomy, interactivity, responsiveness and adaptability, and can independently complete tasks in a complex and changing environment. The core of an intelligent agent lies in its learning and decision-making capabilities. An intelligent agent can be a tank, an infantryman, a mobile phone, etc.

[0015] Step 1 specifically includes:

[0016] Step 1.1: Value network model based on multi-agent reinforcement learning QMIX, in which each agent has a local value network Used to estimate the benefit when agent i takes a certain action;

[0017] Among them, θ i is the neural network parameter of the agent i value network; τ i is the action a taken by the ith agent i A new vector formed by combining with local observation information; Explanation: The value network is used to approximate the value function;

[0018] Step 1.2: Construct the basic QMIX hybrid network model, Q tot (τ, a, s; θ) is used to estimate the benefits of the cooperative actions of all agents, and s is the joint observation value; at the same time, the hybrid network introduces a monotonicity constraint, namely:

[0019]

[0020] For any agent i among the n agents, the cooperative action benefit Q of the agent is transformed into tot Estimated for each agent Complex nonlinear combinations of values;

[0021] The hybrid network is a feedforward neural network that takes the value network output of each agent as input and performs monotonous mixing to obtain Q tot To strengthen the monotonicity constraint, the weights of the hybrid network are restricted to be non-negative, so that the hybrid network can arbitrarily approximate any monotonic function;

[0022] Four super networks are set in the hybrid network to generate weights of each layer of the hybrid network, each super network is composed of a single linear layer, and an absolute value activation function is set to ensure that the weights in the hybrid network are non-negative;

[0023] During training, minimize the loss value L(θ), that is:

[0024]

[0025] Among them, B is the batch size, n is the number of agents, r total is the discounted reward after reshaping, γ is the discount rate, and θ - are the parameters of the value network, which are periodically copied from the history θ records and remain unchanged over multiple iterations.

[0026] Step 2 is as follows:

[0027] Step 2.1: Build the target network, the target network As the target value t(s) is generated, the neural network parameters that are randomly initialized and fixed are used to evenly map the observed state to a feature space. This feature space can ensure that the decision information in the environment is fully mapped and used. At the same time, the generated target value is fixed according to the input state, and there is no need to train the target network.

[0028] in, Indicates that the target network receives the joint observations s of all agents and maps them to the k-dimensional real number domain This real number domain is again represented as the target value f(s);

[0029] The target network consists of two convolutional layers, the convolution kernel size of the first convolutional layer is 2×2, and the convolution kernel size of the second convolutional layer is 2×1. It receives the joint observation values ​​of all agents, followed by two fully connected layers, and outputs the target value f(s);

[0030] Step 2.2: Build a prediction network, which will collect All agents in the same group observe information together and output prediction results Use the same network structure as the target network;

[0031] in, It represents the predicted value output by the prediction network after receiving the joint observation value s of all agents, and ψ is the neural network parameter of the prediction network;

[0032] The prediction network is trained. For the same input, the expected mean square error is minimized by gradient descent to update the parameter ψ of the prediction network, that is:

[0033]

[0034] Step 2.3: Construct an internal reward by calculating the mean square error between the target network and the prediction network for the same input to obtain the internal reward r int ,Right now:

[0035]

[0036] Step 2.4: Reshape the discounted reward r total , in the internal reward r int A constraint variable is added before, so that the internal reward is reshaped into the discounted reward r total The proportion of the external reward r decreases as the number of training times increases. ext Gradually play a leading role, enabling the agent to avoid short-sighted behavior and achieve reasonable exploration of the entire task space; namely:

[0037] r total =r ext +w·r int (5)

[0038] Among them, w is the specific gravity factor, that is:

[0039] w=β·(1-0.25×10 -5 ) step (6)

[0040] β is the initial weight, which is set to 10% of the maximum discount reward that can be obtained in the current scene. The default value is 2. As the time step increases, w will continue to decrease;

[0041] Step 2.5: Reshape the discounted reward clipping. To ensure the applicability of all set hyperparameters and the consistency of rewards, clip the internal reward to 1 based on normalization by dividing it by a running estimate of the standard deviation of the intrinsic return. In the interval, η∈(0,1] is the cropping ratio, Represents the minimum reward value that can be obtained for the current task. represents the maximum reward value that can be obtained for the current task; then the internal reward is:

[0042]

[0043] The final reshaped discounted reward is:

[0044]

[0045] Step 3 is as follows:

[0046] Step 3.1: Build a Behavioral Success Matrix and Behavioral Failure Matrix Where n represents the number of agents, d represents the number of discrete actions of the agent, and the initial matrix values ​​are all 1; at a certain time t, agent i issues a command j among its available actions and receives an immediate reward r ext If r ext >0, then The value of the action j issued by the corresponding agent i is increased by 1, otherwise The action j issued by the corresponding agent i is increased by 1, that is:

[0047]

[0048] Step 3.2: Construct the probability distribution of the joint action behavior of multiple agents to satisfy the Beta distribution Beta(α,β), where α represents the number of successes of the current action and β represents the number of failures. Combine the evaluation value q of each agent value network with the Bayesian method to estimate the posterior distribution of the strategy;

[0049] The q value describes the action value that the agent considers in the current state. At time t, agent i receives local observation information and calculates the value of each action. Then the joint value of all agents is Will Normalize and convert it indirectly into the probability value of each action Right now

[0050]

[0051] in, represents the neural network parameters of the network at time t, O t Represents the local state information observed by the agent at time t;

[0052] During training, it is assumed that the probability distribution of each action has a prior distribution Beta(1,1) and a posterior distribution Beta(α+1,β+1); as the training progresses, the corresponding values ​​of the agent's behavior in the success matrix and the failure matrix continue to accumulate and add 1, α+β will gradually increase, and the Beta distribution will become narrower. If α>β, it means that the current action can bring an advantage in the confrontation, otherwise it means that the current action will produce negative feedback;

[0053] Step 3.2: Exploration of multi-agent joint action behavior. At time t during training, sample from the posterior distribution and obtain the sampling result μ t ,Right now:

[0054]

[0055] By choosing μ t The action with the largest value in each is the joint action of the agent. Right now:

[0056]

[0057] Step 4 is as follows:

[0058] Step 4.1: After the t-th round of multi-agent collaborative confrontation ends, all agents in the t-th round before the start of the t+1 round jointly observe s t , the local observation o of round t before the start of round t+1 t , the value of the network GRU state parameter h before the start of round t+1 t,q , the joint action of each agent Remodeling rewards received All agents in round t+1 jointly observe s t+1 With each agent's local observation o t+1 As a tensor Store it in the experience replay pool; combine all agents in round t before the start of round t+1 to observe s t As a tensor [s t ] is stored in the state replay pool;

[0059] Among them, the experience replay pool and the state replay pool can store m tensors, and m is set between 1e6 and 1e7;

[0060] Step 4.2: During each training of the value network and the hybrid network, a minimum batch of tensors is randomly drawn from the experience replay pool and trained according to the loss value of the hybrid network; similarly, during each training of the prediction network, a minimum batch of tensors is randomly drawn from the state replay pool and trained according to the expected mean square error. The size of the minimum batch is 64.

[0061] Step 5 is as follows:

[0062] Step 5.1: Receive local observations and global observations given by the multi-agent collaborative adversarial environment at the beginning of each round. The local observations are used as the input of the value network, and the global observations are used as the input of the hybrid network, target network, and prediction network.

[0063] Step 5.2: Mapping of the action q-value output by the value network to the instructions in the multi-agent collaborative adversarial environment.

[0064] Among them, when the value network calculates the action probability set, the numbers of the actions that are not feasible for the multi-agent in the current state and the corresponding values ​​are set to negative infinity.

[0065] Step 6 is as follows:

[0066] Step 6.1: Initialize hyperparameters

[0067] Set the experience replay pool size to 1e6, the minimum batch size to 64, the neural network learning rate to 5e-4, the GRU hidden layer size to 64, the discounted return rate for internal and external rewards to 0.99, and the output dimension of the prediction network and target network to 128;

[0068] Step 6.2: According to the number of agents in the current multi-agent collaborative test platform SMAC confrontation scenario, the local observation space dimension, and the action space dimension, through step 1, set the input size of the multi-layer perceptron at the front end of the value network and the output size of the multi-layer perceptron at the end of the value network; through step 2, set the input size of the outer convolution layer of the prediction network and the target network, initialize the neural network parameters of the target network and fix them. Through step 3, initialize the success matrix and failure matrix to all-1 matrices;

[0069] Step 6.3: At the beginning of each round of the adversarial scenario, receive local observation information and input it into the value network of each agent to calculate the q-value of all actions of the agent and will The infeasible actions are set to negative infinity;. Then, each agent The global information is input into the hybrid network to calculate the joint q value Q of the agent tot , according to step 3.2, the success matrix, failure matrix and Q tot Get the posterior distribution Beta(α,β) of the joint action of multiple agents for sampling, select the maximum value of each agent's action to form the joint action of multiple agents, and command the red team to fight against the blue team;

[0070] Step 6.4: Before the end of each round of the confrontation scenario, receive global observation information and external rewards, input the global observation into the target network and the prediction network, calculate the internal reward according to step 2.3, and use the external reward to obtain the reshaped reward according to step 2.5. At the same time, update the success matrix and failure matrix according to step 3.1;

[0071] Step 6.5: The experience replay pool and the state replay pool collect and store training data respectively.

[0072] Step 6.6: For each epoch, extract the minimum batch training policy network and global value network based on the data from the experience replay pool; extract the minimum batch training prediction network based on the data from the state replay pool;

[0073] Step 6.7: After every 5000 epochs, conduct 32 independent confrontations, calculate the average winning rate, and evaluate the intelligence of the current multi-agent;

[0074] Step 6.8: When the average winning rate of the multi-agents reaches 1 or the total number of epochs reaches 1e6, the training is completed and saved as a multi-agent experience exploration collaborative model based on the curiosity mechanism;

[0075] Step 7 is as follows:

[0076] According to the local observation information given by the confrontation scenario, it is input into the value network of each agent in the multi-agent experience exploration collaborative strategy model based on the curiosity mechanism to obtain the action q value of each agent. And input it together with the global information given by the adversarial scenario into the hybrid network to obtain the joint action q value Q tot , each agent chooses Q tot The corresponding actions with the largest q value constitute the joint actions, and the current multi-agent collaborative confrontation strategy is solved.

[0077] The beneficial effects of the present invention are:

[0078] (1) The present invention is based on a multi-agent experience exploration collaborative method under the curiosity mechanism to improve the ability of agents to balance exploration and utilization. By reshaping the external environment rewards and the internal exploration mechanism, multi-agents are encouraged to discover more unfamiliar environmental states and explore potential better strategies. At the same time, when selecting actions, multi-agents can use the feedback experience of action execution results in past trajectories to help multi-agents conduct experienced exploration based on environmental dynamics. By combining the curiosity-driven internal exploration mechanism with the use of the accumulated behavioral experience of multi-agents to guide the learning process, multi-agents can escape more effectively when facing local optimal solutions and converge to the global optimal solution, promoting multi-agents to explore more wise strategies.

[0079] (2) Based on the basic QMIX algorithm, this invention introduces a curiosity and experience exploration mechanism to help multi-agents reshape the reward function during training, dynamically quantify the novelty of the environment state, alleviate the sparsity of the rewards in the adversarial environment, and encourage multi-agents to actively explore "new" states in order to explore unknown action strategies. At the same time, it uses the existing success and failure experience to further explore and explore reasonable or even better adversarial strategies. The multi-agent experience exploration collaborative method based on the curiosity mechanism improves the exploration ability of multi-agents, helps multi-agents to conduct strategy exploration with experience, avoids premature convergence of the algorithm, and improves the learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flow chart of an example of a multi-agent experience exploration collaborative method based on a curiosity mechanism of the present invention;

[0081] Figure 2 It is a schematic diagram of the design idea of ​​the multi-agent experience exploration collaborative method based on the curiosity mechanism of the present invention. DETAILED DESCRIPTION

[0082] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0083] Example 1

[0084] The present invention is based on a multi-agent experience exploration collaborative method under the curiosity mechanism, and the process is as follows Figure 1 , the specific steps are as follows:

[0085] Step 1: Construct a multi-agent decision-making architecture, where the multi-agent includes but is not limited to SMAC platform combat units, multi-vehicle autonomous driving control systems, war game simulation confrontation systems, and other systems that have multiple independent individuals collaborating to achieve a common goal;

[0086] Step 2: Build an experience exploration component suitable for multi-agent decision-making architecture;

[0087] Step 3: Build a curiosity mechanism suitable for multi-agent decision-making architecture;

[0088] Step 4: Build the experience replay pool and state replay pool;

[0089] Step 5: Adapt to the multi-agent collaborative confrontation environment;

[0090] Step 6: training the deep neural network formed by the multi-agent experience exploration decision architecture based on the curiosity mechanism obtained in steps 1 to 5 to obtain a multi-agent experience exploration collaborative strategy model based on the curiosity mechanism;

[0091] Step 7: In the multi-agent experience exploration collaborative strategy model based on the curiosity mechanism, solve the multi-agent collaborative confrontation strategy in the current confrontation scenario.

[0092] Example 2

[0093] 1. Multi-agent collaboration against the environment

[0094] The experimental environment is based on the StarCraft multi-agent challenge (SMAC) environment, a multi-agent collaborative confrontation test platform for Blizzard's real-time strategy game StarCraft II. SMAC contains multiple predefined scenarios of different difficulties for learning, training and testing of multi-agent deep reinforcement learning algorithms. For example, 3Marines (3m), 3Stalkers Vs 3Zealots (3s_vs_3z), etc. In the name of each scenario, the number represents the number of units of the current agent, such as 3s_vs_3z means 3 stalkers and 3 berserkers in asymmetric confrontation. 3m means commanding 3 marines to conduct symmetrical confrontation. This symmetry means that the composition and number of multi-agents commanded by the opposing sides are the same. In each episode, the algorithm to be learned controls the red side in a decentralized manner and fights against all enemy units of the blue side. In the initial state of each episode, the red side commanded by the algorithm to be learned is symmetrical with the initial position of the opposing blue side. The opposing blue side is ordered to approach the position of the red side and attack the red side units.

[0095] In SMAC, the local observation information of a single agent is within its field of view, which includes a circular area around the map with a radius equal to the line of sight. Each agent receives as input a vector consisting of the following features of all units (both allies and enemies) within its field of view: distance, relative x, relative y, and unit type. The global state is hidden from the agent and is a vector consisting of unit features on the entire map. It does not contain the absolute distance between agents and only stores the unit coordinates relative to the center of the map. In addition, the global state includes the health, shield, and attack cooldown of all units. In addition, the global state contains the last actions of all allied agents. The health of Marines, Stalkers, Zealots, and Colossi are 45, 80, 100, and 200, respectively. In addition, Stalkers, Zealots, and Colossi have 80, 50, and 150 shields, respectively. All features, whether local observations or global states, are normalized by their maximum values. For all unit types, the aiming distance and shooting distance of the agent are set to 9 and 6, respectively.

[0096] The action space is represented in discrete form, which may be different depending on the specific scenario, but is usually represented as: move[direction], attack[enemy_id], stop, and noop. Direction has only four directions: east, south, west, and north. Attack[enemy_id] is only allowed when the target unit is within shooting range. Only dead agents can execute noop instructions.

[0097] 2. Reward and Punishment Rules

[0098] The reward comes from the total damage caused by the agent to all enemy units in each time step. In addition, the agent will receive 10 points of reward for each opponent defeated, and 200 points of reward after completely defeating all opponents, and standardized so that the maximum reward that the multi-agent can receive in each episode is 20. Finally, the agent is prohibited from giving feedback on the enemy's attack behavior when idle, so that the agent can only rely on its own exploration decision-making behavior to win.

[0099] Step 1: Build a multi-agent decision-making architecture

[0100] Step 1 specifically includes:

[0101] Step 1.1: Construct a value network model based on multi-agent reinforcement learning QMIX, such as Figure 2 As shown, in QMIX each agent has a local value network Used to estimate the benefit when agent i takes a certain action.

[0102] Among them, θ i is the neural network parameter of the agent i value network; τ i is the action a taken by the ith agent i A new vector formed by combining with local observation information; The description value network is used to approximate the value function.

[0103] In particular, the value network of each agent uses a multi-layer perceptron MLP to receive the local observation of the agent, followed by a recurrent neural network GRU hidden layer with 64-dimensional hidden states, and finally adds a multi-layer perceptron, sets the output result dimension to the action space size, and sets the ReLu activation function to obtain the Q value of each action of the agent.

[0104] Step 1.2: Construct the basic QMIX hybrid network model, Q tot (τ,a,s;θ) is used to estimate the benefits of the cooperative actions of all agents, and s is the joint observation value. At the same time, the hybrid network introduces a monotonicity constraint, namely:

[0105]

[0106] For any agent i among the n agents, the cooperative action benefit Q of the agent is transformed into tot Estimated for each agent Complex nonlinear combinations of values.

[0107] The hybrid network is a feedforward neural network that takes the value network output of each agent as input and performs monotonous mixing to obtain Q tot To enforce the monotonicity constraint, the weights (but not the biases) of the hybrid network are constrained to be non-negative, so that the hybrid network can arbitrarily approximate any monotonic function.

[0108] In particular, four hypernetworks are set in the hybrid network to generate the weights of each layer of the hybrid network. Each hypernetwork consists of a single linear layer, and an absolute value activation function is set to ensure that the weights in the hybrid network are non-negative.

[0109] During training, minimize the loss value L(θ), that is:

[0110]

[0111] In the example, B is the batch size, n is the number of agents, and r is total is the discounted reward after reshaping, γ is the discount rate; θ - are the parameters of the value network, which are periodically copied from the history θ records and remain unchanged over multiple iterations.

[0112] Step 2: Build a curiosity mechanism for multi-agent decision-making architecture

[0113] Step 2 specifically includes

[0114] Step 2.1: Build the target network, the target network As the target value t(s) is generated, the randomly initialized and fixed neural network parameters are used to evenly map the observed state to a feature space, which can ensure that the decision information in the environment is fully mapped and used. At the same time, the generated target value is fixed according to the input state, and there is no need to train the target network.

[0115] in, Indicates that the target network receives the joint observations s of all agents and maps them to the k-dimensional real number domain This real number field is also represented as the target value f(s).

[0116] In particular, the target network consists of two convolutional layers, the convolution kernel size of the first convolutional layer is 2×2, and the convolution kernel size of the second convolutional layer is 2×1, receiving the joint observations of all agents. It is followed by two fully connected layers to output the target value f(s).

[0117] Step 2.2: Build a prediction network, which will collect All agents in the same group observe information together and output prediction results Use the same network structure as the target network.

[0118] in, It represents the predicted value output by the prediction network after receiving the joint observation value s of all agents, and ψ is the neural network parameter of the prediction network.

[0119] In particular, the prediction network will be trained to minimize the expected mean square error for the same input by gradient descent to update the parameters of the prediction network ψ. That is:

[0120]

[0121] In this process, the prediction network will be gradually refined into a pre-set target network;

[0122] Step 2.3: Construct an internal reward by calculating the mean square error between the target network and the prediction network for the same input to obtain the internal reward r int ,Right now:

[0123]

[0124] Among them, S is the joint observation information of all agents in the current state, describing the global information. Then this mean square error will cause multiple agents to obtain higher values ​​in unfamiliar states, but the internal reward will decrease as the access frequency and training process continue to deepen.

[0125] Step 2.4: Reshape the discounted reward r total , in the internal reward r int A constraint variable is added before the internal reward to reshape the reward r total The proportion of the external reward r decreases as the number of training times increases. ext Gradually play a leading role, enabling the agent to avoid short-sighted behavior and achieve reasonable exploration of the entire task space. That is:

[0126] r total =r ext +w·r int

[0127] Among them, w is the specific gravity factor, that is:

[0128] w=β·(1-0.25×10 -5 ) step

[0129] β is the initial weight, which is generally set to 10% of the maximum discounted reward that can be obtained in the current scene. The default value is 2. As the time step increases, w will continue to decrease. Since the internal reward is a description of the agent's behavior exploration, it stimulates the curiosity of multiple agents about unknown states, rather than judging whether the environment has been seen before. Therefore, such a setting can make r int In the early stage of training, it can provide a good exploration motivation for multi-agents and avoid excessive exploration in the later stage. int The calculation of should be done after the multi-agent decision is made and then compared with the actual external reward r received. ext Combined to get the reshaped reward r total .

[0130] Step 2.5: Reshape the discounted reward clipping. Due to different test scenarios and different time points, the size of the reward may vary. Therefore, to ensure the applicability of all set hyperparameters and the consistency of the reward, the intrinsic reward is clipped to 1 based on the normalization of the intrinsic reward divided by the running estimate of the standard deviation of the intrinsic return. In the interval, η∈(0,1] is the cropping ratio, Represents the minimum reward value that can be obtained for the current task. represents the maximum reward value that can be obtained for the current task. Then the internal reward is:

[0131]

[0132] The final remodel discount reward is:

[0133]

[0134] Step 3: Build an experience exploration component suitable for multi-agent decision-making architecture

[0135] Step 3.1: Build a Behavioral Success Matrix and Behavioral Failure Matrix Where n represents the number of agents, d represents the number of discrete actions of the agent, and the initial matrix values ​​are all 1. At a certain time t, agent i issues a command j among its available actions and receives an immediate reward r ext If r ext >0, then The value of the action j issued by the corresponding agent i is increased by 1, otherwise The action j issued by the corresponding agent i is increased by 1, that is:

[0136]

[0137] Step 3.2: Construct the probability distribution of multi-agent joint action behavior to satisfy the Beta distribution Beta(α,β), where α represents the number of successes of the current action and β represents the number of failures. Combine the evaluation value q of each agent value network with the Bayesian method to estimate the posterior distribution of the strategy.

[0138] The q value describes the action value that the agent considers in the current state. At time t, agent i receives local observation information And calculate the value of each action Then the joint value of all agents is Similarly, for It also implies the current joint local observation information. The result greater than or equal to 0 is the same as Combined to get α, The result less than 0 is Combined to get β. Considering different scenarios and different time points, The difference in the results and the influence of its value range (in some cases, the q value is greater than 0 but is very small in the long run) will Normalize and convert it indirectly into the probability value of each action Right now

[0139]

[0140] During training, it is assumed that the probability distribution of each action has a prior distribution Beta(1,1) and a posterior distribution Beta(α+1,β+1). Since the q value describes the agent's cognition of environmental changes, it can help the algorithm make reasonable choices when there is a lack of previous experience, greatly reducing the algorithm's computational cost. At the same time, as training continues, the corresponding values ​​of the agent's behavior in the success matrix and the failure matrix continue to accumulate and add 1, α+β will gradually increase, and the beta distribution will become narrower. If α>β, it means that the current action can bring a confrontational advantage, otherwise it means that the current action will produce negative feedback. Multi-agents will explore based on past experience according to the current uncertainty, which will help discover potential high-quality strategies.

[0141] Step 3.2: Exploration of multi-agent joint action behavior. At time t during training, sample from the posterior distribution and obtain the sampling result μ t ,Right now:

[0142]

[0143] By choosing μ t The action with the largest value in each is the joint action of the agent. Right now:

[0144]

[0145] It is worth noting that the choice of μ t The action with the largest value in is different from the deterministic strategy. This is because during the training process, due to the sampling of uncertainty and posterior distribution, the action with a smaller probability value or q value is still selected with a certain probability. This method helps to explore the action space more comprehensively and avoid over-reliance on the currently estimated optimal action, thereby alleviating the problem of local optimal solution.

[0146] Step 4: Build the experience replay pool and state replay pool.

[0147] Step 4.1: After the t-th round of multi-agent collaborative confrontation ends, all agents in the t-th round before the start of the t+1 round jointly observe s t , the local observation o of round t before the start of round t+1 t , the value of the network GRU state parameter h before the start of round t+1 t,q , the joint action of each agent Remodeling rewards received All agents in round t+1 jointly observe s t+1 With each agent's local observation o t+1 As a tensor Store it in the experience replay pool. Combine all agents in round t before round t+1 to observe s t As a tensor [s t ] is stored in the state replay pool.

[0148] Among them, the experience replay pool and the state replay pool can store m tensors, and m is generally set between 1e6 and 1e7. In this step 3.1, the data content of each tensor stored in the experience replay pool (current global state, current local observation, current value network GRU state parameters, current joint action taken, current reshape reward obtained, new global state after executing the action, new local observation after executing the action) and the data source are described, as well as the data content of each tensor in the state replay pool (current global state) and the data source.

[0149] Step 4.2: During each training of the value network and the hybrid network, a minimum batch of tensors is randomly drawn from the experience replay pool and trained according to the loss value of the hybrid network. Similarly, during each training of the prediction network, a minimum batch of tensors is randomly drawn from the state replay pool and trained according to the expected mean square error. The size of the minimum batch is generally 64.

[0150] In step 4.2, it is described how the experience replay pool is used during hybrid network training, and it is also described how the state revisit pool is used during prediction network training, and a minimum batch is randomly drawn from the experience replay pool and the state revisit pool.

[0151] Step 5: Adapt to the multi-agent collaborative confrontation environment

[0152] Step 5 may specifically include:

[0153] Step 5.1: Receive local observations and global observations given by the multi-agent collaborative adversarial environment at the beginning of each round. The local observations serve as the input of the value network, and the global observations serve as the input of the hybrid network, target network, and prediction network.

[0154] Step 5.2: Mapping of the action q-value output by the value network to the instructions in the multi-agent collaborative adversarial environment.

[0155] Among them, when the value network calculates the action probability set, the numbers of the actions that are not feasible for the multi-agent in the current state and the corresponding values ​​are set to negative infinity (-INF).

[0156] Step 6: Train the deep neural network composed of the multi-agent experience exploration decision-making architecture based on the curiosity mechanism obtained in steps 1 to 5 to obtain a multi-agent experience exploration collaborative strategy model based on the curiosity mechanism.

[0157] Step 6 specifically includes:

[0158] Step 6.1: Initialize hyperparameters.

[0159] Set the experience replay pool size to 1e6, the mini-batch to 64, the neural network learning rate to 5e-4, the GRU hidden layer to 64, the discounted return rate of internal and external rewards to 0.99, and the output dimension of the prediction network and the target network to 128.

[0160] Step 6.2: According to the number of agents in the current SMAC confrontation scenario, the dimension of the local observation space, and the dimension of the action space, through step 1, set the input size of the multilayer perceptron at the front end of the value network and the output size of the multilayer perceptron at the end of the value network. Through step 2, set the input size of the outer convolutional layer of the prediction network and the target network, initialize the neural network parameters of the target network and fix them. Through step 3, initialize the success matrix and failure matrix to all-1 matrices.

[0161] Step 6.3: At the beginning of each round of the adversarial scenario, receive local observation information and input it into the value network of each agent to calculate the q-value of all actions of the agent and will Infeasible actions are set to negative infinity (-INF). The global information is input into the hybrid network to calculate the joint q value Q of the agent tot , according to step 3.2, the success matrix, failure matrix and Q tot The posterior distribution Beta(α, β) of the joint action of multiple agents is obtained for sampling, and the maximum value of each agent's action is selected to form the joint action of multiple agents, and the red team is commanded to fight against the blue team.

[0162] Step 6.4: Before the end of each round of the confrontation scenario, receive global observation information and external rewards, input the global observation into the target network and the prediction network, calculate the internal reward according to step 2.3, and use the external reward to obtain the reshaped reward according to step 2.5. At the same time, update the success matrix and failure matrix according to step 3.1.

[0163] Step 6.5: The experience replay pool and the state replay pool collect and store training data respectively.

[0164] Step 6.6: For each epoch, extract the minimum batch training policy network and global value network based on the data from the experience replay pool; and extract the minimum batch training prediction network based on the data from the state replay pool.

[0165] Step 6.7: After every 5000 epochs, conduct 32 independent confrontations, calculate the average winning rate, and evaluate the intelligence of the current multi-agent.

[0166] Step 6.8: When the average winning rate of the multi-agent reaches 1 or the total number of epochs reaches 1e6, the training is completed and saved as a multi-agent experience exploration collaborative model based on the curiosity mechanism.

[0167] Step 7: In the multi-agent experience exploration collaborative strategy model based on the curiosity mechanism, solve the multi-agent collaborative confrontation strategy in the current confrontation scenario.

[0168] Example 3

[0169] Based on Example 2,

[0170] According to the local observation information given by the confrontation scenario, it is input into the value network of each agent in the multi-agent experience exploration collaborative strategy model based on the curiosity mechanism to obtain the action q value of each agent. And input it together with the global information given by the adversarial scenario into the hybrid network to obtain the joint action q value Q tot , each agent chooses Q tot The corresponding actions with the largest q value constitute the joint actions, and the current multi-agent collaborative confrontation strategy is solved.

[0171] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-agent experience exploration collaborative method based on curiosity mechanism, characterized by: The specific steps are as follows: Step 1: Construct a multi-agent decision-making architecture, where the multi-agent is a system with multiple independent individuals collaborating to achieve a common goal; Step 2: Build an experience exploration component suitable for multi-agent decision-making architecture; Step 3: Build a curiosity mechanism suitable for multi-agent decision-making architecture; Step 4: Build the experience replay pool and state replay pool; Step 5: Adapt to the multi-agent collaborative confrontation environment; Step 6: Train the deep neural network composed of the multi-agent experience exploration decision architecture based on the curiosity mechanism obtained in steps 1 to 5 to obtain a multi-agent experience exploration collaborative strategy model based on the curiosity mechanism; Step 7: In the multi-agent experience exploration collaborative strategy model, solve the multi-agent collaborative confrontation strategy in the current confrontation scenario.

2. The multi-agent experience exploration collaborative method based on curiosity mechanism according to claim 1 is characterized by: Step 1 specifically includes: Step 1.1: Build a value network model based on multi-agent reinforcement learning QMIX, where each agent has a local value network Used to estimate the benefit when agent i takes a certain action; Among them, θ i is the neural network parameter of the agent i value network; τ i is the action a taken by the ith agent i A new vector formed by combining with local observation information; Explanation: The value network is used to approximate the value function; Step 1.2: Construct the basic QMIX hybrid network model, Q tot (τ, a, s; θ) is used to estimate the benefits of the cooperative actions of all agents, and s is the joint observation value; at the same time, the hybrid network introduces a monotonicity constraint, namely: For any agent i among the n agents, the cooperative action benefit Q of the agent is transformed into tot Estimated for each agent Complex nonlinear combinations of values; The hybrid network is a feedforward neural network that takes the value network output of each agent as input and performs monotonous mixing to obtain Q tot To strengthen the monotonicity constraint, the weights of the hybrid network are restricted to be non-negative, so that the hybrid network can arbitrarily approximate any monotonic function; Four super networks are set in the hybrid network to generate weights of each layer of the hybrid network, each super network is composed of a single linear layer, and an absolute value activation function is set to ensure that the weights in the hybrid network are non-negative; During training, minimize the loss value L(θ), that is: Among them, B is the batch size, n is the number of agents, r total is the discounted reward after reshaping, γ is the discount rate, and θ - are the parameters of the value network.

3. The multi-agent experience exploration collaborative method based on curiosity mechanism according to claim 1, characterized in that: Step 2 is as follows: Step 2.1: Build the target network, the target network As the target value t(s) is generated, the neural network parameters that are randomly initialized and fixed are used to evenly map the observed state to a feature space. This feature space can ensure that the decision information in the environment is fully mapped and used. At the same time, the generated target value is fixed according to the input state, and there is no need to train the target network. in, Indicates that the target network receives the joint observations s of all agents and maps them to the k-dimensional real number domain This real number domain is again represented as the target value f(s); The target network consists of two convolutional layers, the convolution kernel size of the first convolutional layer is 2×2, and the convolution kernel size of the second convolutional layer is 2×1. It receives the joint observation values ​​of all agents, followed by two fully connected layers, and outputs the target value f(s); Step 2.2: Build a prediction network, which will collect All agents in the same group observe information together and output prediction results Use the same network structure as the target network; in, It represents the predicted value output by the prediction network after receiving the joint observation value s of all agents, and ψ is the neural network parameter of the prediction network; The prediction network is trained. For the same input, the expected mean square error is minimized by gradient descent to update the parameter ψ of the prediction network, that is: Step 2.3: Construct an internal reward by calculating the mean square error between the target network and the prediction network for the same input to obtain the internal reward r int ,Right now: Step 2.4: Reshape the discounted reward r total , in the internal reward r int A constraint variable is added before, so that the internal reward is reshaped into the discounted reward r total The proportion of the external reward r decreases as the number of training times increases. ext Gradually play a leading role, enabling the agent to avoid short-sighted behavior and achieve reasonable exploration of the entire task space; namely: r total ＝r ext +w·r int (5) Among them, w is the specific gravity factor, that is: w＝β·(1-0.25×10 -5 ) step (6) β is the initial weight, which is set to 10% of the maximum discount reward that can be obtained in the current scene. The default value is 2. As the time step increases, w will continue to decrease; Step 2.5: Reshape the discounted reward clipping. To ensure the applicability of all set hyperparameters and the consistency of rewards, clip the internal reward to 1 based on normalization by dividing it by a running estimate of the standard deviation of the intrinsic return. In the interval, η∈(0,1] is the cropping ratio, Represents the minimum reward value that can be obtained for the current task. represents the maximum reward value that can be obtained for the current task; then the internal reward is: The final reshaped discounted reward is:

4. The multi-agent experience exploration collaborative method based on curiosity mechanism according to claim 1, characterized in that: Step 3 is as follows: Step 3.1: Build a Behavioral Success Matrix and Behavioral Failure Matrix Where n represents the number of agents, d represents the number of discrete actions of the agent, and the initial matrix values ​​are all 1; at a certain time t, agent i issues a certain instruction j among its available actions and obtains an immediate reward r ext If r ext >0, then The value of the action j issued by the corresponding agent i is increased by 1, otherwise The action j issued by the corresponding agent i is increased by 1, that is: Step 3.2: Construct the probability distribution of the joint action behavior of multiple agents to satisfy the Beta distribution Beta(α,β), where α represents the number of successes of the current action and β represents the number of failures. Combine the evaluation value q of each agent value network with the Bayesian method to estimate the posterior distribution of the strategy; The q value describes the action value that the agent considers in the current state. At time t, agent i receives local observation information and calculates the value of each action. Then the joint value of all agents is Will Normalize and convert it indirectly into the probability value of each action Right now in, represents the neural network parameters of the network at time t, O t Represents the local state information observed by the agent at time t; During training, it is assumed that the probability distribution of each action has a prior distribution Beta(1,1) and a posterior distribution Beta(α+1,β+1); as the training progresses, the corresponding values ​​of the agent's behavior in the success matrix and the failure matrix continue to accumulate and add 1, α+β will gradually increase, and the Beta distribution will become narrower. If α>β, it means that the current action can bring an advantage in the confrontation, otherwise it means that the current action will produce negative feedback; Step 3.2: Exploration of multi-agent joint action behavior. At time t during training, sample from the posterior distribution and obtain the sampling result μ t ,Right now: By choosing μ t The action with the largest value in each is the joint action of the agent. Right now:

5. The multi-agent experience exploration collaborative method based on curiosity mechanism according to claim 1, characterized in that: Step 4 is as follows: Step 4.1: After the t-th round of multi-agent collaborative confrontation ends, all agents in the t-th round before the start of the t+1 round jointly observe s t , the local observation o of round t before the start of round t+1 t , the value of the network GRU state parameter h before the start of round t+1 t,q , the joint action of each agent Remodeling rewards received All agents in round t+1 jointly observe s t+1 With each agent's local observation o t+1 As a tensor Store it in the experience replay pool; combine all agents in round t before the start of round t+1 to observe s t As a tensor [s t ] is stored in the state replay pool; Among them, the experience replay pool and the state replay pool can store m tensors, and m is set between 1e6 and 1e7; Step 4.2: During each training of the value network and the hybrid network, a minimum batch of tensors is randomly drawn from the experience replay pool and trained according to the loss value of the hybrid network; similarly, during each training of the prediction network, a minimum batch of tensors is randomly drawn from the state replay pool and trained according to the expected mean square error. The size of the minimum batch is 64.

6. The multi-agent experience exploration collaborative method based on curiosity mechanism according to claim 1, characterized in that: Step 5 is as follows: Step 5.1: Receive local observations and global observations given by the multi-agent collaborative adversarial environment at the beginning of each round. The local observations are used as the input of the value network, and the global observations are used as the input of the hybrid network, target network, and prediction network. Step 5.2: Mapping of the action q-value output by the value network and the instructions in the multi-agent collaborative adversarial environment; Among them, when the value network calculates the action probability set, the numbers of the actions that are not feasible for the multi-agent in the current state and the corresponding values ​​are set to negative infinity.

7. The multi-agent experience exploration collaborative method based on curiosity mechanism according to any one of claims 1 to 6, characterized in that: Step 6 is as follows: Step 6.1: Initialize hyperparameters Set the experience replay pool size to 1e6, the minimum batch size to 64, the neural network learning rate to 5e-4, the GRU hidden layer size to 64, the discounted return rate for internal and external rewards to 0.99, and the output dimension of the prediction network and target network to 128; Step 6.2: According to the number of agents in the current multi-agent collaborative test platform SMAC confrontation scenario, the local observation space dimension, and the action space dimension, through step 1, set the input size of the multi-layer perceptron at the front end of the value network and the output size of the multi-layer perceptron at the end of the value network; through step 2, set the input size of the outer convolutional layer of the prediction network and the target network, initialize and fix the neural network parameters of the target network; through step 3, initialize the success matrix and the failure matrix to all-1 matrices; Step 6.3: At the beginning of each round of the adversarial scenario, receive local observation information and input it into the value network of each agent to calculate the q-value of all actions of the agent and will The infeasible actions are set to negative infinity; then, each agent The global information is input into the hybrid network to calculate the joint q value Q of the agent tot According to step 3.2, the success matrix, failure matrix and Q tot Get the posterior distribution Beta(α,β) of the joint action of multiple agents for sampling, select the maximum value of each agent's action to form the joint action of multiple agents, and command the red team to fight against the blue team; Step 6.4: Before the end of each round of the confrontation scenario, receive global observation information and external rewards, input the global observation into the target network and the prediction network, calculate the internal reward according to step 2.3, and use the external reward to obtain the reshaped reward according to step 2.

5. At the same time, update the success matrix and failure matrix according to step 3.1; Step 6.5: The experience replay pool and the state replay pool collect and store training data respectively; Step 6.6: For each epoch, extract the minimum batch training policy network and global value network based on the data from the experience replay pool; extract the minimum batch training prediction network based on the data from the state replay pool; Step 6.7: After every 5000 epochs, conduct 32 independent confrontations, calculate the average winning rate, and evaluate the intelligence of the current multi-agent; Step 6.8: When the average winning rate of the multi-agent reaches 1 or the total number of epochs reaches 1e6, the training is completed and saved as a multi-agent experience exploration collaborative model based on the curiosity mechanism.

8. The multi-agent experience exploration collaborative method based on curiosity mechanism according to claim 1, characterized in that: Step 7 is as follows: According to the local observation information given by the confrontation scenario, it is input into the value network of each agent in the multi-agent experience exploration collaborative strategy model based on the curiosity mechanism to obtain the action q value of each agent. And input it together with the global information given by the adversarial scenario into the hybrid network to obtain the joint action q value Q tot , each agent chooses Q tot The corresponding actions with the largest q value constitute the joint actions, and the current multi-agent collaborative confrontation strategy is solved.

9. The multi-agent experience exploration collaborative method based on curiosity mechanism according to claim 1, characterized in that: The multi-agents described in step 1 include SMAC platform combat units, multi-vehicle automatic driving control systems, and war game simulation confrontation systems.

Images