Multi-agent collaborative constraint exploration method based on novelty state prediction

Abstract

This invention discloses a multi-agent cooperative constraint exploration method based on novel state prediction, comprising: constructing a multi-agent centralized training distributed execution decision architecture; designing an environment state reshaping reward; constructing and initializing an exploration experience constraint mechanism suitable for the multi-agent centralized training distributed execution decision architecture; training a deep neural network composed of the multi-agent cooperative constraint exploration architecture based on novel state prediction to obtain a multi-agent cooperative constraint exploration model based on novel state prediction; and solving the multi-agent cooperative adversarial strategy under the current adversarial scenario in the model. This invention improves upon the environment dynamics model to measure the novelty of the global joint state, constructs an intrinsic reward mechanism as a driving signal for the agents' cooperative exploration, and combines dynamically adjusted internal and external reward weights to achieve a smooth transition between exploration and exploitation, avoiding getting trapped in local optima.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent deep reinforcement learning technology, specifically involving a multi-agent cooperative constraint exploration method based on novel state prediction. Background Technology

[0002] In multi-agent deep reinforcement learning, policy exploration is the core mechanism that ensures agents can escape local optima and seek better policies. Reasonable policy exploration methods can help agents quickly learn more efficient policies in unknown environments. Injecting random noise is a traditional, classic exploration strategy that can introduce randomness to some extent, helping agents escape local optima. However, in high-dimensional multi-agent scenarios, purely random exploration is extremely inefficient and struggles to capture the most valuable joint actions in collaborative interactions. Furthermore, traditional random strategies tend to waste significant trial-and-error costs within the vast state-action space. Therefore, most current research aims to further optimize exploration methods in multi-agent scenarios from new dimensions to promote collaborative exploration among agents.

[0003] On the other hand, the balance between exploration and exploitation is particularly prominent in multi-agent reinforcement learning. Overexploration can lead to inefficient policy learning or even deviation from the task objective, while overexploitation can easily result in local optima and a lack of policy diversity. To alleviate the problems of blind exploration caused by the inability of external reward signals to provide timely feedback on exploration results, and insufficient feedback in sparse reward scenarios, recent research has focused on designing intrinsic rewards to increase global rewards. Therefore, how to adaptively and dynamically adjust the ratio between intrinsic and extrinsic rewards, and how to guide agents during training to encourage effective rather than blind exploration, remain issues that require further in-depth research. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-agent cooperative constraint exploration method based on novel state prediction, which solves the problem that excessive blind exploration leads to low policy learning efficiency or even deviation from the task objective; while over-exploitation can easily lead to local optima and lack of policy diversity.

[0005] The technical solution adopted in this invention is: a multi-agent cooperative constraint exploration method based on novel state prediction. By designing a reasonable intrinsic reward mechanism, it guides agents to effectively explore underutilized state spaces, improving the efficiency and stability of cooperative policy learning. It improves upon the environmental dynamics model to measure the novelty of the global joint state, constructs an intrinsic reward mechanism as the driving signal for agent cooperative exploration, and combines dynamic adjustment of internal and external reward weights to achieve a smooth transition between exploration and utilization, avoiding getting trapped in local optima. Furthermore, it introduces an experience screening mechanism to constrain and filter exploration experiences with low task relevance, improving experience utilization efficiency and alleviating the problem of blind exploration; specifically as follows: Step 1: Construct a multi-agent centralized training and distributed execution decision-making architecture; Step 2: Construct an environment state reshaping reward suitable for a multi-agent centralized training distributed execution decision-making architecture; Step 3: Construct an exploratory experience constraint mechanism suitable for a multi-agent centralized training and distributed execution decision-making architecture; Step 4: Initialize the current multi-agent centralized training distributed execution decision architecture; Step 5: Train the deep neural network composed of the multi-agent cooperative constraint exploration architecture based on novel state prediction obtained in Steps 1 to 4 to obtain the multi-agent cooperative constraint exploration model based on novel state prediction. Step 6: Solve the multi-agent cooperative adversarial strategy in the current adversarial scenario in the multi-agent cooperative constraint exploration model based on novel state prediction.

[0006] The invention is further characterized in that, The intelligent agents include homogeneous or heterogeneous moving entities with autonomous decision-making capabilities, including tactical combat units in a game environment, passenger cars / commercial vehicles in a road traffic environment, and drones, infantry fighting vehicles, tanks, and unmanned ground vehicles in a military combat environment.

[0007] Step 1 is as follows: Step 1.1: Construct the Actor network for the basic multi-agent proximal policy optimization algorithm In this multi-agent centralized training and distributed execution decision-making architecture, each agent shares the same Actor network. This maps the observations of each agent to the action probability distribution in the discrete action space, or to the mean and standard deviation vector of a multivariate Gaussian distribution in the continuous action space; where, These are the neural network parameters of the Actor network; It is an intelligent agent i Local observation information; The Actor network uses a multi-layer MLP to receive local observations of the agent. Each layer is set with a ReLU activation function, and the results of the last layer are then passed through a SoftMax activation function to obtain the probability value of each action of the agent. During training, the following objective is maximized using policy gradients: (1)

[0008] During updates, a multi-step minimum batch stochastic gradient descent approach is typically used to optimize the ratio. maximize, B It is the batch size. N The number of agents. For the first i Actions of an intelligent agent, For the old parameters of the Actor network, It is the maximum distance of change in the clipping constraint ratio. It is the entropy value of the current strategy. It is a hyperparameter of entropy, used to calculate the total return using the generalized advantage estimation (GAE) method. : (2) (3) in, This is the GAE discount factor. This is the GAE balance bias-variance tradeoff coefficient. To truncate the step index, For instant rewards, yes State value function estimation; Step 1.2: Constructing a centralized critic network for the basic multi-agent proximal policy optimization algorithm By receiving joint local observation information or global state information, the behavior of the Actor network is scored, helping the Actor network to improve its own strategy, and minimizing the following objective using the Bellman equation method: (4) in, These are the neural network parameters of the current Critic network. These are the old parameters for the Critic network. B For batch size, the total return Through action The reward calculated afterward is also known as the unfulfilled discount reward; It is the cutting constraint ratio; The Critic network is configured to use a three-layer MLP to receive global observations from the multi-agent system. The first two layers are set with Tanh activation functions, and the output dimension of the last layer is set to 1.

[0009] Step 2 is as follows: Step 2.1: Extract high-dimensional features of the state. The State Converter network extracts high-dimensional features of the state through a multi-layer fully connected network. and Effective feature extraction was performed to obtain and And it undergoes encoding processing, the dimension of the state space is... ,Right now , It is the state representation after feature extraction, with dimension 1. : (5) Step 2.2: Extract high-dimensional action features and learn the global action space. The Action Converter network extracts high-dimensional action features for each agent through another multi-layer fully connected network and embedding layers. The dimension of the action space is... Right now , It is the action representation after feature extraction, with a dimension of : (6) Step 2.3: Construct the joint state features of all agents in the next time step prediction ,Will Actions of all intelligent agents at any given moment and global state After feature extraction, it is used as joint input. : (7) Step 2.4: Construction Intrinsic rewards of multi-agent systems at different times By using model prediction error, the error between the predicted and actual values ​​of the joint state features of all agents at the next time step is used as the intrinsic reward of the agent at that time step: (8) in, This indicates that the internal reward is scaled and normalized; the intrinsic reward of each agent is obtained by credit allocation based on the global intrinsic reward, that is, the intrinsic reward is generated by the prediction difference of the global state; Step 2.5: Construct and reshape global environment rewards ,exist Add a constraint variable beforehand to make the internal reward reshape the reward. The proportion of external reward decreases as the number of training sessions increases, eventually leading to... Gradually, the agent will play a leading role, enabling it to avoid short-sighted behavior and achieve a reasonable exploration of the entire task space, namely: (9) in, As a weight factor, It is an external reward, namely:

[0010] The initial weight size is used, and as the time step increases, It will continue to decrease; ; TS To train stride length; The final reward for reshaping the global environment is: (10).

[0011] Step 3 is as follows: Step 3.1: Construct a hard screening mechanism, based on prior knowledge of the task, to filter the data in the experience replay pool and remove invalid experiences that are completely irrelevant to the core learning task of the agent; Step 3.2: Construct a soft screening mechanism based on a centralized critic network. Define global value contribution items This is used to evaluate the correlation between each experience sample and the core task, reflecting whether the experience trajectory samples guide the global system to a more valuable state. (11) in, Used to measure the directionality of trajectory experience's true contribution to team goals, maintaining consistency with MARL's ultimate optimization objective: (12) At this point, the update phases of the Actor network and the Critic network will... As weights, they are incorporated into the objective function: (13) Value function update: (14) in, [] represents the expectation operation. It is the cutting constraint ratio; Step 3.3: Construct an Experience Threshold (TF) screening mechanism. Using the total number of training steps as a parameter, the agent is allowed to explore freely without screening during the first 10%-20% of training steps. During this phase, the agent can accumulate information related to environmental dynamics and state transitions. In the mid-to-late stages of training, the screening mechanism is activated to remove rounds without effective interaction, retaining only trajectories that promote effective adversarial interaction, thereby strengthening the interaction between the agent and the adversary. The default TF trigger phase is set to occur at 30%-40% of the maximum training steps. (15) in, These are the minimum and maximum ratios of the training step size interval triggered by TF, respectively.

[0012] Step 4 specifically includes: Step 4.1: Receive the local and global observations given by the multi-agent cooperative adversarial environment at the beginning of each round. The local observations serve as the input to the Actor network, and the global observations serve as the input to the Critic network and the State Converter. The joint action of the Actor network's output serves as the input to the Action Converter. Step 4.2: Map the action probabilities output by the Actor network to the instructions in the multi-agent cooperative adversarial environment. Set the action numbers that are not feasible for the multi-agent in the current state and their corresponding values ​​to 0, and re-execute the Softmax normalization operation.

[0013] Step 5 specifically includes: Step 5.1: Initialize hyperparameters.

[0014] Set the experience replay pool size, set the minimum batch size, set the neural network learning rate, set the hidden layers, set the discounted return rates for internal and external rewards, and set the output dimensions of the State Converter and Action Converter networks; Step 5.2: Based on the number of agents in the current adversarial scenario, the local observation space dimension, and the action space dimension, set the input size of the MLP at the first layer of the Actor network and the output size of the multilayer perceptron at the end of the Actor network through Step 1; set the input size of the outer convolutional layers of the State Converter network and the Action Converter network through Step 2; Step 5.3: Before the end of each round in the adversarial scenario, receive global observation information and external rewards, input the global observations into the State Converter network, input the joint actions into the Action Converter network, and calculate the internal rewards according to Step 2.4 and combine the external rewards to obtain the reshaping rewards; Step 5.4: Experience replay pool and separate collection and storage of training data; Step 5.5: For each epoch, extract the data from the experience replay pool according to step 3.3 to train the policy network and the global value network using a minimum batch. Step 5.6: After every X epochs, conduct Y independent adversarial trials, calculate the average win rate, and evaluate the intelligence of the current multi-agent system; Step 5.7: When the average win rate of the multi-agents reaches 1 or the total number of epochs reaches the preset value, the training is complete and saved as a multi-agent cooperative constraint exploration model based on novel state prediction.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention overcomes the technical bottlenecks of low collaborative exploration efficiency, poor experience utilization, and difficulty in avoiding local optimum traps in traditional multi-agent reinforcement learning. By constructing a global joint state novelty prediction mechanism based on an environmental dynamics model, it achieves accurate perception and efficient exploration guidance of the collaborative state space, significantly improving the convergence speed and stability of collaborative policy learning in multi-agent systems. Simultaneously, through a dynamically adaptive internal and external reward weight adjustment strategy, it achieves a smooth transition and intelligent switching between the exploration and utilization phases, effectively avoiding premature convergence or blind exploration problems caused by fixed reward structures in traditional methods. Furthermore, the introduced experience screening constraint mechanism fundamentally solves the problem of low-quality exploration experience interfering with policy learning, significantly improving the data utility and sample efficiency of experience replay. Overall, this invention demonstrates stronger environmental adaptability, policy robustness, and learning generalization ability in multi-agent collaborative decision-making tasks under complex dynamic environments, providing a more theoretically complete and engineeringally practical technical path for the autonomous collaborative optimization of large-scale multi-agent systems, possessing significant academic innovation value and broad application prospects. Attached Figure Description

[0016] Figure 1 This is a flowchart of the multi-agent cooperative constraint exploration method based on novel state prediction according to the present invention.

[0017] Figure 2 This is a flowchart of the model training process; Detailed Implementation The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0018] Example 1 The application scenarios of the multi-agent cooperative constraint exploration method based on novel state prediction of this invention are as follows: 1. Multi-agent cooperative adversarial environment This experimental environment is based on the StarCraft multi-agent challenge (SMAC) environment, a multi-agent cooperative adversarial testing platform for Blizzard's real-time strategy game StarCraft II. SMAC includes multiple predefined scenarios of varying difficulty for training and testing multi-agent deep reinforcement learning algorithms. Examples include 3 Marines (3m) vs. 5 Marines, and 3 Stalkers vs. 3 Zealots (3s_vs_3z). In each scenario name, the number represents the number of agents currently in use; for example, 3s_vs_3z indicates an asymmetric adversarial battle between 3 Stalkers and 3 Zealots. 3m indicates a symmetric adversarial battle where 3 Marines are commanded, meaning both sides command the same number and composition of multi-agents. In each episode, the algorithm to be trained controls the red team in a distributed manner, fighting against all enemy units of the blue team. Initially, the red team commanded by the algorithm is symmetrical to the initial position of the opposing blue team; the opposing blue team is ordered to move closer to the red team's position and attack red units.

[0019] In SMAC, a single agent's local observation information is contained within its field of view, which includes a circular area around the map with a radius equal to the line-of-sight. Each agent receives a vector as input, composed of the following features of all units (including allies and enemies) within its field of view: distance, relative x, relative y, and unit type. The global state is hidden from the agents and is a vector composed of unit features across the entire map. It does not contain absolute distances between agents, only storing unit coordinates relative to the map center. Furthermore, the global state includes the health, shields, and attack cooldowns of all units. Additionally, the global state contains the last action of all allied agents. Marines, Stalkers, Zealots, and Colossi have health of 45, 80, 100, and 200 respectively. Furthermore, Stalkers, Zealots, and Colossi have shields of 80, 50, and 150 respectively. All features, whether local observations or global state, are normalized to their maximum values. For all unit types, the agent's aiming distance and firing distance are set to 9 and 6 respectively.

[0020] The action space is represented in a discrete form, which may vary depending on the specific scenario, but is typically represented as: move[direction], attack[enemy_id], stop, and noop. There are only four directions: east, south, west, and north. Attack[enemy_id] is only allowed when the target unit is within firing range. Only dead intelligent agents can and can only execute the noop command.

[0021] 2. Reward and Punishment Rules The reward at each time step is derived from the total damage dealt by the agent to all enemy units. Furthermore, the agent receives 10 points for each opponent defeated, and 200 points for defeating all opponents. This reward is standardized so that the maximum reward the multi-agent can obtain in each episode is 20. Finally, the agent is prohibited from responding to enemy attacks while idle, ensuring that the agent can only achieve victory through its own exploratory decision-making.

[0022] like Figure 1 As shown, the specific process is as follows: Step 1: Construct a multi-agent centralized training and distributed execution decision-making architecture; Step 2: Construct an environment state reshaping reward suitable for a multi-agent centralized training distributed execution decision-making architecture; Step 3: Construct an exploratory experience constraint mechanism suitable for a multi-agent centralized training and distributed execution decision-making architecture; Step 4: Initialize the current multi-agent centralized training distributed execution decision architecture; Step 5: Train the deep neural network composed of the multi-agent cooperative constraint exploration architecture based on novel state prediction obtained in Steps 1 to 4 to obtain the multi-agent cooperative constraint exploration model based on novel state prediction. Step 6: Solve the multi-agent cooperative adversarial strategy in the current adversarial scenario in the multi-agent cooperative constraint exploration model based on novel state prediction.

[0023] Example 2: Based on Example 1, step 1 specifically includes: Step 1.1: Constructing the Actor Network for the Basic Multi-Agent Proximal Policy Optimization Algorithm In this multi-agent centralized training and distributed execution decision-making architecture, each agent shares the same Actor network. This maps the observations of each agent to the action probability distribution in the discrete action space, or to the mean and standard deviation vector of a multivariate Gaussian distribution in the continuous action space. These are the neural network parameters of the Actor network; It is an intelligent agent i Local observation information.

[0024] Specifically, the Actor network uses a three-layer MLP to receive local observations of the agent. Each layer is set with a ReLU activation function, and the results of the last layer are then passed through a SoftMax activation function to obtain the probability value of each action of the agent.

[0025] During training, the following objective is maximized using policy gradients:

[0026]

[0027] During updates, a multi-step minimum batch stochastic gradient descent approach is typically used to optimize the ratio. maximize, B It is the batch size. N The number of agents. For the first i Actions of an intelligent agent, For the old parameters of the Actor network, It is the maximum distance of change in the clipping constraint ratio. It is the entropy value of the current strategy. It is a hyperparameter of entropy, used to calculate the total return using the generalized advantage estimation (GAE) method. :

[0028]

[0029] in, This is the GAE discount factor. This is the GAE balance bias-variance tradeoff coefficient. To truncate the step index, For instant rewards, yes State value function estimation; Step 1.2: Constructing a centralized critic network for the basic multi-agent proximal policy optimization algorithm By receiving joint local observation information or global state information, the behavior of the Actor network is scored, helping the Actor network to improve its own strategy, and minimizing the following objective using the Bellman equation method:

[0030] in, These are the neural network parameters of the current Critic network. These are the old parameters for the Critic network. B For batch size, the total return Through action The reward calculated afterward is also known as the unfulfilled reward-to-go.

[0031] Specifically, the Critic network is configured to use a three-layer MLP to receive global observations from the multi-agent, with the first two layers using the Tanh activation function and the last layer having an output dimension of 1.

[0032] Example 3 Based on Example 2, step 2 specifically includes Step 2.1: Extract high-dimensional features of the state. The State Converter network extracts high-dimensional features of the state through a multi-layer fully connected network. and Effective feature extraction was performed to obtain and And it undergoes encoding processing, the dimension of the state space is... Right now , It is the state representation after feature extraction, with dimension 1. :

[0033] Step 2.2: Extract high-dimensional action features and learn the global action space. The Action Converter network extracts high-dimensional action features for each agent through another multi-layer fully connected network and embedding layers. The dimension of the action space is... Right now , It is the action representation after feature extraction, with a dimension of :

[0034] Step 2.3: Construct the joint state features of all agents in the next time step prediction ,Will Actions of all intelligent agents at any given moment and global state After feature extraction, it is used as joint input. :

[0035] Step 2.4: Construction Intrinsic rewards of multi-agent systems at different times By using model prediction error, the error between the predicted and actual values ​​of the joint state features of all agents at the next time step is used as the intrinsic reward of the agent at that time step:

[0036] in, This refers to the internal reward scaling and normalization process, which prevents the internal reward value from being too large or too small, thus affecting the stability of policy updates. The internal reward for each agent is obtained through credit allocation based on the global internal reward; that is, the internal reward is generated from the predicted differences in the global state.

[0037] Step 2.5: Construct and reshape global environment rewards ,exist Add a constraint variable beforehand to make the internal reward reshape the reward. The proportion of external reward decreases as the number of training sessions increases, eventually leading to... Gradually, the agent will play a leading role, enabling it to avoid short-sighted behavior and achieve a reasonable exploration of the entire task space. That is:

[0038] in, As a weight factor, It is an external reward, namely:

[0039] The initial weight is set to 0.5, and as the time step increases, It will continue to decrease. This range makes the training process more robust, ensuring that all agents can fully explore the environment and effectively use the exploration experience for policy utilization in the later stages of training. The default setting is 0.25.

[0040] The final reward for reshaping the global environment is:

[0041] Example 4 Based on Example 3, step 3 specifically includes: Step 3.1: Construction of the hard-filtering mechanism, which filters the data in the experience replay pool based on prior knowledge of the task to remove invalid experiences that are completely irrelevant to the core learning task of the agent.

[0042] Specifically, the goal of the selection process is to prevent the agent from blindly exploring during training, which would lead to excessive concentration of training data on meaningless state transitions, thus affecting the effective learning of the agent's strategy. Taking a cooperative adversarial scenario as an example, each round of training may end in three ways: (1) all agents die, round is lost; (2) all enemies are eliminated, and there are surviving agents, round is won; (3) the round step limit is reached. In the early stages of training, we hope that the agent can freely explore, collect a wide range of state-action pairs from the environment, and establish a preliminary understanding of the dynamics of the environment. At this stage, the agent is allowed to explore irrelevant edge areas and unimportant state transitions. However, as training progresses, the agent's understanding of the environment gradually deepens, and the efficiency of exploration needs to be improved. Therefore, it is necessary to gradually reduce the tolerance for invalid exploration trajectories and ultimately retain only those trajectories that can truly promote the adversarial objective.

[0043] The trajectory data stored in the experience replay pool were differentiated according to the three ending conditions of the training rounds mentioned above, and Boolean variables were set. As a marker for the end of a round, A value of true indicates either a win or a loss in the round, i.e., the first two possibilities. If false, it represents the third case. In the early exploration phase of training, the trajectory data in case (3) means that the agent has not engaged in sufficient and effective confrontation with the enemy, nor has it received effective feedback. It has low value for training and may interfere with the agent's policy convergence process. Therefore, we differentiate and filter the trajectory data in the experience replay pool, and only retain the trajectory data of the first two rounds at the end for policy updates, so as to guide the agent to actively interact and confront the enemy and explore cooperative strategies to alleviate the problem of the agent getting stuck in local optima.

[0044] Step 3.2: Construction of a soft-filtering mechanism based on a centralized critic network Define global value contribution items This is used to evaluate the correlation between each experience sample and the core task, reflecting whether the experience trajectory samples guide the global system to a more valuable state.

[0045] in, Used to measure the directionality of trajectory experience's true contribution to team goals, maintaining consistency with MARL's ultimate optimization objective:

[0046] Soft screening, as a value-perceived weighting mechanism, suppresses the influence of noise and low-quality samples, amplifying information and experience. This is particularly important in multi-agent cooperation, where the update phases of the Actor and Critic networks will... As weights, they are incorporated into the objective function:

[0047] Value function update:

[0048] Step 3.3: Construct an Experience Threshold Filtering (TF) mechanism. Using the total number of training steps as a parameter, the agent is allowed to explore freely without filtering during the first 10%-20% of training steps. During this phase, the agent can accumulate information related to environmental dynamics and state transitions. The filtering mechanism is activated in the mid-to-late stages of training to remove rounds without effective interaction, retaining only trajectories that promote effective adversarial interaction, thereby strengthening the agent's interaction with the adversary. The default TF trigger phase is set to occur at 30%-40% of the maximum training steps.

[0049] in, These are the minimum and maximum ratios of the training step size interval triggered by TF, respectively.

[0050] To maximize the effectiveness of the threshold selection mechanism, the selection threshold can be dynamically adjusted based on the agent's performance at each stage in different scenarios. If the agent performs poorly and exhibits strong exploratory behavior in the initial stages, the end time of the selection can be appropriately extended, allowing the agent more time for effective exploration. Reasonable experience selection can effectively constrain the agent's exploration, enabling it to fully explore collaborative strategies while avoiding getting bogged down in ineffective exploration or local optima in later stages. By setting trigger conditions for the threshold selection mechanism, it can be ensured that the agent continuously acquires valuable experience during training, ultimately learning the globally optimal strategy.

[0051] Example 5: Based on Example 4, step 4 specifically includes: Step 4.1: Receive the local and global observations given by the multi-agent cooperative adversarial environment at the beginning of each round. The local observations serve as the input to the Actor network, and the global observations serve as the input to the Critic network and the State Converter. The joint action of the Actor network's output serves as the input to the Action Converter.

[0052] Step 4.2: Map the action probabilities output by the Actor network to the instructions in the multi-agent cooperative adversarial environment. Set the numbers of actions that are not feasible for the multi-agent in the current state and their corresponding values ​​to 0, and re-execute the Softmax normalization operation.

[0053] Example 6 Based on Example 5, step 5 trains the deep neural network composed of the multi-agent cooperative constraint exploration architecture based on novel state prediction obtained in steps 1 to 4 to obtain a multi-agent cooperative constraint exploration model based on novel state prediction; specifically including: Step 5.1: Initialize hyperparameters.

[0054] Set the experience replay pool size to 1e6, the mini-batch size to 64, the neural network learning rate to 5e-4, the hidden layer size to 64, the discount rate between internal and external rewards to 0.99, and the output dimensions of the StateConverter and ActionConverter networks to 128.

[0055] Step 5.2: Based on the number of agents, the local observation space dimension, and the action space dimension in the current SMAC adversarial scenario, set the input size of the MLP in the first layer of the Actor network and the output size of the multilayer perceptron at the end of the Actor network through Step 1. Set the input size of the outer convolutional layers of the State Converter network and the Action Converter network through Step 2.

[0056] Step 5.3: Before the end of each round in the adversarial scenario, receive global observation information and external rewards, input the global observations into the State Converter network, input the joint actions into the Action Converter network, calculate the internal rewards according to step 2.4, and reshape the external rewards according to step 2.5.

[0057] Step 5.4: Collect and store training data in the experience replay pool and the state replay pool respectively.

[0058] Step 5.5: For each epoch, extract the data from the experience replay pool to train the minimum batch policy network and the global value network according to step 3.3.

[0059] Step 5.6: After every 5000 epochs, conduct 32 independent adversarial matches, calculate the average win rate, and evaluate the intelligence of the current multi-agent system.

[0060] Step 5.7: When the average win rate of the multi-agents reaches 1 or the total number of epochs reaches 1e6, training is complete, and the model is saved as a multi-agent cooperative constraint exploration model based on novel state prediction.

Claims

1. A method for multi-agent collaborative constraint exploration based on novelty state prediction, characterized in that, By designing an intrinsic reward mechanism, the agent is guided to effectively explore the underutilized state space, thereby improving the efficiency and stability of collaborative policy learning. By improving the environmental dynamics model to measure the novelty of the global joint state, an intrinsic reward mechanism is constructed as a driving signal for the agent's collaborative exploration. The weights of internal and external rewards are dynamically adjusted to achieve a smooth transition between exploration and utilization, avoiding getting trapped in local optima. An experience screening mechanism is introduced to constrain and filter exploration experiences with low task relevance, thereby improving the efficiency of experience utilization.

2. The novel state prediction based multi-agent collaborative constraint exploration method according to claim 1, characterized in that, Specifically as follows: Step 1: Construct a multi-agent centralized training and distributed execution decision-making architecture; Step 2: Construct an environment state reshaping reward suitable for a multi-agent centralized training distributed execution decision-making architecture; Step 3: Construct an exploratory experience constraint mechanism suitable for a multi-agent centralized training and distributed execution decision-making architecture; Step 4: Initialize the current multi-agent centralized training distributed execution decision architecture; Step 5: Train the deep neural network composed of the multi-agent cooperative constraint exploration architecture based on novel state prediction obtained in Steps 1 to 4 to obtain the multi-agent cooperative constraint exploration model based on novel state prediction. Step 6: Solve the multi-agent cooperative adversarial strategy in the current adversarial scenario in the multi-agent cooperative constraint exploration model based on novel state prediction.

3. The multi-agent cooperative constraint exploration method based on novel state prediction according to claim 2, characterized in that, The intelligent agents include homogeneous or heterogeneous moving entities with autonomous decision-making capabilities, including tactical combat units in a game environment, passenger cars / commercial vehicles in a road traffic environment, and drones, infantry fighting vehicles, tanks, and unmanned ground vehicles in a military combat environment.

4. The multi-agent cooperative constraint exploration method based on novel state prediction according to claim 2, characterized in that, Step 1 is as follows: Step 1.1: Construct the Actor network for the basic multi-agent proximal policy optimization algorithm In this multi-agent centralized training and distributed execution decision-making architecture, each agent shares the same Actor network. This maps the observations of each agent to the action probability distribution in the discrete action space, or to the mean and standard deviation vector of a multivariate Gaussian distribution in the continuous action space; where, These are the neural network parameters of the Actor network; It is an intelligent agent i Local observation information; The Actor network uses a multi-layer MLP to receive local observations of the agent. Each layer is set with a ReLU activation function, and the results of the last layer are then passed through a SoftMax activation function to obtain the probability value of each action of the agent. During training, the following objective is maximized using policy gradients: （1） During updates, a multi-step minimum batch stochastic gradient descent approach is typically used to optimize the ratio. maximize, B It is the batch size. N For the number of agents, For the first i Actions of an intelligent agent, For the old parameters of the Actor network, It is the maximum change distance of the clipping constraint ratio. It is the entropy value of the current strategy. It is a hyperparameter of entropy, used to calculate the total return using the generalized advantage estimation (GAE) method. : （2） （3） in, This is the GAE discount factor. This is the GAE balance bias-variance tradeoff coefficient. To truncate the step index, For instant rewards, yes State value function estimation; Step 1.2: Constructing a centralized critic network for the basic multi-agent proximal policy optimization algorithm By receiving joint local observation information or global state information, the behavior of the Actor network is scored, helping the Actor network to improve its own strategy, and minimizing the following objective using the Bellman equation method: （4） in, These are the neural network parameters of the current Critic network. These are the old parameters for the Critic network. B For batch size, the total return Through action The reward calculated afterward is also known as the unfulfilled discount reward; It is the cutting constraint ratio; The Critic network is configured to use a three-layer MLP to receive global observations from the multi-agent system. The first two layers are set with Tanh activation functions, and the output dimension of the last layer is set to 1.

5. The multi-agent cooperative constraint exploration method based on novel state prediction according to claim 4, characterized in that, Step 2 is as follows: Step 2.1: Extract high-dimensional features of the state. The State Converter network extracts high-dimensional features of the state through a multi-layer fully connected network. and Effective feature extraction was performed to obtain and And it undergoes encoding processing, the dimension of the state space is... ,Right now , It is the state representation after feature extraction, with dimension 1. : （5） Step 2.2: Extract high-dimensional action features and learn the global action space. The Action Converter network extracts high-dimensional action features for each agent through another multi-layer fully connected network and embedding layers. The dimension of the action space is... Right now , It is the action representation after feature extraction, with a dimension of : （6） Step 2.3: Construct the joint state features of all agents in the next time step prediction ,Will Actions of all intelligent agents at any given moment and global state After feature extraction, it is used as joint input. : （7） Step 2.4: Construction Intrinsic rewards of multi-agent systems at different times By using model prediction error, the error between the predicted and actual values ​​of the joint state features of all agents at the next time step is used as the intrinsic reward of the agent at that time step: （8） in, This indicates that the internal reward is scaled and normalized; the intrinsic reward of each agent is obtained by credit allocation based on the global intrinsic reward, that is, the intrinsic reward is generated by the prediction difference of the global state; Step 2.5: Construct and reshape global environment rewards ,exist Add a constraint variable beforehand to make the internal reward reshape the reward. The proportion of external reward decreases as the number of training sessions increases, eventually leading to... Gradually, the agent will play a leading role, enabling it to avoid short-sighted behavior and achieve a reasonable exploration of the entire task space, namely: （9） in, As a weight factor, It is an external reward, namely: The initial weight size is used, and as the time step increases, It will continue to decrease; ; TS To train stride length; The final reward for reshaping the global environment is: （10）。 6. The multi-agent cooperative constraint exploration method based on novel state prediction according to claim 5, characterized in that, Step 3 is as follows: Step 3.1: Construct a hard screening mechanism, based on prior knowledge of the task, to filter the data in the experience replay pool and remove invalid experiences that are completely irrelevant to the core learning task of the agent; Step 3.2: Construct a soft screening mechanism based on a centralized critic network. Define global value contribution items This is used to evaluate the correlation between each experience sample and the core task, reflecting whether the experience trajectory samples guide the global system to a more valuable state. （11） in, Used to measure the directionality of trajectory experience's true contribution to team goals, maintaining consistency with MARL's ultimate optimization objective: （12） At this point, the update phases of the Actor network and the Critic network will... As weights, they are incorporated into the objective function: （13） Value function update: （14） in, [] represents the expectation operation. It is the cutting constraint ratio; Step 3.3: Construct an Experience Threshold (TF) screening mechanism. Using the total number of training steps as a parameter, the agent is allowed to explore freely without screening during the first 10%-20% of training steps. During this phase, the agent can accumulate information related to environmental dynamics and state transitions. In the mid-to-late stages of training, the screening mechanism is activated to remove rounds without effective interaction, retaining only trajectories that promote effective adversarial interaction, thereby strengthening the interaction between the agent and the adversary. The default TF trigger phase is set to occur at 30%-40% of the maximum training steps. （15） in, These are the minimum and maximum ratios of the training step size interval triggered by TF, respectively.

7. The multi-agent cooperative constraint exploration method based on novel state prediction according to claim 6, characterized in that, Step 4 specifically includes: Step 4.1: Receive the local and global observations given by the multi-agent cooperative adversarial environment at the beginning of each round. The local observations serve as the input to the Actor network, and the global observations serve as the input to the Critic network and the State Converter. The joint action of the Actor network's output serves as the input to the Action Converter. Step 4.2: Map the action probabilities output by the Actor network to the instructions in the multi-agent cooperative adversarial environment. Set the action numbers that are not feasible for the multi-agent in the current state and their corresponding values ​​to 0, and re-execute the Softmax normalization operation.

8. The multi-agent cooperative constraint exploration method based on novel state prediction according to claim 7, characterized in that, Step 5 specifically includes: Step 5.1: Initialize hyperparameters Set the experience replay pool size, set the minimum batch size, set the neural network learning rate, set the hidden layers, set the discounted return rates for internal and external rewards, and set the output dimensions of the State Converter and Action Converter networks; Step 5.2: Based on the number of agents in the current adversarial scenario, the local observation space dimension, and the action space dimension, set the input size of the MLP at the first layer of the Actor network and the output size of the multilayer perceptron at the end of the Actor network through Step 1; set the input size of the outer convolutional layers of the State Converter network and the Action Converter network through Step 2; Step 5.3: Before the end of each round in the adversarial scenario, receive global observation information and external rewards, input the global observations into the State Converter network, input the joint actions into the Action Converter network, and calculate the internal rewards according to Step 2.4 and combine the external rewards to obtain the reshaping rewards; Step 5.4: Experience replay pool and separate collection and storage of training data; Step 5.5: For each epoch, extract the data from the experience replay pool according to step 3.3 to train the policy network and the global value network using a minimum batch. Step 5.6: Calculate the average win rate through multiple independent adversarial chaining exercises to evaluate the intelligence of the current multi-agent system; Step 5.7: When the average win rate of the multi-agents reaches 1 or the total number of epochs reaches the preset value, the training is complete and saved as a multi-agent cooperative constraint exploration model based on novel state prediction.

Images