A multi-agent cooperative interception decision model construction method and system

Abstract

This invention provides a method and system for constructing a multi-agent cooperative interception decision model in the field of agent control. The method includes determining the actions that multiple agents independently execute at the current decision moment; updating the relative motion state based on the actions to obtain the relative motion state at the next decision moment and determining the individual immediate reward; determining the total reward of the agent after executing the action based on the individual immediate reward of the agent and the individual immediate rewards of neighboring agents; generating an experience tuple for the agent at the current decision moment by combining the state, action, total reward, and state at the next moment; calculating a loss function based on the experience tuple and optimizing the parameters of the agent's deep Q-network to obtain an optimized deep Q-network; and determining the multi-agent cooperative interception decision model based on the optimized deep Q-network of each agent. This method improves the efficiency of constructing the cooperative interception decision model.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agent control, specifically relating to a method and system for constructing a multi-agent collaborative interception decision model. Background Technology

[0002] Multi-agent cooperative interception decision-making refers to the decision-making process of a swarm of autonomous intelligent agents, such as drones and robots, in a dynamic adversarial environment, coordinating their actions to jointly complete tasks such as interception and defense against moving or stationary targets. Its core function is to compensate for the shortcomings of individual agents through group collaboration, thereby significantly improving the overall mission success rate, robustness, and resource utilization efficiency when facing complex tasks and strong adversaries. Existing multi-agent cooperative interception decision-making tasks are often modeled as multi-agent game problems, with the core challenge being how to achieve efficient and adaptive cooperative decision-making under conditions of numerous agents, complex environments, and strong adversarial dynamics.

[0003] Based on the above adjustments, existing technologies often employ Independent Q-Learning (IQL), where each agent independently uses a deep Q-network, treating other agents as part of the environment for policy learning. However, this approach results in the environment dynamically changing over training time, making it difficult to converge to an effective cooperative policy and leading to inefficient model building. Summary of the Invention

[0004] To address the existing problems, this invention provides a method and system for constructing a multi-agent collaborative interception decision model.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for constructing a multi-agent cooperative interception decision model includes: A deep Q-network is initialized for each agent. The input of the deep Q-network is the relative motion state of the agent with respect to the interception target and other agents. The output of the deep Q-network is the Q value corresponding to the agent performing multiple actions. At each decision-making moment, a deep Q-network is used to determine the actions that multiple agents can independently execute at the current decision-making moment. The relative motion state of each agent is updated based on its actions to obtain the relative motion state of each agent at the next decision-making moment. According to a preset interception rule, the individual immediate reward corresponding to the action executed by each agent is determined by its relative motion state at the next decision-making moment. The total reward after the agent's action is determined by its individual immediate reward and the individual immediate rewards of its neighboring agents, where neighboring agents are determined by the relative motion states of other agents relative to the agent at the next moment. Each agent generates its experience tuple at the current decision-making moment by combining its relative motion state at the current decision-making moment, the independently executed action, the total reward, and its relative motion state at the next moment. The Q-learning target value of the agent is calculated based on the empirical tuple. The loss function is calculated using the Q-learning target value and the Q-values ​​corresponding to the actions in the empirical tuple. The parameters of the deep Q-network of the agent are optimized based on the loss function calculation results to obtain the optimized deep Q-network. The multi-agent cooperative interception decision model is determined by the optimized deep Q-network of each agent.

[0006] Optionally, the multi-agent cooperative interception decision model construction method provided by the present invention further includes: For each agent, other agents whose relative distance is within a preset range are considered as the agent's neighboring agents.

[0007] Optionally, the multi-agent cooperative interception decision model construction method provided by the present invention further includes: The individual instant reward for each agent is determined based on whether the interception of the target is successful or not, or the change in distance between each agent and the target.

[0008] Optionally, the multi-agent cooperative interception decision model construction method provided by the present invention further includes: Based on a preset reward sharing coefficient, the collaborative reward of an agent is determined by the individual instant rewards of neighboring agents; The total reward for the agent is determined by the combination of collaborative rewards and individual immediate rewards.

[0009] Optionally, the multi-agent cooperative interception decision model construction method provided by the present invention further includes: The experience replay pool is constructed by using the experience tuples obtained from multiple moments of the agent through an online Q-network. Sample from the experience replay pool; determine the agent's Q-learning target value from the total reward in the sampling results through the target Q-network.

[0010] Optionally, the multi-agent cooperative interception decision model construction method provided by the present invention further includes: The Q-values ​​corresponding to the actions in the sampling results are calculated using an online Q-network. The mean squared error loss function is calculated using the Q-learning target value and Q-value of the agent.

[0011] Optionally, the multi-agent cooperative interception decision model construction method provided by the present invention further includes: The parameters of the online Q-network are updated using gradient descent based on the mean squared error loss function to obtain the optimized online Q-network. The parameters of the target Q-network are then softly updated based on the optimized online Q-network parameters to obtain the optimized target Q-network.

[0012] This invention also provides a system for constructing a multi-agent cooperative interception decision model, comprising: The initialization module is used to initialize a deep Q-network for each agent. The input of the deep Q-network is the relative motion state of the agent with the interception target and other agents, and the output of the deep Q-network is the Q value corresponding to the agent performing multiple actions. The experience tuple construction module is used to determine the actions that multiple agents can independently perform at each decision time using a deep Q-network, and update the relative motion state of each agent based on its actions to obtain the relative motion state of each agent at the next decision time. According to preset interception rules, the individual immediate reward corresponding to the action performed by each agent is determined by its relative motion state at the next decision time. The total reward after the agent performs the action is determined by the individual immediate reward of the agent and the individual immediate rewards of neighboring agents, where neighboring agents are determined by the relative motion states of other agents relative to the agent at the next time. Each agent generates its experience tuple at the current decision time by combining its relative motion state at the current decision time, the action performed independently, the total reward, and its relative motion state at the next time. The model building module is used to calculate the Q-learning target value of the agent based on the empirical tuple. The loss function is calculated by using the Q-learning target value and the Q-values ​​corresponding to the actions in the empirical tuple. The parameters of the deep Q-network of the agent are optimized based on the loss function calculation results to obtain the optimized deep Q-network. The multi-agent cooperative interception decision model is determined by the optimized deep Q-network of each agent.

[0013] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the steps in a method for constructing a multi-agent cooperative interception decision model.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, which, when loaded by a processor, is capable of executing any step of a method for constructing a multi-agent cooperative interception decision model.

[0015] The multi-agent cooperative interception decision model construction method provided by this invention has the following beneficial effects: In this invention, at each decision-making moment, each agent independently selects and executes an action based on its current state through the network. All actions collectively drive the simulation environment update, thereby obtaining the new state of each agent at the next moment. After the environment update, the system calculates the individual immediate reward obtained by each agent for its own actions according to the preset interception rules. At the same time, it determines the neighboring agents of each agent based on the updated state and contributes the individual immediate rewards of each agent to its neighbors in a certain way, thereby synthesizing a total reward for each agent that includes both its own gains and the contributions of its neighbors. Finally, the target value of Q-learning is calculated based on the total reward, and the loss is calculated by comparing it with the network's Q-value estimate for the current action. This optimizes the parameters of the deep Q-network of each agent, resulting in a cooperative interception decision model composed of the optimized networks of each agent.

[0016] This invention introduces a reward-sharing mechanism based on dynamic neighbor relationships. When calculating the total reward used for network updates, it includes not only the immediate reward of the agent's own actions but also a portion of the rewards from its neighboring agents. This mechanism ensures that each agent's Q-value update signal implicitly incorporates feedback from the effects of local peer behaviors when optimizing its own policy. The coupling of reward signals establishes an indirect collaborative learning link between agents, stabilizing the dynamic changes in the environment caused by each agent's policy updates. This guides agents to naturally consider the impact of their behavior on their neighbors and the local group during the learning process. It promotes the convergence of dispersed individual policies towards collaboration, solving the problems of collaboration difficulties and low convergence efficiency caused by non-stationary environments in independent Q-learning. Furthermore, it improves the construction efficiency of collaborative interception decision models without requiring centralized coordination or global information exchange. Attached Figure Description

[0017] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a method for constructing a multi-agent cooperative interception decision model according to an embodiment of the present invention; Figure 2This is a schematic diagram of the agent's local perception and attack range model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the dynamic local interaction topology provided in an embodiment of the present invention; Figure 4 This is an example of a deep Q-learning algorithm framework provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a large-scale cluster interception game scenario provided in an embodiment of the present invention; Figure 6 This is an example of the curve showing the change of the average cluster reward with the number of training rounds in a 12v20 scenario provided by an embodiment of the present invention; Figure 7 This is an example of the average power consumption variation curve with the number of training rounds in a 12v20 scenario provided by an embodiment of the present invention; Figure 8 This is an example of the curve showing the change of the average cluster reward with the number of training rounds in a 15v30 scenario provided by an embodiment of the present invention; Figure 9 This is one example of an algorithm robustness analysis graph provided in an embodiment of the present invention; Figure 10 This is the second example of an algorithm robustness analysis diagram provided in an embodiment of the present invention; Figure 11 This is one example of a diagram illustrating the coupling effect of boundary conditions and attack decisions provided in an embodiment of the present invention. Figure 12 This is the second example of a diagram illustrating the coupling effect of boundary conditions and attack decisions in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0020] To address the need for collaborative interception decision-making by intelligent agents, existing technologies often employ centralized planning and control methods such as Model Predictive Control (MPC) and Dynamic Programming (DP) for task modeling. However, these methods rely on centralized optimization using global state information and precise dynamic models, which suffers from high computational complexity, poor real-time performance, and the risk of single points of failure in large-scale scenarios. In contrast, distributed methods based on artificial rules, such as biomimetic clustering algorithms that borrow from the Boids model and interception methods based on geometric guidance laws, coordinate through pre-defined behavioral mechanisms like separation, alignment, and aggregation. These methods offer advantages such as fast response and computational simplicity. However, they lack policy flexibility, and their performance verification depends on… Expert experience alone is insufficient to adapt to unforeseen complex situations, exhibiting low robustness and failing to meet the application requirements of large-scale cluster collaborative interception. Methods based on Multi-Agent Deep Reinforcement Learning (MARL), such as Value Decomposition Networks (VDN) and QMIX, employ centralized training and distributed execution. While they learn a decomposable joint value function using global information during training and make distributed decisions during execution, their strong dependence on global state information during training makes it difficult to scale to ultra-large-scale clusters of more than 50 agents. This results in high communication overhead, high training complexity, and centralized trainers often becoming system bottlenecks. While Independent Q-Learning can reduce communication overhead, this method operates in a non-stationary environment during training, making it difficult to converge to an effective collaborative strategy.

[0021] In summary, there is an urgent need for a fully distributed decision-making method that allows each agent to emerge with efficient global collaborative behavior through autonomous learning, based solely on local perception information and interactions with a limited number of neighbors. To address this, this invention provides a large-scale multi-agent collaborative interception decision-making scheme based on local collaborative deep Q-learning. Each agent, without being aware of the global situation or relying on a centralized controller, defines its interaction scope through "local topological constraints" and designs a "reward local redistribution" mechanism to diffuse the benefits or costs of individual actions to its neighbors. This incentivizes altruism and cooperation at the individual level, ultimately achieving self-organized collaborative intelligence at the group level, thereby improving task success rate while reducing system energy consumption.

[0022] Example 1 This invention provides a method for constructing a multi-agent cooperative interception decision model, specifically as follows: Figure 1 As shown, it includes the following steps: Step 11: Initialize a deep Q-network for each agent. The input of the deep Q-network is the relative motion state of the agent with respect to the interception target and other agents. The output of the deep Q-network is the Q value corresponding to the multiple actions performed by the agent.

[0023] Specifically, a simulation environment for a large-scale cluster interception game is first constructed, and the kinematic model of the agents, the limited-range local perception model of each agent, and the dynamic neighbor relationship are defined. For example, the motion model of each agent can adopt the mass model, and the kinematic equation is shown in formula (1):

[0024] (1) in, Let be the kinematic equation of agent i in the horizontal direction. Let be the kinematic equation of agent i in the vertical direction. Let i be the speed of agent i. Let be the angle between the velocity of agent i and the horizontal direction. During the game evolution, the relative motion relationship between the attacking and defending agents in space is shown in formula (2):

[0025] (2) in, This represents the change in distance between agents. This represents the change in the line-of-sight angle between agents. The distance between agents. The angle between the defender's speed and the horizontal direction. The angle between the attacker's speed and the horizontal. The angle between the two agents. The speed of the attacking agent. For the speed of the defending AI, This represents the horizontal component of the defender's position. This represents the vertical component of the defender's position. This represents the horizontal component of the attacker's position. This represents the vertical component of the attacker's position.

[0026] The local perception and interaction model is used to dynamically perceive and identify neighboring agents around each agent when the agent's position changes. For example, for agent i, it can perceive agents with a radius of 1000 m centered on itself. The system collects information such as the position, velocity, and identity of other agents within a circular region, and forms a dynamic neighbor set for agent i at time t based on the information perceived from other agents. Furthermore, the interaction topology of the local perception and interaction model can be further abstracted into models such as small-world networks, thereby simulating lossy or restricted communication in actual interception decisions.

[0027] For example, such as Figure 2As shown, for an agent, its local perception range model is the area encompassed by its perception radius, while its attack range model is the range between the agent's attack radius and the angles it can attack. The dynamic local interaction topology between multiple agents is illustrated as follows: Figure 3 As shown, for agent 0, there are agents 1 and 2 within its perception radius. These two agents are the neighboring agents of agent 0, defined by the distance of agent 1 relative to agent 0. Angle with direction of motion And the distance between agent 2 and agent 0. Angle with direction of motion Construct the dynamic neighbor set of agent 0 at this moment, and then send the instant reward obtained by agent 0 after performing the action in the previous moment to agent 1 and agent 2 respectively, and send the instant rewards obtained by agent 1 and agent 2 after performing the action in the previous moment to agent 0.

[0028] Specifically, to establish a micro-level mechanism based on local information interaction, this invention can be implemented through two methods: static local interaction and dynamic local interaction. Static local interaction refers to the fact that the identity of each agent interacting is locally invariant, meaning that the identity of the agent's neighbors remains unchanged in long-term decision-making. Dynamic local interaction, on the other hand, refers to the fact that the identities of interacting neighbors change according to specific predetermined rules during the long-term decision-making process. The network topology influences the efficiency of information propagation, the connection methods between nodes, and the information transmission paths, playing a crucial role in the evolution of agent decision-making.

[0029] Classic network structures include small-world networks, scale-free networks, random networks, and regular networks. In complex networks, each node can represent an individual, an organization, or an information propagation unit, while edges represent relationships or connections between nodes. In this invention, each agent can be considered a node, and the edge relationships between nodes represent the pairwise game relationships between two agents. Specific topological attribute constraints are used to analyze the emergence and evolution of the overall system decision-making behavior patterns under different information interaction mechanisms. For example, in small-world networks, the distance between two randomly selected nodes increases logarithmically with the number of nodes in the network; that is, such systems generally have two significant characteristics: a high clustering coefficient and a short average path length (ASPL). In regular networks, each node is connected to the same number of neighboring nodes, meaning the connections in the network are very uniform, exhibiting high symmetry and a long average path length. Due to the regularity of the network, traveling from one node to another may require a relatively long path; therefore, the average path length of regular networks is usually longer than that of random or small-world networks. Due to the simplicity of regular network topology, it is very suitable for modeling, developing and verifying the effectiveness of new algorithms from the perspective of complex systems.

[0030] In typical regular networks, the boundaries are considered interconnected; that is, nodes extending beyond the boundary connect to corresponding nodes on the boundary, forming a surrounding structure. This approach is used to simulate an infinite space within a finite region. This is because boundary effects can significantly influence the system's behavior. By using periodic boundary conditions, the nodes on the boundary are connected in a surrounding manner, reducing the impact of boundary effects and making the system closer to the properties of an infinite space. It also simplifies the model's complexity, making it easier to process and analyze.

[0031] like Figure 4 As shown, once the simulation environment is built, an independent, structurally identical deep Q-network is initialized for each agent participating in cooperative interception. This deep Q-network simultaneously contains a target Q-network or a Q-target network, as well as an online Q-network or a Q-estimation network. Then, the construction of empirical tuples begins. Step 12: At each decision time, a deep Q-network is used to determine the actions that multiple agents can independently execute at the current decision time. The relative motion state of each agent is updated based on its actions to obtain the relative motion state of each agent at the next decision time. According to preset interception rules, the individual immediate reward corresponding to the action executed by each agent is determined by its relative motion state at the next decision time. The total reward after the agent's action is determined by its individual immediate reward and the individual immediate rewards of its neighboring agents. Neighboring agents are determined by the relative motion states of other agents relative to the agent at the next decision time. Each agent generates its experience tuple at the current decision time by combining its relative motion state at the current decision time, the independently executed action, the total reward, and its relative motion state at the next decision time. For example, for each agent, other agents within a preset distance are considered its neighboring agents.

[0032] Step 12 includes: Step 121: Determine the individual instant reward for each agent based on whether the interception of the target is successful or not, or the change in distance between each agent and the target.

[0033] Step 122: Based on a preset reward sharing coefficient, determine the collaborative reward of the agent by the individual instant rewards of neighboring agents; Step 123: Determine the total reward for the agent by combining collaborative rewards and individual immediate rewards.

[0034] Specifically, in this invention, other nodes directly connected to each node are considered as neighboring nodes that the agent can perceive. For example, for a network G, its node set is as follows: The edge set in the network is At this point, for agent i, its neighbor set This will allow all interactions with agent i to be achieved through the edge. Directly connected nodes Summarizing them into a single set, considering the temporal changes in network structure under dynamic topology constraints, the neighbor set at time t is... .

[0035] Furthermore, the local cooperative mechanism established by this invention based on a multi-agent cooperative structure is similar to the Vicsek system. The movement direction of an individual is updated based on the average movement direction of its neighbors, considering only information interaction within the local neighborhood. To ensure that the decision-making behavior of the agent and its interacting neighbors remains consistent, this invention designs the game in matrix evolutionary games to only involve nodes and neighbors playing according to predetermined rules, or guides agents to cooperate locally through reward signals in interception games. The behavior of agent i can be determined by local rules. By definition, this rule depends on the state of i. The status of its neighbors That is, the control input of agent i .

[0036] When the set of neighbors of agent i at time t is determined through local sensing and interaction modules or other communication protocols... Then, the individual immediate reward for agent i after performing the action. Based on the reward redistribution mechanism set up in this invention, each neighbor is ensured All can receive collaborative rewards from i. As shown in formula (3): (3) in, A pre-set reward sharing coefficient is used to control the strength of collaborative incentives. Therefore, for agent i, the total reward it receives at time t is... Self-reward Collaborative rewards from all neighbors in the neighbor set The sum is shown in formula (4):

[0037] (4) Formula (4) is the mathematical core of this invention for driving local cooperative emergence. When agent i makes an action that is beneficial to the whole but may be risky to itself, its neighbor j will also receive a positive reward, thereby encouraging neighbor j to choose an action that can protect or cooperate with agent i, thus forming an evolutionary basis for mutual assistance and cooperation at the group level.

[0038] Based on this, the individual immediate reward after agent i performs an action. When the collaborative decision-making task is collaborative interception, the drones can be guided to evolve from basic survivability to advanced tactical behavior through the weighted average of six key indicators: survival time reward, game time reward, attack induction reward, location boundary reward, and defense distance reward, and through progressive rewards. This ensures that the intelligent agent develops in a balanced manner in multi-objective optimization, improves overall combat effectiveness, and thus obtains a composite reward mechanism that integrates spatiotemporal characteristics and tactical intentions, thereby guiding the drone swarm to learn complex offensive and defensive strategies.

[0039] Specifically, survival time reward The aim is to establish a basic survival awareness and encourage drones to avoid unnecessary sacrifices in combat. To avoid reward instability in the early stages of training, this invention chooses the hyperbolic tangent function tanhx as the reward function, ensuring that the reward value converges smoothly within a certain range. A gain coefficient is used to control the rate of reward growth, ensuring that the long-term survival of drones yields higher rewards over time. In the early stages of combat, priority is given to ensuring unit survival rate, providing more strategic options and resource reserves for subsequent tactical execution. In this way, the system ensures that units do not fall into a passive situation due to premature losses, and survival time rewards are provided. The calculation is shown in formula (5):

[0040] (5) in, The drone's survival time can be measured in steps. This is the gain coefficient used to control the rate of reward growth.

[0041] Game time rewards By setting a time threshold Step by step, simulating the time pressure of real battlefields, this approach balances the conflict between "cautious defense" and "rapid advance" tactics, avoiding unnecessary delays and ensuring timely mission completion. For example, when... As rewards gradually decrease over time, they encourage quick decision-making and task execution. The penalty for exceeding the time limit gradually increases to prevent delays in tactical execution. This design encourages drones to make quick decisions in the early stages of combat and avoids unnecessary delays in the later stages, ensuring mission completion on time and rewarding players with game time. The calculation is shown in formula (6):

[0042] (6) in, The current number of moves in the game. This represents the threshold or number of steps required to complete the task. This is the gain coefficient used to control the rate at which rewards and punishments change over time.

[0043] Induced rewards With punishment It is primarily based on classic tactical principles of air combat, namely angle advantage and close-range engagement. This is achieved by setting angle thresholds. and distance constraints This encourages drones to attack from the enemy's flanks and avoids direct confrontation. Specifically, when the angle is greater than... And the distance is less than When the angle is smaller than 100 degrees, give positive rewards to encourage flanking maneuvers; when the angle is smaller than 100 degrees, give positive rewards to encourage flanking maneuvers ... or equal to and the distance is less than At the same time, negative penalties are imposed to prevent "suicide attacks," thereby effectively guiding drones to avoid direct head-on collisions, leveraging the advantages of flanking maneuvers, and inducing rewards. With punishment The calculation is shown in formula (7):

[0044] (7) in, The relative angle between the drone and the enemy aircraft. The distance between the drone and the enemy aircraft This is the positive reward given when the conditions are met. This is the negative penalty imposed when the conditions are met.

[0045] Location boundary reward The aim is to ensure drones remain within their effective operational area, preventing units from straying out of combat zones and thus reducing the risk of tactical vulnerabilities. A strong penalty mechanism is implemented, imposing penalties for actions that deviate too far from the boundary. The penalty is significantly higher than other rewards; however, within the effective combat area, a small positive reward incentivizes the drone to maintain its proper position on the battlefield, effectively preventing the drone from exceeding the effective combat area due to exploration, thus maintaining mission stability and strategic depth. Position boundary rewards... The calculation is shown in formula (8):

[0046] (8) in, This is the current location of the drone. and The boundaries of the battlefield, for example, can be set. It is 0.5. It is 24. This is the penalty value when the drone goes beyond the boundary. A small positive reward is given when the drone is in an effective battlefield area.

[0047] Defense distance bonus This reward system incentivizes drones to respond to threats from nearby enemy aircraft through an inverse design. The reward increases as the distance to the enemy aircraft decreases, encouraging drones to prioritize defensive maneuvers against approaching enemy aircraft. The absence of a specific threshold makes the reward effective across the entire domain, but it is particularly prominent when approaching enemy aircraft. This reward embodies a "proximity deterrence" strategy, prompting drones to proactively approach enemy aircraft and reduce their maneuverability, thereby enhancing tactical initiative and deterrence. The defensive distance reward... The calculation is shown in formula (9):

[0048] (9) in, This refers to the minimum distance between the aircraft and the enemy aircraft.

[0049] Attack behavior reward The goal is to encourage drones to proactively attack enemies at opportune moments and reward successful attacks. Each time an attack occurs, the drone makes a decision based on its distance from the enemy aircraft, only attacking when the enemy is close enough and within attack range. To incentivize successful attacks, the attacker receives a higher reward when the enemy is defeated, while teammates receive appropriate rewards based on their distance from the attacker and their tactical coordination. This reward design not only encourages individual drones to execute effective attacks but also promotes teamwork, rewarding teammates who provide support when the attacker successfully defeats the enemy. And the rewards given to teammates The calculations are shown in formulas (10) and (11) respectively:

[0050] (10) (11) in, The distance between the drone and the enemy aircraft. This represents the effective attack range of the drone, expressed in distance. To store the reward for each drone, a drone that successfully attacks will receive a reward. To acquire teammates who are closest to the current drone, reward those teammates accordingly.

[0051] Based on this, the present invention embeds the above-mentioned reward-sharing mechanism into the standard deep Q-learning framework, resulting in the following: Figure 4 The diagram illustrates a regional collaborative deep Q-learning algorithm. Each agent possesses a structurally identical but independent deep Q-network, and each deep Q-network includes an online Q-network. and a target Q network .

[0052] During the construction of experience tuples, each agent, based on its current decision-making time and its position, motion, and other state information relative to other agents and the target, determines the Q-value for each action through its own online Q-network, and then selects the action to be performed. After each agent completes the action to be performed at the current decision-making time, the simulation environment is updated. Based on whether the interception was successful or not, or the change in distance relative to the target, the agent's individual immediate reward after taking the action is calculated. This reward is combined with the individual immediate rewards of the agent's neighbors after the environment update to determine the agent's cooperative reward or allocation reward, thus determining the total reward after the agent takes the action. Afterwards, each agent combines its position, velocity, and other state information before taking the action, the action taken, the total reward after taking the action, and the updated position, velocity, and other state information after taking the action into an experience tuple for that moment, and stores them in the independent experience replay pools of each agent.

[0053] Step 13: Calculate the Q-learning target value of the agent based on the empirical tuple. Calculate the loss function using the Q-learning target value and the Q-values ​​corresponding to the actions in the empirical tuple. Optimize the parameters of the agent's deep Q-network based on the loss function calculation results to obtain the optimized deep Q-network. Determine the multi-agent cooperative interception decision model using the optimized deep Q-network of each agent.

[0054] Step 13 includes: Step 131: Construct an experience replay pool from the experience tuples obtained by the agent at multiple time points through the online Q-network; sample from the experience replay pool; determine the Q learning target value of the agent through the target Q-network based on the total reward in the sampling results.

[0055] Step 132: Calculate the Q value corresponding to the action in the sampling results using an online Q network; calculate the mean squared error loss function using the agent's Q learning target value and Q value.

[0056] Step 133: Update the parameters of the online Q-network using gradient descent based on the mean squared error loss function to obtain the optimized online Q-network; then, perform a soft update on the parameters of the target Q-network based on the optimized online Q-network parameters to obtain the optimized target Q-network.

[0057] Specifically, once the experience replay pool for each agent is constructed, samples are periodically taken from the experience replay pool, and the target value is determined through the agent's target Q-network. As shown in formula (12): (12) in, As a discount factor, The next state. Due to the target value Not only including The target Q-network outputs the Q-learning target value based on the sampling results, which also includes the total reward calculated in the experience tuple. The total reward also includes the benefits that the neighbors receive due to the agent's action through a reward sharing mechanism. When evaluating the value of an action, the agent can consider not only its own future benefits but also the benefits that its behavior may bring to its neighbors. Then, the difference between the target value and the online Q-network output in the sampling results is calculated using the mean squared error loss function, as shown in formula (13):

[0058] (13) Here, E represents the mean. Then, the parameters of the online Q-network are calculated using gradient descent. Perform the update, and do so via soft update. The parameters of the target Q-network are gradually synchronized in a stepwise manner. The process involves updating and repeating the training multiple times until the deep Q-network of each agent converges, thus obtaining a decision model for collaborative interception decisions by multiple agents. It is important to emphasize that during the training of the deep Q-network, this invention employs a fully distributed training method. That is, all agents execute the perception-decision-reward sharing and learning update loop in parallel. Agents do not share neural network parameters, gradients, or policies; collaboration is achieved solely through a reward allocation mechanism, thereby reducing communication overhead during model training.

[0059] In summary, the construction method provided by this invention allows each agent to make decisions and learn entirely based on local information, without the need to share strategies, parameters, or global information. Collaboration is achieved only through indirect coupling via environmental rewards, resulting in extremely low system communication requirements, strong scalability, and true distribution with low communication overhead. Furthermore, because this invention incentivizes agents to adopt more collaborative and efficient interception paths, it avoids ineffective individual chasing and energy dissipation, effectively reducing system energy consumption. Moreover, through a local reward sharing mechanism, agents can quickly perceive the additional positive feedback from collaborative behavior, accelerating the learning process of cooperative strategies and significantly improving collaborative efficiency and convergence speed. The distributed architecture of this invention avoids single points of failure. The learning mechanism based on local interaction enables the cluster to have strong adaptability to agent failure and environmental changes. Even if some agents fail, the remaining agents can still adjust based on existing collaborative strategies. Furthermore, the simple local reward sharing rules adopted in this invention enable the spontaneous emergence of advanced tactical behaviors such as division of labor, decoy tactics, and encirclement at the group level without complex programming at the individual level, achieving efficient collaborative interception by multiple agents.

[0060] Example 2 Based on Example 1, this example uses Python 3.9 and the PyTorch 1.12 framework to build as follows. Figure 5 The simulation platform shown is a two-dimensional continuous space interception game of red-blue confrontation. N red defending agents need to protect a high-value target located in a fixed area, and M blue attacking agents attempt to break through the defense and attack the target. The defense area is 4×4km, the game area is 48×36km, the number of defending agents is 12 or 15, and the number of attacking agents is 20 or 30. Each agent has a perception radius of 5km, and the reward sharing coefficient k contributed by other agents to this agent is 0.2.

[0061] Furthermore, this embodiment models the cooperative interception task as a decentralized partially observable Markov decision process (Dec-POMDP). In the Dec-POMDP framework, each agent's decision-making process depends on its local observations and action choices. The state transitions and reward functions of the entire system are combined and represented as follows: ,in, This is the state space, containing the state information of all agents. Let be the action space, containing the set of possible actions for each agent i at each time t. Let be the state transition probability function, describing how the state transitions based on the joint actions of the agent. The reward function defines how the reward for each agent is calculated. This is a discount factor used to measure the impact of future rewards. For each agent i, the local observation space describes the environmental information it can perceive. For local observation functions, define the observation from the global state. Local observation of each agent The mapping.

[0062] In Dec-POMDP, the system's state space It contains the state information of multiple agents. At time t, the global state is... But intelligent agents Unable to obtain global state information, it can only perceive its own local environment. (Intelligent agent) Local observation It only reflects relevant information about agents in its neighborhood and is determined by the environmental state. and intelligent agents neighborhood set The decision is as shown in formula (14):

[0063] (14) in, It is a local observation function. Represents intelligent agents At any moment The observed neighborhood set is usually determined by the sensing radius. The maximum number of neighbors that can be perceived, k, and the observation angle φ are limited, and are defined as shown in formula (15): (15) in, It is an intelligent agent and The distance between them It is the orientation of the intelligent agent. From intelligent agents point to The vector, where φ is the observation angle. It is the radius of perception. It is the maximum number of neighbors that can be perceived. In ec-POMDP, each agent... Based on the current local observations at each time t Choose an action ,in It is an intelligent agent Action space. Combined actions. The state transitions that affect the system are shown in formula (16):

[0064] (16) Where is the state transition probability function, describing the transition from the current state... To the next state The transfer. And the intelligent agent. Instant rewards Based on the current state Intelligent agent actions and the new state after environmental transfer The calculation is performed as shown in formula (17):

[0065] (17) Furthermore, in this embodiment, the goal of all agents is to maximize the cumulative reward of each individual, which is optimized by discounting the reward, as shown in formula (18): (18) in, It is a discount factor, representing the degree to which future rewards are discounted. It is an intelligent agent At any moment The rewards obtained. In the process of maximizing their own rewards, each agent interacts with other agents, and eventually a collective behavior of maximizing their own rewards emerges.

[0066] In a two-dimensional continuous space interception game simulation platform for red-blue team competition, each agent possesses an independent neural network to learn how to take optimal action in a given state. Each agent aims to improve its performance by maximizing its own reward, but also receives limited information from its neighbors, particularly the reward for a successful attack. This setup encourages agents to not only optimize their own decisions but also indirectly influence their strategies through interactions with neighbors. For example, each agent's deep Q-network is a three-layer fully connected network, including an input layer, two hidden layers with 128 neurons each, and an output layer. The input is a normalized vector of the relative state of itself and its neighbors within its perception range, and the output is the Q-value corresponding to discrete actions such as acceleration, deceleration, left turn, right turn, and attack. During training, the learning rate is set to 0.001, the discount factor to 0.99, the experience replay pool capacity to 16384, the batch size to 32, and the target network soft update coefficient τ to 0.005; the exploration rate ε linearly decays from 1.0 to 0.01. The total training steps are 200,000 rounds.

[0067] Each agent's strategy is determined by its own neural network. Each agent in the system... At any moment The state is And select an action as a neural network The agent evaluates the expected reward of different actions using the Q-value output of its neural network. i.e., intelligent agent In state The following actions The Q value, where, It is an intelligent agent The neural network parameters. The neural network learns to make the Q value effectively approximate the optimal value, thereby guiding the agent to make the optimal decision. The agent's policy update adopts the DQN algorithm, and the update rule is shown in formula (19):

[0068] (19) in, It is an intelligent agent At any moment The rewards they receive are usually due to immediate feedback on their actions; It is a discount factor, indicating the importance of future rewards; It is an intelligent agent In the next state Choose the Q value for the optimal action.

[0069] Because in this model, the agent's policy is not only based on its own reward but also limited by its local observations. Each agent can observe the state and behavior of its neighbors and obtain rewards for successful attacks by its neighbors in certain tasks. Assume the agents... At any moment The neighborhood set is perceived as This includes all neighbors within the perception radius whose maximum number of neighbors is k. The reward for a successful attack on each neighbor affects the policy update of agent i. Under this mechanism, the Q-value of agent i depends not only on itself but also on some neighbor information. The policy is adjusted through a neural network to maximize the total reward of the individual agent.

[0070] In a red-blue game scenario with N=12 and M=20, the present invention is compared with the existing VDN method in terms of the average cluster reward and average power consumption as a function of the number of training rounds. The curves are shown below. Figure 6 and Figure 7 As shown. Among them, after approximately 50,000 rounds, as... Figure 6 As shown, the average reward of this invention consistently exceeds VDN, and as... Figure 7 As shown, the interception success rate after training with this invention reaches 87%, higher than the 72% of the VDN method, and the average task energy consumption of this invention is reduced by about 35%. In a red-blue game scenario with N=15 and M=30, the average cluster reward of this invention and the existing VDN method with the number of training rounds is as follows: Figure 8 As shown, the average reward of this invention also consistently outperforms the VDN model.

[0071] And, as Figure 9 A comparison of enemy and friendly loss rates when training scales are 30, 50, and 70, and... Figure 10 The comparison results of win rate and exchange ratio show that a model with stronger adaptability and higher win rate can also be obtained by first training under high pressure, such as training the defender with 70 attackers, and then testing in a low-pressure environment, such as testing with 30 attackers.

[0072] In addition, such as Figure 11 The comparison of enemy and friendly loss rates at decoupled inelastic boundaries, coupled inelastic boundaries, and coupled elastic boundaries, respectively, is shown. Figure 12 The comparison results of win rate and exchange ratio show that decoupling "motion decision" and "attack decision" at the network level, and combining "elastic boundary" conditions, can achieve better performance depending on the task focus, such as pursuing win rate or exchange ratio.

[0073] Example 3 This invention also provides a system for constructing a multi-agent cooperative interception decision model, comprising: The initialization module is used to initialize a deep Q-network for each agent. The input of the deep Q-network is the relative motion state of the agent with the interception target and other agents, and the output of the deep Q-network is the Q value corresponding to the agent performing multiple actions. The experience tuple construction module is used to determine the actions that multiple agents can independently perform at each decision time using a deep Q-network, and update the relative motion state of each agent based on its actions to obtain the relative motion state of each agent at the next decision time. According to preset interception rules, the individual immediate reward corresponding to the action performed by each agent is determined by its relative motion state at the next decision time. The total reward after the agent performs the action is determined by the individual immediate reward of the agent and the individual immediate rewards of neighboring agents, where neighboring agents are determined by the relative motion states of other agents relative to the agent at the next time. Each agent generates its experience tuple at the current decision time by combining its relative motion state at the current decision time, the action performed independently, the total reward, and its relative motion state at the next time. The model building module is used to calculate the Q-learning target value of the agent based on the empirical tuple. The loss function is calculated by using the Q-learning target value and the Q-values ​​corresponding to the actions in the empirical tuple. The parameters of the deep Q-network of the agent are optimized based on the loss function calculation results to obtain the optimized deep Q-network. The multi-agent cooperative interception decision model is determined by the optimized deep Q-network of each agent.

[0074] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a multi-agent cooperative interception decision model construction method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0075] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a multi-agent cooperative interception decision model construction method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0076] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] It should be noted that the above specific embodiments enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for constructing a multi-agent cooperative interception decision model, characterized in that, include: A deep Q-network is initialized for each agent, wherein the input of the deep Q-network is the relative motion state of the agent with respect to the interception target and other agents, and the output of the deep Q-network is the Q value corresponding to the agent performing multiple actions. At each decision-making moment, the deep Q-network determines the actions that multiple agents can independently execute at the current decision-making moment, and updates the relative motion state of each agent based on the actions of each agent to obtain the relative motion state of each agent at the next decision-making moment. According to the preset interception rules, the individual instant reward corresponding to the action independently executed by each agent is determined by the relative motion state of each agent at the next decision-making moment. The total reward after the agent executes the action is determined by the individual instant reward of the agent and the individual instant rewards of neighboring agents, wherein the neighboring agents are determined by the relative motion state of other agents relative to the agent at the next moment. Each agent generates the experience tuple of the agent at the current decision-making moment by taking the relative motion state corresponding to the current decision-making moment, the action independently executed, the total reward, and the relative motion state of each agent at the next moment. The Q-learning target value of the agent is calculated based on the empirical tuple. The loss function is calculated using the Q-learning target value and the Q-values ​​corresponding to the actions in the empirical tuple. The parameters of the deep Q-network of the agent are optimized based on the loss function calculation results to obtain the optimized deep Q-network. The multi-agent cooperative interception decision model is determined by the optimized deep Q-network of each agent.

2. The method for constructing a multi-agent cooperative interception decision model according to claim 1, characterized in that, After obtaining the relative motion state of each agent at the next decision moment, the following is also included: For each agent, other agents whose relative distance is within a preset range are considered as the agent's neighboring agents.

3. The method for constructing a multi-agent cooperative interception decision model according to claim 1, characterized in that, According to the preset interception rules, the individual immediate reward corresponding to the action independently performed by each agent is determined by the relative motion relationship of each agent at the next decision time, including: The individual instant reward for each agent is determined based on whether the interception of the target is successful or not, or the change in distance between each agent and the target.

4. The method for constructing a multi-agent cooperative interception decision model according to claim 1, characterized in that, The total reward for the agent after performing an action is determined by the agent's individual immediate reward and the individual immediate rewards of neighboring agents, including: Based on a preset reward sharing coefficient, the collaborative reward of an agent is determined by the individual instant rewards of neighboring agents; The total reward for the agent is determined by the collaborative reward and the individual immediate reward.

5. The method for constructing a multi-agent cooperative interception decision model according to claim 1, characterized in that, The deep Q-network includes an online Q-network and a target Q-network. The agent's Q-learning target value is calculated based on empirical tuples, including: The experience replay pool is constructed by using the experience tuples obtained from multiple moments of the agent through an online Q-network. Sample from the experience replay pool; determine the agent's Q-learning target value from the total reward in the sampling results through the target Q-network.

6. The method for constructing a multi-agent cooperative interception decision model according to claim 5, characterized in that, The loss function calculation, based on the Q-learning target value and the Q-values ​​corresponding to actions in the experience tuples, includes: The Q-values ​​corresponding to the actions in the sampling results are calculated using an online Q-network. The mean squared error loss function is calculated using the Q-learning target value and Q-value of the agent.

7. The method for constructing a multi-agent cooperative interception decision model according to claim 6, characterized in that, Based on the loss function calculation results, the parameters of the deep Q-network of the agent are optimized to obtain the optimized deep Q-network, which includes: The parameters of the online Q-network are updated using gradient descent based on the mean squared error loss function to obtain the optimized online Q-network. The parameters of the target Q-network are softly updated based on the optimized online Q-network parameters to obtain the optimized target Q-network.

8. A system for constructing a multi-agent cooperative interception decision model, characterized in that, include: An initialization module is used to initialize a deep Q-network for each agent, wherein the input of the deep Q-network is the relative motion state of the agent with the interception target and other agents, and the output of the deep Q-network is the Q value corresponding to the agent performing multiple actions respectively; The experience tuple construction module is used to determine, at each decision time, the actions independently executed by multiple agents through the deep Q-network, and update the relative motion state of each agent based on the actions of each agent to obtain the relative motion state of each agent at the next decision time; according to the preset interception rules, the individual instant reward corresponding to the action independently executed by each agent is determined by the relative motion state of each agent at the next decision time; the total reward after the agent executes the action is determined by the individual instant reward of the agent and the individual instant rewards of neighboring agents, wherein neighboring agents are determined by the relative motion state of other agents relative to the agent at the next time; and each agent generates its experience tuple at the current decision time by combining its relative motion state at the current decision time, the independently executed action, the total reward, and its relative motion state at the next time. The model building module is used to calculate the Q-learning target value of the agent based on the empirical tuple. The loss function is calculated by using the Q-learning target value and the Q-values ​​corresponding to the actions in the empirical tuple. The parameters of the deep Q-network of the agent are optimized based on the loss function calculation results to obtain the optimized deep Q-network. The multi-agent cooperative interception decision model is determined by the optimized deep Q-network of each agent.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method for constructing a multi-agent cooperative interception decision model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to execute the steps of the method for constructing a multi-agent cooperative interception decision model as described in any one of claims 1 to 7.

Images