Time-kill net generation method and system based on multi-agent reinforcement learning

Abstract

The application discloses a kind of time sequence killing net generation method and system based on multi-agent reinforcement learning, belong to artificial intelligence technical field.The application is aimed at the problem of resource allocation and multi-target attack success rate optimization in multi-task combat, adopts multi-agent deep reinforcement learning technology, generates dynamic time sequence killing net by constructing agent system for attack task and combining online training mechanism;By introducing the combat capability probability function of time-varying force in node and edge, the agent decision process is optimized to adapt to complex battlefield situation;Design the reward mechanism that integrates task execution time, priority and deadline, realize the cooperative scheduling among multi-agents, reduce the task missing rate;Online learning and strategy adaptive updating are supported simultaneously, avoid dimension disaster, have strong dynamic response capability and flexibility, so as to improve the overall effectiveness and attack success rate of combat system.

Description

Technical Field

[0001] This invention proposes a method and system for generating temporal kill networks based on multi-agent reinforcement learning, belonging to the field of artificial intelligence technology. Background Technology

[0002] Kill networks, as a type of military Internet of Things (IoT), organically combine battlefield information and resources such as reconnaissance nodes, operational decision-making nodes, strike nodes, and assessment nodes to form a redundant, agile, and complex mesh structure. This enables functions such as integrating operational resources, precision target strikes, and efficient multi-domain collaboration. Modern warfare is complex and ever-changing, with dynamic shifts in the battlefield environment, operational resource status, and target status. Furthermore, operations face multiple constraints, such as accurately describing the battlefield situation, rapidly adapting to battlefield changes, satisfying diverse operational constraints, and closing multiple kill chains. Therefore, generating kill networks has become a pressing challenge.

[0003] Current kill net generation methods primarily target static combat networks, neglecting the temporal attributes and dynamic changes of missions, combat resources, and the combat network itself. Application methods mainly include... Traditional heuristic algorithms such as genetic algorithms and particle swarm optimization are being replaced by deep reinforcement learning algorithms like DQN, PPO, and DDPG, which utilize reward mechanisms, as well as rule-based reasoning methods such as knowledge reasoning. Current research on temporal kill net modeling adds time attributes to traditional kill net structures by incorporating effective equipment time periods and effective connection time periods, further aligning them with real battlefield situations. Current methods primarily assess the timeliness of target engagement and target destructiveness, and apply multi-objective optimization algorithms, including non-dominated sorting genetic algorithms and multi-objective differential evolution algorithms, to balance these two aspects and form a temporal kill net strategy.

[0004] Wu Keyu et al. proposed a formal modeling approach for kill networks based on a stochastic controlled Petri net model to solve the problem of random concurrent target strikes. They solved the problem using deep reinforcement learning methods through an optimal strategy construction paradigm based on Markov decision processes (see "Kill Network Modeling and Optimal Control Methods for Stochastic Concurrent Tasks"). Si Guangya et al. proposed a method for constructing a simulation model of a multi-dimensional dynamic information network for command and control systems, as well as a method for constructing a command and control system effectiveness evaluation model based on hypernet theory (see "Research and Practice on Networked Modeling of Command and Control Systems"). Tan Yuejin et al. proposed a networked description and modeling method for weapon systems, where equipment nodes use functions to describe the influence relationships between equipment and the dynamic changes in equipment performance over time (see "Networked Description and Modeling Methods for Weapon Systems"). Xia Boyuan reviewed research on kill networks in "Mosaic Warfare" from five aspects: operational concepts, system modeling, combinatorial optimization, deep reinforcement learning, and multi-objective optimization (see "Research on Operational Loop Recommendation Methods for Kill Networks in Mosaic Warfare"). Yang Conglin's temporal kill net model adds the attribute of effective time to the nodes and edges of the kill net model, and solves the kill net model using a multi-objective optimization algorithm (see "Research on Modeling Method for Task Allocation Problem Based on Temporal Kill Net"). Zhang Hao constructed a three-layer combat system consisting of a communication network, a main station equipment network, and a support equipment network, and studied the cascading effect of edge failure in the combat system network. He also designed a method for generating combat system network disintegration strategies that consider edge failure (see "Research on Combat System Network Disintegration Strategies Considering Edge Failure").

[0005] Most existing kill net generation methods focus on relatively simple models, primarily integrating and modeling battlefield resources by abstracting combat units into network nodes. Node attributes mainly reflect the current node's specific category, behavioral capabilities, and execution time, neglecting much of the actual combat unit state and action attributes, such as action effective time periods and success probabilities. Existing temporal kill net models, building upon traditional kill net models, add temporal attributes to nodes and edges, define action effective time periods, and incorporate parameters such as node effective time periods, node capacity, and the number of tasks per node. This achieves a more accurate depiction of battlefield situation and resources compared to traditional models.

[0006] Existing research has the following specific drawbacks.

[0007] 1. Current combat network models still lack descriptions of the relationship between the capabilities of combat nodes and time, as well as descriptions of how edge connectivity changes over time.

[0008] 2. Existing kill net generation algorithms are mainly designed for traditional kill net structures, and they neglect the time-sensitivity of node capabilities during strategy selection. The equipment status of nodes can change dynamically over time and may be in different states, with different task processing capabilities in different states; most existing strategy generation methods are only applicable to static kill net structures.

[0009] 3. Traditional reinforcement learning suffers from the "curse of dimensionality" in multi-task policy generation; when building a single agent, its state space and action space both increase exponentially; and it cannot handle an uncertain number of tasks. Summary of the Invention

[0010] The purpose of this invention is to propose a method and system for generating temporal kill networks based on multi-agent deep reinforcement learning, aiming to achieve multi-mission combat resource allocation and optimize the success rate and latency of multi-target strikes in the construction of temporal kill networks. This invention constructs a combat network oriented towards strike missions, designs a multi-agent deep reinforcement learning method integrating online training mechanisms, and generates a temporal kill network with closed kill chains, thereby maximizing the benefits of mission real-time performance and strike success rate.

[0011] To achieve the above objectives, the present invention adopts the following technical solution.

[0012] A method for generating temporal kill networks based on multi-agent reinforcement learning includes the following steps: 1) Create nodes based on the attributes of combat units, and create edges connecting nodes based on the relationships between combat units to construct the combat network; 2) Create missions based on operational requirements, and construct a mission set from all pending missions; 3) Based on the combat network and task set, define the action space, state space and reward function of the agent, and construct a multi-agent reinforcement learning model; 4) During the training phase, a policy network and a value network are constructed for each agent in the multi-agent reinforcement learning model, and the policy network and value network of the agent are trained by the multi-agent deep deterministic policy gradient algorithm. 5) Use the trained multi-agent reinforcement learning model to perform actual tasks, and make action decisions for each agent when performing tasks through online inference.

[0013] Furthermore, in step 1), the attributes of the combat unit include its faction, unit category, number of waiting tasks, and combat capability probability, where the combat capability probability is the probability distribution of the combat unit's combat capability changing over time.

[0014] Furthermore, the relationship between combat units in step 1) includes connection categories and connectivity probabilities, where the connectivity probability is the probability distribution of the connectivity between combat units over time.

[0015] Furthermore, in step 2), each task in the task set consists of a target, a task completion deadline, and a task priority.

[0016] Furthermore, in step 3), the agent follows the following rules: When an agent completes the task of the current node, its action space is the node corresponding to the next subtask type, and the action reward is 0. When an agent is in the process of performing a task or waiting for a node to execute a task, its action space is empty and the action reward is 0. When the agent completes the current subtask, its action space is empty, and the reward value is calculated and the corresponding reward is provided.

[0017] Furthermore, the steps for calculating the reward value include: (1) If only the sub-task is completed but the target strike mission is not completed, the reward value is: in, For indication quantity, Indicates task The subtask was executed successfully. This indicates that the subtask failed to execute; This indicates the time taken from when the task reaches the current node until the task is completed or fails; and These are all constant terms used to adjust the reward values; (2) If the target strike mission is completed, the reward value is: in, For indication quantity, Indicates task The mission to strike the target was successfully completed. This indicates that the mission to strike the target has failed. Indicates task priority. Indicates the deadline for completing the task. Indicates the time when the task is completed. and These are all constant terms used to adjust the reward values; (3) All agents share the same model reward, and the total reward at time t is: Where N represents the number of agents in the model; This represents the actual reward of agent i at time t, determined based on the current task state. or .

[0018] Furthermore, in step 4), constructing a policy network and a value network for each agent means constructing an actor network and a critic network, as well as a corresponding target actor network and a target critic network, for each agent.

[0019] Furthermore, step 4) involves training the agent's policy network and value network, which includes: In each training round, an action is generated for each agent based on the current policy, system state, and the agent's observation information, and a reward is calculated based on the execution result. Each agent's actions, rewards, and new states are stored in the experience replay area for subsequent training. For each agent, random samples are taken, and the expected value of the reward for the target action is calculated using the target network; The parameters of the critic network are updated by minimizing the loss of the expected value of the action reward; The actor network is updated based on the feedback provided by the critic network using the policy gradient method. Use a soft update method to update the parameters of the target actor network and the target critic network; Repeat the training process until the network model converges, meaning the agent can choose the optimal action based on the environmental state.

[0020] Furthermore, after making an action decision in step 5), the action, environmental state, and reward information of each agent are stored in the experience replay area in real time. The agent's strategy is updated by sampling historical data from the experience pool, and the updated strategy is used to continue to optimize the agent's decision-making process.

[0021] A temporal kill net generation system based on multi-agent reinforcement learning includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above method.

[0022] The present invention has achieved the following beneficial effects.

[0023] 1. Adapting to Complex and Ever-Changing Battlefield Situations: This invention further optimizes existing temporal kill net structures by adding probability functions to nodes and edges to describe the changes in combat capabilities of combat units over time. By introducing randomness, this invention enhances the robustness of agent decision-making, enabling it to better adapt to the dynamic environment of real combat.

[0024] 2. Meeting Real Combat Needs: This invention introduces priority and deadline parameters for missions and designs an action reward mechanism that integrates mission targets, mission deadlines, and mission priorities. By unifying action rewards among multiple agents, this invention achieves effective resource scheduling among agents and significantly reduces the mission deadline miss rate.

[0025] 3. Supports online learning and dynamic adjustment: This invention is based on a multi-agent reinforcement learning method, which can learn online from the actual battlefield situation and dynamically adjust the agent's strategy. Compared with traditional multi-objective optimization algorithms, the method of this invention has higher flexibility and can better adapt to dynamic and complex environments.

[0026] 4. Avoiding the curse of dimensionality and maximizing system benefits: This invention introduces a multi-agent deep reinforcement learning method and achieves policy planning and resource balancing among multiple agents by sharing action rewards. This method effectively avoids the curse of dimensionality caused by the exponential growth of the state space and dynamic space, thereby ensuring the maximization of system benefits. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the task scheduling and execution method based on multi-agent reinforcement learning in this embodiment.

[0028] Figure 2 This is a schematic diagram of the combat network in the embodiment. Detailed Implementation

[0029] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings.

[0030] This invention specifically discloses a method for generating temporal kill networks based on multi-agent reinforcement learning, the brief process of which is as follows: Figure 1 As shown, the specific steps are as follows.

[0031] 1. Construct an operational network, such as Figure 2 As shown.

[0032] (1) Create nodes based on the specific attributes of the combat unit.

[0033] The attributes specifically include faction affiliation, unit type, number of pending missions, and combat capability probability, represented as follows: ; Among them, Camp represents the faction. When Camp equals 1, it represents the enemy faction, and when Camp equals 0, it represents our faction. nodeType represents the unit type, which represents the type of mission that the combat unit can undertake in actual combat. There are four unit types: when nodeType=S, it means that the current node type is a reconnaissance node; when nodeType=D, it means that the current node type is a decision node; when nodeType=A, it means that the current node type is an attack node; and when nodeType=E, it means that the current node type is an evaluation node. waitingTasks refers to the number of waiting tasks, indicating the number of tasks currently waiting to be executed on this node; It is a joint probability function that represents the probability distribution of a node’s combat capability over time, where t represents time and c represents the node’s combat capability. This represents the probability that the node has a capability of c at time t.

[0034] (2) Create edges based on the relationships between different combat units and connect the nodes.

[0035] The relationships between combat units, i.e., edge attributes, include connection category and connectivity probability, represented as follows: ; Where Type represents the connection type of the edge; It is a joint probability function that represents the probability distribution of edge connectivity over time; where t represents time and I represents edge connectivity, where I=1 represents connectivity and I=0 represents disconnectivity. This represents the probability that the node has connectivity I at time t.

[0036] 2. Create missions based on operational requirements. Missions are represented as follows: .

[0037] in, Indicates the target of task i; Indicates the deadline for completing the task; The priority of the task is represented by a positive integer. A value of 1 indicates the highest priority, with the priority decreasing sequentially.

[0038] The tasks to be performed are grouped into a task set S. Assume there are N tasks in S. .

[0039] 3. Based on the combat network and mission set, define the action space, state space, and reward function of the agent, and construct a multi-agent reinforcement learning model.

[0040] If the agent has completed the task at the current node, the agent can select the corresponding type of node to connect to the current node based on the type of the next subtask to be performed. This is the agent's action space at that moment, and the action reward is 0.

[0041] If the agent is currently in the process of performing a task or waiting for a node to execute a task, then the agent's action space is empty at that moment, and the action reward is 0.

[0042] If the agent is at the moment of subtask completion, then the agent's action space is empty at that moment, and the action reward is: in, For indication, when When the value is 1, it represents a task. The subtask was executed successfully; when A value of 0 indicates that the subtask failed to execute; This represents the time elapsed from when the task reaches this node until the task is completed or fails. and All are greater than 0, representing constant terms for adjusting reward values.

[0043] When the target is successfully hit, the reward provided by agent i to the system is: in, For indication, when When the value is 1, it represents a task. The subtask was executed successfully; when A value of 0 indicates that the subtask failed to execute. This represents the priority of the task, expressed as a positive integer. Indicates the deadline for completing the task. and All of these are constant terms used to adjust the reward values. This marks the moment the task is completed.

[0044] When the target is successfully attacked or the attack fails and there are no target nodes to attack, the agent terminates its action.

[0045] To maximize overall benefit and avoid agents choosing to maximize individual benefit at the expense of system benefit, all agents share the same overall reward. The overall reward at time t is: in, The reward provided by the system to agent i at time t is determined based on the current task state. or .

[0046] 4. The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is used to train the multi-agent reinforcement learning model.

[0047] During training, the policy network and value network of the model are constructed. That is, for each agent, an actor network and a critic network are constructed, as well as a corresponding target actor network and a target critic network. The actor network and the corresponding target actor network constitute the policy network, and the critic network and the corresponding target critic network constitute the value network.

[0048] The following describes the process of training a multi-agent reinforcement learning model.

[0049] (1) Generate actions and record experience: In each round of training, actions are generated for each agent based on the current policy, system state and agent observation information; after the generated actions are executed, the corresponding new state, reward and related environmental information are recorded and stored in the experience playback area.

[0050] (2) Random sampling experience: Randomly sample a batch of training data from the experience replay area; for each sampled data, use the target network to calculate the expected reward value of the target action as a reference for model update.

[0051] (3) Update network parameters: Use the target reward expectation value of the sampled data to minimize the expected value loss function and update the corresponding critic network parameters; use the policy gradient method to update the corresponding actor network parameters.

[0052] (4) Soft update target network: Using the soft update method, gradually adjust the parameters of the target actor network and the target critic network so that they gradually approach the parameter values ​​of the current actor network and critic network, ensuring the stability of training and ending the current training round.

[0053] (5) Iterative training: Repeat the above training process until the convergence condition is met.

[0054] 5. During the actual deployment phase, based on the actual combat scenario, a pre-trained multi-agent reinforcement learning model is used to generate specific action decisions for each strike mission through online inference. After executing the generated actions, the corresponding actions, environmental states, rewards, and other information are recorded and stored in the experience replay area. Based on the data accumulated in the experience replay area, the reinforcement learning model's strategy is dynamically adjusted and updated to ensure that the model can adapt to changes in the actual combat environment and optimize mission decision-making effectiveness.

[0055] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A method for generating temporal kill networks based on multi-agent reinforcement learning, characterized in that, Includes the following steps: 1) Create nodes based on the attributes of combat units, and create edges connecting nodes based on the relationships between combat units to construct the combat network; 2) Create missions based on operational requirements, and construct a mission set from all pending missions; 3) Based on the combat network and task set, define the action space, state space and reward function of the agent, and construct a multi-agent reinforcement learning model; 4) During the training phase, a policy network and a value network are constructed for each agent in the multi-agent reinforcement learning model, and the policy network and value network of the agent are trained by the multi-agent deep deterministic policy gradient algorithm. 5) Use the trained multi-agent reinforcement learning model to perform actual tasks, and make action decisions for each agent when performing tasks through online inference.

2. The method as described in claim 1, characterized in that, In step 1), the attributes of the combat unit include its faction, unit category, number of waiting tasks, and combat capability probability, where the combat capability probability is the probability distribution of the combat unit's combat capability over time.

3. The method as described in claim 1, characterized in that, The relationships between combat units in step 1) include connection categories and connectivity probabilities, where connectivity probability is the probability distribution of connectivity between combat units over time.

4. The method as described in claim 1, characterized in that, Each task in the task set in step 2) consists of a target, a task completion deadline, and a task priority.

5. The method as described in claim 1, characterized in that, In step 3), the agent follows the following rules: When an agent completes the task of the current node, its action space is the node corresponding to the next subtask type, and the action reward is 0. When an agent is in the process of performing a task or waiting for a node to execute a task, its action space is empty and the action reward is 0. When the agent completes the current subtask, its action space is empty, and the reward value is calculated and the corresponding reward is provided.

6. The method as described in claim 5, characterized in that, The steps for calculating the reward value include: (1) If only the sub-task is completed but the target strike mission is not completed, the reward value is: in, For indication quantity, Indicates task The subtask was executed successfully. This indicates that the subtask failed to execute; This indicates the time taken from when the task reaches the current node until the task is completed or fails; and These are all constant terms used to adjust the reward values; (2) If the target strike mission is completed, the reward value is: in, For indication quantity, Indicates task The mission to strike the target was successfully completed. This indicates that the mission to strike the target has failed. Indicates task priority. Indicates the deadline for completing the task. Indicates the time when the task is completed. and These are all constant terms used to adjust the reward values; (3) All agents share the same model reward, and the total reward at time t is: Where N represents the number of agents in the model; This represents the actual reward of agent i at time t, determined based on the current task state. or .

7. The method as described in claim 1, characterized in that, Step 4) involves constructing a policy network and a value network for each agent, which means constructing an actor network and a critic network, as well as a corresponding target actor network and a target critic network, for each agent.

8. The method as described in claim 7, characterized in that, Step 4) involves training the agent's policy network and value network, which includes: In each training round, an action is generated for each agent based on the current policy, system state, and the agent's observation information, and a reward is calculated based on the execution result. Each agent's actions, rewards, and new states are stored in the experience replay area for subsequent training. For each agent, random samples are taken, and the expected value of the reward for the target action is calculated using the target network; The parameters of the critic network are updated by minimizing the loss of the expected value of the action reward; The actor network is updated based on the feedback provided by the critic network using the policy gradient method. Use a soft update method to update the parameters of the target actor network and the target critic network; Repeat the training process until the network model converges, meaning the agent can choose the optimal action based on the environmental state.

9. The method as described in claim 1, characterized in that, After making an action decision in step 5), the action, environmental state, and reward information of each agent are stored in the experience replay area in real time. The agent's policy is updated by sampling historical data from the experience pool, and the updated policy is used to continue to optimize the agent's decision-making process.

10. A temporal kill network generation system based on multi-agent reinforcement learning, characterized in that, It includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method according to any one of claims 1-9.

Images