A Multi-Agent Distributed Cooperative Decision-Making Method for Air-Sea Confrontation

Abstract

This invention relates to a multi-agent distributed collaborative decision-making method for air-sea combat, belonging to the field of artificial intelligence. The method includes steps such as operational scenario editing, neural network architecture design, agent state space design, action space design, reward / penalty function design, model parameter training, and algorithm testing. This invention proposes a multi-agent distributed collaborative decision-making method for air-sea combat, using the air-sea combat battlefield environment as the operational background. Based on a "centralized training-distributed execution" multi-agent distributed collaborative decision-making framework, it achieves multi-agent distributed collaborative autonomous decision-making in complex and dynamic battlefield environments.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a multi-agent distributed collaborative decision-making method for air-sea combat. Background Technology

[0002] With the continuous advancement of science and technology, reinforcement learning has become a research hotspot in the modern era. It has a wide range of application needs and important theoretical significance in both military and civilian fields. Currently, reinforcement learning has been widely applied in various military application systems, intelligent robots, drones, e-sports, and autonomous driving.

[0003] As reinforcement learning technology is applied in increasingly complex military environments, the number of agents requiring control is growing, and the computational burden of reinforcement learning increases dramatically with the number of agents. Single-agent reinforcement learning can no longer solve decision-making and control problems in large-scale scenarios, thus giving rise to multi-agent reinforcement learning algorithms. Compared to single-agent algorithms, multi-agent reinforcement learning algorithms can apply artificial intelligence to more complex scenarios that are closer to human society, and can also demonstrate excellent collaborative control capabilities when facing high-level task requirements. However, because multi-agent reinforcement learning algorithms require modeling multiple agents, each agent can only observe local situational information around it and cannot obtain global situational information about the battlefield, which is very disadvantageous for decision-making. Summary of the Invention

[0004] (a) Technical problems to be solved The technical problem this invention aims to solve is how to provide a multi-agent distributed collaborative decision-making method for air-sea confrontation, in order to address the problem that single-agent reinforcement learning can no longer solve decision-making and control problems in large-scale scenarios, and that multi-agent reinforcement learning algorithms cannot obtain global situational information of the battlefield.

[0005] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a multi-agent distributed cooperative decision-making method for air-sea confrontation, which includes the following steps: Step 1: Editing the Operational Scenario Based on a certain simulation platform, a combat scenario was designed, and the content included the initialization information of the red and blue combat units, the simulation time, the single-step simulation step size, the launch control authorization of various weapons of the red and blue camps, the weather conditions of the combat area, the combat plan of the blue team's weapons and equipment, and the conditions for determining the victory of the red and blue teams. Step Two: Motion Space Design Based on existing combat scenarios and considering the tactical actions that the intelligent agents can execute in the simulation platform, the action space of the intelligent agents is designed. The executable actions of the intelligent agents are defined as movement, attack, return to base, and idle rotation. To establish the mapping relationship between the intelligent agent actions and the neural network output, the movement action is defined as a one-dimensional one-hot vector with a dimension of 4; the attack action is defined as a one-dimensional one-hot vector with a dimension equal to the number of enemy air defense equipment; the return to base action is defined as a one-dimensional vector with a dimension of 1; and the idle rotation action is defined as a one-dimensional vector with a dimension of 1. All four types of actions are 0-1 discrete variables, where 1 represents the current action being executed and 0 represents the current action not being executed. Step 3: Design of the agent's state space; Based on the established action space design scheme of the intelligent agent, and combined with the situation information interface provided by the simulation platform, the state space of the intelligent agent is designed. Based on the decision framework of "centralized training-distributed execution", two state space representation methods are designed for global situation observation during training and local situation observation during execution. Local situation observation includes three types: position observation relative to red and blue equipment, state information observation of the intelligent agent, and attack condition observation of mounted weapons. All three types of state observation information adopt a continuous one-dimensional vector space design and are normalized before being input into the intelligent agent decision network. Global situation observation includes two types: absolute position observation of red and blue equipment and state observation of all intelligent agents. Both types of observation adopt a continuous one-dimensional vector space design and are normalized before being input into the intelligent agent auxiliary training network. Step 4: Neural Network Architecture Design Based on the design scheme of the action space and state space of the given agents and the decision architecture of "centralized training-distributed execution", a neural network architecture for a multi-agent distributed collaborative decision-making method for air-sea confrontation is designed. The neural network architecture consists of two types of networks: one is the independent behavior control network DRQN ​​for each agent, and the other is the global action evaluation network MIX used in the training phase. The DQRN network structure uses the agent's local situational awareness and the previous action. , [This refers to] network input, combined with the gating unit output from the previous time step. After calculation by the gated loop unit module and two layers of multilayer perceptron modules, the reward vector of all possible actions performed by the agent at the current moment is output. and intermediate information And select the maximum Q value based on the Q-maximization strategy. The corresponding action is used as the prediction output. The action prediction output of this network does not depend on the situational observation of other agents and can independently carry out distributed decision-making and control. The MIX network structure uses global situational information. and the DRQN ​​network output values ​​of each agent [ As input, the data is processed by a parameter-trainable hybrid network, which outputs the reward value of the joint action of each agent at the current time. This network can analyze and evaluate the decision quality of joint actions based on current global situation observations, and guide the improvement of action decision-making in intelligent agent networks; DRQN and MIX networks are tightly coupled based on input and output information, forming the network architecture Qmix. During the training phase, parameters are trained based on the Qmix architecture. After training, the Mix network is discarded, and only the DRQN ​​network is used for agent decision control. Both DRQN ​​and MIX networks update parameters using the Adam gradient descent algorithm until the model converges. Step 5: Design the reward and punishment function; Based on the established action space design scheme and combined with the judgment conditions for the Red Team's victory, a reward and punishment function is designed. The reward and punishment function design is to give a score evaluation to the joint actions of the agent's control decision under the current battlefield situation. The absolute value of the total reward in a single round is controlled to be around 1, that is, the reward and punishment function is normalized. The reward and punishment function R is divided into four categories: distance reward and punishment, attack reward and punishment, return reward and punishment, and mission completion reward and punishment. Among them, the mission completion reward and punishment value is the main reward and punishment value, whose absolute value is significantly greater than the first three items, and this value is positive. Step Six: Model Parameter Training; Based on the established neural network structure design, the model parameters are trained by combining the simulation platform interface and the computer hardware and software environment. Step 7: Algorithm Testing; The trained model is applied to a simulation environment to test its distributed collaborative decision-making capability. Based on the "centralized training - distributed execution" decision architecture, only the agent decision network DRQN ​​needs to be tested. The testing capabilities include two aspects: First, based on a predetermined combat scenario, the agent performs several rounds of autonomous decision-making simulations on the simulation platform, and its win rate is statistically analyzed to see if it meets the target. If the win rate meets the expected target, the test is passed; otherwise, the algorithm is improved to address the problems in the agent's decision-making process, and the above steps are repeated until the test is passed. Second, the model's generalization ability is tested. The scenario environment for algorithm testing varies to different degrees from the training combat scenario to test whether the agent can still perform well when the battlefield environment changes. The testing process still involves the agent performing several rounds of autonomous decision-making simulations on the simulation platform, and its win rate is statistically analyzed to see if it meets the target. If there are abnormal performances in some scenarios, the algorithm needs to be improved to enhance the agent's generalization ability.

[0006] (III) Beneficial Effects This invention proposes a multi-agent distributed collaborative decision-making method for air-sea confrontation. Taking the air-sea confrontation battlefield environment as the combat background, it realizes multi-agent distributed collaborative autonomous decision-making in complex dynamic battlefield environments based on the multi-agent distributed collaborative decision-making framework of "centralized training-distributed execution". Attached Figure Description

[0007] Figure 1 This is a flowchart of the multi-agent distributed collaborative decision-making process for air-sea combat in this invention. Figure 2 This is a flowchart of the model parameter training process for this invention; Figure 3 This is a diagram of the neural network architecture of the present invention, wherein the left side is the Mix network architecture; the middle side is the Qmix network architecture; and the right side is the DRQN ​​network architecture. Detailed Implementation

[0008] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0009] This invention designs a multi-agent distributed collaborative decision-making method for air-sea confrontation. Taking the air-sea confrontation battlefield environment as the combat background, it constructs a multi-agent fighter jet based on deep reinforcement learning technology and proposes a multi-agent distributed collaborative decision-making framework of "centralized training-distributed execution" to realize the technical approach of multi-agent distributed collaborative decision-making in complex and dynamic battlefield environments.

[0010] This paper proposes a multi-agent distributed collaborative decision-making method for air-sea combat. Using the air-sea combat battlefield environment as the operational background, it constructs multi-agent fighter jets based on deep reinforcement learning technology and adopts a "centralized training-distributed execution" multi-agent distributed collaborative decision-making framework to achieve distributed collaborative autonomous decision-making among multi-agents in complex and dynamic battlefield environments. The multi-agent distributed collaborative decision-making method for air-sea combat includes steps such as: operational scenario editing, neural network architecture design, agent state space design, action space design, reward / penalty function design, model parameter training, and algorithm testing. The method steps are as follows: Figure 1 As shown.

[0011] Step 1: Editing the Operational Scenario Based on a certain simulation platform, a combat scenario was designed, and the content included the initialization information of the red and blue combat units (unit type, unit quantity, platform mounting scheme, deployment location, etc.), simulation simulation time, single-step simulation step length, launch control authorization of various weapons of the red and blue camps, meteorological conditions such as rainfall, wind speed, and sea state in the combat area, combat plans of the blue team's weapons and equipment, and the conditions for determining the victory of the red and blue teams.

[0012] Step Two: Motion Space Design Based on existing operational scenarios and considering the tactical actions that intelligent agents can execute in a simulation platform, the action space of the agents is designed. The executable actions of the agents are defined as movement, attack, return to base, and idle rotation. To establish the mapping between agent actions and neural network outputs, the movement action is defined as a one-dimensional one-hot vector with dimension 4; the attack action is defined as a one-dimensional one-hot vector with dimension equal to the number of enemy air defense equipment; the return to base action is defined as a one-dimensional vector with dimension 1; and the idle rotation action is defined as a one-dimensional vector with dimension 1. All four types of actions are 0-1 discrete variables, where 1 represents the current action being executed and 0 represents the current action not being executed.

[0013] Step 3: Design the state space of the agent.

[0014] Based on the established action space design scheme of the intelligent agent, and combined with the situation information interface provided by the simulation platform, the state space of the intelligent agent is designed. Based on the "centralized training-distributed execution" decision framework, two state space representation methods are designed: global situation observation during training and local situation observation during execution. Local situation observation includes three categories: position observation relative to red and blue equipment, state information observation of the intelligent agent, and attack condition observation of mounted weapons. All three types of state observation information adopt a continuous one-dimensional vector space design and are normalized before being input into the intelligent agent decision network. Global situation observation includes two categories: absolute position observation of red and blue equipment and state observation of all intelligent agents. Both types of observation adopt a continuous one-dimensional vector space design and are normalized before being input into the intelligent agent auxiliary training network.

[0015] Step 4: Neural Network Architecture Design Based on the design scheme of the action space and state space of the given agents and the decision-making architecture of "centralized training-distributed execution", a neural network architecture for a multi-agent distributed collaborative decision-making method for air-sea combat is designed. The neural network architecture consists of two types of networks: one is a DRQN ​​(Dedicated Decision-Making Network) for each agent, and the network architecture is as follows. Figure 3 As shown in c), another type is the global action evaluation network MIX used during the training phase, with the network architecture as follows: Figure 3 As shown in a).

[0016] The DQRN network structure uses the agent's local situational awareness and the previous action. , [This refers to] network input, combined with the gating unit output from the previous time step. After calculation by the gated loop unit module and two layers of multilayer perceptron modules, the reward vector of all possible actions performed by the agent at the current moment is output. and intermediate information And select the maximum Q value based on the Q-maximization strategy. The corresponding action is used as the prediction output. Since the action prediction output of this network does not depend on the situation observation of other agents, it can independently carry out distributed decision control. The MIX network structure uses global situational information. and the DRQN ​​network output values ​​of each agent [ As input, the data is processed by a parameter-trainable hybrid network, which outputs the reward value of the joint action of each agent at the current time. This network can analyze and evaluate the decision quality of joint actions based on current global situation observations, and guide the improvement of action decision-making in agent networks.

[0017] DRQN and MIX networks are tightly coupled based on input and output information, and together they form the network architecture Qmix adopted in this paper. The network architecture is as follows: Figure 3 As shown in b), the training phase uses the Qmix architecture for parameter training. After training, the Mix network is discarded, and only the DRQN ​​network is used for agent decision control. Both the DRQN ​​and MIX networks update their parameters using the Adam gradient descent algorithm until the model converges.

[0018] Step 5: Design the reward and punishment function.

[0019] Based on the established action space design scheme and the victory criteria for the Red Team, a reward and penalty function is designed. The reward and penalty function design assigns scores to the joint actions of the agent's control decisions under the current battlefield situation. Since neural networks have a strong ability to fit output values ​​between [-1, +1], the absolute value of the total reward per round in the algorithm design is controlled to be around 1, i.e., the reward and penalty function is normalized. The algorithm's reward and penalty function R is divided into four categories: distance reward and penalty, attack reward and penalty, return reward and penalty, and mission completion reward and penalty. Among them, the mission completion reward is the main reward and penalty value, whose absolute value is significantly greater than the first three items, and this value is positive.

[0020] Step 6: Model parameter training.

[0021] Based on the established neural network structure design, and combined with the simulation platform interface and computer hardware and software environment, model parameters are trained. The process is as follows: Figure 2As shown. First, the computer GPU, deep learning framework modules, neural network parameters, training sample experience pool, etc., are initialized. Then, the simulation platform is started, and the environment is initialized based on the combat scenario. Battlefield situation information is acquired through the simulation platform's interface and mapped into neural network input for calculation to obtain action prediction results. These results are then converted into simulation platform equipment commands to interact with the simulation environment and acquire new battlefield situation information. The quality of the current neural network prediction results is evaluated using the newly acquired battlefield situation combined with reward and punishment function rules. The above interactive process is repeated until a single round of simulation is completed. At this point, the entire round of network input and output, reward and punishment values ​​are constructed into a training sample and stored in the experience pool. When the number of samples in the experience pool reaches the threshold required for training, a batch of samples is randomly selected as the input to the neural network. Based on the network output, gradient calculation is performed using a time difference strategy, and backpropagation is performed to update the network parameters. The above training process is repeated until the model test indicators meet the predetermined requirements, at which point training stops.

[0022] Step 7: Algorithm testing.

[0023] Algorithm testing involves applying the trained model to a simulation environment to test its distributed collaborative decision-making capabilities. Based on the "centralized training - distributed execution" decision architecture, only the agent decision network DRQN ​​needs to be tested. The testing capabilities mainly include two aspects: First, based on a predetermined combat scenario, the agent performs several rounds of autonomous decision-making simulations on the simulation platform, and its win rate is statistically analyzed to see if it meets the target. If the win rate meets the expected target, the test is passed; otherwise, the algorithm is improved to address the problems in the agent's decision-making process, and the above steps are repeated until the test is passed. Second, the model's generalization ability is tested. The scenario environment for algorithm testing varies to different degrees from the training combat scenario to test whether the agent can still perform well when the battlefield environment changes. The testing process still involves the agent performing several rounds of autonomous decision-making simulations on the simulation platform, and its win rate is statistically analyzed to see if it meets the target. If abnormal performance is observed in some scenarios, the algorithm needs to be improved to enhance the agent's generalization ability.

[0024] Example 1: The following detailed description of specific embodiments of the present invention, using concrete examples, illustrates the multi-agent distributed collaborative decision-making process for air-sea confrontation: Step 1: Editing the Operational Scenario Based on the command simulation platform, operational scenarios were established.

[0025] Step 2: Motion space design.

[0026] Algorithm action space design establishes the mapping relationship between the algorithm network's output and the agent's decision-making behavior. It employs a discrete one-hot one-dimensional vector space design, where a network output of 1 indicates the agent executes the action, and an output of 0 indicates the agent does not execute the action. There are four categories: attack actions. Movement Return to base Idle ,Right now =[ , , , ], among which, attack actions To launch an air-to-air missile at an enemy aircraft that meets certain attack conditions, = ... Where n represents the number of enemy aircraft, and its dimension is... Equal to the number of enemy aircraft; movement actions To move one unit distance in any of the four directions (east, south, west, north) while satisfying the movement conditions, = ... Where m represents the number of directions (4), and its dimension is... 4; Return to base action To ensure that the fighter jet executes the return command and returns to the takeoff airport when the return conditions are met, its dimension is... 1; Idle motion The dimension of the action that is forcibly executed after the death of an agent. The total dimension of the action space is 1. It equals the sum of the dimensions of all types of actions, that is: = + + +

[0027] The network outputs actions under certain constraints: when the agent dies, it is only allowed to perform idle actions, i.e. =[0 0 …. 1]; When the agent meets the return-to-home condition, only the return-to-home action is allowed, i.e., a=[0 0 … 0 1 0]; If the agent is alive and its missiles are not exhausted, and no enemy aircraft meeting the attack conditions appears, then the agent autonomously decides to choose a direction to move based on the network input; If the agent is alive and its missiles are not exhausted, and an enemy aircraft meeting the attack conditions exists, then the agent autonomously decides to attack a certain enemy aircraft or choose to continue moving to other positions based on the network input. In the algorithm, the return-to-home condition refers to the aircraft being alive and the air-to-air missiles not being exhausted; the attack condition refers to the distance between the enemy aircraft and the agent being within the effective range of the missiles and the angle between the line connecting the agent and the enemy aircraft and the agent's current heading being within the attack sector angle range of the agent.

[0028] Step 3: Design the state space of the agent.

[0029] Algorithm state space design is to establish a mapping relationship between the input of the algorithm network and the agent's observation information of the environment. The essence of a neural network is to fit the functional relationship between the network's input and output. Therefore, whatever action the algorithm outputs at the network output layer, the corresponding information input needs to be provided at the network input layer. In other words, the network input is sufficient to provide enough useful decision reference information for the network output.

[0030] Since the QMIX network includes both DRQN ​​and MIX networks, separate network input designs are needed for each type of network: global state observation. and local state observation The two networks share some semantic similarities in their inputs, but differ significantly in their structure. Generally, the inputs of the two networks are... Defined as the global battlefield situation information required for algorithmic decision-making. It is defined as the local battlefield situation information observed by the agent itself, which is required for the agent's decision-making.

[0031] Global state observation It comprises two categories: absolute position observation and fighter agent state observation. Both types of observation employ a continuous one-dimensional vector space design, with the single neuron output value o defined as [0,1]. Absolute position observation describes the horizontal and vertical relative positions of all entities on the battlefield, including friendly fighter agents and enemy fighters, ships, etc., relative to the lower left corner of the battlefield space. The dimension of absolute position observation is... The calculation is as follows, where This indicates the number of our fighter jets. Indicates the number of enemy aircraft. Indicates the number of enemy ships. The number of variables describing the absolute position of an entity is generally taken as 2 in two-dimensional space. The fighter jet agent state observation describes the remaining ammunition, remaining fuel, and heading angle information of each agent on the battlefield. The remaining ammunition represents the ratio of the number of available missiles to the total payload of the fighter jet; the remaining fuel represents the ratio of the remaining fuel to the total fuel; and the heading angle is the ratio of the current flight direction of the fighter jet relative to true north to 360°. The dimension of the fighter jet agent state observation is... The calculation method is as follows. Therefore, the global state observation dimension is... The calculation method is as follows: = ( + + )*

[0032] = *3 = +

[0033] Local state observation It comprises three categories: relative position observation, fighter agent state observation, and attack condition observation. All three types of observation employ a continuous one-dimensional vector space design, with the single neuron output value o defined as [0,1]. Relative position observation describes the relative position of the current agent with other friendly and enemy fighters. It is calculated by dividing the absolute value of the position difference between the two by the battlefield length / width. The dimension of the relative position observation is... The calculation method is as follows, where This indicates the number of our fighter jets. Indicates the number of enemy aircraft. Indicates the number of enemy ships. The dimension of the variable describing the absolute position of an entity is 2, typically taken as 2 in two-dimensional space. The state observation of a fighter jet agent describes the ratio of the difference between the agent's current fuel level and the minimum fuel level required for a safe return to its total fuel level, and the ratio of available missiles to the total payload. The dimension is 2; the attack condition observation describes the ratio of the distance between the fighter jet and the enemy aircraft to the length / width of the battlefield, and the ratio of the angle between the line connecting the fighter jet and the enemy aircraft and the fighter jet's current heading to 360°. The calculation method is shown below. Local state observation. dimension The calculation method is as follows: = ( - + + )*

[0034] = * 2 = + +

[0035] When the fighter jet agent dies, its entire observation space is set to 0, i.e. =[0 0 … 0], and at the same time, all descriptions of the fighter jet in the global state observation are also set to 0; on the other hand, if a fighter jet of our side or a fighter jet or ship of the enemy is killed, then all observations of this entity by our agents are set to 0, and similarly, all values ​​of this entity in the global state observation are also set to 0.

[0036] Step 4: Neural Network Architecture Design The neural network architecture of the multi-agent distributed cooperative decision-making method for air-sea confrontation consists of two types of networks: one is the independent behavior control network (DRQN) for each agent, and the other is the global action evaluation network (MIX) used during the training phase. This network combines the control decisions of all agent networks to obtain joint actions and analyzes and evaluates the decision quality of the joint actions based on the current global situation observation. The input of the DQRN network is the current state observation of agent i and the action of the previous time step, and the output is... ( , ), where j = 1, 2, ..., n, and n is the dimension of the agent's action space. ( , ) indicates that the agent at time t is based on Observation execution The reward value that can be obtained is generally what we choose. For the corresponding action 'a', since the network's action prediction output does not depend on the situational observations of other agents, it can make independent decisions and control. The input to the MIX network is all the agents' actions. ( , Values ​​and global situation observation The output is That is, at time t, the algorithm is based on the global situation. The environmental reward value obtained by performing the joint action 'a'.

[0037] Step 5: Design the reward and punishment function.

[0038] The reward and penalty function design evaluates the joint actions of the agent's control decisions under the current battlefield situation. Since neural networks have a strong ability to fit output values ​​between [-1, +1], the absolute value of the total reward in a single round is controlled to be around 1, i.e., the reward and penalty function is normalized. The algorithm's reward and penalty function R is divided into four categories: distance reward and penalty. , crackdown and rewards Return flight rewards and penalties And rewards and penalties for completing the task Among them, the task completion reward / penalty value is the main reward / penalty value, and its absolute value is significantly greater than the previous three items, and this value is positive.

[0039] Distance Rewards and Punishments The design aims to encourage agents to engage the enemy as quickly as possible while avoiding areas of enemy fire threat. It includes two components: an engagement reward. Punishment for approaching dangerous areas The calculation method for the engagement reward function is shown below, where... This represents the distance difference between the agent and enemy aircraft number i. This represents the number of the closest enemy aircraft to the agent at time t. This indicates the approach radius, typically set to 1.5-2 times the enemy's firing range. The penalty for approaching a danger zone is calculated as follows: This represents the active threat radius of enemy threat entity i. This represents the radius of the danger buffer zone, which is generally set to about 1.1 times the enemy's fire range. L is the battlefield height / width. The calculation method is as follows. Since distance rewards are secondary rewards, to prevent agents from exhibiting abnormal states such as greed, all distance rewards are set to negative values.

[0040] =

[0041] = -max{( + ) / L,0} = +

[0042] Punishment and reward The design aims to encourage agents to launch an air-to-air missile to strike enemy aircraft when conditions permit. Since the strike reward is strongly correlated with the main reward, it is set to a positive value, with a significantly larger penalty compared to the distance penalty. Here, the strike penalty is set to +2.

[0043] The purpose of the return-to-base reward is to encourage agents to execute the return-to-base command and return to the airport when the return-to-base conditions are met, namely, insufficient fuel or depleted ammunition, thus avoiding losses due to aircraft being shot down. This reward is a one-time reward and is beneficial to our faction; therefore, the return-to-base command is set to a positive value of +1.

[0044] The purpose of the task completion reward and punishment design is to encourage agents to cooperate to achieve victory in the simulation. The condition for achieving victory in the simulation is to destroy all enemy aircraft. This reward is the main reward, so its value is set to a positive value, and its absolute value must be significantly greater than the values ​​of other types of reward and punishment functions. Here we set it to +100.

[0045] Step 6: Model parameter training.

[0046] Model parameter training involves generating training experience samples through interaction between the agent and the environment, and iteratively updating the neural network parameters based on the gradient descent algorithm to train the network's function fitting ability to meet the expected requirements.

[0047] Since this paper uses a locally observed agent-based collaborative decision-making method, the agents cannot obtain sufficient information to support effective decision-making. Current solutions fall into two main categories: one is to increase communication mechanisms between agents to share observation information; the other is to use historical time-series observation information as input for agent decisions. This paper adopts the latter. Therefore, the algorithm's sample experience pool is no longer based on single-step interaction samples, but rather on a sequence of all single-step samples collected from the start of the agent's interaction with the environment to the end of the current interaction. The neural network training uses batch training, with 32 experience samples taken from the experience pool for gradient updates in each round. The stopping condition for each round of training is when the number of interactions between the agent and the environment reaches a threshold, or when the agent completes its task or all agents die. The ending condition for model parameter training is when the total number of training rounds reaches a threshold, or when the win rate of the agent in the test cases meets the required index.

[0048] Step 7: Algorithm testing.

[0049] Algorithm testing involves applying the trained model to a simulation environment to test its decision-making capabilities. Because the algorithm employs a "centralized training - distributed execution" architecture, the input to the neural network model during distributed execution testing is no longer the joint situational information observed by all agents, but rather the local observation information of a single agent. The test phase uses the agent's win rate in the simulation environment as the passing indicator. If the win rate reaches the expected target, the test is passed; otherwise, algorithm improvements are made to address problems in the agent's decision-making process, and the above steps are repeated until the test is passed. Simultaneously, to test the model's generalization ability, the simulated combat scenarios, including the agent's initial position and enemy equipment conditions, all differ to varying degrees from the training scenario. Therefore, the test assesses whether the agent can still perform well under partially changed battlefield conditions. If abnormal performance is observed in certain scenarios, targeted algorithm improvements are needed to enhance the agent's generalization ability.

[0050] This invention proposes a multi-agent distributed collaborative decision-making method for air-sea confrontation. Taking the air-sea confrontation battlefield environment as the combat background, it realizes multi-agent distributed collaborative autonomous decision-making in complex dynamic battlefield environments based on the multi-agent distributed collaborative decision-making framework of "centralized training-distributed execution".

[0051] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-agent distributed cooperative decision-making method for air-sea confrontation, characterized in that, The method includes the following steps: Step 1: Editing the Operational Scenario Based on a certain simulation platform, a combat scenario was designed, and the content included the initialization information of the red and blue combat units, the simulation time, the single-step simulation step size, the launch control authorization of various weapons of the red and blue camps, the weather conditions of the combat area, the combat plan of the blue team's weapons and equipment, and the conditions for determining the victory of the red and blue teams. Step Two: Motion Space Design Based on existing combat scenarios and considering the tactical actions that the intelligent agents can execute in the simulation platform, the action space of the intelligent agents is designed. The executable actions of the intelligent agents are defined as movement, attack, return to base, and idle rotation. To establish the mapping relationship between the intelligent agent actions and the neural network output, the movement action is defined as a one-dimensional one-hot vector with a dimension of 4; the attack action is defined as a one-dimensional one-hot vector with a dimension equal to the number of enemy air defense equipment; the return to base action is defined as a one-dimensional vector with a dimension of 1; and the idle rotation action is defined as a one-dimensional vector with a dimension of 1. All four types of actions are 0-1 discrete variables, where 1 represents the current action being executed and 0 represents the current action not being executed. Step 3: Design of the agent's state space; Based on the established action space design scheme of the intelligent agent, and combined with the situation information interface provided by the simulation and deduction platform, the state space design of the intelligent agent is carried out. Based on the "centralized training-distributed execution" decision framework, two state-space representation methods are designed for global situation observation during training and local situation observation during execution. Local situation observation includes three categories: position observation relative to red and blue equipment, state information observation of agents, and attack condition observation of mounted weapons. All three types of state observation information adopt a continuous one-dimensional vector space design and are normalized before being input into the agent decision network. Global situation observation includes two categories: absolute position observation of red and blue equipment and state observation of all agents. Both types of observation adopt a continuous one-dimensional vector space design and are normalized before being input into the agent auxiliary training network. Step 4: Neural Network Architecture Design Based on the design scheme of the action space and state space of the given agents and the decision architecture of "centralized training-distributed execution", a neural network architecture for a multi-agent distributed collaborative decision-making method for air-sea confrontation is designed. The neural network architecture consists of two types of networks: one is the DRQN, which is an independent behavior control network for each agent, and the other is the global action evaluation network MIX used in the training phase. The DQRN network structure uses the agent's local situational awareness and the previous action. , [This refers to] network input, combined with the gating unit output from the previous time step. After calculation by the gated loop unit module and two layers of multilayer perceptron modules, the reward vector of all possible actions performed by the agent at the current moment is output. and intermediate information And select the maximum Q value based on the Q-maximization strategy. The corresponding action is used as the prediction output. The action prediction output of this network does not depend on the situational observation of other agents and can independently carry out distributed decision-making and control. The MIX network structure uses global situational information. and the DRQN ​​network output values ​​of each agent [ As input, the data is processed by a parameter-trainable hybrid network, which outputs the reward value of the joint action of each agent at the current time. This network can analyze and evaluate the decision quality of joint actions based on current global situation observations, and guide the improvement of action decision-making in intelligent agent networks; DRQN and MIX networks are tightly coupled based on input and output information, forming the network architecture Qmix. During the training phase, parameters are trained based on the Qmix architecture. After training, the Mix network is discarded, and only the DRQN ​​network is used for agent decision control. Both DRQN ​​and MIX networks update parameters using the Adam gradient descent algorithm until the model converges. Step 5: Design the reward and punishment function; Based on the established action space design scheme and combined with the judgment conditions for the Red Team's victory, a reward and punishment function is designed; the reward and punishment function design is to give a score evaluation to the joint actions of the intelligent agent's control decision under the current battlefield situation. The absolute value of the total reward in a single round is controlled to be around 1, that is, the reward and penalty function is normalized. The reward and penalty function R is divided into four categories: distance reward and penalty, attack reward and penalty, return reward and penalty, and mission completion reward and penalty. Among them, the mission completion reward and penalty value is the main reward and penalty value, whose absolute value is significantly greater than the first three items, and this value is positive. Step Six: Model Parameter Training; Based on the established neural network structure design, the model parameters are trained by combining the simulation platform interface and the computer hardware and software environment. Step 7: Algorithm Testing; The trained model is applied to a simulation environment to test its distributed collaborative decision-making capability. Based on the "centralized training - distributed execution" decision architecture, only the agent decision network DRQN ​​needs to be tested. The testing capabilities include two aspects: First, based on a predetermined combat scenario, the agent performs several rounds of autonomous decision-making simulations on the simulation platform, and its win rate is statistically analyzed to see if it meets the target. If the win rate meets the expected target, the test is passed; otherwise, the algorithm is improved to address the problems in the agent's decision-making process, and the above steps are repeated until the test is passed. Second, the model's generalization ability is tested. The algorithm testing scenario environment varies to different degrees from the training combat scenario setting to test whether the agent can still perform well when the battlefield environment changes. The testing process still involves the agent performing several rounds of autonomous decision-making simulations on the simulation platform, and its win rate is statistically analyzed to see if it meets the target. If there are abnormal performances in some scenarios, the algorithm needs to be improved to enhance the agent's generalization ability.

2. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 1, characterized in that, In step one, the initialization information includes: unit type, unit quantity, platform mounting scheme and deployment location, and meteorological conditions include rainfall, wind speed and sea state.

3. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 1, characterized in that, In step two, a discrete one-hot one-dimensional vector space design is adopted, meaning that a network output of 1 indicates that the agent performs the action, and an output of 0 indicates that the agent does not perform the action. There are four categories: attack actions. Movement Return to base and idling ,Right now =[ , , , ], among which, attack actions To launch an air-to-air missile at an enemy aircraft that meets certain attack conditions, = ... Where n represents the number of enemy aircraft, and its dimension is... Equal to the number of enemy aircraft; movement actions To move one unit distance in any of the four directions (east, south, west, north) while satisfying the movement conditions, = ... Where m represents the number of directions (4), and its dimension is... 4; Return to base action To ensure that the fighter jet executes the return command and returns to the takeoff airport when the return conditions are met, its dimension is... 1; Idle motion The dimension of the action that is forcibly executed after the death of an agent. The value is 1.

4. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 3, characterized in that, Total dimension of the action space It equals the sum of the dimensions of all types of actions, that is: = + + + The network outputs actions under certain constraints: when the agent dies, it is only allowed to perform idle actions, i.e. =[0 0 …. 1]; When the agent meets the return-to-home conditions, only the return-to-home action is allowed, i.e., a=[0 0 … 0 1 0]; If the agent survives and the missiles are not exhausted, and no enemy aircraft that meets the attack conditions appears, then the agent will autonomously decide to choose a direction to perform the movement action based on the network input. If the agent survives and its missiles are not exhausted, and there is an enemy aircraft that meets the attack conditions, the agent will autonomously decide to attack a certain enemy aircraft based on network input or choose to continue moving to another location. The return condition refers to the fighter jet surviving and its air-to-air missiles not being exhausted. The attack condition refers to the distance between the enemy aircraft and the fighter jet agent being within the effective range of the missiles and the angle between the line connecting the fighter jet and the enemy aircraft and the fighter jet's current heading being within the fighter jet's attack sector angle range.

5. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 3 or 4, characterized in that, In step three, since the QMIX network includes two types of networks: DRQN ​​network and MIX network, network input design is performed separately for each type of network: global state observation. and local state observation , Defined as the global battlefield situation information required for algorithmic decision-making. It is defined as the local battlefield situation information observed by the agent itself, which is required for the agent's decision-making. Global state observation It includes two categories: absolute position observation and fighter jet agent state observation. Both types of observation adopt a continuous one-dimensional vector space design, and the output value o of a single neuron is defined as [0,1]. Among them, absolute position observation describes the horizontal and vertical relative positions of all entities in the battlefield, including friendly fighter jet agents and enemy fighter jets and ships, relative to the lower left corner of the battlefield space. Fighter jet agent state observation describes the remaining ammunition, remaining fuel, and heading angle information of each agent in the battlefield. The remaining ammunition represents the ratio of the number of missiles available to the total number of missiles carried by the fighter jet. The remaining fuel represents the ratio of the remaining amount of fuel to the total amount of fuel. The heading angle is the ratio of the angle of the current flight direction of the fighter jet relative to due north to 360°. Local state observation It comprises three categories: relative position observation, fighter agent state observation, and attack condition observation. All three types of observation employ a continuous one-dimensional vector space design, with the single neuron output value o defined as [0,1]. Relative position observation describes the relative position of the current agent with other friendly and enemy aircraft, calculated by dividing the absolute value of the position difference by the battlefield length / width. Fighter agent state observation describes the ratio of the difference between the agent's current fuel level and the minimum fuel level required for a safe return to base to the total fuel level, and the ratio of available missiles to the total payload. Its dimension is... The value is 2; the attack condition observation describes the ratio of the distance between the fighter jet and the enemy aircraft to the length / width of the battlefield, and the ratio of the angle between the line connecting the fighter jet and the enemy aircraft and the fighter jet's current heading to 360°. When the fighter jet agent dies, its entire observation space is set to 0, i.e. =[0 0 … 0], and at the same time, all descriptions of the fighter jet in the global state observation are also set to 0; on the other hand, if a fighter jet of our side or a fighter jet or ship of the enemy is killed, then all observations of this entity by our agents are set to 0, and similarly, all values ​​of this entity in the global state observation are also set to 0.

6. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 5, characterized in that, In step three Dimension of absolute position observation The calculation is as follows, where This indicates the number of our fighter jets. Indicates the number of enemy aircraft. Indicates the number of enemy ships. This represents the number of variables describing the absolute position of an entity, taking the value of 2 in two-dimensional space; Dimension of fighter jet intelligent agent state observation The calculation method is as follows; Global state observation dimension The calculation method is as follows: = （ + + ）* = *3 = + Dimension of relative position observation The calculation method is as follows, where This indicates the number of our fighter jets. Indicates the number of enemy aircraft. Indicates the number of enemy ships. This represents the number of variables describing the absolute position of an entity, taking the value of 2 in two-dimensional space; Attack Condition Observation Dimension The calculation method is as follows; Local state observation dimension The calculation method is as follows: = （ - + + ）* = * 2 = + + 。 7. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 5, characterized in that, In step five, the algorithm's reward and penalty function R is divided into four categories: distance reward and penalty. , crackdown and rewards Return flight rewards and penalties And rewards and penalties for completing the task ; Distance Rewards and Punishments The design aims to encourage agents to engage the enemy as quickly as possible while avoiding areas threatened by enemy fire. It includes Two items: Reward for approaching the enemy Punishment for approaching dangerous areas ; Punishment and reward This is used to encourage agents to launch an air-to-air missile to strike an enemy aircraft when conditions permit; the strike reward is set to a positive value, and the reward / penalty value relative to distance should be significantly larger; The return-to-base reward is designed to encourage agents to execute the return-to-base command and return to the airport when the return-to-base conditions are met, namely when fuel is insufficient or ammunition is exhausted, so as to avoid the loss of the fighter jet due to being shot down. This reward is a one-time reward and is beneficial to our faction, and the return-to-base command is set to a positive value of +1. The task completion reward and penalty is used to encourage agents to cooperate to achieve victory in the simulation. The condition for achieving victory in the simulation is to destroy all enemy aircraft. This reward is the main reward, and its value is set to a positive value, and its absolute value must be significantly greater than the value of other types of reward and penalty functions.

8. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 7, characterized in that, The calculation method for the engagement reward function is shown below, where... This represents the distance difference between the agent and enemy aircraft number i. This represents the number of the closest enemy aircraft to the agent at time t. This indicates the radius of approach to the enemy, set to 1.5-2 times the enemy's fire range; The penalty for approaching a dangerous area is calculated as follows: This represents the active threat radius of enemy threat entity i. This represents the radius of the danger buffer zone, set to approximately 1.1 times the enemy's fire range, where L is the battlefield height / width. The calculation method is as follows; to prevent the agent from entering an abnormal state, all distance rewards and penalties are set to negative values; = = -max{( + ) / L,0} = + 。 9. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 7 or 8, characterized in that, Step six specifically includes: First, initializing the computer GPU, deep learning framework module, neural network parameters, and training sample experience pool; then starting the simulation platform and initializing the environment based on the combat scenario; acquiring battlefield situation information through the simulation platform's interface, mapping it into neural network input for calculation, obtaining action prediction results, and converting them into simulation platform equipment commands to interact with the simulation environment to acquire new battlefield situation information; using the newly acquired battlefield situation combined with reward and punishment function rules to evaluate the quality of the current neural network prediction results; repeating the above interaction process until the end of a single round of simulation; constructing the entire round of network input and output, reward and punishment values ​​into a training sample and storing it in the experience pool; when the number of samples in the experience pool reaches the threshold required for training, randomly selecting a batch of samples as input to the neural network, calculating gradients using a time difference strategy based on the network output, and performing backpropagation to update the network parameters; repeating the above training process until the model test indicators meet the predetermined requirements, and then stopping training.

10. The multi-agent distributed cooperative decision-making method for air-sea confrontation as described in claim 9, characterized in that, The algorithm's training sample experience pool is no longer based on single-step interaction samples of the agent, but on a sequence of all single-step samples from the start of the agent's interaction with the environment to the end of the current round of interaction. This sequence serves as the algorithm's learning experience. The neural network training adopts a batch training method, with 32 experience samples taken from the experience pool for gradient updates in each round of training. The stopping condition for each round of training is that the number of interactions between the agent and the environment reaches a threshold, or the agent completes its task or all agents die. The ending condition for model parameter training is that the total number of training rounds reaches a threshold, or the win rate of the agent in the test cases meets the target requirements.

Images