A method, device and storage medium for optimizing decision-making of drone swarm confrontation based on HAPPO algorithm

Abstract

The present invention discloses a method, device and storage medium for optimizing the decision-making of unmanned aerial vehicle cluster confrontation based on the HAPPO algorithm, and belongs to the field of aircraft technology. The method includes: obtaining the position information of the target point, the position and state information of the red drone, and the position and state information of the blue drone; taking each drone as a decision maker, taking the target point position information, the drone position and situation information as the state, and taking the drone track point as the action, based on the preset reward function and discount factor, converting the pre-constructed unmanned aerial vehicle cluster confrontation decision optimization model into a Markov decision process; each drone uses a pre-trained deep reinforcement neural network to solve the Markov decision process based on the target point position information and the position and state information of the red and blue drones, and obtains the drone cluster confrontation strategy. The present invention can realize the flexible adjustment of the drone cluster under multi-agent control in a complex situation environment, so as to efficiently complete the combat mission objectives.

Description

Technical Field

[0001] The present invention relates to a method, a device and a storage medium for optimizing unmanned aerial vehicle (UAV) cluster confrontation decision based on a HAPPO algorithm, and belongs to the technical field of unmanned aerial vehicle control. Background Art

[0002] With the rapid development of science and technology, drones have shown great advantages in replacing human flight missions, especially the complexity, variability and unpredictability of 4D (Dull, Dirty, Dangerous and Deep) missions, which make small-sized, fast and low-cost drones stand out. Compared with the single drone combat mode, the drone cluster combat mode not only integrates the powerful functions of a single drone, but also pays more attention to the coordinated combat and joint resistance of drone clusters, so as to gather the advantages of single combat capabilities and cluster collaboration capabilities.

[0003] With the continuous development of intelligent platforms such as drones, modern battles are increasingly showing an intelligent development trend. The battlefield situation caused by fast-paced and strong confrontation is changing rapidly. The complex environment such as fire interception and battlefield meteorology and geography faced by unmanned systems requires that the confrontation plan must be able to be dynamically adjusted and generated accordingly, which puts forward very high efficiency requirements for action planning. However, in the face of large-scale, fast-paced intelligent confrontation actions, traditional action planning methods such as mathematical planning or bionic evolution are difficult to model and slow to calculate, and it is difficult to meet the requirements of intelligent action planning tasks. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device and storage medium for optimizing the decision-making of drone swarm confrontation based on the HAPPO algorithm, so as to enable the drone swarm to make real-time and effective dynamic adjustments when facing complex battlefield situations, thereby improving the combat strength and efficiency.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0006] On the one hand, the present invention provides a method for optimizing the decision-making of a drone swarm confrontation, which comprises:

[0007] Obtain the target point location information, the location and status information of the red team’s drone, and the location and status information of the blue team’s drone;

[0008] Taking each drone as a decision maker, the target point location information, the drone's location and situation information as the state, and the track point selected by the drone as the action, the pre-built drone swarm confrontation decision optimization model is transformed into a Markov decision process based on the preset reward function and discount factor;

[0009] Each drone uses a pre-trained deep reinforcement neural network and solves the Markov decision process based on the target point location information, the location and state information of the red drone, and the location and state information of the blue drone to obtain an optimized drone cluster confrontation strategy.

[0010] The present invention utilizes the improved multi-agent algorithm HAPPO for learning. By designing the global state space, action space, reward function, etc. of the agent, and through multiple iterative deductions, the agent can control the UAV to make the optimal trajectory planning and confrontation decision under different situations.

[0011] Optionally, the state of the Markov decision process is expressed as:

[0012] ;

[0013] In the formula, Representing an Agent The state space of Indicates the Red Army's drone The ratio of coordinates to battlefield length, Indicates the Red Army's drone The ratio of coordinates to battlefield width, represents the speed scalar of the Red Team UAV, It represents the ratio of the heading angle of the red drone to , Indicates the distance between the Red Team UAV and the ground target point. Indicates the radius of the Red Team’s drone’s observable range. Indicates the angle between the line connecting the ground target point and the Red UAV and the heading of the Red UAV; Indicates the blue drone The ratio of coordinates to battlefield length, Indicates the blue drone The ratio of coordinates to battlefield width, Represents the speed scalar of the blue team’s drone, Indicates the heading angle of the blue drone and The ratio of It represents the angle between the line connecting the blue drone and the red drone and the heading of the red drone. Indicates the straight-line distance between the blue drone and the red drone;

[0014] The actions of the Markov decision process are: the track points of the drone at the next moment.

[0015] Optionally, the reward function of the Markov decision process is expressed as:

[0016] ;

[0017] In the formula, It represents the comprehensive instant reward obtained by the agent at the moment, Indicates the reward for hitting an enemy aircraft. Indicates the number of enemy aircraft hit; Indicates the penalty for hitting an obstacle or entering a threat area. Indicates the number of teammates. Indicates the penalty for being out of bounds when approaching the boundary. represents the gravitational force of the value field on the drone, Indicates the reward for closing the distance when chasing an enemy aircraft. Indicates the number of pursuits, It means punishment for being threatened directly by enemy aircraft. Indicates the number of enemy aircraft in front. , , and Represents the weight coefficient.

[0018] Optionally, the gravitational force of the value field on the drone is expressed as:

[0019] ;

[0020] In the formula, represents the value of the value field to the agent, represents the gradient of the value field, represents the value scale factor, Indicates the value of the value body. A positive value indicates a reward, while a negative value indicates a threat. Indicates drone Distance from the value body.

[0021] Optionally, the value of the value field to the agent is expressed as:

[0022] .

[0023] Optionally, the pre-trained deep reinforcement neural network is trained using the HAPPO algorithm, wherein each agent has an Actor network and inputs state information , output agent action , all agents share a Critic network and input state information , output state value , the Actor network and the Critic network are both fully connected networks, the corresponding number of network layers and the number of neurons in the network are hyperparameters, and the training process includes:

[0024] Each agent uses the Actor network to interact with the environment and generate a set of sample data, which includes state, action, reward and next state, and defines the action value function of any ordered agent based on the sample data;

[0025] Based on the action-value function, an advantage function of the multi-agent is obtained; and based on the advantage function, a policy gradient of the multi-agent is obtained; and further based on the policy gradient, a loss function of the Actor network is obtained;

[0026] Calculate the state value and target state value output by the Critic network to obtain the loss function of the Critic network;

[0027] Use the optimizer to minimize the corresponding loss function and update the parameters of the relevant neural network model;

[0028] Based on the training results, determine whether the expected requirements are met. If not, adjust the reward function weights and related hyperparameters and continue training; otherwise, save the intelligent agent model in this scenario.

[0029] Optionally, the action value function is expressed as:

[0030] ;

[0031] In the formula, represents the action-value function, represents any ordered agent, represents the agent number, represents the agent action, Indicates strategy, express The combined action value of the agents inside and outside, Indicates the status, express The joint actions of the agents between express The joint actions of other agents, express The strategies of agents other than It represents the mean of the action-value function that calculates the action space to obey the strategy.

[0032] Optionally, based on the action-value function, the advantage function of the multi-agent is obtained, expressed as:

[0033] ;

[0034] In the formula, represents the advantage function, express and The joint action value of the agents in the interval, express The value of the joint action of the agents in the interval, express The joint actions of agents within the interval, express Joint actions of agents within the interval.

[0035] Optionally, based on the advantage function, the policy gradient of the multi-agent is obtained, expressed as:

[0036] ;

[0037] In the formula, represents the policy gradient, express The joint actions of agents within the interval, Representing an Agent The old strategy, Representing an Agent New strategies, represents the advantage function.

[0038] Optionally, the loss function of the Actor network is expressed as:

[0039] ;

[0040] In the formula, represents the loss function of the Actor network, Indicates that the initial state follows the distribution , the agent's actions obey The mean of the advantage function, Indicates that the ratio of the new and old strategies is cut to Within, represents the hyperparameter, Representing an Agent The old strategy, Representing an Agent New strategies, Representing an Agent Actions, It means taking the minimum value among the elements in the brackets.

[0041] Optionally, the loss function of the Critic network is expressed as:

[0042] ;

[0043] In the formula, The neural network parameters are The loss function is It is expressed as the mean of the calculated state value differences, Represents the state value of the Critic network output, Indicates the target state value.

[0044] Optionally, the target state value is expressed as:

[0045] ;

[0046] In the formula, The discount factor Second power, Indicates The reward value of the step, The discount factor Second power, Indicates status The state value of The network parameters are State value function, express The state of the moment, Indicates the initial time.

[0047] In a second aspect, the present invention further provides a drone swarm confrontation decision optimization device, which comprises:

[0048] The information acquisition module is configured to acquire the target point location information, the location and status information of the red party UAV, and the location and status information of the blue party UAV;

[0049] The optimization model conversion module is configured to convert the pre-built UAV swarm confrontation decision optimization model into a Markov decision process based on a preset reward function and a discount factor, taking each UAV as a decision maker, taking the target point location information, the UAV location and situation information as the state, and taking the track point selected by the UAV as the action;

[0050] And, the strategy optimization module is configured to use each drone to use a pre-trained deep reinforcement neural network, and based on the target point location information, the location and state information of the red drone, and the location and state information of the blue drone, to solve the Markov decision process to obtain an optimized drone cluster confrontation strategy.

[0051] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the drone cluster confrontation decision optimization method as described in the first aspect are performed.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The autonomous confrontation decision-making process of drones based on the multi-agent deep reinforcement learning algorithm of the present invention can involve multiple drone agents, and utilizes the advantage function decomposition and the sequential strategy update mechanism of each agent to improve the cooperation between drones. The agents are updated in a certain order, and the subsequent agents can use the strategies that have been updated by the previous agents when updating. The interaction and cooperation between the agents are taken into consideration, so that the agents can control the drone cluster to make the optimal trajectory planning under different situations to complete the task or achieve a specific goal; in addition, the present invention also incorporates the design of the value field into the reward function to make the continuous auxiliary reward smoother, more guiding, and more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of an embodiment of the method for optimizing the decision-making of drone swarm confrontation based on the HAPPO algorithm of the present invention;

[0055] Figure 2 The figure shows a schematic diagram of the application process of the UAV path planning algorithm based on the PPO algorithm of the present invention;

[0056] Figure 3 The figure shows a schematic diagram of the architecture of the UAV trajectory planning algorithm under the control of a single agent of the present invention;

[0057] Figure 4 The figure shows a schematic diagram of the algorithm architecture for generating a UAV confrontation strategy under multi-agent control of the present invention;

[0058] Figure 5 The figure shows the trajectory planning result of a single-agent UAV in the simulation experiment of the present invention;

[0059] Figure 6 Shown is a schematic diagram of the confrontation results between the red and blue drones in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0060] The technical concept of the present invention is as follows: The present invention proposes a drone swarm confrontation scheme in a maritime combat scenario controlled by multiple agents. The reinforcement learning system architecture is based on a general reinforcement learning algorithm interaction architecture and is adapted to battlefield deduction requirements. The system architecture includes intelligent environmental perception, reactive local autonomous decision-making, and global autonomous planning and collaboration. During the reasoning process, each component supports each other to form a closed loop system, deeply perceives the battlefield environment, and outputs a multi-step combat strategy for the battlefield situation to achieve the mission objectives.

[0061] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0062] Example 1

[0063] This embodiment introduces a drone swarm confrontation decision optimization method, combined with Figure 1 , the specific steps are as follows:

[0064] Obtain the target point location information, the location and status information of the red team’s drone, and the location and status information of the blue team’s drone;

[0065] Taking each drone as a decision maker, the target point location information, the drone's location and situation information as the state, and the track point selected by the drone as the action, the pre-built drone swarm confrontation decision optimization model is transformed into a Markov decision process based on the preset reward function and discount factor;

[0066] Each drone uses a pre-trained deep reinforcement neural network and solves the Markov decision process based on the target point location information, the location and state information of the red drone, and the location and state information of the blue drone to obtain an optimized drone cluster confrontation strategy.

[0067] In the specific application of this embodiment, each drone is equivalent to an intelligent agent. Each intelligent agent adjusts the path planning strategy according to the real-time situation information and optimizes the combat effectiveness of the cluster according to the results of the confrontation decision, so that the drone cluster can achieve efficient collaborative combat in a complex battlefield environment, so as to effectively improve the success rate and safety of the operation.

[0068] Example 2

[0069] In order to realize the decision-making optimization process of UAV swarm maritime confrontation under multi-agent control, this embodiment first starts with a single UAV and briefly describes the specific implementation process of the corresponding path planning of the UAV under the control of a single agent during maritime combat.

[0070] 1. UAV trajectory planning under single-agent control based on PPO algorithm

[0071] In order to realize the path planning of the UAV, the state space, action space and reward function need to be modeled accordingly.

[0072] The state space includes: the position of the target point, the position and state of the drone itself, and the enemy information that can be detected, which can be specifically expressed as:

[0073] ;

[0074] In the formula, Representing an Agent The state space of Indicates the Red Army's drone The ratio of coordinates to battlefield length, Indicates the Red Army's drone The ratio of coordinates to battlefield width, represents the speed scalar of the Red Team UAV, It represents the ratio of the heading angle of the red drone to , Indicates the distance between the Red Team UAV and the ground target point. Indicates the radius of the Red Team’s drone’s observable range. Indicates the angle between the line connecting the ground target point and the Red UAV and the heading of the Red UAV.

[0075] The action space is the set of all actions that the agent can perform, which reflects the behavior pattern of the agent and corresponds to the output format of the action network. In this algorithm, the action space is specifically the track point of the drone at the next moment.

[0076] In the process of designing the reward function, in order to ensure the design of a smoother and better guided continuous auxiliary reward, the present invention draws on the classic artificial potential field method in the robot path planning algorithm and incorporates the value field into the reward function.

[0077] Drones operating in actual battlefields should operate within designated areas, that is, the effective movement range of drone confrontation missions is limited to a rectangular battlefield area with fixed length and width. Drones that escape or leave the battlefield area will be punished accordingly.

[0078] In a specific application scenario, in order to make the sparse penalty for leaving the battlefield area continuous, the boundary is set to have a negative value for the agent; secondly, if the enemy aircraft is attacked, a corresponding reward will be obtained, and the enemy aircraft is set to have a positive value for the agent. Therefore, the closer the distance, the higher the absolute value of the positive or negative value of the value field to the agent, which is proportional to the inverse of the distance, and can be specifically expressed as a function:

[0079] ;

[0080] In the formula, represents the value of the value field to the agent, represents the value scale factor, Indicates the value of the value body. A positive value indicates a reward, while a negative value indicates a threat. Indicates drone Distance from the value body.

[0081] Correspondingly, the gravitational force of the value field on the drone is the gradient of the value field, as shown in the following formula:

[0082] ;

[0083] In the formula, represents the gravitational force of the value field on the drone, represents the gradient of the value field, represents the value scale factor, Indicates the value of the value body. A positive value indicates a reward, while a negative value indicates a threat. Indicates drone Distance from the value body.

[0084] In order to guide the drone to chase the enemy, a distance approach reward based on the value field is set; in order to prevent the drone from going out of bounds, a boundary approach penalty based on the value field is set. The design of related rewards and penalties effectively solves the problem of sparse rewards, and the smoothness and guidance of related continuous auxiliary reward functions are also significantly improved.

[0085] In addition, in order to encourage drones to chase enemy aircraft from behind and try to gain an advantageous position rather than attack head-on and die together, a penalty for being threatened head-on by enemy aircraft is set to distinguish between head-on attack and attack after detour. In summary, the design of the relevant reward function can be shown as follows:

[0086] ;

[0087] In the formula, It represents the comprehensive instant reward obtained by the agent at the moment, Indicates the reward for hitting an enemy aircraft. Indicates the number of enemy aircraft hit; Indicates the penalty for hitting an obstacle or entering a threat area. Indicates the number of teammates. Indicates the penalty for being out of bounds when approaching the boundary. represents the gravitational force of the value field on the drone, Indicates the reward for closing the distance when chasing an enemy aircraft. Indicates the number of pursuits, It means punishment for being threatened directly by enemy aircraft. Indicates the number of enemy aircraft in front. , , and Represents the weight coefficient.

[0088] The PPO algorithm has an Actor-Critic network architecture. Specifically, the Actor network is a policy network that inputs state information , output agent action ; The Critic network is a state-value network, with input state , output state value Among them, both Actor and Critic are three-layer fully connected networks, and the number of neurons in the network is a hyperparameter that needs to be determined based on specific scenario experiments.

[0089] In the training process of the PPO algorithm, the policy network needs to interact with the environment to generate a batch of sample data. , and put it into the experience replay pool, and then use this data to calculate the corresponding advantage function. Indicates the current state. Indicates action, Indicates reward, Indicates the next state.

[0090] In the PPO algorithm, the advantage function is used to evaluate how good each action is relative to the average level. The samples within time steps are used to calculate the corresponding advantage function, which can be specifically expressed as:

[0091] ;

[0092] In the formula, express The advantage function at time, express The timing difference error at time, express The timing difference error at time, express The timing difference error at time, express Status at all times The state value of express Status at all times The state value of represents the time index within a trajectory segment of a given length, Represents the discount factor.

[0093] Again, since the PPO algorithm uses the actor-critic architecture, it is necessary to implement the loss functions of the actor and critic networks separately. Specifically, the loss function of the Actor network is expressed as:

[0094] ;

[0095] In the formula, represents the loss function of the Actor network, Indicates that the new strategy is in state Take action The probability of Indicates that the old policy is in state Take action The probability of taking an action is calculated by sampling.

[0096] The loss function of the Critic network is expressed as:

[0097] ;

[0098] In the formula, The discount factor Second power, Indicates The reward value of the step, The discount factor Second power, Indicates status The state value of Indicates network parameters is the state value function, express The state of the moment, Indicates the initial time.

[0099] Finally, use the optimizer to minimize the relevant loss function of the PPO algorithm and update the neural network model parameters. Among them, Adam, SGD and other optimizers are usually used for optimization; mini-batch online update method is used for corresponding training. Repeat the above steps until the preset number of iterations is reached.

[0100] Combination Figure 2 and Figure 3 In this embodiment, when the UAV under the control of a single agent based on the PPO algorithm performs corresponding path planning, it first obtains the target position information, the current position information of the UAV and the environmental information during the assumption or simulation process, and then loads the UAV path planning algorithm model to output the planned UAV waypoints at the next moment; the UAV simulation entity updates the status according to the waypoint information, and the simulation deduction engine completes the update of the battlefield situation accordingly; then it needs to determine whether the environmental information, geographic information and target information are updated, and the path planning process ends after the UAV navigates to the desired target area or target position.

[0101] In summary, in the path planning process of a single UAV in maritime combat, the maneuver decision problem in the UAV offensive mission is modeled as a Markov decision process. The single UAV trajectory planning algorithm based on the proximal policy optimization algorithm (PPO algorithm) interacts with the environment multiple times, regularizes the modeling of threat areas and obstacles, collects empirical data, and updates the intelligent agent strategy in a timely manner, thus realizing the path planning of a single UAV in maritime combat scenarios.

[0102] Furthermore, this embodiment is further advanced to multiple drones of the red and blue sides, detailing the specific implementation process of the corresponding path planning and intelligent confrontation strategy of the drone cluster under multi-agent control during maritime operations.

[0103] 2. Generation of Intelligent Countermeasure Strategy for UAVs under Multi-agent Control Based on HAPPO Algorithm

[0104] First of all, it is worth explaining that the HAPPO algorithm is a multi-agent collaborative algorithm based on the PPO algorithm. This algorithm is a multi-agent trust region method strategy optimization method with theoretical guarantees. It reduces the complexity of the problem through a hierarchical structure and improves the efficiency of the algorithm. It decomposes the task into multiple subtasks, each of which can be trained and optimized separately, making it easier to find effective solutions. At the same time, the hierarchical structure also enables the algorithm to better handle problems of long-term dependencies and sparse rewards.

[0105] It is worth noting that the multi-agent environment includes several drones and several wireless communication devices, and each of the drones can only obtain local observations and needs to establish a wireless communication connection to share the observation results.

[0106] Similarly, similar to the drone trajectory planning algorithm based on the PPO algorithm, the drone intelligent confrontation strategy generation algorithm based on the HAPPO algorithm also needs to model the state space, action space and reward function.

[0107] The state space includes: the position of the target point, the position and state of the red team's own drone, the information of the blue team's enemy drones that can be detected, and the position and state information of the blue team's enemy drones. Specifically, the state of the relevant Markov decision process can be expressed as:

[0108] ;

[0109] In the formula, Representing an Agent The state space of Indicates the Red Army's drone The ratio of coordinates to battlefield length, Indicates the Red Army's drone The ratio of coordinates to battlefield width, represents the speed scalar of the Red Team UAV, It represents the ratio of the heading angle of the red drone to , Indicates the distance between the Red Team UAV and the ground target point. Indicates the radius of the Red Team’s drone’s observable range. Indicates the angle between the line connecting the ground target point and the Red UAV and the heading of the Red UAV; Indicates the blue drone The ratio of coordinates to battlefield length, Indicates the blue drone The ratio of coordinates to battlefield width, Represents the speed scalar of the blue team’s drone, Indicates the heading angle of the blue drone and The ratio of It represents the angle between the line connecting the blue drone and the red drone and the heading of the red drone. Indicates the straight-line distance between the blue drone and the red drone.

[0110] The action space follows the design of the previous section, that is, the action space is specifically the track point of the drone at the next moment.

[0111] Similarly, the reward function can be specifically expressed as:

[0112] ;

[0113] In the formula, It represents the comprehensive instant reward obtained by the agent at the moment, Indicates the reward for hitting an enemy aircraft. Indicates the number of enemy aircraft hit; Indicates the penalty for hitting an obstacle or entering a threat area. Indicates the number of teammates. Indicates the penalty for being out of bounds when approaching the boundary. represents the gravitational force of the value field on the drone, Indicates the reward for closing the distance when chasing an enemy aircraft. Indicates the number of pursuits, It means punishment for being threatened directly by enemy aircraft. Indicates the number of enemy aircraft in front. , , and Represents the weight coefficient.

[0114] In addition, the discount factor used to calculate the long-term return is usually between 0 and 1. When the discount factor is close to 0, the agent pays more attention to the short-term return; when the discount factor is close to 1, the agent pays more attention to the long-term return. The relevant discount factor in this embodiment can be specifically set to 0.99.

[0115] The HAPPO algorithm is based on the PPO algorithm. Similar to the PPO algorithm, each agent has an Actor network. The neural network model Actor is the strategy network, and the input state information , output agent action ; All agents share a Critic network, input state , output state value .

[0116] Both Actor and Critic are fully connected networks. The number of network layers and the number of neurons in the network are hyperparameters and need to be determined based on specific scenario experiments.

[0117] Like the PPO algorithm, HAPPO needs to use a policy network to interact with the environment to generate a set of sample data, including information such as state, action, reward, and next state. The relevant data also needs to be stored in the experience recycling pool for subsequent training.

[0118] In the HAPPO algorithm, for any ordered agent The action value function of is defined as:

[0119] ;

[0120] In the formula, represents the action-value function, represents any ordered agent, represents the agent number, represents the agent action, Indicates strategy, express The combined action value of the agents inside and outside, Indicates the status, express The joint actions of the agents between express The joint actions of other agents, express The strategies of agents other than It represents the mean of the action-value function that calculates the action space to obey the strategy.

[0121] Based on the action value function, the advantage function of the multi-agent is obtained, which is used to evaluate the quality of each action relative to the average level, expressed as:

[0122] ;

[0123] In the formula, represents the advantage function, express and The joint action value of the agents in the interval, express The value of the joint action of the agents in the interval, express The joint actions of agents within the interval, express Joint actions of agents within the interval.

[0124] Using advantage functions helps improve learning efficiency and makes learning more stable. Experience also shows that advantage functions also help reduce variance, and excessive variance is an important factor leading to overfitting.

[0125] Based on the advantage function, the policy gradient of the multi-agent is obtained, which is expressed as:

[0126] ;

[0127] In the formula, represents the policy gradient, express The joint actions of agents within the interval, Representing an Agent The old strategy, Representing an Agent New strategies, represents the advantage function.

[0128] With the above policy gradient, the loss function of the actor network of the HAPPO algorithm can be defined accordingly:

[0129] ;

[0130] In the formula, represents the loss function of the Actor network, Indicates that the initial state follows the distribution , the agent's actions obey The mean of the advantage function, Indicates that the ratio of the new and old strategies is cut to Within, represents the hyperparameter, Representing an Agent The old strategy, Representing an Agent New strategies, Representing an Agent Actions, It means taking the minimum value among the elements in the brackets.

[0131] The loss function of the Critic network is consistent with the PPO algorithm, that is, the loss function of the Critic network can also be expressed as:

[0132] ;

[0133] In the formula, The neural network parameters are The loss function is It is expressed as the mean of the calculated state value differences, Represents the state value of the Critic network output, Indicates the target state value;

[0134] Wherein, the target state value is expressed as:

[0135] ;

[0136] In the formula, The discount factor Second power, Indicates The reward value of the step, The discount factor Second power, Indicates status The state value of The network parameters are State value function, express The state of the moment, Indicates the initial time.

[0137] Furthermore, an optimizer is used to minimize the loss function of HAPPO and update the neural network model parameters. Usually, optimizers such as Adam and SGD are used for optimization. Mini-batch and online update methods can be used for training.

[0138] Finally, based on the training results, determine whether the expected requirements are met. If not, adjust the reward function weight and related hyperparameters and continue training; otherwise, save the agent model in this scenario. The performance of the algorithm can be evaluated using indicators such as the reward function, the average reward of the strategy, and the state value function.

[0139] It is worth noting that the model parameter update of the mth agent in the HAPPO algorithm is calculated based on the results of the previous m-1 updates. During the training process, the upper-level agent makes decisions based on the performance of the lower-level agent and guides the behavior of the lower-level agent. This arrangement of sequential updates of the multi-agent model ensures that the HAPPO multi-agent policy gradient algorithm has the same monotonic convergence characteristics as the single-agent policy gradient algorithm.

[0140] In addition, to verify the effectiveness of the decision optimization method of the present invention, we designed a UAV combat scenario, used the above algorithm to train the corresponding model, found the optimal trajectory and the optimal combat decision, and placed the trained model in each scenario for testing. Figure 3 As shown in the figure, the drone first obtains information about the surrounding environment, stores historical data in the experience playback pool, calculates the current optimal tactical plan through continuous data sampling and strategy update, and finally commands the drone to perform trajectory planning and related tactical decisions. The architecture diagram of the drone confrontation strategy generation algorithm under multi-agent control is shown in the figure. Figure 4 As shown in the figure, the reward function is calculated based on the current state and the action selected by the drone. By maximizing the cumulative round reward value, the optimal tactical deduction result of this scenario is obtained.

[0141] This embodiment also performs simulation analysis on the solution proposed by the present invention. Figure 5 The results of the UAV's trajectory planning in a maritime combat scenario are shown. It can be seen that after multiple iterations of learning, the UAV can perform regularized modeling of obstacles based on the results of load perception and perform obstacle avoidance operations. The UAV will not leave immediately after reaching the destination, and will hover over the target point before completing the mission. The trajectory planning scheme proposed by the present invention is more in line with the actual scenario and the actual flight control situation of fixed-wing UAVs, and the experimental results have better authenticity and effectiveness. Figure 6 The corresponding design diagram of the red-blue confrontation simulation scenario is displayed. The red side attacks the blue side's units in different formations, with the goal of eventually reaching the target area defended by the blue side.

[0142] Example 3

[0143] This embodiment provides a drone swarm confrontation decision optimization device, which includes:

[0144] The information acquisition module is configured to acquire the target point location information, the location and status information of the red party UAV, and the location and status information of the blue party UAV;

[0145] The optimization model conversion module is configured to convert the pre-built UAV swarm confrontation decision optimization model into a Markov decision process based on a preset reward function and a discount factor, taking each UAV as a decision maker, taking the target point location information, the UAV location and situation information as the state, and taking the track point selected by the UAV as the action;

[0146] And, the strategy optimization module is configured to use each drone to use a pre-trained deep reinforcement neural network, and based on the target point location information, the location and state information of the red drone, and the location and state information of the blue drone, to solve the Markov decision process to obtain an optimized drone cluster confrontation strategy.

[0147] The drone swarm confrontation decision optimization device can execute the drone swarm confrontation decision optimization method introduced in the above-mentioned embodiment 1 or 2, and the specific functional implementation of each functional module will not be repeated.

[0148] Example 4

[0149] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the drone cluster confrontation decision optimization method described in Example 1 or Example 2 are implemented.

[0150] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0154] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A UAV swarm confrontation decision optimization method based on HAPPO algorithm, characterized in that: include: Obtain the target point location information, the location and status information of the red team’s drone, and the location and status information of the blue team’s drone; Taking each drone as a decision maker, the target point location information, the drone's location and situation information as the state, and the track point selected by the drone as the action, the pre-built drone swarm confrontation decision optimization model is transformed into a Markov decision process based on the preset reward function and discount factor; Each drone uses a pre-trained deep reinforcement neural network and solves the Markov decision process based on the target point location information, the location and state information of the red drone, and the location and state information of the blue drone to obtain an optimized drone cluster confrontation strategy; Among them, the reward function of the Markov decision process is expressed as: R t =R kill +R die +R bdr +R chs +R thr R kill =α*enemies R die =-β*allies R bdr =-ω*F(uav i ) R chs =ρ*attach R thr =-ρ*face In the formula, R t represents the comprehensive instant reward obtained by the agent at time t, R kill represents the reward for hitting an enemy aircraft, enemies represents the number of enemy aircraft hit; R die indicates the penalty for hitting an obstacle or entering a threat area, allies indicates the number of teammates, R bdr Indicates the penalty for out-of-bounds threat when approaching the boundary, F(uav i ) represents the gravitational force of the value field on the drone, R chs It indicates the reward for chasing an enemy aircraft. Attach indicates the number of pursuits. R thr It represents the penalty for being threatened by enemy aircraft head-on, face represents the number of enemy aircraft head-on, and α, β, ω and ρ represent weight coefficients.

2. The UAV swarm confrontation decision optimization method based on the HAPPO algorithm according to claim 1 is characterized in that: The state of the Markov decision process is expressed as: In the formula, s i represents the state space of agent i, x i represents the ratio of the x-coordinate of the red drone to the length of the battlefield, y i represents the ratio of the y coordinate of the red drone to the width of the battlefield, v i represents the speed scalar of the red drone, θ i represents the ratio of the heading angle of the red drone to π, d tar Indicates the distance between the Red Team UAV and the ground target point, d obs Indicates the radius of the observable range of the Red Team UAV, α tar Indicates the angle between the ground target point and the Red UAV’s connecting line and the Red UAV’s heading; x e Indicates the ratio of the blue drone's x coordinate to the battlefield length, y e represents the ratio of the blue drone's y coordinate to the battlefield width, v e Represents the speed scalar of the blue drone, θ e It represents the ratio of the heading angle of the blue drone to π. It represents the angle between the line connecting the blue drone and the red drone and the heading of the red drone. Indicates the straight-line distance between the blue drone and the red drone; The actions of the Markov decision process are: the track points of the drone at the next moment.

3. The UAV swarm confrontation decision optimization method based on the HAPPO algorithm according to claim 2 is characterized in that: The gravitational force of the value field on the drone is expressed as: In the formula, U(uav i ) represents the value of the value field to the agent, represents the gradient of the value field, ξ represents the value scale factor, val represents the value of the value body, a positive value represents a reward, a negative value represents a threat, d(uav i ) indicates UAV i distance from the value body; The value of the value field to the agent is expressed as:

4. The UAV swarm confrontation decision optimization method based on the HAPPO algorithm according to claim 1 is characterized in that: The pre-trained deep reinforcement neural network is trained using the HAPPO algorithm, where each agent has an Actor network and inputs state information s t , output agent action a t , all agents share a Critic network and input state information s t , output state value V(s t ), the Actor network and the Critic network are both fully connected networks, the corresponding number of network layers and the number of neurons in the network are hyperparameters, and the training process includes: Each agent uses the Actor network to interact with the environment and generate a set of sample data, which includes state, action, reward and next state, and defines the action value function of any ordered agent based on the sample data; Based on the action-value function, an advantage function of the multi-agent is obtained; and based on the advantage function, a policy gradient of the multi-agent is obtained; and further based on the policy gradient, a loss function of the Actor network is obtained; Calculate the state value and target state value output by the Critic network to obtain the loss function of the Critic network; Use the optimizer to minimize the corresponding loss function and update the parameters of the relevant neural network model; Based on the training results, determine whether the expected requirements are met. If not, adjust the reward function weights and related hyperparameters and continue training; otherwise, save the intelligent agent model in the current training scenario.

5. The UAV swarm confrontation decision optimization method based on the HAPPO algorithm according to claim 4 is characterized in that: The action value function is expressed as: In the formula, represents the action value function, i 1:m =(i1,…,i m ) represents any ordered agent, i represents the agent number, a represents the agent action, π represents the strategy, represents the joint action value of agents within and outside [1,m], S represents the state, represents the joint action of the agents between [1,m], represents the joint action of agents outside [1,m], represents the strategy of the agent outside [1,m], It means calculating the mean of the action value function of the action space that obeys the strategy π; Based on the action value function, the advantage function of the multi-agent is obtained, which is expressed as: In the formula, represents the advantage function, represents the joint action value of the agent in the interval [1,k] and [1,m], represents the joint action value of the agents in the interval [1, k], represents the joint action of agents in the interval [1, k], represents the joint action of agents in the interval [1,m]; Based on the advantage function, the policy gradient of the multi-agent is obtained, which is expressed as: Where M π (s,a) represents the policy gradient, represents the joint action of agents in the interval [1,m-1], Representing an Agent The old strategy, Representing an Agent A new strategy, π (s,a) represents the advantage function.

6. The UAV swarm confrontation decision optimization method based on the HAPPO algorithm according to claim 5 is characterized in that: The loss function of the Actor network is expressed as: Where L represents the loss function of the Actor network, Indicates that the initial state follows the distribution ρ π , the agent's actions obey the mean of the advantage function of π, Indicates that the ratio of the new and old strategies is clipped to 1±∈, where ∈ represents a hyperparameter. Represents agent i m The old strategy, Represents agent i m New strategies, Represents agent i m action, min(.) means taking the minimum value of the elements in the brackets.

7. The UAV swarm confrontation decision optimization method based on the HAPPO algorithm according to claim 6 is characterized in that: The loss function of the Critic network is expressed as: Where, L v (θ) represents the loss function with the neural network parameter θ, Expressed as the mean of the calculated state value differences, V θ (s) represents the state value of the Critic network output, V target Indicates the target state value; Wherein, the target state value is expressed as: In the formula, γ i-t represents the it-th power of the discount factor, r i represents the reward value of the i-th step, γ T-t represents the discount factor raised to the power of Tt, V θ (s T ) indicates state s T The state value, V θ (·) indicates that the network parameters are the state value function θ, s T represents the state at time T, and t represents the initial time.

8. A drone swarm confrontation decision optimization device based on the HAPPO algorithm, characterized in that: include: The information acquisition module is configured to acquire the target point location information, the location and status information of the red party UAV, and the location and status information of the blue party UAV; The optimization model conversion module is configured to convert the pre-built UAV swarm confrontation decision optimization model into a Markov decision process based on a preset reward function and a discount factor, taking each UAV as a decision maker, taking the target point location information, the UAV location and situation information as the state, and taking the track point selected by the UAV as the action; And, a strategy optimization module is configured to solve the Markov decision process for each drone using a pre-trained deep reinforcement neural network and based on the target point location information, the location and state information of the red drone, and the location and state information of the blue drone to obtain an optimized drone cluster confrontation strategy; Among them, the reward function of the Markov decision process is expressed as: R t =R kill +R die +R bdr +R chs +R thr R kill =α*enemies R die =-β*allies R bdr =-ω*F(uav i ) R chs =ρ*attach R thr =-ρ*face In the formula, R t represents the comprehensive instant reward obtained by the agent at time t, R kill represents the reward for hitting an enemy aircraft, enemies represents the number of enemy aircraft hit; R die indicates the penalty for hitting an obstacle or entering a threat area, allies indicates the number of teammates, R bdr Indicates the penalty for out-of-bounds threat when approaching the boundary, F(uav i ) represents the gravitational force of the value field on the drone, R chs It indicates the reward for chasing an enemy aircraft. Attach indicates the number of pursuits. R thr It represents the penalty for being threatened by enemy aircraft head-on, face represents the number of enemy aircraft head-on, and α, β, ω and ρ represent weight coefficients.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the drone swarm confrontation decision optimization method based on the HAPPO algorithm described in any one of claims 1 to 7.

Images