Heterogeneous multi-UAV cooperative confrontation strategy generation method based on hierarchical reinforcement learning

Abstract

The present invention relates to a method for generating a heterogeneous multi-UAV collaborative confrontation strategy based on hierarchical reinforcement learning, and belongs to the field of multi-agent deep reinforcement learning. The present invention comprises the following steps: constructing a collaborative confrontation environment; building a heterogeneous multi-UAV collaborative confrontation strategy network model based on hierarchical reinforcement learning; training the heterogeneous multi-UAV collaborative confrontation strategy network model based on the collaborative confrontation environment; loading the trained heterogeneous multi-UAV collaborative confrontation strategy network model to achieve heterogeneous multi-UAV collaborative confrontation. Based on hierarchical reinforcement learning, the present invention constructs a "total-total" type heterogeneous multi-UAV collaborative confrontation strategy network model to achieve the fusion of global state information and individual state information, and balances the individual action value and the overall value through deep neural network fitting, which is conducive to the generation of heterogeneous multi-UAV collaborative confrontation strategies, and the confrontation strategy training process is clear, and the screening process of the optimal confrontation strategy is simple and easy to implement.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent deep reinforcement learning, and in particular to a method for generating a heterogeneous multi-UAV collaborative confrontation strategy based on hierarchical reinforcement learning. Background Art

[0002] In recent years, with the extensive use of unmanned equipment and platforms, the demand for the application of drone control technology has increased. In particular, heterogeneous drones with different functions and performance characteristics can use their respective advantages to cooperate and collaborate with each other to jointly cope with complex confrontation mission requirements. How to use artificial intelligence to achieve intelligent collaborative confrontation of heterogeneous drones has become a key issue in the current research on intelligent strategy generation.

[0003] In the article "Heterogeneous Multi-UAV Collaborative Penetration Method Based on MA2IDDPG Algorithm" published by Chang Xin et al. in Hebei Industrial Science and Technology, inspired by the course learning, each UAV independently carries the same DDPG network framework. After traction training and descriptive training, the DDPG network of each UAV is optimized to form a heterogeneous multi-UAV collaborative penetration capability that integrates expert experience knowledge and artificial intelligence. This method replaces the information interaction between UAVs with traction training based on expert experience knowledge, so that the UAVs can form collaborative capabilities even though the information is independent. This method relies on traction training to form collaborative capabilities, has a high demand for domain knowledge, and is not flexible enough compared to the method of generating strategies from data.

[0004] In the patent "A method for multi-UAV coordinated penetration based on heterogeneous multi-agent reinforcement learning" applied by the 54th Institute of China Electronics Technology Group Corporation, with the authorization announcement number CN113822409B, a traction game scenario and a result description game scenario are constructed. The heterogeneous multi-UAV based on reinforcement learning is first trained in the traction game scenario, and then the network is migrated to the result description game scenario for further training, so that the deep neural network can further explore better confrontation strategies based on objective data on the basis of quickly generating initial strategies. However, the neural networks carried by each UAV in the patent are independent of each other, but they use rewards for evaluating the overall effect, which limits the generation and optimization of the coordinated penetration strategy.

[0005] In the patent "Allocation Method and Device for Coordinated Low-Altitude Penetration of Multiple Heterogeneous UAVs" applied by the Aviation Engineering University of the Strategic Support Force of the Chinese People's Liberation Army, with the authorization announcement number CN117008641B, a UAV strategy allocation model is constructed to effectively allocate tasks and plans to each UAV according to fitness, thereby achieving the coordination of multiple heterogeneous UAVs and resolving allocation conflicts between UAVs. However, this patent is based on preset rules and allocation strategies, and there is a lot of domain knowledge and expert experience. As the number of UAVs increases and the complexity of the scene increases, the preset rules and strategies may become difficult to manage and maintain.

[0006] In the article "Multi-agent Game Strategy Generation Method Based on Hierarchical Reinforcement Learning" published by Chang Xin et al. in Radio Engineering, an intelligent multi-agent strategy generation model framework was constructed based on hierarchical reinforcement learning, and multi-agent game strategies were generated through training. This method maps global information to control information through the top-level strategy control model, and then combines the control information and individual perception information to obtain action value pairs through a deep neural network. Finally, the overall value is estimated by accumulating the maximum individual action value, and based on this, the neural network parameters are optimized through time difference to form an effective game confrontation strategy. However, this method has only been experimented in the StarCraft 2 game environment, which is quite different from the drone application environment, and the direct accumulation to obtain the estimate of the overall value is too simple to express the relationship between each individual and the whole, which is not conducive to strategy generation. Summary of the invention

[0007] In view of this, the present invention proposes a method for generating a heterogeneous multi-UAV collaborative confrontation strategy based on hierarchical reinforcement learning. This method uses a deep neural network and hierarchical reinforcement learning to construct a "total-divide-to-total" type heterogeneous multi-UAV collaborative confrontation strategy network model, balances the relationship between individual action value and overall value, and generates a heterogeneous multi-UAV collaborative confrontation strategy through training.

[0008] The technical solution adopted by the present invention is:

[0009] A method for generating a heterogeneous multi-UAV collaborative confrontation strategy based on hierarchical reinforcement learning includes the following steps:

[0010] Step 1: Build a collaborative confrontation environment;

[0011] Step 2: Build a heterogeneous multi-UAV collaborative confrontation strategy network model based on hierarchical reinforcement learning;

[0012] Step 3: Train the heterogeneous multi-UAV collaborative confrontation strategy network model based on the collaborative confrontation environment;

[0013] Step 4: Load the trained heterogeneous multi-UAV collaborative confrontation strategy network model to realize heterogeneous multi-UAV collaborative confrontation.

[0014] Furthermore, the specific method of step 1 is:

[0015] Step 1-1: Set the collaborative confrontation environment as a two-dimensional rectangular area, the lower left corner of the two-dimensional rectangular area is located at the origin of the coordinate system, and the length and width are along the X-axis and Y-axis respectively; the heterogeneous drones in the collaborative confrontation environment are divided into red and blue parties, each party includes 1 jamming drone, 2 attack drones and 1 base; the position of the red jamming drone is ( , ), the survival flag is ; The position of the Red side's first strike drone is ( , ), the survival flag is ; The position of the Red side's second strike drone is ( , ), the survival flag is ; The location of the Red side's base is ( , ), the survival flag is ; The position of the blue jamming drone is ( , ), the survival flag is ; The position of the Blue side's first strike drone is ( , ), the survival flag is ; The position of the second strike drone of the blue team is ( , ), the survival flag is ; The location of the blue base is ( , ), the survival flag is Jamming drones and strike drones can move, strike drones can destroy jamming drones, strike drones and bases, jamming drones can suppress the destruction range of strike drones, and the survival flag is True by default;

[0016] Step 1-2: Set the destruction rules: Set the destruction distance threshold to , the destruction angle threshold is set to ;

[0017] For the Red side's first strike drone, when the following relationship is satisfied with the Blue side unit m, m = 0, 1, 2, 3, the Red side's first strike drone will delete the Blue side unit m from the collaborative confrontation environment and set the survival flag to Set to False:

[0018]

[0019]

[0020] a = [ x ′ m , t − x 1 , t , y ′ m , t − y 1 , t ]

[0021] b = [ x 1 , t − x 1 , t − 1 , y 1 , t − y 1 , t − 1 ]

[0022] in( , ) is the position of the first strike drone of the Red Army at time t, ( , ) is the position of the first strike drone of the Red Army at time t-1, ( , ) is the position of the blue unit m at time t;

[0023] For the Red side's second strike drone, when the following relationship is satisfied with the Blue side unit m, m=0, 1, 2, 3, the Red side's second strike drone will delete the Blue side unit m from the collaborative confrontation environment and set the survival flag to Set to False:

[0024]

[0025]

[0026] a = [ x ′ m , t − x 2 , t , y ′ m , t − y 2 , t ]

[0027] b = [ x 2 , t − x 2 , t − 1 , y 2 , t − y 2 , t − 1 ]

[0028] in( , ) is the position of the second strike drone of the Red Army at time t, ( , ) is the position of the second strike drone of the Red Army at time t-1, ( , ) is the position of the blue unit m at time t;

[0029] For the Blue side's first strike drone, when the following relationship is satisfied with the Red side unit n, n = 0, 1, 2, 3, the Blue side's first strike drone will delete the Red side unit n from the collaborative confrontation environment and set the survival flag to Set to False:

[0030]

[0031]

[0032] a = [ x ′ 1 , t − x n , t , y ′ 1 , t − y n , t ]

[0033] b = [ x ′ 1 , t − x ′ 1 , t − 1 , y ′ 1 , t − y ′ 1 , t − 1 ]

[0034] in( , ) is the position of the first attack drone of the blue team at time t, ( , ) is the position of the first attack drone of the blue team at time t-1, ( , ) is the position of the red unit n at time t;

[0035] For the Blue side's second strike drone, when the following relationship is satisfied with the Red side unit n, n = 0, 1, 2, 3, the Blue side's second strike drone will delete the Red side unit n from the collaborative confrontation environment and set the survival flag to Set to False:

[0036]

[0037]

[0038] a = [ x ′ 2 , t − x n , t , y ′ 2 , t − y n , t ]

[0039] b = [ x ′ 2 , t − x ′ 2 , t − 1 , y ′ 2 , t − y ′ 2 , t − 1 ]

[0040] in( , ) is the position of the second strike drone of the blue team at time t, ( , ) is the position of the second strike drone of the blue team at time t-1, ( , ) is the position of the red unit n at time t;

[0041] Step 1-3: Set the interference rule: Set the interference distance threshold to , the interference angle threshold is set to ;

[0042] For the red jamming drone, when the following relationship is satisfied with the blue unit m, m=0, 1, 2, 3, the destruction distance threshold of the blue attack drone is Become 1 / 2 of the initial value:

[0043]

[0044]

[0045] a = [ x ′ m , t − x 0 , t , y ′ m , t − y 0 , t ]

[0046] b = [ x 0 , t − x 0 , t − 1 , y 0 , t − y 0 , t − 1 ]

[0047] in( , ) is the position of the Red Army’s jamming drone at time t, ( , ) is the position of the Red Army’s jamming drone at time t-1, ( , ) is the position of the blue unit m at time t;

[0048] For the blue jamming drone, when the following relationship is satisfied with the red unit n, n=0, 1, 2, 3, the destruction distance threshold of the red attack drone Become 1 / 2 of the initial value:

[0049]

[0050]

[0051] a = [ x n , t − x ′ 0 , t , y n , t − y ′ 0 , t ]

[0052] b = [ x ′ 0 , t − x ′ 0 , t − 1 , y ′ 0 , t − y ′ 0 , t − 1 ]

[0053] in( , ) is the position of the blue jammer drone at time t, ( , ) is the position of the blue jammer drone at time t-1, ( , ) is the position of the red unit n at time t;

[0054] Step 1-4: Set the movement rules: except for the base, all other red units move at a two-dimensional speed ( , ) to update the position; set the speed threshold to , the position update rule of the red unit is:

[0055]

[0056]

[0057] Where n=0, 1, 2;

[0058] Step 1-5: Build status information: Assemble the red unit information and opponent information into a list , provided to the intelligent model for strategy generation, where , The corresponding overall status information is

[0059] o whole = [ x 0 , y 0 , a 0 , x 1 , y 1 , a 1 , x 2 , y 2 , a 2 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 3 ]

[0060] Corresponding to the individual status information of the Red Army’s jamming drone, Corresponding to the individual status information of the Red Army's first strike drone, Individual information corresponding to the Red Army's second strike drone:

[0061] o 0 = [ x 0 , y 0 , a 0 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 4 ]

[0062] o 1 = [ x 1 , y 1 , a 1 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 3 ]

[0063] o 2 = [ x 2 , y 2 , a 2 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 3 ]

[0064] Step 1-6: Construct a reward function: Based on the destruction rule, when the Red team's base is within the destruction distance and destruction angle threshold of the Blue team's first strike drone and the second strike drone, set the corresponding reward value, the confrontation ends, the end flag bit done is set to True, and the victory flag bit win is set to False; otherwise, when the Blue team's first strike drone and the second strike drone are destroyed, that is, and When both are False, the corresponding reward value is set, the confrontation ends, the end flag bit done is set to True, and the victory flag bit win is set to True;

[0065] Step 1-7: Set up the blue team rule strategy model:

[0066] The blue team interferes with the drone's movement on the X axis for:

[0067]

[0068] The blue team interferes with the drone's movement on the Y axis for:

[0069]

[0070] The movement of the Blue team's first strike drone on the X axis for:

[0071]

[0072] The movement of the Blue Team's first strike drone on the Y axis for:

[0073]

[0074] The movement of the blue side's second strike drone on the X axis for:

[0075]

[0076] The movement of the blue team's second strike drone on the Y axis for:

[0077] .

[0078] Furthermore, the specific method of step 2 is:

[0079] Step 2-1: Build the top-level control network; top-level control network It uses 3 layers of fully connected layers in series, and the network parameters are , to achieve overall status information Mapping to control information I;

[0080] Step 2-2: Build an individual control network; individual control network A single fully connected layer is used, and the network parameters are , , realize control information I and individual information Individual action value The number of individual discrete actions is , then the output dimension of the individual control network is , that is, the individual action value The dimension is ;

[0081] Step 2-3: Build a collective value network; collective value network A single fully connected layer is used to realize the individual action value of the individual control network Towards collective values The mapping of has a dimension of 1;

[0082] Step 2-4: Based on the top-level control network, individual control network and collective value network, a heterogeneous multi-UAV collaborative confrontation strategy network model is formed to realize the overall state information and individual information Individual action value and collective values Mapping.

[0083] Furthermore, the specific method of step 3 is:

[0084] Step 3-1: Set the number of training times , the number of evaluations is Win rate threshold ;

[0085] Step 3-2: Define the action space for interacting with the collaborative adversarial environment as [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] ;

[0086] Construct random action generation function and intelligent action generation function; random action generation function is from Randomly select a value from and ; The intelligent action generation function converts the overall state information and individual information Obtaining individual action value through heterogeneous multi-UAV cooperative confrontation strategy network model , and obtain the maximum individual action value Corresponding action label , then from , , and In accordance with the label The corresponding order values ​​are assigned to the red side's jamming drone and , the Red Army's first strike drone and With the Red Side's Second Strike Drone and ;

[0087] Step 3-3: Read the overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ;

[0088] Step 3-4: Set random number ϕ ∈ [ 0 , 1 ] , the generated action space is:

[0089] [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] = { Follow machine move do born become letter number ϕ < 0 . 1 wisdom able move do born become letter number ϕ ≥ 0 . 9

[0090] The action space interacts with the collaborative confrontation environment to obtain updated state information, reward value reward_r, end flag done and victory flag win; the updated state information includes the overall state information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ;

[0091] Step 3-5: Combine the state information, action, reward value and state information after executing the action to form experience r= [ o whole , o 0 , o 1 , o 2 , a 0 , a 1 , a 2 , reward _ r , o ′ whole , o ′ 0 , o ′ 1 , o ′ 2 ] , stored in the experience pool;

[0092] Step 3-6: When the number of experiences r stored in the experience pool is greater than L, B pieces of experience data are randomly extracted from it, and the neural network parameters are updated. Based on the temporal difference algorithm, the loss value is defined as:

[0093] loss = Q 0 ( a 0 ) + Q 1 ( a 1 ) + Q 2 ( a 2 ) − Q total ( o ) − α [ reward _ r + max Q ′ 0 + max Q ′ 1 + max Q ′ 2 − Q total ( o ′ ) ]

[0094] in, o = [ o whole , o 0 , o 1 , o 2 ] It is the status information at the current moment; For The collective value of The action label to be executed at the current state The corresponding individual action value; is the reward decreasing function; reward_r is the reward value after the action label is executed; o ′ = [ o ′ whole , o ′ 0 , o ′ 1 , o ′ 2 ] It is the status information after executing the action; For The maximum individual action value; For The collective value of

[0095] Based on the loss value, the neural network parameters are optimized through the Adam optimizer;

[0096] Step 3-7: When the end flag done is True, the number of training times num_train is increased by 1 to determine whether the current number of training times has reached the preset number of training times If the preset number of training times is not reached, the collaborative confrontation environment is initialized and the process returns to step 3-4; if the preset number of training times is reached, the number of training times num_train, the number of tests num_eval, and the number of wins num_win are cleared, and the process goes to step 3-8;

[0097] Step 3-8: Read the current overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ;

[0098] Step 3-9: Generate action space using intelligent action generation function [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] , interact with the collaborative confrontation environment and obtain updated status information, reward value reward_r, end flag done and victory flag win; the updated status information includes the overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ;

[0099] Step 3-10: When the end flag bit done is True, the number of tests num_eval is increased by 1; if the confrontation is successful, that is, the victory flag bit win is True, the number of wins num_win is increased by 1, otherwise return to step 3-9;

[0100] Step 3-11: Determine whether the current test number num_eval reaches the evaluation number , if the number of evaluations is reached , then calculate the winning rate num_win / num_eval and go to step 3-12, otherwise return to step 3-9;

[0101] Step 3-12: If the winning rate reaches the preset winning rate threshold, save the model parameters and complete the training; otherwise, return to step 3-3.

[0102] Furthermore, the specific method of step 4 is:

[0103] Step 4-1: Read the trained model parameters;

[0104] Step 4-2: Read the overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ;

[0105] Step 4-3: Generate action space using intelligent action generation function [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] , interact the action space with the collaborative confrontation environment, obtain the updated state information, reward value reward_r, end flag done and victory flag win; the updated state information includes the overall state information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ;

[0106] Step 4-4: If the end flag done is True, the confrontation is completed; otherwise, return to step 4-3.

[0107] The beneficial effects of the present invention are:

[0108] 1. The confrontation strategy training process of the present invention is clear, and the screening process of the optimal confrontation strategy is simple and easy to implement.

[0109] 2. The present invention adopts a "total-division-total" type heterogeneous multi-UAV collaborative confrontation strategy network model, which can realize the fusion of global state information and individual state information, and balances the individual action value and the overall value through deep neural network fitting, which is conducive to the generation of heterogeneous multi-UAV collaborative confrontation strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0111] Figure 2 Flow chart of step 3 in the embodiment of the present invention.

[0112] Figure 3 This is a diagram showing the relationship between the number of repetitions and the winning rate.

[0113] Figure 4 Schematic diagram of the relationship between the number of repetitions and rewards.

[0114] Figure 5 A roadmap for collaborative confrontation decision making. DETAILED DESCRIPTION

[0115] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0116] The overall process of generating a strategy for heterogeneous multi-UAV cooperative confrontation based on hierarchical reinforcement learning is as follows: Figure 1 As shown, the specific steps include:

[0117] Step 1: Build a collaborative confrontation environment;

[0118] Step 1-1: Set the scene scope and heterogeneous drones;

[0119] The collaborative confrontation scene is a two-dimensional rectangular area. The length and width of the confrontation scene are along the X-axis and Y-axis respectively. The length of the confrontation scene is , ; The width of the adversarial scene is , =10.

[0120] In the confrontation scenario, the heterogeneous drones are divided into the red side and the blue side. Each side includes 1 jamming drone, 2 attack drones and 1 base, so the total number of drones on each side N is 3. The positions of the jamming drones on the red side are and , survival flag ; The positions of the Red side attacking drone 1 are and , the survival flag is ; The positions of the Red side's attack drone 2 are and , the survival flag is ; The location of the Red side's base is and , the survival flag is ; The positions of the blue jammer drones are and , the survival flag is ; The positions of the blue team's attack drone 1 are and , the survival flag is ; The positions of the blue team's attack drone 2 are and , the survival flag is ; The location of the blue base is and , the survival flag is Among them, jamming drones and strike drones can move, strike drones can destroy jamming drones, strike drones and bases, and jamming drones can suppress the destruction range of strike drones.

[0121] Step 1-2: Destroy the rules;

[0122] The destroy distance threshold is set to =2, the destruction angle threshold is set to =30°.

[0123] For the Red side's first strike drone, when the following relationship is satisfied with the Blue side unit m, m = 0, 1, 2, 3, the Red side's first strike drone will delete the Blue side unit m from the collaborative confrontation environment and set the survival flag to Set to False:

[0124]

[0125]

[0126] a = [ x ′ m , t − x 1 , t , y ′ m , t − y 1 , t ]

[0127] b = [ x 1 , t − x 1 , t − 1 , y 1 , t − y 1 , t − 1 ]

[0128] in( , ) is the position of the first strike drone of the Red Army at time t, ( , ) is the position of the first strike drone of the Red Army at time t-1, ( , ) is the position of the blue unit m at time t;

[0129] For the Red side's second strike drone, when the following relationship is satisfied with the Blue side unit m, m=0, 1, 2, 3, the Red side's second strike drone will delete the Blue side unit m from the collaborative confrontation environment and set the survival flag to Set to False:

[0130]

[0131]

[0132] a = [ x ′ m , t − x 2 , t , y ′ m , t − y 2 , t ]

[0133] b = [ x 2 , t − x 2 , t − 1 , y 2 , t − y 2 , t − 1 ]

[0134] in( , ) is the position of the second strike drone of the Red Army at time t, ( , ) is the position of the second strike drone of the Red Army at time t-1, ( , ) is the position of the blue unit m at time t;

[0135] For the Blue side's first strike drone, when the following relationship is satisfied with the Red side unit n, n = 0, 1, 2, 3, the Blue side's first strike drone will delete the Red side unit n from the collaborative confrontation environment and set the survival flag to Set to False:

[0136]

[0137]

[0138] a = [ x ′ 1 , t − x n , t , y ′ 1 , t − y n , t ]

[0139] b = [ x ′ 1 , t − x ′ 1 , t − 1 , y ′ 1 , t − y ′ 1 , t − 1 ]

[0140] in( , ) is the position of the first attack drone of the blue team at time t, ( , ) is the position of the first attack drone of the blue team at time t-1, ( , ) is the position of the red unit n at time t;

[0141] For the Blue side's second strike drone, when the following relationship is satisfied with the Red side unit n, n = 0, 1, 2, 3, the Blue side's second strike drone will delete the Red side unit n from the collaborative confrontation environment and set the survival flag to Set to False:

[0142]

[0143]

[0144] a = [ x ′ 2 , t − x n , t , y ′ 2 , t − y n , t ]

[0145] b = [ x ′ 2 , t − x ′ 2 , t − 1 , y ′ 2 , t − y ′ 2 , t − 1 ]

[0146] in( , ) is the position of the second strike drone of the blue team at time t, ( , ) is the position of the second strike drone of the blue team at time t-1, ( , ) is the position of the red unit n at time t;

[0147] Step 1-3: Interference rules;

[0148] The interference distance threshold is set to =3, the interference angle threshold is set to =30°.

[0149] For the red jamming drone, when the following relationship is satisfied with the blue unit m, m=0, 1, 2, 3, the destruction distance threshold of the blue attack drone is Become 1 / 2 of the initial value:

[0150]

[0151]

[0152] a = [ x ′ m , t − x 0 , t , y ′ m , t − y 0 , t ]

[0153] b = [ x 0 , t − x 0 , t − 1 , y 0 , t − y 0 , t − 1 ]

[0154] in( , ) is the position of the Red Army’s jamming drone at time t, ( , ) is the position of the Red Army’s jamming drone at time t-1, ( , ) is the position of the blue unit m at time t;

[0155] For the blue jamming drone, when the following relationship is satisfied with the red unit n, n=0, 1, 2, 3, the destruction distance threshold of the red attack drone Become 1 / 2 of the initial value:

[0156]

[0157]

[0158] a = [ x n , t − x ′ 0 , t , y n , t − y ′ 0 , t ]

[0159] b = [ x ′ 0 , t − x ′ 0 , t − 1 , y ′ 0 , t − y ′ 0 , t − 1 ]

[0160] in( , ) is the position of the blue jammer drone at time t, ( , ) is the position of the blue jammer drone at time t-1, ( , ) is the position of the red unit n at time t;

[0161] Steps 1-4: Maneuvering rules;

[0162] Except for the base, which cannot move, all entities have a 2D velocity. and Update the position. Set the speed threshold to is 0.5, the position update rule of the red entity is:

[0163]

[0164]

[0165] Step 1-5: State information construction;

[0166] Assemble each drone n information and opponent information into a list , provided to the intelligent model for strategy generation. Corresponding to the overall status information, specifically:

[0167] o whole = [ x 0 , y 0 , a 0 , x 1 , y 1 , a 1 , x 2 , y 2 , a 2 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 3 ]

[0168] Corresponding to the individual status information of the Red Army’s jamming drone, Corresponding to the individual status information of the Red Army's strike drone 1, Corresponding to the individual information of the Red Team's strike drone 2:

[0169] o 0 = [ x 0 , y 0 , a 0 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 4 ]

[0170] o 1 = [ x 1 , y 1 , a 1 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 3 ]

[0171] o 2 = [ x 2 , y 2 , a 2 , x 3 , y 3 , a 3 , x ′ 0 , y ′ 0 , a ′ 0 , x ′ 1 , y ′ 1 , a ′ 1 , x ′ 2 , y ′ 2 , a ′ 2 , x ′ 3 , y ′ 3 , a ′ 3 ]

[0172] Step 1-6: Reward function construction;

[0173] Based on the destruction rules of step 1-2, when the Red team's base is within the destruction distance and destruction angle threshold of the Blue team's attack drone 1 and attack drone 2, the reward value reward_r is set to -10, the confrontation ends, the end flag bit done is set to True, and the victory flag bit win is set to False; otherwise, when the Blue team's attack drone 1 and attack drone 2 are destroyed, that is, and When both are False, the reward value reward_r is set to 100, the confrontation ends, the end flag done is set to True, and the victory flag win is set to True.

[0174] Step 1-7: Load the blue team rule strategy model for the red team intelligent model to generate adversarial strategies through training. The blue team rule strategy model is set as follows:

[0175] The blue team interferes with the drone's movement on the X axis for:

[0176]

[0177] The blue team interferes with the drone's movement on the Y axis for:

[0178]

[0179] The movement of the blue attack drone 1 on the X axis for:

[0180]

[0181] The movement of the blue attack drone 1 on the Y axis for:

[0182]

[0183] The movement of the blue team's attack drone 2 on the X axis for:

[0184]

[0185] The movement of the blue team's attack drone 2 on the Y axis for:

[0186] .

[0187] Step 2: Build a heterogeneous multi-UAV collaborative confrontation strategy network model based on hierarchical reinforcement learning;

[0188] Step 2-1: Build the top-level control network;

[0189] Top-level control network It is built based on a deep neural network, and its neural network parameters are , to achieve overall status information Mapping to control information I:

[0190]

[0191] The top-level control network is built in series with three fully connected layers, and the parameters are shown in the following table:

[0192]

[0193] Step 2-2: Build an individual control network;

[0194] Individual Control Network It is built based on a deep neural network, and its neural network parameters are , , realize control information I and individual information Individual action value The number of individual discrete actions is , then the output dimension of the individual control network is =4, that is, the value of individual action The dimension is :

[0195]

[0196] The individual control network uses a single fully connected layer, and the parameters are shown in the following table:

[0197]

[0198] Step 2-3: Build a collective value network;

[0199] Collective Value Network Based on deep neural network construction, the individual action value of the individual control network is realized Towards collective values The mapping of has dimension 1:

[0200]

[0201] The collective value network uses a single fully connected layer with the parameters shown in the following table:

[0202]

[0203] Step 2-4: Construct a heterogeneous multi-UAV collaborative confrontation strategy network model;

[0204] Based on the top-level control network, individual control network and collective value network, a heterogeneous multi-UAV collaborative confrontation strategy network model is formed to realize the overall state information and individual information Individual action value and collective values Mapping:

[0205] { Q 0 = f 0 [ o 0 , f I ( o whole ) ] Q 1 = f 1 [ o 1 , f I ( o whole ) ] Q 2 = f 2 [ o 2 , f I ( o whole ) ] Q total = f total ( Q 0 , Q 1 , Q 2 )

[0206] Step 3: Train the network model based on the heterogeneous UAV collaborative confrontation environment. The process is as follows: Figure 2 As shown, specifically including:

[0207] Step 3-1: Set training parameters;

[0208] Set the number of training times 90, the number of evaluations =10 and win rate threshold is 100%.

[0209] Step 3-2: Define the action space for interacting with the collaborative adversarial environment as [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] .

[0210] Construct random action generation function and intelligent action generation function; random action generation function is from Randomly select a value from and .

[0211] The intelligent action generation function converts the overall state information and individual information Obtaining individual action value through heterogeneous multi-UAV cooperative confrontation strategy network model , get the maximum individual action value Corresponding action label , then from , , and In accordance with the label The corresponding order values ​​are assigned to the red side's jamming drone and , Red side attacks drone 1 and With Red Team Attack Drone 2 and .

[0212] Step 3-3: Get status information;

[0213] Read overall status information , Red Army jamming drone individual status information 、Individual status information of Red side attack drone 1 Individual information of Red Team's strike drone 2 .

[0214] Step 3-4: Set random number ϕ ∈ [ 0 , 1 ] , the generated action space is:

[0215] [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] = { Follow machine move do born become letter number ϕ < 0 . 1 wisdom able move do born become letter number ϕ ≥ 0 . 9

[0216] The action space interacts with the environment to obtain updated state information, reward value reward_r, end flag done and victory flag win. The updated state information includes the overall state information , Red Army jamming drone individual status information 、Individual status information of Red side attack drone 1 Individual information of Red Team's strike drone 2 .

[0217] Step 3-5: Store the state information, action, reward value and state information after executing the action in the experience pool. The stored experience r is [ o whole , o 0 , o 1 , o 2 , a 0 , a 1 , a 2 , reward _ r , o ′ whole , o ′ 0 , o ′ 1 , o ′ 2 ] .

[0218] Step 3-6: When the number of experiences r stored in the experience pool is greater than L=2000, B=64 pieces of experience data are randomly extracted from it to update the neural network parameters.

[0219] Based on the temporal difference algorithm, the loss value is defined as:

[0220] loss = Q 0 ( a 0 ) + Q 1 ( a 1 ) + Q 2 ( a 2 ) − Q total ( o ) − α [ reward _ r + max Q ′ 0 + max Q ′ 1 + max Q ′ 2 − Q total ( o ′ ) ]

[0221] in, o = [ o whole , o 0 , o 1 , o 2 ] It is the status information at the current moment; For The collective value of The action label to be executed at the current state The corresponding individual action value; is the reward decreasing function; reward_r is the reward value after the action label is executed; o ′ = [ o ′ whole , o ′ 0 , o ′ 1 , o ′ 2 ] It is the status information after executing the action; For The maximum individual action value; For The collective value at that time. Based on the loss value loss, the neural network parameters are optimized by the Adam optimizer, and the optimizer learning rate is is 0.01.

[0222] Step 3-7: When the end flag done is True, the number of training times num_train is increased by 1 to determine whether the current number of training times has reached the preset number of training times ; If the preset number of training times is not reached, the initialization scene returns to step 3-4; if the preset number of training times is reached, the number of training times num_train is cleared, the number of tests num_eval is cleared, the number of wins num_win is cleared, and step 3-8 is entered.

[0223] Step 3-8: Get status information

[0224] Read the current overall status information , Red Army jamming drone individual status information 、Individual status information of Red side attack drone 1 Individual information of Red Team's strike drone 2 .

[0225] Step 3-9: Generate action space using intelligent action generation function [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] , the action space interacts with the environment and obtains the updated state information, reward value reward_r, end flag done and victory flag win. The updated state information includes the overall state information , Red Army jamming drone individual status information 、Individual status information of Red side attack drone 1 Individual information of Red Team's strike drone 2 .

[0226] Step 3-10: When the end flag bit done is True, the number of tests num_eval increases by 1. If the confrontation is successful, that is, the victory flag bit win is True, the number of wins num_win increases by 1.

[0227] Step 3-11: Determine whether the current test number num_eval reaches the evaluation number If the number of evaluations is reached , then calculate the winning rate num_win / num_eval, otherwise, return to step 3-9. The relationship between the number of repetitions and the winning rate is as follows Figure 3 As shown in Figure 4 As shown. Figure 3 It can be seen that after training, a collaborative confrontation strategy is generated with a winning rate of 100%.

[0228] Step 3-12: If the winning rate reaches the preset winning rate threshold, save the model; otherwise, return to step 3-3;

[0229] Step 4: Load the trained neural network model to achieve collaborative confrontation of heterogeneous multi-UAVs.

[0230] Step 4-1: Read the trained neural network model parameters;

[0231] Step 4-2: Get status information;

[0232] Read overall status information , Red Army jamming drone individual status information 、Individual status information of Red side attack drone 1 Individual information of Red Team's strike drone 2 .

[0233] Step 4-3: Generate action space based on intelligent action generation function and interact with the environment;

[0234] Generate action space using intelligent action generation function [ x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ] , the action space interacts with the environment to obtain the updated state information, reward value reward_r, end flag done and victory flag win. The updated state information includes the overall state information , Red Army jamming drone individual status information 、Individual status information of Red side attack drone 1 Individual information of Red Team's strike drone 2 .

[0235] Step 4-4: If the end flag bit done is True, the confrontation is completed; otherwise, return to step 4-3.

[0236] Coordinated confrontation decision-making roadmap Figure 5 shown.

[0237] The present invention is based on hierarchical reinforcement learning to construct a "total-division-total" type heterogeneous multi-UAV collaborative confrontation strategy network model, realizes the fusion of global state information and individual state information, and balances the individual action value and the overall value through deep neural network fitting, which is conducive to the generation of heterogeneous multi-UAV collaborative confrontation strategies. The confrontation strategy training process is clear, and the screening process of the optimal confrontation strategy is simple and easy to implement.

[0238] In summary, the present invention uses deep neural networks for decision-making, optimizes the deep neural networks that meet the winning rate conditions to achieve strategy generation, and realizes the collaborative confrontation of heterogeneous multi-UAVs.

Claims

1. A method for generating a cooperative countermeasure strategy for heterogeneous multi-UAVs based on hierarchical reinforcement learning, characterized in that: The following steps are involved: Step 1: Build a collaborative confrontation environment; the specific methods are: Step 1-1: Set the collaborative confrontation environment as a two-dimensional rectangular area, the lower left corner of the two-dimensional rectangular area is located at the origin of the coordinate system, and the length and width are along the X-axis and Y-axis respectively; the heterogeneous drones in the collaborative confrontation environment are divided into red and blue parties, each party includes 1 jamming drone, 2 attack drones and 1 base; the position of the red jamming drone is ( , ), the survival flag is ; The position of the Red side's first strike drone is ( , ), the survival flag is ; The position of the Red side's second strike drone is ( , ), the survival flag is ; The Red side's base location is ( , ), the survival flag is ; The position of the blue jamming drone is ( , ), the survival flag is ; The position of the first strike drone of the blue team is ( , ), the survival flag is ; The position of the second strike drone of the blue team is ( , ), the survival flag is ; The location of the blue base is ( , ), the survival flag is Jamming drones and strike drones can move, strike drones can destroy jamming drones, strike drones and bases, jamming drones can suppress the destruction range of strike drones, and the survival flag is True by default; Step 1-2: Set the destruction rules: Set the destruction distance threshold to , the destruction angle threshold is set to ; For the Red side's first strike drone, when the following relationship is satisfied with the Blue side unit m, m = 0, 1, 2, 3, the Red side's first strike drone will delete the Blue side unit m from the collaborative confrontation environment and set the survival flag to Set to False: ； ； ； ； in( , ) is the position of the first strike drone of the Red Army at time t, ( , ) is the position of the first strike drone of the Red Army at time t-1, ( , ) is the position of the blue unit m at time t; For the Red side's second strike drone, when the following relationship is satisfied with the Blue side unit m, m=0, 1, 2, 3, the Red side's second strike drone will delete the Blue side unit m from the collaborative confrontation environment and set the survival flag to Set to False: ； ； ； ； in( , ) is the position of the second strike drone of the Red Army at time t, ( , ) is the position of the second strike drone of the Red Army at time t-1, ( , ) is the position of the blue unit m at time t; For the Blue side's first strike drone, when the following relationship is satisfied with the Red side unit n, n = 0, 1, 2, 3, the Blue side's first strike drone will delete the Red side unit n from the collaborative confrontation environment and set the survival flag to Set to False: ； ； ； ； in( , ) is the position of the first attack drone of the blue team at time t, ( , ) is the position of the first attack drone of the blue team at time t-1, ( , ) is the position of the red unit n at time t; For the Blue side's second strike drone, when the following relationship is satisfied with the Red side unit n, n = 0, 1, 2, 3, the Blue side's second strike drone will delete the Red side unit n from the collaborative confrontation environment and set the survival flag to Set to False: ； ； ； ； in( , ) is the position of the second strike drone of the blue team at time t, ( , ) is the position of the second strike drone of the blue team at time t-1, ( , ) is the position of the red unit n at time t; Step 1-3: Set the interference rule: Set the interference distance threshold to , the interference angle threshold is set to ; For the red jamming drone, when the following relationship is satisfied with the blue unit m, m=0, 1, 2, 3, the destruction distance threshold of the blue attack drone is Become 1 / 2 of the initial value: ； ； ； ； in( , ) is the position of the Red Army’s jamming drone at time t, ( , ) is the position of the Red Army’s jamming drone at time t-1, ( , ) is the position of the blue unit m at time t; For the blue jamming drone, when the following relationship is satisfied with the red unit n, n=0, 1, 2, 3, the destruction distance threshold of the red attack drone Become 1 / 2 of the initial value: ； ； ； ； in( , ) is the position of the blue jammer drone at time t, ( , ) is the position of the blue jammer drone at time t-1, ( , ) is the position of the red unit n at time t; Step 1-4: Set the movement rules: except for the base, all other red units move at a two-dimensional speed ( , ) to update the location; Set the speed threshold to , the position update rule of the red unit is: ； ； Where n=0, 1, 2; Step 1-5: Build status information: Assemble the red unit information and opponent information into a list , provided to the intelligent model for strategy generation, where , The corresponding overall status information is ； Corresponding to the individual status information of the Red Army’s jamming drone, Corresponding to the individual status information of the Red Army's first strike drone, Individual information corresponding to the Red Army's second strike drone: ； ； ； Step 1-6: Construct a reward function: Based on the destruction rule, when the Red team's base is within the destruction distance and destruction angle threshold of the Blue team's first strike drone and the second strike drone, set the corresponding reward value, the confrontation ends, the end flag bit done is set to True, and the victory flag bit win is set to False; otherwise, when the Blue team's first strike drone and the second strike drone are destroyed, that is, and When both are False, the corresponding reward value is set, the confrontation ends, the end flag bit done is set to True, and the victory flag bit win is set to True; Step 1-7: Set up the blue team rule strategy model: The blue team interferes with the drone's movement on the X axis for: ； The blue team interferes with the drone's movement on the Y axis for: ； The movement of the Blue Team's first strike drone on the X-axis for: ； The movement of the Blue Team's first strike drone on the Y axis for: ； The movement of the blue side's second strike drone on the X axis for: ； The movement of the blue team's second strike drone on the Y axis for: ； Step 2: Build a heterogeneous multi-UAV collaborative confrontation strategy network model based on hierarchical reinforcement learning; the specific method is: Step 2-1: Build the top-level control network; top-level control network It uses 3 layers of fully connected layers in series, and the network parameters are , to achieve overall status information Mapping to control information I; Step 2-2: Build an individual control network; individual control network A single fully connected layer is used, and the network parameters are , , realize control information I and individual information Individual action value The number of individual discrete actions is , then the output dimension of the individual control network is , that is, the individual action value The dimension is ; Step 2-3: Build a collective value network; collective value network A single fully connected layer is used to realize the individual action value of the individual control network Towards collective values The mapping of has a dimension of 1; Step 2-4: Based on the top-level control network, individual control network and collective value network, a heterogeneous multi-UAV collaborative confrontation strategy network model is formed to realize the overall state information and individual information Individual action value and collective values Mapping; Step 3: Train the heterogeneous multi-UAV collaborative confrontation strategy network model based on the collaborative confrontation environment; the specific method is: Step 3-1: Set the number of training times , the number of evaluations is Win rate threshold ; Step 3-2: Define the action space for interacting with the collaborative adversarial environment as ; Construct random action generation function and intelligent action generation function; random action generation function is from Randomly select a value from and ; The intelligent action generation function converts the overall state information and individual information Obtaining individual action value through heterogeneous multi-UAV cooperative confrontation strategy network model , and obtain the maximum individual action value Corresponding action label , then from , , and In accordance with the label The corresponding order values ​​are assigned to the red side's jamming drone and , the Red Army's first strike drone and With the Red Side's Second Strike Drone and ; Step 3-3: Read the overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ; Step 3-4: Set random number , the generated action space is: ； The action space interacts with the collaborative confrontation environment to obtain updated state information, reward value reward_r, end flag done and victory flag win; the updated state information includes the overall state information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ; Step 3-5: Combine the state information, action, reward value and state information after executing the action to form experience r= , stored in the experience pool; Step 3-6: When the number of experiences r stored in the experience pool is greater than L, B pieces of experience data are randomly extracted from it, and the neural network parameters are updated. Based on the temporal difference algorithm, the loss value is defined as: ； in, It is the status information at the current moment; For The collective value of The action label to be executed at the current state The corresponding individual action value; is the reward decreasing function; reward_r is the reward value after the action label is executed; It is the status information after executing the action; For The maximum individual action value; For The collective value of Based on the loss value, the neural network parameters are optimized through the Adam optimizer; Step 3-7: When the end flag done is True, the number of training times num_train is increased by 1 to determine whether the current number of training times has reached the preset number of training times If the preset number of training times is not reached, the collaborative confrontation environment is initialized and the process returns to step 3-4. If the preset number of training times is reached, the number of training times num_train, the number of tests num_eval, and the number of wins num_win are cleared, and the process goes to step 3-8. Step 3-8: Read the current overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ; Step 3-9: Generate action space using intelligent action generation function , interact with the collaborative confrontation environment and obtain updated status information, reward value reward_r, end flag done and victory flag win; the updated status information includes the overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ; Step 3-10: When the end flag bit done is True, the number of tests num_eval is increased by 1; if the confrontation is successful, that is, the victory flag bit win is True, the number of wins num_win is increased by 1, otherwise return to step 3-9; Step 3-11: Determine whether the current test number num_eval reaches the evaluation number , if the number of evaluations is reached , then calculate the winning rate num_win / num_eval and go to step 3-12, otherwise return to step 3-9; Step 3-12: If the winning rate reaches the preset winning rate threshold, save the model parameters and complete the training; otherwise, return to step 3-3; Step 4: Load the trained heterogeneous multi-UAV collaborative confrontation strategy network model to realize heterogeneous multi-UAV collaborative confrontation.

2. The method for generating a heterogeneous multi-UAV collaborative confrontation strategy based on hierarchical reinforcement learning according to claim 1 is characterized in that: The specific method of step 4 is: Step 4-1: Read the trained model parameters; Step 4-2: Read the overall status information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ; Step 4-3: Generate action space using intelligent action generation function , interact the action space with the collaborative confrontation environment, obtain the updated state information, reward value reward_r, end flag done and victory flag win; the updated state information includes the overall state information , Red Army jamming drone individual status information , Individual status information of the Red side's first strike drone And the individual information of the Red Army's second strike drone ; Step 4-4: If the end flag done is True, the confrontation is completed; otherwise, return to step 4-3.

Images