Multi-UAV coordinated attack and defense method and system based on local observation information

Abstract

The present invention discloses a method and system for coordinated attack and defense against multiple drones based on local observation information. The method includes: establishing a multi-drone attack and defense simulation training environment; determining information trajectory groups based on the local observation information of the drones in the simulation training environment; determining weights for the information trajectory groups based on the local observation information, and generating new trajectory information based on the weights; and determining drone actions based on the new trajectory information to control the drones. By reorganizing and integrating the information trajectories collected by the drones, the present invention provides more accurate representation of the local trajectory information.

Description

Technical Field

[0001] The present invention relates to the field of drone confrontation technology, and in particular to a multi-drone collaborative attack and defense confrontation method and system based on local observation information. Background Art

[0002] Drone confrontations will be a key component of future unmanned combat systems. Drones, with their low cost and high maneuverability, are primarily used for reconnaissance and strike missions. A single drone can only perform limited operations. To execute more complex missions and improve operational efficiency, multiple drones are often deployed to collaborate and coordinate offensive and defensive operations. Compared to single-drone control, multi-drone state and action space is larger, and mutual interference between drones complicates multi-drone control.

[0003] Traditional multi-UAV control methods often require the pre-determination of a large amount of rule information. These methods perform well in simple environments but struggle in complex ones. Reinforcement learning, a classic machine learning method for solving sequential decision-making, offers a promising approach to solving the multi-UAV coordinated attack and defense problem. This problem can be solved using a multi-agent reinforcement learning model. By constructing an interactive simulation environment and designing a suitable reward function, the swarm can learn an autonomous collaborative strategy to efficiently complete the task. Compared to traditional rule-based UAV control methods, multi-agent reinforcement learning methods offer greater robustness and flexibility.

[0004] Problems with existing multi-agent reinforcement learning:

[0005] 1. Due to the limitations of the natural environment and combat needs, it is difficult for drones to communicate with each other;

[0006] 2. The drone’s representation of the information it perceives is not accurate enough. Summary of the Invention

[0007] In order to solve the above problems, the present invention provides a multi-UAV collaborative attack and defense confrontation method and system based on local observation information.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A multi-UAV coordinated attack and defense method based on local observation information, comprising:

[0010] Construct a multi-UAV attack and defense simulation training environment; the simulation training environment includes: two UAV swarms, multiple obstacles, and the attack target locations of each party;

[0011] Determining an information trajectory group based on local observation information of the UAV in the simulation training environment;

[0012] determining a weight of the information trajectory group based on the local observation information, and generating new trajectory information based on the weight;

[0013] The drone action is determined based on the new trajectory information, and the drone is controlled.

[0014] Optionally, determining an information trajectory group based on local observation information of the UAV in the simulation training environment specifically includes:

[0015] Decomposing the local observation information;

[0016] Encode the decomposed local observation information;

[0017] The encoded information is input into the corresponding recurrent neural network to obtain an information trajectory group.

[0018] Optionally, determining a weight of the information trajectory group based on the local observation information, and generating new trajectory information based on the weight, specifically includes:

[0019] Determining a weight of the information trajectory group and a bias term corresponding to the weight based on the local observation information;

[0020] A Hadamard product is performed on the weight and the information trajectory group, and the product is added to the bias term to generate new trajectory information.

[0021] Optionally, determining the drone action based on the new trajectory information and controlling the drone specifically includes:

[0022] Determine a state action value according to the new trajectory information;

[0023] determining a global state action based on the state action value;

[0024] The drone is controlled based on the global state action.

[0025] The present invention also provides a multi-UAV coordinated attack and defense confrontation system based on local observation information, comprising:

[0026] A simulation training environment construction module is used to construct a multi-UAV attack and defense simulation training environment; the simulation training environment includes: two UAV clusters, multiple obstacles, and the attack target locations of each party;

[0027] An information trajectory group generation module is used to determine an information trajectory group based on local observation information of the UAV in the simulation training environment;

[0028] a trajectory weight generation module, configured to determine a weight of the information trajectory group based on the local observation information, and generate new trajectory information based on the weight;

[0029] The control module is used to determine the drone action based on the new trajectory information and control the drone.

[0030] Optionally, the information trajectory group generation module specifically includes:

[0031] a decomposition unit, configured to decompose the local observation information;

[0032] An encoding unit, used to encode the decomposed local observation information;

[0033] The input unit is used to input the encoded information into the corresponding recurrent neural network to obtain an information trajectory group.

[0034] Optionally, the trajectory weight generation module specifically includes:

[0035] a weight generating unit, configured to determine a weight of the information trajectory group and a bias term corresponding to the weight based on the local observation information;

[0036] The new trajectory information generating unit is configured to perform a Hadamard product on the weight and the information trajectory group, and add the product to the bias term to generate new trajectory information.

[0037] Optionally, the control module specifically includes:

[0038] a state-action value determining unit, configured to determine a state-action value according to the new trajectory information;

[0039] a global state action determining unit, configured to determine a global state action based on the state action value;

[0040] A control unit is used to control the drone based on the global state action.

[0041] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0042] This paper addresses the issue of insufficient utilization of local information in reinforcement learning control of multiple drones. It proposes a multi-drone collaborative attack and defense method and system based on local observation information. By reorganizing and integrating the information trajectories collected by the drones, the representation of local trajectory information is more accurate, significantly improving the algorithm's effectiveness. Compared to previous multi-drone collaborative attack and defense methods, this proposed method is suitable for complex adversarial environments where communication between drones is limited or even impossible, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 Flowchart of the multi-UAV coordinated attack and defense method based on local observation information provided by the present invention;

[0045] Figure 2 This is a schematic diagram of the multi-UAV coordinated attack and defense method based on local observation information provided by the present invention;

[0046] Figure 3 This is the overall flow chart of the multi-UAV collaborative attack and defense method based on local observation information provided by the present invention;

[0047] Figure 4 Schematic diagram of the multi-UAV attack and defense simulation training environment provided by the present invention;

[0048] Figure 5 A flow chart for generating an information trajectory group provided by the present invention;

[0049] Figure 6 A flow chart for generating weights of information trajectory groups provided by the present invention;

[0050] Figure 7 A flow chart for generating state-action values ​​provided by the present invention; DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example 1

[0054] like Figure 1-Figure 3 As shown, the multi-UAV coordinated attack and defense confrontation method based on local observation information provided by the present invention includes the following steps:

[0055] Step 101: Construct a multi-UAV attack and defense simulation training environment; the simulation training environment includes: two UAV clusters, multiple obstacles, and attack target locations of each party.

[0056] Construct a multi-UAV attack and defense simulation training environment. The simulation training environment mainly includes two UAV clusters, red and blue, multiple obstacles, and attack target locations of each party. Figure 2 As shown, m red drones and n blue drones will be randomly generated in the area, and the position coordinates of the i-th red drone are marked as The position coordinates of the j-th UAV of the blue team are marked as Three obstacle areas are randomly generated, and the obstacle coordinates are recorded as The drone must be able to actively avoid obstacles during flight; the red team and the blue team each have a headquarters R and B. The drone must protect its own base from being attacked by the enemy drone cluster while destroying the enemy headquarters to win.

[0057] Construct a multi-agent reinforcement learning model. Since drones are constrained by sensors, have some observability issues, and each drone needs to make independent decisions, the multi-drone control problem can be modeled as a multi-agent Dec-POMDP model for solution. This model consists of the following parts:

[0058] (1) Construct joint observation information O = {o1, o2, ..., o n} and global state s, where o n Represents each drone's observation of local information, including geographic information and the positions of friendly and enemy drones within the current field of view; the observable range of each drone is centered on its own coordinates and has a radius of r obs The global state s represents the set of all information in the environment at the current time t.

[0059] (2) Construct joint action information U = {u1, u2, ..., u n}, where u n Represents the actions that each drone can take. Drones can perform 5 actions: north, south, west, east, stop, and attack the target. Each drone has a health value of 5, an attack power of 1, a headquarters health value of 10, and an attack range of r attack , and satisfy the following constraints:

[0060]

[0061] (3) Construct the reward function R, which can be divided into the following parts: if the enemy drone is shot down, the reward is +1, and if the enemy headquarters is destroyed, the reward is +50; if our drone is shot down, the reward is -1, and if our headquarters is destroyed, the reward is -50; if the drone collides or hits an obstacle, the reward is -10.

[0062] (4) Construct the transfer function P(s'|s, U), which represents the transition probability of the UAV transferring to the next state s' after performing an action in state s.

[0063] Step 102: Determine an information trajectory group based on the local observation information of the UAV in the simulation training environment.

[0064] (1) Decompose the local observation information of the UAV. Figure 5 As shown in the figure, although the drone's own observation is limited by the sensor and the observation range is limited, the local observation still contains rich information. In order to represent the information more accurately, we first convert the local information that drone i can observe at time t into According to the component relationship, it is decomposed into three parts: status information about its own situation Status information of friendly drones within its field of view And the status information of enemy drones within its field of view As shown in the following formula:

[0065]

[0066] (2) Generate information trajectory group. Due to the limited sensing range of its own sensors, the drone can only observe the circle with itself as the center and radius r. obs To overcome this problem, Figure 5 As shown, first decompose the observation information of each part Encode and convert to the corresponding Then put it into the corresponding recurrent neural network to obtain the corresponding information trajectory group containing historical trajectory information and the corresponding hidden layer For input at the next moment, the overall process is as follows:

[0067]

[0068] Step 103: Determine the weight of the information trajectory group based on the local observation information, and generate new trajectory information based on the weight.

[0069] (1) Generate trajectory group weights. To more accurately represent trajectory information, providing a weight for each information trajectory group can more effectively emphasize the importance of each part in the current decision. Figure 6 As shown, the local range information that UAV i can perceive at time t is As input, a weight generation network generates three weights for the information trajectory group and the corresponding bias term b.

[0070] (2) Recombining the trajectories. According to the following formula, the three weights Each historical information track in the corresponding information track group Do the Hadamard product and compare it with b t Adding together, we can get the reorganized new trajectory information

[0071]

[0072] Step 104: Determine the drone action based on the new trajectory information and control the drone.

[0073] like Figure 7 As shown, according to the reorganized new trajectory information Input it into a multi-layer perceptron model, which is a neural network whose input is trajectory information This generates the corresponding state action value

[0074] like Figure 3 As shown, the state action value function generated by each drone is Input into the hybrid network and obtain the global state action value function Q according to the following calculation formula tot (τ, u, s; θ), where w i (s t ) and b(s t ) is the global information s at the current moment t The weights and biases of the corresponding value function are obtained through a hypernetwork. A hypernetwork is also a neural network.

[0075]

[0076] Combine the current moment reward r and the global state action value function at the next moment as the update target, recorded as: The time difference method is used to update the network, and the objective function is recorded as:

[0077]

[0078] Among them, r is the reward reward obtained by the drone group through the reward function R at the current time t; θ is the hybrid network Q totThe network parameters, the calculation method of the network target maximum function at the next moment and the global state action value function Q tot The calculation method of (τ, u, s; θ) remains the same, the difference is that the parameters are changed to the next moment

[0079] L is the standard temporal difference loss function for multi-agent reinforcement learning, which is used to train and optimize the parameters of all networks in this invention.

[0080] Compared with the prior art, the present invention has the following advantages:

[0081] 1. To address the problem of insufficient utilization of local information in reinforcement learning control of multiple UAVs, this paper proposes a multi-UAV collaborative attack and defense method based on local observation information. By reorganizing and integrating the information trajectories collected by the UAVs, the representation of local trajectory information is made more accurate, which can significantly improve the algorithm effect.

[0082] 2. Compared with previous multi-UAV collaborative attack and defense methods, the method proposed in this invention is suitable for complex confrontation environments where communication between UAVs is limited or even impossible, and has broad application prospects.

[0083] Example 2

[0084] In order to execute the method corresponding to the above embodiment 1 and achieve the corresponding functions and technical effects, a multi-UAV collaborative attack and defense confrontation system based on local observation information is provided below.

[0085] The system includes:

[0086] The simulation training environment construction module is used to build a multi-UAV attack and defense simulation training environment.

[0087] The information trajectory group generation module is used to determine the information trajectory group based on the local observation information of the drone in the simulation training environment. Specifically, it includes a decomposition unit for decomposing the local observation information; an encoding unit for encoding the decomposed local observation information; and an input unit for inputting the encoded information into the corresponding recurrent neural network to obtain the information trajectory group.

[0088] The trajectory weight generation module is configured to determine the weights of the information trajectory group based on the local observation information and to generate new trajectory information based on the weights. Specifically, it includes: a weight generation unit, configured to determine the weights of the information trajectory group and the corresponding bias terms based on the local observation information; and a new trajectory information generation unit, configured to perform a Hadamard product between the weights and the information trajectory group and add the product to the bias term to generate new trajectory information.

[0089] The control module is configured to determine the drone's actions based on the new trajectory information and control the drone. Specifically, it includes: a state action value determination unit configured to determine a state action value based on the new trajectory information; a global state action determination unit configured to determine a global state action based on the state action value; and a control unit configured to control the drone based on the global state action.

[0090] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0091] This article uses specific examples to illustrate the principles and implementation methods of the invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

Claims

1. A multi-UAV coordinated attack and defense method based on local observation information, characterized in that: include: Build a multi-UAV attack and defense simulation training environment; The simulation training environment includes: two drone swarms, multiple obstacles, and target locations for attack by each party; Determining an information trajectory group based on local observation information of the UAV in the simulation training environment; determining a weight of the information trajectory group based on the local observation information, and generating new trajectory information based on the weight; Determine the drone's motion based on the new trajectory information and control the drone; Based on the local observation information in the joint observation information, the information trajectory group is determined, specifically including: Decomposing the local observation information; Encode the decomposed local observation information; The encoded information is input into the corresponding recurrent neural network to obtain an information trajectory group.

2. The multi-UAV coordinated attack and defense method based on local observation information according to claim 1 is characterized in that: Determining the weight of the information trajectory group based on the local observation information, and generating new trajectory information based on the weight, specifically including: Determining a weight of the information trajectory group and a bias term corresponding to the weight based on the local observation information; A Hadamard product is performed on the weight and the information trajectory group, and the product is added to the bias term to generate new trajectory information.

3. The multi-UAV coordinated attack and defense method based on local observation information according to claim 1 is characterized in that: Determining the drone's motion based on the new trajectory information and controlling the drone specifically includes: Determine a state action value according to the new trajectory information; determining a global state action based on the state action value; The drone is controlled based on the global state action.

4. A multi-UAV coordinated attack and defense system based on local observation information, characterized in that: include: A simulation training environment construction module, used to build a multi-UAV attack and defense simulation training environment; The simulation training environment includes: two drone swarms, multiple obstacles, and target locations for attack by each party; An information trajectory group generating module, configured to determine an information trajectory group based on local observation information of the UAV in the simulation training environment; a trajectory weight generation module, configured to determine a weight of the information trajectory group based on the local observation information, and generate new trajectory information based on the weight; A control module, configured to determine the UAV's motion based on the new trajectory information and control the UAV; The information trajectory group generation module specifically includes: a decomposition unit, configured to decompose the local observation information; An encoding unit, used to encode the decomposed local observation information; The input unit is used to input the encoded information into the corresponding recurrent neural network to obtain an information trajectory group.

5. The multi-UAV coordinated attack and defense system based on local observation information according to claim 4 is characterized in that: The trajectory weight generation module specifically includes: a weight generating unit, configured to determine a weight of the information trajectory group and a bias term corresponding to the weight based on the local observation information; The new trajectory information generating unit is configured to perform a Hadamard product on the weight and the information trajectory group, and add the product to the bias term to generate new trajectory information.

6. The multi-UAV coordinated attack and defense system based on local observation information according to claim 4 is characterized in that: The control module specifically includes: a state-action value determining unit, configured to determine a state-action value according to the new trajectory information; a global state action determining unit, configured to determine a global state action based on the state action value; A control unit is used to control the drone based on the global state action.

Images