Unmanned aerial vehicle cluster autonomous cooperative anti-interference method based on hierarchical reinforcement learning

Abstract

The invention relates to the technical field of unmanned aerial vehicle path planning, and discloses an unmanned aerial vehicle cluster autonomous cooperative anti-interference method based on hierarchical reinforcement learning. According to the method, a multi-objective optimization model is established, constraints of energy consumption, communication power and channel selection are set for the model, a layered decoupling framework is adopted, and path planning, role allocation and communication control are divided into an upper strategy and a lower strategy for optimization. According to upper-layer path planning, a heterogeneous graph attention network serves as a graph neural network to achieve state representation, and then a robust trajectory is generated in combination with topological value prediction and a soft actor-commentator algorithm; and the lower-layer strategy network realizes dynamic role reconstruction and adaptive spectrum interference avoidance by constructing parallel role allocation and communication control sub-networks. And finally, efficient cooperation and network toughness maintenance of the large-scale unmanned aerial vehicle cluster in a strong interference environment are realized in a centralized training and decentralized execution mode.

Description

A hierarchical reinforcement learning-based autonomous cooperative anti-interference method for UAV swarms Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) path planning technology, specifically a hierarchical reinforcement learning-based autonomous collaborative anti-interference method for UAV swarms, which addresses the problem that UAV networks are highly susceptible to various forms of malicious interference, self-interference, and environmental interference threats. Background Technology

[0002] Current research indicates that anti-jamming systems are based on physical hardening, but physical hardening itself is static, passive, and non-adaptive. This makes it unable to independently cope with increasingly intelligent jamming threats.

[0003] To compensate for the passivity of physical hardening, the research introduced a centralized collaboration paradigm. While the centralized collaboration paradigm can theoretically achieve global optimization, in practice, it faces not only the fatal challenge of a "single point of failure" but also fundamental flaws in scalability. Therefore, in scenarios involving ultra-large-scale drone swarms, the system performance of the centralized collaboration paradigm will drop sharply, or even fail completely.

[0004] As for the single-agent paradigm, while it solves the "single point of failure" problem of the centralized collaboration paradigm, it falls into the dilemma of "local optima" and the challenge of "non-steady-state" environment. Such dilemmas and challenges limit the application prospects of the agent paradigm in multi-agent systems.

[0005] By combining the advantages of the previous two approaches and avoiding their disadvantages, agent reinforcement learning (MARL) achieves more robust and efficient anti-interference capabilities than single-agent learning. It can achieve more comprehensive situational awareness and more accurate joint decision-making. However, due to the difficulty in convergence of traditional reinforcement learning algorithms, computational complexity and memory consumption increase exponentially, leading to excessively long training times for agent reinforcement learning, even becoming uncomputable in ultra-large-scale clusters. Besides this theoretical scalability challenge, balancing the quantity and quality of information sharing while ensuring collaborative effects is also a practical challenge for MARL. Summary of the Invention

[0006] This invention proposes a hierarchical reinforcement learning-based method for autonomous cooperative anti-interference in unmanned aerial vehicle (UAV) swarms. This method aims to address the challenges of achieving efficient autonomous cooperation, task execution, and network resilience maintenance in large-scale UAV swarms operating under decentralized control and highly dynamic, complex interference environments. By decoupling complex decision-making problems into hierarchical strategies at different time scales, and utilizing advanced graph neural networks and multi-agent reinforcement learning techniques, this method enables UAV swarms to generate globally optimal cooperative behavior relying only on local information.

[0007] A hierarchical reinforcement learning-based autonomous cooperative anti-interference method for UAV swarms includes the following steps:

[0008] Step 1: Establish a collaborative anti-interference system and physical model for UAV swarms, define the UAV ensemble and its state variables, establish regional division and grid set, construct task and data flow models, establish communication links and dynamic topology, and set kinematic, energy, communication power and channel selection constraints;

[0009] Step 2: Construct a hierarchical and decoupled distributed multi-objective optimization problem, define optimization objectives such as task completion degree, network survivability, and task completion time, and divide the decision-making process into upper-level path planning and lower-level role allocation and communication control;

[0010] Step 3: Design a state representation method based on heterogeneous graph attention network. At each time step, construct a local heterogeneous graph and aggregate neighborhood information through a meta-path-guided attention mechanism to obtain low-dimensional state embeddings.

[0011] Step 4: Design a higher-level path planning method based on topological value internalization, establish a topological value prediction model, and use the soft actor-commentator algorithm (SAC) to learn the path planning strategy.

[0012] Step 5: Design a lower-level decompositional policy network for functional reconfiguration, construct a role allocation sub-policy network and an anti-interference communication sub-policy network, and output the UAV functional roles and communication parameters respectively;

[0013] Step 6: Adopt a centralized training and decentralized execution paradigm based on semantic awareness. In the training phase, a centralized commentator network with semantic enhancement is introduced, and in the execution phase, a distributed actor network makes independent decisions to achieve autonomous collaborative anti-interference of the drone swarm.

[0014] Preferably, step 1 further includes:

[0015] Step 1.1: Define the drone swarm and state variables The state of each drone at any given time is determined by its location. ,speed acceleration ,attitude Remaining energy Local data buffer status and functional roles constitute;

[0016] Step 1.2: Establish the task and data flow model: [This refers to the reconnaissance area.] Discretized into a set of grids The drone generates data packets based on the role when covering the grid. And it achieves single-hop or multi-hop transmission through communication links;

[0017] Step 1.3: Establish a physical model of communication and interference: The quality of the communication link is quantified by the SINR at the receiver, at time... , from Send to The signal, in The received SINR value at the location is denoted as The calculation formula is as follows:

[0018]

[0019] in: It is a drone In the channel Upward drone The power of the transmitted signal; It is a drone The transmitting antenna is relative to Direction angle Antenna gain; It is a drone The receiving antenna is relative to Direction angle Antenna gain; From arrive Channel gain; Representative receiver Background Gaussian white noise power at that location; Refers to the receiver In the channel Above, received signals from other drones within the cluster (excluding useful signal sources). The sum of co-channel interference (excluding external interference); its specific calculation formula is:

[0020]

[0021] in: Represents any other communication link in the cluster. It is an indicator function; This is the sum of interference caused by malicious or non-malicious interference sources outside the cluster; assume there exists a set consisting of M external interference sources. Each interference source Its state is determined by its position and transmission power. and the frequency set of attacks To characterize; then the receiver In the channel The total power of external interference received is:

[0022]

[0023] in: Is the interference source antenna in Gain in direction, From the source of interference to Channel gain;

[0024] Channel selection is derived from a limited set of spectrum resources, and link transmission rate is calculated based on Shannon's formula;

[0025]

[0026] Step 1.4: Establish a dynamic network topology model: Abstract the communication relationships of UAVs into a dynamic graph at any given time. And impose kinematic, energy, communication power, and channel selection constraints:

[0027] , ,

[0028]

[0029] .

[0030] Preferably, step 2 further includes:

[0031] Step 2.1: Define optimization goals: including maximizing task completion rate.

[0032]

[0033] Maximize the network survivability index

[0034]

[0035] and minimize task completion time

[0036] ;

[0037] Step 2.2: Propose a hierarchical decoupling optimization framework. The upper-layer strategy performs path planning at the macro time scale, while the lower-layer strategy performs functional role allocation and anti-interference communication at the micro time scale, so as to balance task execution efficiency and network long-term survivability.

[0038] Preferably, step 3 further includes:

[0039] Step 3.1: Constructing a Local Heterogeneous Graph The node is a drone, and its characteristics include: Position, speed Remaining energy Buffer occupancy rate ,Role And network contribution;

[0040]

[0041] Side attributes include channel, SINR, and link capacity. ;

[0042] Step 3.2: Use the Heterogeneous Graph Attention Network (HetGAT) to aggregate neighbor information, distinguish different communication semantics through the meta-path mechanism, and generate low-dimensional state embeddings.

[0043] Preferably, step 4 further includes:

[0044] Step 4.1: Construct the Topological Value Prediction Model (TVPM) using graph convolutional networks and recurrent neural networks for dynamic graphs.

[0045]

[0046] Embedded temporal modeling is used to predict the future value of local network topology;

[0047] Step 4.2: Quantitative definition and model training of topological value, quantifying topological value:

[0048]

[0049] Then, supervised learning is used to train TVPM;

[0050] Step 4.3: Based on predicted topological value: We designed a reward shaping mechanism; we will predict the value As a potential function Embedding reinforcement learning reward functions;

[0051]

[0052] Step 4.4: Select the Soft Actor-Critic Algorithm Learning Path (SAC) for policy learning. By using a dual critic network, the target value is always estimated using a smaller Q value, which alleviates the problem of Q value overestimation and improves the stability and robustness of the value learning process and path planning.

[0053] Preferably, step 5 further includes:

[0054] Step 5.1: Design the instant reward function

[0055]

[0056] Includes rewards for successful communication, penalties for communication failure, and penalties for transmission power.

[0057] Step 5.2: Construct a role allocation sub-policy network and use the Gumbel-Softmax method to implement discrete role selection for collectors, relayers, and backhaulers;

[0058] Step 5.3: Construct an anti-interference communication sub-strategy network, output channel selection and power allocation parameters, and realize spectrum awareness and adaptive interference avoidance.

[0059] Preferably, step 6 further includes:

[0060] Step 6.1: During the training phase, a semantically enhanced centralized critic network is used. We take the global state and joint action as input and combine role information to perform joint value estimation.

[0061] Step 6.2: The execution phase is run independently by the distributed actor network. Each drone generates state embeddings based on the local heterogeneous graph and outputs flight actions, role selection and communication parameters, thereby achieving autonomous collaboration under decentralized control conditions.

[0062] Compared with the prior art, the beneficial effects of the present invention are:

[0063] 1. This invention proposes a hierarchical reinforcement learning-based autonomous collaborative anti-interference method for UAV swarms. This method aims to address the challenges of achieving efficient autonomous collaboration, task execution, and network resilience maintenance in large-scale UAV swarms operating under decentralized control and highly dynamic, complex interference environments.

[0064] 2. This method decouples complex decision-making problems into hierarchical strategies at different time scales and utilizes advanced graph neural networks and multi-agent reinforcement learning techniques to enable UAV swarms to generate globally optimal cooperative behavior with only local information. Attached Figure Description

[0065] Figure 1 is a flowchart of the method of the present invention;

[0066] Figure 2 is a schematic diagram of the autonomous collaborative anti-interference of the UAV swarm according to the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Referring to Figures 1 and 2, a hierarchical reinforcement learning-based autonomous cooperative anti-interference method for UAV swarms includes the following steps:

[0069] Step 1: Establish a collaborative anti-interference system and physical model for drone swarms; we define a drone swarm. Each drone unit The state at time t is determined by the tuple.

[0070] This indicates that the control input is... Introducing regional division With grid sets Establish a task and data flow model, with the communication link consisting of... By definition, network topology is abstracted as a dynamic graph. And it follows kinematic, energy, communication power and channel selection constraints.

[0071] Step 2: Construct a hierarchical and decoupled distributed multi-objective optimization problem: We constructed a total of three objectives, namely: control policy sequence Collaborative optimization of task completion Network survivability and task completion time In addition, to address the coupling of the decision space, we divide the problem into upper-level path planning and decision-making. Assignment to lower-level roles Communication control .

[0072] Step 3: Design a state representation method based on heterogeneous graph attention network; we construct a local heterogeneous graph and node set at each time t. Each node corresponds to a drone, and the initial feature vector of the node... Edge type set Distinguish link semantics; aggregate neighborhood information through a meta-path-guided attention mechanism, and finally obtain a unified low-dimensional state embedding. .

[0073] Step 4: Design a higher-level path planning method based on topological value internalization; we first construct a topological value prediction model. Input a local dynamic image Output predicted topology value The predicted value is used as the potential function. Combined with potential rewards to form:

[0074]

[0075] Then, the soft actor-critic algorithm (SAC) is used to learn the path planning strategy. Through a dual critic network , Realize the value estimate.

[0076] Step 5: Design a lower-level decompositional policy network oriented towards functional refactoring; we can use an immediate reward function:

[0077]

[0078] Construct a role allocation sub-policy network, inputting... Then output the character These correspond to {collector, relayer, and returner} respectively; simultaneously, we construct an anti-interference communication sub-policy network in parallel and output communication parameters. ,in For channel selection, For power allocation.

[0079] Step 6: Adopt a semantically aware centralized training and decentralized execution paradigm; we introduce a semantically enhanced centralized commentator during the training phase. ,by The input is used to evaluate value; and during the execution phase, distributed actors are retained, with each drone generating a state embedding based on a local heterogeneous graph. Output , , Ultimately, this will achieve fully decentralized autonomous collaboration.

[0080] Furthermore, step 1 is detailed as follows:

[0081] Step 1.1: System Structure and Definition of State Variables: First, we define a large-scale cluster consisting of N drones, denoted as a set. , where N is the total number of drone units in the cluster. At any given time... Each drone unit in the cluster The instantaneous state is composed of a tuple. Provide a complete description:

[0082]

[0083] In this state tuple, each component is defined as follows:

[0084] It is a drone A spatial position vector in a three-dimensional Cartesian coordinate system, with the unit being meters (m), used to describe its precise geographical location. It is a drone The velocity vector is expressed in meters per second (m / s), and the first derivative of its position vector is represented by the velocity vector. It is a drone The acceleration vector, measured in meters per second squared (m / s²), represents the rate of change of its velocity vector. It describes drones The parameters of an aircraft's attitude are typically represented using quaternions or rotation matrices to determine its orientation in space. It is a drone The remaining energy, measured in joules (J), is a non-negative real number that represents the endurance of the system to continue performing its mission. It is a drone The local data buffer is typically a first-in-first-out (FIFO) queue used to temporarily store data packets that have been collected or are waiting to be relayed. It is a drone At any moment The functional role undertaken. Roles are selected from a predefined discrete set, including... These represent the data collector, data forwarder, and data relay, respectively.

[0085] Accordingly, we define each drone At any moment The controllable decision variable, i.e., its control input, is determined by a control vector. express:

[0086]

[0087] The components of this control vector are defined as follows:

[0088] Represents flight control inputs, used to directly control the physical motion of the drone. This includes target acceleration. Or target angular velocity, and perform path planning by changing the position and attitude of the UAV. These represent communication strategy parameters used to control the radio transmission and reception behavior of the drone, and will be defined in detail in the communication model. The role-switching command is a discrete decision used to determine the drone's next action. The new character I'm about to switch to .

[0089] Step 1.2: Task and Data Flow Model: The core task addressed by this invention is regional reconnaissance and information feedback. First, we define the wide geographical area to be reconnoitered. Discretized into a finite set of non-overlapping grid cells, forming a set. , where K is the total number of grid cells.

[0090] The data generation process is defined as follows: when a data generator is a "data collector" ( drones The ground projection position at time Located in a grid cell where information has not yet been collected Within that time, the drone will automatically generate a data packet. Each data packet has a fixed size. (Unit: bits), immediately stored in the drone after generation. Local data buffer Waiting for transmission.

[0091] The data transmission process is defined as follows: Data packets stored in the buffer need to be routed through the UAV network via a wireless communication link, using single-hop or multi-hop relay, and are ultimately transmitted to one or more designated receiving nodes. The receiving node can be a UAV returning after completing its mission, a designated aerial aggregation node, or a ground base station.

[0092] The criterion for determining task completion is: when all grid cells... All of them were collected by at least one drone, and all their corresponding data packets The reconnaissance mission is considered complete once all data has been successfully transmitted to the designated receiving node. The moment of mission completion is recorded as... .

[0093] Step 1.3: Communication and Interference Physical Model: This step details the physical process of point-to-point electromagnetic wave interaction between UAVs and models complex internal and external interference. This applies to any two different UAVs in the cluster. and Whether a communication link can be established between them depends on the sender. Communication decision And complex wireless channel environments. We specifically define the communication decision vector as... ,in It is from a finite set of available channels The selected communication channel frequency for this link is shown in the figure. This is the transmission power allocated for this transmission, measured in watts (W).

[0094] It is a drone In the channel Upward drone The power of the transmitted signal. It is a drone The transmitting antenna is relative to Direction angle Antenna gain, a dimensionless parameter, describes the antenna's ability to directionally radiate energy in space. It is a drone The receiving antenna is relative to Direction angle Antenna gain. From arrive Channel gain is a comprehensive parameter that reflects the power attenuation of a signal during propagation, encompassing various physical effects such as path loss, shadowing fading, and multipath fading. Its value is closely related to the distance between UAVs, their attitude, and obstacles in the environment. Representative receiver The background Gaussian white noise power at that location is typically considered a constant related to the ambient temperature. Refers to the receiver In the channel Above, received signals from other drones within the cluster (excluding useful signal sources). The sum of co-channel interference (from outside). The specific calculation formula is:

[0095]

[0096] The definitions and explanations of the physical quantities in the above formula are as follows:

[0097] Represents any other communication link in the cluster. It is an indicator function, when the link Channel used With Link Channel used If the frequencies are the same, the value is 1; otherwise, it is 0. This means that only signals of the same frequency will cause interference. This represents the sum of interference caused by malicious or non-malicious interference sources outside the cluster. Assume there exists a set consisting of M external interference sources. Each interference source Its state is determined by its position and transmission power. and the frequency set of attacks To characterize it. Then the receiver In the channel The total power of external interference received is:

[0098]

[0099] in: Is the interference source antenna in Gain in direction, From the source of interference to Channel gain.

[0100] A communication link At any moment The condition for being considered successfully established is that its The value must be greater than or equal to the minimum SINR threshold required for the demodulator of the communication system to function properly. We use a Boolean flag. To indicate the connectivity status of the link:

[0101]

[0102] For a successfully established link, the theoretically achievable maximum data transmission rate is... (Unit: bits per second, bps), given by Shannon's formula, it is related to the channel bandwidth. and Related:

[0103]

[0104] This rate determines the time required for a data packet to be transmitted from the sender's buffer queue to the receiver.

[0105] Step 1.4: Dynamic Network Topology Model and Physical Constraints: At any given time We abstract the communication network of the entire drone swarm as an undirected dynamic graph. The vertex set of the graph This refers to a collection of drones. Edge set By all at any moment A successfully established communication link consists of, i.e. .

[0106] In a distributed system, each drone unit Unable to obtain global network topology information This is a core constraint. It can only perceive and acquire information about its directly connected neighbor nodes, that is, its one-hop neighbor set. .

[0107] Furthermore, the entire system must strictly adhere to a series of profound physical constraints to ensure the model's validity. These constraints include, but are not limited to:

[0108] Kinematic constraints: The flight speed, acceleration, and angular velocity of the drone cannot exceed its physical performance limits, i.e. , , .

[0109] Energy Consumption Constraint: Remaining Energy of the Drone rate of change over time It depends on flight power consumption, communication power consumption, and the basic power consumption of onboard equipment:

[0110]

[0111] in It is the flight power consumption related to thrust. It is the power consumption for communication transmission. This is the base power consumption of the onboard computing unit. The total energy consumption must not exceed the initial energy, meaning it must meet certain conditions. .

[0112] Communication power constraints: Transmit power Must be within the minimum power allowed by the hardware and maximum power Between. That is: .

[0113] Channel selection constraints: Communication channel It is necessary to use a limited, pre-allocated set of discrete spectrum resources Select from the options. That is: .

[0114] Furthermore, step 2 is detailed as follows:

[0115] Step 2.1: Define conflicting multiple optimization objectives: The ultimate goal of this invention is to find a set of fully distributed control strategy sequences that evolve over time. This strategy sequence needs to collaboratively optimize the following three interrelated and often conflicting performance metrics:

[0116] Objective 1: Maximize task completion rate or communication success rate. This objective aims to ensure the effective execution of reconnaissance missions. We quantify the objective as the ratio of the number of packets successfully transmitted back to the receiving node to the total number of packets generated within the mission area:

[0117]

[0118] Objective 2: Maximize the network survivability and resilience index ( This objective aims to maintain the robustness of the network topology and prevent network fragmentation caused by individual "greedy" decisions. We use the expected value of the minimum node degree of all nodes in the network as a survivability index. A higher minimum degree means that there are no isolated nodes or nodes with fragile connections in the network.

[0119]

[0120] in, Indicates drone At any moment The node degree is the number of its one-hop neighbors.

[0121] Objective 3: Minimize task completion time ( This goal is related to the efficiency of task execution, and is directly defined as the task completion time. .

[0122]

[0123] Step 2.2: Coupling Challenges and Hierarchical Decoupling Approach in the Decision Space: This approach stems from the profound inherent contradictions among the three optimization objectives mentioned above. These include: in order to complete the task as quickly as possible (minimize...) Drones may need to employ a dispersed, high-speed flight strategy to cover a wide area, but this consumes a lot of energy and can disrupt network connectivity, thereby compromising network survivability. Conversely, in order to maintain a tight network topology (maximizing...) Drones may need to maintain clustered flight, which reduces the efficiency of area exploration and thus extends mission time. ).

[0124] Furthermore, the decision vector for each agent It is a hybrid action space containing continuous variables, namely flight control, and discrete variables, namely role and channel selection, with extremely high dimensionality. Directly optimizing this large and coupled joint action space will face the challenge of the "curse of dimensionality," making it difficult for the algorithm to converge.

[0125] Therefore, this invention proposes a hierarchical decoupled decision-making framework. Its core idea is to decompose the complex joint decision-making process into two levels with different time scales and decision priorities:

[0126] Upper-level strategy: Upper-level strategy operates on a longer time scale (macro scale) and is responsible for making strategic path planning decisions. The goal is to make this strategy "foresightful" in order to improve task execution efficiency. ) and network long-term survivability The fundamental conflict between them.

[0127] Lower-level strategies operate on a shorter timescale (microscale) and are responsible for making tactical adjustments in real time. They assign functional roles based on the physical location context provided by the upper-level strategies. ) and anti-jamming communication ( Its core objective is to ensure the resilience of business data flow in the current local environment and to directly contribute to communication success rate. ).

[0128] The aforementioned layered decoupling approach is also the guiding principle for the design of all subsequent algorithms in this invention.

[0129] Furthermore, step 3 is detailed as follows:

[0130] Step 3.1: Construct a local heterogeneous information graph and define node features; at each decision time... Each drone intelligent agent All of them jump to the neighbor collection And model itself as a local heterogeneous graph In a locally heterogeneous graph, the node set is... Each node corresponds to a drone. To characterize the overall state of the nodes, we define an initial feature vector for each node v.

[0131]

[0132] in Indicates location, Indicates speed, Indicates the remaining energy. Indicates the buffer occupancy rate. This indicates one-hot encoding for the character. Indicates network contribution.

[0133] Edge set From all those that satisfy the connectivity condition Communication link It consists of [a structure / structure]. Each edge also has associated attributes. ,in Indicates the selected channel. This indicates the signal-to-interference-plus-noise ratio (SIR) of the link under this channel. This is the link capacity obtained from Shannon's formula. The edge set defined in this way can simultaneously reflect spectrum allocation, interference conditions, and transmission capacity.

[0134] Building upon this, we employ a heterogeneous graph attention network (HetGAT) as the graph neural network (GNN). The general form of a GNN is: given a graph... and node feature matrix Then through the adjacency structure With parameterized functions Iterative update of node embedding This enables the aggregation of neighbor information. HetGAT introduces attention weights in this framework to distinguish between different edges. The importance of this is emphasized, and the meta-path mechanism is used to guide the aggregation process. Ultimately, all neighbor information is compressed into low-dimensional state embedding vectors. This serves as the unified input for subsequent upper and lower layer policy networks.

[0135] Step 3.2: Information aggregation based on meta-path-guided attention mechanism; in local heterogeneous graphs In the middle, each edge With attributes , representing the channel used by the link, the signal-to-interference-plus-noise ratio (SIR) under that channel, and the link capacity, respectively. To distinguish the importance of different communication relationships in the state representation, we introduce the concept of a meta-path based on edge attributes. This includes defining the meta-path's "..." same The corresponding neighbor set represents the set of neighbors with... Neighbors on the same channel and with available link quality; metapaths can also be defined. The set of neighbors corresponding to "maximum" represents the link neighbors with the highest capacity.

[0136] Heterogeneous Graph Attention Network (HetGAT) builds upon this by learning attention weights. Differential weights are assigned to information from different neighbors. The update rule is as follows:

[0137]

[0138] in Let represent the neighbor node embeddings of layer l, W represent the learnable weight matrix, and σ represent the non-linear activation function. Attention weights. By the neighbor Features and edge attributes The decision is made jointly, and normalization is achieved through softmax.

[0139] By defining multiple such meta-paths, HetGAT can learn the importance of neighbors at different semantic levels and perform weighted summation of information from different neighbors. Ultimately, it compresses and encodes high-dimensional neighborhood information into a highly condensed low-dimensional state embedding vector that is highly relevant to the current task. .

[0140] Furthermore, step 4 is detailed as follows:

[0141] Step 4.1: Constructing the Topological Value Prediction Model (TVPM): To enable agents to predict future network states, we constructed a Topological Value Prediction Model (TVPM). This model is a hybrid deep learning model based on temporal graph convolutional networks, used to dynamicalize and anticipate static, local network metrics.

[0142] The TVPM model consists of two core modules: a Graph Convolutional Network (GCN) and a Gated Recurrent Unit (GRU). At each time step, the agent... First, utilize your own state. and its one-hop neighbor set The information is used to construct its local dynamic graph:

[0143]

[0144] The structure of this diagram includes:

[0145] Node set: Edge set: Node feature matrix: :for any node in ,That 3D feature vector Defined as: ,here It is the remaining energy of the node. It is the status of its buffer occupancy. This is its current role. Edge weight matrix: :side weight Based on node link capacity Decide.

[0146] Next, the GCN module is responsible for performing convolution operations on the local graph at the current time step: aggregating neighborhood information, extracting instantaneous spatial topological features, and then generating a graph embedding vector for that time step. This spatial topological feature propagation rule is quantized as follows:

[0147]

[0148] in , It is the first Layer node embedding matrix, It is the learnable weight matrix of this layer. It is an adjacency matrix with self-loops added. It is the identity matrix. yes The angle matrix.

[0149] After these two layers of propagation, the embedding of each node aggregates information from its two-hop neighbors. Finally, through the last layer... All node embeddings are read out at the graph level, i.e., mean pooling, to obtain the graph embedding vector at that time. .

[0150] Subsequently, these graph embedding vector sequences generated at consecutive time steps We input this into the GRU module. The GRU module is a recurrent neural network that excels at capturing dependencies in time series data. Through its internal update and reset gates, it effectively learns dynamic patterns of topological evolution over time. In the GRU, each time step... The update process is as follows:

[0151]

[0152] in It's GRU in The hidden state at all times and These are the update door and the reset door.

[0153] Step 4.2: Quantitative Definition and Model Training of Topological Value: The final output of TVPM is a scalar, namely the predicted topological value. We will define the topological value... Strictly defined as: for intelligent agents In state Perform flight maneuvers Afterwards, in the future The expected discounted cumulative sum of the minimum node degree in the local network within each time step. To train the TVPM model, we collect data in a simulation environment and calculate the true topological value as the target (label) for supervised learning using the following formula:

[0154]

[0155] The definitions and explanations of the physical quantities in the above formula are as follows:

[0156] This represents the expected operation, because the future evolution of the network is random. It is a hyperparameter that represents the length of the predicted future time window. It is a discount factor used to represent the weight of future value relative to current value. The smaller the value, the more attention is paid to the recent topological value. It is an intelligent agent In the future A set of neighbors for a single jump at a given moment. In the future At any moment, intelligent agent The degree of the neighbor with the lowest degree among all its neighbors. This metric directly reflects the most vulnerable connection in the local network and is key to measuring network robustness.

[0157] After minimizing the mean square error between the predicted value of TVPM and the aforementioned true supervision target, we can train a model that can accurately predict the long-term impact of any candidate flight action on the future network topology.

[0158] Step 4.3: Designing a Potential-Based Reward Shaping Mechanism: To integrate the topological value predicted by TVPM into the reward function of reinforcement learning, thus "internalizing" the consideration of network connectivity, we designed a theoretically policy-invariant, potential-based reward shaping (PBRS) mechanism. Therefore, we will... At any moment Total reward Defined as the sum of two parts:

[0159]

[0160] in, These are intrinsic task rewards directly related to the mission, including positive rewards for a drone covering a new grid cell, or negative rewards based on energy and time consumption. This is the shaping reward we designed to guide agents in maintaining network topology.

[0161] According to PBRS theory, in order to ensure that reward shaping does not change the set of optimal strategies for the original problem, reward shaping is necessary. It must have the following form:

[0162]

[0163] in It is only related to state The relevant real-valued function is called the potential function.

[0164] The key innovation of this invention lies in our use of the potential function. It is directly defined as the predicted topological value output by the TVPM model trained in step 4.1. .Right now:

[0165]

[0166] Through this design, we creatively transform the long-term, abstract goal of topology maintenance into an immediate, intensive reward signal. Whenever the agent performs an action, it will change its state... Transferred to If this action increases the future topological value predicted by TVPM (i.e., It will immediately receive a positive shaping reward. Conversely, a negative reward is obtained if the reward is not obtained. This greatly alleviates the learning efficiency problem in sparse task reward environments and effectively guides the agent to learn a forward-looking movement strategy with network topology.

[0167] Step 4.4: Learning a Path Planning Strategy Based on the Soft Actor-Critic Algorithm: Guided by the aforementioned fusion reward function, this step employs a Soft Actor-Critic (SAC) deep reinforcement learning algorithm to learn the optimal upper-level path planning strategy. The SAC algorithm demonstrates superior sample efficiency, stability, and exploration capabilities when handling high-dimensional continuous action spaces, including the three-dimensional acceleration control problem of UAVs. The SAC framework mainly consists of three neural networks: one policy network (Actor) and two Q-value networks (Critics). The specific inputs and outputs of the networks are as follows:

[0168] The input to the policy network (Actor) is the current state representation. (From step three), the output is the flight control action. The probability distribution.

[0169] The input to the Q-value network (Critic) is a state table. and an action The output is an estimate of the long-term cumulative reward (Q value) that can be obtained by performing this action in this state.

[0170] To alleviate the common problem of overestimation of Q-values ​​in reinforcement learning, we chose the SAC algorithm. SAC employs a Twin-Critic network structure: that is, two independent Critic networks are trained simultaneously. In the input state... and actions (Right now The dual-value network will output two independent Q-value estimates. and Furthermore, when calculating the target value for policy updates, the target value y is always pessimistically chosen as the smaller of the two Q-value estimates. This mechanism makes the value learning process more stable and robust. That is:

[0171]

[0172] in It's a fusion reward. It is the action of the next state. These are the parameters of the target network. It is the entropy regularization coefficient. This alleviates the problem of overestimation of the Q value.

[0173] The purpose of choosing the SAC algorithm is to maximize the cumulative reward of an entropy regularization. The introduction of the entropy regularization term encourages the policy network to output actions with higher randomness, thereby promoting the agent to explore the environment more fully and avoiding premature convergence to a local optimum. In this way, through alternating iterative optimization of the Actor and Critic networks, the agent can eventually learn an optimal path planning strategy with stochastic exploration that can both efficiently complete the task and actively maintain the network topology.

[0174] Furthermore, step 5 is detailed as follows:

[0175] Step 5.1: Design of the lower-level immediate reward function: To provide a clear and quantifiable optimization objective for the learning of the lower-level policy, we design a reward function for each agent. The following instant reward function was designed. The reward function is directly related to the quality and resource consumption of the communication link, and is used to incentivize the agent to maintain the resilience of the data flow.

[0176]

[0177] The reward function consists of three parts:

[0178] Part One The reward for successful communication is as follows: For indicator functions, This is the weighting coefficient for that term. When the agent... with his neighbors The link SINR is higher than a preset threshold When the link is successful, the AI ​​receives a positive reward.

[0179] Part Two It is a penalty for communication failure. We assign weights to these factors. We set a penalty for the agent when the link SINR falls below a threshold. This incentivizes the agent to proactively avoid interference and select appropriate communication partners and channels.

[0180] Part Three It is a penalty for transmission power consumption. We assign weights to them. We determine the total power emitted by the agent to all its neighbors. Punishment is imposed to encourage agents to achieve energy-efficient communication while meeting communication needs.

[0181] Step 5.2: Design of the Role Assignment Sub-Policy Network: This sub-policy network is responsible for deciding on the functional role transitions of the UAV. The network structure consists of two parts: an input layer and an output layer. The input layer of this network takes the state embedding vector as input. The output layer is a three-neuron Softmax layer, corresponding to the probabilities of switching to the roles of {collector, relayer, and returner}. The learning objective of the role allocation sub-policy is to maximize the long-term cumulative value of immediate rewards. Through learning, the role allocation sub-policy network can understand the relationship between roles, network states, and task requirements, and then output subsequent decisions with high probability. This includes when an agent discovers its "network contribution" within the network. When a node is significantly larger than its neighbors and its own buffer is empty, the policy network should output a decision to "switch to relay" with a high probability, in order to assume the responsibility of the network hub.

[0182] Meanwhile, to handle discrete role selection actions in end-to-end gradient descent training, we employ the Gumbel-Softmax reparameterization technique. During training, this technique samples noise from the Gumbel distribution and combines it with the network's output probability to generate a vector representing the role selection; during execution, it directly selects the role with the highest probability (argmax operation). This method cleverly solves the challenge of high-variance gradient estimation when training discrete action policies in deep reinforcement learning.

[0183] Step 5.3: Design of the Anti-interference Communication Sub-Policy Network: This sub-policy network makes decisions in parallel with the role assignment network and is responsible for outputting communication parameters. This includes channel selection (discrete) and power allocation (continuous). Network structure: This network also uses state embedding. It serves as the input, but its output layer is designed as two separate "heads" to handle the parameterized mixed action space.

[0184] The first header is responsible for channel selection. It outputs a K-dimensional logit vector (K being the number of available channels), and also uses the Gumbel-Softmax technique to select a specific communication channel. .

[0185] The second head is responsible for power distribution. It outputs one or more power sources. Continuous values ​​within a range, which are then linearly mapped to the hardware-allowed power range. Internally, as the transmit power to one or more neighbors. .

[0186] The goal of this network is to minimize interference to the network and its own energy consumption while meeting the SINR threshold. This is due to state embedding. It contains information about the channels that neighbors are using. This policy allows the network to learn to proactively perform spectrum awareness and avoidance, selecting the channel with the least interference for communication.

[0187] Furthermore, step 6 is detailed as follows:

[0188] Step 6.1: Design a semantically enhanced centralized critic network: In the CTDE framework, the training and execution phases are asymmetric. During the training phase, we introduce a centralized critic network to evaluate the value of the global state and joint actions. This allows for a more accurate assessment of the value of joint actions, thereby guiding the learning of the distributed actor networks.

[0189] The key innovation of this invention lies in our "semantic enhancement" of the centralized Critic. In traditional CTDE methods, including MADDPG, the Critic's input is typically a set of observations and actions from all agents. In our semantically enhanced Critic, the input not only includes this information, but also additionally contains explicit role information for each agent. , forming input pairs After this, the critic network uses this joint input-output value estimate to guide subsequent parameter updates for the various distributed actor networks.

[0190] Step 6.2: Implement a fully decentralized actor network execution: During execution, the centralized commentator network is completely discarded, retaining only the local policy network of each drone. Each drone... Its decision-making is entirely based on its local information. Specifically, it first collects neighborhood information through sensors and local communication to construct a local heterogeneous graph; then, using the HetGAT network in step three, it encodes the information on the local heterogeneous graph into state embedding vectors. Finally, we will use this low-dimensional vector... The input is fed into its local, pre-trained Actor network, which directly outputs flight actions. Role switching decision and communication parameters .

[0191] This decision-making process does not rely on any central controller or global information, ensuring the method's high scalability and robustness in communication-constrained environments. This paradigm of "centralized training of intelligence and decentralized execution of strategies" is the core mechanism by which this invention achieves complex collaborative anti-interference capabilities in large-scale UAV swarms.

[0192] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0193] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for autonomous cooperative anti-interference of unmanned aerial vehicle (UAV) swarms based on hierarchical reinforcement learning, characterized in that, The process includes the following steps: Step 1: Establish a collaborative anti-interference system and physical model for UAV swarms, define the UAV ensemble and its state variables, establish region partitioning and grid sets, construct task and data flow models, establish communication links and dynamic topology, and set kinematic, energy, communication power, and channel selection constraints; Step 2: Construct a hierarchical decoupled distributed multi-objective optimization problem, define optimization objectives such as task completion, network survivability, and task completion time, and divide the decision-making process into upper-level path planning and lower-level role allocation and communication control; Step 3: Design a state representation method based on heterogeneous graph attention networks, construct a local heterogeneous graph at each time step, and guide attention through meta-paths. The mechanism aggregates neighborhood information to obtain low-dimensional state embeddings; Step 4: Design an upper-level path planning method based on topological value internalization, establish a topological value prediction model, and use the soft actor-commentator algorithm (SAC) to learn the path planning strategy; Step 5: Design a lower-level decompositional policy network oriented towards functional reconstruction, construct a role allocation sub-policy network and an anti-interference communication sub-policy network, and output the UAV functional roles and communication parameters respectively; Step 6: Adopt a semantically aware centralized training and decentralized execution paradigm, introduce a semantically enhanced centralized commentator network in the training phase, and make independent decisions by the distributed actor network in the execution phase to achieve autonomous collaborative anti-interference of the UAV swarm.

2. The method for autonomous cooperative anti-interference of UAV swarms based on hierarchical reinforcement learning according to claim 1, characterized in that, Step 1 further includes: Step 1.1: Defining a drone swarm and state variables The state of each drone at any given time is determined by its location. ,speed acceleration ,attitude Remaining energy Local data buffer status and functional roles Composition; Step 1.2: Establishing a task and data flow model: [This involves] defining the reconnaissance area. Discretized into a set of grids The drone generates data packets based on the role when covering the grid. And achieve single-hop or multi-hop transmission through the communication link; Step 1.3: Establish a physical model of communication and interference: The quality of the communication link is quantified by the SINR of the receiver, at time... ,from Send to The signal, in The received SINR value at the location is denoted as The calculation formula is as follows: in: It is a drone In the channel Upward drone The power of the transmitted signal; It is a drone The transmitting antenna is relative to Direction angle Antenna gain; It is a drone The receiving antenna is relative to Direction angle Antenna gain; From arrive Channel gain; Representative receiver Background Gaussian white noise power at that location; Refers to the receiver In the channel Above, received signals from other drones within the cluster (excluding useful signal sources). The sum of co-channel interference (excluding external interference); its specific calculation formula is: in: Represents any other communication link in the cluster. It is an indicator function; This is the sum of interference caused by malicious or non-malicious interference sources outside the cluster; assume there exists a set consisting of M external interference sources. Each interference source Its state is determined by its position and transmission power. and the frequency set of attacks To characterize; then the receiver In the channel The total power of external interference received is: in: Is the interference source antenna in Gain in direction, From the source of interference to The channel gain; channel selection is derived from a limited set of spectrum resources, and the link transmission rate is calculated based on Shannon's formula; Step 1.4: Establish a dynamic network topology model: Abstract the communication relationships of UAVs into a dynamic graph at any given time. And impose kinematic, energy, communication power, and channel selection constraints: ， ， 。 3. The method for autonomous cooperative anti-interference of UAV swarms based on hierarchical reinforcement learning according to claim 2, characterized in that, Step 2 further includes: Step 2.1: Define the optimization objective: including maximizing task completion rate. Maximize the network survivability index and minimize task completion time Step 2.2: Propose a hierarchical decoupling optimization framework. The upper-layer strategy performs path planning at the macro time scale, while the lower-layer strategy performs functional role allocation and anti-interference communication at the micro time scale, so as to balance task execution efficiency and network long-term survivability.

4. The method for autonomous cooperative anti-interference of UAV swarms based on hierarchical reinforcement learning according to claim 3, characterized in that, Step 3 further includes: Step 3.1: Constructing a local heterogeneous graph The node is a drone, and its characteristics include: Position, speed Remaining energy Buffer occupancy rate ,Role And network contribution; Side attributes include channel, SINR, and link capacity. Step 3.2: The Heterogeneous Graph Attention Network (HetGAT) is used to aggregate neighbor information and distinguish different communication semantics through the meta-path mechanism to generate low-dimensional state embeddings.

5. The method for autonomous cooperative anti-interference of UAV swarms based on hierarchical reinforcement learning according to claim 1, characterized in that, Step 4 further includes: Step 4.1: Constructing a Topological Value Prediction Model (TVPM), using graph convolutional networks and recurrent neural networks to analyze dynamic graphs. Embedding is used for time-series modeling to predict the future local network topology value; Step 4.2: Quantitative definition and model training of topology value, quantifying topology value: Then, supervised learning is used to train the TVPM; Step 4.3: Based on predicted topological value: We designed a reward shaping mechanism; we will predict the value As a potential function Embedding reinforcement learning reward functions; Step 4.4: Select the Soft Actor-Critic Algorithm Learning Path (SAC) for policy learning. By using a dual critic network, the target value is always estimated using a smaller Q value, which alleviates the problem of Q value overestimation and improves the stability and robustness of the value learning process and path planning.

6. The method for autonomous cooperative anti-interference of UAV swarms based on hierarchical reinforcement learning according to claim 1, characterized in that, Step 5 further includes: Step 5.1: Designing an immediate reward function. It includes a communication success reward, a communication failure penalty, and a transmit power penalty; Step 5.2: Construct a role allocation sub-policy network and use the Gumbel-Softmax method to realize the discrete role selection of collectors, relayers, and backhaulers; Step 5.3: Construct an anti-interference communication sub-policy network, output channel selection and power allocation parameters, and realize spectrum awareness and adaptive interference avoidance.

7. A method for autonomous cooperative anti-interference of UAV swarms based on hierarchical reinforcement learning according to claim 6, characterized in that, Step 6 further includes: Step 6.1: During the training phase, a semantically enhanced centralized commentator network is used. We take the global state and joint actions as inputs and combine role information to perform joint value estimation; Step 6.2: During the execution phase, the distributed actor network runs independently. Each UAV generates state embeddings based on the local heterogeneous graph and outputs flight actions, role selection and communication parameters, thereby achieving autonomous coordination under decentralized control conditions.

Images