A narrow-beam unmanned aerial vehicle cluster directional discovery and cooperative navigation method based on deep reinforcement learning
Patent Information
- Application Number
- CN202611149839.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-30
- Publication Date
- 2026-10-16
AI Technical Summary
[0003]然而，在大规模无人机集群的实际部署中，通信系统的性能直接决定了集群协同的上限
[0068] Through the above process, a closed-loop collaborative mechanism is formed, consisting of "local state awareness - scanning tactical selection - beam direction adjustment - bidirectional neighbor discovery - route state update - topology cooperative control - local state feedback". Each UAV re-makes scanning tactical decisions based on the updated local network state at each time step, enabling the UAV swarm to continuously perform neighbor rediscovery and link reconstruction during dynamic topology disturbances caused by takeoff, stable flight, and obstacle avoidance, thereby achieving distributed dynamic networking and topology self-recovery under 6° narrow beam communication conditions.
Smart Images
Figure CN122924355A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) swarm cooperative control and ad hoc network directional communication, and specifically relates to a narrow-beam UAV swarm directional discovery and cooperative navigation method based on deep reinforcement learning. Background Technology
[0002] With the rapid development of modern aviation technology and artificial intelligence, unmanned aerial vehicle (UAV) swarms, with their high flexibility, strong robustness, and low-cost redundancy, have demonstrated enormous application potential in complex tasks such as target search, collaborative penetration, long-range reconnaissance, and area control. When performing such tasks, multi-UAV systems can achieve higher operational efficiency and damage resistance than single-unit systems in dynamic and unknown scenarios through distributed collaboration.
[0003] However, in the actual deployment of large-scale drone swarms, the performance of the communication system directly determines the upper limit of swarm coordination. Traditional drone swarms mostly use omnidirectional antennas for information exchange, but omnidirectional communication has inherent drawbacks: first, energy radiation throughout space results in weak anti-interference capabilities, making it easily masked by enemy noise in strong electromagnetic warfare environments; second, the wide signal radiation range leads to poor performance with low interception and detection probabilities, making the swarm's location easily exposed. Therefore, adopting narrow-beam directional communication technology has become the core means to improve the swarm's stealth and survivability.
[0004] Despite the significant advantages of directional narrow beams, achieving efficient "neighbor discovery" and "topology maintenance" in three-dimensional dynamic space faces challenging bottlenecks:
[0005] First, the "small error, big difference" effect of narrow beam alignment. Under the constraint of a 6° narrow beam, the main lobe width of the UAV antenna is narrow, and the alignment error tolerance space is only ±3°. When large-scale swarms perform long-range penetration missions, the underlying navigation algorithm calculates the trajectory in real time based on the target guidance force and obstacle avoidance repulsion force. When large-scale UAV swarms maneuver in complex environments with multiple large obstacles, frequent attitude adjustments and position jumps can cause the beam to miss the target instantly. Especially during the takeoff phase and the strong maneuver obstacle avoidance phase, the transient displacement caused by physical inertia can cause "maneuvering tears" in the link, causing the swarm network to instantly shatter into a large number of isolated nodes that are not interconnected.
[0006] Second, the dynamic complexity of large-scale cluster topologies. As the number of drones continues to increase, the potential combinations of links in space grow exponentially. Interference between nodes, occlusion, and frequent disconnections and reconnections make topology management extremely complex. Traditional routing protocols and neighbor discovery algorithms are mostly based on first-order kinematic assumptions. However, in real physical simulations, due to acceleration limitations and inertia, if disconnected nodes still rely on outdated coordinates in their routing tables for tracking, severe loss and erroneous tracking will occur. This phenomenon can cause drones to be pulled backward by "false neighbors," resulting in spinning in place or deadlock at the origin, severely hindering mission progress.
[0007] Third, the convergence dilemma of reinforcement learning in narrow-beam environments. Using multi-agent deep reinforcement learning to train nodes' link selection and tactical decision-making is currently the mainstream trend. However, in a 6° narrow-beam environment, due to the high difficulty of alignment, the probability of agents establishing effective connections through random exploration in the early stages of training is almost zero, leading to an extreme "reward sparsity" problem. If training begins directly in a 6° environment, the model often gets stuck in local optima due to the lack of positive feedback for a long time, and may even learn negative strategies such as "collective suicide" to avoid network fragmentation penalties.
[0008] Fourth, there is a lack of curriculum-based learning in developing link capabilities. Existing research mostly focuses on strategy optimization with fixed beamwidths, lacking a progressive training paradigm that gradually increases in difficulty. For agents with link selection capabilities, directly mastering alignment strategies under a 6° beamwidth is unrealistic. How to guide agents to gradually acquire the ability to accurately capture teammate afterimages, predict maneuver trends, and optimize beam pointing in real time in complex obstacle environments through a step-by-step curriculum learning from 30° to 16° and then to 6° is a research gap that urgently needs to be addressed in the field of UAV swarms.
[0009] In summary, existing UAV swarm collaboration methods lack comprehensive consideration of large-scale nodes, narrow beam constraints, complex 3D obstacle avoidance, and intelligent decision-making and self-healing. Therefore, developing a decision-making framework that combines high-fidelity physical kinematics, heuristic survival mechanisms, and curriculum-based reinforcement learning is of significant scientific research value and practical importance for improving the collaborative performance of UAV swarms in harsh environments in my country. Summary of the Invention
[0010] The purpose of this invention is to address the dynamic self-healing problem of highly maneuverable narrow-beam swarms of unmanned aerial vehicles (UAVs) by proposing a deep reinforcement learning-based method for narrow-beam UAV swarm orientation discovery and cooperative navigation (Communication-Aware Multi-Agent Proximal Policy Optimization, CA-MAPPO). To achieve this objective, the steps employed in this invention are as follows:
[0011] Step 1: Perform static system modeling and ground initialization of the UAV swarm on the ground; before the mission begins, initialize the UAVs in the takeoff area in the form of a discrete grid.
[0012] 3D coordinate initialization: Initial position p of each UAV i i，0 It is set on a plane at a height of z, and random perturbations are added to simulate the actual takeoff attitude error.
[0013] Ground-based network establishment: During the ground stationary phase before takeoff, sufficient link establishment time is simulated; each UAV... i Based on the initial Euclidean distance d ij，0 =||p j，0 -p i，0 ||2. Initialize its initial active neighbor set; if the drone U j With U drones i The initial distance does not exceed the maximum communication radius R c Then U j Join U i Initial set of active neighbors:
[0014]
[0015] in U-shaped drone i The set of active neighbors at the initial moment, R c To maximize the communication radius; synchronize the global initial routing table via a highly reliable ground link to ensure that each agent holds the initial coordinates p of its teammates. init With zero velocity vector v init =[0, 0, 0] T .
[0016] Step 2: Using high-fidelity physical evolution derivation based on Newtonian physics, the "non-physical instantaneous movement" caused by traditional first-order kinematics and the resulting narrow-beam maneuver tearing problem are solved. This invention implements a rigorous second-order dynamic integral.
[0017] Calculation of resultant and partial force fields: In each simulation step t, the environment engine calculates the resultant force on UAV i in real time. In particular, to eliminate the "ghost anchoring effect," this invention implements an active link filtering algorithm to ensure topology maintenance force f. topo It is generated only by physically connected neighbors; let A ij，t As a dynamic adjacency matrix index, the resultant force formula is defined as follows:
[0018]
[0019] Where f guideThe guiding force of the drone for the mission objective. Let f be the topological force vector between UAV i and UAV j. obs Let β be the repulsive force vector against the obstacle. G β T β O β is the scaling factor. G +β T +β O =1.
[0020] Acceleration saturation cutoff and integration: mapping the resultant force to acceleration And implement physical power limiting:
[0021]
[0022] Where m is the mass of the drone, and A max This is the maximum acceleration limit.
[0023] Update velocity and position using second-order integrals:
[0024]
[0025] p t+1 =p t +v t+l ·Δt (5)
[0026] Where v t Let be the velocity vector of the UAV at time t. Let V be the acceleration vector of drone i. max To limit the maximum flight speed, p t Let t be the position of the drone at time t.
[0027] Step 3: Implement narrow-beam three-dimensional directional communication and real-time information synchronization for UAV swarms, with beam pointing vector b. i，t Driven by yaw angle ψ and pitch angle θ.
[0028] Geometric alignment calculation: The system calculates the normalized relative position vector r between all node pairs (i, j). ij，t =(p j,t -p i，t ) / ||p j,t -p i，t ||2; The cosine constraint for determining the establishment of a physical link is:
[0029]
[0030] Where θ half It is the half-power angle.
[0031] MAC layer sticky keep-alive state machine: Assigning counter H to each link i，j When formula (6) is not satisfied, the link enters a countdown decay H. i,j =H i,j -1; in H i，j During periods >0, the drone executes the routing switching protocol:
[0032]
[0033] Where p j The location field of drone j is recorded in the routing table, v j Record the speed field of drone j in the routing table, t seen p is the timestamp of the last time drone j was observed or updated. true,j (t) represents the true position of UAV j at time t, v true，j (t) represents the actual velocity of UAV j at time t.
[0034] This ensures that the coordinates in the routing table are always the actual physical coordinates, rather than relying on afterimages from first-order predictions.
[0035] Step 4: Implement a survival mechanism based on heuristic action shielding. When the cluster becomes a "lone wolf" due to obstacle avoidance or strong maneuvering, the expert intervention logic is activated.
[0036] Action interceptor logic: Define the action space C = {c0, c1, c2, c3}, where c0 is prediction tracking, and c1, c2, and c3 are frequency sweeps.
[0037] If the number of currently active neighbors of drone i is detected |N active,i If |=0 and the deep neural network outputs the action of predicting and tracking c0, then masking resampling is implemented:
[0038] c i，actual =Random({c1c2, c3}) (8)
[0039] This operation forces the Lone Wolf drone to abandon blind tracking based on outdated routes and switch to wide-area radar frequency scanning, significantly increasing the probability of self-healing after network outages.
[0040] Step 5: Calculation of the full subnet joint liability reward function; To guide the UAV to maintain large-scale network connectivity, a full subnet joint liability reward function with amplitude limiting is used, and the following multi-dimensional coupled reward is designed:
[0041] Progress and connectivity rewards: based on high water level P min_dist Calculate penetration score:
[0042] r prog =ω p ·max(0, P)min_dist -||p i，t -P target ||) (9)
[0043] Subnet Split Penalty Truncation: Calculating the Subnet Number K using the Disjoint-Set Union Algorithm sub Implementing bandwidth-limiting penalties to curb "suicidal strategies":
[0044] R subnet =max((K) sub -1)·γ sub R limit (10)
[0045] Where K sub γ represents the number of subnets in the current communication topology. sub As the penalty base, R limit To truncate the lower limit.
[0046] Through |R limit |<<R collision The constraint of | ensures that the agent retains its breakthrough momentum when the network breaks down, rather than actively destroying obstacles to avoid long-term point deductions, where R collision This is the collision penalty value when a drone collides with another drone. It is usually negative and is used to penalize dangerous behaviors such as drones colliding with obstacles, colliding with other drones, or triggering a crash.
[0047] Step 6: Training is conducted using a multi-stage course learning strategy; to overcome the reward sparsity problem under narrow beam conditions, this embodiment adopts a tiered course learning paradigm:
[0048] Beam contraction path:
[0049] Phase 1: The beamwidth is set to 30° to guide the agent to learn basic swarm navigation and obstacle avoidance capabilities.
[0050] Phase 2: The beamwidth is reduced to 16° to train the agent's link maintenance accuracy during maneuvers.
[0051] The third stage: the beamwidth is reduced to the final target of 6°, and rapid convergence is achieved through the pre-trained weights.
[0052] Gradient update parameters: in calculating generalized advantage estimation Then, the Actor parameters are updated through the PPO proxy objective function:
[0053]
[0054] in For Actor policy network parameters, denoted as the ratio of the probabilities of actions in the new and old strategies, and ∈ is the PPO shearing coefficient, which controls the magnitude of policy updates.
[0055] Step 7: Distributed online execution and cluster topology closed-loop collaborative control; After completing the above course learning and training, save the converged Actor policy network parameters, and deploy the shared policy network to each UAV node using a centralized training and distributed execution approach; During the online execution phase, each UAV does not need to obtain the complete global state of the UAV cluster, but independently completes scanning tactical decisions based only on its own motion state, local neighbor connection state, obstacle perception information, and local routing history information.
[0056] For the i-th UAV, the local observation state o is acquired at time step t. i，t The results are then fed into the trained Actor policy network to obtain the action probability distributions for four structured scanning tactics:
[0057]
[0058] in, and Let represent the probabilities of UAV i choosing predictive scanning, forward scanning, lateral scanning, and backward scanning tactics, respectively, and satisfy the following conditions:
[0059]
[0060] Considering that narrow-beam neighbor discovery requires complementary spatial scanning directions among different UAVs, this invention does not employ a deterministic execution method where all nodes uniformly select the action with the highest probability. Instead, it performs tactical sampling according to the action probability distribution output by the policy network, i.e.:
[0061]
[0062] Among them, c i,t This indicates the scanning tactic selected by UAV i at time step t; the UAV calculates the corresponding beam azimuth and elevation angles based on the selected tactic, and adjusts the main lobe direction of the directional antenna to perform narrow beam directional scanning.
[0063] After completing the beam direction update, the UAV performs a two-way beam handshake determination based on the spatial distance between nodes and the beam coverage status of both the transmitter and receiver. When UAV i and UAV j simultaneously meet the communication distance constraint and the two-way beam alignment condition, a valid neighbor link is established and the link keep-alive timer is refreshed. If the link fails to re-align the beam within the specified keep-alive time, the corresponding node is removed from the active neighbor set.
[0064] For rediscovered neighbor nodes, the UAV writes their current location, speed, and discovery time into the local routing information table and uses the current valid neighbor links to synchronize routing status. The updated set of active neighbors participates in the topology retention calculation for the next time step, and the routing history information is used for the location prediction of lost neighbors and the execution of predictive scanning tactics.
[0065] Let the overall state of the drone swarm at time step t be S. t The scanning tactics set for each UAV is C. t ={c 1,t c 2，t c N，t The communication topology formed after completing the two-way beam handshake is G. l The drone cluster state update process can be represented as follows:
[0066]
[0067] in, This represents the state transition process of a drone swarm, which consists of directional neighbor discovery, link keep-alive update, routing information synchronization, topology retention calculation, target guidance, and obstacle avoidance maneuvers.
[0068] Through the above process, a closed-loop collaborative mechanism is formed, consisting of "local state awareness - scanning tactical selection - beam direction adjustment - bidirectional neighbor discovery - route state update - topology cooperative control - local state feedback". Each UAV re-makes scanning tactical decisions based on the updated local network state at each time step, enabling the UAV swarm to continuously perform neighbor rediscovery and link reconstruction during dynamic topology disturbances caused by takeoff, stable flight, and obstacle avoidance, thereby achieving distributed dynamic networking and topology self-recovery under 6° narrow beam communication conditions. Attached Figure Description
[0069] Figure 1 Topology diagram of a 6° narrow-beam UAV swarm;
[0070] Figure 2 This is a block diagram of the potential field motion control for the unmanned aerial vehicle (UAV).
[0071] Figure 3 A geometric model for the beam pointing of a drone;
[0072] Figure 4 The training diagram for heuristic actions is hidden.
[0073] Figure 5 For link keep-alive state transition diagram;
[0074] Figure 6 This is the result of the CA-MAPPO cluster collaborative penetration performance evaluation. Detailed Implementation
[0075] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0076] In the following description, the narrow-beam UAV swarm orientation discovery and cooperative navigation method based on deep reinforcement learning proposed in this invention will be abbreviated as CA-MAPPO. This method has been implemented in Python, and simulation results demonstrate its effectiveness. The specific implementation steps of CA-MAPPO are as follows:
[0077] Step 1: Perform static system modeling and ground initialization of the UAV swarm. Before the mission begins, initialize the UAVs in the takeoff area using a discrete grid.
[0078] 3D coordinate initialization: Initial position p of each UAV i i,0 It is set on a plane at a height of z, and random perturbations are added to simulate the actual takeoff attitude error.
[0079] Ground-based network establishment: During the ground stationary phase before takeoff, sufficient link-establishment time is simulated. Each UAV... i Based on the initial Euclidean distance d ij，0 =||p j，0 -p i,0 ||2. Initialize its initial set of active neighbors. If the drone U j With U drones i The initial distance does not exceed the maximum communication radius R c Then U j Join U i Initial set of active neighbors:
[0080]
[0081] in U-shaped drone i The set of active neighbors at the initial moment, R c To achieve the maximum communication radius, a globally initial routing table is synchronized via a highly reliable ground link to ensure that each agent holds the initial coordinates p of its teammates. init With zero velocity vector v init =[0, 0, 0] T .
[0082] Step 2: Employing high-fidelity physical evolution derivation based on Newtonian physics, the "non-physical instantaneous movement" caused by traditional first-order kinematics and the resulting narrow-beam maneuver tearing problem are resolved, as shown in the attached diagram. Figure 2 As shown, the present invention implements a strict second-order dynamical integral.
[0083] Calculation of resultant and partial force fields: In each simulation step t, the environment engine calculates the resultant force on UAV i in real time. In particular, to eliminate the "ghost anchoring effect," this invention implements an active link filtering algorithm to ensure topology maintenance force f. topo It is generated only by physically connected neighbors. Let A ij，t As a dynamic adjacency matrix index, the resultant force formula is defined as follows:
[0084]
[0085] Where f guide The guiding force of the drone for the mission objective. Let f be the topological force vector between UAV i and UAV j. obs Let β be the repulsive force vector against the obstacle. G β T β O β is the scaling factor. G +β T +β O =1.
[0086] Acceleration saturation cutoff and integration: mapping the resultant force to acceleration And implement physical power limiting:
[0087]
[0088] Where m is the mass of the drone, and A max This is the maximum acceleration limit.
[0089] Update velocity and position using second-order integrals:
[0090]
[0091] p t+1 =p t +v t+1 ·Δt (20)
[0092] Where v t Let be the velocity vector of the UAV at time t. Let V be the acceleration vector of drone i. max To limit the maximum flight speed, p t Let t be the position of the drone at time t.
[0093] Step 3: Implement narrow-beam three-dimensional directional communication and real-time information synchronization for UAV swarms, as shown in the attached document. Figure 3 As shown, the beam pointing vector b i，t Driven by yaw angle ψ and pitch angle θ.
[0094] Geometric alignment calculation: The system calculates the normalized relative position vector r between all node pairs (i, j). ij，t =(pj，t -p i，t ) / ||p j，t -p i，t 2. The cosine constraint for determining the establishment of a physical link is:
[0095]
[0096] Where θ half It is the half-power angle.
[0097] MAC layer sticky keep-alive state machine: Assigning counter H to each link i，j When formula (21) is not satisfied, the link enters a countdown decay H. i，j =H i，j -1. In H i，j During periods >0, the drone executes the routing switching protocol:
[0098]
[0099] Where p j The location field of drone j is recorded in the routing table, v j Record the speed field of drone j in the routing table, t seen p is the timestamp of the last time drone j was observed or updated. true,j (t) represents the true position of UAV j at time t, v true,j (t) represents the actual velocity of UAV j at time t.
[0100] This ensures that the coordinates in the routing table are always the actual physical coordinates, rather than relying on afterimages from first-order predictions.
[0101] Step 4: Implement a survival mechanism based on heuristic action shielding, as shown in the attached document. Figure 4 As shown, when the cluster becomes a "lone wolf" due to obstacle avoidance or strong maneuvering, CA-MAPPO initiates expert intervention logic.
[0102] Action interceptor logic: Define the action space C = {c0, c1, c2, c3}, where c0 is prediction tracking, and c1, c2, and c3 are frequency sweeping.
[0103] If the number of currently active neighbors of drone i is detected |N active,i If |=0 and the deep neural network outputs the action of predicting and tracking c0, then masking resampling is implemented:
[0104] c i，actual =Random({c1,c2,c3}) (23)
[0105] This operation forces the Lone Wolf drone to abandon blind tracking based on outdated routes and switch to wide-area radar frequency scanning, significantly increasing the probability of self-healing after network outages.
[0106] Step 5: Calculation of the full sub-network joint liability reward function. To guide the UAV to maintain large-scale network connectivity, a full sub-network joint liability reward function with amplitude limiting is used, and the following multi-dimensional coupled reward is designed:
[0107] Progress and connectivity rewards: based on high water level P min_dist Calculate penetration score:
[0108] r prog =ω p ·max(0, P) min_dist -||p i，t -P target ||) (24)
[0109] Subnet Split Penalty Truncation: Calculating the Subnet Number K using the Disjoint-Set Union Algorithm sub Implementing bandwidth-limiting penalties to curb "suicidal strategies":
[0110] R subnet =max((K) sub -1)·γ sub R limit (25)
[0111] Where K sub γ represents the number of subnets in the current communication topology. sub As the penalty base, R limit To truncate the lower limit.
[0112] Through |R limit |<<R collision The constraint of | ensures that the agent retains its breakthrough momentum when the network breaks down, rather than actively destroying obstacles to avoid long-term point deductions, where R collision This is the collision penalty value when a drone collides with another drone. It is usually negative and is used to penalize dangerous behaviors such as drones colliding with obstacles, colliding with other drones, or triggering a crash.
[0113] Step 6: Training is conducted using a multi-stage course learning strategy. To overcome the reward sparsity problem under narrow beam conditions, this embodiment adopts a tiered course learning paradigm:
[0114] Beam contraction path:
[0115] Phase 1: The beamwidth is set to 30° to guide the agent to learn basic swarm navigation and obstacle avoidance capabilities.
[0116] Phase 2: The beamwidth is reduced to 16° to train the agent's link maintenance accuracy during maneuvers.
[0117] The third stage: the beamwidth is reduced to the final target of 6°, and rapid convergence is achieved through the pre-trained weights.
[0118] Gradient update parameters: in calculating generalized advantage estimation Then, the Actor parameters are updated through the PPO proxy objective function:
[0119]
[0120] in For Actor policy network parameters, denoted as the ratio of the probabilities of actions in the new and old strategies, and ∈ is the PPO shearing coefficient, which controls the magnitude of policy updates.
[0121] Step 7: Distributed Online Execution and Cluster Topology Closed-Loop Cooperative Control. After completing the above training, save the converged Actor policy network parameters and deploy the shared policy network to each UAV node using a centralized training and distributed execution approach. During the online execution phase, each UAV does not need to obtain the complete global state of the UAV cluster; it independently completes scanning tactical decisions based solely on its own motion state, local neighbor connection state, obstacle perception information, and local routing history information.
[0122] For the i-th UAV, the local observation state o is acquired at time step t. i,t The results are then fed into the trained Actor policy network to obtain the action probability distributions for four structured scanning tactics:
[0123]
[0124] in, and Let represent the probabilities of UAV i choosing predictive scanning, forward scanning, lateral scanning, and backward scanning tactics, respectively, and satisfy the following conditions:
[0125]
[0126] Considering that narrow-beam neighbor discovery requires complementary spatial scanning directions among different UAVs, this invention does not employ a deterministic execution method where all nodes uniformly select the action with the highest probability. Instead, it performs tactical sampling according to the action probability distribution output by the policy network, i.e.:
[0127]
[0128] Among them, c i，t This indicates the scanning tactic selected by UAV i at time step t. Based on the selected tactic, the UAV calculates the corresponding beam azimuth and elevation angles, adjusts the main lobe direction of the directional antenna, and performs a narrow-beam directional scan.
[0129] After completing the beam direction update, the UAV performs a two-way beam handshake based on the spatial distance between nodes and the beam coverage status of both the transmitter and receiver. When UAV i and UAV j simultaneously meet the communication distance constraint and the two-way beam alignment condition, a valid neighbor link is established and the link keep-alive timer is refreshed; if the link fails to re-align the beam within the specified keep-alive time, the corresponding node is removed from the active neighbor set.
[0130] For rediscovered neighbor nodes, the drone writes their current location, speed, and discovery time into the local routing information table and synchronizes routing states using currently active neighbor links. The updated set of active neighbors participates in the topology-keeping calculation for the next time step, and routing history information is used for predicting the location of lost neighbors and executing predictive scanning tactics.
[0131] Let the overall state of the drone swarm at time step t be S. t The scanning tactics set for each UAV is C. t ={c 1，t， c 2，t c N，t The communication topology formed after completing the two-way beam handshake is G. t The drone cluster state update process can be represented as follows:
[0132]
[0133] in, This represents the state transition process of a drone swarm, which consists of directional neighbor discovery, link keep-alive update, routing information synchronization, topology retention calculation, target guidance, and obstacle avoidance maneuvers.
[0134] Through the above process, a closed-loop collaborative mechanism is formed, consisting of "local state awareness - scanning tactical selection - beam direction adjustment - bidirectional neighbor discovery - route state update - topology cooperative control - local state feedback". At each time step, each UAV re-makes its scanning tactical decision based on the updated local network state, enabling the UAV swarm to continuously perform neighbor rediscovery and link reconstruction during dynamic topology disturbances caused by takeoff, stable flight, and obstacle avoidance. This achieves distributed dynamic networking and topology self-recovery under 6° narrow-beam communication conditions.
[0135] The performance of the proposed deep reinforcement learning-based narrow-beam UAV swarm orientation discovery and cooperative navigation method has been verified in a Python-based 3D physical simulation environment. (Appendix) Figure 6 This is the result of the CA-MAPPO cluster collaborative penetration performance evaluation.
[0136] Topological evolution and self-healing analysis: as attached Figure 6 (a) and appendix Figure 6As shown in (b), at the moment of takeoff, due to the physical maneuvering tear, the number of subnets instantly surged from 1 to over 40. Subsequently, relying on the heuristic frequency sweeping mechanism in step four, the average number of neighbors rapidly climbed from 0.5 and stabilized at 3 to 5, and the number of subnets fell back to a dynamic equilibrium range of less than 5.
[0137] Tactical selection preference statistics: as attached Figure 6 As shown in (c), the agent learned autonomous link selection capability in a 6° environment. Tactic 0 (predictive sniping) accounted for 36.4%, with a link establishment success rate of 87.7%, demonstrating the superiority of predictive tactics based on real-time physical coordinate synchronization in stable flight segments. During periods of high contact loss, the link establishment success rates of flanking and rearward frequency sweeping tactics reached 82.6% and 86.7%, respectively, supporting rapid network reconstruction of the cluster.
[0138] Long-range penetration trajectory performance: (attached) Figure 6 (d) shows the two-dimensional trajectory of the drone swarm in an environment with three obstacles. The gray solid line represents a stable directional network state, and the blue dots represent brief disconnections during obstacle avoidance. The results show that the swarm did not deadlock after bypassing the obstacles, and all drones accurately hit the yellow star marker at a distance of 30,000 meters, achieving a 100% mission success rate, with the progress curve converging perfectly.
[0139] The contents not described in detail in this specification are existing technologies known to those skilled in the art.
Claims
1. A method for directional discovery and cooperative navigation of narrow-beam UAV swarms based on deep reinforcement learning, comprising the following steps: Step 1: Perform static system modeling and ground initialization of the UAV swarm on the ground; before the mission begins, initialize the UAVs in the takeoff area in the form of a discrete grid; 3D coordinate initialization: Initial position p of each UAV i i，0 It is set on a plane at a height of z, and random perturbations are added to simulate the actual takeoff attitude error; Ground-based network establishment: During the ground stationary phase before takeoff, sufficient link establishment time is simulated; each UAV... i Based on the initial Euclidean distance d ij，0 =||p j，0 -p i,0 ‖2. Initialize its initial active neighbor set; if U_tw is j With U drones i The initial distance does not exceed the maximum communication radius R c Then U j Join U i Initial set of active neighbors: in U-shaped drone i The set of active neighbors at the initial moment, R c To maximize the communication radius; synchronize the global initial routing table via a highly reliable ground link to ensure that each agent holds the initial coordinates p of its teammates. init With zero velocity vector v init =[0, 0, 0] T ; Step 2: Using high-fidelity physical evolution derivation based on Newtonian physics, the "non-physical instantaneous movement" caused by traditional first-order kinematics and the resulting narrow-beam maneuver tearing problem are solved. This invention implements strict second-order dynamic integration. Calculation of resultant and partial force fields: In each simulation step t, the environment engine calculates the resultant force on UAV i in real time. In particular, to eliminate the "ghost anchoring effect," this invention implements an active link filtering algorithm to ensure topology maintenance force f. topo It is generated only by physically connected neighbors; let A ij，t As a dynamic adjacency matrix index, the resultant force formula is defined as follows: Where f guide The guiding force of the drone for the mission objective. Let f be the topological force vector between UAV i and UAV j. obs Let β be the repulsive force vector against the obstacle. G β T β O β is the scaling factor. G +β T +β O =1; Acceleration saturation cutoff and integration: mapping the resultant force to acceleration And implement physical power limiting: Where m is the mass of the drone, and A max Maximum acceleration limit; Update velocity and position using second-order integrals: p t+1 =p t +v t+1 ·Δt (5) Where v t Let be the velocity vector of the UAV at time t. Let V be the acceleration vector of drone i. max To limit the maximum flight speed, p t Let t be the position of the UAV. Step 3: Implement narrow-beam three-dimensional directional communication and real-time information synchronization for UAV swarms, with beam pointing vector b. i，t Driven by yaw angle ψ and pitch angle θ; Geometric alignment calculation: The system calculates the normalized relative position vector r between all node pairs (i, j). ij，t =(p j，t -p i，t ) / ||p j，t -p i，t ||2; The cosine constraint for determining the establishment of a physical link is: and Where θ half It is the half-power angle; MAC layer sticky keep-alive state machine: Assigning counter H to each link i，j When formula (6) is not satisfied, the link enters a countdown decay H. i，j =H i，j -1; in H i，j During periods >0, the drone executes the routing switching protocol: Where p j The location field of drone j is recorded in the routing table, v j Record the speed field of drone j in the routing table, t seen p is the timestamp of the last time drone j was observed or updated. true，j (t) represents the true position of UAV j at time t, v true，j (t) represents the actual velocity of UAV j at time t; This ensures that the coordinates in the routing table are always the actual physical coordinates, rather than relying on the afterimages of first-order predictions; Step 4: Implement a survival mechanism based on heuristic action shielding. When the cluster becomes a "lone wolf" due to obstacle avoidance or strong maneuvering, the expert intervention logic is activated. Action interceptor logic: Define the action space C = {c0, c1, c2, c3}, where c0 is prediction tracking, and c1, c2, and c3 are frequency sweeping; If the number of currently active neighbors of drone i is detected |N active，i If |=0 and the deep neural network outputs the action of predicting and tracking c0, then masking resampling is implemented: c i，actual =Random({c1,c2,c3}) (8) This operation forces the Lone Wolf drone to abandon blind tracking based on outdated routes and switch to wide-area radar frequency scanning, which greatly improves the probability of self-healing after network outage. Step 5: Calculation of the full subnet joint liability reward function; To guide the UAV to maintain large-scale network connectivity, a full subnet joint liability reward function with amplitude limiting is used, and the following multi-dimensional coupled reward is designed: Progress and connectivity rewards: based on high water level P min_dist Calculate penetration score: r prog =ω p ·max(0,P min_dist -||p i，t -p target ||) (9) Subnet Split Penalty Truncation: Calculating the Subnet Number K using the Disjoint-Set Union Algorithm sub Implement bandwidth-limiting penalties to crack down on "suicidal strategies": R subnet =max((K sub -1)·c sub ,R limit ) (10) Where K sub γ represents the number of subnets in the current communication topology. sub As the penalty base, R limit To truncate the lower limit; Through |R limit |<<|R collision The constraint of | ensures that the agent retains its breakthrough momentum when the network breaks down, rather than actively destroying obstacles to avoid long-term point deductions, where R collision This is the collision penalty value when a drone collides with another drone. It is usually negative and is used to punish dangerous behaviors such as drones colliding with obstacles, colliding with other drones, or triggering a crash. Step 6: Training is conducted using a multi-stage course learning strategy; to overcome the reward sparsity problem under narrow beam conditions, this embodiment adopts a tiered course learning paradigm: Beam contraction path: Phase 1: The beamwidth is set to 30° to guide the agent to learn basic swarm navigation and obstacle avoidance capabilities; Phase 2: The beamwidth is reduced to 16° to train the agent's link maintenance accuracy during maneuvers; The third stage: the beamwidth is reduced to the final target of 6°, and rapid convergence is achieved through the pre-trained weights in the early stage; Gradient update parameters: in calculating generalized advantage estimation Then, the Actor parameters are updated through the PPO proxy objective function: in For Actor policy network parameters, The ratio of the probability of actions under the new and old strategies is given, and ε is the PPO shearing coefficient, which controls the magnitude of strategy updates. Step 7: Distributed online execution and cluster topology closed-loop collaborative control; After completing the above course learning and training, save the converged Actor policy network parameters, and deploy the shared policy network to each UAV node using a centralized training and distributed execution approach; During the online execution phase, each drone does not need to obtain the complete global state of the drone cluster. Instead, it independently completes scanning tactical decisions based on its own motion state, local neighbor connection state, obstacle perception information, and local routing history information. For the i-th UAV, the local observation state o is acquired at time step t. i，t The results are then fed into the trained Actor policy network to obtain the action probability distributions for four structured scanning tactics: in, Let represent the probabilities of UAV i choosing predictive scanning, forward scanning, lateral scanning, and backward scanning tactics, respectively, and satisfy the following conditions: Considering that narrow-beam neighbor discovery requires complementary spatial scanning directions among different UAVs, this invention does not employ a deterministic execution method where all nodes uniformly select the action with the highest probability. Instead, it performs tactical sampling according to the action probability distribution output by the policy network, i.e.: Among them, c i，t This indicates the scanning tactic selected by UAV i at time step t; the UAV calculates the corresponding beam azimuth and elevation angles based on the selected tactic, and adjusts the main lobe direction of the directional antenna to perform narrow beam directional scanning; After completing the beam direction update, the UAV performs a two-way beam handshake determination based on the spatial distance between nodes and the beam coverage status of both the transmitter and receiver. When UAV i and UAV j simultaneously meet the communication distance constraint and the two-way beam alignment condition, a valid neighbor link is established and the link keep-alive timer is refreshed. If the link fails to re-align the beam within the specified keep-alive time, the corresponding node is removed from the active neighbor set. For rediscovered neighbor nodes, the UAV writes their current location, speed, and discovery time into the local routing information table and uses the currently valid neighbor links to synchronize routing status; the updated set of active neighbors participates in the topology retention calculation for the next time step, and the routing history information is used for the location prediction of lost neighbors and the execution of predictive scanning tactics; Let the overall state of the drone swarm at time step t be S. t The scanning tactics set for each UAV is C. t ={c 1，t c 2，t c N，t The communication topology formed after completing the two-way beam handshake is G. t The drone cluster state update process can be represented as follows: in, This represents the state transition process of a drone swarm, which consists of directional neighbor discovery, link keep-alive update, routing information synchronization, topology preservation calculation, target guidance, and obstacle avoidance maneuvers. Through the above process, a closed-loop collaborative mechanism is formed, consisting of "local state awareness - scanning tactical selection - beam direction adjustment - bidirectional neighbor discovery - route state update - topology cooperative control - local state feedback". Each UAV re-makes scanning tactical decisions based on the updated local network state at each time step, enabling the UAV swarm to continuously perform neighbor rediscovery and link reconstruction during dynamic topology disturbances caused by takeoff, stable flight, and obstacle avoidance, thereby achieving distributed dynamic networking and topology self-recovery under 6° narrow beam communication conditions.