Unmanned aerial vehicle cluster swarming motion navigation method based on reinforcement learning

Abstract

The invention provides an unmanned aerial vehicle cluster swarming motion navigation method based on reinforcement learning, and relates to the technical field of unmanned aerial vehicle cluster motion navigation. The method comprises the steps of firstly initializing an unmanned aerial vehicle cluster environment, and determining a state space and an action space of an unmanned aerial vehicle; then defining a reward function to measure the performance of the unmanned aerial vehicle cluster in different states and executing different actions; and the award function is optimized to guide the swarming motion navigation of the unmanned aerial vehicle cluster. An HGTD3 algorithm is adopted to construct a deep reinforcement learning model so as to realize the fleeing and navigation of the unmanned aerial vehicle; the NIFA algorithm is adopted to calculate and execute the next action for the lost unmanned aerial vehicle, and unmanned aerial vehicle cluster motion control is kept; and finally, through interaction with the environment, training the constructed deep reinforcement learning model. According to the method, the unmanned aerial vehicle cluster can be trained more efficiently by using the HGTD3-NIFA algorithm, and the task can still be stably and accurately executed in the face of complex and changeable network interference conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) cluster motion navigation, and in particular to a UAV cluster swarming motion navigation method based on reinforcement learning. Background Art

[0002] Over the past 20 years, the use of drones has grown exponentially in various fields, including military and civilian applications such as aerial photography, search and rescue missions, target tracking, forest fire prevention, crowd monitoring, and agricultural spraying. To achieve complex and large-scale tasks, it is necessary to develop from a single drone system to a multi-drone collaborative system. Among them, team collaboration between drones and autonomous obstacle avoidance pose key and fundamental challenges to multi-drone systems. The concept of swarm motion is an important basis for achieving multi-drone collaborative behavior. In a swarm, a group of drones with a common goal perform collective motion in seamless coordination, such as moving from a starting point to a predetermined destination. Swarm behavior occurs when a group moves according to simple rules for a common goal, namely collision avoidance, speed matching, and cluster centering.

[0003] In actual environments, drones are inevitably interfered with, resulting in inconsistent information sharing between drones. If a drone that has lost communication with other drones is defined as a lost drone, then the lost drone cannot effectively perform a clustering algorithm based on information sharing because it cannot receive the required information from other drones, reducing the overall clustering performance. Current research focuses on communications dominated by strong line-of-sight links between drones and cellular base stations, and pays less attention to interference within drone clusters. To this end, it is necessary to propose a clustering algorithm for drones in an interference environment.

[0004] Traditional drone swarm predefined structure control methods rely on rule-based algorithms, which are usually tailored for uncommon environments in reality, which limits their applicability. Therefore, the TD3 algorithm based on reinforcement learning is used, which is more adaptable than traditional methods and does not require a fixed cluster configuration. In addition, to improve training efficiency, reinforcement learning is combined with heuristic algorithms to solve the long learning cycle of typical reinforcement learning. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a swarming motion navigation method of UAV clusters based on reinforcement learning in view of the deficiencies of the above-mentioned prior art, so as to realize multi-UAV navigation and obstacle avoidance in an unknown random environment.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a swarming motion navigation method of drone clusters based on reinforcement learning, comprising the following steps:

[0007] Step 1: Initialize the UAV cluster environment, including setting the initial position, target position and sensor information of the UAV; at the same time, determine the range of the UAV's state space and action space;

[0008] Set obstacles as circles with different radii in the simulated flight environment; the initial position of the drone will not be in the circular area or the target point area; when the drone encounters an obstacle during flight, the training process of the drone ends;

[0009] The state space refers to a set of variables used to describe the current state of the drone cluster, including the position coordinates, speed and attitude of the drone; the action space refers to the set of actions that the drone cluster can choose;

[0010] Step 2: Define a reward function to measure the performance of the drone swarm in different states and when performing different actions; and guide the swarming navigation of the drone swarm by optimizing the reward function; the reward function takes into account multiple factors such as the distance between drones, target achievement, obstacle avoidance, and energy consumption;

[0011] The reward function is defined into 5 categories: target approach reward, cluster reward, avoidance reward, target arrival reward and step reward;

[0012] The target proximity reward is used to guide the UAV to navigate to the target point. First, the distance between the current position of the UAV i and the target point is calculated. Then, this distance is subtracted from the distance between the position of the UAV at the previous moment and the target point. The target proximity reward r1 is defined as:

[0013]

[0014] Among them, w mov is a normal number, determined according to the time requirement of the drone cluster. is the position of UAV i at time t-1, is the position of UAV i at time t, <x tar ,y tar > is the target point position;

[0015] The cluster reward is used to help drones maintain a set distance from other drones, thereby promoting the cluster state between multiple drones; the cluster reward r2 is defined as:

[0016]

[0017] Among them, w con is a positive number, determined according to the time requirement of the drone cluster, d1 is the minimum safe distance of the drone, d3 is the farthest communication distance of the drone, N is the number of adjacent drones, is the position of UAV j adjacent to UAV i at time t;

[0018] The avoidance reward is used to help the drone maintain a safe distance from static obstacles, interference and neighboring drones to promote collision avoidance. The interference source is defined as a circular area, where the inner circle represents severe interference and the outer circle represents insignificant interference. The avoidance reward of the drone for neighboring drones r3, obstacle avoidance r4 and interference source r5 is defined as:

[0019]

[0020]

[0021] Among them, w nei 、w obs 、w int1 and w int2 are all negative constants, determined according to the time requirements of the drone cluster. <x m ,y m > is the position of the mth obstacle, O is the number of obstacles, <x int ,y int > is the location of the interference source, d2 is the safe distance between the drone and the obstacle, d5 and d6 are the inner and outer circle distances of the drone affected by the interference source;

[0022] The target arrival reward is used to grant the drone a target arrival reward after it reaches the specified target point. The target point is represented as a circular area with a radius of , and the target arrival reward r6 is:

[0023]

[0024] Among them, w tar is a positive constant, determined according to the time requirement of the drone cluster, and d4 is the radius of the target point;

[0025] In addition to the previously mentioned reward functions for clustering and navigation behaviors, an additional step reward function is introduced to improve the training performance of the drone. The step reward r7 is defined as:

[0026] r7=w step

[0027] Among them, w step is a positive number, determined according to the time requirements of the drone cluster;

[0028] After each action taken by the drone, the cumulative reward is calculated as a sum; therefore, for drone i in T time steps, the comprehensive reward function is as follows:

[0029]

[0030] in, is the cumulative reward of drone i at time t, η n The weights of various rewards;

[0031] Step 3: Use the TD3 algorithm combined with the heuristic algorithm to form the HGTD3 algorithm to build a deep reinforcement learning model to achieve drone swarming and navigation;

[0032] The drone cluster is assumed to include communication drones and lost drones. The communication drone is equipped with a deep reinforcement learning model to provide decision-making services for cluster movement. The state information required for the training process is extracted from the environment, and the control strategy of the drone is output in a timely manner using the pre-trained deep reinforcement learning model.

[0033] When the lost drone loses communication and cannot obtain status information from the environment, the drone uses sensors to detect the location of nearby drones; uses the NIFA algorithm to calculate and execute the next action and maintain control of the drone cluster movement; and uses a deep reinforcement learning model to provide decision-making services when the lost drone's communication is restored;

[0034] In order to speed up the training of deep reinforcement learning models, heuristic guidance is introduced on the basis of TD3 algorithm to realize the swarming and navigation of drones. The heuristic guidance includes three parts: drone guidance, drone obstacle avoidance, and drone clustering.

[0035] In order to guide the drone to move to the target, the angle θ1 between the drone and the target point is calculated, and the current flight angle of the drone is And convert both angles into the range between π and -π, and then determine the angular velocity acceleration φ1 guided by the drone, as shown in the following formula:

[0036]

[0037] In the context of drone avoidance, safety is ensured by avoiding obstacles and interference sources within the drone's perception range. The angle θ2 between the drone and the obstacle or interference source is calculated, and the angular velocity acceleration φ2 of the drone's obstacle avoidance is determined, as shown in the following formula:

[0038]

[0039] When considering the UAV swarm behavior, the average angular velocity of all UAVs is calculated, and the flight angle w of each UAV is calculated. t i Compared with the average angular velocity, the angular velocity acceleration φ3 of the drone cluster is determined as shown in the following formula:

[0040]

[0041] In addition, in the context of UAV group movement, the average flight speed of UAVs is calculated, and the flight speed of each UAV is Compare and determine the flight acceleration of the drone As shown in the following formula:

[0042]

[0043] The final action of the UAV is determined by considering the directions of the three behaviors of guidance, obstacle avoidance, and clustering. The change of the UAV's flight angular velocity acceleration is controlled by adding the effects of the three behaviors of guidance, obstacle avoidance, and clustering, which is recorded as It remains within a specific range; the change in the UAV's flight speed is recorded as As shown in the following formula:

[0044]

[0045] The method for the lost drone to use the NIFA algorithm to calculate and execute the next action and maintain the motion control of the drone cluster is:

[0046] The NIFA algorithm uses the sparrow search algorithm to control the cluster motion of the UAVs that have lost contact with each other within the unmanned cluster. In order to maintain the cluster behavior of the UAVs after being disturbed, a fitness function that satisfies both the cluster maintenance and obstacle avoidance conditions is designed. In addition, the UAVs use sensors to detect the positions of their neighboring UAVs and obstacles, so that they can make the right decision about their next flight position. The NIFA algorithm consists of three parts: UAV clustering, UAV obstacle avoidance, and distance optimization.

[0047] Aiming at the problem that individual drones cannot adopt collective swarming rules in the regular communication process, a fitness function that complies with the boid rule is designed;

[0048] Assume l ns Represents the distance between the current drone and its neighbors, defined as follows:

[0049]

[0050] in, x ,s y >Indicates the possible location of the missing drone. The locations of the drones around the lost drone;

[0051] The fitness function F1 of the drone swarm is defined as follows:

[0052]

[0053] ​By measuring the distance from the center of the interference source to the UAV, it is evaluated whether the UAV and the interference source are about to collide, so as to ensure that the UAV cluster can navigate around the interference source while maintaining cluster behavior; if the distance is lower than the predetermined safety threshold, a negative fitness value is assigned, indicating that a collision has occurred; let l is Indicates the distance from the current drone to the interference source, defined as follows:

[0054]

[0055] The fitness function F2 for interference source avoidance is defined as follows:

[0056]

[0057] In order to minimize the cumulative flight distance of each UAV while maintaining its clustering behavior throughout the mission, the shortest flight distance among all feasible next positions is calculated based on the UAV’s current position, but still maintaining the clustering state; the fitness function F3 that minimizes the cumulative flight distance is defined as follows:

[0058]

[0059] in, The location of the missing drone;

[0060] The fitness function of the final NIFA algorithm is composed of all the above fitness functions, which enables the interfered UAV to comply with the clustering behavior and obstacle avoidance rules and locate to the most suitable position;

[0061] The final fitness function F of the lost drone is obtained by summing up the fitness functions of each part:

[0062]

[0063] Among them, w a is the weight of each fitness function;

[0064] Step 4: Train the deep reinforcement learning model. The constructed deep reinforcement learning model is trained through interaction with the environment. At each time step, the drone cluster selects an action based on the current state and executes the action. Then, based on the execution results and the feedback of the reward function, the parameters of the deep reinforcement learning model are updated to improve the performance of the drone cluster.

[0065] First, initialize the Actor network and Critic network, and create the target network;

[0066] Then enter the training loop; in each training cycle, obtain sample data from the environment; first, compare the random sampling value of the Actor network with the set threshold to determine whether to use heuristics to select an action or use the current Actor network to select an action. If it is less than the threshold, the heuristic is applied, otherwise the Actor network is used to select an action and apply it to the environment; then observe the next state and immediate reward;

[0067] Update the Critic network; first, calculate the Q value of the next state action on the target network, where the target network parameters are updated by periodically executing a soft update strategy; then, use the difference between these Q values ​​and the Q value of the current state action to calculate the Critic loss function, and minimize the function to update the Critic network parameters; then, update the Actor network; use the current Actor network to select an action and use the Critic network to evaluate the Q value of the action; then, adjust the Actor network parameters through gradient ascent to maximize the Q value of the selected action to optimize the Actor objective function;

[0068] After completing the update of the Actor and Critic networks, before taking the soft update strategy to gradually bring the target network parameters closer to the current network parameters, continue to obtain sample data and iteratively optimize the Critic and Actor networks;

[0069] Repeat the above training process until the stopping condition is met.

[0070] The beneficial effect of adopting the above technical solution is that: the present invention provides a method for swarming motion navigation of unmanned aerial vehicle clusters based on reinforcement learning, which realizes the swarming and navigation of unmanned aerial vehicle clusters in a communication-restricted environment. First, in order to improve the training efficiency, in the initial reinforcement learning process, the heuristic is combined with the unmanned aerial vehicle action selection, and an improved heuristic-guided TD3 algorithm, namely the HGTD3 algorithm, is proposed to enhance convergence and improve the swarming, navigation and obstacle avoidance effects. In order to further improve the performance of unmanned aerial vehicle clusters in a network interference environment, a network interference clustering algorithm NIFA is also designed. In a network interference environment, the algorithm uses a sparrow search algorithm to maintain the swarming behavior and obstacle avoidance of disconnected unmanned aerial vehicles, and designs an adaptive function to meet the conditions of swarming maintenance and obstacle avoidance at the same time, thereby ensuring the swarming and navigation efficiency of the entire swarm. In addition, the TD3 algorithm in the reinforcement learning technology is used in the method of the present invention to optimize the training efficiency. By using the HGTD3-NIFA algorithm, unmanned aerial vehicle clusters can be trained more efficiently, and tasks can still be performed stably and accurately in the face of complex and changeable network interference situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1A schematic diagram of drone swarming provided by an embodiment of the present invention;

[0072] Figure 2 A schematic diagram of a UAV flight motion model provided by an embodiment of the present invention;

[0073] Figure 3 A schematic diagram of the framework of the UAV swarm motion and navigation method provided by an embodiment of the present invention;

[0074] Figure 4 A flowchart of the reinforcement learning algorithm HGTD3 provided in an embodiment of the present invention;

[0075] Figure 5 A schematic diagram of a heuristic algorithm provided by an embodiment of the present invention;

[0076] Figure 6 An algorithm flow chart for solving the swarming problem of drone clusters in interference environments by combining the reinforcement learning algorithm HGTD3 provided in an embodiment of the present invention with the NIFA algorithm for maintaining swarming of lost drones. DETAILED DESCRIPTION

[0077] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0078] In this embodiment, a drone swarming navigation method based on reinforcement learning includes the following steps:

[0079] Step 1: Initialize the drone cluster environment, including setting the initial position, target position and sensor information of the drone; at the same time, determine the range of the drone's state space and action space so that the subsequent reinforcement learning algorithm can be trained;

[0080] Set obstacles as circles with different radii in the simulated flight environment; the initial position of the drone will not be in the circular area or the target point area; when the drone encounters an obstacle during flight, the training process of the drone ends;

[0081] The state space refers to a set of variables used to describe the current state of the drone cluster, including the position coordinates, speed and attitude of the drone. When initializing the environment, the range and value of these state variables need to be clearly defined to provide them to the reinforcement learning algorithm. The speed of the drone is limited to 2m / s-6m / s, and the angular velocity of the drone is limited to The main purpose is to prevent the drone from flying backwards.

[0082] The action space refers to the set of actions that the drone cluster can choose; when initializing the environment, it is necessary to determine the range of action space for each drone, such as setting the maximum speed and maximum steering angle. This helps limit the range of actions of the drone during navigation and enables the reinforcement learning algorithm to better train and make decisions.

[0083] Step 2: Define a reward function to measure the performance of the drone swarm in different states and when performing different actions; and guide the swarming navigation of the drone swarm by optimizing the reward function; the reward function takes into account multiple factors such as the distance between drones, target achievement, obstacle avoidance, and energy consumption;

[0084] The reward function is defined into 5 categories: target approach reward, cluster reward, avoidance reward, target arrival reward and step reward;

[0085] The target proximity reward is used to guide the drone to navigate to the target point; first calculate the distance between the current position of drone i and the target point; then subtract this distance from the distance between the drone's position and the target point at the previous moment; the larger the magnitude of this difference, the longer the distance covered by the drone in a single step. Our goal is to maximize this difference. The target proximity reward r1 is defined as:

[0086]

[0087] Among them, w mov is a normal number, determined according to the time requirement of the drone cluster. is the position of UAV i at time t-1, is the position of UAV i at time t, <x tar ,y tar > is the target point position;

[0088] The cluster reward is used to help drones maintain a set distance from other drones, thereby promoting the cluster state among multiple drones. Specifically, it ensures that the current drone i maintains an appropriate distance from its neighboring drone j. The cluster reward r2 is defined as:

[0089]

[0090] Among them, w con is a positive number, determined according to the time requirement of the drone cluster, d1 is the minimum safe distance of the drone, d3 is the farthest communication distance of the drone, N is the number of adjacent drones, is the position of UAV j adjacent to UAV i at time t;

[0091] The avoidance reward is used to help the drone maintain a safe distance from static obstacles, interference and neighboring drones, and promote collision avoidance; collision conflicts will lead to the termination of this reinforcement learning training, and negative rewards will be used as penalties; similarly, if the distance between drones is less than the set threshold, a negative reward value is given, but the training is not terminated, and the drone is encouraged to adjust its position accordingly; the interference source is defined as a circular area, where the inner circle represents severe interference and the outer circle represents insignificant interference; the avoidance reward of the drone for the neighboring drone r3, the obstacle r4 and the interference source r5 is defined as:

[0092]

[0093] Among them, w nei 、w obs 、w int1 and w int2 are all negative constants, determined according to the time requirements of the drone cluster. <x m ,y m > is the position of the mth obstacle, O is the number of obstacles, <x int ,y int > is the location of the interference source, d2 is the safe distance between the drone and the obstacle, d5 and d6 are the inner and outer circle distances of the drone affected by the interference source;

[0094] The target arrival reward is used to grant the target arrival reward after the drone reaches the specified target point; once the drone i successfully reaches the target point, it will receive a substantial reward, marking the completion of the training process. The target point is represented as a circular area with a radius of , and the target arrival reward r6 is:

[0095]

[0096] Among them, w tar is a positive constant, determined according to the time requirement of the drone cluster, and d4 is the radius of the target point;

[0097] In addition to the reward functions for clustering and navigation behaviors mentioned above, an additional step reward function is introduced to improve the training performance of the UAV; this reward function is designed to encourage the UAV to complete the clustering and navigation tasks in a shorter time and guide the UAV to complete the clustering motion task in less time. The step reward r7 is defined as:

[0098] r7=w step

[0099] Among them, w step is a positive number, determined according to the time requirements of the drone cluster;

[0100] After each action taken by the drone, the cumulative reward is calculated as a sum; therefore, for drone i in T time steps, the comprehensive reward function is as follows:

[0101]

[0102] in, is the cumulative reward of drone i at time t, η n The weights of various rewards;

[0103] Step 3: Use the TD3 algorithm combined with the heuristic algorithm to form the HGTD3 algorithm to build a deep reinforcement learning model to achieve drone swarming and navigation;

[0104] Figure 1 The figure is a schematic diagram of supporting drone swarming in an embodiment of the present invention. The drones fly to the target point in the form of a cluster, and autonomously adjust their flight speed and direction during the flight, avoid obstacles during the flight, and maintain the cluster state. The interfered drones cannot share the detailed information of other drones and maintain the cluster state through simple location.

[0105] Figure 2 Schematic diagram of the UAV motion model used in the embodiment of the present invention. The time interval between two consecutive time steps is fixed to Δt = 1 second. Figure 2 As shown, at point A, Determine the turning signal and control the flight direction of the drone i, Control flight speed. (in rad / s) and (in m / s), negative values ​​indicate deceleration. Assume that the turn and throttle signals take effect immediately. Point B represents the position of drone i at t+1.

[0106] Each drone is equipped with a GPS device to obtain its position coordinates. The wireless transceiver carried by the drone facilitates communication and information exchange, including relevant data such as position and speed. We assume that all drones adopt a fixed-altitude flight mode, that is, only their movement trajectory in the horizontal plane needs to be determined.

[0107] Figure 3 The framework of the UAV swarm motion and navigation system used in the embodiment of the present invention is shown in FIG. UAVs can be divided into two categories, namely, communication UAVs and lost UAVs.

[0108] Communication UAV: ​​The communication UAV is equipped with a deep reinforcement learning (DRL) model to provide decision-making services for swarm motion. It extracts the state information required for the model training process from the environment and uses the pre-trained deep reinforcement learning model to output the control strategy of the UAV in a timely manner.

[0109] Lost drones: In the event of communication loss, when it is unable to obtain status information from the environment, the drone uses sensors to detect the location of nearby drones; calculates and executes the next action through the NIFA algorithm, and maintains control of the movement of the lost drone cluster;

[0110] After the introduction of the HGTD3-NIFA algorithm, both the communicating drones and the lost drones exhibited clustering behavior. By adopting methods such as shared data to train and distribute the drones, each drone can make independent decisions. The drones are trained using the HGTD3 algorithm, taking the information of neighboring drones as input, and using the critic network to calculate the q-values ​​of all actions. Based on the estimated q-values, the actuator network selects appropriate actions for each drone to execute. In the event of interference, NIFA is applied to maintain the performance of the entire cluster and continues until the network returns to normal. Subsequently, the reinforcement learning algorithm can restart the task.

[0111] Figure 4 The following is a flowchart of the reinforcement learning algorithm HGTD3 used in the embodiment of the present invention. Participants act as strategies in the training phase to interact with the environment and data collection. In order to fully explore the environment, random noise sampled from a normal distribution is added to the actions to be performed. The formula is as follows:

[0112]

[0113] The critic network is a value function estimator that estimates the t and a state s t Afterwards, the expectation of the cumulative reward that can be obtained by following the current actor (strategy) in the future is:

[0114]

[0115] Among them, τ represents the trajectory generated based on the transition probability p and the current strategy π, Q is the long-term reward, and R is the current reward value. TD3 uses a recursive update rule called Bellman to update the network parameters as follows:

[0116]

[0117] The target network makes the training process more stable by alleviating moving targets. The specific implementation is as follows:

[0118] θ←τθ+(1-τ)θ′

[0119] φ←τφ+(1-τ)φ′

[0120] To address the overestimation problem and enhance robustness, a dual Q-learning approach is used to replace the target value with the smaller of two target value networks, both of which are equipped with a value function estimator, as shown below:

[0121]

[0122] The policy is then updated to improve performance. To update the policy network, the policy gradient is calculated as shown in the following formula, and then the policy is updated by gradient ascent.

[0123]

[0124] To get an accurate estimate of the value function, the policy network is updated less frequently than the value network. Specifically, the policy is updated once after updating the value function twice. This approach ensures a high-quality estimate of the value function.

[0125] The TD3 algorithm differs from other algorithms in that it incorporates exploration variance during training. The TD3 algorithm is enhanced by introducing variable exploration noise. To control exploration, the exploration noise is gradually reduced at different stages of training.

[0126] First, a large exploration noise is used to encourage the drone to take random actions and collect relevant training data. As training progresses, this exploration noise is gradually reduced to promote smoother transitions. In the later stages of training, the exploration noise is minimized to promote convergence.

[0127] Before training, use a constantly changing exploration noise and policy noise. When the training iteration reaches the starting point of the change, the exploration noise begins to decrease linearly from the maximum to the minimum value, and the policy noise also needs to be adjusted. Set the value of the exploration noise to 2 times. When the exploration noise is reduced to the minimum, no further adjustment is required to make the training reach a stable state. During the end-to-end change training process, the exploration noise is kept at the minimum value until the end of training.

[0128] Figure 5 The figure is a schematic diagram of the heuristic guidance of the drone swarm used in an embodiment of the present invention. In order to improve the drone's ability to explore the environment and improve the training efficiency, several heuristic rules are combined with the greedy strategy to select actions. However, in the early stages of training, the network's training completeness level is relatively low, which often leads to incorrect action selection and low exploration efficiency. Therefore, some heuristic rules are defined to help the agent in the early stages of model training. This includes implementing path planning and obstacle avoidance, which allows more valuable training experience to be collected in the early stages and convergence to be reached faster. During the training process, a heuristic auxiliary action selection module is added to the action selection. This module has three main functions: drone guidance, drone avoidance, and drone swarming.

[0129] In order to guide the drone to move to the target, we calculate the angle θ1 between the drone and the target point. To simplify the calculation, the angle of the drone's flight is converted into a range between π and -π. The current flight angle of the drone. A small difference from θ1 indicates that the drone is flying towards the target point and needs to make minimal adjustments to its direction. Need to increase The angular velocity acceleration φ1 generated by the heuristic algorithm depends on θ1 and If the angle between the two is large, it means that the drone is heading off the correct direction and needs to be adjusted. When the angle between the two is small, it means that the drone is flying towards the target point and there is no need to adjust the direction of the drone excessively. In addition, if θ1 is less than You need to reduce The calculation formula is as follows:

[0130]

[0131] In the context of drone avoidance, safety is ensured by avoiding obstacles and interference sources that appear within the drone’s sensing range. When the angle θ2 between the drone and the obstacle or interference source approaches , it indicates that the drone is currently approaching an obstacle or interference source and needs to change the turning direction to avoid collision. The closer the alignment between θ and θ2, the higher the risk of collision with obstacles or interference sources, and the more significant adjustments to the turning angle are needed. If θ and θ2 show significant differences, there is no need to make substantial adjustments to the turning angle. In summary, the angular velocity acceleration calculation formula generated by the heuristic algorithm is:

[0132]

[0133] Furthermore, in the context of drone group motion, the similarity of the drones’ flight directions plays a key role. According to Boid’s first rule, the closer the flight directions between drones are aligned, the more ideal the clustering is. If the difference exceeds the average angular velocity, it should be reduced proportionally, and the larger the difference, the more drastic the adjustment. If the value is lower than the average, the same operation is performed to keep in sync. The angular velocity acceleration calculation formula generated by the heuristic algorithm is as follows:

[0134]

[0135] When considering the swarm behavior of drones, the flight speed of drone i at time t is and Two motion factors are crucial. To achieve effective clustering, all drones should have similar flight speeds. The closer the flight speeds of the drones are, the better it is for achieving the desired clustering effect. We calculate the average flight speed of all drones and then compare the speeds of each drone. If If the difference between the two speeds is greater, the change will be greater. If the value is lower than the average value, the same operation is performed to maintain consistency; in summary, the calculation formula for the flight acceleration of the drone is as follows:

[0136]

[0137] The final motion of the drone is determined by considering the above three directions. The change of angular velocity acceleration is controlled by adding up the effects of these situations, recorded as It remains within a specific range. The change in the flight speed of the drone is recorded as The final formula is calculated as:

[0138]

[0139] The method for the lost drone to use the NIFA algorithm to calculate and execute the next action and maintain the motion control of the drone cluster is:

[0140] The NIFA algorithm uses the sparrow search algorithm to control the cluster motion of the UAVs that have lost contact with each other within the unmanned cluster. In order to maintain the cluster behavior of the UAVs after being disturbed, a fitness function that satisfies both the cluster maintenance and obstacle avoidance conditions is designed. In addition, the UAVs use sensors to detect the positions of their neighboring UAVs and obstacles, so that they can make the right decision about their next flight position. The NIFA algorithm consists of three parts: UAV clustering, UAV obstacle avoidance, and distance optimization.

[0141] In order to maintain the flocking behavior of drones and recognize the importance of simulating the observed flocking pattern, this principle is incorporated into the design of the fitness function of the sparrow search algorithm to ensure that any interference encountered by the drone will not affect its flocking state; in view of the problem that individual drones cannot adopt the collective flocking rule during regular communication, a fitness function that obeys the boid rule is designed; let l ns Represents the distance between the current drone and its neighbors, defined as follows:

[0142]

[0143] in, <sx ,s y >Indicates the possible location of the missing drone. The locations of the drones around the lost drone;

[0144] The fitness function F1 of the drone swarm is defined as follows:

[0145]

[0146] By measuring the distance from the center of the interference source to the UAV, it is evaluated whether the UAV and the interference source are about to collide, so as to ensure that the UAV cluster can navigate around the interference source while maintaining cluster behavior; if the distance is lower than the predetermined safety threshold, a negative fitness value is assigned, indicating that a collision has occurred; let l is Indicates the distance from the current drone to the interference source, defined as follows:

[0147]

[0148] The fitness function F2 for interference source avoidance is defined as follows:

[0149]

[0150] In order to minimize the cumulative flight distance of each drone while maintaining its clustering behavior throughout the mission; since each step of the drone flight mission is restricted to a limited range, the lost drone is in a circular area as the center point, and the above rules may lead to suboptimal positioning. Therefore, the shortest flight distance among all feasible next positions is calculated based on the current position of the drone, but still maintaining the clustering state; the fitness function F3 that minimizes the cumulative flight distance is defined as follows:

[0151]

[0152] in, The location of the missing drone;

[0153] The fitness function of the final NIFA algorithm is composed of all the above fitness functions, which enables the interfered UAV to comply with the clustering behavior and obstacle avoidance rules and locate to the most suitable position;

[0154] The final fitness function F of the lost drone is obtained by summing up the fitness functions of each part:

[0155]

[0156] Among them, w a is the weight of each fitness function;

[0157] Step 4: Train the deep reinforcement learning model; train the constructed reinforcement learning model through interaction with the environment; at each time step, the drone cluster selects an action based on the current state and executes the action; then based on the execution results and the feedback of the reward function, update the parameters of the deep reinforcement learning model to improve the performance of the drone cluster;

[0158] Figure 6 The flowchart of the reinforcement learning algorithm HGTD3 used in the embodiment of the present invention is shown in FIG. 1. At the beginning of the algorithm, the Actor network and the Critic network are initialized, and the target network is created;

[0159] Then enter the training loop; in each training cycle, obtain sample data from the environment; first, compare the random sampling value of the Actor network with the set threshold to determine whether to use heuristics to select an action or use the current Actor network to select an action. If it is less than the threshold, the heuristic is applied, otherwise the Actor network is used to select an action and apply it to the environment; then observe the next state and immediate reward;

[0160] Update the Critic network; first, calculate the Q value of the next state action on the target network, where the target network parameters are updated by periodically executing a soft update strategy; then, use the difference between these Q values ​​and the Q value of the current state action to calculate the Critic loss function, and minimize the function to update the Critic network parameters; then, update the Actor network; use the current Actor network to select an action and use the Critic network to evaluate the Q value of the action; then, adjust the Actor network parameters through gradient ascent to maximize the Q value of the selected action to optimize the Actor objective function;

[0161] After completing the update of the Actor and Critic networks, before taking the soft update strategy to gradually bring the target network parameters closer to the current network parameters, continue to obtain sample data and iteratively optimize the Critic and Actor networks;

[0162] Repeat the above training process until the stopping condition is met.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A swarming navigation method for drone swarms based on reinforcement learning, characterized by: The following steps are involved: Step 1: Initialize the UAV cluster environment, including setting the initial position, target position and sensor information of the UAV; at the same time, determine the range of the UAV's state space and action space; Step 2: Define a reward function to measure the performance of the drone swarm in different states and when performing different actions; and guide the swarming navigation of the drone swarm by optimizing the reward function; the reward function takes into account multiple factors such as the distance between drones, target achievement, obstacle avoidance, and energy consumption; Step 3: Use the TD3 algorithm combined with the heuristic algorithm to form the HGTD3 algorithm to build a deep reinforcement learning model to achieve swarming and navigation of drones; use the NIFA algorithm to calculate and execute the next action for the lost drones, and maintain the control of the drone cluster movement; Step 4: Train the deep reinforcement learning model; train the constructed deep reinforcement learning model through interaction with the environment; in each time step, the drone cluster selects an action based on the current state and executes the action; then based on the execution results and the feedback of the reward function, update the parameters of the deep reinforcement learning model to improve the performance of the drone cluster.

2. The method for swarming motion navigation of drone clusters based on reinforcement learning according to claim 1, characterized in that: In step 1, the obstacles are set as circles with different radii in the simulated flight environment; the initial position of the drone will not be initially within the circular area, nor will it be initially within the target point area; when the drone encounters an obstacle during flight, the training process of the drone is terminated; The state space refers to a set of variables used to describe the current state of the drone cluster, including the position coordinates, speed, and attitude of the drone; The action space refers to the set of actions that a drone swarm can choose.

3. The method for swarming motion navigation of drone clusters based on reinforcement learning according to claim 2, characterized in that: Step 2 defines the reward function into five categories: target approach reward, cluster reward, avoidance reward, target arrival reward and step reward; The target proximity reward is used to guide the drone to navigate to the target point. First, the distance between the current position of drone i and the target point is calculated; This distance is then subtracted from the distance between the drone’s previous position and the target point; the target proximity reward r1 is defined as: Among them, w mov is a normal number, determined according to the time requirement of the drone cluster. is the position of UAV i at time t-1, is the position of UAV i at time t, <x tar ,y tar > is the target point position; The cluster reward is used to help drones maintain a set distance from other drones, thereby promoting the cluster state between multiple drones; the cluster reward r2 is defined as: Among them, w con is a positive number, determined according to the time requirement of the drone cluster, d1 is the minimum safe distance of the drone, d3 is the farthest communication distance of the drone, N is the number of adjacent drones, is the position of UAV j adjacent to UAV i at time t; The avoidance reward is used to help the drone maintain a safe distance from static obstacles, interference and neighboring drones to promote collision avoidance. The interference source is defined as a circular area, where the inner circle represents severe interference and the outer circle represents insignificant interference. The avoidance reward of the drone for neighboring drones r3, obstacle avoidance r4 and interference source r5 is defined as: Among them, w nei 、w obs 、w int1 and w int2 are all negative constants, determined according to the time requirements of the drone cluster. <x m ,y m > is the position of the mth obstacle, O is the number of obstacles, <x int ,y int > is the location of the interference source, d2 is the safe distance between the drone and the obstacle, d5 and d6 are the inner and outer circle distances of the drone affected by the interference source; The target arrival reward is used to grant the drone a target arrival reward after it reaches the specified target point. The target point is represented as a circular area with a radius of , and the target arrival reward r6 is: Among them, w tar is a positive constant, determined according to the time requirement of the drone cluster, and d4 is the radius of the target point; In addition to the previously mentioned reward functions for clustering and navigation behaviors, an additional step reward function is introduced to improve the training performance of the drone. The step reward r7 is defined as: r7＝w step Among them, w step is a positive number, determined according to the time requirements of the drone cluster; After each action taken by the drone, the cumulative reward is calculated as a sum; therefore, for drone i in T time steps, the comprehensive reward function is as follows: in, is the cumulative reward of drone i at time t, η n The weights of various rewards.

4. The method for swarming motion navigation of drone clusters based on reinforcement learning according to claim 3 is characterized by: In step 3, it is assumed that the drone cluster includes communication drones and lost drones; wherein the communication drone is equipped with a deep reinforcement learning model to provide decision-making services for cluster movement; the state information required for the training process is extracted from the environment, and the control strategy of the drone is output in time using the pre-trained deep reinforcement learning model; in order to speed up the training of the deep reinforcement learning model, heuristic guidance is introduced on the basis of the TD3 algorithm to realize the swarming and navigation of the drone; When the lost drone loses communication and cannot obtain status information from the environment, the drone uses sensors to detect the location of nearby drones; the NIFA algorithm is used to calculate and execute the next action and maintain control of the drone cluster movement; when the lost drone's communication is restored, the deep reinforcement learning model is used to provide decision-making services.

5. The method for swarming motion navigation of drone clusters based on reinforcement learning according to claim 4, characterized in that: The heuristic guidance includes three parts, namely, guidance of drones, obstacle avoidance of drones, and clustering of drones; In order to guide the drone to move to the target, the angle θ1 between the drone and the target point is calculated, and the current flight angle of the drone is And convert both angles into the range between π and -π, and then determine the angular velocity acceleration φ1 guided by the drone, as shown in the following formula: In the context of drone avoidance, safety is ensured by avoiding obstacles and interference sources within the drone's perception range. The angle θ2 between the drone and the obstacle or interference source is calculated, and the angular velocity acceleration φ2 of the drone's obstacle avoidance is determined, as shown in the following formula: When considering the behavior of drone swarms, the average angular velocity of all drones is calculated, and the flight angle of each drone is Compared with the average angular velocity, the angular velocity acceleration φ3 of the drone cluster is determined as shown in the following formula: In addition, in the context of UAV group movement, the average flight speed of UAVs is calculated, and the flight speed of each UAV is Compare and determine the flight acceleration of the drone As shown in the following formula: The final action of the UAV is determined by considering the directions of the three behaviors of guidance, obstacle avoidance, and clustering. The change of the UAV's flight angular velocity acceleration is controlled by adding the effects of the three behaviors of guidance, obstacle avoidance, and clustering, which is recorded as It remains within a specific range; the change in the UAV's flight speed is recorded as As shown in the following formula:

6. The method for swarming motion navigation of drone clusters based on reinforcement learning according to claim 4, characterized in that: The method for the lost drone to use the NIFA algorithm to calculate and execute the next action and maintain the motion control of the drone cluster is: The NIFA algorithm uses the sparrow search algorithm to control the cluster motion of the UAVs that have lost contact with each other within the unmanned cluster. In order to maintain the cluster behavior of the UAVs after being disturbed, a fitness function that satisfies both the cluster maintenance and obstacle avoidance conditions is designed. In addition, the UAVs use sensors to detect the positions of their neighboring UAVs and obstacles, so that they can make the right decision about their next flight position. The NIFA algorithm consists of three parts: UAV clustering, UAV obstacle avoidance, and distance optimization. Aiming at the problem that individual drones cannot adopt collective swarming rules in the regular communication process, a fitness function that complies with the boid rule is designed; Assume l ns Represents the distance between the current drone and its neighbors, defined as follows: in, x ,s y >Indicates the possible location of the missing drone. The locations of the drones around the lost drone;​ The fitness function F1 of the drone swarm is defined as follows: By measuring the distance from the center of the interference source to the UAV, it is evaluated whether the UAV and the interference source are about to collide, so as to ensure that the UAV cluster can navigate around the interference source while maintaining cluster behavior; if the distance is lower than the predetermined safety threshold, a negative fitness value is assigned, indicating that a collision has occurred; let l is Indicates the distance from the current drone to the interference source, defined as follows: The fitness function F2 for interference source avoidance is defined as follows: In order to minimize the cumulative flight distance of each UAV while maintaining its clustering behavior throughout the mission, the shortest flight distance among all feasible next positions is calculated based on the UAV’s current position, but still maintaining the clustering state; the fitness function F3 that minimizes the cumulative flight distance is defined as follows: in, The location of the missing drone; The fitness function of the final NIFA algorithm is composed of all the above fitness functions, which enables the interfered UAV to comply with the clustering behavior and obstacle avoidance rules and locate to the most suitable position; The final fitness function F of the lost drone is obtained by summing up the fitness functions of each part: Among them, w a is the weight of each fitness function.

7. The method for swarming motion navigation of drone clusters based on reinforcement learning according to claim 4, characterized in that: The specific method of step 4 is: First, initialize the Actor network and Critic network, and create the target network; Then enter the training loop; in each training cycle, obtain sample data from the environment; first, compare the random sampling value of the Actor network with the set threshold to determine whether to use heuristics to select an action or use the current Actor network to select an action. If it is less than the threshold, the heuristic is applied, otherwise the Actor network is used to select an action and apply it to the environment; then observe the next state and immediate reward; Update the Critic network; first, calculate the Q value of the next state action on the target network, where the target network parameters are updated by periodically executing a soft update strategy; then, use the difference between these Q values ​​and the Q value of the current state action to calculate the Critic loss function, and minimize the function to update the Critic network parameters; then, update the Actor network; use the current Actor network to select an action and use the Critic network to evaluate the Q value of the action; then, adjust the Actor network parameters through gradient ascent to maximize the Q value of the selected action to optimize the Actor objective function; After completing the update of the Actor and Critic networks, before taking the soft update strategy to gradually bring the target network parameters closer to the current network parameters, continue to obtain sample data and iteratively optimize the Critic and Actor networks; Repeat the above training process until the stopping condition is met.

Images