Unmanned aerial vehicle cluster intelligent coverage method and system based on improved reinforcement learning algorithm

Abstract

The application discloses an unmanned aerial vehicle cluster intelligent covering method and system based on an improved reinforcement learning algorithm, relates to the technical field of unmanned aerial vehicle communication, and comprises the following steps: a multi-objective optimization system model corresponding to an unmanned aerial vehicle cluster is constructed; the model is used for maximizing the unmanned aerial vehicle cluster system covering period (i.e. maximizing the minimum residual energy), taking into account the fairness of ground mobile users, and simultaneously satisfying the real-time coverage rate requirement of the ground mobile users; environment information (i.e. unmanned aerial vehicle position information and energy consumption information, ground mobile user position information) for executing a target task is acquired; according to the multi-objective optimization system model and the environment information, a particle swarm intelligent algorithm is used to initialize a state and assist in determining a deterministic deep reinforcement learning algorithm of a single-step action of a cooperative agent, and the deployment position of the unmanned aerial vehicle cluster is calculated. The application is used for maximizing the unmanned aerial vehicle cluster system covering period, taking into account the fairness of ground mobile users, and simultaneously satisfying the real-time coverage rate requirement of the ground mobile users.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) communication technology, and in particular to a method and system for intelligent coverage of UAV swarms based on an improved reinforcement learning algorithm. Background Technology

[0002] Over the past few years, the adoption of unmanned aerial vehicles (UAVs) has seen tremendous growth. Mobile base stations (UBS) mounted on UAVs can be deployed in target areas where there is insufficient existing infrastructure to support them, thereby improving the coverage and performance of terrestrial mobile user service networks. Due to the key characteristics of UAVs—high mobility, rapid deployment, reliable line-of-sight (LoS) links, and easy access to remote areas—some researchers have explored issues related to improving the quality of on-demand services and increasing the duration of energy-efficient systems.

[0003] With the increasing use of drones, they have made significant strides in rapidly deploying networks, such as providing on-demand wireless coverage in specific geographic areas. The mobility and uneven distribution of terrestrial mobile user equipment (UEs) in hotspot areas place considerable pressure on UEs serving terrestrial networks in these areas. A key challenge is the dynamic, on-demand, tiered deployment of multiple cooperating drones, which can effectively improve coverage and the quality of service for terrestrial mobile users.

[0004] Unmanned aerial vehicles (UAVs) offer a promising technology for providing efficient on-demand services to ground mobile users. Due to their flexibility and mobility, UAV base stations (UAV-BSs) can be deployed based on the distribution of ground mobile users and their Quality of Service (QoS) requirements, such as fairness, coverage, and lifespan. However, extending the lifespan of UAV swarms is a challenge due to the limited battery capacity and coverage range of drones. Simultaneously, ensuring a certain level of coverage while maintaining fairness in service is also crucial. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for intelligent coverage of unmanned aerial vehicle (UAV) swarms based on an improved reinforcement learning algorithm, in order to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] This invention provides a method for intelligent coverage of unmanned aerial vehicle (UAV) swarms based on an improved reinforcement learning algorithm, comprising:

[0008] A multi-objective optimization system model corresponding to the UAV swarm is constructed. The multi-objective optimization system model is used to maximize the coverage period of the UAV swarm system (i.e., maximize the minimum remaining energy), take into account the fairness of ground mobile users, and at the same time meet the real-time coverage requirements of ground mobile users.

[0009] Obtain environmental information for executing the target task; the environmental information includes the location information of the UAV, the energy consumption information of the UAV, and the location information of ground mobile users within the target area;

[0010] Based on the multi-objective optimization system model and environmental information, a deterministic deep reinforcement learning algorithm with particle swarm intelligence algorithm for initializing the state and assisting cooperative agents in single-step actions is used to calculate the deployment location of the UAV swarm.

[0011] This invention also provides an intelligent coverage system for unmanned aerial vehicle (UAV) swarms based on an improved reinforcement learning algorithm, comprising:

[0012] The model building module is used to build a multi-objective optimization system model corresponding to the UAV swarm. The multi-objective optimization system model is used to maximize the coverage period of the UAV swarm system (i.e., maximize the minimum remaining energy), take into account the fairness of ground mobile users, and meet the real-time coverage requirements of ground mobile users.

[0013] The environmental information acquisition module is used to acquire environmental information for performing the target task; the environmental information includes the location information of the UAV, the energy consumption information of the UAV, and the location information of ground mobile users within the target area;

[0014] The deployment location calculation module is used to calculate the deployment location of the UAV swarm based on the multi-objective optimization system model and environmental information, employing a particle swarm intelligence algorithm to initialize the state and a deterministic deep reinforcement learning algorithm to assist the cooperative agents in single-step actions.

[0015] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0016] This invention maximizes the minimum remaining energy of UAVs in a UAV swarm system by using a deterministic deep reinforcement learning algorithm that optimizes the system model and environmental information through multi-objective optimization, initializes the state using particle swarm intelligence, and assists cooperative agents in single-step actions. This approach balances fairness for ground mobile users while meeting their real-time coverage requirements, thereby maximizing the coverage period of the UAV swarm system. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This is a system architecture diagram of an intelligent coverage method for UAV swarms based on an improved reinforcement learning algorithm provided in an embodiment of the present invention.

[0019] Figure 2 A flowchart illustrating the intelligent coverage method for UAV swarms based on an improved reinforcement learning algorithm provided in an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of persistent drone swarm coverage provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of environmental information collection provided in an embodiment of the present invention;

[0022] Figure 5 A simplified reinforcement learning model diagram provided in an embodiment of the present invention;

[0023] Figure 6 This is a detailed structural diagram of the current reinforcement learning method provided in the embodiments of the present invention;

[0024] Figure 7 A schematic diagram of a state and action neural network provided for embodiments of the present invention;

[0025] Figure 8 A schematic diagram illustrating the continuous motion principle of a drone provided in an embodiment of the present invention;

[0026] Figure 9 This is a diagram illustrating the auxiliary deep reinforcement learning structure provided in an embodiment of the present invention.

[0027] Figure 10 This is a schematic diagram of the optimized structure of the auxiliary learning system provided in an embodiment of the present invention;

[0028] Figure 11 The cumulative return curve of the reinforcement learning algorithm provided in this embodiment of the invention is compared with the cumulative return curve of other algorithms. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] To meet the multi-objective requirements of on-demand services, this invention must coordinate various indicators. Therefore, Pareto dominance theory is introduced to optimize conditionally constrained multi-objective optimization problems. A multi-objective cooperative optimization system model (MCOP) is designed to maximize the coverage period of the UAV swarm system (i.e., maximize the minimum residual energy), while ensuring fairness for ground mobile users and meeting the real-time coverage requirements for ground mobile users. To address this issue, a novel assisted deep reinforcement learning (ADRL) model is developed, employing the particle swarm optimization (PSO) algorithm from swarm intelligence algorithms to assist cooperative multi-agent deep reinforcement learning (DRL). Multi-agent cooperative DRL is the most challenging because traditional DRL is not suitable for multi-agent cooperation in high-dimensional spaces. ADRL consists of two phases: 1) PSO-based initialization, and 2) PSO-assisted learning step-by-step following a certain probability with decay.

[0032] The system architecture of the UAV swarm intelligent coverage method based on an improved reinforcement learning algorithm provided in this embodiment mainly consists of three steps, as follows: Figure 1 As shown.

[0033] Step 1: Model Building (Target Task). Design the UAV flight mission model (multiple intelligent UAVs cooperating to cover the target area). Design the mission model, including the basic model, coverage model, and energy model.

[0034] Step 2: Information Acquisition (Method 1). Obtain environmental information for the target task. The sensing system (wireless sensors) collects environmental information (ground mobile user information and UAV deployment information in the target area), uploads it to the cloud server, and performs information parsing (converting the data format to suit the auxiliary learning model).

[0035] Step 3: Intelligent Deployment (Method 2). Based on environmental information and executing the target task. The intelligent deployment module reads data from the information parsing module, and then assists the learning model (centralized training - distributed action) in learning and decision-making simultaneously, with the aim of executing the constructed target task. The intelligent drone swarm distributes its actions and updates the environmental deployment. Return to Step 2 until the task ends. To avoid the cost of actual drone exploration and training of neural network models, the initial intelligent deployment involves a specified number of simulation training sessions before transitioning to actual intelligent deployment.

[0036] The intelligent coverage method for UAV swarms based on an improved reinforcement learning algorithm provided in this embodiment is as follows: Figure 2 As shown, the steps include the following.

[0037] Step 100: Construct a multi-objective optimization system model corresponding to the UAV swarm; the multi-objective optimization system model is used to maximize the coverage period of the UAV swarm system, take into account the fairness of ground mobile users, and meet the real-time coverage requirements of ground mobile users.

[0038] Currently, due to the limited onboard energy of drones and the uneven energy consumption of drones during mission deployment, the remaining energy of drones after deployment is uneven. The duration of drone swarm deployment is determined by the drone with the minimum remaining energy after deployment to the target area. Therefore, extending the coverage duration of the drone swarm (system coverage cycle) requires maximizing the minimum remaining energy of drones after deployment. Figure 3 A schematic diagram illustrating persistent coverage by a drone swarm is shown.

[0039] The problem of constructing the multi-objective optimization system model MCOP is: to provide on-demand services to ground mobile users in a two-dimensional area, maximize the system coverage period for a limited number of drones, meet a certain ground mobile user coverage rate, take into account the fairness of ground mobile users, and maximize the minimum remaining energy of drones.

[0040] Step 1.1: Basic Model. Define model notation: [x, y] represents the range of the target region, where x1≤x≤x2, y1≤y≤y2. m Represents ground mobile users within the target area, where 1 ≤ m ≤ M. Represents ground mobile user b m The coordinates at time t. The duration of the coverage task is T, T m k represents the total time that a ground mobile user m is covered during a coverage mission. m =1 indicates that the ground mobile user b m Covered, k m =0 indicates that the mobile user b is on the ground. m Not covered. Represents the set of locations of a drone swarm. Let represent the position of the i-th drone at time t, where 1 ≤ i ≤ N. h = [h1, h2] represents the drone's deployment altitude range, and the drone's coverage radius R = h(θ) i =45°). C con =2h² represents the connectivity radius of the UAV. f con Denotes the connectivity index, if f con =1 indicates disconnecting the network connection; otherwise, f con =0 indicates that the connection has not been broken. out Denotes the out-of-bounds index, if f out =1 indicates that the drone is outside the target area; otherwise, f out =0 indicates that the boundary has not been exceeded. f covThis represents the coverage index, if f cov =1 indicates that the coverage does not meet the requirements; otherwise, f cov =0 indicates that the requirement is met.

[0041] Step 1.2: Coverage Model. Coverage score c for terrestrial mobile users. m The specific coverage model is determined by the coverage time, as shown in formulas (1)-(2).

[0042]

[0043]

[0044]

[0045] Among them, c m Represents ground mobile user b m Coverage score, f c Jain Fairness Index, representing coverage for all terrestrial mobile users, f p This indicates the real-time coverage rate for ground mobile users.

[0046] Step 1.3: Energy Model. The constructed energy model is not limited to any specific energy consumption model; this embodiment uses a general linear model. As shown in formula (4).

[0047] E up E hori E hove E down :=Γ,E hove :=w·e unit (4).

[0048] Among them, E up E represents the uplink energy consumption per unit distance. down E represents the downlink energy consumption per unit distance. hori E represents the horizontal energy consumption per unit distance. hove e represents the energy consumption per unit time of hovering. unit Γ represents unit energy consumption, and w represents energy consumption weight.

[0049] Defined residual energy index f e As shown in formula (5). p i This indicates the location of drone i. This represents the energy consumed by drone i from its takeoff from the starting base station to its current location. E represents the energy consumed during drone initialization.

[0050]

[0051]

[0052]

[0053] in, σ[e] represents the expected energy consumed by drone i during deployment. i ] represents the standard deviation of the remaining energy of drone i during deployment, Var[e i ] represents the variance of the remaining energy of drone i during deployment.

[0054] Step 1.4: Task (problem) model; Maximum coverage period deployment objective function F as shown in formula (8).

[0055]

[0056]

[0057] f p >=Th p (10);

[0058] Among them, Th p P represents the specified coverage rate; P represents the deployment location of the drone swarm when the objective function F is maximized.

[0059] Step 200: Obtain environmental information for executing the target task; the environmental information includes the location information of the UAV within the target area, the energy consumption information of the UAV, and the location information of ground mobile users.

[0060] In special circumstances, such as natural disasters like earthquakes or temporary communication network deployments, where ground-based equipment cannot provide convenient communication services to mobile users, drones can be used to achieve communication coverage. This involves using IoT sensors (miniature sensor nodes, which cannot provide communication services to mobile users within the target area but only acquire location information for individual mobile users) distributed throughout the target area to obtain the location information of mobile users within that area. IoT sensors offer advantages such as low cost, large quantity, small size, and ease of deployment. Environmental information from the drone (i.e., location and energy consumption information) is transmitted to a cloud server via a self-organizing topology network using onboard sensors. Information collection methods include... Figure 4 As shown, the specific steps are as follows:

[0061] Step 2.1: Obtain environmental information of the target area by using existing IoT sensors or temporarily deployed micro sensor nodes in the target area.

[0062] Step 2.2: Collect location information of mobile users on the ground using miniature sensor nodes scattered throughout the target area.

[0063] Step 2.3: Optimize the network information transmission lines of the sensor system, select an energy-saving communication path (multi-hop communication path), and upload the information collected by the miniature sensor nodes to the cloud server (signal base station). Figure 4 In the diagram, the dashed lines between the miniature sensor nodes represent the communication paths.

[0064] Step 2.4: The drone swarm constructs an independent communication topology network, using multi-hop communication to transmit the location and energy consumption information of each drone to the cloud server (signal base station). Figure 4 In the diagram, the dashed lines between drones represent communication paths.

[0065] Step 300: Based on the multi-objective optimization system model and environmental information, a deterministic deep reinforcement learning algorithm is used to initialize the state and assist the cooperative agents in single-step actions, and the deployment location of the UAV swarm is calculated.

[0066] Before performing step 300, the following steps also need to be performed.

[0067] Step (1): The information parsing module reads data from the cloud server, including the location information of ground mobile users, the location information of UAVs and the energy consumption information of UAVs, and converts it into the data format required by the auxiliary learning model. The converted data includes the data information corresponding to the models (basic model, coverage model, energy consumption model and task model) constructed in step 100.

[0068] Step (2): The information parsing module will generate the ground mobile user fairness f c Average score of ground mobile users The remaining energy index f of drones e Connectivity index f con Cross-boundary index f out and real-time coverage rate of ground mobile users f p It is passed to the auxiliary learning model.

[0069] Step 300 specifically includes:

[0070] Step (1): Use particle swarm optimization algorithm to determine the initial observation status information of each UAV; the observation status information includes the position information of UAV i, the position information of other UAVs, the information of ground mobile users, the remaining energy information of UAV i, the fairness index of ground mobile users, the real-time coverage rate of ground mobile users and the remaining energy index of UAVs.

[0071] Step (2): Based on the multi-objective optimization system model and the observation status information of each UAV, the state is initialized by the particle swarm intelligence algorithm and the deterministic deep reinforcement learning algorithm is used to assist the cooperative intelligent agent in single-step action to calculate the deployment position of the UAV swarm.

[0072] The basic model process of reinforcement learning is as follows: Figure 5 As shown, the drone (agent) acquires its state and gains from the environment, calculates its next action through decision-making, and then acts on the environment, forming a cyclical system. Since drone flight operations are a continuous flight control problem, this embodiment employs a deterministic deep reinforcement learning algorithm (DDPG algorithm for short).

[0073] A centralized training and distributed execution strategy is adopted. The environmental states observed by multiple agents (multiple UAVs) are input into the central agent's training network for unified learning. The learned decision network parameters are then imported into the decision network of each UAV (agent), and each UAV executes independently according to the training strategy. Assisted learning (i.e., assisted deep reinforcement learning) uses a swarm intelligence algorithm (Particle Swarm Optimization, PSO) to assist the central agent's training, improving the quality of learned experience and accelerating system convergence.

[0074] The auxiliary learning module primarily addresses the problem of current Deep Reinforcement Learning (DRL) failing to achieve system convergence in high-dimensional state spaces. Current DRL, due to its stochastic learning mechanism, struggles to plan efficient action policies in high-dimensional spaces. Auxiliary learning directly generates high-quality data experience (single-step optimal decision-making) rather than relying on random actions. In other words, given the large number of state combinations in high-dimensional spaces, random actions make it difficult for neural networks to summarize patterns and accurately adjust the parameters of the action policy network to enable the agent to make efficient actions that lead to system convergence.

[0075] Step 3.1: The intelligent deployment module reads and parses information, including the coverage fairness index fc and the average score of terrestrial mobile users. Residual energy index fe, connectivity index f con Crossing the boundary index f out Real-time coverage rate of ground mobile users f p The purpose is to implement the model constructed in step 100.

[0076] Step 3.2: Traditional reinforcement learning (this embodiment uses the DDPG algorithm).

[0077] Reinforcement learning (RL) is a learning method that can handle multi-state decision-making processes when an agent interacts with its system environment over a series of discrete time steps. At each time step t, the agent observes state s. t Execute action a t And receive a reward r t .

[0078] The goal of reinforcement learning is to learn a mapping π(s) from state to action policy. t a tThe goal is to maximize the cumulative gains from future discounts. Where T is the number of time steps, the discount factor γ∈[0,1], and r(.) is the revenue function at time point t′.

[0079] Classical reinforcement learning algorithms are based on Q-learning, which allows an agent (drone) to optimally interact with the environment represented by a Markov Decision Process (MDP). However, Q-learning is only applicable to a finite discrete action state space. DRL, also known as a Deep Q-Network (DQN), uses a DNN (or multiple DNNs) as an approximation of the estimation function Q(.). State-Action Value (Q-value) Q(s) t a t ) is used to represent based on state s t and action a t Long-term cumulative rewards,

[0080] Traditional neural networks are trained by minimizing the difference between the estimated value and the label value. DRL has no prior information (label value), which is replaced by the target value. DQN is trained by minimizing the following loss function, as shown in Equation (11).

[0081]

[0082] Among them, y t θ represents the target value. Q This represents the weight vector of DQN.

[0083] The foundation and core of reinforcement learning is the Bellman equation, which contains three main concepts: policy function, state-value function, and state-action-value function. The time difference (TD) method is used to address the update problem of the state-action-value function. The TD method combines Monte Carlo sampling with guidance from dynamic programming, using the value function of subsequent states to estimate the current value function. Q-learning using the TD method is a widely used model-free RL algorithm. The Q function of DQN is still updated using the TD method. The calculation of the Q value is shown in Equation (12).

[0084] Q(s, a) ← Q(s, a) + α(R) t+1 +γ·Q(s′,a′)-Q(s,a)) (12).

[0085] Among them, R t+1 +γ·Q(s′,a′) represents the TD target, R t+1+γ·Q(s′,a′)-Q(s,a) is called the TD bias, α is the learning rate, γ is the reward decay coefficient, and the leftmost Q(s,a) represents the target Q value. The target Q value can also be estimated, as shown in formula (13).

[0086] y t =r(s t a t )+γ·Q(S t+1 ,π(S t+1 |θ π )|θ Q (13).

[0087] The basic idea of ​​DDPG is to maintain a parameterized Actor function π(s) t a t This function specifies the current policy by deterministically mapping states to specific actions, while maintaining a Critic function Q(s, a|θ). Q This simulates the correlation between Q-values ​​and state-action pairs. Updates are driven by action policies, which can be applied to the Actor network parameters θ. π The cumulative reward J is used to update the Actor network according to the chain rule, as shown in Equation (14).

[0088]

[0089] The target network can update its parameters using formula (15).

[0090] θ = αθ + (1-α)θ (15).

[0091] Here, α represents the update rate, which determines the degree of influence of newly generated parameters in the evaluation network on the parameters in the target network. The purpose of using the target network is to prevent overestimation.

[0092] The current detailed structure and data flow of reinforcement learning are as follows: Figure 6 As shown. The deployment system has N drones, with drone observation state O and system action A. First, the system initialization module obtains the initial environmental observation state O1 and calculates the initial action A1, and then applies the experience (O1) to the initial observation state O1. t+1 O t R t Stored in the playback cache, where, Central training network requirements The central network mainly consists of Actor and Critic modules, each including a target network (target_net) and an estimation network (eval_net). The central network training has two objectives: first, to minimize the difference (TD_error) between the actual Q-value (target_q) and the estimated value (eval_q); second, to calculate the policy (Policy_grads) corresponding to minimizing the action loss. The central network is then trained to perform actions... It acts on the environment and will affect the corresponding (O) action. t+1 O t R t Save to the replay buffer. See Algorithm 1 for the detailed execution flow. The neural network of state and action is as follows: Figure 7 As shown, the Actor neural network and the Critic neural network each have 5 intermediate layers, with the number of neurons in each layer being 400, 200, 100, 64, and 32, respectively. The activation function for each layer is ReLU, and the optimizer is Adam. The output layer of the Actor neural network has 3 neurons, and the output layer of the Critic neural network has 1 neuron.

[0093] Step 3.3: Construct the basic elements of reinforcement learning: state, action, and reward, as shown below:

[0094] The detailed descriptions of the basic elements (state, action, and reward) of the ADRL algorithm are as follows.

[0095] 1) State: For each UAV i, its observations And the system observation space: The observation space of UAV i at time t consists of five parts, namely the position of the UAV ( and ), Status of ground mobile users Remaining energy Coverage Index (Fairness for Ground Mobile Users) f c and the remaining energy index f of drones e .

[0096] 2) Actions: such as Figure 8 As shown, for each UAV i in time slot t, its Φ i ∈[0,π] and These are the polar angle and azimuth angle of the UAV's flight direction, i, respectively. d i ∈[0, d max ] is the flight distance, d mmax That is the maximum permissible flight distance. Therefore For a task It is the space for action.

[0097] 3) Rewards: As described in formula (8) above, the definition of rewards is shown in formula (16).

[0098]

[0099] Where t∈[0,T], p con This represents the penalty for losing the connection. cov p represents the penalty for falling below the required coverage rate. out This represents penalties for drones crossing boundaries.

[0100] Step 3.4: Assisted Learning (ADRL, which builds upon Step 3.2)

[0101] To address the challenge of large state space dimensionality and difficulty in convergence in current deep reinforcement learning models, this paper proposes an improved deep reinforcement learning model by introducing swarm intelligence-based algorithms to assist in model training. Figure 9 As shown. The improvements are mainly twofold: First, the current environment state S is initialized using a swarm intelligence optimization algorithm (Particle Swarm Optimization, PSO), replacing the traditional random initialization; second, an auxiliary action strategy based on the PSO algorithm is introduced, which determines the specific action by utilizing a selector that satisfies certain probability rules. There are two types of actions: one is the action strategy calculated by the agent's decision layer, and the other is the action strategy calculated by the auxiliary learning module. For the specific operation flow, please refer to the proposed Algorithm 1.

[0102] Algorithm 1: ADRL (Auxiliary Deep Reinforcement Learning).

[0103] 1: Randomly initialize the Critic network Q(o, a|θ) Q ) and Actor network π(o|θ) π The weight θ Q and θ π ;

[0104] 2: Initialize the weights θ of the target network Q′ and π′ Q′ =θ Q θ π′ =θ π ;

[0105] 3: Initialize the replay cache Gaussian noise Greed factor ε and decay exponent ∈;

[0106] 4: For loop number episode = 1 to loop end Ep: This is the loop; / / Ep is the number of learning rounds;

[0107] 5: Randomly initialize O1 using Algorithm 2;

[0108] 6: For time step epoch = 1 to end of time step T: loop here; / / T is the task duration in step 1.1;

[0109] 7: If random number but: / / It is a random number, and ∈ will gradually decrease;

[0110] 8: Action / / ε probability of taking random action;

[0111] 9: Else:

[0112] 10: / / action Obtained by Algorithm 3;

[0113] 11: Endif;

[0114] 11: Execution of Action a t Obtain the observed state o t+1 and revenue r t ;

[0115] 12: Ifr con == 1 or f out == 1 or f cov ==1 item:

[0116] 13: Profit r t -(f con ·p con +fo ut ·p out +f cov -p cov / / Benefits minus penalties

[0117] 14: The intelligent agent drone will not carry out this operation;

[0118] 15: Endif;

[0119] 16: Storing transitional samples (o t a t r t o t+1 To replay cache middle;

[0120] 17: From cache The minimum batch size of H samples is randomly sampled (o j a j r j oj+1 );

[0121] 18: Calculation The value;

[0122] 19: Utilize minimizing the loss value To update the weights θ of the Critic network Q(·) Q ;

[0123] 20: Utilizing policy gradients Update the weights θ of the Actor network π(·) π ;

[0124] 21: Update the parameters of the target networks for Actor and Critic: θ Q′ =α·θ Q +(1-α)·θ Q′ θ π′ =α·θ Q +(1-α)·θ π′ ;

[0125] 22: Endfor;

[0126] 24: Update the decay exponent ∈ = ∈·Θ; / / Θ represents the defined update step size, for example, Θ = 0.98;

[0127] 25: Endfor.

[0128] Step 3.4.1: PSO Initialization Algorithm: The problem mentioned above is a large-scale optimization problem, which means that the algorithm may require a large number of iterations when the performance of the initial solution is not high. Therefore, improving the performance of the initial solution is of paramount importance. According to evolutionary theory, Particle Swarm Optimization (PSO) is considered a superior computational model compared to other evolutionary algorithms. Furthermore, the Pareto optimality principle also applies to multi-objective optimization problems (MOP). Therefore, the Pareto-constrained multi-objective PSO (MOPSO) algorithm is applied to the UAV initialization deployment problem, i.e., a Pareto-constrained multi-objective particle swarm optimization algorithm is used to determine the initial observation state information for each UAV.

[0129] Definition 1 (Pareto Dominance): A vector superior (represented as) If and only if u is greater than v, i.e., u i >v i .

[0130] Definition 2 (Pareto optimal set): For a given multi-objective optimization problem Pareto optimal set It is defined as formula (17).

[0131]

[0132] Definition 3 (Pareto Front): For a given multi-objective optimization problem Pareto optimal set Pareto Frontier It is defined as formula (18).

[0133]

[0134] Definition 4 (Pareto constraint dominance): For a given MOP vector And f i (x) Pareto dominance is only considered when the given constraints are not met.

[0135] The normal procedure for generating a Pareto front is to calculate feasible points Ω and their corresponding f(Ω). When a sufficient number are available, non-dominated points can be identified, and a Pareto front can be generated. Coverage probability Th p , connectivityTh con The target region is a Pareto constraint. Once all conditions are met, the purpose of MOP is to maximize the value of F.

[0136] Algorithm 2: MOPSO initializes drone coverage deployment.

[0137] Input: N, M, Th p CN; / / CN represents the set number of loops, N represents the number of drones, and M represents the number of mobile ground users;

[0138] Output: GBEST; / / Globally optimal initial drone coverage deployment, addressing the issues addressed in step 1.4.

[0139] 1: For i == 1 to PN: This loops; / / PN represents the number of particles;

[0140] 2: Initialize particle velocity VEL[i]; / / Random initialization, where VEL[i]∈[-d mmax d inax ];

[0141] 3: Initialize particle POP[i]; / / POP represents a particle swarm, where each particle consists of the positions of N drones;

[0142] 4: Evaluate the fitness of each particle in the Particle Swarm Optimization (POP); / / Fitness is equivalent to the reward in step 3.3;

[0143] 5: Store non-dominated particles in the repository REP; / / Store initially superior particles in the external repository REP;

[0144] 6: Endfor;

[0145] 7: For i == 1 to PN: This is a loop;

[0146] 8: The initial optimal solution for each particle, PBESTS[i] := POP[i];

[0147] 9: Endfor;

[0148] 10: While (True):

[0149] 11: The optimal particle GBEST in REP is selected using the roulette wheel algorithm;

[0150] 12: Update speed VEL[i]=η·VEL[i]+ζ·(PBESTS[i]-POP[i])+ξ·(GBEST-POP[i]);

[0151] 13: Update the particle swarm POP[i] = POP[i] + VEL[i];

[0152] 14: Maintain the original state of particles in the search space, such as if they go out of bounds or are disconnected;

[0153] 15: Evaluate the fitness of each particle in the Particle Swarm Optimization (POP);

[0154] 16: Update the particles in the repository REP;

[0155] 17: If particle i has a fitness better than the historical best solution for that particle:

[0156] 18: Update PBESTS[i] = POP[i];

[0157] 19: Endif;

[0158] 20: If circle>CN: / / Initialize circle==1;

[0159] 21: Exit the while loop;

[0160] 22: Endif;

[0161] 23: The number of cycles is circle = circle + 1;

[0162] 24: Endwhile;

[0163] 25: If coverage f p <Th p :

[0164] 26: Number of drones N = N + 1; / / The current number of drones cannot meet the set coverage requirement, initialize N == 1;

[0165] 27: Return to step 1 of Algorithm 2;

[0166] 28: Endif;

[0167] Step 3.4.2: PSO-assisted learning.

[0168] DRL (Directed Learning) explores the optimal solution in an environment without prior knowledge, continuously summarizing experience and optimizing actions. This may lead to getting stuck in local solution spaces, resulting in a degree of blindness in the search direction. Especially for multi-objective, multi-agent cooperative tasks, current DRL cooperative algorithms neglect how the combination of multiple agents achieves optimal system performance, essentially making action decisions based on stochastic state estimations.

[0169] Algorithm 3: MOPSO-assisted decision-making action, which uses a Pareto-constrained multi-objective particle swarm optimization algorithm to assist in decision-making on the optimal cooperative action based on the multi-objective optimization system model and the observation status information of each UAV, and calculates the deployment position of the UAV swarm.

[0170] The action decision details are mainly based on Algorithm 2, with only 4 modifications:

[0171] Step 1: Set the random initialization speed range from step 2 to...

[0172] Step 2: Replace the initial particle swarm in step 3 with PN particles randomly generated within the velocity range, based on the current deployment location of the drone swarm, with increments (or decrements) including the current deployment location;

[0173] Step 3: Set the loop count CN of step 20 to 10;

[0174] Step 4: Delete steps 25-28.

[0175] Algorithm 3 uses MOPSO to make the agent's next action decision (single-step action search). Its main purpose is to obtain the action with the highest fitness, which helps to generate high-quality experience data and thus improve the learning efficiency of the DRL model.

[0176] Example 2

[0177] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, an intelligent coverage system for UAV swarms based on an improved reinforcement learning algorithm is provided below.

[0178] This embodiment provides an intelligent coverage system for drone swarms based on an improved reinforcement learning algorithm, including:

[0179] The model building module is used to build a multi-objective optimization system model corresponding to the UAV swarm. The multi-objective optimization system model is used to maximize the coverage period of the UAV swarm system (i.e., maximize the minimum remaining energy), take into account the fairness of ground mobile users, and meet the real-time coverage requirements of ground mobile users.

[0180] The environmental information acquisition module is used to acquire environmental information for performing the target task; the environmental information includes the location information of the UAV, the energy consumption information of the UAV, and the location information of ground mobile users within the target area.

[0181] The deployment location calculation module is used to calculate the deployment location of the UAV swarm based on the multi-objective optimization system model and environmental information, employing a particle swarm intelligence algorithm to initialize the state and a deterministic deep reinforcement learning algorithm to assist the cooperative agents in single-step actions.

[0182] In this invention, the following operations were performed:

[0183] 1. Design the system model MCOP, including the basic model, coverage model, energy consumption model, and task model (objective function F).

[0184] 2. The remote cloud service system collects environmental information from target devices (ground mobile users and aerial drones).

[0185] 3. Analyze and obtain the coverage information and coverage score of ground mobile users, as well as the location information and energy consumption of drones.

[0186] 4. Design the basic elements of reinforcement learning, including state model, action model, and reward model.

[0187] 5. The swarm intelligence algorithm (Particle Swarm Optimization, PSO) initializes the coverage deployment status of the UAV system;

[0188] 6. PSO intelligent assisted learning improves the quality of training experience.

[0189] 7. Construct a Pareto-constrained dominant model to solve a multi-objective optimization problem (with conflicting objectives).

[0190] 8. Simulation experiments demonstrate that Assisted Deep Reinforcement Learning (ADRL) outperforms existing advanced Deep Reinforcement Learning (DRL) algorithms.

[0191] Compared to traditional structures, the assisted learning system architecture adds intelligent initialization and intelligent assisted learning modules to the traditional system model structure. Furthermore, for multi-objective optimization problems, it also adds a Pareto-constrained dominance module, such as... Figure 10 As shown.

[0192] The drone and ground mobile users possess environmental information within the target area. This information is collected by sensors and uploaded to a cloud service system. An information parser then retrieves coverage information, which is imported into a reinforcement learning system. Based on a multi-objective coverage optimization objective function F, a Pareto-constrained dominance strategy is used to initialize the system state. The initialization aims to accelerate the deployment of the system to its optimal state. The reinforcement learning system employs a centralized training and distributed execution strategy, while also incorporating an intelligent auxiliary learning module with Pareto constraint dominance. This module aims to improve the quality of learning experience and accelerate neural network training convergence, primarily by using a swarm intelligence algorithm (Particle Swarm Optimization, PSO) to assist the agent in making decisions within the system, thereby generating high-quality experience. Finally, the agent (drone) determines its next action and updates its deployment within the environment through an action network.

[0193] Simulation experiment diagram as follows Figure 11 As shown, the target area is 200×200 units in size, the drone deployment altitude range is [20,40] units, the number of users M = 100, and the user random movement and movement step size are 0.4 units. Coverage Th p =50%, energy consumption weight w=20, energy consumption weight Γ=1.4:1.2:1:0.8. Training Ep=600, each iteration T=600 steps. Reinforcement learning reward decay value γ=0.9, reinforcement learning update rate α=0.001, decay exponent ∈=1. Noise Greed factor ε = 0.4, cache space quantity Batch sampling size H = 200. Penalty p con =p cov =p out =10. The initial iteration count for the PSO (Particle Swarm Optimization) algorithm is CN = 100, and the number of particles is PN = 100. The weight coefficients for the PSO algorithm are η = 0.6, ζ = 0.4, and ξ = 0.4. The particle swarm velocity step size is d. mmax =1 unit, particle swarm external warehouse capacity REP=200. Specific parameters should be optimized and adjusted according to the implementation example.

[0194] In the simulation experiment, the number of test samples was 600. Experimental results demonstrate that the improved ADRL algorithm's cumulative gain is significantly higher than other algorithms (DDPG, DDQN, PER_DQN), and the growth of cumulative gain gradually slows down and tends to plateau as the number of test steps increases. The cumulative gain of the four algorithms in the initial step sequence is less than 1; therefore, the logarithmic value is less than 0. In conclusion, the improved ADRL algorithm outperforms current popular deep reinforcement learning models.

[0195] Comparison list:

[0196] DDPG: A currently popular deep reinforcement learning model algorithm based on deterministic deep policy gradients; ADRL: A proposed improved deep reinforcement learning model algorithm based on DDPG; DDQN: A currently popular deep reinforcement learning model algorithm based on two deep Q networks; PER_DQN: A currently popular deep reinforcement learning model algorithm based on priority experience replay deep Q networks.

[0197] Compared to current research, the research question, model design, and theoretical methods of this invention are all proposed for the first time.

[0198] To address the issues with current two-dimensional spatial coverage periods, a maximum coverage period deployment model is proposed. The objective remains to maximize the remaining energy of the UAV in the minimum coverage system. Deep reinforcement learning, as an advanced method for multi-objective, multi-agent cooperative optimization deployment, has limitations in addressing the problems proposed in this invention. Therefore, an improved auxiliary learning model is proposed to better solve the proposed problems.

[0199] This invention constructs a data acquisition system, from edge-layer environmental perception to key data collection and then to data uploading to a cloud server. Traditional data acquisition modules use drones as edge-layer perception devices, employing onboard sensors to capture ground mobile user information through a carpet search of the target area. Because ground mobile user signals may be intermittent (or probabilistic), drone detection of ground mobile users has vulnerabilities, and carpet searching of ground mobile users wastes onboard resources.

[0200] Traditional data acquisition in distributed systems suffers from issues of data consistency, integrity, and real-time performance, leading to significant errors in subsequent neural network learning results. This hinders efficient decision-making by intelligent drones and consequently impacts the deployment efficiency of coverage systems. The designed data acquisition system (sensor system), however, utilizes sensor nodes distributed across the target area to collect ground mobile user information in real time. It then plans energy-efficient transmission paths for these sensor nodes to upload this information to a cloud server. This not only improves data collection efficiency but also aids in the action decisions (drone flight maneuvers) of the policy neural network, ultimately enhancing the deployment performance of the coverage system.

[0201] The main reason for the poor learning efficiency of current deep reinforcement learning networks in high-dimensional state spaces is the difficulty in learning superior training experiences (experiences that can accelerate system convergence) from random experiences generated by random actions, thus affecting the training quality of neural network parameters. Existing reinforcement learning methods, such as Prioritized Experience Replay Deep Q Networks (PER_DQN), attempt to select high-quality experiences from random experiences for training; however, random experiences are unlikely to generate high-quality experiences. Assisted learning (i.e., assisted deep reinforcement learning) employs swarm intelligence algorithms (Particle Swarm Optimization, PSO) to generate high-quality experiences, thereby improving network training efficiency and accelerating system convergence.

[0202] From a theoretical and methodological perspective, assisted deep reinforcement learning outperforms current deep reinforcement learning. However, reinforcement learning faces challenges in constructing optimization objective functions for multi-objective, multi-agent cooperation. The optimal value of the objective function is difficult to define because the benefits of multiple objectives inevitably conflict, with interests at odds. To address this conflict, the Pareto dominance criterion is introduced. Since practical coverage problems have certain technical indicators (e.g., coverage requirements, connectivity, etc.), which only need to be met, when these "constraints" are satisfied, they are no longer considered as factors for Pareto dominance; this is termed "Pareto constraint dominance." The deployment process of UAVs involves continuous control; however, most current literature uses discrete control schemes, which do not reflect the actual flight characteristics of UAVs. A continuous action space for the UAV is designed, supporting actions at arbitrary angles and distances. For the proposed maximum periodic coverage deployment objective, the environmental state space of the UAV and the corresponding benefit model for the objective function are designed.

[0203] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0204] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligent coverage of unmanned aerial vehicle (UAV) swarms based on an improved reinforcement learning algorithm, characterized in that, include: A multi-objective optimization system model corresponding to the UAV swarm is constructed. The multi-objective optimization system model is used to maximize the coverage period of the UAV swarm system, take into account the fairness of ground mobile users, and meet the real-time coverage requirements of ground mobile users. The multi-objective optimization system model for constructing the drone swarm specifically includes: Construct a coverage model; the coverage model is: ； ； ； in, Indicates ground mobile users Coverage score, Indicates ground mobile users within the target area. , This represents the total time that a ground mobile user m is covered during a coverage mission. T Indicates the duration of the coverage task. The Jain Fairness Index represents the coverage of all terrestrial mobile users within the target area. Indicates the coverage rate of terrestrial mobile users within the target area. Indicates ground mobile users Covered Indicates ground mobile users Not covered; Construct an energy model; the energy model is as follows: ； s.t. ; ； in, Indicates the remaining energy index. Indicates drone i The expected energy consumed during deployment. Indicates drone i The energy consumed from takeoff from the starting base station to the current location. Indicates drone i Location, This indicates the energy level at which the drone initializes. Indicates drone i The standard deviation of remaining energy during deployment. Indicates drone i The variance of remaining energy during deployment; Construct a multi-objective optimization system model; the multi-objective optimization system model is as follows: P = arg ; s.t. ; ； in, Indicates the specified coverage rate; P Describe the objective function The deployment location of the drone swarm corresponding to the maximum value; Obtain environmental information for executing the target task; the environmental information includes the location information of the UAV, the energy consumption information of the UAV, and the location information of ground mobile users within the target area; Convert the multi-objective optimization system model and environmental information into the data format required by the auxiliary deep reinforcement learning model; The fairness index for ground mobile users Average score of ground mobile users Remaining energy index of drones Connectivity index Boundary crossing index Real-time coverage of ground mobile users The algorithm is passed to an auxiliary deep reinforcement learning model; the auxiliary deep reinforcement learning model is a model obtained by fusing particle swarm optimization algorithm with deterministic deep reinforcement learning algorithm. Based on a multi-objective optimization system model and environmental information, a deterministic deep reinforcement learning algorithm is used to initialize the state and assist the cooperative agents in single-step actions, specifically calculating the deployment location of the UAV swarm. The particle swarm optimization algorithm is used to determine the initial observation state information for each UAV; the observation state information includes the UAV... i Location information, location information of other drones, information of ground mobile users, drones i Remaining energy information, ground mobile user fairness index, ground mobile user coverage rate, and drone remaining energy index; Based on the multi-objective optimization system model and the observation status information of each UAV, a deterministic deep reinforcement learning algorithm is used to initialize the state and assist the cooperative agent in single-step actions to calculate the deployment position of the UAV swarm.

2. The intelligent coverage method for UAV swarms based on an improved reinforcement learning algorithm according to claim 1, characterized in that, Obtain the environment information for executing the target task, specifically including: The location information of mobile users on the ground is collected by using miniature sensor nodes scattered throughout the target area; The drone's location and energy consumption information are collected using onboard sensors.

3. The intelligent coverage method for UAV swarms based on an improved reinforcement learning algorithm according to claim 1, characterized in that, The particle swarm optimization algorithm is used to determine the initial observation state information for each UAV, specifically including: Based on a Pareto-constrained multi-objective particle swarm optimization algorithm, the initial observation state information of each UAV is determined.

4. The intelligent coverage method for UAV swarms based on an improved reinforcement learning algorithm according to claim 1, characterized in that, Based on the multi-objective optimization system model and the observation state information of each UAV, a deterministic deep reinforcement learning algorithm is used to initialize the state and assist the cooperative agents in single-step actions, specifically including: Based on the multi-objective optimization system model and the initial observation state information of each UAV, a deterministic deep reinforcement learning algorithm based on Pareto constraint-dominated multi-objective particle swarm intelligence algorithm is used to initialize the state and assist the cooperative agent in single-step action to calculate the deployment position of the UAV swarm.

5. A drone swarm intelligent coverage system based on an improved reinforcement learning algorithm, characterized in that, The aforementioned intelligent coverage system for UAV swarms based on an improved reinforcement learning algorithm applies the intelligent coverage method for UAV swarms based on an improved reinforcement learning algorithm as described in any one of claims 1 to 4, wherein the system comprises: The model building module is used to build a multi-objective optimization system model corresponding to the UAV swarm; the multi-objective optimization system model is used to maximize the coverage period of the UAV swarm system, take into account the fairness of ground mobile users, and meet the real-time coverage requirements of ground mobile users. The environmental information acquisition module is used to acquire environmental information for performing the target task; the environmental information includes the location information of the UAV, the energy consumption information of the UAV, and the location information of ground mobile users within the target area; The deployment location calculation module is used to calculate the deployment location of the UAV swarm based on the multi-objective optimization system model and environmental information, employing a particle swarm intelligence algorithm to initialize the state and a deterministic deep reinforcement learning algorithm to assist the cooperative agents in single-step actions.

Images