Multi-uav content caching and trajectory planning method based on federated deep reinforcement learning

Abstract

The application provides a multi-unmanned aerial vehicle content caching and trajectory planning method based on federal deep reinforcement learning, comprising the following steps: constructing an air-ground wireless communication caching network, and obtaining cumulative average throughput of the air-ground wireless communication caching network; constructing an optimization objective function of an unmanned aerial vehicle trajectory based on the cumulative average throughput; constructing a caching replacement model in the unmanned aerial vehicle as a first MDP model, and constructing a trajectory design of the unmanned aerial vehicle as a second MDP model; based on the optimization objective function, using an algorithm of federal deep reinforcement learning to calculate global parameters of the first MDP model and the second MDP model, and performing multi-unmanned aerial vehicle trajectory planning and caching replacement based on the global parameters. The application can perform more sufficient exploration under the premise of protecting privacy, so as to solve the problems of caching hit and fair communication of users.

Description

Technical Field

[0001] The present invention belongs to the field of adaptive decision-making technology for drones in caching service scenarios, and in particular to a multi-drone content caching and trajectory planning method based on federated deep reinforcement learning. Background Art

[0002] In recent years, the rapid development of the Internet of Things (IoT) and smart devices has fueled a surge in data-intensive applications. However, this surge in request data traffic has also resulted in enormous network load. Therefore, edge caching has emerged as an effective solution to alleviate backhaul congestion caused by duplicate content transmission. By caching popular content at the edge, user access latency can be reduced and service quality improved. However, static edge servers may not be able to provide reliable content delivery in the face of dynamically changing network topologies, resulting in a poor user experience. Deploying multiple edge servers also incurs additional costs.

[0003] Drones, acting as aerial base stations, can provide wider coverage and higher transmission rates, offering flexible wireless access to wireless networks. Augmenting terrestrial cellular networks with drones can expand service coverage in rural and remote areas, hotspots, and emergency situations. However, drones have limited cache and storage capabilities and may not be able to meet user requests. Therefore, high-altitude platforms can serve as complementary components of drone networks, assisting drones in covering larger areas. Furthermore, caching-enabled drones can be dynamically deployed to provide services as needed, improving caching efficiency.

[0004] Traditional trajectory optimization algorithms require global information, including user location, channel status, and content popularity distribution, some of which is often unavailable in dynamically changing environments. Furthermore, information such as user location and request content is private, leading to privacy concerns in centralized approaches. Therefore, federated learning, which enables collaborative training without sharing raw data, has been introduced as a promising approach. Summary of the Invention

[0005] The purpose of the present invention is to propose a multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning, to study the content caching and trajectory planning scenarios of multiple UAVs, to propose a federated learning framework in order to protect user privacy, and to use a method based on deep reinforcement learning to optimize the cache replacement strategy and the trajectory of the UAV. In order to improve the cache hit rate, the UAV downloads a pre-trained cache replacement model from a high-altitude platform to make local replacement decisions. At the same time, the UAV will perform distributed training locally, optimize the trajectory by maximizing fair throughput, and upload the model weights to the high-altitude platform. The high-altitude platform aggregates the models of all UAVs through a federated learning algorithm, and uses a global model to assist in updating the local model of the UAV. The method proposed in this embodiment can conduct more thorough exploration under the premise of protecting privacy to solve the problems of cache hits and fair communication of users.

[0006] To achieve the above objectives, the present invention provides a multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning, comprising:

[0007] An air-ground wireless communication cache network is constructed to obtain a cumulative average throughput of the air-ground wireless communication cache network; wherein the air-ground wireless communication cache network includes: M drones and M service groups;

[0008] Based on the cumulative average throughput, construct an optimization objective function for the UAV trajectory;

[0009] The cache replacement model in the UAV is constructed as the first MDP model, and the trajectory design of the UAV is constructed as the second MDP model;

[0010] Based on the optimization objective function, a federated deep reinforcement learning algorithm is used to calculate global parameters of the first MDP model and the second MDP model, and multi-UAV content caching and trajectory planning are performed based on the global parameters.

[0011] Optionally, obtaining the cumulative average throughput of the air-ground wireless communication cache network includes:

[0012] Get the cumulative throughput of user n in the first t time slots;

[0013] Define the throughput ratio of user n;

[0014] Obtaining a fairness index based on a Jain fairness index using the throughput ratio;

[0015] The cumulative average throughput is obtained based on the time cumulative throughput and the fairness index in the first t time slots.

[0016] Optionally, obtaining the time cumulative throughput of user n in the first t time slots includes:

[0017] Calculate the request probability of the content to simulate the user request probability;

[0018] Based on the user request probability, obtain the cache hit probability of drone m for user n in time slot t;

[0019] Get the downlink data transmission rate between the drone and the user;

[0020] Based on the cache hit probability of drone m for user n in time slot t and the downlink data transmission rate, the time cumulative throughput of user n in the first t time slots is obtained.

[0021] Optionally, the time cumulative throughput in the first t time slots is:

[0022]

[0023] Among them, D n (t) is the time cumulative throughput, is the cache hit rate of drone m for user n in time slot t, R m,n (t) is the data transmission rate between UAV m and user n in time slot t; i is the variable in the accumulation function, which is used to refer to t;

[0024] The throughput ratio of user n is:

[0025]

[0026] Among them, f n (t) is the throughput ratio of user n, N is the number of users;

[0027] The fairness index is:

[0028]

[0029] in, is the fairness index;

[0030] The cumulative average throughput is:

[0031]

[0032] Among them, D f is the cumulative average throughput, and T is the total number of time slots.

[0033] Optionally, the cache hit probability of drone m for user n in time slot t is:

[0034]

[0035] in, is the cache hit probability of drone m for user n in time slot t, Ψ m(t) is the cache space of UAV m, ξ is the cache replacement strategy of UAV, a m,n (t) is the access decision in time slot t, is the number of cached contents hit, is the total number of requests;

[0036] The request probability of the content is:

[0037]

[0038] Among them, p i represents the request probability of the content, K represents the number of cached content blocks, j represents {1,...,K}, and λ represents the popularity coefficient;

[0039] The downlink data transmission rate between the drone and the user is:

[0040]

[0041] Among them, R m,n (t) represents the downlink data transmission rate between the UAV and the user, a m,n (t) represents the access decision in time slot t, b m,n (t) represents the channel bandwidth allocated to user n, P0 and N0 represent the transmission power and noise power spectral density respectively, represents the average path loss of the UAV-user link.

[0042] Optionally, the optimization objective function of the UAV trajectory is:

[0043]

[0044] stC1:

[0045] C2:

[0046] C3:

[0047] C4:

[0048] C5:

[0049] C6:

[0050] Among them, D f represents the cumulative average throughput of the system, p m (t) represents the trajectory of the UAV, q nrepresents the location of the group users, C1 and C2 represent the constraints of the limited area, C3 and C4 represent the flight speed constraints of the drone, and its maximum speed is v max , the maximum acceleration is a max , C5 represents the UAV's airborne energy constraint, x m (t) represents the horizontal coordinate of UAV m at time slot t, X max Indicates the range of user distribution area, It means that for any UAV m, Indicates that for any time slot t, y m (t) represents the horizontal ordinate of UAV m at time slot t, Y max Indicates the range of user distribution area, v m (t) represents the speed of UAV m at time slot t, a m (t) represents the acceleration of UAV m at time slot t, E init represents the initial energy value of the drone, E(t) represents the remaining energy of the drone at time slot t, K represents the number of content blocks, represents the cache decision of UAV m in time slot t, and C represents the size of the cache space.

[0051] Optionally, the method for obtaining the remaining energy of the drone in time slot t is:

[0052] Get the thrust of the rotor during the flight of the drone;

[0053] Based on the thrust of the rotor during the flight of the drone, the energy consumed by the drone during the flight is obtained;

[0054] Based on the energy consumed by the UAV during flight, the remaining energy of the UAV in time slot t is obtained.

[0055] Optionally, the position of drone m in the next time slot is:

[0056]

[0057] Among them, p m (t) is the position of UAV m at time slot t, v m (t) is the velocity vector of UAV m at time slot t, a m (t) is the acceleration vector of UAV m at time slot t, and Δ is the length of each time slot.

[0058] The thrust of the rotor during the flight of the drone is:

[0059]

[0060] Where m is the mass of the drone, ρ is the air density, v = ‖v‖ is the absolute value of the drone's speed, S FP is the equivalent plate area, g is the gravitational acceleration vector, T h (v,a) is the thrust of the rotor during the flight of the UAV;

[0061] The energy consumed by the drone during flight is:

[0062]

[0063] Among them, P(v,T h ) is the energy consumed by the UAV during flight, nr is the number of rotors, Th is the thrust of the rotors during flight, τ c is the climb angle, d0 is the fuselage drag ratio of each rotor, c s is the solidity of each rotor, A is the disc area of ​​each rotor, δ is the drag coefficient of the local blade section, c T is the thrust coefficient based on the brake disc area, c f is the correction factor of the induced power increment;

[0064] The remaining energy of the drone at time slot t is:

[0065]

[0066] Among them, E(t) is the remaining energy of the UAV at time slot t, E init is the initial energy of the UAV, and Δ is the length of the time slot.

[0067] Optionally, the first MDP model includes: a first state space, a first action space, and a first reward design;

[0068] The first state space is:

[0069] s t ={V0,V1,...,V C}

[0070] Among them, V i ,i=0,...,C is the feature vector of the request content, which consists of three parts: short-term feature v si , mid-term characteristics v mi and long-term characteristics v li , respectively, represent the total number of requests for the corresponding content within a specific time window;

[0071] The first action space is:

[0072] a t =ξ∈{1,...,C}

[0073] Where ξ=1,...,C means that the content of the current request is used to replace the content cached at the corresponding location;

[0074] The first reward is designed as follows:

[0075]

[0076] Where h(t) is the cumulative hit count of each cached content in the t-th decision period, and the weight w is used i to highlight the differences on each cache slot;

[0077] The second MDP model includes: a second state space, a second action space, and a second reward design;

[0078] The second state space is:

[0079] s t =(L m,n (t),v(t),v p (t))

[0080] Among them, L m,n (t) is the relative position of the drone and the user, v(t) is the flight speed of the drone, and v p (t) is the flight direction of the UAV;

[0081] The second action space is:

[0082] a t ={a(t),a p (t)}

[0083] Among them, a(t) is the acceleration of the drone, a p (t) is the direction of the UAV’s acceleration;

[0084] The second reward is designed as follows:

[0085] r t =r t th +r t pe

[0086] Among them, r t th is the throughput reward, r t pe It is a constrained negative incentive.

[0087] Optionally, the global parameters are calculated using the federated deep reinforcement learning algorithm:

[0088] ω global (ε)=ρ i ω i(ε)

[0089] Among them, ρ i is the importance weight of the ith agent, ω global (ε) represents the global model parameters, ω i represents the local model parameters of the i-th distributed agent.

[0090] The present invention has the following beneficial effects:

[0091] This paper studies the content caching and trajectory planning scenarios of multiple drones. To protect user privacy, a federated learning framework is proposed, and a deep reinforcement learning-based method is used to optimize the cache replacement strategy and drone trajectories. To improve the cache hit rate, the drone downloads a pre-trained cache replacement model from a high-altitude platform to make local replacement decisions. At the same time, the drone will perform distributed training locally, optimize the trajectory by maximizing fair throughput, and upload the model weights to the high-altitude platform. The high-altitude platform aggregates the models of all drones through a federated learning algorithm and uses a global model to assist in updating the local models of the drones. The method proposed in this paper can conduct more comprehensive exploration to solve the problems of cache hits and fair communication for users while protecting privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0093] Figure 1 This is a flowchart of a method for multi-UAV content caching and trajectory planning based on federated deep reinforcement learning according to an embodiment of the present invention;

[0094] Figure 2 Schematic diagram comparing the convergence of three algorithms according to an embodiment of the present invention;

[0095] Figure 3 Schematic diagram of the performance of various methods under different user scales according to an embodiment of the present invention;

[0096] Figure 4 Schematic diagram of the effect of cache space size on the overall cache miss rate according to an embodiment of the present invention. DETAILED DESCRIPTION

[0097] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0098] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0099] like Figure 1 As shown, this embodiment proposes a multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning, including:

[0100] An air-ground wireless communication cache network is constructed to obtain a cumulative average throughput of the air-ground wireless communication cache network. The air-ground wireless communication cache network in this embodiment includes: M drones and M service groups.

[0101] Based on the cumulative average throughput of this embodiment, an optimization objective function for the UAV trajectory is constructed;

[0102] The cache replacement model in the UAV is constructed as the first MDP model, and the trajectory design of the UAV is constructed as the second MDP model;

[0103] Based on the optimization objective function of this embodiment, the federated deep reinforcement learning algorithm is used to calculate the global parameters of the first MDP model and the second MDP model of this embodiment, and multi-UAV trajectory planning and cache replacement are performed based on the global parameters of this embodiment.

[0104] Furthermore, obtaining the cumulative average throughput of the air-ground wireless communication cache network includes:

[0105] Get the cumulative throughput of user n in the first t time slots;

[0106] Define the throughput ratio of user n;

[0107] Using the throughput ratio of this embodiment, a fairness index based on the Jain fairness index is obtained;

[0108] The cumulative average throughput of this embodiment is obtained based on the time cumulative throughput and the fairness index in the first t time slots.

[0109] Furthermore, obtaining the time cumulative throughput of user n in the first t time slots includes:

[0110] Calculate the request probability of the content to simulate the user request probability;

[0111] Based on the user request probability, obtain the cache hit probability of drone m for user n in time slot t;

[0112] Get the downlink data transmission rate between the drone and the user;

[0113] Based on the cache hit probability and downlink data transmission rate of drone m to user n in time slot t, the time cumulative throughput of user n in the first t time slots is obtained.

[0114] Furthermore, the method for obtaining the remaining energy of the drone in time slot t in this embodiment is as follows:

[0115] Get the position of UAV m in the next time slot;

[0116] Get the thrust of the rotor during the flight of the drone;

[0117] Based on the thrust of the rotor during the flight of the drone, the energy consumed by the drone during the flight is obtained;

[0118] Based on the energy consumed by the UAV during flight, the remaining energy of the UAV in time slot t is obtained.

[0119] This embodiment studies the content caching and trajectory planning scenarios of multiple drones. In order to protect user privacy, a federated learning framework is proposed, and a deep reinforcement learning-based method is used to optimize the cache replacement strategy and drone trajectories. Figure 1 The specific steps are as follows:

[0120] 1. Build an air-ground wireless communication caching network consisting of M drones and M service groups. Each group of N users is provided with caching services by a drone. The total execution time is divided into T time slots, t∈{1,...,T}, and each time slot has the same length Δ. The flight altitude of the drone is fixed at h uav , at time slot t, the coordinates of the mth UAV can be obtained using p m (t)=(x m (t),y m (t),h uav ) to indicate that the user’s location is defined as q n =(x n ,y n ,h user ), so the distance between the user and the drone can be defined as d m,n (t)=‖p m (t)-q n ‖.

[0121] Consider a requested content library F = {f1,f2,...,f K}, which consists of K content blocks of equal size. i The request probability is denoted as p i ,i=1,...,K, it satisfies Zipf's law, so the content f i The request probability can be expressed as:

[0122]

[0123] Where λ is the popularity coefficient. A larger λ means a more concentrated distribution of content files. Conversely, a smaller λ means a more uniform distribution of content request probabilities.

[0124] 2. The drone has a communication range d com , when d m,n (t)<d com When , user n can request content from drone m. Compared with ground communication propagation, the drone-to-ground channel depends on altitude and elevation angle. In the complex environment of the city, obstacles also affect signal propagation. Therefore, the channel can be modeled as two types: LoS channel and NLoS channel to simulate the actual situation. In terms of propagation model, the log-normal shadow channel is adopted, in which the LoS link and NLoS link can be modeled by the corresponding channel parameters. Therefore, for the mth drone and the nth user, the LoS-NLoS probability formula of the transmission link between them is defined as follows:

[0125]

[0126] Where c1 and c2 are parameters related to the environment. In addition, the elevation angle is given by It is concluded that the LoS and NLoS path losses between the UAV and the user can be expressed as:

[0127]

[0128] in is the inverse of the free space transmission loss, c is the speed of light, f c is the carrier frequency, β LoS and β NLoS are the path loss coefficients under different environments. Therefore, the average path loss of the UAV-user link can be expressed as:

[0129]

[0130] The downlink data transmission rate between the drone and the user can be expressed as:

[0131]

[0132] Among them, a m,n (t) represents the access decision in time slot t. If user n accesses the drone and sends a request in time slot t, then a m,n (t)=1, otherwise a m,n (t) = 0. b m,n (t) is the channel bandwidth allocated to user n. P0 and N0 represent the transmission power and noise power spectral density, respectively.

[0133] 3. UAVs have high maneuverability, and in this study, the flight altitude of the UAV is fixed, so its velocity vector can be defined as v = (v x ,v y ), where 0≤‖v‖≤v max Similarly, the drone acceleration vector a can be defined as (a x ,a y ), where 0≤‖a‖≤a max Because the duration of each time slot is very short, it can be considered that the drone is moving with uniform acceleration within the time slot. Therefore, for drone m, its speed and acceleration in the next time slot can be expressed as v m (t+1)=v m (t)+a m (t)Δ, Therefore, its position in the next time slot can be expressed as:

[0134]

[0135] During the flight of a drone, the thrust equation of the rotor can be expressed as a function of the drone's acceleration and velocity. Therefore, the thrust equation of each rotor can be obtained by the following formula:

[0136]

[0137] where n r is the number of rotors, m is the mass of the UAV, ρ is the air density, v=‖v‖ is the absolute value of the UAV speed, S FP is the equivalent plate area, g is the gravitational acceleration vector, and the energy consumed by the drone during flight can be expressed as:

[0138]

[0139] Where δ is the local blade section drag coefficient, c T is the thrust coefficient based on the brake disc area, A is the disc area of ​​each rotor, c s is the solidity of each rotor, c f is the correction coefficient of the induced power increment, τ c is the climb angle, and d0 is the fuselage drag ratio of each rotor.

[0140] Assume that the initial energy value of the drone is E init ,According to the above energy consumption formula, the remaining energy of the UAV in time slot t can be obtained as:

[0141]

[0142] 4. Assuming that each drone has a limited cache storage capacity C, the cache space of drone m can be expressed as in Indicates whether the drone has cached the content f i When a drone provides services to a user, each time a cache miss occurs, the drone needs to make a replacement decision ξ∈{1,...,C}, where ξ represents the corresponding cache location. This decision will affect subsequent cache hits. For user n, the cache hit probability of drone m for user n in time slot t is:

[0143]

[0144] in is the number of cached contents hit, is the total number of requests. and It is the same variable, the subscript in the lower right corner Ψm(t),ξ express This variable is affected by these factors, but this influence can't be expressed in a formula, so it's expressed in superscript form. Simply put, the hit rate is calculated as the number of hits divided by the total number of hits. However, the number of hits is definitely affected by cache space and policies. If the content is cached, the number of hits may increase, but whether this increase actually occurs depends on whether users request the content.

[0145] 5. The cumulative throughput of user n in the first t time slots is

[0146]

[0147] Maximizing system throughput may cause the drone to hover near some users with good channel conditions, thus neglecting to serve other users. In order to balance fairness and throughput, the throughput ratio of user n is defined as:

[0148]

[0149] The fairness index based on Jain's fairness index is:

[0150]

[0151] in The closer the fairness index is to 1, the fairer the communication is. The total fair throughput is defined as:

[0152]

[0153] Therefore, the trajectory optimization problem of the UAV can be expressed as:

[0154]

[0155] stC1:

[0156] C2:

[0157] C3:

[0158] C4:

[0159] C5:

[0160] C6:

[0161] Among them, C1 and C2 represent the constraints of the limited area, C3 and C4 represent the flight speed constraints of the UAV, and its maximum speed is v max , the maximum acceleration is a max , C5 represents the onboard energy constraint of the UAV.

[0162] 6. The problem proposed is a non-convex optimization problem, which is difficult to solve within the polynomial computational complexity. Therefore, this paper adopts a deep reinforcement learning method based on TD3 to optimize the trajectory of the UAV. Considering that the parameters of the cache space are discrete, the present invention adopts a DQN-based method to optimize the cache replacement strategy of the UAV, so as to better cooperate with TD3 to solve the above problem.

[0163] The cache replacement model in the drone is modeled as an MDP, and the decision-making process when a cache miss occurs is defined. The cache state, system action, and feedback reward are as follows:

[0164] State space: To limit the state space, this embodiment only extracts the feature vectors V of the cached content and the current request content. i As the state, i=0,1,...,C. The index ranges from 0 to the cache capacity C. The index of the current request content is 0, and the index of the cache content is 1 to C, indicating the content at the corresponding position in the cache space. The feature vector consists of three parts: short-term feature v si , mid-term characteristics v mi and long-term characteristics v li , respectively represent the total number of requests for the corresponding content in a specific time window. Therefore, at each time slot t, the state of the cache is defined as s t ={V0,V1,...,V C}.

[0165] Action space: To limit the action space, this embodiment assumes that in each decision period, the drone can only replace a cached content with the currently requested content, or keep the cached state unchanged.t =ξ∈{1,...,C} is the action selected during the decision-making period, ξ=1,...,C means replacing the content cached at the corresponding location with the content of the current request.

[0166] Reward design: The cache hit rate affects the optimization goal of this embodiment and can therefore be used as an evaluation metric for the cache strategy. The goal of this embodiment is to optimize the overall hit rate using short-term feedback. The reward for each decision period depends on the short-term and long-term cache hit rates. Therefore, the reward function is defined as:

[0167]

[0168] Where h(t) is the cumulative number of hits for each cached content in the t-th decision period, and the weight w i To highlight the differences on each cache slot, you can favor the most recently selected operations by increasing the weight of the new cache content.

[0169] This embodiment also models the trajectory design of the UAV as an MDP, defining the UAV to make a movement decision at each time slot. The model is defined as follows:

[0170] State Space: In wireless communication systems, the distance between the transmitter and receiver significantly impacts communication performance. Therefore, this embodiment uses the relative position between the drone and the users in the group as the state variable. Position information allows the drone to optimize its position to improve system performance, while using relative position allows for greater generalization of the model. As mentioned above, the vertical height of the drone remains constant, so it can be represented by its horizontal relative position, defined as follows:

[0171] L m,n (t)=(x m (t)-x n ,y m (t)-y n )

[0172] Considering that the flight decision is also affected by the current flight state of the UAV, the state variables should also include the UAV's flight speed v = ‖v‖ and the UAV's flight direction v p =arctan(v y / v x ). In summary, the state space of the drone can be expressed as:

[0173] s t =(L m,n (t),v(t),v p (t))

[0174] Action space: The action variables determine the movement of the drone, and thus the trajectory of the drone.p Represent the acceleration and direction of the drone respectively. The action space of the drone can be expressed as:

[0175] a t ={a(t),a p (t)}

[0176] Reward design: Through reward design, this embodiment can transform the non-convex optimization problem into maximizing cumulative returns. Therefore, this embodiment sets:

[0177]

[0178] where κ th is a constant coefficient. In addition, this embodiment also sets rewards related to boundary values. When the drone exceeds the allowed flight range or runs out of energy and cannot return, this embodiment sets corresponding penalties:

[0179] r t pe =ζ B (t)P B (Tt)+ζ E (t)P E

[0180] ζ B (t),ζ E (t)∈{0,1} respectively indicates whether the drone has exceeded the boundary and whether the energy is exhausted. When the drone exceeds the boundary, ζ B (t)=1, otherwise ζ B (t) = 0, ζ E (t) Similarly. P B ,P E are two negative fixed coefficients, representing the penalties for exceeding the boundary and energy exhaustion, respectively.

[0181] In summary, the reward function of this embodiment is designed as follows:

[0182] r t =r t th +r t pe

[0183] 7. In centralized DRL approaches, the status and action information of all drones must be centralized in a central controller. This increases the computational and communication burden, especially in large-scale networks, where excessive latency can reduce the real-time performance and stability of the system. Therefore, this paper proposes a distributed deep reinforcement learning algorithm based on federated learning to improve system stability.

[0184] In the proposed federated deep reinforcement learning algorithm, each UAV is treated as a distributed agent to make real-time trajectory decisions. Meanwhile, the high-altitude platform collects local model training parameters from all distributed agents to perform federated averaging. In the εth round of training, after local training is complete, the network parameters ω(ε) are sent to the high-altitude platform. After collecting the parameters sent by all UAVs, the high-altitude platform calculates the global parameters using the federated averaging algorithm:

[0185] ω global (ε)=ρ i ω i (ε)

[0186] where ρ i is the importance weight of the ith agent, satisfying the constraint Determined by the specificity of the agent. In this paper, since the number of each user group is assumed to be equal, we can set ρ i = 1 / M. Each drone agent downloads the global parameters from the high-altitude platform to update the local network and then starts the next round of training.

[0187] like Figure 2 As shown in the figure, all three algorithms gradually converge with increasing training rounds. The proposed algorithm performs best. Although it experiences fluctuations due to federation, it is more likely to explore higher reward values. In contrast, the centralized DRL algorithm, due to the increased dimensionality of both states and actions, makes decision-making more difficult, potentially lingering near suboptimal solutions and requiring more exploration time. Because the reward is related to both fair throughput and cache hit rate, the baseline algorithm, which removes the cache decision, performs worse than the two aforementioned algorithms due to the influence of the cache hit rate.

[0188] Figure 3 The performance of various methods at different user scales is shown. Fair throughput improves with increasing user scale, due to the increase in the number of users within the drone's communication range. However, the performance gap between federated DRL and centralized DRL becomes more pronounced as the number of users increases. This is because centralized DRL requires excessive communication and computational resources, which affects the algorithm's learning efficiency. Therefore, our proposed federated DRL is more suitable for large-scale drone networks.

[0189] Figure 4The effect of cache size on the overall cache miss rate is shown. As can be seen from the figure, the miss rates of all replacement strategies decrease as cache capacity increases. When the cache capacity is small, the performance of the four algorithms is very similar. As the cache capacity increases, the gap between the proposed algorithm and the other baseline algorithms first increases and then decreases. When the cache capacity is large enough to accommodate most popular content, the performance of the algorithms gradually converges. Generally speaking, our proposed method consistently achieves lower miss rates than other methods.

[0190] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning, characterized by: include: An air-ground wireless communication cache network is constructed, and a cumulative average throughput of the air-ground wireless communication cache network is obtained; wherein the air-ground wireless communication cache network includes: M drones and M service groups; Based on the cumulative average throughput, construct an optimization objective function for the UAV trajectory; The cache replacement model in the UAV is constructed as the first MDP model, and the trajectory design of the UAV is constructed as the second MDP model; Based on the optimization objective function, a federated deep reinforcement learning algorithm is used to calculate global parameters of the first MDP model and the second MDP model, and multi-UAV content caching and trajectory planning are performed based on the global parameters.

2. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 1 is characterized in that: Obtaining the cumulative average throughput of the air-ground wireless communication cache network includes: Get the cumulative throughput of user n in the first t time slots; Define the throughput ratio of user n; Obtaining a fairness index based on a Jain fairness index using the throughput ratio; The cumulative average throughput is obtained based on the time cumulative throughput and the fairness index in the first t time slots.

3. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 2 is characterized in that: Obtaining the cumulative time throughput of user n in the first t time slots includes: Calculate the request probability of the content to simulate the user request probability; Based on the user request probability, obtain the cache hit probability of drone m for user n in time slot t; Get the downlink data transmission rate between the drone and the user; Based on the cache hit probability of drone m for user n in time slot t and the downlink data transmission rate, the time cumulative throughput of user n in the first t time slots is obtained.

4. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 2 is characterized in that: The cumulative throughput in the first t time slots is: Among them, D n (t) is the time cumulative throughput, is the cache hit rate of drone m for user n in time slot t, R m,n (t) is the data transmission rate between UAV m and user n in time slot t; The throughput ratio of user n is: Among them, f n (t) is the throughput ratio of user n, N is the number of users; The fairness index is: in, is the fairness index; The cumulative average throughput is: Among them, D f is the cumulative average throughput, and T is the total number of time slots.

5. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 3 is characterized in that: The cache hit probability of drone m for user n in time slot t is: in, is the cache hit probability of drone m to user n in time slot t, Ψ m (t) is the cache space of UAV m, ξ is the cache replacement strategy of UAV, a m,n (t) is the access decision in time slot t, is the number of cached contents hit, is the total number of requests; The request probability of the content is: Among them, p i represents the request probability of the content, K represents the number of cached content blocks, j represents {1,...,K}, and λ represents the popularity coefficient; The downlink data transmission rate between the drone and the user is: Among them, R m,n (t) represents the downlink data transmission rate between the UAV and the user, a m,n (t) represents the access decision in time slot t, b m,n (t) represents the channel bandwidth allocated to user n, P0 and N0 represent the transmission power and noise power spectral density respectively, represents the average path loss of the UAV-user link.

6. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 1 is characterized in that: The optimization objective function of the UAV trajectory is: Among them, D f represents the cumulative average throughput of the system, p m (t) represents the trajectory of the UAV, q n represents the location of the group users, C1 and C2 represent the constraints of the limited area, C3 and C4 represent the flight speed constraints of the drone, and its maximum speed is v max , the maximum acceleration is a max , C5 represents the UAV's airborne energy constraint, x m (t) represents the horizontal coordinate of UAV m at time slot t, X max Indicates the range of user distribution area, It means that for any UAV m, Indicates that for any time slot t, y m (t) represents the horizontal ordinate of UAV m at time slot t, Y max Indicates the range of user distribution area, v m (t) represents the speed of UAV m at time slot t, a m (t) represents the acceleration of UAV m at time slot t, E init represents the initial energy value of the drone, E(t) represents the remaining energy of the drone at time slot t, K represents the number of content blocks, represents the cache decision of UAV m in time slot t, and C represents the size of the cache space.

7. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 6 is characterized in that: The method for obtaining the remaining energy of the drone in time slot t is: Get the thrust of the rotor during the flight of the drone; Based on the thrust of the rotor during the flight of the drone, the energy consumed by the drone during the flight is obtained; Based on the energy consumed by the UAV during flight, the remaining energy of the UAV in time slot t is obtained.

8. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 7 is characterized in that: The position of drone m in the next time slot is: Among them, p m (t) is the position of UAV m at time slot t, v m (t) is the velocity vector of UAV m at time slot t, a m (t) is the acceleration vector of UAV m at time slot t, and Δ is the length of each time slot; The thrust of the rotor during the flight of the drone is: Where m is the mass of the drone, ρ is the air density, v = ‖v‖ is the absolute value of the drone's speed, S FP is the equivalent plate area, g is the gravitational acceleration vector, T h (v,a) is the thrust of the rotor during the flight of the UAV; The energy consumed by the drone during flight is: Among them, P(v,T h ) is the energy consumed by the UAV during flight, nr is the number of rotors, Th is the thrust of the rotors during flight, τ c is the climb angle, d0 is the fuselage drag ratio of each rotor, c s is the solidity of each rotor, A is the disc area of ​​each rotor, δ is the drag coefficient of the local blade section, c T is the thrust coefficient based on the brake disc area, c f is the correction factor of the induced power increment; The remaining energy of the drone at time slot t is: Among them, E(t) is the remaining energy of the UAV at time slot t, E init is the initial energy of the UAV, and Δ is the length of the time slot.

9. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 1 is characterized in that: The first MDP model includes: a first state space, a first action space, and a first reward design; The first state space is: s t ={V0,V1,...,V C } Among them, V i ,i=0,...,C is the feature vector of the request content, which consists of three parts: short-term feature v si , mid-term characteristics v mi and long-term characteristics v li , respectively, represent the total number of requests for the corresponding content within a specific time window; The first action space is: a t =ξ∈{1,...,C} Where ξ=1,...,C means that the content of the current request is used to replace the content cached at the corresponding location; The first reward is designed as follows: Where h(t) is the cumulative hit count of each cached content in the t-th decision period, and the weight w is used i to highlight the differences on each cache slot; The second MDP model includes: a second state space, a second action space, and a second reward design; The second state space is: s t =(L m,n (t),v(t),v p (t)) Among them, L m,n (t) is the relative position of the drone and the user, v(t) is the flight speed of the drone, and v p (t) is the flight direction of the UAV; The second action space is: a t ={a(t),a p (t)} Among them, a(t) is the acceleration of the drone, a p (t) is the direction of the UAV’s acceleration; The second reward is designed as follows: r t ＝r t th +r t pe Among them, r t th is the throughput reward, r t pe It is a constrained negative incentive.

10. The multi-UAV content caching and trajectory planning method based on federated deep reinforcement learning according to claim 1 is characterized in that: The global parameters calculated using the federated deep reinforcement learning algorithm are: oh global (e)=r i oh i (e) Among them, ρ i is the importance weight of the ith agent, ω global (ε) represents the global model parameters, ω i represents the local model parameters of the i-th distributed agent.

Images