Deep reinforcement learning method for combining unmanned aerial vehicle deployment and resource allocation

Abstract

The invention discloses a deep reinforcement learning method for combining unmanned aerial vehicle deployment and resource allocation, and the method comprises the steps: building a communication network model, and initializing a communication environment, the number of unmanned aerial vehicles and the number of users; determining an optimization target and a constraint condition according to the signal-to-noise ratio of the user and the system capacity; converting the optimization problem into a Markov decision process, and determining an intelligent agent, a state space, an action space and a reward function by using a deep reinforcement learning method; each agent continuously interacts with the environment through a D3QN algorithm to optimize and update the own network, so that the agent obtains an optimal resource allocation strategy; personalized federated deep reinforcement learning is utilized to maximize long-term network throughput, protect user privacy and adapt to a time-varying network state at the same time, and in a federated learning framework of PFDRL, a deep reinforcement learning algorithm is adopted, so that an unmanned aerial vehicle can decide resource allocation and position adjustment in real time based on local observation, and a global optimal solution is realized at the same time.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and particularly to a deep reinforcement learning method for joint UAV deployment and resource allocation. Background Art

[0002] The sixth-generation (6G) wireless network aims to provide users with ultra-real-time experiences anytime and anywhere, effectively expanding the communication coverage. This requires dynamic coverage and connection expansion by leveraging innovative wireless nodes. Due to the inherent advantages of UAV communication, UAVs can serve as aerial mobile base stations in remote areas or as temporary infrastructure in areas where conventional network infrastructures are damaged by disasters such as earthquakes and tsunamis.

[0003] However, the application of UAVs as aerial navigation systems faces many challenges. On the one hand, more flexible base station deployment schemes are crucial for UAV base stations (BSs) because the management and scheduling of multiple UAVs become more difficult, which will significantly affect the coverage and transmission performance. On the other hand, resource allocation in UAV networks is coupled with UAV deployment, so these two problems need to be jointly solved to optimize network performance. In other words, traditional resource allocation strategies are not applicable to UAV dynamic scenarios.

[0004] The joint deployment and resource allocation scheme of UAV BSs is particularly important, which requires a large amount of accurate network state information. However, in practical applications, due to the mobility of users and UAVs, dynamic quality of service (QoS) requirements, and unstable wireless channel states, network states are usually time-varying, so static optimization becomes ineffective in solving problems.

[0005] Therefore, an innovative solution is needed that can improve network performance while reducing energy consumption, especially in UAV networks. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides a deep reinforcement learning method for joint UAV deployment and resource allocation, including the following steps:

[0007] S1. Establish a communication network model, initialize the communication environment, the number of UAVs, and the number of users;

[0008] S2. Determine the optimization objective and constraints according to the signal-to-noise ratio and system capacity of users;

[0009] S3. Convert the optimization problem into a Markov decision process, and use the deep reinforcement learning method to determine the agent, state space, action space, and reward function;

[0010] S4. Each agent continuously interacts with the environment through the D3QN algorithm to optimize and update its own network, so that the agent obtains the optimal resource allocation strategy.

[0011] The further limited technical solution of the present invention is:

[0012] Further, in step S1, the communication network model includes a leading UAV, M UAVs, and N ground users. The leading UAV communicates with other UAVs, and the UAV deployment and resource allocation are executed in a time-slot manner; the user locations and requested services are fixed in specific time slots, and a user communicates with only one UAV within one time slot; it is assumed that only the users within the beam coverage area of the UAV communicate with the UAV:

[0013]

[0014] Among them, represents the horizontal distance between user i and UAV j, H represents the height of the UAV, and θ j represents the half-power beamwidth of UAV j.

[0015] As described above, in a deep reinforcement learning method for joint UAV deployment and resource allocation, in step S1, the service request of user i within a time slot is defined as Among them, represents the amount of data to be transmitted to the UAV, represents the transmission delay constraint of this service request; all UAVs share the spectrum resources of the system; a binary variable is defined to represent whether UAV j uses sub-channel k to serve user i at time slot t. If so, then Otherwise, That is, user i can only be served by one UAV using a sub-channel for user i at time slot t.

[0016] As described above, in a deep reinforcement learning method for joint UAV deployment and resource allocation, in step S2, the interference signals received by the UAVs are determined, and the signal-to-noise ratio information is calculated; the UAV communication link follows a probabilistic path loss model, where the line-of-sight LoS and non-line-of-sight NLoS links have different occurrence probabilities. In time slot t, the probability of the existence of a LoS path between UAV j and user i is:

[0017]

[0018] Among them, α and β represent environment-related parameters; the probability of the existence of an NLoS link is represents the size of the data to be transmitted to the UAV;

[0019] The signal-to-noise ratio of the drone j and the user i on the sub-channel k is as follows:

[0020]

[0021] Among them, represents the interference of the drone j. Only the ground users covered repeatedly by multiple drones will interfere with the other drones that are not communicating with this ground user. Therefore, the transmission rate of the drone j and the user on the time slot of the sub-channel k is expressed as

[0022] As described above, a deep reinforcement learning method for joint drone deployment and resource allocation. In step S2, the optimization objective is to maximize the overall throughput of the system by optimizing the resource allocation strategy. The problem is formulated as maximizing the overall throughput of the system, which is defined as:

[0023]

[0024] The constraint conditions are as follows:

[0025]

[0026]

[0027] Among them, and α = {α1, α2,..., α M} respectively represent access control, drone position, and the beam width of each drone; the constraint condition (a) means that a user can only access one drone on one sub-channel in a time slot; the constraint condition (b) means that the dynamic quality-of-service requirements of the user are satisfied by the drone serving it; the constraint condition (c) means that the served user is within the coverage area of the relevant drone; the constraint conditions (d) and (e) respectively constrain the beam width of the drone antenna and the safety distance between drones.

[0028] As described above, a deep reinforcement learning method for joint drone deployment and resource allocation. In step S3, considering the time-varying constraints and long-term optimization objectives, the optimization problem is set as a sequential decision-making problem, and the network environment in future time slots is unknown; since each drone can only use the local observation information within its coverage area to adjust its position, beam width, and access control to maximize the network throughput, the problem is modeled as a decentralized partially observable Markov decision process, where:

[0029] Agent: The drone j (1 ≤ j ≤ J) is regarded as an agent. Each agent independently updates its strategy and selects the sub-carrier and transmission power by itself;

[0030] State space: Defined as Among them respectively represent the current position of the UAV and the user position, represent the current beamwidth of the UAV;

[0031] Action space: The action selected by UAV j at time slot t is a j , where and respectively represent the beamwidth of UAV j and the access control parameter, represent the movement of UAV j;

[0032] Reward function: When the agent executes an action and satisfies the constraint conditions (a), (b), and (c), a reward value will be obtained; according to the observation exchange range defined by the state, the reward functions for any primary user or secondary user are respectively defined:

[0033]

[0034] If the UAV does not meet the safety distance requirement, a penalty of ξ1 will be imposed on the UAV; if the associated user does not meet the dynamic quality of service requirement, a penalty of ξ2 will be imposed on the UAV; otherwise, the reward is defined as the throughput.

[0035] As described above, in a deep reinforcement learning method for joint UAV deployment and resource allocation, in step S3, deep learning is incorporated into the federated learning framework, and an integrated distillation and dynamic regularization mechanism is added to the framework to form a federated deep reinforcement learning framework PFRL with integrated distillation and dynamic regularization; based on the global objective and the local loss function, the personalized model PFRL of UAV j is optimized. The global objective is to aggregate the features of the local models of each UAV, and the local loss function F j (θ j ) is used to extract features from local data;

[0036] The global objective is incorporated as a regularization term, and the optimization problem of the personalized model PFRL of each UAV j∈M is expressed as:

[0037]

[0038] Among them, the parameter λ controls the similarity between the personalized model PFRL and the global model. When λ = 0, the personalized model PFRL degenerates into a local model. The larger λ is, the more similar the personalized model PFRL is to the global model; the larger the value of τ is, the more heterogeneous the environment faced by each UAV is; when n → ∞ and λ* → 0, that is, when each UAV has an infinite number of local observations, no federated learning is performed, and the training of the personalized model PFRL only depends on local observations; when n → 0 and λ* → ∞, the training of the personalized model PFRL depends on the global model; when τ → ∞ and λ → 0, that is, when the local data of each UAV is completely uncorrelated, the training of the personalized model PFRL can only depend on local data samples; when τ → 0 and λ → ∞, that is, when the local data distributions of all UAVs are the same, the training of the personalized model PFRL depends on the global model.

[0039] As described above, in a deep reinforcement learning method for joint UAV deployment and resource allocation, in step S4, within a time slot, each UAV selects an action according to the current observation of the environment. When the UAV takes an action, the environmental state switches to another state, and the UAV obtains a corresponding reward; in the D3QN algorithm, the target value is calculated as follows:

[0040] y t = r t+1 + γq(s t+1 , argmax a q(s t+1 , a; α e ) ; α t )

[0041] That is, the evaluation network is used to obtain the action corresponding to the optimal action value in the s t+1 state, and then the target network is used to calculate the action value of this action, so as to obtain the target value; where θ e and α t represent the parameters of the evaluation network and the target network respectively.

[0042] As described above, in a deep reinforcement learning method for joint UAV deployment and resource allocation, in step S4, after a round of local model training, each UAV transmits the weight vector of its local model to the leading UAV for global model aggregation; train a personalized model, which performs local training without uploading and aggregating the model; use the global model as regularization to modify the training process; introduce the parameter λ, and adjust the personalization degree of the algorithm by dynamically changing the value of λ; represent the personalized target to be solved as:

[0043]

[0044] Among them, g r represents the global model;

[0045] Integrate the action selection strategies of all drones to obtain a global model and correct the strategies obtained from local observations; when the leading drone uploads the local model to the drones, it will first perform parameter averaging to obtain a preliminary local model, which is defined as:

[0046]

[0047] where P j represents the amount of training data of drone j, represents the total amount of training data of all drones; after averaging the parameters, the second step of aggregating the global model is performed; the global model is modified using set distillation; the initial global model is used as the student model, and each local model uploaded by the drones is used as the teacher model. First, convert the Q-value output of each teacher model into a selection behavior strategy with a Boltzmann distribution, which is expressed as:

[0048]

[0049] where T em is the distillation temperature parameter, and the higher the value of T em the softer the probability distribution of different actions taken by the current policy; represents the local model uploaded by drone j of the policy network.

[0050] As described above, in a deep reinforcement learning method for joint drone deployment and resource allocation, in step S4, in order to minimize the gap between the policy of the preliminary student global model and the teacher network the loss function is defined as the KL divergence between the student model and the policy corresponding to each teacher network ; the loss function is expressed as:

[0051]

[0052] where, represents the policy network of the teacher model, represents the local model uploaded by drone j of the policy network;

[0053] Update with the loss function between different teacher models and student models is expressed as:

[0054]

[0055] The beneficial effects of the present invention are:

[0056] (1) In the present invention, an adaptive and flexible joint UAV deployment and resource allocation scheme is proposed. This scheme utilizes personalized federated deep reinforcement learning (referred to as PFDRL), aiming to maximize the long-term network throughput while protecting user privacy and adapting to time-varying network states. In the federated learning framework of PFDRL, a deep reinforcement learning (DRL) algorithm is adopted, enabling UAVs to make real-time decisions on resource allocation and position adjustment based on local observations while achieving a global optimal solution.

[0057] (2) In the present invention, the Dueling Double Deep Q Network (D3QN) algorithm incorporates the idea of the Double DQN algorithm based on the Dueling DQN algorithm. Its only difference from the Dueling DQN algorithm lies in the way of calculating the target value. In the Dueling DQN algorithm, the target value is calculated by using the target network to obtain the action values of all actions in the S t+1 state, and then calculating the target value based on the optimal action value. Due to this maximization operation, the algorithm has an "overestimation" problem, affecting the accuracy of decision-making. In the D3QN algorithm, the target value is calculated by using the evaluation network to obtain the action corresponding to the optimal action value in the S t+1 state, and then using the target network to calculate the action value of this action, thereby obtaining the target value. Through the interaction of the two networks, the "overestimation" problem of the algorithm is effectively avoided, which makes it perform well in scenarios of joint UAV deployment resource allocation algorithms that require long-term planning and decision-making.

[0058] (3) In the present invention, in the UAV deployment network, while ensuring the quality of user services, federated learning and reinforcement learning algorithms are utilized. This method promotes information sharing among UAVs through the federated learning framework while protecting user privacy, and uses the reinforcement learning algorithm to optimize the strategy of each UAV, thereby effectively improving the overall throughput of the system, enhancing energy efficiency, and significantly reducing power consumption. This adaptive strategy can not only cope with the complex challenges in the dynamic network environment but also achieve the efficient utilization and optimal allocation of network resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a schematic diagram of the overall process of the present invention;

[0060] Figure 2 is a schematic diagram of the communication system model in an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of the competitive network structure in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0062] A deep reinforcement learning method for joint UAV deployment and resource allocation provided by this embodiment is as follows Figure 1 shown in the following steps:

[0063] S1. Establish a communication network model, and initialize the communication environment, the number of UAVs, and the number of users.

[0064] As Figure 2 shown, the communication environment in this embodiment mainly includes a leading UAV, M UAVs, and N ground users. It is assumed that the leading UAV can communicate with other UAVs to promote the cooperation among the UAV swarm; considering that the user locations and the requested services may be time-varying, it is assumed that the UAV deployment and resource allocation are executed in a time-slot manner; the user locations and the requested services are fixed in a specific time slot, and a user communicates with only one UAV in one time slot; the method of this embodiment mainly adopts the reinforcement learning method to allocate resources for each UAV-user pair, and maximizes the long-term network throughput by optimizing the resource allocation strategy.

[0065] To protect the effective communication between the user and the UAV, it is assumed that only the users within the beam coverage area of the UAV can communicate with the UAV:

[0066]

[0067] where represents the horizontal distance between user i and UAV j, H represents the height of the UAV, and θ j represents the half-power beam width of UAV j.

[0068] Considering the delay-constrained communication between the user and the UAV, such as exchanging time-critical command and control messages, real-time video streams, etc.; since the services required by the users may change over time, the service request of user i in the time slot is defined as where represents the amount of data to be transmitted to the UAV, represents the transmission delay constraint of this service request; all UAVs share the spectrum resources of the system, and interference will inevitably occur.

[0069] Define a binary variable to indicate whether UAV j uses sub-channel k to serve user i in time slot t. If so, otherwise, that is, user i can only be served by one UAV using a sub-channel for user i in time slot t.

[0070] S2. Determine the optimization objectives and constraints according to the signal-to-noise ratio and system capacity of the user; determine the interference signals received by the UAV, and calculate the signal-to-noise ratio information; the UAV communication link follows a probabilistic path loss model, where the line-of-sight LoS and non-line-of-sight NLoS links have different occurrence probabilities. In time slot t, the probability that there is a LoS path between UAV j and user i is:

[0071]

[0072] where α and β represent environment-related parameters; the probability of the existence of an NLoS link is represents the data size to be transmitted to the UAV.

[0073] The signal-to-interference plus noise ratio (SINR) between UAV j and user i on subchannel k is:

[0074]

[0075] where represents the interference of UAV j. In this embodiment, the interference from ground users within the coverage of UAV j is considered. That is, only the ground users covered by multiple UAVs will interfere with the other UAVs that are not communicating with this ground user. Therefore, the transmission rate between UAV j and the user on subchannel k in time slot t is expressed as

[0076] In this work, the optimization objective is to maximize the overall throughput of the system by optimizing the resource allocation strategy; in order to better achieve the long-term energy-saving operation of the network, the problem to be considered is further expressed as maximizing the overall throughput of the system, which is defined as:

[0077]

[0078] The constraints are as follows:

[0079]

[0080]

[0081] where and α = {α1, α2,..., α M} represent access control, UAV position, and the beamwidth of each UAV respectively; Constraint (a) means that a user can only access one UAV on one subchannel in a time slot; Constraint (b) means that the user's dynamic quality of service requirements are satisfied by the UAV it is served by; Constraint (c) means that the served user is within the coverage area of the relevant UAV; Constraints (d) and (e) respectively constrain the beamwidth of the UAV antenna and the safety distance between UAVs.

[0082] S3. Convert the optimization problem into a Markov decision process. Since the channel state information is unknown, use the deep reinforcement learning method to determine the agent, state space, action space, and reward function.

[0083] Considering the time-varying constraints and long-term optimization objectives, the optimization problem in this embodiment is a sequential decision problem; the network environment in future time slots (including the mobility of users and UAVs, dynamic service requirements, wireless channel state, etc.) is unknown. Since each UAV can only use the local observation information within its coverage area (including user QoS requirements, user position, and wireless channel state, etc.) to adjust its position, beamwidth, and access control to maximize the network throughput, the problem is modeled as a decentralized partially observable Markov decision process.

[0084] Build a reinforcement learning model, introduce a Markov model, and determine the agent, state space, action space, and reward function. Use the deep reinforcement learning algorithm to train and assign an optimal policy to each agent.

[0085] Agent: Consider the UAV j (1 ≤ j ≤ J) as an agent. Each agent independently updates its policy. Each agent can collect information from its own area and explore the network environment. Each agent can independently select subcarriers and transmit power.

[0086] State space (State): It is defined as where represent the current position of the UAV and the user position respectively, represents the current beamwidth of the UAV.

[0087] Action space (Action): The action selected by UAV j at time slot t is a j , where and represent the beamwidth and access control parameter of UAV j respectively, represents the movement of UAV j; create a discretized action space by discretizing the variables in the action space. Each UAV selects an action from the discrete action space according to the current policy and the observed information.

[0088] Reward function: To maximize the long-term throughput of the communication network while ensuring the quality of user services, the reward is defined as:

[0089] When the agent executes an action and satisfies the constraint conditions (a), (b), and (c), a reward value will be obtained; the reward is a quantitative feedback from the environment, used to evaluate the effectiveness of the executed action based on the observed state; to reflect the optimization goal of the resource allocation task, a reward function form similar to the energy efficiency η is designed, where the reward is a quantitative feedback from the environment, used to evaluate the effectiveness of the executed action based on the observed state.

[0090] According to the observation exchange range defined by the state, the reward functions for any primary user or secondary user are defined respectively:

[0091]

[0092] If the drone does not meet the safety distance requirement, a penalty ξ1 will be imposed on the drone; if the associated user does not meet the dynamic quality of service requirement, a penalty ξ2 will be imposed on the drone; otherwise, the reward is defined as the throughput.

[0093] In the fully decentralized mode, since there is no information sharing among drones, it is difficult to achieve the global optimal solution. As a privacy-preserving collaborative learning framework, FL can achieve information sharing among drones to a certain extent while protecting user privacy; therefore, in this embodiment, deep learning is incorporated into the federated learning framework to solve the above problems.

[0094] In addition, since the training data in each drone's policy network consists of the observations, actions, and rewards obtained by itself from the environment, that is, for drone j, the input of its local DRL model is its observations, rewards, and the relevant Q values in its experience replay pool.

[0095] Due to the uneven distribution of user locations and the variability of user service requests, the strategies that each drone needs to learn are different; therefore, the training data of a single drone may be similar but actually different, which will lead to the heterogeneity of the drone policy network; obviously, personalized learning methods can effectively improve system performance.

[0096] Since the local models need to be continuously uploaded and aggregated in FDRL, it will be difficult for the local models to adapt to the specific environment of each specific drone.

[0097] Considering the heterogeneity of local models caused by the differences in the environments of individual drones, this embodiment further proposes to incorporate an integrated distillation and dynamic regularization mechanism into the framework to promote the collaboration of heterogeneous local models; in this way, this embodiment develops a personalized federated deep reinforcement learning framework with integrated distillation and dynamic regularization, called PFRL.

[0098] To better address the heterogeneity of local models caused by the environmental differences faced by individual drones, learn the personalized and drone-specific models across the network for drone j.

[0099] This embodiment considers two objectives to formulate the optimization problem of the personalized model PFRL for drone j. One is the global objective, which is to aggregate the features of the local models of each drone; the other is the local loss function F j (α j ), whose purpose is to extract features from local data.

[0100] To combine these two objectives, the global objective is incorporated as a regularization term to encourage the personalized model to approach the optimal global model; the optimization problem of the personalized model PFRL for each drone j ∈ M is expressed as:

[0101]

[0102] where the parameter λ controls the similarity between the personalized model PFRL and the global model. When λ = 0, the personalized model PFRL degenerates into a local model, and the larger λ is, the more similar the personalized model PFRL is to the global model.

[0103] By adjusting the value of λ to better optimize the training process, it is necessary to find a suitable λ*, and λ* can be expressed by deriving the Bayesian estimator as:

[0104]

[0105] where λ* is the optimal λ with the minimum loss function value on drone j, representing the minimum value of the empirical risk of drone j.

[0106] Note that the larger the τ value is, the more heterogeneous the environment faced by each UAV is; when n→∞ and λ*→0, that is, when each UAV has an infinite number of local observations, federated learning is not carried out, and the training of the personalized model PFRL only depends on local observations; on the other hand, when n→0 and λ*→∞, that is, when the local observations of each UAV are very few, the training of the personalized model PFRL can mainly rely on the global model; when τ→∞ and λ→0, that is, when the local data of each UAV is completely uncorrelated, the training of the personalized model PFRL can only rely on local data samples; on the other hand, when τ→0 and λ→∞, that is, when the local data distributions of all UAVs are the same, the training of the personalized model PFRL can mainly rely on the global model.

[0107] S4. Each agent continuously interacts with the environment through the D3QN algorithm to optimize and update its own network, so that the agent obtains the optimal resource allocation strategy.

[0108] Training the D3QN of the local model: In a time slot, each UAV selects an action according to the current observation of the environment. When the UAV takes an action, the environmental state switches to another state, and the UAV obtains a corresponding reward; in this way, each UAV continuously interacts with the environment, aiming to learn a strategy that can obtain the best decision in any observation state and obtain the maximum long-term reward during the learning process.

[0109] The Dueling Double Deep Q Network (D3QN) algorithm incorporates the idea of the Double DQN algorithm on the basis of the Dueling DQN algorithm. The only difference between it and the Dueling DQN algorithm lies in the way of calculating the target value; in the Dueling DQN algorithm, the calculation method of the target value is:

[0110] y t =r t+1 +γmax a q(s t+1 ,a;α t )

[0111] That is, the action values of all actions in the s t+1 state are obtained by using the target network, and then the target value is calculated based on the optimal action value; due to the maximization operation here, the algorithm has an "overestimation" problem, which affects the accuracy of decision-making; where ω t represents the target network parameters.

[0112] In the D3QN algorithm, the calculation method of the target value is:

[0113] y t =r t+1 +γq(st+1 , argmax a q(s t+1 , a; α e ); θ t )

[0114] That is, the evaluation network is used to obtain the action corresponding to the optimal action value in the s t+1 state, and then the target network is used to calculate the action value of this action, so as to obtain the target value; through the interaction of the two networks, the "overestimation" problem of the algorithm is effectively avoided; where θ e and θ t represent the parameters of the evaluation network and the target network respectively; where the competitive network structure diagram is as Figure 3 shown.

[0115] After one round of local model training, each UAV transmits the weight vector of its local model to the leading UAV for global model aggregation; the global model and local model training of the personalized model are mixed. Since the local model needs to be continuously uploaded and summarized, it will be difficult for the local model to adapt to the specific environment of each specific UAV. Therefore, a personalized model is trained, and this model performs local training without uploading and aggregating the model; the global model is used as regularization to modify the training process.

[0116] Since it is difficult to quantify the appropriate heterogeneity level of the personalized method, the parameter λ is introduced. By dynamically changing the value of λ, the personalization degree of the algorithm is adjusted to better adapt to the heterogeneity level of the environment; the personalized goal to be solved is expressed as:

[0117]

[0118] where, g r represents the global model; the personalized model constructed in this way not only has the adaptability to the current environment where the UAV is located, but also has the generalization of the global model.

[0119] Since each UAV can only observe a part of the entire environmental state, the Q value obtained from local observation is not equal to the Q value obtained from observing the entire environment. Therefore, the optimal operation selected to maximize the Q value is not necessarily the optimal operation across environmental states; therefore, the action selection strategies of all UAVs are integrated to obtain a global model to correct the strategy obtained from local observation.

[0120] Specifically, the aggregation of the global model is performed in two steps; when the leading UAV obtains the local model uploaded to the UAV, it will first perform parameter averaging to obtain a preliminary local model, which is defined as:

[0121]

[0122] Among them, P j represents the amount of training data of drone j, and represents the total amount of training data of all drones; due to the different specific environments faced by different drones, their models are heterogeneous, and the generalization effect of the global model obtained by directly averaging the parameters cannot be guaranteed; therefore, after averaging the parameters, the second aggregation of the global model is performed.

[0123] To enable the global model to better generalize, the global model is modified using ensemble distillation; using the initial global model as the student model, and the local models uploaded by each drone as the teacher models. First, convert the Q-value output of each teacher model into a selection behavior strategy of a Boltzmann distribution, which is expressed as:

[0124]

[0125] Among them, T em is the distillation temperature parameter, and the higher the value of T em indicates that the probability distribution of different actions taken by the current policy is softer; use the parameter T em to control the importance of each action and generate an appropriate Softmax ensemble of different actions, so that the student model can learn more knowledge from the teacher models; represents the local model uploaded by drone j of the policy network.

[0126] To minimize the gap between the policy of the preliminary student global model and the teacher network , the loss function is defined as the KL divergence between the student model and the policy corresponding to each teacher network ; the loss function is expressed as:

[0127]

[0128] Among them, represents the policy network of the teacher model, represents the local model uploaded by drone j of the policy network.

[0129] Update with the loss function between different teacher models and the student model is expressed as:

[0130]

[0131] Due to the time-varying characteristics of the multi-UAV-assisted wireless network, the complexity of UAV deployment and resource allocation has become a real challenge. In the personalized federated deep reinforcement learning (PFDRL) scheme, deep reinforcement learning is used to train local models and personalized models on UAVs, and a two-level parameter aggregation scheme is adopted on the leading UAV to form a global model. The personalized model can adapt to a specific environment while leveraging the generalization of the global model to accelerate learning convergence.

[0132] In this embodiment, the communication environment is initialized by simulating the communication environment as a wireless communication scenario within a range of 1500m * 1500m, where ground users are randomly distributed in this area. It is assumed that all UAVs fly at a fixed altitude H = 150m.

[0133] The noise power is set to -80dBm, the carrier frequency f = 2GHz, the available bandwidth is 100MHz, and the average additional losses of the Los and NLoS links are η LoS = 1, η NLoS = 20.

[0134] The network parameters are initialized. The D3QN algorithm network has a two-layer fully connected network, and the number of neurons in the two hidden layers is 64 and 128 respectively. The network learning rate is set to 0.00001, and the discount rate is 0.9.

[0135] The number of training rounds T for the agent is set to 10000, and the number of training steps step per round is set to 100 steps. The neural network parameters are optimized using the Adma optimizer every 50 steps. The reward situation is recorded according to the training process of the agent. The agent continuously optimizes its own strategy according to the proposed algorithm, and finally obtains the optimal resource allocation scheme. Finally, the trained model is applied to the actual scenario, and users can independently select the optimal resource allocation scheme.

[0136] In the communication scenario of UAV joint deployment, traditional reinforcement learning methods may lead to a huge state space even for a medium number of UAVs and users. Since the observations and actions of a specific UAV will affect the observations and actions of other UAVs, it is unrealistic to obtain the transition probability and observation function. In addition, the huge state space and action space in the Markov decision process will result in a large amount of local data being required for model training.

[0137] Therefore, sending the local data of all UAVs to the leading UAV is quite expensive, which makes it difficult to apply traditional centralized DRL methods. In the fully decentralized mode, it is difficult to achieve the global optimal solution due to the lack of information sharing among UAVs. Federated learning (FL), as a privacy-preserving collaborative learning framework, can achieve information sharing among UAVs while protecting user privacy to a certain extent.

[0138] Therefore, in this embodiment, DRL is incorporated into the FL framework to solve the above problems; in addition, since the training data in each UAV policy network consists of the observations, actions, and rewards obtained by itself from the environment, that is, for UAV j, the input of its local DRL model is its observations, rewards, and the relevant Q-values in its experience replay pool.

[0139] Due to the non-uniformity of user location distribution and the variability of user service requests, the policies that each UAV needs to learn are different; therefore, the training data of a single UAV may be similar but actually different, which will lead to the heterogeneity of the UAV policy network; obviously, the personalized learning method can effectively improve the system performance.

[0140] In summary, this embodiment proposes an adaptive and flexible joint UAV deployment and resource allocation scheme, which uses personalized federated deep reinforcement learning (called PFDRL), aiming to maximize the long-term network throughput while protecting user privacy and adapting to the time-varying network state; in the federated learning framework of PFDRL, the deep reinforcement learning (DRL) algorithm is adopted to enable the UAV to make real-time decisions on resource allocation and position adjustment based on local observations, while achieving the global optimal solution.

[0141] In addition to the above embodiments, the present invention may have other implementation manners. All technical solutions formed by equivalent replacement or equivalent transformation fall within the protection scope required by the present invention.

Claims

1. A deep reinforcement learning method for joint UAV deployment and resource allocation, characterized in that: It includes the following steps: S1. Establish a communication network model, initialize the communication environment, the number of drones and the number of users; S2. Determine the optimization objective and constraints according to the signal-to-noise ratio of the users and the system capacity; S3. Convert the optimization problem into a Markov decision process, and use the deep reinforcement learning method to determine the agent, the state space, the action space and the reward function; S4. Each agent continuously interacts with the environment through the D3QN algorithm to optimize and update its own network, so that the agent obtains the optimal resource allocation strategy.

2. The deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 1, wherein: In the step S1, the communication network model includes a leading drone, M drones and N ground users. Communication is carried out between the leading drone and other drones. The deployment and resource allocation of the drones are executed in a time slot manner; the user positions and requested services are fixed in specific time slots, and a user communicates with only one drone in one time slot; it is assumed that only the users within the beam coverage area of the drone communicate with the drone: Among them, represents the horizontal distance between user i and UAV j, H represents the height of the UAV, and θ j represents the half-power beamwidth of UAV j.

3. The deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 2, wherein: In the step S1, the service request of user i within a time slot is defined as wherein represents the amount of data to be transmitted to the UAV, represents the transmission delay constraint of this service request; all UAVs share the spectrum resources of the system; a binary variable is defined to indicate whether UAV j uses sub-channel k to serve user i at time slot t. If so, otherwise, that is, user i can only be served by one UAV using a sub-channel at time slot t.

4. A deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 1, characterized in that: In the step S2, determine the interference signal received by the drone and calculate the signal-to-noise ratio information; the drone communication link follows a probabilistic path loss model, where the line-of-sight LoS and non-line-of-sight NLoS links have different occurrence probabilities. In time slot t, the probability that there is a LoS path between drone j and user i is: where α and β represent environment-related parameters; the probability of the existence of an NLoS link is denotes the data size to be transmitted to the UAV; The signal-to-noise ratio between drone j and user i on subchannel k is: Among them, represents the interference of UAV j. Only the ground users repeatedly covered by multiple UAVs will interfere with the remaining UAVs that are not communicating with this ground user. Therefore, the transmission rate between UAV j and the user in sub-channel k time slot is expressed as 5. A deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 1, characterized in that: In the step S2, determine that the optimization objective is to maximize the overall throughput of the system by optimizing the resource allocation strategy. The problem is formulated as maximizing the overall throughput of the system, which is defined as: The constraints are as follows: Among them, and α = {α1, α2,..., α M} represent access control, UAV position, and beamwidth of each UAV, respectively; Constraint (a) means that a user can only access one UAV on one sub-channel in a time slot; Constraint (b) means that the dynamic quality-of-service requirements of the user are satisfied by the UAV it is served by; Constraint (c) means that the served user is within the coverage area of the relevant UAV; Constraints (d) and (e) respectively constrain the beamwidth of the UAV antenna and the safety distance between UAVs.

6. The deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 5, characterized in that: In the step S3, considering the time-varying constraints and the long-term optimization objective, set the optimization problem as a sequential decision problem, and the network environment in future time slots is unknown; since each drone can only use the local observation information within its coverage area to adjust its position, beam width and access control to maximize the network throughput, model the problem as a decentralized partially observable Markov decision process, where: Agent: Take drone j (1 ≤ j ≤ J) as the agent. Each agent independently updates its policy and selects the subcarrier and transmission power by itself; State space: defined as where respectively represent the current position of the UAV and the user's position, represents the current beam width of the UAV; Action space: The action selected by UAV j at time slot t is a j , where and represent the beamwidth and access control parameter of UAV j respectively, represents the movement of UAV j; Reward function: When the agent executes an action and the constraint conditions (a), (b), and (c) are restricted, a reward value will be obtained; according to the observation exchange range defined by the state, the reward functions of any primary user or secondary user are defined respectively: If the drone does not meet the safety distance requirement, the drone will be punished by ξ1; if the associated user does not meet the dynamic quality of service requirement, the drone will be punished by ξ2; otherwise the reward is defined as the throughput.

7. A deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 1, characterized in that: In the step S3, deep learning is incorporated into the federated learning framework, and an integrated distillation and dynamic regularization mechanism is added to the framework to form a federated deep reinforcement learning framework PFRL with integrated distillation and dynamic regularization; the personalized model PFRL of the drone j is optimized based on the global objective and the local loss function, the global objective is to aggregate the features of the local models of each drone, and the local loss function F j (α j ) is used to extract features from local data; Incorporate the global objective as a regularization term, and represent the personalized model PFRL optimization problem of each drone j ∈ M as: Among them, the parameter λ controls the similarity between the personalized model PFRL and the global model. When λ = 0, the personalized model PFRL degenerates into a local model. The larger λ is, the more similar the personalized model PFRL is to the global model; the larger the value of τ is, the more heterogeneous the environment faced by each UAV is; when n → ∞ and λ* → 0, that is, when each UAV has an infinite number of local observations, no federated learning is performed, and the training of the personalized model PFRL only depends on local observations; when n → 0 and λ* → ∞, the training of the personalized model PFRL depends on the global model; when τ → ∞ and λ → 0, that is, when the local data of each UAV is completely uncorrelated, the training of the personalized model PFRL can only depend on local data samples; when τ → 0 and λ → ∞, that is, when the local data distributions of all UAVs are the same, the training of the personalized model PFRL depends on the global model.

8. A deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 1, characterized in that: In step S4, within a time slot, each UAV selects an action according to the current observation of the environment. When the UAV takes an action, the environmental state switches to another state, and the UAV obtains a corresponding reward. In the D3QN algorithm, the calculation method of the target value is as follows: y t = r t+1 + γq(s t+1 , argmax a q(s t+1 , a; α e ); α t ) That is, the optimal action corresponding to the action value in the s state is obtained by using the evaluation network, and then the action value of this action is calculated by using the target network to obtain the target value; where θ t+1 and α e respectively represent the parameters of the evaluation network and the target network. t ​ 9. A deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 8, characterized in that: In step S4, after a round of local model training, each UAV transmits the weight vector of its local model to the leading UAV for global model aggregation. Train a personalized model that performs local training without uploading and aggregating the model. Use the global model as regularization to modify the training process. Introduce the parameter λ, and adjust the personalization degree of the algorithm by dynamically changing the value of λ. The personalized objective to be solved is expressed as: Among them, g r represents the global model; Integrate the action selection strategies of all UAVs to obtain a global model and correct the strategy obtained from local observations. When the leading UAV obtains the local model and uploads it to the UAV, it will first perform parameter averaging to obtain a preliminary local model, which is defined as: Among them, P j represents the amount of training data of drone j, represents the total amount of training data of all drones; after averaging the parameters, the second aggregation of the global model is performed; the global model is modified using set distillation; the issued global model is used as the student model, and the local models uploaded by each drone are used as the teacher models. First, the Q-value output of each teacher model is converted into a selection behavior strategy of a Boltzman distribution, which is expressed as: Among them, T em is the distillation temperature parameter, and the higher the value of T em , the softer the probability distribution of different actions taken by the current policy; represents the local model uploaded by drone j 's policy network.

10. A deep reinforcement learning method for joint UAV deployment and resource allocation according to claim 9, characterized in that: In the step S4, in order to minimize the strategy of the preliminary student global model and the teacher network the loss function is defined as the KL divergence between the strategy of the student model and the strategy corresponding to each teacher network The loss function is expressed as: Among them, represents the policy network of the teacher model, represents the local model uploaded by drone j 's policy network; Update using the loss function between different teacher models and student models Expressed as:

Images