Unmanned aerial vehicle resource allocation optimization method and system based on deep reinforcement learning

Abstract

The invention provides an unmanned aerial vehicle resource allocation optimization method and system based on deep reinforcement learning, and the method comprises the steps: carrying out the modeling of an application scene, calculating the throughput of a whole model, taking the maximization of the throughput as a target function, and formulating a joint resource optimization problem of path trajectory optimization and power allocation optimization; clustering the user terminals by using a clustering algorithm, and considering the user capacity limitation of the unmanned aerial vehicle in the clustering process; constructing and training a deep reinforcement learning model, introducing a dual architecture into the deep reinforcement learning model to obtain a value function and a dominant function, introducing a target network in the training process, and cooperatively updating target network parameters and main network parameters; and solving a joint optimization problem by using the trained deep reinforcement learning model to obtain a final path trajectory optimization and power distribution optimization strategy. According to the invention, the UAV flight path and power resource allocation can be intelligently realized, and the throughput of the system and the QoS satisfaction of the user are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication, and particularly relates to a method and system for optimizing UAV resource allocation based on deep reinforcement learning. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development and iteration of 5G technology, the unmanned aerial vehicle (UAV) communication system has gradually become an important part of future communication networks. Especially in 5G and future 6G networks, UAV communication, as a key technology, has received extensive attention. Due to its flexibility, low cost, and high mobility, UAVs have shown great application potential in multiple fields such as emergency communication and the Internet of Things. Although the UAV communication system has broad application prospects, its resource allocation problem still faces many challenges. Among them, spectrum management is one of the key issues. Due to the continuous increase in the number of UAVs, the demand for its spectrum resources has increased sharply. How to efficiently allocate limited wireless spectra to avoid interference is the focus of current research.

[0004] Trotta et al. proposed an aerial grid network deployment scheme based on game theory and swarm movement methods. This scheme considered the degree of cooperation among UAVs and formulated corresponding UAV charging scheduling strategies. Lai et al. proposed a scheme for adaptively and fairly deploying UAVs to solve the imbalance problem in the association between UAVs and ground UAVs. Li et al. proposed a UAV-assisted cooperative transmission network, which used the alternative optimization (AO) and Dinkelbach methods to reformulate the original problem into an easy-to-handle form. By jointly optimizing UAV-user association, UAV movement trajectories, and resource allocation, the network energy efficiency was maximized. These schemes used heuristic algorithms, and these schemes needed to use global channel state information, increasing the implementation difficulty of the algorithms in actual applications in different environments. Liu et al. used convex optimization methods to find the optimal hovering positions and power allocations of NOMA-enhanced single UAVs. Although the NOMA technology was used to save spectrum resources, the user movement scenario and multi-UAV scenario were not considered. During the deployment of multiple UAVs, the ability to quickly adapt to complex environments is crucial for achieving effective distributed control of UAVs and efficiently completing communication coverage tasks.

[0005] In recent years, DRL (Deep Reinforcement Learning), as an advanced technology that combines deep learning and reinforcement learning, has performed excellently in solving complex decision-making problems. DRL combines DL (Deep Learning) and RL (Reinforcement Learning) and has made significant progress in many fields in recent years. RL aims to learn decision-making strategies by interacting with the environment, while DL can extract meaningful features from raw data. The application of DRL in wireless network resource allocation has attracted increasing research attention. Sun et al. proposed a block selection method based on 3D convolutional DRL and an energy-aware DRL-based block search algorithm for UAV path planning. Liu Jiangbin et al. proposed a new hybrid network architecture that uses unmanned aerial vehicles as aerial mobile base stations and flies periodically along the cell edge. The movement of users is ignored in these models, resulting in the loss of the function of UAVs to track and better serve mobile users. Liu et al. proposed a framework based on decentralized deep reinforcement learning (DRL) to control each UAV in a distributed manner to maximize the time-averaged coverage fraction achieved by all UAVs in a task and minimize the total energy consumption. However, it only considers the flight of UAVs at a fixed altitude to complete the service for users. In actual UAV communication scenarios, there are line-of-sight and non-line-of-sight situations. In order to increase the total throughput of UAV services, it is also very important for UAVs to be able to autonomously control their flight altitude.

[0006] Generally speaking, UAV communication networks have advantages such as flexible deployment, dynamic adjustment, and wide coverage. However, in practical applications, resource allocation problems of UAVs, including path planning and power allocation, still face huge challenges. Summary of the Invention

[0007] To solve the above problems, the present invention proposes a method and system for optimizing UAV resource allocation based on deep reinforcement learning. The present invention can intelligently realize the flight trajectory of UAVs and allocate power resources, greatly improving the throughput of the system and the QoS satisfaction of users.

[0008] According to some embodiments, the present invention adopts the following technical solutions:

[0009] A method for optimizing UAV resource allocation based on deep reinforcement learning, comprising the following steps:

[0010] Model the application scenario, calculate the signal-to-interference-plus-noise ratio and transmission rate according to the communication channel parameters of each UAV and user terminal, and then calculate the throughput of the entire model. Taking the maximization of throughput as the objective function, formulate a joint resource optimization problem for path trajectory optimization and power allocation optimization;

[0011] Use a clustering algorithm to cluster user terminals, and consider the user capacity limit of UAVs during the clustering process;

[0012] Model the joint optimization problem as a Markov decision process, design the state space and action space of each agent, and design a reward function based on the objective function;

[0013] Construct and train a deep reinforcement learning model. The deep reinforcement learning model introduces a dual architecture to obtain a value function and an advantage function, and during the training process, a target network is introduced, and the parameters of the target network and the main network are updated collaboratively;

[0014] Use the trained deep reinforcement learning model to solve the joint optimization problem to obtain the final path trajectory optimization and power allocation optimization strategy.

[0015] As an alternative implementation, the process of modeling the application scenario includes:

[0016] Construct an application scenario model according to the number of drones and user terminals in the application scenario, their initial position relationships, and the total drone service time. In the application scenario model, the initial positions of the drones are fixed, the user terminals are randomly distributed within the coverage of the scenario, and the user terminals can move freely within the scenario;

[0017] The non-orthogonal frequency division multiplexing multiple access technology is used for communication between the drones and the user terminals.

[0018] As an alternative implementation, the process of calculating the signal-to-interference-plus-noise ratio (SINR) and transmission rate according to the communication channel parameters of each drone and user terminal includes: calculating the path loss according to the number of drones and user terminals and their initial position relationships. The path loss includes the path loss in the line-of-sight link case and the path loss in the non-line-of-sight link case, and then calculating the average path loss when each user communicates through the drone;

[0019] Considering the average path loss of communication, channel gain, inter-cluster interference, and intra-cluster interference, determine the signals received by each user. Use the channel interference noise ratio as the criterion for determining the decoding order to decode the signals, calculate the SINR and transmission rate, and determine the throughput of the entire model according to the SINR and transmission rate.

[0020] As an alternative implementation, the process of formulating the joint resource optimization problem of path trajectory optimization and power allocation optimization includes taking the maximization of throughput within the entire model setting time as the objective function, and the joint resource optimization problem includes the following constraints: the three-dimensional position constraint of the unmanned aerial vehicle (UAV), which must be above the service area, within the operable altitude range, avoid collisions or interference with other UAVs, ensure normal operation between it and other communication devices outside the cellular network, and prevent collisions between UAVs; each user terminal must be served by only one UAV at the same time, the transmission power and consumed power constraints of the UAV, and the minimum quality of service constraint that each user needs to meet for communication.

[0021] As an alternative implementation, the process of clustering user terminals using a clustering algorithm and considering the user capacity limit of the UAV during the clustering process includes: using the K-Means algorithm to cluster user terminals, taking the positions of each user terminal as the observation set and inputting them into one of the UAVs in the UAV swarm, and then broadcasting the user clustering to all UAVs through the control channel. First, place the cluster centers at arbitrary positions, then assign each user to the nearest cluster, and recalculate the centroid of each cluster. When the number of users in any cluster exceeds the load capacity of the UAV, if there are extra users in a certain user cluster, the farthest user will be removed and assigned to another nearest cluster, and repeat the above operations until the users in all clusters meet the load capacity or the maximum number of users of the UAV.

[0022] As an alternative implementation, the process of modeling the joint optimization problem as a Markov decision process, designing the state space and action space of each agent, and designing the reward function based on the objective function includes: the local state information consists of the position of the UAV, the channel gain of the user terminal, the positions of the remaining UAVs, and the inter-cluster interference;

[0023] The actions are designed as a set of discrete flight actions of the UAV and discrete powers. The agent selects actions from the action space through a policy. The flight actions of the UAV include flying horizontally left, horizontally right, horizontally forward, horizontally backward, vertically upward, vertically downward, and hovering;

[0024] When the UAV is about to fly to the edge of the service area, force its flight action to avoid flying out of the service area.

[0025] As an alternative implementation, in the process of introducing a dual architecture into the deep reinforcement learning model to obtain the value function and the advantage function, the value function is used to measure the goodness or badness of being in a certain state s t at that time, and the advantage function A(s t , a t ) describes being in state st Under the following circumstances, compared with other possible actions, action a t has an advantage; and the last layer of the deep reinforcement learning model is divided into two independent layers, with parameters β and φ respectively. β is used to represent the neuron parameters of the value function, and φ represents the neuron parameters of the advantage function for each action.

[0026] As an alternative implementation, during the training process of the deep reinforcement learning model, a target network is introduced, and the process of jointly updating the target network parameters and the main network parameters includes: during the training process of the deep reinforcement learning model, a target network with a parameter set of θ - is introduced. By selecting an action to maximize the Q value of the target network, the objective function is modified to:

[0027]

[0028] y t is the objective function at time t, θ - is the parameter set, Q(s t ,a t ; θ) is the value function, r t is the reward at time t, s t is the local observation state at time t, a t is the action at time t, γ is the discount factor; after a certain number of time steps, the target network parameter set θ - is updated by directly synchronously copying the main network parameter set θ.

[0029] As an alternative implementation, during the training process, each drone selects its action as a separate agent and does not know the action selections of other agents. Multiple drones are allowed to connect to the same neural network to enable sharing of experiences among agents. When the drones provide services, the parameters of the neural network are copied to each agent;

[0030] The agent can choose to locally update its own neural network or only use the neural network for prediction without updating the parameters. Each agent regards the positions of other agents as part of the environment when making decisions, and a certain distance is maintained between agents.

[0031] A drone resource allocation optimization system based on deep reinforcement learning, comprising:

[0032] A modeling module, configured to model the application scenario, calculate the signal-to-interference-plus-noise ratio and transmission rate according to the communication channel parameters of each drone and user terminal, and then calculate the throughput of the entire model. Taking the throughput maximization as the objective function, a joint resource optimization problem of path trajectory optimization and power allocation optimization is formulated;

[0033] The clustering module is configured to cluster user terminals using a clustering algorithm and consider the user capacity limit of the UAV during the clustering process;

[0034] The design module is configured to model the joint optimization problem as a Markov decision process, design the state space and action space of each agent, and design a reward function based on the objective function;

[0035] The optimization module is configured to construct and train a deep reinforcement learning model. The deep reinforcement learning model introduces a dual architecture to obtain a value function and an advantage function, and introduces a target network during the training process, and the target network parameters and the main network parameters are updated collaboratively; use the trained deep reinforcement learning model to solve the joint optimization problem to obtain the final path trajectory optimization and power allocation optimization strategy.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] The present invention uses Dueling DQN to optimize the UAV flight path and power allocation in the UAV communication system, thereby maximizing the system throughput. And by introducing user clustering through the K-Means algorithm, the UAV can learn the optimal allocation strategy under the cooperation relationship, and can intelligently realize the UAV flight trajectory and allocate power resources, greatly improving the system throughput and the QoS satisfaction of users.

[0038] The present invention aims to maximize the system throughput. After satisfying the QoS requirements of users and power constraints, it completes training optimization in continuous interaction with the environment, enabling each UAV to perform trajectory planning and power allocation distributively. It provides a highly feasible solution for the application of DRL algorithms in UAV-assisted communication, and thus has good application prospects.

[0039] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The specification drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0041] Figure 1 It is a schematic diagram of a system model of an embodiment;

[0042] Figure 2 It is an implementation diagram of a Dueling DQN algorithm of an embodiment;

[0043] Figure 3It is an implementation diagram of an improved K-Means algorithm in an embodiment;

[0044] Figure 4 It is a diagram of a Dueling DQN network model in an embodiment;

[0045] Figure 5 It is the convergence of the loss function when a method in an embodiment is trained at different learning rates;

[0046] Figure 6 It is the UAV trajectory and the movement of the user derived from a method in an embodiment;

[0047] Figure 7 It is a comparison diagram of the execution effects of different methods in an embodiment;

[0048] Figure 8 It is a change diagram of the throughput of the entire system when a method in an embodiment uses different numbers of UAVs. Detailed implementation manner

[0049] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further descriptions of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0051] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0053] Embodiment 1

[0054] An optimization method for UAV resource allocation based on deep reinforcement learning includes the following steps:

[0055] Step 1: Model the application scenario adopted and formulate a joint resource optimization problem of UAV path trajectory optimization and power allocation optimization.

[0056] Such as Figure 1As shown in the figure, assume that there are K drones and M terminal users in the scenario, and the total drone service time is T. The initial positions of the drones are fixed, and the terminal users are randomly distributed within the coverage of the scenario and can move freely within the scenario. Let K = {1,..., K} and M = {1,..., M} represent all the drones and terminal users respectively. The initial position of the k-th drone is represented by and its position during the service process can be expressed as u k (t) = (x k (t), y k (t), z k (t)) ∈ R 3 , where t ∈ [0, T], and we assume that the flight speed of the drone is constant at v. Assume that the user's height is 0, so the position of the user can be expressed as l m (t) = (x m (t), y m (t), 0) ∈ R 3 . Therefore, at time slot t, the distance between the m-th user and the k-th drone during communication is:

[0057]

[0058] Both the distance between the drone and the user and the communication channel between the drone and the user adopt 3GPP specifications Release 15. The path loss is divided into line-of-sight (LoS) and non-line-of-sight (NLoS) links. The path loss in the case of the LoS link is shown in equation (2):

[0059]

[0060] where z k (t) represents the flight height of the k-th drone, and f c represents the carrier frequency.

[0061] In the case of the NLoS link, the path loss is as follows. For the sake of convenient representation, we first introduce an intermediate auxiliary path loss, which can be expressed as:

[0062]

[0063] From equation (3), the path loss expression in the case of the NLoS link can be obtained as:

[0064] PL NLoS (t) = max{PL LoS (t), PL′(t)} (4)

[0065] We define the horizontal distance between the drone and the user as:

[0066]

[0067] Therefore, the probability of path loss under the line-of-sight link can be expressed as:

[0068]

[0069] where d 0 = max[294.05·log 10 z k (t) - 432.94, 18], p 1 = 233.98·log 10 h u (t) - 0.95. Then the probability of path loss under the non-line-of-sight link can be expressed as:

[0070] P NLoS (t) = 1 - P LoS (t) (7)

[0071] Therefore, the average path loss when the m-th user communicates through the k-th drone at time slot t can be expressed as:

[0072]

[0073] Considering the impact of small-scale fading, the channel gain when the m-th user communicates through the k-th drone at time slot t can be expressed as:

[0074]

[0075] where, is expressed as the fading coefficient.

[0076] We use to represent the user-drone association indicator function. When the k-th drone serves the m-th user otherwise Therefore, the superimposed transmission signal x k (t) of the drone can be expressed as:

[0077]

[0078] where represents the power allocated by the drone to user m, then represents the transmission signal from drone k to user m.

[0079] Since we have clustered all users according to the number of UAVs, the interference received by user m consists of two parts: inter-cluster interference and intra-cluster interference. The inter-cluster interference is caused by other UAVs using the same resource block when user m communicates with UAV k. We use to represent the inter-cluster interference. Therefore, the composition of the inter-cluster interference can be expressed as:

[0080]

[0081] Among them, represents the channel gain between user m and other UAV s, and P s (t) represents the total power consumption of other UAV s, which is specifically expressed as:

[0082]

[0083] The intra-cluster interference is the interference received from other users within the cluster when user m communicates with UAV k, which is represented by :

[0084]

[0085] Among them, M' represents the total number of users within a cluster, represents the channel gain between intra-cluster user n and UAV k, represents the power consumption allocated by UAV k to intra-cluster user n.

[0086] Therefore, through equations (8), (9), (11), and (13), the signal received by user m can be obtained as:

[0087]

[0088] Among them represents Additive Gaussian White Noise (AWGN).

[0089] In this embodiment, SIC is used to decode the signal received by user m. In order to implement SIC, we use the Channel to Interference plus Noise Ratio (CINR) as the criterion for determining the decoding order, which is expressed as:

[0090]

[0091] Among them, i (i ∈ M) represents the index sorted according to the CINR value.

[0092] Without Loss of Generality (w.l.o.g.), in this embodiment, the following sorting is performed according to the CINR value from largest to smallest where and represent the strongest channel and the weakest channel users respectively. According to the principle of SIC technology, within the same cluster, we allocate higher power to users with lower CINR. Therefore, the power allocated to each user can be expressed in size as After sorting, we perform step-by-step decoding from largest to smallest according to the allocated power. First, start decoding the signal with power , and regard the signals of users with power less than as interference, and then start normal decoding. After successful decoding, remove the signal with power from the superimposed signal, and the next step is to decode the signal with power , and so on. Therefore, the Signal to Interference & Noise Ratio (SINR) of user i can be expressed as:

[0093]

[0094] So, at time slot t, when the m-th user communicates through the k-th UAV, assuming that the index of its CINR at this time is i, the transmission rate of the m-th user can be expressed as:

[0095]

[0096] where B represents the bandwidth of the k-th UAV.

[0097] Therefore, at time slot t, the transmission rates of all UAVs can be expressed as:

[0098]

[0099] Finally, the throughput of the entire system during the T period can be expressed as:

[0100]

[0101] Our goal is to maximize the total throughput of the entire system by optimizing the UAV trajectory and power allocation strategy. Among them, the user-UAV association indication coefficient is used to represent the user association situation, U = {u k (t), k ∈ K, 0 ≤ t ≤ T} represents the flight position of the UAV during the service time 0 ≤ t ≤ T, and P = {p mThe power allocation strategy of the UAV for each user is represented by {P(t), m ∈ M, 0 ≤ t ≤ T}. Therefore, the optimization objective can be written as:

[0102]

[0103] Constraints C1 - C3 represent the three - dimensional position constraints of the UAV. It must be above the service area, within the operable altitude range, avoid collisions or interference with other UAVs, and ensure normal operation between it and other communication devices outside the cellular network. Constraint C4 ensures that there are no collisions between UAVs. Constraint C5 ensures that each user can only be served by one UAV at the same time, avoiding multiple UAVs serving the same user simultaneously. Constraint C6 limits the transmission power P k of each UAV, ensuring that the power consumed does not exceed the upper limit, thus preventing excessive energy consumption. Constraint C7 requires that each user needs to meet the minimum QoS for communication. The optimization objective has been proven to be an NP - hard problem, and due to the mobility of UAVs and users, this problem is highly dynamic. Therefore, traditional convex optimization methods are difficult to effectively solve this highly dynamic and multi - constraint problem. Therefore, this embodiment will study a data - driven deep reinforcement learning algorithm to solve this optimization problem.

[0104] Step 2. Considering that the user capacity of a single UAV is limited, the K - Means algorithm is used to achieve user clustering.

[0105] The K - Means algorithm is a heuristic clustering algorithm that has been proven to perform very well in user clustering in wireless communication. K - Means has a low computational complexity and can find the clustering result in a timely manner in UAV communication.

[0106] In the optimization problem modeling of this embodiment, the location of user m can be input as an observation set into one of the UAVs in the UAV swarm, and then the user clustering can be broadcast to all UAVs through the control channel. The traditional K - Means algorithm initially places the clustering centers at arbitrary positions, then assigns each user to the nearest cluster, and recalculates the centroid of each cluster. The algorithm iterates this process until the centroids of all clusters no longer change. However, the traditional K - Means algorithm does not consider that the user capacity of a single UAV is limited. Therefore, when the number of users in any cluster exceeds the load capacity of the UAV, the K - Means algorithm needs to be supplemented and modified.

[0107] As Figure 3As shown in the figure, assuming that there are redundant users in a certain user cluster, the farthest user will be removed and assigned to another nearest cluster. And this operation is repeated until the users in all clusters meet the load capacity or the maximum number of users of the UAV.

[0108] Step 3. Design the key elements of DRL, including the state space, action space, reward function, and network model.

[0109] In the scenario of the UAV communication system, the optimization goal is to maximize the throughput of the entire system by optimizing the UAV trajectory and power allocation. Since the Dueling DQN model needs to calculate the Q value of the action according to the input state information, and it is difficult to obtain the position information during the random movement of users, the channel gain is used to represent the state of the user and also as the basis for power allocation. In this model, the local state information consists of the position u k (t) of the UAV and the channel gain of the user the positions u i (t) (i≠k) of the remaining UAVs, and the inter-cluster interference . At time t, the local observation state is expressed as:

[0110]

[0111] In the joint trajectory and power optimization problem, the action is designed as a set of discrete UAV movements A t and discrete power P t d . At time t, the agent selects an action from the action space through the policy π:

[0112] a t = [A t , P t d (22)

[0113] where A t represents the flight action of the UAV, including seven flight actions: horizontal left, horizontal right, horizontal forward, horizontal backward, vertical up, vertical down, and hover. It can be expressed as:

[0114]

[0115] where Δ = vτ, where v represents the flight speed of the UAV and τ represents the unit time. And when the UAV is about to fly to the edge of the service area, we force its flight action to avoid flying out of the service area. The discrete transmit power P t d of the UAV is quantized into L power levels for convenient control:

[0116] P t d ∈ {p 1 , p 2 ,..., p L [[ID=10}]}(24)

[0117] During the training process, a decreasing ε-greedy strategy is adopted as the exploration strategy to guide the agent to select actions. This strategy enables the agent to use the policy function with a probability of ε, and randomly select an action from the action space with a probability of 1 - ε.

[0118] Through the above design, the state and action space of each agent can be obtained.

[0119] During the reinforcement learning process, the agent will make a decision to maximize the reward based on the feedback of the UAV communication system scenario. Therefore, the design of the reward often aligns with the optimization goal. And, in order to meet the constraint conditions, the following reward function is adopted:

[0120]

[0121] where R(t) is the total transmission rate and λ is the penalty coefficient. When the constraint C7 is not satisfied, the penalty coefficient will increase, and when increasing it cannot reduce the number of steps that do not meet the QoS requirements, the increase of the penalty coefficient will stop.

[0122] Step 4. Based on the method provided in this embodiment (i.e., the UAV resource allocation algorithm based on the multi-agent Dueling DQN combined with the K-Means algorithm, hereinafter referred to as the MDDRA algorithm, mainly using the Dueling DQN network architecture), through multiple rounds of training, continuously optimize the Q-value network to obtain a power allocation strategy and a UAV flight path planning that meet the QoS requirements of the UE and the constraint requirements.

[0123] The operations of the traditional DQN include the online deep Q-learning stage and the offline deep neural network (DNN) construction stage. The online deep Q-learning stage refers to the agent interacting with the environment, collecting experience data, and using the main network to select actions. The offline DNN construction stage is the process of training the main network using the experience data, calculating the target Q-value using the target network, and approximating the Q-value function by training the deep neural network.

[0124] The main network and the target network are mainly used in the offline DNN construction stage, and are used to predict the Q-value and calculate the target Q-value respectively.

[0125] The offline DNN construction stage is used to learn the value function Q(s t , a t; θ). Usually, θ is optimized and updated by minimizing the following loss function:

[0126] L(θ) = E[(y t -Q(s t , a t ; θ)) 2 (26)

[0127] where y t is the target function of the Q-network, γ is the discount factor, θ - is the set of network parameters of the target network, and y t is expressed as:

[0128]

[0129] However, in DQN, when updating the action value, the target network is responsible for both selecting actions and evaluating the Q-values of these actions, resulting in an overestimation problem. Therefore, the Double Deep Q-Network (Double DQN) algorithm is proposed to alleviate the above problem. During the training process, a target network with a parameter set of θ - is introduced, and the Q-value of the target network is maximized by selecting an action. Therefore, the target function in (27) is replaced with the following target function:

[0130]

[0131] After a certain number of time steps, the target network parameter set θ - is updated by directly synchronously copying the main network parameter set θ.

[0132] The Dueling DQN algorithm is a reinforcement learning algorithm improved from the DQN algorithm. As Figure 2 shown, it includes a main network and a target network. In reinforcement learning, for many states, it is unnecessary to estimate the value of each action selection. Therefore, we only need to use the Q-value function considering taking a specific action a t in the state s t . So, we introduce a Dueling Architecture into the neural network to obtain the value function V(s t ) and the advantage function A(s t , a t ). The value function V(s t ) is used to measure the goodness or badness of being in a specific state s t . The advantage function A(s t , a t ) describes, in the state s t , compared with other possible actions, the action at Advantages. The advantage function A(s t , a t ) is expressed as:

[0133] A(s t , a t ) = Q(s t , a t ) - V(s t ) (29)

[0134] Therefore, compared with the traditional DQN, as Figure 4 shown, the difference of Dueling DQN is that the last layer of DQN is divided into two independent layers with parameters β and φ respectively. β is used to represent the neuron parameters of the value function V(s t |θ, β), while φ represents the neuron parameters of the advantage function A(s t , a t |θ, φ) for each action. However, the formula (29) cannot obtain accurate and relatively unique values of V(s t |θ, β) and A(s t , a t |θ, φ). For identifiability, the Q value in Dueling DQN can be expressed as:

[0135]

[0136] where |A| represents the size of the action space. The principle of the Dueling DQN algorithm is as Figure 2 shown.

[0137] In the Dueling DQN model, each UAV acts as an individual agent to select its action and does not know the action selections of other agents. However, during the training process, multiple UAVs are allowed to connect to the same neural network.

[0138] Although the experiences of each agent are different, they can be unified into a standard form and then these experiences are used to train a shared neural network. It can also be regarded that the experiences of each agent can be indirectly learned by other agents through the shared neural network, thereby reducing the training time and increasing the training efficiency. Moreover, the experiences are only shared during the training process. When the UAV provides services, the parameters of the neural network will be copied to each agent. The agent can choose to locally update its own neural network or only use the neural network for prediction without updating the parameters. Although each UAV, as an individual agent, does not know the action selections of other agents. However, when making decisions, the UAV regards the positions of other agents as part of the environment, and the UAVs can maintain a certain distance from each other to avoid excessive inter-cluster interference, thus achieving cooperation to a certain extent.

[0139] The Dueling DQN uses a DNN network as the Q-network, which includes two hidden layers and a Dueling branch consisting of a state-value function branch and an advantage function branch. Finally, the state-value function branch and the advantage function branch are combined to calculate the final Q-value. In addition, the Q-network also includes an input layer with the same size as the state dimension and an output layer corresponding to the Q-values of actions. Except for the output layer, the activation function of other neurons is the ReLU function.

[0140] The following comparative experiments were also conducted in this embodiment:

[0141] Experiments related to the allocation of UAV flight trajectories and power resources based on the Multi-Agent Dueling DQN combined with the K-Means algorithm (MDDRA):

[0142] Computer simulation experiments were carried out to verify the performance of the proposed UAV flight trajectory and power resource allocation algorithm. The scenario of the downlink UAV communication system was simulated, and the communication channels of the system scenario all adopted 3GPP specifications Release 15. Users were randomly distributed in a 500m×500m service area, and the users moved randomly within the service area with a moving speed less than or equal to the maximum moving speed of the users. The UAV was deployed near the boundary of the cellular network at an initial height of 100m at the initial moment. The number of neurons in the first two hidden layers was 64 and 32 respectively, the number of neurons in the value function was 16 and 1, and the number of neurons in the advantage function was 16. In addition, the activation function used was the rectified linear unit, and the loss function selected was the mean squared error. The Adam optimizer was applied when training the neural network. The ε value of the greedy action strategy decreased linearly from 0.9 to 0. Unless otherwise specified, the remaining default environment simulation parameters are listed in Table 1 below.

[0143] Table 1 Simulation environment parameters

[0144] Parameter Value Service area 500m × 500m Number of drones 3 Number of users 6 Flight speed of drones 5m / s Flight altitude of drones [20m, 150m] Maximum moving speed of users 0.5m / s Carrier frequency fc 2GHz Bandwidth B of each resource block 15kHz Maximum transmission power 29dBm

[0145] The software and hardware specifications for simulation experiments and the MDDRA algorithm simulation parameters are shown in Table 2. The simulation environment was implemented using Python, the optimization algorithm and the network model were based on the TensorFlow framework, and GPU acceleration used CUDA.

[0146] Table 2 Experimental platform specifications and software versions

[0147] Parameter Value CPU Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz GPU NVIDIA GeForce RTX 3060 Memory 16GB @ 2666MHz Hard disk 500GB SSD Operating system Microsoft Windows 10 0.0.19045 Python version 3.7.11 TensorFlow version 2.7.1 CUDA version 12.0 Capacity D of the experience replay pool 10000 Discount factor 1 Learning rate 0.001 Update frequency 600 Sampling sample size 128

[0148] The method provided in this embodiment can simultaneously plan the flight trajectory of the UAV and allocate power resources to optimize the energy efficiency of the system. The magnitude of the learning rate in Dueling DQN determines the magnitude of the policy update and is an important parameter affecting the optimization performance. Figure 5 It shows the convergence of the loss function when MDDRA is trained at different learning rates. As can be seen from the figure, too small a learning rate will lead to slow convergence, increase the time to find the optimal value, and it is difficult to obtain the optimal result. While too large a learning rate may cause the model to oscillate near the optimal solution and may even fail to converge. Through simulation experiments, the learning rate of the method provided in this embodiment is 0.001, achieving good convergence performance.

[0149] As Figure 6 shown, the UAV trajectory derived by the method provided in this embodiment and the movement of the user. The user follows a random movement model. Generally, it first chooses to fly on the horizontal plane and then chooses to descend vertically to approach the user. Since each UAV is an agent, when choosing flight actions in the later stage, the data rate of each step during the movement is considered, and each UAV has different selection schemes. Finally, the throughput of the overall system is maximized.

[0150] As Figure 7 shown, by using DQN, DDQN, and Random to verify and compare the advantages of the method provided in this embodiment in optimizing the path planning and power allocation of the downlink UAV communication system. It can be seen that the method provided in this embodiment can have better system throughput than other algorithms through training. Among them, the Random algorithm is used as a benchmark allocation strategy to adapt to scenarios where resource allocation optimization is not carried out in practice.

[0151] As Figure 8 shown, it is a graph of the change in the throughput of the entire system when different numbers of UAVs are used under the method provided in this embodiment. When there is only one UAV, there is a lack of inter-cluster interference in the entire system, resulting in limited growth of the system throughput. Generally, using more UAVs can obtain greater throughput. However, as the number of UAVs increases, the gain of the entire system gradually becomes effective. And, when the scale of UAVs increases more, it is certain that the overall throughput of the UAV communication system will decrease. Therefore, this method has good scalability and can provide good communication services to ground users under different numbers of UAVs according to different scenarios.

[0152] Embodiment 2

[0153] A UAV resource allocation optimization system based on deep reinforcement learning, including:

[0154] A modeling module, configured to model an application scenario, calculate the signal-to-interference-plus-noise ratio (SINR) and transmission rate based on the communication channel parameters of each unmanned aerial vehicle (UAV) and user terminal, and then calculate the throughput of the entire model. Taking the maximization of throughput as the objective function, a joint resource optimization problem of path trajectory optimization and power allocation optimization is formulated.

[0155] A clustering module, configured to cluster user terminals using a clustering algorithm, and consider the user capacity limit of UAVs during the clustering process.

[0156] A design module, configured to model the joint optimization problem as a Markov decision process, design the state space and action space of each agent, and design a reward function based on the objective function.

[0157] An optimization module, configured to construct and train a deep reinforcement learning model. The deep reinforcement learning model introduces a dual architecture to obtain a value function and an advantage function, and in the training process, a target network is introduced, and the parameters of the target network and the main network are updated collaboratively. The trained deep reinforcement learning model is used to solve the joint optimization problem to obtain the final path trajectory optimization and power allocation optimization strategy.

[0158] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0159] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more flows or multiple flows and / or blocks Figure 1 one or more blocks or multiple blocks.

[0160] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions specified in one or more of the processes and / or blocks Figure 1 one or more of the processes and / or blocks Figure 1 specified in the block or blocks.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the processes and / or blocks Figure 1 one or more of the processes and / or blocks Figure 1 specified in the block or blocks.

[0162] The foregoing are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art without creative efforts within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for optimizing UAV resource allocation based on deep reinforcement learning, characterized in that: The following steps are involved: Model the application scenario, calculate the signal-to-interference-noise ratio and transmission rate based on the communication parameters of each drone and user terminal, and then calculate the throughput of the entire model. Take throughput maximization as the objective function, and formulate a joint resource optimization problem of path trajectory optimization and power allocation optimization; The user terminals are clustered using clustering algorithms, and the user capacity limit of drones is considered during the clustering process; Modeling the joint optimization problem as a Markov decision process, designing the state space and action space of each agent, and designing a reward function based on the objective function; Constructing and training a deep reinforcement learning model, wherein the deep reinforcement learning model introduces a dual architecture to obtain a value function and an advantage function, and during the training process, introduces a target network, and the target network parameters and the main network parameters are updated collaboratively; The trained deep reinforcement learning model is used to solve the joint optimization problem and obtain the final path trajectory optimization and power allocation optimization strategy.

2. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that: The process of modeling an application scenario includes: According to the number of drones and user terminals in the application scenario, the initial position relationship, and the total drone service time, an application scenario model is constructed. In the application scenario model, the initial position of the drone is fixed, the user terminals are randomly distributed within the coverage of the scenario, and the user terminals can move freely within the scenario. Non-orthogonal frequency division multiple access technology is used for communication between the drone and the user terminal.

3. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that: The process of calculating the signal to noise ratio and the transmission rate according to the communication parameters of each UAV and the user terminal includes: calculating the path loss according to the number and initial position relationship of the UAV and the user terminal, wherein the path loss includes the path loss in the case of line-of-sight link and the path loss in the case of non-line-of-sight link, and then calculating the average path loss when each user communicates through the UAV; Considering the average path loss, channel gain, inter-cluster interference and intra-cluster interference of communication, the signal received by each user is determined, and the channel interference and noise ratio is used as the criterion for determining the decoding order. The signal is decoded, and the signal-to-interference-noise ratio and transmission rate are calculated. The throughput of the entire model is determined based on the signal-to-interference-noise ratio and transmission rate.

4. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that: The process of formulating the joint resource optimization problem of path trajectory optimization and power allocation optimization includes taking the throughput maximization within the entire model setting time as the objective function, and the joint resource optimization problem includes the following constraints: the three-dimensional position constraint of the drone, which must be above the service area and within the operable altitude range, and avoid collision or interference with other drones, and ensure its normal operation with other communication equipment outside the cellular network, and there will be no collision between drones; each user terminal must be served by only one drone at the same time, the drone's transmission power and power consumption constraints, and the minimum service quality constraints that each user needs to meet in communication.

5. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that: The user terminals are clustered by using a clustering algorithm, and in the clustering process, the user capacity limitation of the UAV is considered, including: using a K-Means algorithm to cluster the user terminals, inputting the position of each user terminal as an observation set into a UAV in the UAV group, and then broadcasting the user clusters to all UAVs through a control channel, first placing the cluster center at an arbitrary position, and then assigning each user to the nearest cluster, and recalculating the centroid of each cluster. When the number of users in any cluster exceeds the load capacity of the UAV, if there are redundant users in a user cluster, the farthest user will be removed and assigned to another nearest cluster, and the above operation will be repeated until the users in all clusters meet the load capacity of the UAV or the maximum number of users.

6. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that: The joint optimization problem is modeled as a Markov decision process, and the state space and action space of each agent are designed. Based on the objective function, the process of designing the reward function includes: the local state information is composed of the position of the UAV and the channel gain of the user terminal, the positions of the other UAVs, and the inter-cluster interference; The action design is a set of discrete UAV flight actions and discrete powers. The agent selects actions from the action space through strategies. The UAV flight actions include horizontal left, horizontal right, horizontal forward, horizontal backward, vertical upward, vertical downward and hovering. When the drone is about to fly to the edge of the service area, its flight action is forced to avoid flying out of the service area.

7. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that: The deep reinforcement learning model introduces a dual architecture to obtain the value function and advantage function. The value function is used to measure the state s. t The degree of goodness when the advantage function A(s t ,a t ) describes the state s t Next, compared to other possible actions, action a t and the last layer of the deep reinforcement learning model is divided into two independent layers with parameters β and φ, β is used to represent the neuron parameters of the value function, and φ represents the neuron parameters of the advantage function of each action.

8. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that: During the training process of the deep reinforcement learning model, the target network is introduced, and the process of collaboratively updating the target network parameters and the main network parameters includes: during the training process of the deep reinforcement learning model, the parameter set θ is introduced. - The target network, by selecting an action to maximize the Q value of the target network, the objective function is modified as follows: y t =r t +γQ(s t+1 ,argmaxQ(s t+1 ,a t+1 ;i);i - ); a t+1 y t is the objective function at time t, θ - is the parameter set, Q(s t ,a t ; θ) is the value function, r t is the reward at time t, s t is the local observation state at time t, a t is the action at time t, γ is the discount factor; after a certain number of time steps, the target network parameter set θ - The update is performed by directly synchronously copying the master network parameter set θ.

9. The method for optimizing UAV resource allocation based on deep reinforcement learning as claimed in claim 1, characterized in that During the training process, each drone selects its actions as a separate agent and is unaware of the action choices of other agents. This allows multiple drones to be connected to the same neural network to share experience between agents. When a drone provides a service, the parameters of the neural network are copied to each agent. The agent can choose to update its own neural network locally, or only use the neural network for prediction without updating parameters. Each agent considers the position of other agents as part of the environment when making decisions, and maintains a certain distance between agents.

10. A drone resource allocation optimization system based on deep reinforcement learning, characterized in that: include: The modeling module is configured to model the application scenario, calculate the signal-to-interference-noise ratio and transmission rate according to the communication parameters of each UAV and user terminal, and then calculate the throughput of the entire model. Taking the throughput maximization as the objective function, a joint resource optimization problem of path trajectory optimization and power allocation optimization is formulated; A clustering module is configured to cluster the user terminals using a clustering algorithm, and in the clustering process, consider the user capacity limit of the drone; A design module is configured to model the joint optimization problem as a Markov decision process, design the state space and action space of each agent, and design a reward function based on the objective function; The optimization module is configured to build and train a deep reinforcement learning model, which introduces a dual architecture to obtain a value function and an advantage function, and introduces a target network during the training process, and the target network parameters and the main network parameters are updated collaboratively; the trained deep reinforcement learning model is used to solve the joint optimization problem to obtain the final path trajectory optimization and power allocation optimization strategy.

Images