A diversity resource scheduling method for relay drones based on improved DQN algorithm

Abstract

This patent discloses a method for relay drone diversity resource scheduling based on an improved DQN algorithm, which includes the following steps: establishing a mathematical model for drone relay communication scenarios with diverse requirements; converting the mathematical model into a Markov decision model; and applying the improved DQN algorithm to the decision model. The disclosed improved DQN algorithm makes three improvements compared to the original DQN algorithm: introducing a dual network structure to alleviate the existing over-estimation problem; adjusting random sampling to an importance sampling strategy, introducing a priority experience replay mechanism, reducing the bias in sample selection, and improving the training effect and convergence of the algorithm; replacing the linear layer in the evaluation network with a noise layer to help the algorithm balance exploration and development and increase the generalization ability of the model. The present invention can achieve higher data throughput and user service levels, and can better adapt to complex and changeable drone relay communication scenarios with diverse requirements.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle relay communication, and in particular relates to a relay unmanned aerial vehicle diversity resource scheduling method based on an improved DQN algorithm. Technical Background

[0002] In traditional communication networks, due to factors such as geographical limitations and infrastructure construction costs, there are problems of insufficient communication coverage or poor communication quality, which is particularly prominent in scenarios such as emergency rescue, large-scale events and remote areas. The emergence of drone relay communication technology fills this gap. Drone relay communication technology is a technology that uses drones as relay communication nodes to forward communication data for users in complex environmental areas, enhancing the reliability and transmission performance of wireless communications in the area. Through the flexible mobility and rapid deployment capabilities of drones, it can provide communication support in a timely and effective manner, providing strong help for people's production, life and emergency rescue.

[0003] With the continuous development and maturity of drone technology, drone relay communication is becoming one of the research hotspots in the field of communications, attracting the attention and investment of many researchers. However, most of the problems they solve ignore the diversity of user needs and the level of user urgency. Therefore, we use drones as relay nodes and carry out work in scenarios that fully consider the diversity of communication needs and the level of user urgency. This is more in line with actual application scenarios and has very important application value and research significance.

[0004] The method disclosed in this patent takes into account the diversity of communication needs and user urgency levels. Although it is closer to the actual application scenario, it often makes the studied problem more difficult. There are three main challenges here: the allocation and management of communication resources in the scenario of diverse communication needs requires ensuring the efficient use of communication resources while meeting diverse needs; after considering the user urgency level, the problem changes from a single-objective optimization problem to a multi-objective optimization problem, which requires trade-offs and compromises between different goals, resulting in a more complex solution space; many algorithms perform well when solving problems in known and static scenarios, but perform poorly for problems in unknown and changing scenarios. These make solving problems more difficult and difficult to handle with traditional optimization techniques. Therefore, it is necessary to design a more adaptive and effective method to dynamically adjust resource scheduling and drone movement trajectories to solve such problems.

[0005] In the service scenario of drone relay communication, the optimization of different goals by controlling drone trajectory and allocating available communication resources has attracted extensive attention from researchers. At present, the mainstream methods commonly used for path planning and resource allocation include decomposition and coordination optimization methods, heuristic methods, and reinforcement learning methods. Among them, the decomposition and tuning method can well reduce the complexity of the problem and simplify the solution process, but it is not suitable for solving our problem. The optimization goals of the problem we studied are interrelated and interfered, and the problem scenario is complex and changing. It is difficult to find a suitable decomposition strategy to ensure that the decomposed sub-problems are relatively independent. The implementation and application of heuristic methods are relatively simple, suitable for fast solution and experience-based problems, but they show relatively weak adaptability in unknown, changing, and complex environments, and are not suitable for solving the problems we studied. In addition to decomposition and tuning methods and heuristic methods, the use of deep reinforcement learning algorithms to optimize drone behavior decisions is a current research hotspot. The relay environment is unknown and changing for drones, and there is also the characteristic of delayed rewards. The reinforcement learning algorithm is good at understanding the environment through continuous interaction with the unknown environment and learning the changing laws of the environment to obtain the maximum long-term reward. Therefore, it is a very appropriate choice to use reinforcement learning algorithms to solve this problem model.

[0006] Deep reinforcement learning is a cutting-edge research field that combines deep learning and reinforcement learning. Its core concept is to learn the optimal behavior strategy through interaction with the environment. DQN is one of the classic algorithms of deep reinforcement learning. It has achieved great success in solving many complex problems. However, the DQN algorithm also has some problems and challenges, such as high dependence on experience replay, selective bias, highly sensitive hyperparameters, overestimation, etc. This patent is based on deep reinforcement learning and noisy neural networks, and discloses a DDQN (NPD-DQN) method with prioritized experience replay of noisy networks, which is used to solve the problem of drone relay communication in scenarios with diverse user needs. Summary of the invention

[0007] This patent proposes the NPD-DQN algorithm based on the original DQN algorithm to solve the problem of drone relay communication in scenarios with diverse user needs. Compared with the traditional DQN algorithm, this method makes the following adjustments: In response to the transition estimation problem of DQN, a dual network structure is introduced, namely the main network and the target network. The main network selects the action with the largest Q value, and the target network evaluates the selected action. In response to the original random sampling strategy of DQN, an importance sampling mechanism is introduced, and the sample priority is updated using TD error. During training, it will be more inclined to select experiences that are more beneficial to the current learning task for replay, and better utilize limited training resources. The linear layer in the evaluation network is replaced by a noise layer to increase the robustness and exploration ability of the model and avoid over-reliance on the current optimal action. The NPD-DQN algorithm has better adaptability, stability and generalization ability, and can better solve problems in drone relay scenarios.

[0008] A relay drone diversity resource scheduling method based on an improved DQN algorithm includes the following steps:

[0009] Step 1: Establish a mathematical model for drone relay communication scenarios with diverse requirements based on actual application scenarios;

[0010] Step 2, modeling the UAV relay communication mathematical model as a Markov decision process model;

[0011] In step 3, the proposed NPD-DQN algorithm is used to solve step 2, and a series of simulation experiments are used to demonstrate the limitations of the NPD-DQN algorithm compared to other methods.

[0012] In step 1, the modeling process of the UAV relay communication scenario with specific diverse communication requirements is divided into the following sub-steps:

[0013] Step 1.1, make specific assumptions about the research scenario: assume that there is a slightly distant base station A and that base station A is capable of helping the overloaded or damaged base station B in the area to complete the communication task; since there are few objects blocking the sky, it is assumed that the signal propagation is line-of-sight link LoS propagation; during the service time, the drone is able to access any user in the area; since it is too difficult to dynamically control the movement of drones in a continuous three-dimensional space, the service area map is rasterized; users occupy communication resources in a non-preemptive manner, and the system can only reclaim the allocated resources after the time slice requested by the user is consumed; the scenario under study does not consider user movement, and the flight altitude of the drone will be limited; each user can obtain his or her own location through GPS and can share it with the drone synchronously; the user's position remains unchanged throughout the service cycle; we will use frequency division multiplexing to connect users to communication, and assume that the channels in the channel set are orthogonal channels; under the setting of the above scenario, define the drone relay service scenario studied in this paper.

[0014] Step 1.2, define and explain the variables and symbols of the drone relay communication scenario with diverse communication requirements:

[0015] The user set is represented as K = {1, 2, 3, ..., k}, where k users are distributed in the communication service area Ψ; the position of the drone in the service period T is defined as In order to present the defined model more clearly to the readers, the physical meanings of the important symbols used in this patent are summarized in Table 1;

[0016] Table 1: Physical meaning of important parameters

[0017]

[0018] Step 1.3, establishing a mathematical model for the relay communication system model;

[0019] Furthermore, step 1.3 can be divided into the following four sub-steps:

[0020] Step 1.31, modeling user sub-model:

[0021] User tolerance time constraints, arbitrary user waiting time The maximum waiting time cannot be exceeded The constraints are shown in formula (1);

[0022]

[0023] The scenario we studied fully considers the urgency of the user's relay communication. Some users waiting for communication have urgent communication needs. We define the total number of communication requests initiated by users in this communication task as n total, the number of communication requests initiated again in addition to the first communication request is defined as n req ;n req With n total The relationship is specifically shown in formula (2), and the definition of the user relay communication expectation is specifically shown in formula (3). The indicator EU average is used to reflect the level of consideration of the user communication expectation during the entire service cycle, and the specific expression is shown in formula (4);

[0024]

[0025] Step 1.32, model the drone sub-model:

[0026] Without loss of generality, the UAV starts from the starting position Ψ in the relay service area, and the starting position is recorded as Q s , after providing a series of relay services to the user for a service time T, it finally returns to the starting point. The specific constraints are shown in formula (5);

[0027] Q s =Q0=Q N-1 (5)

[0028] The drone must not fly out of the service area to perform relay communication tasks at any time slot during the entire service cycle. To describe this constraint, let’s assume that the boundary equation of the area Ψ is The constraints are specifically shown in formula (6);

[0029]

[0030] The relay service area is a continuous three-dimensional space. The service area map is rasterized, and the constraints are as shown in formula (7), where Δx and Δy are unit distances;

[0031]

[0032] There is a constraint that the total bandwidth resources and power resources occupied by the user cannot exceed the maximum bandwidth and maximum power of the drone. The specific constraint is shown in formula (8);

[0033]

[0034] Step 1.33, modeling channel sub-model:

[0035] The Euclidean distance between the drone and user k in the service area at the tth time slot is recorded as The specific expression is shown in formula (9);

[0036]

[0037] The communication model between the UAV and user k follows the free space path loss model and is denoted as Based on the assumption of free space propagation without obstacles and interference, the specific expression is shown in formula (10);

[0038]

[0039] In formula (10), ρ0 is the ideal channel power gain per unit distance under line-of-sight link; in the field of wireless communication, the signal-to-noise ratio It is an indicator used to measure the relative strength between signal and noise. The specific expression is shown in formula (11);

[0040]

[0041] The transmission rate obtained by ground user k in time slot t is denoted as The specific expression is shown in formula (12);

[0042]

[0043] Step 1.34, the description of the UAV relay communication problem P with diverse communication requirements is specifically shown in formula (13);

[0044]

[0045] In problem P, (a), (b), (c) and (g) are the formal descriptions of the UAV's position movement constraints; (d) and (e) are the formal descriptions of the UAV's bandwidth resource and power resource constraints, respectively; and (f) is the constraint on the waiting time after the user initiates a communication request.

[0046] In step 2, the mathematical problem of drone relay communication is transformed into a Markov decision process as follows, which can be divided into two steps:

[0047] Step 2.1, the Markov decision process can be defined as a five-tuple MDP =<S,A,R,P,Υ> , where S is the discrete state space set, A is the action set of the agent, R is the immediate reward matrix, P is the agent state transition probability matrix, and γ is the discount coefficient; in the Markov decision process, the next state S t+1 Depends only on the current state S t and the current action a t , its specific expression is shown in formula (14);

[0048] P(S t+1 |S t+1 ,a t )=P(S t+1 |S1S2S3…S t ,a1a2a3…a t ) (14)

[0049] Step 2.2, convert the UAV relay communication problem into a five-tuple. Since the environment of the problem to be solved is changing, the definition of the state transition probability matrix P in the five-tuple is complex and difficult to give directly. Replace the state transfer matrix in the original quintuple and consider Constitute a non-stationary MDP;

[0050] Furthermore, step 2.2 can be divided into the following six steps:

[0051] Step 2.21, Drone status It consists of 3 parts in, Indicates the current stage of relay service and whether there is a return plan; Record the location of the drone; Used to record the user currently communicating;

[0052] Step 2.22, the drone’s actions in each time slot where a t Contains (Ω t ,Λ t ),Ω t ={0,1,…,8},

[0053] Step 2.23, transform the objective function and constraints of the UAV relay communication problem with diversity requirements into the reward function R t The function consists of three parts, as shown in formula (15);

[0054]

[0055] In formula (15), the first part is the data throughput R t Rewards, Part 2 is the user urgency level reward, and the third part is the penalty PEN for the given task P constraint. t ;

[0056] Step 2.24, the first part is the data throughput R t The reward is normalized and the specific expression is shown in formula (16);

[0057]

[0058] Step 2.25, part 2 is the user urgency level reward, and it is normalized. The specific expression is shown in formula (17);

[0059]

[0060] Step 2.26, transform the problem constraints into a reward function, as shown in formula (18);

[0061]

[0062] The drone will be penalized if it crosses the area boundary, as shown in formula (19);

[0063]

[0064] If the user's waiting time exceeds the tolerance time, he will be punished, as shown in formula (20);

[0065]

[0066] When the service end time is reached at the current time, if the user fails to return to the starting position, a penalty will be imposed, as shown in formula (21);

[0067]

[0068] If the total number of communication resources allocated to a user exceeds the maximum number of resources it has, a penalty will be imposed, as shown in formula (22);

[0069]

[0070] Formulas (19), (20), (21), and (22) respectively represent the constraints that the UAV cannot fly out of the service area, the communication resource allocation cannot exceed the total communication resources held, the user must be connected to the communication within the user tolerance time as much as possible, and the UAV must return to the starting position when the service time ends.

[0071] In step 3, the NPD-DQN algorithm is used to solve the Markov decision process model defined in step 2 as follows;

[0072] Step 3.1, this patent proposes the NPD-DQN algorithm based on DQN, and makes three adjustments compared to the original DQN algorithm;

[0073] We introduce a noise layer into the fully connected network. Assume that in the original network, this layer has p inputs and q outputs, which is specifically expressed as y = ωx + b; After introducing noise into this layer, the definition of this layer is shown in formula (23);

[0074]

[0075] The experience buffer uses SumTree to update the sample priority and select the samples to be learned, improving learning efficiency and stability. The sample priority is updated using TD-error. The specific formula is shown in formula (24).

[0076]

[0077] The NPD-DQN algorithm uses a dual network structure to select and evaluate actions to alleviate its over-estimation problem. The details of the loss function are shown in formula (25);

[0078]

[0079] Among them, part (a) represents the estimation of future rewards. The main network selects the action with the maximum Q value, and the target network evaluates the Q value of the action; part (b) represents the target Q value; part (c) is the estimated Q function value with noise.

[0080] Step 3.2, apply the proposed NPD-DQN algorithm to the UAV relay communication scenario;

[0081] Furthermore, step 3.2 is divided into the following nine steps:

[0082] Step 3.21, initialize the relay system environment, drone status, communication list, experience buffer, and neural network parameters;

[0083] Step 3.22, determine whether the maximum number of iterations has been reached, if so, end the algorithm, if not, execute 3.23;

[0084] Step 3.23, determine whether the maximum number of steps has been reached, if not, execute step 3.24, if yes, execute 3.22;

[0085] Step 3.24, check the communication list c Is there a user whose time slice has been used up? If so, remove him / her from the list. c , and release the communication resources, if not, no processing will be done; check the waiting list l w and urgent communication list u , is there a waiting time T w Exceeding tolerance time T r If there is a user, remove him from the queue and give him a certain penalty; check the new communication demand in the current time slot environment, if the user's request already exists in l w Then move it to l u Otherwise, add the user to l w , according to (24) to give the urgency index, if the user already exists in l w, then the urgency index is increased according to formula (24);

[0086] Step 3.25, get the current state of the drone, input the current state into the main network, get the Q value of different actions, and execute the action a with the maximum Q value t ;

[0087] Step 3.26, in s t Execute a in the state t Action, interacting with the environment, the drone transitions to the next state s t+1 , reward R t And whether the next state is the end state end, and the experience {s t ,a t ,r t ,s t+1 ,end}Based on the maximum sample priority, store it in the experience buffer;

[0088] Step 3.27, using the importance sampling mechanism, extract N in the experience buffer m For each sample, the target value is constructed by using the main network to select actions and the target network to evaluate actions.

[0089] Step 3.28, use formula (24) to calculate the TD error and update the sample priority; use formula (25) to calculate the mean square error and update the parameters of the main neural network;

[0090] Step 3.29, determine whether the update frequency of the target network has been reached. If so, copy the parameters of the main network to the target network. If not, proceed to step 3.23.

[0091] Beneficial effects of the present invention:

[0092] Based on deep reinforcement learning, the NPD-DQN algorithm is proposed for drone relay communication scenarios with diverse requirements. This method alleviates the traditional DQN transition estimation problem, makes better use of limited training resources, improves the robustness and exploration ability of the algorithm, and can better solve the core problems in drone relay scenarios and provide users with better relay communication services. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is an architecture diagram of the NPD-DQN algorithm applied to the UAV relay communication scenario in an embodiment of the present invention;

[0094] Figure 2 is a diagram describing an application scenario of an embodiment of the present invention;

[0095] Figure 3 is a Markov decision process model diagram;

[0096] Figure 4 is an action selection diagram for UAV communication resource allocation and trajectory optimization according to an embodiment of the present invention;

[0097] Figure 5 This is a simulation experiment scene diagram of an embodiment of the present invention;

[0098] Figure 6 It is a convergence diagram of the NPD-DQN algorithm applied to the UAV relay communication system according to an embodiment of the present invention;

[0099] Figure 7 This is a user throughput statistics chart of 30 executions of the NPD-DQN algorithm according to an embodiment of the present invention;

[0100] Figure 8 It is a statistical graph of user urgency levels when the NPD-DQN algorithm according to an embodiment of the present invention is executed 30 times;

[0101] Fig. 9 The effect of the embodiment of the present invention on the effect of the NPD-DQN algorithm under different population distributions;

[0102] Fig.10 It is a statistical diagram of data throughput of the NPD-DQN algorithm proposed in the embodiment of the present invention under different population distribution conditions;

[0103] Fig.11 It is a service quality statistics chart of the NPD-DQN algorithm proposed in the embodiment of the present invention under different population distribution conditions. DETAILED DESCRIPTION

[0104] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now selected and described in detail below; it is obvious that the implementation cases described are part of the embodiments of the present invention, rather than all the embodiments, and cannot be understood as limiting the scope of implementation of the present invention; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0105] Embodiment 1: Based on deep reinforcement learning, an NPD-DQN algorithm is proposed and applied to diverse drone relay communication scenarios to solve the problems of communication resource allocation and drone trajectory planning.

[0106] Research problem scenarios such as Figure 2 As shown in the diagram, the algorithm architecture is as follows Figure 1 As shown in the figure, the specific implementation method is introduced in three parts: the first part is the description of the simulation experiment scene and the setting of parameters, the second part is the specific application process of the NPD-DQN algorithm, and the third part is the experimental results and effect verification.

[0107] Part 1: Simulation experiment scenario simulation and parameter setting;

[0108] In our simulation, we set the 2 According to the problem definition, we divide the service area into 2500 blocks of 20m 2 The drone flies at an altitude of 30 meters and provides a total of 120 time slots of relay service.

[0109] There are 100 users in the relay communication area. The number of relay service requests for each user during the service period follows the Poisson distribution X ap ~Pois(3), the number of communication resource requests follows a normal distribution X rn ~N(4,1 2 ), the communication duration follows the normal distribution X ct ~N(3,1 2 ), the tolerance time follows the normal distribution X rt ~N(5,1 2 ); In our simulation experiment platform, the experimental scene diagram is Figure 6 , the important parameter settings are shown in Table 2 below:

[0110] Table 2: Simulation experiment parameter settings

[0111]

[0112] Part II: Practical application of NPD-DQN algorithm in diverse UAV relay communication system;

[0113] The specific application of the NPD-DQN algorithm is divided into two steps;

[0114] Step 1: Build a UAV relay system simulation environment with diverse communication resources, which is divided into the following sub-steps:

[0115] Step 1.1, complete the user simulation module in the environment;

[0116] Step 1.11, 10 locations are randomly generated in the service area, and 100 users are evenly distributed in a circle with a radius of 60m and the 10 locations as the center. The users are divided into 10 clusters to achieve user location simulation;

[0117] Step 1.12, the simulation scenario has a user communication demand generator to simulate 100 users generating communication demands. The number of communication demands of 100 users within the service time satisfies the Poisson distribution X ap ~Pois(3), the number of resources requested for each communication satisfies the normal distribution X rn ~N(4,1 2), the duration of each communication task satisfies the normal distribution X ct ~N(3,1 2 ), the tolerance time satisfies the normal distribution X rt ~N(5,1 2 );

[0118] Step 1.2, complete the penalty detection module in the environment;

[0119] Step 1.21, detect the current state of the drone s t Took action a t After that, check whether the boundary has been crossed. If so, a penalty will be imposed. If not, no action will be taken.

[0120] Step 1.22, UAV relay communication resource detection, check the communication resource allocation, whether it exceeds the remaining resources of the UAV, if so, give corresponding penalties;

[0121] Step 1.23, user waiting timeout detection, if there is a user in the waiting list and the urgent list whose waiting time has exceeded the timeout, a penalty will be imposed;

[0122] Step 1.24, check whether the drone has returned to its initial position;

[0123] Step 1.3, complete the environmental feedback module;

[0124] Step 1.31, calculate the position of the drone and the users in the communication list, and calculate the current communication throughput;

[0125] Step 1.32, calculate the current user urgency level;

[0126] Step 1.33, record whether the four tests occur, and if so, record the penalty;

[0127] Step 1.34, normalize the reward values ​​of step 1.31 and step 1.32, combine with the penalty value of step 1.33, and give the drone the total feedback for this action;

[0128] Step 1.4, user communication management module, there are three types of users that need to be managed, which is implemented in three sub-steps;

[0129] Step 1.41, check the communication list c Is there a user whose time slice has been used up? If so, remove him / her from the list. c , and release the communication resources, if not, no processing will be done;

[0130] Step 1.42, check the waiting list w and urgent communication list u , is there a waiting time T w Exceeding tolerance time Tr If there are any users, they will be removed from the queue and given a certain penalty;

[0131] Step 1.43, finally check the new communication demand in the current time slot environment. If the user's request already exists in l w Then move it to l u Otherwise, add the user to l w , according to (24) to give the urgency index, if the user already exists in l w , then the urgency index is increased according to formula (24);

[0132] Step 2: Apply the proposed NPD-DQN algorithm to the UAV relay communication scenario, which is completed in 9 steps:

[0133] Step 2.1, initialize the relay system environment, drone status, communication list, experience buffer, and neural network parameters;

[0134] Step 2.2: Determine whether the maximum number of iterations has been reached. If so, terminate the algorithm and output the optimal strategy sequence π and the maximum total reward value r. total , the maximum total throughput R total , average urgency level Otherwise, execute 2.3;

[0135] Step 2.3, determine whether the maximum number of steps has been reached, if not, execute step 2.4, if yes, execute 2.2;

[0136] Step 2.4, manage the communication list, waiting list and urgent communication list;

[0137] Step 2.5, get the current state of the drone, input the current state into the main network, get the Q value of different actions, and execute the action a with the maximum Q value t ;

[0138] Step 2.6, in s t Execute a in the state t Action, interacting with the environment, the drone transitions to the next state s t+1 , reward R t And whether the next state is the end state end, and the experience {s t ,a t ,r t ,s t+1 ,end}Based on the maximum sample priority, store it in the experience buffer;

[0139] Step 2.7, using the importance sampling mechanism, extract N mFor each sample, the target value is constructed by using the main network to select actions and the target network to evaluate actions.

[0140] Step 2.8, use formula (24) to calculate the TD error and update the sample priority; use formula (25) to calculate the mean square error and update the parameters of the main neural network;

[0141] Step 2.9, determine whether the update frequency of the target network has been reached. If so, copy the parameters of the main network to the target network. If not, proceed to step 2.3.

[0142] Part 3: Simulation experiment results presentation and results analysis;

[0143] Figure 1 shows the simulation results: Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.11 ;

[0144] from Figure 6 It can be concluded that after the algorithm becomes stable, the NPD-DQN algorithm we proposed is superior to the other seven algorithms; in terms of reward value, it is 5.0% higher than the DDQN algorithm, 4.2% higher than DuelingDQN, 18.1% higher than NoisyNet, 4.5% higher than P-DQN, and 35.5% higher than the original DQN, which is significantly better than the Q learning algorithm and the Greedy algorithm; in terms of stability, algorithms with a priority experience replay mechanism, such as NPD-DQN and P-DQN, show higher stability during the convergence process than the other six algorithms;

[0145] from Figure 7 The statistical results show that the communication throughput index of the NPD-DQN algorithm is better than that of the other seven algorithms. Specifically, compared with PDQN, it is improved by 16.4%; compared with DQN, NPD-DQN is improved by 36.3%; compared with DDQN, it is improved by 19.8%; compared with DuelingDQN, NPD-DQN is improved by 10.5%; compared with NoisyDQN, it is improved by 25.6%;

[0146] from Figure 8 The statistical results show that the NPD-DQN algorithm outperforms the other seven algorithms in terms of user urgency level indicators. Specifically, compared with PDQN, it is improved by 26.7%; compared with DQN, it is improved by 46.1%; compared with DDQN, it is improved by 16.3%; compared with Dueling DQN, it is improved by 21.2%; compared with Noisy DQN, it is improved by 29.5%;

[0147] from Fig. 9 From the statistical results, it can be concluded that as the number of user clusters decreases, the performance of the Greedy algorithm and the Q-learning algorithm does not change much, and is always poor and stable, while the effects of other algorithms decrease significantly; when the number of user clusters is 8, 6, and 4, although the performance of the NPD-DQN algorithm also shows a downward trend, it is still significantly better than other methods; it is worth noting that when the number of user clusters is reduced to 2, Dueling DQN performs best, surpassing other algorithms, including NPD-DQN;

[0148] from Fig.10 From the statistical results, it can be concluded that the effect of changing the number of user clusters on the optimization goal of communication throughput is not significant in general; when the number of users changes from 10 clusters to 4 clusters, the communication throughput decreases slightly overall, and when the number of user clusters decreases to 2, the total communication throughput generally increases, but is still slightly lower than the level when the number of user clusters is 10 and 8; it is particularly worth noting that the communication throughput of the NPD-DQN algorithm is higher than that of other algorithms in all cluster numbers; when the number of user clusters is 2, the communication throughput of the Greedy algorithm exceeds that of other cluster numbers;

[0149] from Fig.11 It can be concluded from the statistical results that the number of user clusters has a significant impact on user satisfaction; as the number of user clusters decreases, the user satisfaction of all methods shows a clear downward trend; although the user satisfaction of the NPD-DQN algorithm also decreases when the number of user clusters is 10, 8, 6 and 4, it is still better than other algorithms; however, when the number of user clusters is 2, the user satisfaction of the Dueling DQN method is better than that of other algorithms.

[0150] In Example 1, a method called NPD-DQN is proposed, which is based on the DQN algorithm and successfully applied to the UAV relay communication scenario with diverse requirements. By comparing with algorithms such as DQN, DDQN, DQN, Q-learning, Greedy, Dueling DQN, and NoisyNet DQN with experience priority replay, it is proved that the NPD-DQN algorithm has better stability and convergence under the same experimental parameter settings, and can obtain higher reward values; therefore, the NPD-DQN algorithm is more suitable for solving problems related to complex and changeable UAV relay communication scenarios.

[0151] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of the present invention to be protected is defined by the attached claims and their equivalents.

Claims

1. A relay drone diversity resource scheduling method based on an improved DQN algorithm, characterized in that: It includes the following 2 steps: Step 1: According to the actual application scenarios, a mathematical model is established for the UAV relay communication scenarios with diverse requirements, and the mathematical model is converted into a Markov decision process model; Step 2, apply the proposed NPD-DQN algorithm to the Markov decision process model defined in step 1; Among them, the NPD-DQN algorithm proposed in step 2 makes three improvements compared with the traditional DQN algorithm; First, a noise layer is introduced into the fully connected network. Assume that in the original network, this layer has p inputs and q outputs, which is specifically expressed as y = ωx + b; After the noise is introduced into this layer, the definition of this layer is as shown in formula (1); Secondly, the experience buffer uses SumTree to update the sample priority and select the samples to be learned, improving learning efficiency and stability; the sample priority is updated using TD-error, as shown in formula (2); Finally, the NPD-DQN algorithm uses a dual network structure to select and evaluate actions to alleviate its overestimation problem. The loss function is shown in (3). In formula (3), part (a) represents the estimation of future rewards. The main network selects the action with the maximum Q value; the target network evaluates the Q value of the action; part (b) represents the target Q value; part (c) is the estimated Q function value with noise; Step 2.1, initialize the relay system environment, drone status, communication list, experience buffer, and neural network parameters; Step 2.2, determine whether the maximum number of iterations has been reached, if so, end the algorithm, if not, execute 2.3; Step 2.3, determine whether the maximum number of steps has been reached, if not, execute step 2.4, if yes, execute 2.2; Step 2.4, manage the communication list, waiting list and urgent communication list; Step 2.5, get the current state of the drone, input the current state into the main network, get the Q value of different actions, and execute the action a with the maximum Q value t ; Step 2.6, in s t Execute a in the state t Action, interacting with the environment, the drone transitions to the next state s t+1 , reward R t And whether the next state is the end state end, and the experience {s t ,a t ,r t ,s t+1 ,end}Based on the maximum sample priority, store it in the experience buffer; Step 2.7, using the importance sampling mechanism, extract N m For each sample, the target value is constructed by using the main network to select actions and the target network to evaluate actions. Step 2.8, use formula (2) to calculate the TD error and update the sample priority; use formula (3) to calculate the mean square error and update the parameters of the main neural network; Step 2.9, determine whether the update frequency of the target network has been reached. If so, copy the parameters of the main network to the target network. If not, proceed to step 2.

3.

2. According to claim 1, a relay drone diversity resource scheduling method based on an improved DQN algorithm is characterized in that: The step 1 specifically includes the following sub-steps: Step 1.1, make specific assumptions about the relay scenario; assume that there is a base station A in the distance and that base station A is able to help base station B that is overloaded or damaged in the area to complete the communication task; since there are few objects blocking the sky, it is assumed that the signal propagation is line-of-sight link LoS propagation; during the service time, the drone is able to access any user in the area; since it is too difficult to dynamically control the movement of drones in a continuous three-dimensional space, the service area map is rasterized; users occupy communication resources in a non-preemptive manner, and the system can only reclaim the allocated resources after the time slice requested by the user is consumed; the scenario under study does not consider user movement, and the flight altitude of the drone will be limited; each user can obtain his or her own location through GPS and can share it with the drone synchronously; the user's location remains unchanged during the entire service cycle; The experimental scenario will use frequency division multiplexing to connect users to communication, and assume that the channels in the channel set are orthogonal channels; under the setting of the above scenario, the drone relay service scenario is defined; UAV relay communication scenario with diverse communication requirements Variables and symbols: The user set is represented by K = {1, 2, 3, ..., k}, where k users are distributed in the communication service area Ψ, and the position of the UAV in the service period T is defined as Step 1.2, establish the UAV relay communication problem model and transform the problem model into a Markov decision process model.

3. According to claim 2, a relay drone diversity resource scheduling method based on an improved DQN algorithm is characterized in that: The step 1.2 specifically includes the following sub-steps: Step 1.21, modeling the user sub-model of the relay communication system; User tolerance time constraints, arbitrary user waiting time The maximum waiting time cannot be exceeded The constraints are specifically shown in formula (4); The UAV relay service scenario fully considers the urgency of the user's relay communication. Some users waiting for communication have urgent communication needs, so the total number of communication requests initiated by the users of this communication task is defined as n total , the number of communication requests initiated again in addition to the first communication request is defined as n req ;n req With n total The relationship is as shown in formula (5); The definition of the expected degree of user relay communication is specifically shown in formula (6); The indicator EU average is used to reflect the level of consideration of user communication expectations during the entire service cycle, as shown in formula (7); Step 1.22, model the relay communication drone sub-model; Without loss of generality, the UAV starts from the starting position Ψ in the relay service area, and the starting position is recorded as Q s , after providing a series of relay services to the user for a service time T, it finally returns to the starting point. The specific constraints are shown in formula (8); Q s =Q0=Q N-1 (8) The drone must not fly out of the service area to perform relay communication tasks at any time slot during the entire service cycle. To describe this constraint, let’s assume that the boundary equation of the area Ψ is The constraints are specifically shown in formula (9); The relay service area is a continuous three-dimensional space. The service area map is rasterized. There are movement constraints for each time slot of the UAV. The specific constraints are shown in formula (10), where Δx and Δy are unit distances; There is a constraint that the total bandwidth resources and power resources occupied by the user cannot exceed the maximum bandwidth and maximum power of the drone. The specific constraint is shown in formula (11); Step 1.23, modeling the relay communication channel sub-model; The Euclidean distance between the drone and user k in the service area at the tth time slot is recorded as The specific expression of Euclidean distance is shown in formula (12); The communication model between the UAV and user k follows the free space path loss model and is denoted as Based on the assumption of free space propagation without obstacles and interference, the specific expression is shown in formula (13); In formula (13), ρ0 is the ideal channel power gain per unit distance under line-of-sight link; in the field of wireless communication, the signal-to-noise ratio It is an indicator used to measure the relative strength between signal and noise. The specific expression is shown in formula (14); represents the transmission rate obtained by ground user k in time slot t, and the specific expression is shown in formula (15); Step 1.24, the description of the UAV relay communication problem P with diverse communication requirements is specifically shown in formula (16); In problem P, (a), (b), (c) and (g) are the formal descriptions of the UAV's position movement constraints; (d) and (e) are the formal descriptions of the UAV's bandwidth resource and power resource constraints, respectively; (f) is the constraint on the waiting time after the user initiates a communication request; Step 1.25, convert the UAV relay communication model with the defined diverse communication requirements into a Markov decision process model.

4. According to claim 3, a relay drone diversity resource scheduling method based on an improved DQN algorithm is characterized in that: The step 1.25 specifically includes the following sub-steps: Step 1.251, convert the UAV relay communication problem into a 5-tuple; the Markov decision process can be defined as a 5-tuple MDP =<S,A,R,P,Υ> , where S is the discrete state space set, A is the action set of the agent, R is the immediate reward matrix, P is the agent state transition probability matrix, and γ is the discount coefficient; in the Markov decision process, the next state S t+1 Depends only on the current state S t and the current action a t , its specific expression is shown in formula (17); P(S t+1 |S t+1 ,a t )=P(S t+1 |S1S2S3…S t ,a1a2a3…a t ) (17) Since the environment of the problem being solved is changing, the definition of the state transition probability matrix P in the 5-tuple is complex and difficult to give directly. Replace the state transfer matrix in the original quintuple and consider Constitute a non-stationary MDP; Drone status It consists of 3 parts in, Indicates the current stage of relay service and whether there is a return plan; Record the location of the drone; Used to record the user currently communicating; Drone movements at each time slot where a t Contains (Ω t ,Λ t ),Ω t ={0,1,…,8}, Step 1.252, transform the objective function and constraints of the UAV relay communication problem with diversity requirements into a reward function; the specific expression of the reward function is shown in formula (18); In formula (18), the first part is the data throughput R t Rewards, Part 2 is the user's urgency level reward, and the third part is the penalty PEN for the given task P constraint. t ; The first part is the data throughput R t The reward is normalized as shown in formula (19); Among them, r max Indicates the signal-to-noise ratio of the channel between the drone and the user under ideal conditions; Part 2 is the user urgency level reward, and it is normalized as shown in formula (20); In the third part, the problem constraints are transformed into reward functions, and there are four penalties for the behavior of the drone, as shown in formula (21); The drone will be penalized if it crosses the boundary of the area, as shown in formula (22); If the user's waiting time exceeds the tolerance time, he will be punished, as shown in formula (23); If the total number of communication resources allocated to the user exceeds the maximum number of resources owned by the drone, a penalty will be imposed, as shown in formula (24); When the service end time is reached at the current time, if the user fails to return to the starting position, a penalty will be imposed, as shown in formula (25); Formulas (22), (23), (24) and (25) respectively represent the constraints that the UAV cannot fly out of the service area, the communication resource allocation cannot exceed the total communication resources held, the user must be connected to the communication within the user tolerance time as much as possible, and the UAV must return to the starting position when the service time ends.

5. According to claim 1, a relay drone diversity resource scheduling method based on an improved DQN algorithm is characterized in that: The step 2.4, management of the communication list, waiting list and urgent communication list specifically includes the following sub-steps: Step 2.41, check the communication list c Is there a user whose time slice has been used up? If so, remove him / her from the list. c , and release the communication resources, if not, no processing will be done; Step 2.42, check the waiting list w and urgent communication list u , is there a waiting time T w Exceeding tolerance time T r If there are any users, they will be removed from the queue and given a certain penalty; Step 2.43, finally check the new communication demand in the current time slot environment. If the user's request already exists in l w Then move it to l u Otherwise, add the user to l w , according to (2) to give the urgency index, if the user already exists in l w , then increase the urgency index according to formula (2).

Images