Access control and trajectory planning method and device for emergency communication network

Abstract

The invention relates to an access control and trajectory planning method and device for an emergency communication network. According to the method, communication requirements of an uplink on throughput and a downlink on timeliness and reliability in an emergency communication network are comprehensively considered, a joint optimization model of access control and unmanned aerial vehicle trajectory planning problems is modeled, and joint optimization of differentiated communication requirements of the uplink and the downlink can be realized by solving the joint optimization model. And a priority-based access control algorithm and a weight generation network enhancement-based QMIX trajectory planning algorithm are provided to solve the joint optimization model, so that a better result can be obtained through convergence at a higher speed. According to the method, efficient communication and resource management in an emergency communication scene can be realized.

Description

Technical Field

[0001] The present application relates to the field of emergency communication technology for unmanned aerial vehicles, and in particular to an access control and trajectory planning method and device for an emergency communication network. Background Art

[0002] Unmanned aerial vehicles (UAVs) are an integral part of future mobile networks, aiming to establish 3D seamless communications. UAVs can be used as aerial base stations to expand network coverage and capacity. Compared with traditional ground base stations, UAV base stations are highly maneuverable, flexible to deploy, and low-cost, making them ideal for emergency communications in emergency situations such as disaster relief. When ground infrastructure fails, UAVs can serve as communication relays between remote control centers and ground nodes, providing reliable, low-cost communication services for ground nodes.

[0003] In disaster relief scenarios, ensuring the quality of service (QoS) in communications is crucial because it directly affects the progress of rescue operations. Currently, many works consider optimizing the achievable data rate or latency. However, in some special scenarios, it is not enough to evaluate the performance of the entire network from a single dimension. For example, in disaster and rescue scenarios, ground robots can be used as ground nodes to complete specific tasks. By carrying some sensors such as cameras and radars, ground robots can replace humans to perform tasks such as disaster area monitoring and life detection. Since the ground infrastructure is damaged, drone clusters can usually be used as communication relays to transmit business information between the remote control center and the ground robot. In this scenario, the ground robot transmits the collected environmental information back to the control center through the UAV. At the same time, the UAV also needs to forward the control commands issued by the control center to the ground robot. Therefore, ensuring the QoS quality of the remote control center and the ground robot is the key to disaster relief. Specifically, the link from the ground robot to the drone needs to provide sufficient channel capacity to transmit environmental information, while the link from the UAV to the ground robot needs to ensure the timeliness and reliability of the control information. Therefore, it is very necessary to simultaneously ensure the differentiated communication requirements of the uplink and downlink.

[0004] However, the communication resources and coverage of a single UAV are limited. To overcome these limitations, multiple UAVs are used to collaboratively provide services to ground nodes to expand network coverage and improve communication capabilities. Unlike a single UAV, multiple UAV base stations use the same spectrum resources, which introduces additional interference. The location of the UAV, communication resource allocation, and node scheduling can be adjusted to improve the quality of service to ground nodes. Therefore, joint optimization of UAV trajectories, communication resources, and node scheduling is necessary. However, this problem exhibits a high degree of complexity. Traditional optimization-based methods rely on global information and calculate UAV trajectories and resource allocation schemes offline. However, this approach has great limitations. On the one hand, it is impractical to obtain global information at all times for disaster scenarios where existing ground infrastructure is destroyed. On the other hand, when the node position changes, the complex solution process needs to be repeatedly calculated, which is very time-consuming.

[0005] Fortunately, the emergence of reinforcement learning has brought more possibilities for multi-UAV assisted communication. Multi-agent deep reinforcement learning (MADRL) allows each UAV to act as an independent decision maker to adapt to the distributed needs of wireless networks. At present, many researchers have used MADRL for the joint optimization of UAV trajectory and communication. Although most MADRL algorithms show satisfactory performance in small-scale networks, when the number of network nodes increases, the state-action space will increase exponentially with the increase in the number of UAVs and nodes. In addition, due to the limited perception ability of UAV agents, each agent is usually observed locally. The nodes they observe at different times are changing, which will cause the convergence of MADRL to deteriorate. Existing MADRL algorithms require a large number of empirical samples to achieve convergence in large-scale scenarios when solving the joint optimization problem of UAV trajectory and communication. Summary of the invention

[0006] Based on this, it is necessary to provide an access control and trajectory planning method and device for emergency communication networks to address the technical problems of high differentiated communication requirements, complex calculations and poor convergence in the joint optimization problem of uplink and downlinks under the conditions of limited frequency resources in the above-mentioned emergency communication scenarios, so as to achieve efficient communication and resource management in emergency communication scenarios.

[0007] An access control and trajectory planning method for an emergency communication network, the method comprising:

[0008] Build a multi-UAV-assisted emergency communication network and obtain uplinks and downlinks within the network; the emergency communication network consists of multiple UAVs, multiple ground nodes, and a remote control center;

[0009] Construct limited perception model, channel model and communication model between UAV and ground nodes based on emergency communication network;

[0010] By jointly considering the communication requirements of uplink throughput and downlink timeliness and reliability, and based on the limited perception model, channel model and communication model, a joint optimization model of access control and UAV trajectory planning problems is constructed;

[0011] The joint optimization model is described as a de-neutralized partially observable Markov process, and a priority-based access control algorithm and a weight generation network-enhanced QMIX trajectory planning algorithm are used to solve the joint optimization model to obtain the optimal access decision and the optimal trajectory of the UAV.

[0012] In one of the embodiments, in an emergency communication network, each ground node is used to collect and obtain environmental information and transmit it to a remote control center. The remote control center is used to make decisions based on the received environmental information and send command and control information including mobile control and mission instructions to the ground node. Each drone serves as an aerial base station and is used to forward the transmitted information between the ground node and the remote control center. The uplink in the network is defined as a link for transmitting environmental information from the ground node to the drone, and the downlink is defined as a link for transmitting command and control information from the remote control center to the ground node.

[0013] In one embodiment, the limited perception model is described by using the perception variables between the UAV and the ground node, and the perception variables between the UAV m and the ground node u at time slot t are defined as c m,u (t), and c m,u (t)∈{0,1}; if the ground node u is within the sensing range of the drone m at time slot t, c m,u (t) = 1; otherwise c m,u (t)=0.

[0014] In one embodiment, the channel model is described by the channel gain between the ground node and the UAV, and the channel gain is calculated as follows:

[0015] According to the pitch angle from the ground node u to the UAV m at time slot t, the probability of the line-of-sight (LoS) link and the non-line-of-sight (NLoS) link between the ground node u and the UAV m is obtained, which are expressed as

[0016]

[0017] in, represents the probability of the LoS link between ground node u and UAV m at time slot t, represents the probability of the NLoS link between the ground node u and the UAV m at time slot t, a and b are parameters determined by the environment type, represents the pitch angle from ground node u to UAV m at time slot t, H represents the fixed altitude of the UAV, and q u is the position of ground node u, q m (t) is the position of UAV m at time slot t;

[0018] According to the positions of the ground node u and the UAV m at time slot t, the path loss between the ground node u and the UAV m is obtained, which is expressed as

[0019]

[0020] Among them, η ξ represents the average additional path loss of the LoS link or NLoS link, f c is the carrier frequency, c is the speed of light, and ξ represents a LoS link or a NLoS link;

[0021] according to And the path loss corresponding to each link is calculated to obtain the channel gain between the ground node u and the UAV m, which is expressed as

[0022]

[0023] in, represents the path loss of the LoS link between ground node u and UAV m at time slot t, represents the path loss of the NLoS link between the ground node u and the UAV m at time slot t.

[0024] In one embodiment, the communication model includes:

[0025] For the uplink, the communication model is described by uplink signal-to-interference-noise ratio, uplink data rate, and throughput; where the uplink signal-to-interference-noise ratio from ground node u to drone m at time slot t is Expressed as

[0026]

[0027] Among them, g u,m (t) represents the channel gain between ground node u and UAV m at time slot t, g u′,m (t) represents the channel gain between another ground node u′ and UAV m at time slot t, α u,m (t) represents the access control variable between ground node u and UAV m at time slot t, α u′,m′ (t) represents the access control variable between another ground node u′ and another UAV m′ at time slot t, n0 represents the power spectral density of additive Gaussian white noise, p uis the transmission power of ground node u on each subchannel, B is the subchannel bandwidth, M represents the number of UAVs, and U represents the number of ground nodes;

[0028] Uplink data rate from ground node u to UAV m at time slot t Expressed as

[0029]

[0030] The total throughputs obtained by the ground node u and the entire emergency communication network in the past t time slots are expressed as and Where t' = {1, 2, ..., t} is any time slot in the past t time slots, δ is the time slot length; meanwhile, the average throughput of ground node u in the past t time slots is expressed as

[0031] For the downlink, the communication model is described by the downlink signal-to-interference-noise ratio, downlink data rate, and decoding error probability; where the downlink signal-to-interference-noise ratio from drone m to ground node u at time slot t is Expressed as

[0032]

[0033] Among them, α m,u (t) represents the access control variable between UAV m and ground node u at time slot t, α m',u' (t) represents the access control variable between another UAV m′ and another ground node u′ at time slot t, g m,u (t) represents the channel gain between UAV m and ground node u at time slot t, g m',u (t) represents the channel gain between another UAV m′ and the ground node u at time slot t, p m is the transmission power of UAV m in each sub-channel, For drone collection, is the ground node set;

[0034] Downlink data rate from UAV m to ground node u at time slot t Expressed as

[0035]

[0036] Among them, τ is the transmission delay of the command and control packet (the packet of command and control information), Q -1 (·) is the inverse function of the Q function, ε max is the maximum decoding error probability, V represents the channel dispersion, which can be expressed as

[0037] When UAV m transmits a command packet to ground node u at time slot t, the decoding error probability of ground node u is expressed as

[0038]

[0039] Among them, τ max represents the maximum tolerable delay, S represents the given packet length; and ε u (t)≤ε max .

[0040] In one embodiment, the joint optimization model of access control and UAV trajectory planning problem is expressed as

[0041]

[0042] The joint optimization model is a mixed integer nonlinear programming problem consisting of a continuous variable q and a discrete variable α, where λ is a weight coefficient used to balance the relative importance of the uplink and downlink. Indicates the service period of the ground node in the UAV The total fair throughput is used to consider the communication demand for throughput of the uplink; service period It is divided into T time slots of equal length, each time slot length is δ; represents the fairness coefficient at time slot t, is the average throughput of ground node u in the past t time slots, and U represents the number of ground nodes; represents the uplink data rate of ground node u at time slot t; represents the number of downlinks that meet QoS (QoS-DL), which is used to consider the communication requirements of downlinks for timeliness and reliability; 1(·) represents the indicator function, which is specifically expressed as ε max is the maximum decoding error probability, ε u (t) represents the decoding error probability of ground node u at time slot t; M represents the number of UAVs;

[0043] The joint optimization model satisfies the following constraints:

[0044]

[0045] Among them, constraint (1) means that only ground nodes within the UAV’s sensing range can access the UAV, α m,u (t) represents the access control variable between UAV m and ground node u at time slot t, For drone collection, is the ground node set, represents the service period; constraint (2) indicates that each ground node can only access one UAV at most; constraint (3) indicates that the number of ground nodes accessed by each UAV cannot exceed the number of subchannels, K is the total number of subchannels; constraint (4) indicates the value range of access control variables and perception variables, where c m,u (t) represents the perception variable between UAV m and ground node u at time slot t; constraint (5) indicates that the position of the UAV cannot exceed the boundary of the target area, where x m (t) and y m (t) represents the horizontal and vertical coordinates of UAV m at time slot t, X and Y represent the horizontal and vertical boundaries of the target area, respectively; constraint (6) means that the distance between any two UAVs is greater than the minimum distance D min , to avoid collision, where q i (t) and q j (t) represent the positions of UAV i and UAV j at time slot t, respectively.

[0046] In one embodiment, the joint optimization model is described as a de-neutralized partially observable Markov process, including:

[0047] The feature vectors of UAV m and ground node u at time slot t are obtained respectively, which are expressed as

[0048]

[0049] Among them, the feature vector of drone m at time slot t is Including the current position q of drone m m (t) and the total number of ground nodes accessed by UAV m in the previous time slot The characteristic vector of ground node u at time slot t Including the location q of the ground node n u , whether the drone was connected in the previous time slot Average uplink throughput up to time slot t-1 and the decoding error probability ε of time slot t-1 u (t-1);

[0050] According to the feature vectors of the two entities, the UAV and the ground node, the joint optimization model is described as a de-neutralized partially observable Markov process, which is described by the environment state, the local observation information of the UAV, the UAV action and the reward function. Among them, the environment state is described by the feature vector set of all entities; the local observation information of the UAV is described by the feature vector of the UAV itself, the feature vector set of neighboring UAVs within the perception range of the UAV and the feature vector set of ground nodes; the UAV action is described by the action decision of the UAV's moving direction and the ID of the ground node connected to the UAV; the reward function is described by the global reward and the individual penalty value of the UAV, and the global reward is the weighted sum of the total fair throughput and QoS-DL.

[0051] In one embodiment, the priority-based access control algorithm includes:

[0052] For ground nodes, in each time slot, each ground node gives priority to accessing the nearest UAV;

[0053] For drones, the throughput priority principle is adopted for the uplink, that is, in the first t time slots, the ground nodes with less throughput are given priority to access the drone; the distance priority principle is adopted for the downlink, that is, the ground nodes with the closest distance are given priority to access the drone; by taking the weighted sum of the normalized throughput and distance, the priority ranking of the drone for the set of ground nodes within the sensing range is obtained, which is expressed as

[0054]

[0055] Among them, P m is the set of ground nodes within the sensing range of UAV m , sort(·) indicates that each UAV sorts the ground nodes according to the weighted sum of the normalized throughput and distance. The ground nodes with higher priority are preferentially connected to the UAV, and for each ground node connected to the UAV, the corresponding UAV will allocate the subchannel with the least interference in the uplink and downlink at the current moment to it; Γ u (t) is the total throughput obtained by ground node u in the past t time slots, is the maximum value of the total throughput, λ is the weight coefficient, q u is the position of ground node u, q m (t) is the position of UAV m at time slot t, and u′ represents the set Any ground node in Assemble for drones.

[0056] In one of the embodiments, a QMIX trajectory planning algorithm enhanced by a weight generation network is composed of several agent networks, a hybrid network and a super network;

[0057] The agent network is used to obtain the local observation information of the corresponding drone and fit the individual value of the drone agent. The agent network consists of three parts: the input layer, the middle layer and the output layer. The input layer consists of two weight generation networks and a multi-layer perceptron. The two weight generation networks are used to extract the feature vector sets of neighboring drones within the drone's perception range and the feature vector sets of ground nodes in the drone's local observation information. The multi-layer perceptron is used to extract the feature vectors of the drone itself in the drone's local observation information. The input layer finally outputs an embedded feature by merging the outputs of the two weight generation networks and a multi-layer perceptron. The middle layer uses a gated recursive unit, which takes the hidden state of the previous time slot and the embedded features output by the input layer as input, and outputs the hidden state of the current time slot. The output layer uses a multi-layer perceptron, which calculates by obtaining the hidden state of the current time slot and outputs the individual value of the drone. The drone uses the ∈-greedy strategy to make action decisions based on its own individual value.

[0058] The hybrid network is used to take the individual value of each drone as input and obtain the joint action value through forward propagation;

[0059] The hypernetwork is used to calculate the weights and biases of the hybrid network based on the state of the environment.

[0060] An access control and trajectory planning device for an emergency communication network, the device comprising:

[0061] A network construction module is used to build a multi-UAV-assisted emergency communication network and obtain uplinks and downlinks within the network; the emergency communication network consists of multiple UAVs, multiple ground nodes, and a remote control center;

[0062] A model building module is used to build a limited perception model, channel model and communication model between the UAV and the ground node based on the emergency communication network;

[0063] The problem modeling module is used to construct a joint optimization model of access control and UAV trajectory planning problems by jointly considering the communication requirements of uplink throughput and downlink timeliness and reliability, and based on the limited perception model, channel model and communication model;

[0064] The joint solution module is used to describe the joint optimization model as a de-neutralized partially observable Markov process, and adopts the priority-based access control algorithm and the QMIX trajectory planning algorithm enhanced by the weight generation network to solve the joint optimization model to obtain the optimal access decision and the optimal trajectory of the UAV.

[0065] The access control and trajectory planning method and device for emergency communication networks comprehensively consider the communication requirements of uplink throughput and downlink timeliness and reliability in emergency communication networks, and model the joint optimization model of access control and UAV trajectory planning problems. By solving the joint optimization model, the joint optimization of differentiated communication requirements of uplink and downlink can be achieved, and a priority-based access control algorithm and a QMIX trajectory planning algorithm enhanced by a weight generation network are proposed to solve the joint optimization model. Among them, the priority-based access control algorithm realizes node access through a heuristic method, effectively reduces the state action space of the QMIX algorithm, and solves the problem of poor convergence; the QMIX trajectory planning algorithm enhanced by a weight generation network uses a weight generation network to improve the QMIX algorithm, solves the problem of low training efficiency caused by dynamic observation, and can converge to a better result at a faster speed. This method can realize efficient communication and resource management in emergency communication scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a flow chart of an access control and trajectory planning method for an emergency communication network in one embodiment;

[0067] Figure 2 A schematic diagram of an emergency communication network assisted by multiple drones in one embodiment;

[0068] Figure 3 A schematic diagram of a weight generation network in one embodiment;

[0069] Figure 4 A schematic diagram of the architecture of a QMIX trajectory planning algorithm enhanced based on a weight generation network in one embodiment;

[0070] Figure 5 Schematic diagram of the process of the PW-QMIX algorithm in one embodiment;

[0071] Figure 6 It is a schematic diagram of a long-term reward convergence curve under different learning rate conditions in one embodiment;

[0072] Figure 7 A schematic diagram showing a comparison of decision performances between a PW-QMIX algorithm and a QMIX algorithm in one embodiment;

[0073] Figure 8A schematic diagram of convergence curves of solutions with / without dynamic observation processing under conditions of different numbers of ground nodes in one embodiment;

[0074] Fig. 9 is a schematic diagram of convergence curves of different observation processing schemes in one embodiment;

[0075] Fig.10 is a schematic diagram of the relationship between λ and fair throughput under different charge packet lengths in one embodiment;

[0076] Fig.11 Schematic diagram of the relationship between λ and QoS-DL under different charge packet lengths in one embodiment;

[0077] Fig.12 A schematic diagram of the relationship between the number of drones and fair throughput for different algorithms in one embodiment;

[0078] Fig.13 Schematic diagram of the relationship between the number of drones and QoS-DL for different algorithms in one embodiment. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0080] In one embodiment, Figure 1 As shown, a method for access control and trajectory planning for an emergency communication network is provided, comprising the following steps:

[0081] Step S1, constructing a multi-UAV-assisted emergency communication network and obtaining uplinks and downlinks within the network; wherein the emergency communication network is composed of multiple UAVs, multiple ground nodes and a remote control center.

[0082] Specifically, Figure 2As shown, the present application considers a multi-drone assisted emergency communication network in a disaster rescue scenario. When the ground communication infrastructure (such as a 5G base station) is destroyed, the drone is deployed as an aerial base station to quickly provide emergency communication services. The rescue robot, as a ground node (GN), can collect and obtain front-line environmental information and transmit the environmental information in the form of images or videos to the remote control center. The remote control center is used to make decisions based on the received environmental information and send command and control information including mobile control and mission instructions to the ground node. Then, the rescue robot performs search and rescue missions based on the received command and control center. Each drone acts as an aerial base station for forwarding the transmitted information between the ground node and the remote control center. The uplink in the network is defined as the link for transmitting environmental information from the ground node to the drone, and the downlink is defined as the link for transmitting command and control information from the remote control center to the ground node.

[0083] Assume that there are M drones and U ground nodes in the emergency communication network. and Represents the drone set and the ground node set respectively. This application considers that the drone provides a time period of Service cycle is divided into T equal time slots, each time slot length is δ. The drone flies at a fixed altitude H and a constant speed v. A 3D Cartesian coordinate system is used to describe the positions of the drone and the ground nodes. At time slot t, the position of drone m is expressed as The position of the ground node u is expressed as In a certain time slot, the ground node can only communicate with one drone. To avoid collision, drones need to maintain a safe distance D. min , expressed as

[0084] Step S2, constructing a limited perception model, a channel model and a communication model between the UAV and the ground node based on the emergency communication network.

[0085] Limited perception model: When temporarily deployed drones provide communication services for a large unknown area, they are usually unable to know the status information of all nodes in advance. In an actual emergency communication network, ground nodes can periodically send location and communication request information through a common channel. In addition, due to signal attenuation, drones can only receive messages from ground nodes within a certain distance. Similarly, drones can only perceive neighboring drones within a certain range. Assume that the perception distance of each drone to the ground node and neighboring drones is D GN and D UAV In time slot t, only when UAV m can only sense ground node u when When t, UAV i and UAV j can perceive each other. It can be seen that each UAV can only obtain partial status information of the ground node in real time. Due to the change of the position of the UAV, its observation area is also constantly changing. In order to characterize the perception relationship between the UAV and the ground node, the limited perception model constructed in this application uses the perception variables between the UAV and the ground node to describe it. Specifically, the perception variable between UAV m and ground node u at time slot t is defined as c m,u (t), and c m,u (t)∈{0,1}; if the ground node u is within the sensing range of the drone m at time slot t, c m,u (t) = 1; otherwise c m,u (t)=0.

[0086] Channel model: The channel model constructed in this application is described by the channel gain between the ground node and the drone. The calculation process of the channel gain is:

[0087] This application considers a general air to ground (A2G) model that depends on the probability of occurrence of line-of-sight (LoS) and non-line-of-sight (NLoS) links between UAVs and ground nodes.

[0088] According to the pitch angle from the ground node u to the UAV m at time slot t, the probability of the line-of-sight (LoS) link and the non-line-of-sight (NLoS) link between the ground node u and the UAV m is obtained, which are expressed as

[0089]

[0090] in, represents the probability of the LoS link between ground node u and UAV m at time slot t, represents the probability of the NLoS link between the ground node u and the UAV m at time slot t, a and b are parameters determined by the environment type, represents the pitch angle from ground node u to UAV m at time slot t, H represents the fixed altitude of the UAV, and q u is the position of ground node u, q m (t) is the position of UAV m at time slot t;

[0091] According to the positions of the ground node u and the UAV m at time slot t, the path loss between the ground node u and the UAV m is obtained, which is expressed as

[0092]

[0093] Among them, η ξrepresents the average additional path loss of the LoS link or NLoS link, f c is the carrier frequency, c is the speed of light, and ξ represents a LoS link or a NLoS link;

[0094] according to And the path loss corresponding to each link is calculated to obtain the channel gain between the ground node u and the UAV m, which is the mathematical expectation of the two cases of LoS and NLoS, specifically expressed as

[0095]

[0096] in, represents the path loss of the LoS link between ground node u and UAV m at time slot t, represents the path loss of the NLoS link between the ground node u and the UAV m at time slot t.

[0097] Communication model: To minimize interference, each drone provides uplink and downlink access to the ground node it accesses through Orthogonal Frequency Division Multiple Access (OFDMA) technology. At the same time, the uplink and downlink work in Frequency Division Duplexing (FDD) mode, that is, the uplink and downlink of each ground node work simultaneously on different frequency bands. Assume that each drone has K UL =K DL = K uplink subchannels and downlink subchannels with the same bandwidth of B, respectively expressed as The transmission power of each UAV and ground node in each subchannel is p m and p u . Define access control variable α u,m (t) describes the access relationship between UAV m and ground node u. When ground node u accesses UAV m at time slot t, α u,m (t)=1, otherwise α u,m (t)=0.

[0098] For the uplink, the communication model is described by uplink signal-to-interference-plus-noise ratio, uplink data rate, and throughput; where the uplink signal-to-interference-plus-noise ratio (SINR) from ground node u to drone m at time slot t is Expressed as

[0099]

[0100] Among them, gu,m (t) represents the channel gain between ground node u and UAV m at time slot t, g u′,m (t) represents the channel gain between another ground node u′ and UAV m at time slot t, α u,m (t) represents the access control variable between ground node u and UAV m at time slot t, α u′,m′ (t) represents the access control variable between another ground node u′ and another UAV m′ at time slot t, n0 represents the power spectrum density (PSD) of additive white Gaussian noise (AWGN), p u is the transmission power of ground node u on each subchannel, and B is the subchannel bandwidth.

[0101] Considering the environmental information of the uplink transmission, the achievable data rate can be calculated using the Shannon formula. The uplink data rate from ground node u to UAV m at time slot t is Expressed as

[0102]

[0103] According to the above formula, the total throughput obtained by the ground node u and the entire emergency communication network in the past t time slots are respectively expressed as and Where t' = {1, 2, ..., t} is any time slot in the past t time slots, δ is the time slot length; meanwhile, the average throughput of ground node u in the past t time slots is expressed as

[0104] For the downlink, the communication model is described by the downlink signal-to-interference-noise ratio, downlink data rate, and decoding error probability. For the downlink, the command and control information sent to the ground node usually has strict requirements on timeliness and reliability. The QoS requirement of the command and control information can be expressed as the maximum tolerable delay τ max and the maximum decoding error probability ε max To quantify. In addition, the length of the command packet is usually short. In short packet communication, since the law of large numbers does not apply, the result of information theory obtained from Shannon capacity does not apply. Therefore, for any value of the signal-to-noise ratio, the decoding error probability ε is non-zero. In order to accurately characterize the transmission performance of the command packet, this application uses an approximate value of the downlink data rate that can be achieved by short packets.

[0105] Specifically, the downlink signal-to-interference-noise ratio from UAV m to ground node u at time slot t is Expressed as

[0106]

[0107] Among them, α m,u (t) represents the access control variable between UAV m and ground node u at time slot t, α m',u' (t) represents the access control variable between another UAV m′ and another ground node u′ at time slot t, g m,u (t) represents the channel gain between UAV m and ground node u at time slot t, g m',u (t) represents the channel gain between another UAV m′ and the ground node u at time slot t, p m is the transmission power of UAV m on each sub-channel.

[0108] Downlink data rate from UAV m to ground node u at time slot t Expressed as

[0109]

[0110] Among them, τ is the transmission delay of the charge packet, Q -1 (·) is the inverse function of the Q function, ε max is the maximum decoding error probability, V represents the channel dispersion, which can be expressed as

[0111] The end-to-end delay of the command ignores the queuing delay and only considers the transmission delay. For a given packet length S, the maximum tolerable delay τ max Transmit a packet within, this application can obtain When UAV m transmits a command packet to ground node u at time slot t, the decoding error probability of ground node u is expressed as

[0112]

[0113] Among them, τ max represents the maximum tolerable delay, S represents the given packet length; the meaning of the above formula is that, given the packet length S and the maximum tolerable delay τ max Under these conditions, the decoding error probability ε can be calculated by the above formula u (t). Therefore, in order to meet the timeliness and reliability requirements of the command packet, the decoding error probability is set to not exceed the threshold ε max , which can be expressed as ε u (t)≤ε max .

[0114] Step S3, by jointly considering the communication requirements of the uplink for throughput and the downlink for timeliness and reliability, and based on the limited perception model, channel model and communication model, a joint optimization model of access control and UAV trajectory planning problems is constructed.

[0115] Specifically, for the uplink, only pursuing the maximization of the throughput of the emergency communication network may result in only a few ground nodes being able to continuously obtain communication services most of the time, while other nodes are in a communication blind spot. To avoid this situation, the Jain fairness coefficient is introduced to evaluate the fairness of the communication services obtained by the ground nodes. The fairness coefficient is calculated based on the average throughput of each ground node in the time slot before time slot t. At time slot t, the calculation formula of the fairness coefficient is

[0116]

[0117] According to the Cauchy-Schwarz inequality, η(t) always satisfies 1 / U≤η(t)≤1. The higher the fairness coefficient, the smaller the average throughput difference between different ground nodes, and the better the service fairness in the emergency communication network. In order to maximize the throughput of the emergency communication network while ensuring the fairness of ground node services, the fair throughput is defined as the product of the throughput of each time slot and the fairness coefficient. Therefore, the ground node has a service period of The total fair throughput can be expressed as

[0118]

[0119] When the downlink satisfies ε u (t)≤ε max When the delay and reliability requirements of the command packet are met at the same time. In a service cycle, this application hopes to establish more links that meet the communication requirements. Therefore, the number of downlinks satisfying QoS (QoS-DL) is selected as the evaluation index, and its expression is

[0120]

[0121] Among them, 1(·) represents the indicator function, which is specifically expressed as

[0122] Since differentiated communication requirements for uplink and downlink need to be met, the utility of the emergency communication network is defined as the weighted sum of fair throughput and QoS-DL. Both fair throughput and QoS-DL are affected by the location of the UAV and the access control strategy. In order to maximize these two indicators simultaneously, the UAV trajectory planning and access control problem is modeled as

[0123]

[0124] Among them, λ is the weight coefficient, which is used to balance the relative importance of uplink and downlink; the total fair throughput is used to consider the communication requirements of uplink for throughput; QoS-DL is used to consider the communication requirements of downlink for timeliness and reliability.

[0125] The joint optimization model satisfies the following constraints:

[0126]

[0127] Among them, constraint (1) means that only ground nodes within the UAV’s sensing range can access the UAV, α m,u (t) represents the access control variable between UAV m and ground node u at time slot t, For drone collection, is the ground node set, represents the service period; constraint (2) indicates that each ground node can only access one UAV at most; constraint (3) indicates that the number of ground nodes accessed by each UAV cannot exceed the number of subchannels, K is the total number of subchannels; constraint (4) indicates the value range of access control variables and perception variables, where c m,u (t) represents the perception variable between UAV m and ground node u at time slot t; constraint (5) indicates that the position of the UAV cannot exceed the boundary of the target area, where x m (t) and y m (t) represents the horizontal and vertical coordinates of UAV m at time slot t, X and Y represent the horizontal and vertical boundaries of the target area, respectively; constraint (6) means that the distance between any two UAVs is greater than the minimum distance D min , to avoid collision, where q i (t) and q j (t) represent the positions of UAV i and UAV j at time slot t, respectively.

[0128] The joint optimization model is a mixed integer nonlinear programming problem consisting of a continuous variable q and a discrete variable α. Due to the destruction of basic communication facilities, it is usually difficult for the remote control center to obtain the location information of all ground nodes. Therefore, this application needs to design a distributed solution.

[0129] In step S4, the joint optimization model is described as a de-neutralized partially observable Markov process, and the priority-based access control algorithm and the QMIX trajectory planning algorithm enhanced by the weight generation network are used to solve the joint optimization model to obtain the optimal access decision and the optimal trajectory of the UAV.

[0130] The decentralized partially observable Markov decision process (De-POMDP) ​​consists of a five-tuple Composition, of which Indicates the state of the environment. represents the joint action space of M agents, represents local observation information, is the state transfer function, is the reward function. In the Dec-POMDP model, at time slot t, the environment information is represented as Each drone acts as an independent intelligent agent. Get a local observation information from the current environment information s(t) Then, each agent uses local observation information o m (t) Select an action The actions of all agents form a joint action Then, the environment is transformed into a state according to the probability function Transition to next state In the fully cooperative setting, all agents share the same reward function That is, r(t)=r1(t)=…=r M (t). Ultimately, the goal of MADRL is to learn the optimal strategy to maximize the long-term return Among them, γ∈[0,1] represents the discount factor.

[0131] Specifically, the joint optimization model is described as a de-neutralized partially observable Markov process, including:

[0132] In the scenario where multiple drones provide communication services to ground nodes, there are two types of entities: drones and ground nodes. The feature vector of each entity contains environmental information related to its own category. The feature vectors of drone m and ground node u at time slot t are obtained respectively, which are expressed as

[0133]

[0134] Among them, the feature vector of drone m at time slot t is Including the current position q of drone m m (t) and the total number of ground nodes accessed by UAV m in the previous time slot The characteristic vector of ground node u at time slot t Including the position q of the ground node u u , whether the drone was connected in the previous time slot Average uplink throughput up to time slot t-1 and the decoding error probability ε of time slot t-1 u (t-1). and Each feature of has a key impact on the optimization objective of the joint optimization model. Next, the key components of Dec-POMDP will be defined based on the feature vectors of the above entities.

[0135] Specifically, according to the feature vectors of the two entities, the UAV and the ground node, the joint optimization model is described as a de-neutralized partially observable Markov process, which is described by the environment state, the local observation information of the UAV, the UAV action and the reward function.

[0136] The environment state is described by the set of feature vectors of all entities. Specifically, at time slot t, the environment state s(t) contains all entity state information related to the problem solution, which is composed of the feature vectors of the entities. Let and They represent the set of feature vectors of all drones and all ground nodes respectively. Therefore, the environment state can be represented as the set of feature vectors of all entities:

[0137] s(t)=[s UAV (t),s GN (t)].

[0138] The local observation information of the drone is described by the drone’s own feature vector, the feature vector set of neighboring drones within the drone’s perception range, and the feature vector set of ground nodes. Specifically, each drone can only perceive the distance D GN and D UAV Therefore, each drone agent can only observe some ground nodes and exchange their observation information with drones within the perception range. Therefore, at time slot t, drone m observes neighboring drones and ground nodes They are respectively expressed as follows

[0139]

[0140] make represents the eigenvector of the drone θ itself, Represents the set of feature vectors of neighboring drones within the sensing range of drone θ, represents the feature vector set of ground nodes within the sensing range of drone m. Therefore, the local observation information of drone θ can be the features of all three, which can be expressed as

[0141]

[0142] The drone action is described by the action decision of the drone's moving direction and the ID of the ground node accessing the drone. Specifically, in each time slot, the actions that the drone can take include movement and access control of ground nodes. Since the drone flies at a constant speed v and a fixed height H, the movement of the drone can be simplified to movement in a certain direction in a two-dimensional plane. This application discretizes the horizontal movement direction, and the movement direction is expressed as Ψ = {0, π / 2, π, 3π / 2}. In addition, the scheduled ground node must be a ground node within the perception range. The action of drone θ in time slot t can be expressed as

[0143]

[0144] in, represents the action decision of the moving direction of UAV m, represents the ID of the ground node connected to UAV m. The displacement of UAV m in this time slot is

[0145] △q m (t)=(△x m (t),△y m (t),0)=(δ×vcosψ, δ×vsinψ,0).

[0146] The reward function is described by a global reward and individual penalty values ​​of drones. The global reward is the weighted sum of the total fair throughput and QoS-DL. Specifically, all drones have a common goal, which is to improve the utility of the entire emergency communication network. Therefore, this application selects the sum of the utilities generated by all drones as the global reward. Similar to the optimization goal of the joint optimization model, considering the differentiated communication requirements of uplink and downlink, the weighted sum of the two is selected as the reward. At time slot t, the global reward can be expressed as the weighted sum of the total fair throughput and QoS-DL, expressed as

[0147]

[0148] In addition, in order to avoid collisions between drones and considering that drones need to always fly within the service area, a binary vector ξ(t)∈{0,1} is defined to describe the situation where a drone collides or flies out of the service area. m (t) indicates whether UAV m is beyond the collision or beyond the area at time slot t. m If (t) = 1, a penalty value μ is added to the reward value, otherwise the penalty value is 0. In summary, the complete reward function is obtained by combining the global reward and the penalty value of the individual drone, which can be expressed as

[0149] r(t)=rg (t)-ξ m (t)·μ.

[0150] The priority-based access control algorithm includes: during the drone service, there may be a situation where the ground node u is within the perception range of multiple drones, and the number of nodes within the perception range of drone m exceeds the number of available channels. Therefore, it is necessary to formulate an effective access control strategy to meet the optimization goals and ensure the optimization of communication services under limited resources. The core idea of ​​the access control strategy proposed in this application is: in each time slot, each ground node gives priority to accessing the nearest drone, and each drone gives priority to providing services to nodes with higher priority.

[0151] For ground nodes, in each time slot, each ground node gives priority to accessing the nearest UAV. Specifically, when a ground node u is within the coverage of multiple UAVs, it will give priority to accessing the UAV m closest to itself. ★ Send access request information, that is

[0152]

[0153] in, Represents the set of drones within the sensing range of the ground node u.

[0154] For drones, it is necessary to prioritize ground nodes by comprehensively considering the throughput requirements of the uplink and the QoS requirements of the downlink. For the uplink, considering the throughput requirements of the ground nodes and service fairness, the node with the lowest throughput at the current moment is set to be served first. For the downlink, considering the requirements of the command information for transmission reliability and timeliness, the ground nodes closer to the drone have greater channel gain and are more inclined to meet the above two requirements. Specifically, the throughput priority principle is adopted for the uplink, that is, in the first t time slots, the ground nodes with less throughput are given priority to access the drone; the distance priority principle is adopted for the downlink, that is, the ground nodes with the closest distance are given priority to access the drone. In order to optimize the consistency of the goals, the priority is also defined in the form of weighted sum. In addition, since the dimensions of "throughput" and "distance" are different, the two are normalized. The throughput of the ground nodes within the perception range of drone m is divided by the maximum throughput. Similarly, the distance is also divided by the maximum distance from the ground node to the drone. By taking the weighted sum of the normalized throughput and distance, the priority ranking of the drone for the set of ground nodes within the perception range is obtained, which is expressed as

[0155]

[0156] Among them, P m is the set of ground nodes within the sensing range of UAV m sort(·) indicates that each UAV sorts the ground nodes according to the weighted sum of the normalized throughput and distance. The ground nodes with higher priority are given priority to access the UAV, and for each ground node connected to the UAV, the corresponding UAV will allocate the subchannel with the least uplink and downlink interference at the current moment.

[0157] The architecture of the QMIX trajectory planning algorithm enhanced by the weighted generation network is as follows Figure 4 shown.

[0158] Among them, the Multilayer Perceptron (MLP) has a strong ability in data fitting, but it is difficult to adapt to the change of input dimension. Therefore, this application designs an automatic weight generation mechanism, hoping to achieve dynamic adjustment of the weight matrix to adapt to the change of the number of input elements. i Each corresponds to a sub-weight matrix w i In order to make the dimension of the weight matrix change synchronously with the number of input elements, construct a Figure 3 As shown in the figure, the weight generation network (WGN) consists of a single hidden layer. WGN transforms each entity feature x i As input, generate the corresponding sub-weight matrix w i . To this end, the sub-weight matrix can be expressed as

[0159]

[0160] Furthermore, the output of the entire network can be expressed as

[0161]

[0162] Where L is the number of entities.

[0163] like Figure 3 As shown, the features of all entities are fed into a shared neural network, and after forward propagation, the output weight matrix w i Then, each weight matrix is ​​associated with the corresponding input feature x i Perform matrix multiplication (matmul). Aggregate all outputs using sum operation to get feature representation y. It is worth noting that the weight generation network The input and output dimensions of the network only depend on the feature dimensions of each entity, not on the number of input entities. Therefore, the weight generation network can flexibly handle the dynamic change of input dimensions caused by the change of the number of input entities, thereby maintaining the robustness and generalization ability of the network. In addition, the weight generation network can generate a weight matrix based on the feature vector of each entity to measure the impact of each observed entity on the current agent.

[0164] Figure 4 As shown in the figure, the QMIX trajectory planning algorithm enhanced by the weight generation network consists of several intelligent agent networks, a hybrid network and a super network.

[0165] Among them, the agent network is used to obtain the local observation information of the corresponding drone. m , fitting and outputting the individual value Q of the UAV agent m According to the observation vector, each drone’s observation includes the feature vectors of neighboring drones and ground nodes within the sensing range. m The dimension will change with the change of the number of entities within the perception range. The agent network consists of three parts: the input layer, the middle layer and the output layer. In the classic QMIX algorithm, the agent network uses MLP to process observation information. However, as mentioned earlier, MLP is difficult to deal with the problem of local observation dimensions. In order to solve this problem, this application selects WGN to generate weights for features with a variable number of entities. Among them, the input layer is composed of two weight generation networks and a multi-layer perceptron. The two weight generation networks are used to extract features from the feature vector sets of neighboring drones within the perception range of the drone in the local observation information of the drone and the feature vector sets of ground nodes. In addition, since the observation vector dimension of the agent itself has always been unchanged, this application still uses MLP to extract features from the feature vectors of the drone itself in the local observation information of the drone. The input layer finally merges the outputs of the two weight generation networks and a multi-layer perceptron to output an embedded feature. This process can be expressed as

[0166]

[0167] in, They are functions implemented by the neural network for the corresponding entity types.

[0168] In order to keep the memory of historical observations and actions, the middle layer uses a gated recurrent unit (GRU) with the hidden state h of the previous time slot m (t-1) and the embedded features e output by the input layer m (t) is taken as input and outputs the hidden state h of the current time slot m(t). The GRU unit introduces historical information by establishing a dependency relationship between each time slot and the previous time slot. Based on historical information, each drone agent can better infer changes in the global environment state and make better decisions. This process can be expressed as

[0169]

[0170] in, Represents the function implemented by GRU.

[0171] The output layer uses a multi-layer perceptron, which obtains the hidden state h of the current time slot m (t) is calculated and the output is the individual value of the drone, which is expressed as

[0172]

[0173] in, represents the output function implemented by MLP, θ a Represents the network parameters of the intelligent agent. According to the obtained individual value Q m , the agent uses the ∈-greedy strategy to make action decisions, which can be expressed as the following formula

[0174]

[0175] After each agent chooses an action, the utility of each agent to the environment is calculated. Then, the reward r(t) is calculated according to the above reward function, and the environment will transfer to the next state s(t+1).

[0176] During the training phase, the hybrid network is used to transform the individual value Q of each drone m As input, the joint action value Q is obtained through forward propagation tot As a special neural network, the hypernetwork is used to calculate the weights and biases of the hybrid network according to the state of the environment. Specifically, the hybrid network is a two-layer feedforward neural network that converts the local action value Q of each agent into m (o m (t),a m (t);θ a ) as input, and the joint action value Q is obtained through forward propagation tot (s(t),a(t);θ mix ,θ h ), where θ mix and θ h Denote the parameters of the hybrid network and the hypernetwork respectively. The joint action value Q tot represents the expected long-term cumulative return achieved under the joint strategy. In contrast, the individual value Q mQuantifies the contribution of each agent to this cumulative reward. tot While more accurately approximating the true joint action value, the hybrid network can more effectively adjust individuals via back-propagation.

[0177] In order to ensure the consistency between the global optimal strategy and the local optimal strategy of a single agent, the IGM (Individual-Global-Max) criterion needs to be met:

[0178]

[0179] The above formula shows that the joint action value Q tot and individual value Q m There is a monotonic relationship between tot And each Q m The following monotonicity constraints also need to be met:

[0180]

[0181] Specifically, Figure 4 As shown, through the design of hybrid network and super network, QMIX ensures the joint action value Q tot is the individual value Q of each agent m A non-negative monotonic function of . Hypernetwork Take the environment state s(t) as input and generate parameters θ for the hybrid network mix ={w1,w2,b1,b2}, where w1,w2,b1,b2 represent the weights and biases of the first and second layers of the hybrid network respectively. Therefore, the present application can obtain The values ​​output by the hypernetwork are activated by the absolute value function to obtain weights w1 and w2. This means that w1 and w2 are non-negative, thus ensuring the monotonicity constraint. The bias b1 is generated by a single-layer MLP, while the bias b2 is generated by a two-layer MLP with a ReLU activation function. Finally, the joint action value function can be expressed as

[0182] Q tot (s(t),a(t);θ mix ,θ h )=w2·φ ELU (w1·Q+b1)+b2;

[0183] where Q = [Q1(o1(t),a1(t);θ a ),…,Q M (o M (t),a M (t);θ a )],φ ELU (·) represents the ELU activation function.

[0184] Furthermore, the present application adopts the DQN (deep Q network) method to train all networks, that is, to train two neural networks of the same structure at the same time, one of which is an online network responsible for real-time evaluation of Q values, and its weights are updated every time slot; the other is a target network responsible for calculating the target Q value and periodically synchronizing weights with the online network. The parameters of the target network's agent network, hybrid network, and super network are respectively expressed as All networks are trained by minimizing the following loss function:

[0185]

[0186] in, Represents a sample randomly sampled from the playback buffer. Represents the global target Q value.

[0187] The above-mentioned priority-based access control algorithm and weight-generated network-enhanced QMIX trajectory planning algorithm are collectively referred to as the PW-QMIX algorithm (priority-based access control and WGN-enhanced QMIX-basedtrajectory planning algorithm). The PW-QMIX algorithm can achieve efficient communication services for ground nodes by dynamically selecting the access node for each time slot and the moving direction of the drone in each time slot. The process of the PW-QMIX algorithm is as follows Figure 5 As shown, the specific steps are as follows:

[0188] Step 1. Parameter initialization: Randomly initialize the online network parameters θ a ,θ mix ,θ h and target network parameters Initialize the playback buffer

[0189] Step 2: Initialize the environment state: Before entering each round, initialize the environment state s(t).

[0190] Step 3: Obtain local observations: In the training round, each agent obtains its local observation information o at each time slot t m (t).

[0191] Step 4: Make action decisions: Each agent makes action decisions on the direction of movement according to the ∈-greedy strategy

[0192] Step 5: Calculate the joint action value: Combine the environment state s(t) and the individual value Q of each drone agent m Input into the hybrid network to calculate the joint action value Q tot .

[0193] Step 6: Make access control decisions: Each ground node calculates the distance of the drones within the sensing range; then selects the drone closest to it, i.e., m ★ ; Each UAV calculates the priority P of the ground nodes within the sensing range m ; If the UAV has an idle channel, a sub-channel is allocated to the ground node; After all UAVs have made control decisions, the access control variable α is updated.

[0194] Step 7: Execute actions and transfer the environment state: Each agent performs movement actions and access actions. The environment state transfers from s(t) to the next state s(t+1), and each agent updates the local observation information o m (t+1).

[0195] Step 8. Calculate the global reward: Calculate the reward function r(t).

[0196] Step 9: Store experience samples: End a training round and store experience samples of one round in the playback buffer.

[0197] Step 10: Update online network parameters: If the playback buffer size Larger than the mini-batch size The sampling size from the playback buffer is samples; then update the online network parameters according to the loss function.

[0198] Step 11: Update the target network parameters: If the target network parameter update frequency is reached, the parameters of the online network are assigned to the target network.

[0199] Step 12: Output the agent network parameters: After the training is completed, output the parameters of the agent network.

[0200] Furthermore, in order to verify the beneficial effects of the present application, the PW-QMIX algorithm proposed in the present application was compared with other algorithms, and a series of simulation experiments were conducted to verify the significance of the modeled joint optimization problem and the superiority of the proposed algorithm.

[0201] The network simulation scenario is set as follows: UAV nodes are randomly deployed in a rectangular area of ​​1500m×1500m. The area is divided into four non-overlapping hotspot areas, each containing an equal number of ground nodes. At the beginning of each service period, the positions of U ground nodes and M UAVs are reset and randomly distributed within the area. The positions of the ground nodes remain unchanged during each service period. The service period is divided into 150 time slots. At the end of each time slot, the UAV makes decisions on its flight direction and ground node access control until the end of the service period. The parameters of the path loss are set according to the urban scene model. The environmental parameters a and b are set to 9.61 and 0.16, respectively, and the additional path loss η LoS , η NLoS Set to 1dB, 20dB, path loss index ζ is 2, drone speed v is 10m / s, flight altitude H is 100m, carrier frequency f c is 2GHz, the subchannel bandwidth B is 200KHz, the time slot length δ is 10s, the number of subchannels K is 5, the power spectral density of additive Gaussian white noise n0 is -174dbm / Hz, and the maximum tolerable delay τ max is 2ms, the maximum decoding error probability ε max For 10 -5 , the size of the charge packet S is 64Bytes (bytes), the transmission power of the drone and the ground node is p m 、p u They are 0.5W and 0.2W respectively. The sensing range of the drone and the ground node is D UAV , D GN The safe distance between drones is 200m and 200m respectively. min is 5m.

[0202] First, the convergence is verified: In order to illustrate the convergence of the PW-QMIX algorithm, this application simulates the relationship between long-term rewards and training rounds under different learning rate settings. The long-term reward convergence curve under different learning rate conditions is as follows: Figure 6 As shown in the figure, with the increase of training rounds, the rewards of all cases gradually increase. When lr = 0.001, 0.0001, the two cases achieved convergence after 15,000 and 30,000 training rounds respectively. However, the training efficiency of the case of lr = 0.00001 is relatively low. Considering that the case of lr = 0.001 converges to a higher reward value faster, this application sets the learning rate lr = 0.001 to implement the following simulation.

[0203] Comparison and verification for different access methods: This application compares the proposed PW-QMIX algorithm with the QMIX algorithm alone for trajectory and node access decision making. The decision performance comparison results of the PW-QMIX algorithm and the QMIX algorithm are as follows: Figure 7As shown, from Figure 7 It can be seen that the priority-based access control algorithm converged after about 15,000 episodes of training, while QMIX did not converge. This is because the QMIX algorithm is completely based on the reinforcement learning algorithm to make decisions on node access and drone trajectories. The huge state-action space makes it impossible for the algorithm to learn effective strategies within a limited number of rounds. Therefore, the algorithm has not converged. In contrast, the priority-based access control algorithm proposed in this application uses a heuristic method to make node access. Compared with the QMIX algorithm, which only needs to make decisions on trajectory optimization, the algorithm proposed in this application greatly reduces the state-action space. Therefore, the algorithm proposed in this application is significantly better than using only the QMIX algorithm.

[0204] Verification of dynamic observation processing capabilities: In order to verify the processing capabilities of the algorithm proposed in this application for local observation problems, ablation experiments were first carried out under different node numbers. In addition, this application also compares the proposed PW-QMIX algorithm with other SOTA (existing state-of-the-art) algorithms. Set the number of drones to 4, and the number of ground nodes to 20, 40, and 60, respectively. MLP is selected as the input layer of QMIX to process observation features as a benchmark algorithm. And only the priority-based access control algorithm proposed in this application is adopted, denoted as P-QMIX. Then, this application compares PW-QMIX with P-QMIX to verify the performance with / without local observation processing schemes. For ground node access, both algorithms adopt the priority-based access control algorithm proposed in this application.

[0205] Figure 8 The convergence curves of the schemes with and without dynamic observation processing under different ground node numbers are described. Figure 8 As can be seen from the figure, all algorithms have achieved convergence with the increase of training rounds. Regardless of the number of nodes being 20, 40 or 60, PW-QMIX has a higher return than P-QMIX without local observation processing. This is because WGN dynamically generates a weight matrix for the features of each entity, which enhances the representation ability of the model. In particular, as the number of nodes increases, the performance advantage of PW-QMIX becomes more obvious. Specifically, when the number of nodes is 20, 40, and 60, the return of PW-QMIX is about 6%, 9%, and 13% higher than that of P-QMIX, respectively. This is because, in an area of ​​limited size, when the number of nodes increases, the nodes observed by the local observation of the drone without local observation processing scheme in each time slot will change significantly due to the denser nodes within the perception range. The inefficient MLP structure is difficult to effectively handle the high dynamic changes of the input features. On the contrary, PW-QMIX can make better and more reasonable decisions in each time slot, improving the quality of service provided by drones to ground nodes.

[0206] In order to further evaluate the performance of WGN, this application further selected three commonly used dynamic observation processing schemes for comparison, including Attention (attention mechanism), GNN (graph neural network) and Sets (set mechanism). The specific introduction is as follows:

[0207] (1) PA-QMIX: This algorithm uses the Attention mechanism to enhance the input layer of QMIX. The attention mechanism processes input features through flexible weighted summation. The main advantage of the attention mechanism is that it can operate independently of the fixed input dimension. This allows it to dynamically assign weights to the features of any number of entities, effectively solving the problem of dynamic observation.

[0208] (2) PG-QMIX: This algorithm is the result of integrating GNN into QMIX. GNN uses a graph structure to model drones and ground nodes and their relationships. It is inherently robust to changes in the number of entities because its performance is not affected by changes in the input dimension.

[0209] (3) PS-QMIX: This algorithm is obtained by integrating the Sets mechanism into QMIX. The Sets mechanism uses a shared neural network to extract feature representations of entities. These feature representations are then aggregated through a summation operation for subsequent processing.

[0210] In addition, the above baseline adopts the priority-based access control algorithm proposed in this application. The number of ground nodes is set to 80.

[0211] Fig. 9 The convergence curves of different observation processing schemes such as PW-QMIX, PA-QMIX, PG-QMIX, PS-QMIX and P-QMIX are shown. Fig. 9 As shown in the figure, the algorithms with dynamic observation processing schemes are better than P-QMIX. Specifically, PW-QMIX achieves the best convergence performance, with a final return value of about 1110, which is about 5.1% higher than PA-QMIX and PS-QMIX, and about 20% higher than the less efficient P-QMIX and PG-QMIX.

[0212] Table 1 Results of different observation processing schemes

[0213] algorithm Fair Throughput (Gbits) QoS-DL P-QMIX 9.58 2391 PA-QMIX 11.4 2670 PS-QMIX 11.0 2722 PG-QMIX 8.91 2415 PW-QMIX 12.2 2773

[0214] As shown in Table 1, this performance improvement stems from the enhancement of uplink fair throughput and QoS-DL. First, PW-QMIX achieves the highest values ​​in all indicators, verifying its overall advantage in improving system utility. Second, PW-QMIX exhibits faster convergence. It converges after about 15,000 rounds, while PA-QMIX and PS-QMIX require about 25,000 rounds to achieve convergence. In addition, PW-QMIX exceeds the convergence values ​​of all other algorithms after only 10,000 training times. These results show that WGN has better convergence performance in dynamic observation processing.

[0215] Verification of the importance of the optimization target: In order to verify the importance of the joint optimization of uplink and downlink, the present application changes the weight factor λ in the optimization target. The present application sets λ to 0, 0.5, and 1, respectively, indicating the cases of considering only the downlink, considering both the uplink and downlink, and considering only the uplink. Considering that the QoS requirement of the downlink is a key factor affecting the optimization target, the present application performs simulations under different charge packet lengths. Fig.10 Schematic diagram of the relationship between λ and fair throughput under different charge packet lengths. Fig.11 Schematic diagram of the relationship between λ and QoS-DL under different charge packet lengths.

[0216] from Fig.10 and Fig.11It can be seen that as λ increases, the fair throughput of all packet lengths improves and QoS-DL gradually decreases. These results show that when only one type of link is optimized, the performance of the unconsidered links will degrade significantly. Specifically, when the metric of the considered link reaches its optimal value, the metric of the unconsidered link reaches its worst. However, for the disaster relief scenario considered in this application, it is unacceptable to focus on only one type of link. Joint optimization of uplink and downlink is a more reasonable choice. Compared with optimizing only for one type of link requirements, joint optimization may result in performance loss of one type of metric. However, considering the gains achieved, this loss is acceptable. For example, when S=64Bytes, the fair throughput of λ=0.5 is about 9% lower than that of λ=1, but about 37.6% higher than that of λ=0. Similarly, the QoS-DL at λ=0.5 is about 0.6% lower than that at λ=0, but 183% higher than that at λ=1. In addition, the length of the charge packet has a key impact on the system utility. For example, when λ=0.5, as the command packet length increases, the metrics of both the uplink and downlink decrease. This is because larger command packets require higher quality links to meet the quality of service requirements. Since the drone makes decisions based on a comprehensive consideration of both links, changes in the downlink requirements will also affect the uplink quality of service. Through this simulation, the application analyzes the relationship between system utility and λ under different command packet lengths, and verifies the importance of joint uplink and downlink optimization.

[0217] Comparative verification of the scalability of the algorithm: Further analyze the scalability of the PW-QMIX algorithm proposed in this application under the conditions of different numbers of intelligent agents. The VDN (Value Decomposition Network) algorithm is a classic value-based algorithm that is widely used to solve multi-agent collaboration tasks. However, different algorithms give different collaborative capabilities to agents. Therefore, this simulation selects the VDN algorithm as a comparison algorithm to verify the scalability of the proposed algorithm in multi-UAV collaboration. To ensure a fair comparison, the VDN algorithm also uses the same dynamic observation processing scheme as PW-QMIX. Therefore, the algorithm is named PW-VDN. The number of ground nodes is set to 80.

[0218] In order to demonstrate the scalability of the proposed algorithm, this application compares the two algorithms under different numbers of drones. Fig.12 and Fig.13The results of the number of drones, fair throughput, and QoS-DL for the PW-QMIX and PW-VDN algorithms are shown respectively. When M = 4, 6, 8, 10, the fair throughput of PW-QMIX is about 4.3%, 3.2%, 8.6%, and 10.1% higher than that of PW-VDN, respectively. At the same time, the number of downlinks that meet QoS requirements is also higher than that of PW-VDN. This result directly verifies the advantage of the PW-QMIX algorithm in scalability.

[0219] In addition, as far as a single algorithm is concerned, as the number of drone agents increases, the fair throughput of the two algorithms also gradually increases. However, the growth rate is gradually decreasing. For example, when the number of drones increases from 4 to 6, the fair throughput of PW-QMIX and PW-VDN increases by about 15.5% and 16.8%, respectively. When it increases from 6 to 8, the fair throughput of the two algorithms increases by about 14% and 8.9%, respectively. When it increases from 8 to 10, it increases by about 7.4% and 6.1%. The growth rate of fair throughput has decreased to varying degrees. On the one hand, due to the increase in the number of drones, the number of access nodes also increases, which in turn leads to an increase in interference on the same channel. On the other hand, the increase in the number of drones leads to an exponential increase in the state space, and the cooperation between drones becomes more complicated. Therefore, the performance of the decision-making has declined. Similarly, QoS-DL has a similar trend with fair throughput.

[0220] In summary, the present application comprehensively considers the differentiated communication requirements of the emergency communication system composed of drones and ground nodes. In view of the communication requirements of uplink throughput and downlink timeliness and reliability, the present application models a multi-drone trajectory planning and access control problem. By optimizing the trajectory of the drone and the nodes accessed, the fair throughput and QoS-DL of the system are maximized. Then, the present application proposes a priority-based access control algorithm and a QMIX trajectory planning algorithm enhanced by a weight generation network. The priority-based access control algorithm realizes node access through a heuristic method, which effectively reduces the state-action space. At the same time, the weight generation network is used to improve QMIX, which solves the problem of low training efficiency caused by dynamic observation. The simulation results verify the significance of the optimization goal of this application. At the same time, compared with other algorithms, the algorithm proposed in this application can converge to better results at a faster speed and has stronger scalability.

[0221] It should be understood that although Figure 1 and Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 and Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0222] In one embodiment, an access control and trajectory planning device for an emergency communication network is provided, comprising:

[0223] A network construction module is used to build a multi-UAV-assisted emergency communication network and obtain uplinks and downlinks within the network; the emergency communication network consists of multiple UAVs, multiple ground nodes, and a remote control center;

[0224] A model building module is used to build a limited perception model, channel model and communication model between the UAV and the ground node based on the emergency communication network;

[0225] The problem modeling module is used to construct a joint optimization model of access control and UAV trajectory planning problems by jointly considering the communication requirements of uplink throughput and downlink timeliness and reliability, and based on the limited perception model, channel model and communication model;

[0226] The joint solution module is used to describe the joint optimization model as a de-neutralized partially observable Markov process, and adopts the priority-based access control algorithm and the QMIX trajectory planning algorithm enhanced by the weight generation network to solve the joint optimization model to obtain the optimal access decision and the optimal trajectory of the UAV.

[0227] For the specific limitations of the access control and trajectory planning device for emergency communication networks, please refer to the limitations of the access control and trajectory planning method for emergency communication networks mentioned above, which will not be repeated here. Each module in the above-mentioned access control and trajectory planning device for emergency communication networks can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0228] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0229] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An access control and trajectory planning method for an emergency communication network, characterized in that: The method comprises: Constructing a multi-UAV-assisted emergency communication network and obtaining uplinks and downlinks within the network; wherein the emergency communication network is composed of multiple UAVs, multiple ground nodes, and a remote control center; Based on the emergency communication network, a limited perception model, a channel model and a communication model between the UAV and the ground node are constructed; By jointly considering the communication requirements of the uplink for throughput and the downlink for timeliness and reliability, and based on the limited perception model, channel model and communication model, a joint optimization model of access control and UAV trajectory planning problems is constructed; The joint optimization model is described as a deneutralized partially observable Markov process, and a priority-based access control algorithm and a weight generation network-enhanced QMIX trajectory planning algorithm are used to solve the joint optimization model to obtain the optimal access decision and the optimal trajectory of the UAV.

2. The method according to claim 1, characterized in that In the emergency communication network, each ground node is used to collect and obtain environmental information and transmit it to the remote control center. The remote control center is used to make decisions based on the received environmental information and send command and control information including mobile control and mission instructions to the ground node. Each drone serves as an aerial base station and is used to forward the transmitted information between the ground node and the remote control center. The uplink in the network is defined as the link for transmitting environmental information from the ground node to the drone, and the downlink is defined as the link for transmitting command and control information from the remote control center to the ground node.

3. The method according to claim 1, characterized in that The finite perception model is described by the perception variables between the UAV and the ground node. The perception variables between the UAV m and the ground node u at time slot t are defined as c m,u (t), and c m,u (t)∈{0,1}; if the ground node u is within the sensing range of the drone m at time slot t, c m,u (t) = 1; otherwise c m,u (t)=0.

4. The method according to claim 1, characterized in that: The channel model is described by the channel gain between the ground node and the UAV. The calculation process of the channel gain is: According to the pitch angle from the ground node u to the UAV m at time slot t, the probability of the line-of-sight (LoS) link and the non-line-of-sight (NLoS) link between the ground node u and the UAV m is obtained, which are expressed as in, represents the probability of the LoS link between ground node u and UAV m at time slot t, represents the probability of the NLoS link between the ground node u and the UAV m at time slot t, a and b are parameters determined by the environment type, represents the pitch angle from ground node u to UAV m at time slot t, H represents the fixed altitude of the UAV, and q u is the position of ground node u, q m (t) is the position of UAV m at time slot t; According to the positions of the ground node u and the UAV m at time slot t, the path loss between the ground node u and the UAV m is obtained, which is expressed as Among them, η ξ represents the average additional path loss of the LoS link or NLoS link, f c is the carrier frequency, c is the speed of light, and ξ represents a LoS link or a NLoS link; according to And the path loss corresponding to each link is calculated to obtain the channel gain between the ground node u and the UAV m, which is expressed as in, represents the path loss of the LoS link between ground node u and UAV m at time slot t, represents the path loss of the NLoS link between the ground node u and the UAV m at time slot t.

5. The method according to claim 1, characterized in that The communication model includes: For the uplink, the communication model is described by uplink signal-to-interference-noise ratio, uplink data rate and throughput; where the uplink signal-to-interference-noise ratio from ground node u to drone m at time slot t is Expressed as Among them, g u,m (t) represents the channel gain between ground node u and UAV m at time slot t, g u′,m (t) represents the channel gain between another ground node u′ and UAV m at time slot t, α u,m (t) represents the access control variable between ground node u and UAV n at time slot t, α u′,m′ (t) represents the access control variable between another ground node u′ and another UAV m′ at time slot t, n0 represents the power spectral density of additive Gaussian white noise, p u is the transmission power of ground node u on each subchannel, B is the subchannel bandwidth, For drone collection, is the ground node set; Uplink data rate from ground node u to UAV m at time slot t Expressed as The total throughputs obtained by the ground node u and the entire emergency communication network in the past t time slots are expressed as and Where t' = {1, 2, ..., t} is any time slot in the past t time slots, δ is the time slot length; meanwhile, the average throughput of ground node u in the past t time slots is expressed as U is the number of ground nodes; For the downlink, the communication model is described by downlink signal-to-interference-noise ratio, downlink data rate and decoding error probability; where the downlink signal-to-interference-noise ratio from drone m to ground node u at time slot t is Expressed as Among them, α m,u (t) represents the access control variable between UAV m and ground node u at time slot t, α m',u' (t) represents the access control variable between another UAV m′ and another ground node u′ at time slot t, g m,u (t) represents the channel gain between UAV m and ground node u at time slot t, g m',u (t) represents the channel gain between another UAV m′ and the ground node u at time slot t, p m is the transmission power of UAV m in each sub-channel; Downlink data rate from UAV m to ground node u at time slot t Expressed as Among them, τ is the transmission delay of the charge packet, Q -1 (·) is the inverse function of the Q function, ε max is the maximum decoding error probability, V represents the channel dispersion, which can be expressed as When UAV m transmits a command packet to ground node u at time slot t, the decoding error probability of ground node u is expressed as Among them, τ max represents the maximum tolerable delay, S represents the given packet length; and ε u (t)≤ε max .

6. The method according to claim 1, characterized in that The joint optimization model of access control and UAV trajectory planning problem is expressed as The joint optimization model is a mixed integer nonlinear programming problem consisting of a continuous variable q and a discrete variable α, where λ is a weight coefficient used to balance the relative importance of the uplink and downlink. Indicates the service period of the ground node in the UAV The total fair throughput is used to consider the communication demand for throughput of the uplink; service period It is divided into T time slots of equal length, each time slot length is δ; represents the fairness coefficient at time slot t, is the average throughput of ground node u in the past t time slots, and U represents the number of ground nodes; represents the uplink data rate of ground node u at time slot t; represents the number of downlinks that meet QoS (QoS-DL), which is used to consider the communication requirements of downlinks for timeliness and reliability; 1(·) represents an indicator function, which is specifically expressed as ε max is the maximum decoding error probability, ε u (t) represents the decoding error probability of ground node u at time slot t; M represents the number of UAVs; The joint optimization model meets the following constraints: Among them, constraint (1) means that only ground nodes within the UAV’s sensing range can access the UAV, α m,u (t) represents the access control variable between UAV m and ground node u at time slot t, For drone collection, is the ground node set, represents the service period; constraint (2) indicates that each ground node can only access one UAV at most; constraint (3) indicates that the number of ground nodes accessed by each UAV cannot exceed the number of subchannels, K is the total number of subchannels; constraint (4) indicates the value range of access control variables and perception variables, where c m,u (t) represents the perception variable between UAV m and ground node u at time slot t; constraint (5) indicates that the position of the UAV cannot exceed the boundary of the target area, where x m (t) and y m (t) represents the horizontal and vertical coordinates of UAV m at time slot t, X and Y represent the horizontal and vertical boundaries of the target area, respectively; constraint (6) means that the distance between any two UAVs is greater than the minimum distance D min , to avoid collision, where q i (t) and q j (t) represent the positions of UAV i and UAV j at time slot t, respectively.

7. The method according to claim 6, characterized in that The joint optimization model is described as a de-neutralized partially observable Markov process, including: The feature vectors of UAV m and ground node u at time slot t are obtained respectively, which are expressed as Among them, the feature vector of drone m at time slot t is Including the current position q of drone m m (t) and the total number of ground nodes accessed by UAV m in the previous time slot The characteristic vector of ground node u at time slot t Including the position q of the ground node u u , whether the drone was connected in the previous time slot Average uplink throughput up to time slot t-1 and the decoding error probability ε of time slot t-1 u (t-1); According to the feature vectors of the two entities, the UAV and the ground node, the joint optimization model is described as a de-neutralized partially observable Markov process, which is described by the environment state, the local observation information of the UAV, the UAV action and the reward function; wherein the environment state is described by the feature vector set of all entities; the local observation information of the UAV is described by the feature vector of the UAV itself, the feature vector set of neighboring UAVs within the perception range of the UAV and the feature vector set of ground nodes; the UAV action is described by the action decision of the UAV's moving direction and the ID of the ground node connected to the UAV; the reward function is described by the global reward and the individual penalty value of the UAV, and the global reward is the weighted sum of the total fair throughput and QoS-DL.

8. The method according to claim 1, characterized in that Priority-based access control algorithms, including: For ground nodes, in each time slot, each ground node gives priority to accessing the nearest UAV; For drones, the throughput priority principle is adopted for the uplink, that is, in the first t time slots, the ground nodes with less throughput are given priority to access the drone; the distance priority principle is adopted for the downlink, that is, the ground nodes with the closest distance are given priority to access the drone; by taking the weighted sum of the normalized throughput and distance, the priority ranking of the drone for the set of ground nodes within the sensing range is obtained, which is expressed as Among them, P m is the set of ground nodes u within the sensing range of UAV m m , sort(·) indicates that each UAV sorts the ground nodes according to the weighted sum of the normalized throughput and distance. The ground nodes with higher priority are preferentially connected to the UAV, and for each ground node connected to the UAV, the corresponding UAV will allocate the subchannel with the least interference in the uplink and downlink at the current moment to it; Γ u (t) is the total throughput obtained by ground node u in the past t time slots, is the maximum value of the total throughput, λ is the weight coefficient, q u is the position of ground node u, q m (t) is the position of UAV m at time slot t, and u′ represents the set Any ground node in Assemble for drones.

9. The method according to claim 1, characterized in that: The QMIX trajectory planning algorithm based on weighted generation network enhancement consists of several agent networks, a hybrid network and a super network; The agent network is used to obtain the local observation information of the corresponding drone and fit the individual value of the drone agent. The agent network consists of an input layer, an intermediate layer and an output layer. The input layer consists of two weight generation networks and a multi-layer perceptron. The two weight generation networks are used to extract features from the feature vector sets of neighboring drones within the drone's perception range and the feature vector sets of ground nodes in the drone's local observation information. The multi-layer perceptron is used to extract features from the drone's own feature vectors in the drone's local observation information. The input layer finally outputs an embedded feature by merging the outputs of the two weight generation networks and a multi-layer perceptron. The intermediate layer uses a gated recursive unit, which takes the hidden state of the previous time slot and the embedded features output by the input layer as input, and outputs the hidden state of the current time slot. The output layer uses a multi-layer perceptron, which calculates by obtaining the hidden state of the current time slot and outputs the individual value of the drone. The drone uses the ∈-greedy strategy to make action decisions based on its own individual value. The hybrid network is used to take the individual value of each drone as input and obtain the joint action value through forward propagation; The super network is used to calculate and obtain the weights and biases of the hybrid network according to the environmental state.

10. An access control and trajectory planning device for an emergency communication network, characterized in that: The device comprises: A network construction module, used to construct a multi-UAV-assisted emergency communication network and obtain uplinks and downlinks within the network; wherein the emergency communication network is composed of multiple UAVs, multiple ground nodes and a remote control center; A model building module, used to build a limited perception model, a channel model and a communication model between the UAV and the ground node based on the emergency communication network; A problem modeling module, configured to construct a joint optimization model of access control and UAV trajectory planning problems by jointly considering the communication requirements of the uplink for throughput and the downlink for timeliness and reliability, and based on the limited perception model, channel model and communication model; The joint solution module is used to describe the joint optimization model as a de-neutralized partially observable Markov process, and adopt a priority-based access control algorithm and a QMIX trajectory planning algorithm enhanced by a weight generation network to solve the joint optimization model to obtain the optimal access decision and the optimal trajectory of the UAV.

Images