Disaster area emergency unmanned aerial vehicle networking method based on reinforcement learning

Abstract

The invention relates to the technical field of unmanned aerial vehicle dispatching, in particular to a disaster area emergency unmanned aerial vehicle networking method based on reinforcement learning, which comprises the following steps: taking an unmanned aerial vehicle as an intelligent agent, and taking horizontal and vertical coordinates, speed, direction, residual energy consumption and user throughput of the unmanned aerial vehicle as an observation space of the unmanned aerial vehicle; taking the speed scheduling scheme and the bandwidth allocation scheme of the unmanned aerial vehicle as an action space of the unmanned aerial vehicle; introducing a graph attention mechanism to aggregate observation features of adjacent unmanned aerial vehicles; based on the aggregated observation features, bandwidth allocation actions and corresponding rewards of the unmanned aerial vehicle under each speed scheduling are obtained; and introducing a Jain's fair index to measure the fairness of the service quality of the unmanned aerial vehicle in the region, and optimizing the speed scheduling and bandwidth allocation of the unmanned aerial vehicle. Under the condition of limited resources, the communication bandwidth is reasonably allocated, and the service coverage rate of the affected user is maximized.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle dispatching, and more specifically to a disaster area emergency unmanned aerial vehicle networking method based on reinforcement learning. Background Art

[0002] Unmanned Aerial Vehicle (UAV), commonly known as drone, is an aircraft that can be remotely controlled or driven by an autonomous system. In recent years, it has been widely studied and applied in many fields such as military and agriculture, especially in disaster rescue. When a disaster occurs, quickly rescuing the victims and transferring them to a safe place becomes the top priority. However, disasters often cause serious damage to ground-based telecommunications facilities and transportation networks, making it difficult for traditional ground rescue methods to be effectively carried out. In this case, the time cost of waiting for the restoration of facilities is unacceptable for the rescue operation, and how to search and rescue trapped people in a timely manner becomes a problem that needs to be solved urgently.

[0003] Compared with traditional rescue methods, drones have become a hot topic in disaster rescue research due to their high flexibility, low cost and easy deployment. By carrying a communication module, drones can act as aerial base stations (ABS), build emergency communication networks, and provide necessary communication services to users on the ground. Thanks to the rapid development of high altitude platform stations (HAPS) and satellite communication technology, drones can quickly establish reliable communication links in post-disaster rescue.

[0004] Drone aerial base stations can provide better wireless communication services in disaster areas with complex terrain. However, due to the high price of airborne base stations and the suddenness of disasters, the number of drone base stations available in disaster areas is usually limited. At the same time, due to the limitation on the number of drone base stations, disaster area rescue needs to consider the allocation of limited resources.

[0005] Existing drone deployment algorithms have two deficiencies in disaster area emergency networking. On the one hand, existing technologies are mainly based on static deployment, and drones are deployed as fixed base stations. With limited drone base stations, the service range of drone base stations may not cover the disaster-stricken area, causing some users to be unable to obtain services. On the other hand, most existing technologies focus on improving the performance of drones. However, when a disaster occurs, some users are restricted in their mobility due to the impact of the disaster and are distributed in relatively scattered locations. At this time, focusing only on traditional network indicators may cause users in evacuated locations to be unable to obtain effective communication services for a long time, thereby missing the golden rescue time. In this scenario, the fairness of quality of service (QoS) becomes a key factor that needs to be considered urgently.

[0006] In addition, during disaster rescue operations, global information is usually not available in a timely manner, and drone swarms can only make decisions based on partial information they observe. Existing technologies based on global information cannot be applied to emergency rescue missions in disaster areas.

[0007] Therefore, how to maximize the service coverage for disaster-affected users, reasonably allocate communication bandwidth under limited resource conditions, and ensure that each user can get due support is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0008] In view of this, the present invention provides a disaster area emergency drone networking method based on reinforcement learning, which can reasonably allocate communication bandwidth under limited resource conditions and maximize the service coverage of disaster-stricken users.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A disaster area emergency drone networking method based on reinforcement learning includes the following steps:

[0011] The UAV is regarded as an intelligent agent, and its horizontal and vertical coordinates, speed, direction, remaining energy consumption and user throughput are used as the observation space of the UAV; the speed scheduling plan and bandwidth allocation plan of the UAV are used as the action space of the UAV;

[0012] Introducing a graph attention mechanism to aggregate the observation features of neighboring drones;

[0013] Based on the aggregated observation features, the bandwidth allocation actions and corresponding rewards of the drone under each speed scheduling are obtained;

[0014] Jain's fairness index is introduced to measure the fairness of drone service quality in a region and optimize the speed scheduling and bandwidth allocation of drones.

[0015] Furthermore, the observation space O of the drone i The drone state vector O UAV and the observed user state vector O POI Splicing composition, represented by o i =[O UAV ||O POI ], where || represents concatenation operation;

[0016] The length of the drone state vector is l UAV , including the horizontal and vertical coordinates, speed, direction and remaining energy consumption of the drone;

[0017] The observed user state vector is of fixed length l POI .

[0018] Furthermore, the speed scheduling of the UAV is defined as a discrete action with finite dimensions, and the bandwidth allocation scheme of the UAV is defined as a continuous action;

[0019] Assuming that the drones have the same action space A, the action space of the problem is a mixed action space containing discrete actions and continuous actions, which consists of two parts: speed scheduling and bandwidth allocation, and is expressed by the following formula:

[0020] A={(k,x k )|x k ∈X for all k∈K}

[0021] Among them, k represents the selected discrete action, which indicates the speed scheduling plan; x k represents the selected continuous action, indicating the bandwidth allocation scheme; K represents the discrete action set, which contains K discrete actions; X represents the continuous action set, which means the continuous action set after taking speed scheme k, which can be expressed as a continuous action set with a length of C. k The vector describes .

[0022] Furthermore, the graph attention mechanism is introduced to aggregate the observation features of adjacent drones, including:

[0023] Use the shared encoder to observe the UAV’s observation space o at time t t Encode and get the encoded embedding te i ;

[0024] The drone swarm is modeled as a topological graph, with drones as nodes and the encoded embedding t is regarded as the feature information of the node, and the adjacent drones that can communicate are regarded as connected edges. The adjacency matrix is ​​defined as d t ; Use the graph attention network to aggregate the information of drones and neighboring nodes to obtain the observation features of drone i The neighbor node set of drone i is defined as N UAV (u i ), neighbor node j∈N UAV (u i ).

[0025] Furthermore, for any drone u i , extract the two-hop neighbor node information as its observation feature, expressed as:

[0026]

[0027] in, represents the observation feature of drone i at time t; σ represents the nonlinear activation function; m represents the mth attention head, M represents the total number of attention heads, represents the attention coefficient of the mth attention head; W m represents a linear transformation matrix; represents the embedding of drone j.

[0028] Furthermore, the process of determining the bandwidth allocation action of the UAV under each speed scheduling includes:

[0029] The aggregated observation features of each UAV are fed into the Param Q network as state input. The Param Q network maps the state and discrete actions into continuous parameters to obtain the most suitable bandwidth allocation action for each speed scheduling coupling.

[0030] The Q network obtains the final speed suitable for the current observation and the corresponding bandwidth allocation action based on the current observation state of the drone and the coupled action plan obtained by the Param Q network.

[0031] Furthermore, the calculation process of the reward corresponding to the bandwidth allocation action of the drone under each speed scheduling includes:

[0032] Calculate the total data throughput of each user within the drone coverage area at time t;

[0033] Based on the total data throughput, the reward generated by each user for the drone's action at time t and the total reward generated by the drone for the current step are calculated.

[0034] Furthermore, the drone service process is divided into several time slots, each time slot length is τ, and each drone is equipped with a wireless base station and a satellite communication module. When the drone provides communication services, users within the coverage area transmit data according to the allocated bandwidth. The calculation formula for the total data throughput of user j at time t is:

[0035]

[0036] in, represents the total data throughput of user j at time t; v represents the time slot v before t; n represents the number of drones; i represents the i-th drone; represents the data transmission rate of user j at time t.

[0037] Furthermore, the calculation formula for each user's reward for the drone's current action and the total reward for the drone's current action is:

[0038]

[0039] Where, j represents the jth user; G represents the total user set; represents the reward generated by the drone serving user j in time slot t; p represents a hyperparameter.

[0040] Furthermore, the calculation formula of Jain's fairness index is:

[0041]

[0042] Among them, J t represents the fairness index at time t; G represents the total set of users in the disaster area, and |G| represents the modulo operation on the total set of users.

[0043] It can be seen from the above technical solution that compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Construction of emergency communication network: The present invention can be used to construct an efficient drone emergency communication network, which can be used as a mobile base station to provide real-time communication guarantee for disaster-stricken areas. Relying on the unique advantages of drones such as high flexibility and easy deployment, the emergency communication network composed of drones can change the service scope through the dynamic deployment of drones, that is, the networked drone swarm will go to the non-stop hovering point to provide services to ground users during the continuous movement. By constantly changing the service location and networking form of the drone swarm during the communication mission, the drone can serve all users within the disaster area, can quickly adapt to complex terrain and dynamic disaster situations, and ensure that in the event of damage to infrastructure, the affected people can keep in touch with the outside world in a timely manner and obtain rescue information.

[0045] 2. Resource optimization management: The present invention regards the UAV as an intelligent agent by adopting the method of reinforcement learning, and regards the speed scheduling and resource allocation scheme of the UAV as the action space of the intelligent agent. Among them, the UAV flies to different hovering points through speed scheduling and provides communication services according to the bandwidth allocation scheme. Different speed scheduling schemes will affect the user status of the UAV coverage range when hovering, and produce different allocation schemes. Therefore, it can be considered that the speed scheduling action of the UAV is associated with the bandwidth allocation action, and the speed scheduling affects the selection of bandwidth allocation action. The present invention will reasonably reduce the speed and direction scheduling, and define it as a discrete action of finite dimensions to accelerate the convergence speed of the algorithm; while the more influential bandwidth allocation ratio is still defined as a continuous action to provide dynamic communication services to ground users in a fine-grained manner. Relying on the proposed PDGN network to solve the action decision of the mixed action space, the coupling scheme of the UAV speed scheduling and resource allocation action is completed to ensure that the limited UAV base stations can fairly serve all affected users. This function not only improves the coverage of communication services, but also ensures the efficient use of communication resources, especially in the case of uneven user distribution, to minimize service blind spots.

[0046] 3. Real-time decision support: By utilizing partial observation information, the present invention can realize real-time decision-making in the drone group and dynamically adjust the task allocation and action strategy of the drone based on local environmental changes. This intelligent decision-making ability of drone tasks improves the response speed and service quality of drones in disaster area rescue, significantly enhancing the efficiency of rescue operations.

[0047] 4. Fairness index optimization: The present invention also introduces a fairness evaluation mechanism for service quality, so that during the rescue process, all users can obtain balanced communication services within a limited time. This feature ensures that users who are restricted by the disaster will not be ignored, thereby maximizing the overall benefits of the rescue operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A flowchart of the disaster area emergency drone networking method based on reinforcement learning provided by the present invention;

[0050] Figure 2 A scene diagram for providing communication services for the dynamically deployed drone provided by the present invention;

[0051] Figure 3 This is a schematic diagram of the reinforcement learning process provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] like Figure 1 As shown, the embodiment of the present invention discloses a disaster area emergency drone networking method based on reinforcement learning, comprising the following steps:

[0054] S1. Model the communication control problem of the UAV as a Markov random process, regard the UAV as an intelligent agent, and use the horizontal and vertical coordinates, speed, direction, remaining energy consumption and user throughput of the UAV as the observation space of the UAV; use the speed scheduling plan and bandwidth allocation plan of the UAV as the action space of the UAV; initialize the disaster area environment, user status and UAV status;

[0055] S2, introduce graph attention mechanism to aggregate the observation features of neighboring drones;

[0056] S3. Based on the aggregated observation features, the bandwidth allocation action and corresponding reward of the drone under each speed scheduling are obtained;

[0057] S4. Introduce Jain’s fairness index to measure the fairness of drone service quality in the region and optimize the speed scheduling and bandwidth allocation of drones.

[0058] Next, combine Figure 2 and Figure 3 The above steps of the present invention are further described.

[0059] S1. Determination of observation space and action space.

[0060] 1) Observation space: In the area where natural disasters occur, according to existing research, the affected users are scattered in the disaster area according to the zipf distribution, and the drone swarm conducts observations and executes speed scheduling and resource allocation plans. The drone observation content includes the state vector O UAV , O UAV is encoded as a length of l UAV The vector is composed of the horizontal and vertical coordinates, speed, direction and remaining energy consumption of the drone. In addition, the observation content also includes the throughput O of users within the observation range. POI , because O UAV The user coordinate information is the same within the observation range of the same drone coordinates, that is, the drone coordinates contain the user coordinate information. Therefore, in order to reduce the size of the observation space, only the user throughput information is concerned in the observation.

[0061] In addition, since the number of users observed by the drone changes dynamically with the location, the length of the user throughput list observed by the drone at different locations is different. To ensure dimensional consistency, the user throughput O POI Supplemented to a fixed length l POI , which is calculated based on the zipf distribution and the observation range, and is equivalent to the maximum number of user observations of the drone. Finally, the drone state vector O UAV and the observed user state vector O POI Splicing, get the observation space o i The length is l UAV +l POI , represented by oi =[O UAV ||O POI ], where || represents a concatenation operation.

[0062] 2) Action space: In the action space part, the drone first performs speed scheduling to fly to the service point, and then provides communication services according to the bandwidth allocation plan. Different speed scheduling plans will affect the user status of the drone's coverage area when hovering, and produce different allocation plans. Therefore, it can be considered that the speed scheduling action of the drone is associated with the bandwidth allocation action, and speed scheduling affects the selection of bandwidth allocation actions. The present invention will reasonably reduce the scheduling of speed and direction, and define it as a discrete action of finite dimensions to speed up the convergence of the algorithm; while the more influential bandwidth allocation plan is still defined as a continuous action to provide dynamic communication services to ground users in a fine-grained manner.

[0063] Assuming that the drones have the same action space A, the action space of the problem is a mixed action space containing discrete actions and continuous actions, which consists of two parts: speed scheduling and bandwidth allocation, and is expressed by the following formula:

[0064] A={(k,x k )|x k ∈X for all k∈K}

[0065] Among them, k represents the selected discrete action, which indicates the speed scheduling plan; x k represents the selected continuous action, indicating the bandwidth allocation scheme; K represents the discrete action set, which contains K discrete actions; X represents the continuous action set, which means the continuous action set after taking speed scheme k, which can be expressed as a continuous action set with a length of C. k The vector describes .

[0066] The present invention regards the UAV as an intelligent agent and uses the reinforcement learning method to learn the speed scheduling and resource allocation scheme of the UAV. The specific learning process is as follows: Figure 2 As shown in the figure. In the reinforcement learning process, the PDGN network model is constructed, which mainly includes two modules: (1) Observation feature extraction module: Introducing the graph attention mechanism to aggregate neighbor information and extract the local observation of the drone and the observation features of the drone's neighbor nodes. (2) Hybrid action space decision module: Through the extracted observation features, the discrete actions and continuous actions are jointly optimized to improve the quality of the strategy, so that the intelligent agent can act more sensitively and efficiently in complex environments. The functions of the two modules are respectively reflected in S2 and S3. S2 and S3 are further explained below.

[0067] S2. Observation feature extraction:

[0068] Use the shared encoder to observe the UAV’s observation space o at time t tEncode and get the encoded embedding The shared encoder ENC uses a multi-layer perceptron.

[0069] The drone swarm is modeled as a topological graph, with drones as nodes and the encoded embedding t is regarded as the feature information of the node, and the adjacent drones that can communicate are regarded as connected edges. The adjacency matrix is ​​defined as d t ; Use the graph attention network to aggregate the information of drones and neighboring nodes to obtain the observation features of drone i The neighbor node set of drone i is defined as N UAV (u i ), neighbor node u j ∈N UAV (u i ). The weighted observation features are expressed as:

[0070]

[0071] Where k∈N UAV (u i ), α ij Indicates drone u i The neighbor set of ij is the attention coefficient of drone i and j, W is the learned linear transformation matrix, η is the learnable weight vector, and σ is the nonlinear activation function.

[0072] In order to stably improve the attention expression effect, the present invention further adopts a multi-head attention mechanism. Each node aggregates information through multiple attention heads and learns different attention weight distributions. For any drone i, the two-hop neighbor node information is extracted as its observation feature, that is, after the output of the single-layer graph attention network, it passes through another layer of graph attention network to extract features, which can be expressed as:

[0073]

[0074] in, represents the observation feature of drone i at time t; σ represents the nonlinear activation function; m represents the mth attention head, M represents the total number of attention heads, represents the attention coefficient of the mth attention head; W m represents a linear transformation matrix; represents the embedding of drone j.

[0075] S3, action decision in mixed action space:

[0076] After the observation feature extraction module aggregates the observation features of adjacent drones, the hybrid action space module couples the speed scheduling and resource allocation actions through Param Q and Q networks, including:

[0077] Defining the action space The action value function under is expressed as: Assume that the discrete action selected by the drone at time t is k t , the continuous action corresponding to the coupling is The Bellman equation is updated to:

[0078]

[0079] In the above formula, the continuous action x of the drone k is the action associated with the discrete action k. Before calculating the action value, we need to calculate Get the continuous action coupled to each discrete action k Then choose the one that maximizes the value

[0080] When the Q network is fixed, the upper bound of the continuous parameter is calculated Considered as a function mapping from drone state to continuous parameters The Bellman equation changes to:

[0081]

[0082] PDGN uses the Q(·;ω) network to approximate the action-value function Q(s,k,x k ), using ParamQ(·;θ) network approximation function mapping

[0083] The aggregated observation features of each UAV are fed into the Param Q network as state input. The Param Q network maps the state and discrete actions into continuous parameters to obtain the most suitable bandwidth allocation action for each speed scheduling coupling.

[0084] The Q network obtains the final speed suitable for the current observation and the corresponding bandwidth allocation action based on the current observation state of the drone and the coupled action plan obtained by the Param Q network.

[0085] Determine the selected speed and resource allocation scheme, and complete the action decision in the mixed action space.

[0086] The hybrid action space decision part consists of four networks: Param Q network, target Param Q network, Q network and target Q network. Among them, the Param Q network is used to map states and discrete actions into continuous parameters, and obtain a list of continuous parameters for each discrete action coupling.

[0087] x t =X θ (s t )=[x 11 ,...,x 1C ,...,x KC ]

[0088] X in the formula θ Represents the Param Q network. x t is a continuous parameter list containing the continuous actions corresponding to each discrete action; [x 11 ,...,x 1C ,...,x KC ], the subscripts K and C represent the number of discrete actions and the length of continuous actions in the action space, respectively.

[0089] The Q network then maps the state and continuous parameters to actual values ​​and determines the appropriate discrete actions:

[0090] q t =Q(s t ,x t ;ω)=[q1,...,q K ]

[0091] q t The maximum index k of the elements in t = argmax i {q i} as discrete actions, ω represents the parameters of the Q network, x t Represents a list of consecutive parameters, starting from x t Continuous parameters intercepted in As continuous actions, a discrete-continuous action plan of the policy is obtained.

[0092] Finally, PDGN uses gradient descent to minimize the Bellman mean square error to learn network parameters. Assume that in the tth step, the parameters of the Q network and Param Q network are ω t and θ t , combined with the multi-step temporal difference algorithm, the difference target can be calculated as:

[0093]

[0094] In addition, when ω is fixed, by k(s; θ); ω) is maximized to calculate θ, and the two loss functions of the present invention are expressed as:

[0095]

[0096] The other two target networks are used to reduce the volatility of model updates and stabilize the training process. By updating the model every few training steps, the volatility of the model during training can be reduced.

[0097] Specifically, when making action decisions, it is necessary to calculate the reward for each action decision. Therefore, the present invention constructs a reward function. The specific construction process includes:

[0098] The drone service process is divided into several time slots, each time slot length is τ, and each drone is equipped with a wireless base station and a satellite communication module. When the drone provides communication services, users within the coverage area transmit data according to the allocated bandwidth. The total data throughput of user j at time t is calculated using the following formula:

[0099]

[0100] in, represents the total data throughput of user j at time t; v represents the time slot v before t; n represents the number of drones; i represents the i-th drone; represents the data transmission rate of user j at time t.

[0101] Calculate the reward generated by each user's drone action at time t based on the total data throughput And the total reward r for the drone to perform the current action t , the calculation formula is:

[0102]

[0103] Where, j represents the jth user; G represents the total user set; represents the reward generated by the drone serving user j in time slot t; p represents the hyperparameter of the algorithm, which is set and compared in the experiment and can be adjusted.

[0104] S4. Introduce Jain’s fairness index to measure the fairness of drone service quality in a region.

[0105] The purpose of the present invention is to maximize the fairness of service quality. If the unmanned cluster only considers traditional network indicators, users will be unable to obtain communication services for a long time and miss the golden rescue time. Therefore, the present invention measures the fairness of service quality by introducing Jain's fairness index:

[0106]

[0107] in, is the total data throughput of user j at time t; G represents the total set of users in the disaster area, and |G| represents the modulo operation on the user set, that is, the number of users.

[0108] As shown in the formula, Jain's fairness index is the target, which is a global indicator and requires the status information of each user to be calculated. However, when performing communication tasks in disaster areas, drones can only obtain information about users within the observation range, and cannot directly use Jain's fairness index as a reward function. In reinforcement learning, the reward function is an evaluation of drone actions. The higher the reward for drone actions, the more inclined the drone is to take that action. However, because Jain's fairness index cannot be used directly as a reward, a reward function based on local observations is designed to approximate the fairness index. Jain's fairness index is usually used to measure the fairness of the entire network. The more uniform the user throughput of the network, the closer the Jain's fairness index is to 1, which means that the drone service is more fair.

[0109] In general, the present invention assumes that a group of drones fly in an area affected by a natural disaster to complete emergency communication tasks. There are several affected users in the area scattered in various locations according to the zipf distribution, waiting for drones to provide communication services, and the user throughput value at the initial moment is 0. Considering that drones may be stored in different warehouses, the present invention assumes that the drone group is initially located at a random position in the area. Due to the suddenness of the disaster, the drone is unknown to the affected users in the area and can only make decisions based on the local information observed by itself. Each drone is equipped with a wireless base station and a satellite communication module, and has the ability to provide emergency communication services. When the drone provides communication services, users within the coverage area transmit data according to the allocated bandwidth and obtain disaster relief information. If the unmanned cluster only considers traditional network indicators, users will be unable to obtain communication services for a long time and miss the golden rescue time. Therefore, Jain's fairness index is introduced to measure the fairness of drone service quality in the region.

[0110] Finally, the goal of this invention is to maximize Jain's fairness index by jointly optimizing the drone swarm, speed scheduling, and bandwidth allocation ratio of users within the coverage area. This invention models the service quality as throughput within a finite time, uses Jain's fairness index to measure the fairness of user service quality, and optimizes drone speed scheduling and resource allocation through the proposed PDGN model, so that all users in the network can get high-quality communication services.

[0111] Finally, the performance of the proposed technology was evaluated through experimental simulation and compared with the existing technology. The proposed technology improved the fairness index by about 6.8% and the throughput index by 1.2% compared with the existing algorithm. At the same time, it was also better than the existing technology in comparative experiments in different scenarios.

[0112] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0113] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A disaster area emergency drone networking method based on reinforcement learning, characterized in that: The following steps are involved: The UAV is regarded as an intelligent agent, and its horizontal and vertical coordinates, speed, direction, remaining energy consumption and user throughput are used as the observation space of the UAV; the speed scheduling plan and bandwidth allocation plan of the UAV are used as the action space of the UAV; Introducing a graph attention mechanism to aggregate the observation features of neighboring drones; Based on the aggregated observation features, the bandwidth allocation actions and corresponding rewards of the drone under each speed scheduling are obtained; Jain's fairness index is introduced to measure the fairness of drone service quality in a region and optimize the speed scheduling and bandwidth allocation of drones.

2. The disaster area emergency drone networking method based on reinforcement learning according to claim 1 is characterized in that: UAV observation space O i The drone state vector O UAV and the observed user state vector O POI Splicing composition, represented by o i =[O UAV ||O POI ], where || represents concatenation operation; The length of the drone state vector is l UAV , including the horizontal and vertical coordinates, speed, direction and remaining energy consumption of the drone; The observed user state vector is of fixed length l POI .

3. The disaster area emergency drone networking method based on reinforcement learning according to claim 1 is characterized in that: The speed scheduling of the UAV is defined as a discrete action of finite dimensions, and the bandwidth allocation scheme of the UAV is defined as a continuous action; Assuming that the drones have the same action space A, the action space of the problem is a mixed action space containing discrete actions and continuous actions, which consists of two parts: speed scheduling and bandwidth allocation, and is expressed by the following formula: A={(k,x k )|x k ∈X for all k∈K} Among them, k represents the selected discrete action, which indicates the speed scheduling plan; x k represents the selected continuous action, indicating the bandwidth allocation scheme; K represents the discrete action set, which contains K discrete actions; X represents the continuous action set, which means the continuous action set after taking speed scheme k, which can be represented by a length of C k The vector describes .

4. The disaster area emergency drone networking method based on reinforcement learning according to claim 1 is characterized in that: The graph attention mechanism is introduced to aggregate the observation features of neighboring drones, including: Use the shared encoder to observe the UAV’s observation space o at time t t Encode and get the encoded embedding The drone swarm is modeled as a topological graph, with drones as nodes and the encoded embedding t is regarded as the feature information of the node, and the adjacent drones that can communicate are regarded as connected edges. The adjacency matrix is ​​defined as d t ; The graph attention network is used to aggregate the information of drones and neighboring nodes to obtain the observation features of drone i The neighbor node set of drone i is defined as N UAV (u i ), neighbor node j∈N UAV (u i ).

5. The disaster area emergency drone networking method based on reinforcement learning according to claim 4 is characterized in that: For any UAV i, extract the two-hop neighbor node information as its observation feature, expressed as for: in, represents the observation feature of drone i at time t; σ represents the nonlinear activation function; m represents the mth attention head, M represents the total number of attention heads, represents the attention coefficient of the mth attention head; W m represents a linear transformation matrix; represents the embedding of drone j.

6. The disaster area emergency drone networking method based on reinforcement learning according to claim 3 is characterized in that: The process of determining the bandwidth allocation action of the drone under each speed scheduling includes: The aggregated observation features of each UAV are fed into the Param Q network as state input. The Param Q network maps the state and discrete actions into continuous parameters to obtain the most suitable bandwidth allocation action for each speed scheduling coupling. The Q network obtains the final speed suitable for the current observation and the corresponding bandwidth allocation action based on the current observation state of the drone and the coupled action plan obtained by the Param Q network.

7. The disaster area emergency drone networking method based on reinforcement learning according to claim 1 is characterized in that: The calculation process of the reward corresponding to the bandwidth allocation action of the drone under each speed scheduling includes: Calculate the total data throughput of each user within the drone coverage area at time t; Based on the total data throughput, the reward generated by each user for the drone's action at time t and the total reward generated by the drone for the current step are calculated.

8. The disaster area emergency drone networking method based on reinforcement learning according to claim 7 is characterized in that: The drone service process is divided into several time slots, each time slot length is τ, and each drone is equipped with a wireless base station and a satellite communication module. When the drone provides communication services, users within the coverage area transmit data according to the allocated bandwidth. The calculation formula for the total data throughput of user j at time t is: in, represents the total data throughput of user j at time t; v represents the time slot v before t; n represents the number of drones; i represents the i-th drone; represents the data transmission rate of user j at time t.

9. The disaster area emergency drone networking method based on reinforcement learning according to claim 8 is characterized in that: The calculation formula for each user's reward for the drone's current action and the total reward for the drone's current action is: Where, j represents the jth user; G represents the total user set; represents the reward generated by the drone serving user j in time slot t; p represents a hyperparameter.

10. The disaster area emergency drone networking method based on reinforcement learning according to claim 7 is characterized in that: The calculation formula of Jain's fairness index is: Among them, J t represents the fairness index at time t; G represents the total set of users in the disaster area, and |G| represents the modulo operation on the total set of users.

Images