Unmanned aerial vehicle arrival and departure sorting method based on FCFS algorithm and reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle arrival and departure sorting method based on an FCFS algorithm and reinforcement learning, and the method comprises the steps: enabling an unmanned aerial vehicle to enter a waiting layer of a terminal region, creating an initial execution time table, and obtaining the initial data of the unmanned aerial vehicle; obtaining an initial state matrix; determining a to-be-executed action set of the unmanned aerial vehicle based on the airport operation limiting condition; calculating a reward value of each to-be-executed action in the to-be-executed action set; the unmanned aerial vehicle executes the to-be-executed action with the maximum reward value; repeating the above steps until the accumulated reward value is maximum, and obtaining the sequence of the to-be-executed actions corresponding to the accumulated reward value; optimizing the optimal time execution table based on the sorting of the actions to be executed and the sorting result of the FCFS algorithm; and the unmanned aerial vehicle executes corresponding actions according to the optimal time execution table. According to the optimized take-off and landing sequence, the utilization rate of the take-off and landing channels can be effectively improved, the situation that the take-off and landing channels are idle or congested is reduced, and the safety risk caused by collision of the unmanned aerial vehicles is reduced to a great extent.

Description

Technical Field

[0001] The present application belongs to the field of drone control technology, and specifically relates to a drone entry and departure sorting method based on FCFS algorithm and reinforcement learning. Background Art

[0002] As an emerging economic form, low-altitude economy mainly relies on low-altitude airspace for various aircraft flight activities. As an important carrier for the development of low-altitude economy, drones are widely used in urban logistics, remote area distribution and emergency transportation. With the vigorous development of the e-commerce industry and the sharp increase in logistics business volume, traditional distribution methods have been unable to meet the needs of fast and efficient logistics. With its advantages of high distribution efficiency and low labor costs, logistics drones can solve the problems encountered in the development of traditional ground logistics and become an effective choice for solving the "last mile".

[0003] For the take-off and landing scheduling of drones, the First-Come, First-Served (FCFS) algorithm is generally adopted. The basic principle of FCFS is to process requests in the order in which they arrive, that is, the earliest request will be served first, and the newly arrived request will wait at the end of the queue. This algorithm is simple to operate and fair to every drone. However, in the actual logistics drone delivery process, the take-off and landing order of drones is not completely fair. It is necessary to take into account the occurrence of some emergency landing situations when each drone takes off and lands, such as the drone needs to land first due to insufficient battery power, and the individual orders need to take off and deliver first due to urgent delivery time. According to the take-off and landing order obtained by this algorithm, the utilization rate of the take-off and landing channels is insufficient, which will cause the take-off and landing channels to be idle or congested.

[0004] Currently, there is a lack of research on the entry and departure sequencing of multiple logistics drones. The traditional aircraft entry and departure sequencing strategy is not applicable to logistics drones with high traffic volume in low-altitude airspace and frequent delivery tasks. Summary of the invention

[0005] The present application provides a method for sorting UAV arrival and departure based on the FCFS algorithm and reinforcement learning, so as to solve the technical problem that the traditional aircraft arrival and departure sorting strategy is not applicable to logistics UAVs with high traffic volume and frequent delivery tasks in low-altitude airspace.

[0006] In order to solve the above technical problems, a technical solution adopted in this application is: a method for sorting UAV entry and departure based on FCFS algorithm and reinforcement learning, including:

[0007] The drone enters the waiting layer of the terminal area, creates an initial execution schedule, and obtains the initial data of the drone; the initial data includes the type of drone, the type of courier, the initial take-off and landing times assigned by the FCFS algorithm, and the priority of the delivery task;

[0008] Based on the initial data of the UAV, the initial state matrix is ​​obtained;

[0009] Based on the airport operation constraints and the starting state matrix, determine the set of actions to be performed by the drone;

[0010] Calculate the reward value of each action to be executed in the set of actions to be executed, and obtain the action to be executed with the largest reward value;

[0011] The drone executes the pending action with the largest reward value and enters the next state;

[0012] Repeat the above steps until the cumulative reward value is the largest, and obtain the ranking of the actions to be executed corresponding to the cumulative reward value;

[0013] Optimize the best time execution table based on the order of actions to be executed and the order results of the FCFS algorithm;

[0014] The drone performs corresponding actions according to the optimal time execution table.

[0015] Further, airport operation restrictions include:

[0016] When the drone battery power is lower than the preset power threshold, the drone will land first;

[0017] Since the take-off and landing channel is occupied, the drone hovers and waits;

[0018] Since delivery missions have the highest priority, drones take off or land first.

[0019] Furthermore, the method for calculating the reward value of each action to be executed in the set of actions to be executed includes:

[0020] According to formula (1), the reward value is obtained; wherein formula (1) is:

[0021]

[0022] Among them, r multi is a multi-target reward; i is the i-th target weight; r i is the reward of the i-th target; M is the number of targets; Gt is defined as the expected long-term return, γ is the discount coefficient, and its value is in [0,1]. When γ is close to 1, more attention is paid to long-term returns; when γ is close to 0, more attention is paid to short-term returns.

[0023] Further, the method for calculating the cumulative reward value includes:

[0024] Based on formula (2), an iterative formula for calculating the cumulative reward is obtained;

[0025] Q(s,a)=Q(s,a)+α(r+γV(s')-Q(s,a)) (2);

[0026] Among them, α represents the learning step size; γ represents the reward discount factor; s' represents the transition to state s' after executing action a in state s, and V(s') represents the value function of state s'.

[0027] Furthermore, based on the order of the actions to be executed and the ordering result of the FCFS algorithm, the method for optimizing the best time execution table includes:

[0028] Determine whether the drone encounters any restriction; if there is no restriction, the optimal time execution table is the initial execution schedule; if at least one restriction is encountered, the optimal time execution table is regenerated according to the order of the actions to be executed.

[0029] The beneficial effect of the present application is that compared with the traditional aircraft entry and departure sorting algorithm, the logistics drone entry and departure sorting method based on FCFS algorithm and reinforcement learning proposed in this patent is more suitable for actual logistics drone delivery scenarios. FCFS sorting based on arrival and take-off time, combined with reinforcement learning algorithm, in the entry and landing scenario, takes into account the situation that the drone is in urgent need of landing due to insufficient battery power, so that it can land first; in the departure and take-off scenario, the priority of the generated delivery order is taken into account, so that the high priority takes off first. The optimized take-off and landing sequence can effectively improve the utilization rate of the take-off and landing channels, reduce the idleness or congestion of the take-off and landing channels, and greatly reduce the safety risks caused by drone collisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of an embodiment of a method for sorting UAV entry and departure based on FCFS algorithm and reinforcement learning of the present application;

[0031] Figure 2 This is a schematic diagram of the drone’s entry and departure based on the FCFS algorithm and reinforcement learning in this application;

[0032] Figure 3 This is a flowchart of accumulating reward values ​​for actions to be executed in an embodiment of a method for sorting drone entry and departure based on FCFS algorithm and reinforcement learning of the present application. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0035] See also Figure 1 , Figure 1 : is a flowchart of an embodiment of a method for sorting drones entering and leaving an airport based on the FCFS algorithm and reinforcement learning of the present application. The method includes:

[0036] Step S1. The drone enters the waiting layer of the terminal area, creates an initial execution schedule, and obtains the initial data of the drone; wherein the initial data includes the type of drone, the type of express, the initial take-off and landing times assigned by the FCFS algorithm, and the priority of the delivery task.

[0037] Specifically, express delivery orders are divided into two types: large and small, corresponding to two sizes of delivery drone models. The initial data includes the type of drone, the type of express delivery, the initial take-off and landing times assigned by the FCFS algorithm, and the priority of the delivery task.

[0038] The terminal area of ​​the take-off and landing field is the boundary of the airspace under jurisdiction. When a drone enters the terminal area, it is included in the control range of the drone take-off and landing field. The waiting layer is an air area in the terminal area for drones to hover and wait. When a drone enters the terminal area of ​​the take-off and landing field, its initial data is obtained, the initial execution schedule is created, and the initial state is obtained.

[0039] Since the number of take-off and landing channels is less than the number of drones, it is necessary to arrange the take-off and landing order of each drone reasonably. The take-off and landing order in this initial execution schedule refers to the order in which multiple drones enter the take-off and landing airport and leave or land through the take-off and landing channels.

[0040] Step S2: Based on the initial data of the UAV, obtain the initial state matrix.

[0041] Specifically, the initial state matrix is ​​determined based on initial data such as the model of the drone used to transport logistics, the initial take-off and landing times assigned by FCFS, and the priority of the delivery task.

[0042] Step S3: Based on the airport operation constraints and the starting state matrix, determine the set of actions to be performed by the drone.

[0043] Specifically, the set of actions to be performed A is determined based on various operational restrictions during the operation of the departing flight in the terminal area. Among them, it is necessary to consider the following operational restrictions: if the take-off and landing channel is occupied, the drone will hover and wait; if the drone battery is insufficient, the drone will land first; if the delivery order has a high priority, the drone will take off first.

[0044] Step S4: Calculate the reward value of each action to be executed in the set of actions to be executed, and obtain the action to be executed with the largest reward value.

[0045] Specifically, in the take-off or landing scenario, calculate any action a in the set of actions to be executed A i The corresponding reward value is calculated, and the action to be performed is selected based on the reward value.

[0046] The reinforcement learning algorithm is a value-based method. It does not directly output the probability of the corresponding action in each state, but calculates the value of taking a certain action at each step and selects the action with the maximum action value (maximum cumulative benefit) to obtain the best strategy. The algorithm generates a Q-table (execution timetable) consisting of states and actions, where each row represents a state s and each column represents an action a. i Each value represents the Q(s,a) of executing a pending action a in the current state s. Through the process of interaction between the intelligent program and the environment, the environment's feedback is continuously received, and the Q-table is continuously updated until convergence.

[0047] Among them, strategy π is the mapping from state to action, and strategy π(a|s) is a random strategy, that is, for each action a i The probability of being in state s is different. The random strategy is described by probability, that is, the probability of performing this action in a certain state:

[0048] π(a|s)=P[A t =a|S t =s];

[0049] The definition of reward value is to judge the immediate return in an interaction. In this embodiment, the optimization factors that the intelligent program needs to consider include the order of drone arrival at the terminal area, the remaining power of the drone at the terminal area, the priority of the drone's own mission, the drone model, and collision-free, but are not limited to these factors. It can also be customized according to specific task requirements to meet specific optimization goals. The multi-objective reward is expressed as a mathematical model, as shown in formula (1):

[0050]

[0051] Among them, r multi is a multi-target reward; i is the i-th target weight; r iis the reward of the i-th target; M is the number of targets; Gt is defined as the expected long-term return (Return), γ is the discount coefficient, and its value is in [0,1]. When γ is close to 1, more attention is paid to long-term returns; when γ is close to 0, more attention is paid to short-term returns.

[0052] Step S5: The drone executes the pending action with the largest reward value and enters the next state.

[0053] Specifically, in the result calculated according to the reward value matrix, if the reward value of a certain execution action in the set is greater than the reward value of the current action, the above-mentioned action to be executed is executed and jumps to the next state.

[0054] The expected long-term return is:

[0055]

[0056] r represents the return reward value of the action to be executed, which is an immediate return. The state value function describes the weighted sum of the reward values ​​of all possible actions to be executed based on the current state. Define V π (s) is the long-term expected return of strategy state s.

[0057] The value function V of state s is:

[0058] V π (s) = E π [R t |S t =s];

[0059] The action value function describes the weighted sum of all possible long-term rewards for a given state and active. π (s,a) is the strategy in state s, perform action a i The long-term expected return. The action value function Q of taking action a in state s is:

[0060] Q π (s,a)=E π [G t |S t =s,A t =a];

[0061] Convert from value function to Q function and execute action a i Will transfer from state s to the next state s', then the Q function can be expressed as:

[0062] Q π (s,a)=E[r+γV(s′)|s,a];

[0063] At this time, the Q function indicates that action a is taken in state si The immediate reward r obtained after adding the discounted value of future returns is the iterative formula.

[0064] Step S6: Repeat the above steps until the cumulative reward value is the largest, and obtain the ranking of the actions to be executed corresponding to the cumulative reward value.

[0065] Specifically, steps S4-S5 are repeated until the cumulative reward value is the maximum and the reward value iteration result reaches convergence. The cumulative reward values ​​of all actions to be executed are calculated, and the order of the actions to be executed is generated according to the size of the cumulative reward values.

[0066] See also Figure 3 , the specific method of accumulating reward values ​​uses the following iterative formula;

[0067] Q(s,a)=Q(s,a)+α(r+γV(s')-Q(s,a)) (2);

[0068] Among them, α represents the learning step size; γ represents the reward discount factor; s' represents the execution of action a in state s i Then it changes to state s', V(s') represents the value function of state s'.

[0069] That is, action a is executed in state s i The environment will give a feedback r, and then it will move to the next state, and there will be an expected cumulative return in this state, forming a recursive formula. Through this recursive process, the Q-table is continuously updated to obtain a final optimal Q value table (optimal execution table), so that the intelligent program can obtain the Q value of the corresponding action in each state, and thus select the action with the largest Q value to complete the optimization of the take-off and landing sequence of the drone entering and leaving the airport.

[0070] Step S7: Optimize the best time execution table based on the order of the actions to be executed and the ordering result of the FCFS algorithm.

[0071] Specifically, it is determined whether the drone encounters a restriction condition; if there is no restriction condition, the optimal time execution table is the initial execution schedule; if at least one restriction condition is encountered, the optimal time execution table is regenerated according to the order of the actions to be executed.

[0072] For example, the take-off and landing field is planned according to the destination of the drone delivery mission, different flight routes, types of drones, etc. The take-off and landing field is divided into α docking areas, each area has 8 aprons, and each docking area is equipped with β landing entrances and θ take-off channels. The take-off and landing channels are separated and do not interfere with each other. β drones can land and θ drones can take off at the same time. Each area can dock drones from the corresponding routes without interfering with each other, that is, the take-off and landing field can simultaneously land α·β drones and take off α·θ drones. Drones in a single parking area land in sequence according to the sorting algorithm.

[0073] Assume that during the landing process, all drones entering the terminal area waiting for landing are in a reasonable waiting position. At the waiting altitude layer of the take-off and landing field, there are n drones waiting to land. The remaining power of the drones, the order of drones arriving at the terminal area, the priority of drone delivery tasks, and the type of drones will affect the landing order of the drones. Assuming that n drones descend in the order of 1 to n, which drones should be assigned to take off and land in which order to make the drone entry and departure procedures most efficient.

[0074] First, determine whether the take-off and landing channel is occupied by other drones, whether the battery is sufficient, and whether there is a higher priority task. If the take-off and landing channel is idle, the drone has sufficient battery and the task priority is not high, then take off and land directly according to the initial schedule assigned by FCFS. If the take-off and landing channel is occupied, or the battery is lower than the preset value, or the task priority is high, it will hover and wait in the waiting layer, and take off and land according to the optimal time execution table optimized by the reinforcement learning algorithm.

[0075] Step S8. The drone performs corresponding actions according to the optimal time execution table.

[0076] Specifically, when the drone enters the take-off and landing layer, the optimal time execution table is frozen. According to the optimization results, the optimized take-off and landing sequence is determined, and the drone enters the landing or take-off channel and takes off or lands according to the optimized sequence. At this time, the take-off and landing time sequence is not allowed to be changed.

[0077] Compared with the traditional entry and exit drone sorting algorithm, this application is more suitable for actual logistics drone delivery scenarios. Based on the FCFS sorting of arrival and take-off time, combined with the reinforcement learning algorithm, in the entry and landing scenario, the situation that the drone is in urgent need of landing due to insufficient power is considered, so that it can land first; in the departure and take-off scenario, the priority of the generated delivery order is considered, so that the high priority takes off first. The reinforcement learning algorithm can handle the uncertainty and randomness of the environment, continuous state and action space, and can cope with the complex dynamic process of logistics drones entering and leaving. The optimized take-off and landing sequence can effectively improve the utilization rate of the take-off and landing channels, reduce the idleness or congestion of the take-off and landing channels, and greatly reduce the safety risks caused by drone collisions. In addition, in addition to the arrival and take-off time, the consideration of drone power and delivery task priority is also added, which can effectively reduce drone loss and delivery costs, shorten customer waiting time, improve customer satisfaction with drone delivery services, increase customer recognition, promote the promotion of logistics drone delivery business, and promote the development of low-altitude economy.

[0078] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for sorting UAV arrival and departure based on FCFS algorithm and reinforcement learning, characterized in that: include: The drone enters the waiting layer of the terminal area, creates an initial execution schedule, and obtains the initial data of the drone; wherein the initial data includes the type of the drone, the type of express delivery, the initial take-off and landing times assigned by the FCFS algorithm, and the priority of the delivery task; Based on the initial data of the UAV, an initial state matrix is ​​obtained; Determining a set of actions to be performed by the UAV based on the airport operation restriction conditions and the initial state matrix; Calculate the reward value of each action to be executed in the set of actions to be executed, and obtain the action to be executed with the largest reward value; The drone executes the pending action with the largest reward value and enters the next state; Repeat the above steps until the cumulative reward value is the largest, and obtain the ranking of the actions to be executed corresponding to the cumulative reward value; Optimizing the best time execution table based on the order of the actions to be executed and the order results of the FCFS algorithm; The drone performs corresponding actions according to the optimal time execution table.

2. The method according to claim 1, characterized in that The airport operation restrictions include: Based on the battery level of the drone being lower than a preset battery level threshold, the drone is given priority to land; Based on the fact that the take-off and landing channel is occupied, the UAV hovers and waits; Since the delivery task has the highest priority, the drone takes off or lands first.

3. The method according to claim 1, characterized in that The method for calculating the reward value of each action to be performed in the set of actions to be performed includes: According to formula (1), the reward value is obtained; wherein the formula (1) is: Among them, r multi is a multi-target reward; i is the i-th target weight; r i is the reward of the i-th target; M is the number of targets; Gt is defined as the expected long-term return, γ is the discount coefficient, and its value is in [0,1]. When γ is close to 1, more attention is paid to long-term returns; when γ is close to 0, more attention is paid to short-term returns.

4. The method according to claim 1, characterized in that: The method for calculating the cumulative reward value includes: Based on formula (2), an iterative formula for calculating the cumulative reward is obtained; wherein the formula (2) is: Q(s,a)=Q(s,a)+α(r+γV(s')-Q(s,a)) (2); Among them, α represents the learning step size; γ represents the reward discount factor; s' represents the transition to state s' after executing action a in state s, and V(s') represents the value function of state s'.

5. The method according to claim 1, characterized in that: The method for optimizing the best time execution table based on the sorting of the actions to be executed and the sorting results of the FCFS algorithm includes: Determine whether the drone encounters the restriction condition; if there is no restriction condition, the optimal time execution table is the initial execution schedule; if at least one restriction condition is encountered, regenerate the optimal time execution table according to the sequence of the actions to be executed.

Images