Unmanned aerial vehicle charging scheduling method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of unmanned aerial vehicle control, and particularly relates to an unmanned aerial vehicle charging scheduling strategy based on deep reinforcement learning. The charging recommendation problem of the unmanned aerial vehicles needs to consider the potential resource competition between the unmanned aerial vehicles and the occupation condition of the charging pile in the charging process. MAPPO is an algorithm for multi-agent reinforcement learning, shows excellent performance on some models, but also has certain limitation, namely, the stability of the algorithm is relatively weak, and the algorithm is easy to fall into local optimum. In order to solve the problems, the invention proposes that a Traley PPO algorithm is applied to multi-agent reinforcement learning, the defects of a multi-agent near-end optimization algorithm MAPPO are overcome, meanwhile, an attention level system is introduced into a centralized Critic network, the optimization direction is adaptively guided through a dynamic gradient weighting strategy, and the problem that the state action space is too large is avoided. According to the method provided by the invention, the unmanned aerial vehicle can be efficiently distributed to the most suitable charging station.

Description

Technical Field

[0001] The present application belongs to the field of drone control technology, and in particular relates to a drone charging scheduling method based on deep reinforcement learning. Background Art

[0002] Drones are gradually expanding from aerial photography toys to application areas in various industries. Globally, their market size is growing steadily. After completing the planning of drone charging stations, due to the limited charging infrastructure and the spatial and temporal imbalance of charging demand, in order to realize large-scale automated application of drones, the intelligent scheduling problem of handling charging requests needs to be solved urgently. Different from traditional recommendation tasks, the charging recommendation problem needs to consider the potential resource competition between drones and the occupancy of charging piles during the charging process.

[0003] With the development of distributed artificial intelligence, multi-agent systems have become the forefront and hotspot in the field of dynamic scheduling research. Multi-agent technology gives manufacturing resource entities and functional entities a certain intelligence, enabling them to have the ability of self-perception, reasoning, self-learning and self-decision-making. MAPPO is a centralized training and decentralized execution algorithm for multi-agent reinforcement learning. Although the MAPPO algorithm has been widely regarded as the most advanced method in the field of reinforcement learning and has shown excellent performance on some models, it also has certain limitations. One of the problems is that it cannot limit the probability ratio of the strategy, affecting the stability of the algorithm. In addition, the policy constraints based on the probability ratio metric will lead to a lack of exploration of the strategy, resulting in a tendency to fall into the local optimum. Summary of the invention

[0004] This paper proposes a UAV charging scheduling method based on deep reinforcement learning to deal with the charging recommendation problem of UAVs and efficiently allocate UAVs to appropriate charging stations. The present invention mainly includes the following steps:

[0005] Step S1: Consider each charging station as an independent agent, describe the problem as a multi-objective multi-agent reinforcement learning task, mathematically model the problem, and set the state space, action space, and reward function.

[0006] Step S2: Drawing on the Actor-Critic framework, we adopt a centralized training and decentralized execution approach to coordinate the behaviors of multiple Actors through a centralized Critic network.

[0007] Step S3: According to the urgency of the mission, the drones are divided into emergency and non-emergency categories, and the cumulative priority queue APQ is used to simulate the process of drones queuing for charging at the charging station.

[0008] Step S4: Introduce the attention level system into the centralized critic network, and adaptively guide the optimization direction through the dynamic gradient weighting strategy to avoid the problem of too large state action space.

[0009] Step S5: Apply the Truly PPO algorithm to multi-agent reinforcement learning to solve the shortcomings of the multi-agent proximal optimization algorithm MAPPO.

[0010] Preferably, when mathematically modeling the problem in step S1, a set of N charging stations A = {a1, a2, ..., a n} and a set of charging requests R. The drone can send a charging request at any time, and a charging request can be represented by r t =<l t , T t , T d , p i >, where l t is the location of the charging request, T t is the request time, T d is the charging duration, p i Indicates the growth rate of the user's priority. The charging success rate is defined as the ratio of the number of requests that accept the recommendation and are successfully charged to the total number of requests that accept the recommendation, and the charging waiting time is defined as the sum of the driving time from the request location to the charging station and the waiting time in the queue for charging. When the previous drone is fully charged, the request with the highest priority is selected from the set for charging. The ultimate goal is to develop a real-time intelligent charging recommendation algorithm to assign each drone that makes a charging request to a suitable charging station, minimize the overall waiting time, and improve the charging success rate of accepting the system recommendation.

[0011] Preferably, the centralized training and distributed execution in step S2 means: each charging station i∈I is regarded as an intelligent agent, and it is assumed that in the centralized training stage, the intelligent agent can observe the environmental state and share a centralized Critic network; this network uses the environmental state and joint action as input to evaluate the behavior of each intelligent agent; in the distributed execution stage, the Actor network of the intelligent agent makes decisions based on its own observations. This method can make full use of global information and reduce the demand for communication and calculation to balance global optimization and execution efficiency.

[0012] Preferably, the cumulative priority queue method in step S3 means that the drones are divided into two priorities: emergency and non-emergency. Consider a drone queuing system with two priority situations and the number of charging stations greater than or equal to 1, which has a rate λ P and λ s Poisson arrival process with probability b P and bP Accumulation priority, where b P ≥ b s ≥0. If a drone arrives and a charging station is available, it can charge immediately. When all charging stations are occupied, the priority accumulates at a rate of b′. Therefore, if a drone arrives at time t and is still in the queue at time t′, the priority of the drone is b′(t′-t). The cumulative priority queue method can adjust the relative waiting time of drones and arrange the charging priority of drones more fairly.

[0013] Preferably, the dynamic gradient weighted strategy in step S4 means that for a drone that needs to be charged, only the nearest charging station needs to be considered, and the Critic with focused attention learning is used to determine which nearby charging stations are involved, and the scoring function of the additive model is used for scoring. Then, the weight α of each agent can be obtained from the scoring function. When calculating the weight of the charging station, the competition of drones for charging piles is taken into consideration. For a drone requesting charging, during the time it goes to the charging station and queues, there may be other drones with higher priorities charging before it. This potential competition is difficult to quantify. To solve this problem, from the moment the request is made, the number of charging stations c is counted every m minutes. i The number of available charging posts (can be negative) until charging starts, which is used as the future competition situation, recorded as (I i ) t . Through a fully connected layer to obtain c i Regarding potential competition for this request:

[0014]

[0015] Where W fc is the learning factor, the potential competition is concatenated into the vector x in the scoring function mentioned above t The goal of introducing the attention mechanism is to solve the problem of large state and action space and reduce the amount of calculation.

[0016] Preferably, in step S5, since the PPO algorithm in MAPPO does not guarantee the stability and exploratory nature of the training process, the Truly PPO algorithm is used instead. By using the KL divergence, Truly PPO limits the strategy distribution between the new and old strategies to be proportional to the KL divergence distance, making the strategy update more stable and reliable. At this time, the optimization goal of each agent i is:

[0017]

[0018] By limiting the strategy distribution, the algorithm strategy update is made more stable and more exploratory. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the UAV charging scheduling strategy based on deep reinforcement learning described in the present invention.

[0020] Figure 2 Request scheduling model for charging.

[0021] Figure 3 This is a state transition diagram.

[0022] Figure 4 Schematic diagram of centralized training distribution execution.

[0023] Figure 5 The following is a comparison chart of the training results. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0025] Attached Figure 1 This is a flow chart of a UAV charging scheduling strategy based on deep reinforcement learning according to the present invention. The specific implementation steps of the present invention include:

[0026] Step S1: Consider a set of N charging stations A = {a1, a2, ..., a n} and a set of charging requests R. The drone can send a charging request at any time, and a charging request can be represented by r t =<l t , T t , T d , p i >, where l t is the location of the charging request, T t is the request time, T d is the charging duration, and pi represents the growth rate of the user's priority. The service cycle is measured in days, and the time interval uses the real time of the real world. There are two states of charging requests: success, or charging failure due to exceeding the maximum waiting time. The charging success rate is defined as the ratio of the number of requests that accept the recommendation and charge successfully to the total number of requests that accept the recommendation, and the charging waiting time is defined as the sum of the driving time from the request location to the charging station and the waiting time in the queue for charging. Defined as c iThe set of drones waiting for charging at time t, when the previous drone is finished charging, select the request with the highest priority from the set for charging. The problem studied in this paper is to develop a real-time intelligent charging recommendation algorithm to assign each drone that makes a charging request to a suitable charging station, minimize the overall waiting time, and improve the success rate of charging recommended by the system. The charging request scheduling model is as follows Figure 2 As shown. The state space, action space and reward function are set as follows:

[0027] State space: For the state at time T t Incoming charging request t The information that each agent i can observe is the agent number, the current time T t , the number of idle charging piles at the charging station, the number of charging requests around the charging station in the future, and the estimated time to arrive at the charging station, recorded as (o i ) t . The information observed by all agents is used as a model in T t The state at the moment is denoted as s t =(o 1 ) t , (o 2 ) t , ..., (o i ) t .

[0028] Action space: The purpose of the present invention is to recommend a suitable charging station for each charging request, so a charging request has only two options for a charging station: select or not select. Only one charging station can be selected. It is defined as the joint action at time t. In order to coordinate the decisions of each intelligent agent, the action value of the charging station with shorter queuing time and higher charging success rate is set to 1, and the others are 0.

[0029] State transition: For each agent, the period from the completion of the previous charging request to the arrival of the next charging request is called the state transition period, that is, the state transition period of agent i from arrive During this time in between. Figure 3 As shown, at time T1, a drone requires charging. According to the above, all charging stations are connected according to the status at this time. Select a suitable charging station to charge it and perform joint action ac1. When T2, the drone is fully charged, c i The status of does not change, but is updated to the status of the next charging request.

[0030] Reward function: A delayed reward settlement scheme, that is, the total reward during the period is calculated after the next request arrives; if a charging request r t Successfully, the environment returns the negative value of the waiting time as part of the reward. t If the waiting time WT exceeds a certain threshold, the recommendation is considered a failure, and the environment returns a smaller reward as a penalty to increase the charging success rate; in general, the reward function is defined as follows:

[0031]

[0032] where ∈ WT is the penalty reward; the above reward is a global reward, shared by all agents; during the state transition, multiple drones may complete charging, so the discounted rewards obtained by completing all requests during the state transition are added together as the cumulative reward. The cumulative reward from T1 to T2 can be written as:

[0033]

[0034] Where T t is the time it takes for each drone to complete charging within the time period, and γ is the discount factor.

[0035] Step S2: Use centralized training and decentralized execution to coordinate the behaviors of multiple Actors through a centralized Critic network, such as Figure 4 As shown in the figure, the specific details are as follows: The process is divided into three parts: centralized training, delayed access strategy combining future information, and decentralized execution process. The centralized Critic network takes the states and joint actions of all agents as input. During the training process, the agents work together and continuously adjust strategies to maximize the overall η(π). During training, each agent makes independent decisions based on the environmental information it obtains, and does not communicate with each other. Charging requests will be assigned to the charging station with the highest reward. In the decentralized execution process, each agent does not need all the information during training, which improves execution efficiency. Specifically, for a charging request, agent i∈I uses the following formula in its observation Based on action.

[0036]

[0037] Step S3: Process the charging results of each charging station when the request appears, that is, when the drone is finished charging, the corresponding charging station will release the corresponding number of charging piles. When the drone is recommended to a charging station with free charging piles, the charging is successful and the environment will return a reward; if the charging station is full, the drone will wait in line and accumulate priority. If the maximum waiting time specified by the priority is exceeded, the charging fails. The next charging drone will choose the one with the highest priority in the waiting queue. For the request r at time t t , will Deposit into the experience replay pool.

[0038] Cumulative Priority Queuing (APQ) refers to dividing drones into two priorities: emergency and non-emergency. Consider a drone queuing system with two priority cases and the number of charging stations greater than or equal to 1, which has a rate λ P and λ s Poisson arrival process with probability b P and b P Accumulation priority, where b P ≥ b s ≥ 0. If a drone arrives and a charging station is available, it can charge immediately. When all charging stations are occupied, the priority accumulates at a rate of b′. Therefore, if a drone arrives at time t and is still in the queue at time t′, the priority of this drone is b′(t′-t). After completing the last charging request, the drone with the highest priority in the queue starts charging.

[0039] Step S4: In MA-TPPO, the centralized critic network takes the states and joint actions of all agents as input, which will cause large action space and large state space problems. So for a drone that needs to be charged, only a few nearby charging stations will respond to the request, regardless of those that are too far away. In order to determine which nearby charging stations are involved, the attention mechanism is used to learn the centralized critic to solve the problem. The scoring function of the additive model is used for scoring:

[0040]

[0041] where v and W a is the learned parameter, x i is a vector composed of the state, action and future information of agent i. Next, the weight α of each agent can be obtained by the scoring function:

[0042]

[0043] When calculating the weight of charging stations, the competition of drones for charging piles is taken into account. For a drone requesting charging, during the time it goes to the charging station and queues, there may be other drones with higher priority charging before it. This potential competition is difficult to quantify. To solve this problem, the number of drones at charging station c is counted every m minutes from the moment the request is made. i The number of available charging posts (can be negative) until charging starts, which is used as the future competition situation, recorded as (I i ) t . Through a fully connected layer to obtain c i Regarding potential competition for this request:

[0044]

[0045] Where W fc is the learning factor, the potential competition is concatenated into the vector x in the scoring function mentioned above t middle.

[0046] Step S5: During the training process, each agent cooperates and continuously adjusts the strategy to maximize the overall η(π), but the optimization of random strategies is often unstable, and it cannot be guaranteed that the strategy update will always be improved. Therefore, MAPPO imposes restrictions on poorly performing strategies through the clipping function, making the training and learning process more stable and efficient. However, the PPO algorithm does not guarantee the stability and exploratory nature of the training process, so the Truly PPO algorithm is used instead. The optimization goal of each agent i is:

[0047]

[0048] Step S6: Carry out simulation experiments according to the steps above. Due to the lack of large-scale application of drones and the lack of charging stations, the data in this article is obtained by referring to the data of electric vehicle charging in reality. The number of charging stations and available charging piles is crawled in a publicly accessible software. Convert the map coordinates into a 230*170 area, and divide the area into 1×1 grids. 58 charging stations are set up in the map, and the number of charging requests in the surrounding area in the next 15 minutes (that is, the grid where the charging station is located and the 8 adjacent grids) is summarized as the future demand of the corresponding charging station. The charging request data is randomly generated according to a Poisson distribution with an expectation of 0.2. It is generated for 30 days, and a total of 94,889 requests are used as training data, of which first-level requests account for 20% and second-level requests account for 80%. Take 10 days of training data and generate 10 days of requests again, a total of 20 days of new and old data for testing. Penalty reward ε WTAll experiments were conducted on a Linux server with two NVIDA2090 GPUs. b = 0.4, m = 30, discount factor γ = 0.99, the number of hidden layers of the actor and critic networks was set to 256, the replay buffer size was 1000, the batch size was 32, and the hyperparameter α was 0.2.

[0049] Two evaluation indicators are defined to evaluate the performance of the model. First, R f is the set of first-level priority requests, R s is the set of second-level priority requests, and and are the sets of successful charging requests for the two priorities. In order to evaluate the recommended total charging waiting time, the average waiting time of all charging requests (W m ) and charging success rate CSR.

[0050] In order to verify the effectiveness of the model in this paper, it is compared with the following four methods: random, greedy-5, MAPP, and IPPO. The table shows the experimental results of the method of the present invention and other methods used for comparison on two indicators. Overall, among all the methods, the method proposed in this paper comprehensively surpasses the other four algorithms. Compared with IPPO and MAPPO, the method proposed in this paper improves the average waiting time and charging success rate by 30.08%, 5.51% and 14.4%, 2.59% respectively, which proves the effectiveness of the algorithm. This is because these two algorithms have problems with large state and action space, and it is difficult to learn a satisfactory strategy. Adding the attention mechanism to the Critic network can effectively avoid this problem. Figure 5 It can be seen that this algorithm is superior to MAPPO in terms of both convergence speed and reward.

[0051] Table 1 Performance of different algorithms on Wm and CSR

[0052]

[0053] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, they can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A UAV charging scheduling method based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1: Consider each charging station as an independent agent, describe the problem as a multi-objective multi-agent reinforcement learning task, mathematically model the problem, and set the state space, action space, and reward function; Step S2: Drawing on the Actor-Critic framework, we adopt a centralized training and decentralized execution approach to coordinate the behaviors of multiple Actors through a centralized Critic network. Step S3: According to the urgency of the mission, the drones are divided into emergency and non-emergency categories, and the cumulative priority queue APQ is used to simulate the process of drones queuing for charging at the charging station; Step S4: Introduce the attention level system into the centralized critic network, and adaptively guide the optimization direction through the dynamic gradient weighting strategy to avoid the problem of too large state action space; Step S5: Apply the Truly PPO algorithm to multi-agent reinforcement learning to solve the shortcomings of the multi-agent proximal optimization algorithm MAPPO.

2. The UAV charging scheduling strategy based on deep reinforcement learning according to claim 1 is characterized in that: The state space in step S1 is expressed as: t Incoming charging request t The information that each agent i can observe is the agent number, the current time T t , the number of idle charging piles at the charging station, the number of charging requests around the charging station in the future, and the estimated time to arrive at the charging station, recorded as (o i ) t ; All the information observed by the agents is used as a model in T t The state at the moment is denoted as s t =(o 1 ) t , (o 2 ) t , ..., (o i ) t .

3. The UAV charging scheduling strategy based on deep reinforcement learning according to claim 1 is characterized in that: The action space in step S1 is represented as follows: a charging request has only two options for a charging station: select or not select, that is, and can only select one charging station; therefore It is defined as the joint action at time t. In order to coordinate the decisions of each intelligent agent, the action value of the charging station with shorter queuing time and higher charging success rate is set to 1, and the others are 0.

4. The UAV charging scheduling strategy based on deep reinforcement learning according to claim 1 is characterized in that: The reward function in step S1 is expressed as: a delayed reward settlement scheme, that is, the total reward during the calculation period is calculated after the next request arrives; if a charging request r t Success, the environment returns the negative value of the waiting time as part of the reward; if an r t If the waiting time WT exceeds a certain threshold, the recommendation is considered a failure, and the environment returns a smaller reward as a penalty to increase the charging success rate; in general, the reward function is defined as follows: where ∈ WT is the penalty reward; the above reward is a global reward, shared by all agents; during the state transition, multiple drones may complete charging, so the discounted rewards obtained by all requests completed during the state transition are added together as the cumulative reward.

5. The UAV charging scheduling strategy based on deep reinforcement learning according to claim 1 is characterized in that: The core of the cumulative priority queue APQ in step S3 is that when a drone is recommended to a charging station with an empty charging pile, the charging is successful and the environment returns a reward; if the charging station is full, the drone will wait in line and accumulate priority. If the maximum waiting time specified by the priority is exceeded, the charging fails; the next drone to charge will choose the one with the highest priority in the waiting queue; for the request at time t r t, will Deposit into the experience replay pool.

6. The UAV charging scheduling strategy based on deep reinforcement learning according to claim 1 is characterized in that: The core of the dynamic gradient weighted strategy in step S4 is that, for a drone that needs to be charged, only the nearest charging station needs to be considered, and the Critic with focused attention learning is used to determine which nearby charging stations are involved; and the scoring function of the additive model is used for scoring: where v and W a is the learned parameter, x i is a vector composed of the state, action and future information of agent i; next, the weight α of each agent can be obtained by the scoring function:

7. The UAV charging scheduling strategy based on deep reinforcement learning according to claim 1 is characterized in that: The core of the Truly PPO algorithm in step S5 is that by using KL divergence, Truly PPO limits the strategy distribution between the new and old strategies to be proportional to the KL divergence distance. The optimization goal of each agent i is:

Images