Unmanned aerial vehicle path planning method based on offline reinforcement learning

Abstract

The invention belongs to the technical field of unmanned aerial vehicles, and discloses an unmanned aerial vehicle path planning method based on offline reinforcement learning, and the method comprises the steps: building a communication time delay model and an unmanned aerial vehicle energy consumption model of an unmanned aerial vehicle and a vehicle, carrying out the simulation of a traffic flow through employing an SUMO traffic simulator, recording the state data of the unmanned aerial vehicle and the vehicle, and carrying out the calculation of the unmanned aerial vehicle path. Generating an off-line training data set for off-line reinforcement learning training, designing an unmanned aerial vehicle state space, an action space and a reward function for deep reinforcement learning according to the off-line training data set, training an off-line reinforcement learning model, and combining with the initial position of the unmanned aerial vehicle to obtain an unmanned aerial vehicle state space, an action space and a reward function; and outputting the flight path of the unmanned aerial vehicle based on the trained offline reinforcement learning model. The method can effectively reduce the communication time delay while reducing the energy consumption of the unmanned aerial vehicle, thereby remarkably improving the task completion rate of the unmanned aerial vehicle in a complex dynamic environment.

Description

Technical Field

[0001] The present application belongs to the field of drone technology, and specifically relates to a drone path planning method based on offline reinforcement learning. Background Art

[0002] With the rapid development of science and technology, drones (UAVs) have been widely used in fields such as logistics and the military due to their unique advantages. This has made UAV path planning technology a highly sought-after research topic. Traditional UAV path planning algorithms include several classic methods. The Dijkstra and A* algorithms, based on geometric methods, construct graph structures and follow specific rules to search for the shortest path. Evolutionary algorithms and particle swarm optimization algorithms, on the other hand, fall into the category of heuristic search algorithms, simulating natural evolution or swarm behavior to find optimal paths. These traditional algorithms can, to a certain extent, plan reasonable paths for UAVs in static or relatively simple dynamic environments. However, with the increasing complexity of application scenarios, the limitations of these traditional algorithms are becoming increasingly apparent. When UAVs face complex and dynamically changing environments, such as those with numerous moving obstacles and constantly updating environmental information, traditional algorithms often fail to perform satisfactorily in terms of real-time obstacle avoidance and emergency response. Furthermore, in practical applications, UAVs must complete path planning tasks without increasing energy consumption. Traditional algorithms, however, lack sufficient consideration for energy optimization, making them difficult to meet this requirement.

[0003] To improve the performance of drone path planning, path planning methods that combine deep learning and reinforcement learning are gaining popularity. Reinforcement learning, particularly offline reinforcement learning, effectively reduces the computational overhead required for online learning by leveraging pre-collected historical data for training, while also mitigating the potential risks of online learning and demonstrating improved adaptability and efficiency. However, existing path planning methods that combine deep learning and reinforcement learning primarily focus on path planning itself, failing to fully consider factors such as communication between the drone and the outside world and energy consumption.

[0004] In the intelligent transportation system supported by UAVs, the UAVs often need to communicate with multiple vehicles simultaneously to obtain global traffic information for optimal path planning and task allocation. Therefore, the communication delay will affect the coordination capability of the entire system. If the communication delay is too large, the UAVs may not be able to obtain accurate traffic data in time, thereby affecting their global decision-making capability in complex traffic environments. Especially in emergency scenarios, such as road risk warning or emergency vehicle priority, communication delay is directly related to the success or failure of the task. In addition, in the path planning of UAVs, energy consumption is also one of the key factors affecting the efficiency and endurance of UAV task execution. Traditional path planning algorithms often focus on finding the shortest path or avoiding obstacles, while ignoring the impact of energy consumption. However, factors such as flight distance, flight altitude, speed, and load of UAVs will directly affect their energy consumption, especially when dealing with complex terrain and dynamic environments, efficient use of energy becomes even more critical. Excessive energy consumption not only leads to task failure, but also forces the UAV to return early due to insufficient power, and cannot complete the scheduled task.

[0005] Therefore, there is an urgent need for a path planning method that can consider both the communication delay of UAVs and vehicles and the energy consumption of UAVs. SUMMARY

[0006] In view of the problems of large communication delay and high energy consumption in the path planning of UAVs in the prior art, the present application proposes a UAV path planning method based on offline reinforcement learning. This method aims to learn offline through historical flight data and expert knowledge, construct an implicit action space policy model with disturbance, and dynamically optimize path selection to reduce communication delay and energy consumption of UAVs, thereby significantly improving the task completion rate of UAVs in complex dynamic environments, and providing strong support for efficient operation of UAVs in various practical application scenarios.

[0007] In order to achieve the above technical purpose, the present application is realized by the following technical solutions:

[0008] In one aspect of the present application, a UAV path planning method based on offline reinforcement learning is provided, comprising the following steps:

[0009] A communication delay model of UAVs and vehicles and a UAV energy consumption model are established; the communication delay model uses queuing theory modeling with preemptive M / G / 1 priority queue for high-priority safety warning signals and low-priority state update signals, calculates the queuing delay of the two types of signals, and the total communication delay is the sum of transmission delay and queuing delay; the UAV energy consumption model includes communication energy consumption, flight energy consumption, and calculation energy consumption;

[0010] According to the communication delay model and the UAV energy consumption model, a traffic flow is simulated by using a SUMO traffic simulator, and state data of the UAV and the vehicle are recorded to generate an offline training data set for offline reinforcement learning training;

[0011] According to the offline training data set, a UAV state space, an action space and a reward function for deep reinforcement learning are designed, and an offline reinforcement learning model is trained;

[0012] In combination with an initial position of the UAV, a flight path of the UAV is output based on the trained offline reinforcement learning model.

[0013] In an embodiment, for the ith UAV, the communication delay model satisfies that an average communication delay of a vehicle within a communication range is:

[0014]

[0015] wherein ρ1 represents an occupancy rate of the first type of message, ρ2 represents an occupancy rate of the second type of message, λ j represents an arrival rate of the jth type of message, μ j represents a service rate of the jth type of message, j ∈ {1, 2}; E[B j ] represents an average service time of the jth type of message; represents an average value of the square of the service time; E[W j ] represents an average waiting time of the jth type of message; d i represents a message transmission path length; C i represents a channel transmission rate.

[0016] In an embodiment, the UAV energy consumption model is:

[0017]

[0018] wherein, represents a set of all vehicle clients participating in federated learning within a communication range of the UAV i i; S i represents a message size to be sent by the UAV i i; represents a transmission power of the UAV i, represents a reachable data transmission rate of a channel between the UAV i i and the vehicle j; C and φ are respectively an energy consumption coefficient of a computing system, a computing period required for each byte of computation, a frequency of a CPU clock of the UAV, and S(w i ) is a packet size sent to the client i; dMM' representing the UAV i distance between two consecutive positions, V i representing the UAV i velocity, P i representing the UAV i at velocity V i motion power.

[0019] In one embodiment, the reward function is:

[0020]

[0021] wherein, R W , R E , respectively related to road vehicle density, UAV-vehicle communication delay, UAV energy consumption, average risk time of vehicles and average congestion time of vehicles, R W = W, R E = E, p v denotes road vehicle density, W denotes total communication delay of all UAV-vehicles, E denotes total energy consumption of all UAVs, t r denotes average risk time of all vehicles, t b denotes average congestion time of all vehicles. w W , w E , are corresponding weights respectively.

[0022] In one embodiment, a training experience pool is constructed during training to store an offline training data set (S, A, R, S'), wherein S is the current state of the UAV, A is the action taken by the UAV, R is the reward obtained, and S' is the next state of the UAV; data is sampled in batches from the experience pool in a batch training manner, and the Q value neural network parameters are updated in combination with a gradient descent algorithm.

[0023] In one embodiment, the loss function used by the gradient descent algorithm is:

[0024] (y t - Q t (S, A; θ)) 2 ;

[0025] wherein y t = R + γmaxQ(S', A; θ'), parameter γ represents a discount factor of the reward function R value; S is the current state of the UAV, S' is the next state of the UAV; A is the action taken by the UAV; θ represents the current state model parameter of the UAV, and θ' represents the next state model parameter of the UAV.

[0026] In one embodiment, the offline reinforcement learning model adopts a perturbed implicit action space policy model.

[0027] In one embodiment, the action selection adopted in the training process follows an ε-greedy strategy: according to a predetermined probability ε, the optimal action that can obtain the maximum reward function value is selected; with a probability (1-ε), a random action is selected from all possible actions.

[0028] In one embodiment, in the process of simulating traffic flow by using the SUMO traffic simulator, the Krauss following movement model is selected as the simulation model of the following behavior of the longitudinal motion of the vehicle, and the LC2013 lane changing model is used to simulate the lane changing behavior of the vehicle.

[0029] In one embodiment, in the SUMO traffic simulator setting, the default flight path of the UAV is set as an elliptical path.

[0030] The beneficial effects of the present application are:

[0031] 1) The dynamic path planning method based on deep reinforcement learning adopted by the present application can quickly adjust the path planning strategy according to the real-time environmental data of the UAV during flight, effectively improving the real-time performance and response speed of path planning in complex environments, and ensuring that the UAV can quickly make planning decisions in a changing environment.

[0032] 2) By considering the communication delay between the UAV and the ground vehicle, the present application can dynamically adjust the flight strategy and task scheduling of the UAV during path planning, reducing unnecessary communication waiting time, thereby achieving more efficient collaborative work between the UAV and the vehicle and improving the efficiency of the overall communication network.

[0033] 3) The present application accurately evaluates and optimizes the energy consumption of the UAV during path planning, thereby effectively reducing the energy consumption of the UAV during task execution and maximizing the endurance time of the UAV, especially suitable for complex task scenarios with long time and high load.

[0034] 4) By training a perturbed implicit action space policy model, the present application can ensure the stability and global optimality of path planning when facing dynamic, multi-obstacle and uncertain complex environments, thereby significantly reducing the risk of path deviation and task failure of the UAV due to unexpected situations.

[0035] 5) The present application supports multiple UAVs working together, and through the combination of global and local path planning, it can achieve task allocation and path coordination among multiple UAVs in complex scenarios, avoid conflicts between UAVs, and ensure the implementation of the overall optimal path planning strategy during task execution.

[0036] 6) The path planning algorithm of the present application can adapt to different types of unmanned aerial vehicles and changes in task requirements, and can be quickly deployed and applied in different scenarios such as logistics distribution, disaster relief, traffic monitoring, etc. The algorithm module can be flexibly adjusted according to the needs of different application scenarios, and has good scalability and practicality. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 An offline reinforcement learning data collection and model test simulation environment diagram for an embodiment of the present application;

[0038] Figure 2 An unmanned aerial vehicle elliptical path diagram for an embodiment of the present application;

[0039] Figure 3 An unmanned aerial vehicle path prediction flowchart for an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions of the present application will be described in detail below in conjunction with specific embodiments, but those skilled in the art will understand that the following described embodiments are part of the embodiments of the present application, not all embodiments, and are only used to illustrate the present application, and should not be regarded as limiting the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0041] The present application proposes an unmanned aerial vehicle path planning method based on offline reinforcement learning, which simultaneously considers the communication delay between unmanned aerial vehicles and vehicles and the energy consumption of unmanned aerial vehicles, and trains an implicit action space policy model with disturbance through offline learning of historical flight data and expert knowledge. By optimizing path selection, the energy consumption and communication delay of unmanned aerial vehicles can be effectively reduced, and the task completion rate in complex dynamic environments can be significantly improved.

[0042] In one embodiment, an unmanned aerial vehicle path planning method based on offline reinforcement learning includes the following steps:

[0043] Step 1: Mathematical modeling of the communication delay between unmanned aerial vehicles and vehicles and the energy consumption of unmanned aerial vehicles.

[0044] 1) Wireless communication delay modeling

[0045] In the communication process between the UAV and the vehicle, the signals are divided into high-priority safety warning type signals and low-priority state update type signals. The high-priority safety warning type signals usually involve emergency or safety-related information, which need to be transmitted to the vehicle as soon as possible to ensure the safety of driving. The low-priority state update type signals mainly contain the state information of the UAV, such as position, speed, etc., although the real-time requirement of communication is relatively low, but also need to be transmitted regularly.

[0046] In order to analyze the queuing delay of signals at the UAV end, queuing theory is used for modeling, and M / M / 1 priority queue model is used in particular. For the queuing delay of high-priority signals, since high-priority signals have priority processing right, when they arrive at the UAV, they will be inserted into the front end of the processing queue (if the UAV is processing other low-priority signals, the current processing process will be suspended). Therefore, the queuing delay of high-priority signals mainly depends on whether there is other high-priority signal being processed at present. For the queuing delay of low-priority signals, low-priority signals will be queued after arriving at the UAV, waiting for all signals in front to be processed, and if high-priority signals arrive during processing, the current task will be interrupted and the high-priority signals will be processed. Therefore, the queuing delay of low-priority signals may be long and will be affected by the arrival of high-priority signals. The total communication delay is the sum of the transmission delay from the UAV to the vehicle and the queuing delay at the UAV end.

[0047] In some embodiments, for the ith UAV, for the jth message, the communication delay model satisfies that the average communication delay of vehicles within the communication range is:

[0048]

[0049] Wherein:

[0050] ρ1 represents the occupation rate of the first type of message, and ρ2 represents the occupation rate of the second type of message, λ j represents the arrival rate of the jth message, μ j represents the service rate of the jth message, j ∈ {1, 2};

[0051] E[B j ] represents the average service time of the jth message;

[0052] represents the average value of the square of the service time of the jth message;

[0053] E[W j ] represents the average waiting time of the jth message;

[0054] d i represents the length of the message transmission path;

[0055] C i denotes the channel transmission rate.

[0056] 2) UAV energy consumption modeling

[0057] The UAV energy consumption is mainly composed of communication energy consumption, flight energy consumption, and computing energy consumption. The communication energy consumption is the energy consumed by the UAV when sending and receiving signals, which is related to factors such as signal transmission distance, transmission rate, modulation mode, etc. The flight energy consumption is the energy consumed by the UAV during flight, which is related to factors such as flight speed, flight altitude, flight time, etc. The computing energy consumption is the energy consumed by the UAV when processing signals (including queuing, scheduling, encoding and decoding, etc.), which is related to factors such as processor performance and algorithm complexity of the UAV.

[0058] In some embodiments, for the UAV i , the energy consumption is the sum of the computing energy consumption, the communication energy consumption, and the flight energy consumption:

[0059]

[0060] wherein, denotes the set of all vehicle clients participating in federated learning within the communication range of the UAV i ;

[0061] S i denotes the size of the message to be sent by the UAV i ; denotes the transmission power of the UAV i, denotes the achievable data transmission rate of the channel between the UAV i and the vehicle j; k, C, and φ are the energy consumption coefficient of the computing system, the number of computing cycles required per byte of computation, and the frequency of the UAV CPU clock, respectively, while S(w i ) is the size of the data packet sent to client i; d MM' denotes the distance between two consecutive positions of the UAV i ; i denotes the speed of the UAV i ; i denotes the motion power of the UAV i at speed V i .

[0062] Step 2: According to the communication delay model and the UAV energy consumption model, use the SUMO traffic simulator to simulate the traffic flow, record the UAV and vehicle state data, and generate an offline training data set for offline reinforcement learning training.

[0063] SUMO (Simulation of Urban MObility) is an open-source, microscopic, multi-modal traffic simulation software that can simulate the behavior of vehicles, pedestrians, and other traffic participants in urban traffic. To simulate real traffic scenarios, record the state data of drones and vehicles to generate an offline training dataset. At the same time, the dataset is divided into 80% training set and 20% test set for testing. The training set is used to train the offline reinforcement learning model, so that it learns the optimal decision-making strategy of drones and vehicles in different states; the test set is used to evaluate the performance of the model and ensure that the model also performs well on unseen data.

[0064] In some embodiments, the Krauss moving model is used to simulate the longitudinal movement of vehicles. The Krauss model is a commonly used car-following model that adjusts the speed of the vehicle based on the speed and distance of the preceding vehicle to maintain a safe following distance. The LC2013 lane-changing model is used to simulate the lane-changing behavior of vehicles, which covers four types of lane-changing: strategic lane-changing, which is performed by vehicles to reach their destination or choose a better path; cooperative lane-changing, which is performed by vehicles to help other vehicles (such as yielding); speed-enhancing lane-changing, which is performed by vehicles to obtain higher driving speed; and emergency vehicle (such as ambulance) avoidance lane-changing, which is performed by vehicles to give way to emergency vehicles.

[0065] In some embodiments, in the simulation, the drone flies along an elliptical path. The simulation of the drone flying along an elliptical path is a scenario in which the drone patrols or monitors a certain area, ensuring that the drone can cover a certain spatial range and effectively communicate with vehicles.

[0066] Step three: design the drone state space, action space, and reward function for deep reinforcement learning based on the offline training dataset, and train the offline reinforcement learning model to optimize the communication and energy consumption performance between the drone and the vehicle.

[0067] The offline reinforcement learning model used in this application avoids the high risk and high cost of real-time interaction in the actual environment by training on pre-collected historical data, significantly improving training safety and efficiency compared to traditional path planning algorithms.

[0068] The state space contains various information describing the UAV and its surrounding environment. The state space contains the UAV's own coordinates, direction, vehicle information within the communication range, risk and congestion information. Among them, the UAV's own coordinates include horizontal coordinates, vertical coordinates, and height coordinates, which are used to determine the position of the UAV in three-dimensional space. The direction indicates the flight direction information of the UAV. The vehicle information within the communication range indicates the number of vehicles within the communication range, the average communication delay of the vehicles, and the average energy consumption, reflecting the communication status between the UAV and the vehicles. The risk and congestion information is the average risk time and the average congestion time of the vehicles within the communication range; the average risk time is the time length of the vehicle driving on the risk road without timely lane changing; the average congestion time is the time length of the vehicle driving in front of an ambulance, police car and other emergency vehicles without timely lane changing, which helps to evaluate the impact of traffic conditions on communication and UAV operation.

[0069] The action space defines the actions that the UAV can take, including the movable distance in longitude, latitude and vertical direction. The UAV can move in three-dimensional space according to these action parameters as optional actions.

[0070] The goal of the reward function is to guide the reinforcement learning algorithm to learn in the optimization direction, including maximizing the road vehicle density to reasonably utilize road resources and improve traffic efficiency; and minimizing the UAV-vehicle communication delay, UAV energy consumption, average risk time and average congestion time of vehicles to improve communication quality, reduce UAV energy consumption and improve traffic safety.

[0071] In some embodiments, the reward function is:

[0072]

[0073] wherein, R W , R E , are related to the road vehicle density, the UAV-vehicle communication delay, the UAV energy consumption, the average risk time and the average congestion time of vehicles, respectively, R W , R E , ρ v denotes the road vehicle density, W denotes the total communication delay of all UAV-vehicle, E denotes the total energy consumption of all UAVs, t r denotes the average risk time of all vehicles, t b denotes the average congestion time of all vehicles; w W , w E , are the corresponding weights, respectively.

[0074] In some embodiments, a training experience pool is constructed during the training process, storing an offline training dataset (S, A, R, S'), where S is the current state of the UAV, A is the action taken by the UAV, R is the reward obtained, and S' is the next state of the UAV; data is sampled in batches from the experience pool in a batch training manner, and the Q-value neural network parameters are updated in combination with the gradient descent algorithm. The loss function used by the gradient descent algorithm is:

[0075] (y t -Q t (S,A;θ)) 2 ;

[0076] where y t =R+γmaxQ(S',A;θ'), the parameter γ represents the discount factor of the reward function R value; S is the current state of the UAV, S' is the next state of the UAV; A is the action taken by the UAV; θ represents the current state model parameter of the UAV, and θ' represents the next state model parameter of the UAV.

[0077] In some embodiments, the offline reinforcement learning model adopts a latent action space policy model (PLAS, Policy in the Latent Action Space) with perturbation. The policy learning is optimized by mapping the complex original action space to a low-dimensional latent space. This model uses an encoder to convert high-dimensional actions to a latent space, reducing redundancy and noise and making the planning of the policy more stable and efficient; by introducing a perturbation mechanism, the actions in the latent action space are slightly randomly disturbed or adjusted, increasing the robustness of the policy and avoiding getting stuck in a local optimal solution, which can better cope with uncertainty and improve the generalization performance of the policy and its performance in practical applications.

[0078] During the training process, the action selection follows the ε-greedy strategy: according to the predetermined probability ε, the optimal action that can obtain the maximum reward function value is selected; with probability (1-ε), a random action is selected from all possible actions.

[0079] Step four: based on the initial position of the UAV, output the flight path of the UAV based on the trained offline reinforcement learning model.

[0080] Specifically, the following steps are included:

[0081] 1) Determine the starting coordinates of the UAV in three-dimensional space, which usually includes coordinate values in the horizontal direction (such as longitude), vertical direction (such as latitude), and height direction. This initial position is the starting point for the UAV flight path planning, and all subsequent calculations and movements are based on this position.

[0082] 2) Input the current time's UAV's state parameters into the trained offline reinforcement learning model. These state parameters can include the horizontal coordinate, vertical coordinate, height coordinate, direction, the number of vehicles in the communication range, the average communication delay of vehicles, the average energy consumption, the average risk time and average congestion time of vehicles in the communication range, etc. The specific parameters are consistent with the state space defined in step three.

[0083] 3) The offline reinforcement learning model calculates according to the input state parameters, and outputs the predicted movement of the UAV at the current time step. The movement is reflected as the lateral (e.g. longitude direction) and vertical (e.g. latitude direction) offset between the current position and the next position, and can also include the change in height.

[0084] 4) Add the offset output by the model to the current position of the UAV to calculate the new position; the UAV adjusts its position and attitude according to the calculated new position to achieve the new position.

[0085] 5) After reaching the new position, input the new state parameters into the offline reinforcement learning model again, and repeat the process of calculating the predicted movement and moving according to the predicted value. By continuously updating the state parameters of the UAV, calculating the predicted movement and performing the movement operation, the UAV can gradually fly along the path planned by the trained offline reinforcement learning model until the predetermined flight target is reached or the termination condition is met. For example, the flight time reaches the upper limit or reaches the specified destination.

[0086] Embodiment

[0087] 1) Wireless communication delay modeling

[0088] Consider two types of signals sent by UAVs to vehicles, one is high-priority safety warning type signal, and the other is state update type low-priority signal. The queuing delay is modeled by queuing theory, and the queuing time of the two types of signals is obtained respectively. In particular, this queue is a M / G / 1 priority queue with preemption. For the i-th UAV, the average communication delay of all vehicles in the communication range for the j-th message is calculated as follows:

[0089]

[0090] 2) UAV energy consumption modeling

[0091] For UAV i , the energy consumption is the sum of the calculation energy consumption, communication energy consumption and flight energy consumption:

[0092]

[0093] 3) Data collection phase

[0094] The considered simulation environment consists of multiple UAVs and ground vehicles, as Figure 1 shown, a 3-lane, 4-kilometer long highway is simulated, with four UAVs flying above it. The UAVs can fly at a horizontal speed of 30 km / h, a vertical speed of 10 km / h, and a flying height between 100 and 150 meters. The movement of the vehicles is achieved through the SUMO traffic simulator, using the Krauss movement model and the LC2013 lane changing model. The LC2013 model encompasses four types of lane changing: strategic lane changing, cooperative lane changing, speed boost lane changing, and lane changing for emergency vehicles (e.g., ambulances). As Figure 2 shown, the UAVs fly along an elliptical path in the simulation, providing monitoring data of the overall traffic situation. Each UAV is responsible for an area and communicates with the vehicles within its area of responsibility through a wireless channel, assisting in the driving of the vehicles. The UAVs fly above the road and communicate with the vehicles through a wireless network, sending signals in real time.

[0095] The offline reinforcement learning training dataset is obtained through simulation, which contains 10,000 samples, i.e., a simulation duration of 66 minutes and 40 seconds, of which 80% is used for training and 20% is used for testing. The observation data o = {o1, o2, o3, o4} of each sample is the state parameters of UAV1, UAV2, UAV3, and UAV4 at the current time.

[0096] At time t, the state parameters of UAV i are:

[0097]

[0098] where,

[0099] x i [t] is the horizontal coordinate of UAV i at time t;

[0100] y i [t] is the vertical coordinate of UAV i at time t;

[0101] z i [t] is the height coordinate of UAV i at time t;

[0102] θ i [t] is the direction indication of UAV i at time t;

[0103] ρ vi is the number of vehicles within the communication range of UAV i at time t;

[0104] is the total time t in the UAV i Average communication delay of vehicles within the communication range;

[0105] It's a UAV i Average energy consumption at time t;

[0106] and is the time t for all the i Average risk time and average blocking time for cars within communication range.

[0107] 4) Model training phase

[0108] During the training process, a training experience pool is constructed to store experience data (S, A, R, S'). Data is sampled in batches from the experience pool according to the batch training method. The training provides the neural network parameters with Q values. The training process uses the gradient descent algorithm to update the parameters.

[0109] The specific loss function used is: (y t -Q t (S,A;θ)) 2 , where y t =R+γmaxQ(S',A;θ'), where the parameter γ represents the reduction coefficient of the reward function R value, and is 0.95 in this example.

[0110] The action selection during training follows the ε-greedy strategy, that is, the action that can obtain the maximum reward value is selected with probability ε, and an action in the action space is randomly selected with probability (1-ε).

[0111] In order to optimize both the UAV-vehicle communication delay and the UAV energy consumption when planning the UAV path, this application designs a sophisticated offline reinforcement learning reward function, which includes the road vehicle density, UAV-vehicle communication delay, UAV energy consumption, the average risk time of the vehicle and the average blocking time: reward function in, R W =W,R E =E, The goal of this reward function is to maximize the vehicle density ρ v , while minimizing other parameters (i.e. W, E, t r ,t b ).

[0112] 5) UAV path planning stage

[0113] Based on the trained offline reinforcement learning model, the flight path is planned for the drone. The specific steps are as follows: Figure 3 As shown:

[0114] 1) Initial position setting: In the first iteration, the initial position (x, y coordinates) of the UAV needs to be input as the initial state of the model. This position represents the initial position of the UAV in the current environment, so that the model can make subsequent trajectory predictions based on this position;

[0115] 2) UAV path prediction: The current parameters of the UAV, i.e. the observation data at the current time, are input. As mentioned earlier, at time t, is the state parameter of the UAV i . Through the trained offline reinforcement learning model, the predicted movement of the UAV at the current time step is output, i.e. the lateral and longitudinal offset between the current position and the next position;

[0116] 3) UAV moves according to the predicted value: According to the offset output by the model, it is added to the current position of the UAV to obtain the new position of the UAV after the current time step;

[0117] 4) Loop update: The updated new position will be used as the starting position for the next iteration to predict the next position offset. This process is repeated until the termination condition is reached.

[0118] The advantage of the present application is that by integrating the communication delay optimization and energy management module, combined with the offline reinforcement learning model, the delay and energy consumption of the UAV in the communication process with the vehicle are effectively reduced, while achieving efficient path planning and strategy optimization under low computational complexity, significantly improving the real-time performance and energy utilization efficiency of the model, and having good scalability and flexibility.

[0119] Although the embodiments of the present application are described above in combination with the drawings, the present application is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and guiding, but not limiting. Those skilled in the art can make many forms under the inspiration of the present specification and without departing from the scope protected by the claims of the present application, which are all within the scope of protection of the present application.

Claims

1. The UAV path planning method based on offline reinforcement learning is characterized by: include: Establish a communication delay model between drones and vehicles and a drone energy consumption model; The communication delay model uses queuing theory modeling with preemptive M / G / 1 priority queues for high-priority safety warning signals and low-priority status update signals to calculate the queuing delays of the two types of signals. The total communication delay is the sum of the transmission delay and the queuing delay. The UAV energy consumption model includes communication energy consumption, flight energy consumption and computing energy consumption; Based on the communication delay model and the UAV energy consumption model, the traffic flow is simulated using the SUMO traffic simulator, the UAV and vehicle status data are recorded, and an offline training dataset is generated for offline reinforcement learning training; Designing a drone state space, action space, and reward function for deep reinforcement learning based on the offline training dataset, and training an offline reinforcement learning model; Combined with the initial position of the UAV, the flight path of the UAV is output based on the trained offline reinforcement learning model.

2. The UAV path planning method according to claim 1, characterized in that: For the i-th UAV and the j-th message, the communication delay model satisfies the average communication delay of vehicles within the communication range: Among them, ρ1 represents the occupancy rate of the first type of messages, ρ2 represents the occupancy rate of the second type of messages, λ j represents the arrival rate of the j-th type of message, μ j represents the service rate of the jth type of message, j∈{1,2}; E[B j ] represents the average service time of the jth type of message; represents the average value of the square of the service time; E[W j ] represents the average waiting time of the jth type of message; d i Indicates the length of the message transmission path; C i Indicates the channel transmission rate.

3. The UAV path planning method according to claim 1, characterized in that: For the i-th drone, the drone energy consumption model is: in, Indicates that in UAV i The set of all vehicle clients participating in federated learning within the communication range; S i Indicates UAV i The size of the message to be sent, represents the transmission power of UAV i, Indicates UAV i The achievable data transmission rate of the channel between vehicle j; k, C, and φ are the energy consumption coefficient of the computing system, the computing cycles required for each byte of computing, and the frequency of the drone CPU clock, respectively. i ) is the size of the data packet sent to client i; d MM' Indicates UAV i The distance between two consecutive positions, V i Indicates UAV i Speed, P i Indicates UAV i At speed V i The power of movement under.

4. The UAV path planning method according to claim 1, characterized in that: The reward function is: in, R W 、R E 、 They are respectively related to road vehicle density, UAV-vehicle communication delay, UAV energy consumption, average risk time and average blocking time of vehicles.

5. The UAV path planning method according to claim 1, characterized in that: During the training process, a training experience pool is constructed to store the offline training data set (S, A, R, S'), where S is the current state of the drone, A is the action taken by the drone, R is the reward obtained, and S' is the next state of the drone; data is sampled in batches from the experience pool according to the batch training method, and the Q-value neural network parameters are updated in combination with the gradient descent algorithm.

6. The UAV path planning method according to claim 5, characterized in that: The loss function used by the gradient descent algorithm is: (y t -Q t (S,A;θ)) 2 ; Among them, y t =R+γmaxQ(S',A;θ'), where the parameter γ represents the reduction coefficient of the reward function R; S is the current state of the drone, S' is the next state of the drone; A is the action taken by the drone; θ represents the model parameters of the current state of the drone, and θ' represents the model parameters of the next state of the drone.

7. The UAV path planning method according to claim 1, characterized in that: The offline reinforcement learning model adopts an implicit action space policy model with perturbations.

8. The UAV path planning method according to claim 7, characterized in that: The action selection used in the training process follows the ε-greedy strategy: the optimal action that can obtain the maximum reward function value is selected according to the predetermined probability ε; an action is randomly selected from all possible actions with probability (1-ε) to execute.

9. The UAV path planning method according to claim 1, characterized in that: In the process of simulating traffic flow using the SUMO traffic simulator, the Krauss car-following mobility model is selected as the car-following behavior simulation model of vehicle longitudinal motion, and the LC2013 lane-changing model is used to simulate the vehicle's lane-changing behavior.

10. The UAV path planning method according to claim 1, characterized in that: In the SUMO traffic simulator settings, the default flight path of the drone is set to an elliptical path.

Images