Digital twinning training method for reinforcement learning driven unmanned aerial vehicle trajectory planning

Through a digital twin training method for reinforcement learning, combined with GPS and sensor information, optimized drone trajectory planning, the problem that traditional methods are difficult to cope with dynamics and uncertainties in complex environments is solved, and more efficient and safer Drone mission execution.

CN119026458APending Publication Date: 2024-11-26BEIJING INST OF TECH
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411010482.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-26

Smart Images

  • Figure CN119026458A_ABST
    Figure CN119026458A_ABST
Patent Text Reader

Abstract

The invention provides a digital twin training method for reinforcement learning driving unmanned aerial vehicle trajectory planning, and belongs to the field of communication, and the method comprises the steps: building a digital twin model # imgabs0 # of a task environment in a central server based on GPS information and unmanned aerial vehicle sensor perception information, and building a digital twin virtual-real interaction channel; a ground node set # imgabs1 # responsible for each unmanned aerial vehicle is determined, and the center position of the ground node set # imgabs2 # serves as the initial position of task execution of the unmanned aerial vehicle; designing a state space S, an action space A and a revenue function R of reinforcement learning for a specific task executed by the unmanned aerial vehicle, and determining a reinforcement learning model M; constructing a digital twinborn deduction simulation platform DT-S < im > to obtain empirical data B; constructing a digital twinborn decision model DT-Dec, and assisting the pre-training of the reinforcement learning model M; and performing reinforcement learning training. The method can effectively improve the convergence speed, reduce the security threat of the unmanned aerial vehicle, and improve the task security and strategy efficiency of the unmanned aerial vehicle in the communication task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of communications, and in particular relates to a digital twin training method for reinforcement learning driven unmanned aerial vehicle trajectory planning. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in the field of communications. In the military field, drones can not only be used for reconnaissance and surveillance, but also for battlefield support and air strikes. In the civilian field, drones also play an important role in emergency communications, environmental testing, modern agriculture, low-altitude economy and other fields. However, in complex environments, traditional drone trajectory planning methods are usually based on mathematical models or empirical rules, which make it difficult to fully consider the dynamics and uncertainty of the real environment. Reinforcement learning technology, as a method of updating strategies through the interaction between intelligent agents and the environment, can update strategies in a deterministic dynamic environment and improve mission effectiveness.

[0003] However, the acquisition of experience data by traditional reinforcement learning methods depends on the interaction between the agent and the environment. In complex tasks, the cost of collecting experience data will increase rapidly as the scale of experience data increases. Corresponding to drone-related tasks, relying on drone flights to obtain experience data will not only greatly increase the difficulty of obtaining experience data, but also bring unbearable drone losses. Especially in special scenarios in urban environments with densely distributed buildings, drones face many challenges in performing tasks, such as obstacle occlusion and flight safety threats. Summary of the invention

[0004] The purpose of this invention is to propose a digital twin training method for reinforcement learning driven UAV trajectory planning, which can effectively improve the convergence speed, reduce the security threats of UAVs, and improve the mission safety and strategy effectiveness of UAVs in communication tasks.

[0005] The present invention is achieved through the following technical solutions:

[0006] A digital twin training method for reinforcement learning driven UAV trajectory planning, comprising the following steps:

[0007] Step S1: Based on GPS information and drone sensor perception information, a digital twin model of the mission environment is constructed on the central server. Digital Twin Model Including mission area, ground node model Drone Model Building Model And the drone sensor perception results

[0008] Step S2: construct a digital twin virtual-real interaction channel DT-InC to realize two-way interaction between the digital twin model and the real world;

[0009] Step S3: Based on the digital twin model Ground node model and drone models According to the K-nearest neighbor algorithm clustering results, the set of ground nodes that each drone is responsible for is determined. Ground node collection The center position of is taken as the initial position of the UAV to perform the task, where n U is the number of drones;

[0010] Step S4: Design the state space S, action space A and benefit function R of reinforcement learning for the specific task to be performed by the drone, and determine the reinforcement learning model M;

[0011] Step S5: construct a digital twin simulation platform DT-Sim to obtain empirical data B;

[0012] Step S6: construct a digital twin decision model DT-Dec to assist in pre-training of the reinforcement learning model;

[0013] Step S7: Based on the digital twin model The state space S of reinforcement learning, action space A, reward function R, reinforcement learning model M, digital twin virtual-reality interaction channel DT-InC, digital twin deduction and simulation platform DT-Sim, experience data B, and digital twin decision model DT-Dec perform reinforcement learning training.

[0014] Furthermore, in step S1, the task area is a square space with a length, width and height of L×W×H respectively, and the ground node model Including ground node locations and ground node communication requirements, the UAV model Including drone location, the building model Including building distribution information, the drone sensor perception results Includes the drone's sensor perception of the surrounding environment.

[0015] Furthermore, in step S2, the actual mission environment and the physical entity environment composed of drones establish a communication link with the central server through satellite or 5G communication to realize the digital twin model. Two-way interaction with the real world.

[0016] Furthermore, in step S4, the specific tasks performed by the drone include drone safe flight tasks and drone communication coverage tasks.

[0017] Furthermore, in step S4, when the drone performs a drone safety flight mission:

[0018] The state of the drone at time step t is expressed as Among them, p t represents the position of the drone at time step t, p tar represents the target position of the defined UAV, d t =||p t -p tar ||2 is the distance between the drone and the target position at time step t, d0=||p o -p tar ||2 is the initial distance between the drone and the target position, It is the perception information, which is represented as a 27-dimensional vector. If we move forward in a specific direction in the next time step, If there is an obstacle in the memory, the value of the position element is 1, otherwise it is 0. T0 is the time step length, v = [v x ,v y ,v z ] is the flight speed of the drone in the x, y, and z directions, a t =[a x ,a y ,a z ] is the moving direction of the drone in the x, y, and z axes, a x ,a y ,a z ={0,±1};

[0019] The drone’s action a at time step t t Then there are 27 forward directions of the current position, expressed as a t ∈{1,2,L,27}, where the drone performs different actions corresponding to different moving directions in x, y, and z, specifically [x, y, z] = {1, 0, -1; 1, 0, -1; 1, 0, -1}, and the action a at time step t t The value of is a random permutation of the elements in [x,y,z] = {1,0,-1; 1,0,-1; 1,0,-1}, there are 3 3 =27 directions;

[0020] The revenue function of the drone at time step t is r t =r 1,t +r 2,t +r 3,t +r 4,t +r 5,t ,in, represents the distance benefit, ω1 represents the distance benefit weight, Δd represents the distance change between the UAV and the target position, represents the stationary penalty, ω2 represents the stationary penalty weight, represents the crash penalty, ω3 represents the crash penalty weight, represents the out-of-bounds penalty, ω4 represents the out-of-bounds penalty weight, represents the reward for reaching the end point, and ω5 represents the weight of the end point reward.

[0021] Furthermore, in step S4, when the drone performs the drone communication coverage task:

[0022] The state of the drone at time step t is in, represents the ground node coverage status at time step t, represents the ground node service status at time step t, represents the information age of adjacent time steps, κ c K represents the information age accumulated in a single time step during the process of ground nodes being served by UAVs. c represents the number of users served at the current time step t, χ loss represents the information age offset of the served ground nodes, K G Indicates the number of ground nodes;

[0023] The drone’s action a at time step t t Then there are 27 forward directions of the current position, expressed as a t ∈{1,2,L,27}, where the drone performs different actions corresponding to different moving directions in x, y, and z, specifically [x, y, z] = {1, 0, -1; 1, 0, -1; 1, 0, -1}, and the action a at time step t t The value of is a random permutation of the elements in [x,y,z] = {1,0,-1; 1,0,-1; 1,0,-1}, there are 3 3 =27 directions;

[0024] The revenue function of the drone at time step t is Among them, Tt represents the remaining time after completing the network coverage conditions for all users as the reward for task completion, p beyond Indicates the penalty a drone receives for crossing the boundary.

[0025] Furthermore, the step S5 specifically includes: in the digital twin model The sampled state information s∈S is based on the output of the reinforcement learning model in the action space a t ∈A, select an action and get the state s of the next time step t+1 and evaluate The action execution reward r in t,DT =R(st ,a t ), get the empirical data s t ,a t ,s t+1 ,r t,DT >∈B.

[0026] Furthermore, the step S6 is specifically as follows: the digital twin model Based on the empirical data obtained in step S5, the reinforcement learning model M is iteratively updated.

[0027] Furthermore, the step S7 specifically includes:

[0028] Step S71: In the digital twin model and physical entity drones, denoted as M DT and M Re , the number of training rounds N ep =0;

[0029] Step S72: Digital twin model Based on the reinforcement learning model M DT Select the action to perform a t ∈A, observe the state s of the next time step t+1 , evaluate the action execution reward r t,DT =R(s t ,a t ); if in Execute action a t If the UAV is not threatened after ∈A, then execute step S74, otherwise execute step S73;

[0030] Step S73: Execute action a t After the drone faces a threat, the reward is r DT =r DT -1000, will s t ,a t ,s t+1 ,r t,DT >Save in B and go to step S72;

[0031] Step S74: Use DT-InC to convert M DT The parameters are copied to M Re , and based on M Re Select the physical entity drone to perform action a t , evaluate the action execution reward r t,Re =R(s t ,a t ), observe the next time step state s t+1 , and the empirical data <st ,a t ,s t+1 ,r t,Re >Sent to via DT-InC Save in

[0032] Step S75: Update model M according to updated experience data B DT , the number of training rounds N ep +1, jump to step S72 until the maximum number of training rounds is reached

[0033] The present invention has the following beneficial effects:

[0034] The present invention combines the reinforcement learning method with the digital twin technology. Through the digital twin technology, a digital model of the real communication environment can be constructed, including geographic information, signal propagation characteristics, obstacle distribution, etc., to achieve a high degree of restoration and accurate simulation of the real environment, introduce the dynamic changes of the real environment into the simulated environment, effectively deepen the convergence speed of the deep reinforcement learning model, and improve the accuracy and efficiency of the trajectory planning results, thereby reducing the security threat of drones and improving the mission safety and strategy effectiveness of drones in communication missions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention is further described in detail below with reference to the accompanying drawings.

[0036] Figure 1 It is a flow chart of the present invention.

[0037] Figure 2 This is a schematic diagram of the range of perception results of the drone sensor of the present invention.

[0038] Figure 3 This is a schematic diagram of the benefit convergence of the present invention in the UAV safe flight mission.

[0039] Figure 4 This is a schematic diagram of the convergence of the number of crashes in the safe flight mission of the UAV of the present invention.

[0040] Figure 5 This is a schematic diagram of the mission time convergence of the present invention in the UAV safe flight mission.

[0041] Figure 6 This is a schematic diagram of the benefit convergence of the present invention in the UAV communication coverage mission.

[0042] Figure 7 This is a schematic diagram of the convergence of the number of coverage nodes in the UAV communication coverage task of the present invention.

[0043] Figure 8This is a schematic diagram of the task time convergence of the present invention in the UAV communication coverage task. DETAILED DESCRIPTION

[0044] like Figure 1 As shown in FIG, the digital twin training method for reinforcement learning driven UAV trajectory planning includes the following steps:

[0045] Step S1: Based on GPS information and drone sensor perception information, a digital twin model of the mission environment is constructed on the central server. Digital Twin Model Including mission area, ground node model Drone Model Building Model And the drone sensor perception results

[0046] Specifically, the mission area is a square space with a length, width, and height of L×W×H. Including ground node location and ground node communication requirements, drone model Including drone location, the building model Including building distribution information, drone sensor perception results Includes information about the drone's surrounding environment from its sensors;

[0047] In this embodiment, L = 500m, W = 500m, H = 200m, K G = 200 ground nodes are evenly and randomly distributed on the ground of the mission area. The position of the ground node is expressed as [x g ,y g ,z g ],n U =5 The UAV model is represented in the same way as the ground node model, including the UAV position, expressed as [x u ,y u ,z u ], the building model is saved as K B Building information vector, represented as UAV sensor perception results Indicators indicating whether there are obstacles in the different directions of the drone's x, y, and z axes, such as Figure 2 shown.

[0048] Step S2: construct a digital twin virtual-real interaction channel DT-InC to realize two-way interaction between the digital twin model and the real world;

[0049] Specifically, the actual mission environment and the physical environment composed of drones establish a communication link with the central server through satellite or 5G communication to realize the digital twin model. Two-way interaction with the real world, i.e. realization of digital twin model Real-time collaboration and lossless interaction with the mission environment.

[0050] Step S3: Based on the digital twin model Ground node model and drone models According to the K-nearest neighbor algorithm clustering results, the set of ground nodes that each drone is responsible for is determined. Ground node collection The center position of is taken as the initial position of the UAV to perform the task, where n U The process of determining the set of ground nodes that each drone is responsible for based on the clustering result of the K nearest neighbor algorithm is an existing technology.

[0051] Step S4: Design the state space S, action space A and benefit function R of reinforcement learning for the specific task to be performed by the drone, and determine the reinforcement learning model M;

[0052] Specifically, the specific tasks performed by drones include drone safe flight tasks and drone communication coverage tasks. During the training process, the state space, action space and benefit function are necessary inputs for the training of the reinforcement learning model. The reinforcement learning model needs to identify the state in the state space, and then select the action in the action space according to its own output, and then calculate the benefit based on the benefit function and the state just identified and the selected action, and update the model parameters based on this benefit. Through continuous training iterations, the reinforcement learning model will converge.

[0053] When the UAV is performing a UAV safe flight mission:

[0054] Define the starting position p of the drone o 、Target position p tar , UAV sensor perception results The state space of the Markov process of the drone safe flight mission is composed of the drone position information, target position distance information and perception information, specifically:

[0055] At time step t, the distance d between the drone and the target position t and the initial distance d0 between them according to p o and p tar The calculation result is d t =||p t -p tar ||2,d0=||p o -ptar ||2, where p t represents the position of the UAV at time step t;

[0056] Assume v = [v x ,v y ,v z ] represents the flight speed of the drone in the x, y, and z axes, and the length is x, y, and z. Define a t =[a x ,a y ,a z ] is the moving direction of the drone in the x, y, and z axes, a x ,a y ,a z ={0,±1}, perception information Represented as a 27-dimensional vector, if the range is moved in a specific direction at the next time step If there is an obstacle, the value of the position element is 1, otherwise it is 0;

[0057] Therefore, the state at time step t is expressed as

[0058] The action a of the drone at time step t t Then there are 27 forward directions of the current position, expressed as a t ∈{1,2,L,27}, where the drone performs different actions corresponding to different moving directions in x, y, and z, specifically [x, y, z] = {1, 0, -1; 1, 0, -1; 1, 0, -1}, and the action a at time step t t The value of is a random permutation of the elements in [x,y,z] = {1,0,-1; 1,0,-1; 1,0,-1}, there are 3 3 =27 directions;

[0059] The benefits after executing the action at time step t include distance benefit, stationary penalty, crash penalty, out-of-bounds penalty and arrival reward, which are as follows:

[0060] Distance gain: represents the change in distance between the drone and the target position at each time step after the maneuver, expressed as Among them, ω1 represents the distance benefit weight, Δd represents the distance change between the UAV and the target position;

[0061] Static penalty: indicates that the action output by the drone before reaching the target position is the benefit of keeping still. The expression is Among them, ω2 represents the static penalty weight;

[0062] Crash penalty: indicates that the drone collides with an obstacle, and the gain under the condition that there is an obstacle within the range of the drone's motion in the time slot T_0 is expressed as Among them, ω3 represents the crash penalty weight;

[0063] Out-of-bounds penalty: indicates that the drone exceeds the scope of the mission area S during movement, and the expression is Among them, ω4 represents the out-of-bounds penalty weight;

[0064] Reward for reaching the destination: represents the reward for the drone when it reaches the destination. The expression is Among them, ω5 represents the weight of reaching the destination;

[0065] In summary, the revenue function of a drone performing a safe flight mission is defined as r t =r 1,t +r 2,t +r 3,t +r 4,t +r 5,t .

[0066] When the UAV performs the UAV communication coverage mission:

[0067] The air-to-ground link channel is defined as Line-of-Sight (LoS) link and Non-LoS (NloS) link. The system will automatically detect the communication link between the drone and the ground node in the urban environment model. u,g Is there a building blocking the link? u,g The channel gain is h u,g , ignoring interference, the signal-to-noise ratio (SNR) of the link is modeled as γ u,g = p·h u,g / σ 2 , where p is the signal transmission power, σ 2 is the noise power. Let c LoS is the channel gain under the condition of unit distance d0=1, d u,g is the distance between the UAV and the corresponding ground node, η LoS is the path loss index, Y LoS is the channel random reflection fading distribution parameter under LoS channel. LoS The calculation expression is in, is the free space loss, so under the LoS channel, the channel gain can be further expressed as

[0068] The signal-to-noise ratio is expressed as Among them, G LoSis a constant, which includes the communication channel characteristics. The NLoS channel modeling method is similar to LoS, and it is also necessary to introduce the channel parameter η under the NLoS channel. NLoS and Y NLoS In addition, the ground node needs to satisfy the signal-to-noise ratio of the network γ ≥ γ th Only when the condition is met can the network be accessed. Specifically in this embodiment, specifically in this example, p = 0.1dBW, σ 2 =-75dBm, η LoS =0.1dB,η NLoS =21dB, f c =2.4GHz,λ th =0dB.

[0069] The state space of the Markov process of the UAV communication coverage task includes location information, network coverage information and information age information. At different time steps t, the state vector s t Representation, specifically including: the drone position p at time step t t UAV network coverage information at time step t, including ground node coverage status Ground node service status If the ground node is covered by the network or served by the drone, the value is 1, otherwise it is 0.

[0070] Define the information age of time step t. Assuming that the initial value of the information age of all ground nodes is the same, the information age of adjacent time slots is defined as κ c K represents the information age accumulated in a single time step during the process of ground nodes being served by UAVs. c represents the number of users served at the current time step t, χ loss represents the information age offset of the served ground nodes, K G represents the number of ground users;

[0071] In summary, the state space of the UAV communication coverage task at time step t is

[0072] The action space of the drone at time step t is the same as the action space when performing the drone safe flight mission, that is, action a t Then there are 27 forward directions of the current position, expressed as a t ∈{1,2,L,27}, where the drone performs different actions corresponding to different moving directions in x, y, and z, specifically [x, y, z] = {1, 0, -1; 1, 0, -1; 1, 0, -1}, and the action a at time step t t The value of is a random permutation of the elements in [x,y,z] = {1,0,-1; 1,0,-1; 1,0,-1}, there are 33 =27 directions;

[0073] The profit function of the drone at time step t is based on the information age, constructs the profit gain of each action, and fits it with the tanh function, which is expressed as Among them, Tt represents the remaining time after completing the network coverage conditions for all users as the reward for task completion, p beyond Indicates the penalty a drone receives for crossing the boundary.

[0074] In this embodiment, κ c =10, χ loss =1.

[0075] The goal of both the drone safe flight mission and the communication coverage mission is to minimize the mission time.

[0076] Step S5: construct a digital twin simulation platform DT-Sim to obtain empirical data B;

[0077] Specifically, in the digital twin model Sampling status information t ∈S, select action a in the action space A based on the output of the reinforcement learning model t ∈A, get the state s of the next time step t+1 and evaluate The action execution reward r in t,DT =R(s t ,a t ), get empirical data t ,a t ,s t+1 ,r t,DT >∈B. The action execution reward is the profit function r of the drone safe flight mission and drone communication coverage mission in step S4 t Specifically in this embodiment, step S5 is only executed for a specific task, namely, a drone safe flight task or a drone communication coverage task.

[0078] Step S6: construct a digital twin decision model DT-Dec to assist in pre-training of the reinforcement learning model;

[0079] Specifically, the digital twin model Based on the empirical data obtained in step S5 t ,a t ,s t+1 ,r t,DT >∈B, iteratively update the reinforcement learning model M.

[0080] ​​In this embodiment, step S6 is only executed for a specific task, namely, the drone safe flight task or the drone communication coverage task. The Double Deep Q-Learning (DDQN) algorithm is used as the deep reinforcement learning model, D = 500000, δ = 0.99, D s =512,α=0.0004.

[0081] Step S7: Based on the digital twin model The state space S of reinforcement learning, the action space A, the benefit function R, the reinforcement learning model M, the digital twin virtual-real interaction channel DT-InC, the digital twin deduction and simulation platform DT-Sim, the experience data B, and the digital twin decision model DT-Dec perform reinforcement learning training;

[0082] The specific steps include:

[0083] Step S71: In the digital twin model and physical entity drones, denoted as M DT and M Re , the number of training rounds N ep =0;

[0084] Step S72: Digital twin model Based on the reinforcement learning model M DT Select the action to perform a t , observe the state s of the next time step t+1 , evaluate the action execution reward r t,DT , where the action reward is the revenue function r of the drone safe flight mission and drone communication coverage mission in step S4 t . ; If in If the drone is not threatened after executing action a*, then execute step S74, otherwise execute step S73;

[0085] Step S73: Execute action a t After the drone faces a threat, the reward is r t,DT =r t,DT -1000, will t ,a t,t+1 ,r t,DT >Save in B and go to step S72;

[0086] Step S74: Use DT-InC to convert M DT The parameters are copied to M Re , and based on M Re Select the physical entity drone to perform action a t , evaluate the action execution reward r t,Re ​=R(s t ,a t ), where the action reward is the revenue function r of the drone safe flight mission and drone communication coverage mission in step S4 t . Observe the next time step state s t+1 , and the empirical data t ,a t,t+1 ,r t,Re >Sent to via DT-InC Save in

[0087] Since the threat to the drone is eliminated in step S72, the drone can continue to work in the subsequent steps;

[0088] Step S75: Update model M according to updated experience data B DT , the number of training rounds N ep +1, jump to step S72 until the number of training rounds reaches the maximum limit

[0089] In this embodiment, step S7 is only executed for a specific task, namely, a drone safe flight task or a drone communication coverage task. UAV communication coverage mission

[0090] Figure 3 , Figure 4 , Figure 5 They are respectively schematic diagrams of the convergence of the round reward (income), the number of crashes and the mission time in the application of the present invention in the safe flight mission of the unmanned aerial vehicle; Figure 3 , Figure 4 , Figure 5 The horizontal axis in the middle is the number of training rounds, with a value of 1-3500 at a time, and the vertical axis is the training results of revenue, number of crashes and mission time in different rounds. Figure 3 , Figure 4 , Figure 5 It can be seen from the digital twin framework of the present invention and the traditional framework without the present invention that the present invention can effectively accelerate the model convergence progress;

[0091] Figure 6 , Figure 7 , Figure 8 They are respectively schematic diagrams of the convergence of the round reward (income), the number of covered nodes and the task time in the UAV communication coverage task applied by the present invention; Figure 6 , Figure 7 , Figure 8 It can be seen from the digital twin framework that the present invention is applied to and the traditional framework that the present invention is not applied to that the present invention can effectively accelerate the model convergence progress.​

[0092] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A digital twin training method for reinforcement learning driven UAV trajectory planning, characterized by: The steps include: Step S1: Based on GPS information and drone sensor perception information, a digital twin model of the mission environment is constructed on the central server. Digital Twin Model Including mission area, ground node model Drone Model Building Model and the UAV sensor perception result S; Step S2: construct a digital twin virtual-real interaction channel DT-InC to realize two-way interaction between the digital twin model and the real world; Step S3: Based on the digital twin model Ground node model and drone models According to the K-nearest neighbor algorithm clustering results, the set of ground nodes that each drone is responsible for is determined. Ground node collection The center position of is taken as the initial position of the UAV to perform the task, where n U is the number of drones; Step S4: Design the state space S, action space A and benefit function R of reinforcement learning for the specific task to be performed by the drone, and determine the reinforcement learning model M; Step S5: construct a digital twin simulation platform DT-Sim to obtain empirical data B; Step S6: construct a digital twin decision model DT-Dec to assist in pre-training of the reinforcement learning model M; Step S7: Based on the digital twin model The state space S of reinforcement learning, action space A, reward function R, reinforcement learning model M, digital twin virtual-reality interaction channel DT-InC, digital twin deduction and simulation platform DT-Sim, experience data B, and digital twin decision model DT-Dec perform reinforcement learning training.

2. A digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 1, characterized in that: In step S1, the task area is a square space with a length, width and height of L×W×H respectively, and the ground node model Including ground node locations and ground node communication requirements, the UAV model Including drone location, the building model Including building distribution information, the drone sensor perception results Includes the drone's sensor perception of the surrounding environment.

3. A digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 2, characterized in that: In step S2, the actual mission environment and the physical entity environment composed of drones establish a communication link with the central server through satellite or 5G communication to realize the digital twin model. Two-way interaction with the real world.

4. The digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 3 is characterized by: In step S4, the specific tasks performed by the drone include a drone safe flight task and a drone communication coverage task.

5. A digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 4, characterized in that: In step S4, when the drone performs a drone safe flight mission: The state of the drone at time step t is expressed as Among them, p t represents the position of the drone at time step t, p tar represents the target position of the defined UAV, d t =||p t -p tar ||2 is the distance between the UAV and the target position at time step t, d0=||p o -p tar ||2 is the initial distance between the drone and the target position, It is the perception information, which is represented as a 27-dimensional vector. If we move forward in a specific direction in the next time step, If there is an obstacle in the memory, the value of the position element is 1, otherwise it is 0. T0 is the time step length, v = [v x ,v y ,v z ] is the flight speed of the drone in the x, y, and z axes, a t =[a x ,a y ,a z ] is the moving direction of the drone in the x, y, and z axes, a x ,a y ,a z ={0,±1}; The action a of the drone at time step t t Then there are 27 forward directions of the current position, expressed as a t ∈{1,2,L,27}, where the drone performs different actions corresponding to different moving directions in x, y, and z, specifically [x, y, z] = {1, 0, -1; 1, 0, -1; 1, 0, -1}, and the action a at time step t t The value of is a random permutation of the elements in [x,y,z] = {1,0,-1; 1,0,-1; 1,0,-1}, there are 3 3 =27 directions; The revenue function of the drone at time step t is r t =r 1,t +r 2,t +r 3,t +r 4,t +r 5,t ,in, represents the distance benefit, ω1 represents the distance benefit weight, Δd represents the distance change between the UAV and the target position, represents the stationary penalty, ω2 represents the stationary penalty weight, represents the crash penalty, ω3 represents the crash penalty weight, represents the out-of-bounds penalty, ω4 represents the out-of-bounds penalty weight, Represents the reward for reaching the end point, and ω5 represents the reward weight for reaching the end point.

6. A digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 4, characterized in that: In step S4, when the drone performs the drone communication coverage task: The state of the drone at time step t is in, represents the ground node coverage status at time step t, represents the ground node service status at time step t, represents the information age of adjacent time steps, κ c K represents the information age accumulated in a single time step during the process of ground nodes being served by UAVs. c represents the number of users served at the current time step t, χ loss represents the information age offset of the served ground nodes, K G Indicates the number of ground nodes; The action a of the drone at time step t t Then there are 27 forward directions of the current position, expressed as a t ∈{1,2,L,27}, where the drone performs different actions corresponding to different moving directions in x, y, and z, specifically [x, y, z] = {1, 0, -1; 1, 0, -1; 1, 0, -1}, and the action a at time step t t The value of is a random permutation of the elements in [x,y,z] = {1,0,-1; 1,0,-1; 1,0,-1}, there are 3 3 =27 directions; The revenue function of the drone at time step t is Among them, Tt represents the remaining time after completing the network coverage conditions for all users as the reward for task completion, p beyond Indicates the penalty a drone receives for crossing the boundary.

7. A digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 5 or 6, characterized in that: The step S5 specifically includes: in the digital twin model Sampling status information t ∈S, select action a in the action space A based on the output of the reinforcement learning model t ∈A, get the state s of the next time step t+1 and evaluate The action execution reward r in t,DT =R(s t ,a t ), get empirical data t ,a t ,s t+1 ,r t,DT >∈B.​ 8. The digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 7, characterized in that: The step S6 specifically includes: the digital twin model Based on the empirical data obtained in step S5, the reinforcement learning model M is iteratively updated.

9. A digital twin training method for reinforcement learning driven UAV trajectory planning according to claim 8, characterized in that: The step S7 specifically includes: Step S71: In the digital twin model and physical entity drones, denoted as M DT and M Re , the number of training rounds N ep =0; Step S72: Digital twin model Based on the reinforcement learning model M DT Select the action to perform a t , observe the state s of the next time step t+1 , evaluate the action execution reward r t,DT ; If in Execute action a t If the drone is not facing a threat, then execute step S74, otherwise execute step S73; Step S73: Execute action a t After the drone faces a threat, the reward is r t,DT =r t,DT -1000, will t ,a t ,s t+1 ,r t,DT >Save in B and go to step S72;​ Step S74: Use DT-InC to convert M DT The parameters are copied to M Re , and based on M Re Select the physical entity drone to perform action a t , evaluate the action execution reward r t,Re =R(s t ,a t ), observe the next time step state s t+1 , and the empirical data t ,a t ,s t+1 ,r t,RE >Sent to via DT-InC Save in​ Step S75: Update the model M according to the updated experience data B DT , the number of training rounds N ep +1, jump to step S72 until the maximum number of training rounds is reached

Citation Information

Cited By

  • Variable-pitch unmanned aerial vehicle simulation control method based on digital twinning and unmanned aerial vehicle

    CN119148554A

  • Motion mechanism time statistical analysis method and device based on unidirectional digital twinning

    CN119927896A

  • Evaluation method and device of trajectory planning strategy, electronic equipment and storage medium

    CN122133502A