Adaptive unmanned aerial vehicle trajectory optimization and resource allocation method based on improved DT
Through the improved DecisionTransformer model and constraint reinforcement learning algorithm, combined with cost variables and pre-training fine-tuning technology, the adaptability problem of drone edge computing in long-term constraint changes is solved, and efficient and secure drone trajectory optimization and resource allocation are achieved.
Patent Information
- Application Number
- CN202411739041.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing drone edge computing technology is difficult to effectively solve the random optimization problem in the time-varying environment of mobile terminal demand, especially when long-term constraint changes, and traditional RL algorithms are difficult to quickly adapt to and meet constraints.
Using the improved DecisionTransformer (DT) model, by introducing cost variables into the DT input module, combining the constraint reinforcement learning algorithm, decision actions are generated under the constraints of different UAV battery capacity thresholds, and rapid migration and generalization in different scenarios are achieved through pre-training and fine-tuning of the model.
It realizes good generalization adaptability when the constraints of wireless environments change, improves the efficiency and security of the edge computing model of the drone, can effectively meet long-term constraints, and at the same time, it is better than traditional methods in task performance.
Smart Images
Figure CN119729628A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicle edge computing technology, and in particular to an adaptive unmanned aerial vehicle trajectory optimization and resource allocation method based on improved DT. Background Art
[0002] 6G edge computing technology is pursuing the goal of providing low-latency, efficient computing services for mobile terminals. Drones are widely used to provide flexible computing services due to their high connectivity and flexibility. In an environment where mobile terminal demands vary, traditional algorithms such as convex optimization cannot solve random optimization problems. However, the Reinforcement Learning (RL) algorithm has been widely used in edge computing scenarios due to its ability to interact with the environment.
[0003] RL algorithms can be divided into traditional RL algorithms represented by value function learning and policy function learning. However, there are a series of problems in deployment. First, the interaction between the RL algorithm and the environment will cause a lot of delays, which is difficult to deploy in high-real-time systems. Secondly, because the RL algorithm in the wireless environment needs to meet some long-term constraints, such as the battery consumption of the drone does not exceed the maximum battery capacity. The RL algorithm takes the cumulative expectation maximization of short-term rewards as the learning goal. When dealing with long-term constraints, it can only convert the long-term cost caused by battery consumption into a short-term penalty term of the reward function, which is difficult to effectively ensure the satisfaction of long-term constraints.
[0004] Constrained RL algorithms provide a new approach to maximize system benefits and effectively meet long-term constraints. However, existing constrained RL algorithms such as Constrained Policy Optimization (CPO) usually require online interaction with the environment to learn and are only applicable to fixed constraints. When faced with changes in constraints, it is difficult to adapt quickly.
[0005] In solving the above challenges, Decision Transformer (DT), as an emerging deep learning method, provides a new idea. DT is derived from the core model Transformer of the large language model, and has shown great potential in offline RL tasks. After large-scale offline pre-training, DT has great advantages in the ability of migration and generalization in new scenarios. It adopts the sequential decision-making method of generative artificial intelligence, and can quickly migrate and generalize with only a small number of samples for fine-tuning when environmental parameters change. However, how to apply DT in edge computing environments to adapt to the complex constraints of wireless environments and achieve rapid migration and generalization in different scenarios is an urgent problem to be solved. Summary of the invention
[0006] The purpose of the present invention is to provide an adaptive UAV trajectory optimization and resource allocation method based on improved DT, which improves the efficiency and security of the edge computing model and achieves good generalization adaptability when the constraints of the wireless environment change.
[0007] To achieve the above objectives, the present invention provides an adaptive UAV trajectory optimization and resource allocation method based on improved DT, the steps comprising:
[0008] S1. Improve the DT model and introduce cost variables into the DT input module;
[0009] S2. Use the constrained reinforcement learning algorithm to generate decision actions under the constraints of different UAV battery power thresholds and combine the current state information to calculate the system throughput reward and UAV energy consumption cost corresponding to the current decision action;
[0010] S3. Use the constrained reinforcement learning algorithm to collect data to pre-train the improved model. After the training, under the new battery power threshold, collect new state information data, use the pre-trained improved model to generate preliminary exploration of drone decision actions, calculate the system throughput reward and drone energy consumption cost corresponding to the current action, and combine the state, action, reward, and cost variables into a complete trajectory segment;
[0011] S4. Using the trajectory segments obtained in step S3, fine-tune the improved model to optimize the improved model's drone trajectory selection, user association and band allocation decisions under the current environmental conditions, while maximizing the reward and minimizing the cost.
[0012] Preferably, the status information in step S2 includes user task queue, user processing delay and task processing time limit.
[0013] Preferably, in step S2, a constrained reinforcement learning algorithm is used to generate a decision action based on the formula:
[0014] a(t)={a n (t),n∈N};
[0015] Where N is the number of ground users and a(t) is the action at different times t.
[0016] Preferably, calculating the reward corresponding to the current decision action in step S2 includes:
[0017] Calculate all end-user-based fairness indices using the following formula:
[0018]
[0019] In the formula, i(t) represents the fairness index; K(·) represents the exponential function, which returns 1 when the condition is met, otherwise it returns 0; D n (t) represents the user task queue, δ-δ1(t) represents the task processing time limit, Indicates user processing delay;
[0020] Based on I(t), the system throughput at the current moment is calculated using the formula:
[0021]
[0022] Preferably, the energy consumption cost of the drone is calculated in step S2 based on the formula:
[0023]
[0024] In the formula, E M (t) represents the hovering energy consumption, E T (t) represents the transmission energy consumption, E C (t) represents the computational energy consumption.
[0025] Preferably, the improved model pre-training in step S3 updates the parameter θ through the mean square error, and the formula is:
[0026] L(θ)=[α * (t)-a(t)] 2 ;
[0027] In the formula, a * (t) represents the expert action that satisfies the constraints generated by the constrained reinforcement learning algorithm.
[0028] Preferably, in step S4, the LoRa lightweight method is used to fine-tune the improved model.
[0029] Therefore, the present invention adopts the above-mentioned adaptive UAV trajectory optimization and resource allocation method based on improved DT, which has the following beneficial effects:
[0030] (1) The DT model is improved by adding cost variable input on the basis of DT, so that more abundant information can be learned from historical data to more effectively meet the constraints of the environment and achieve rapid generalization in different scenarios;
[0031] (2) Adopt LoRa lightweight fine-tuning technology to adaptively fine-tune and improve DT model parameters to achieve rapid migration and generalization in different scenarios;
[0032] (3) Improved the efficiency and security of the drone edge computing model.
[0033] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0035] Figure 2 A method architecture diagram of an embodiment of the present invention;
[0036] Figure 3 A comparison chart of experimental results of cumulative energy consumption of rounds under energy consumption constraint thresholds of the present invention and other algorithms according to an embodiment of the present invention;
[0037] Figure 4 This is a comparison chart of system throughputs of the present invention and other algorithms in different scenarios according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0039] Example
[0040] Reference Figure 1-2 The present invention provides an adaptive UAV trajectory optimization and resource allocation method based on improved DT, the steps comprising:
[0041] S1. Improve the DT model and introduce cost variables in the DT input module. By adding cost input to describe the cost brought by the battery power constraint of the drone in the wireless environment, the improved model can learn how to meet the constraints of the wireless optimization environment.
[0042] S2. Use the constrained reinforcement learning algorithm Constrained Policy Optimization (CPO) to generate the trajectory selection, user association and band allocation decisions of the drone under the constraints of different drone battery power thresholds (such as battery power of 100kJ or 200kJ), combined with the current state information, that is, generate decision actions, and calculate the system throughput reward and drone energy consumption cost corresponding to the current decision action.
[0043] In this embodiment, the status information includes user task queue, user processing delay, and task processing time limit, which can be collected through drone sensors.
[0044] In this embodiment, the decision action is generated based on the formula:
[0045] a(t)={a n (t),n∈N};
[0046] Where N is the number of ground users and a(t) is the action at different times t.
[0047] In this embodiment, calculating the reward corresponding to the current decision action includes:
[0048] Calculate all end-user-based fairness indices using the following formula:
[0049]
[0050] Where I(t) represents the fairness index; K(·) represents the exponential function, which returns 1 when the condition is met and 0 otherwise; D n (t) represents the user task queue, δ-δ1(t) represents the task processing time limit, Indicates user processing delay.
[0051] Based on I(t), the system throughput at the current moment is calculated using the formula:
[0052]
[0053] In this embodiment, the energy consumption cost of the drone is calculated based on the formula:
[0054]
[0055] In the formula, E M (t) represents the hovering energy consumption, E T (t) represents the transmission energy consumption, E C (t) represents the computational energy consumption.
[0056] S3. Use the CPO algorithm to collect data to pre-train the improved model to build a CADT model. After the training, under the new battery power threshold condition (such as the battery power is 300kJ), collect new state information data, use the pre-trained improved model to generate preliminary exploration of drone decision actions, calculate the system throughput reward and drone energy consumption cost corresponding to the current action, and combine the state, action, reward, and cost variables into a complete trajectory segment.
[0057] The improved model pre-training in this embodiment updates the parameter θ through the mean square error, and its formula is:
[0058] L(θ)=[a * (t)-a(t)] 2 ;
[0059] In the formula, a * (t) represents the expert action generated by the CPO algorithm that satisfies the constraints.
[0060] S4. Using the trajectory segments obtained in step S3, fine-tune the improved model to optimize the improved model's drone trajectory selection, user association and band allocation decisions under the current environmental conditions, while maximizing the reward and minimizing the cost.
[0061] In this embodiment, the LoRa lightweight method is used to fine-tune the improved model, and the DT model parameters are adaptively fine-tuned to achieve generalization under the constraints of the current wireless environment.
[0062] In order to verify the effectiveness of the method of the present invention, the following experiment is set up for verification.
[0063] Model setup
[0064] Proximal Policy Optimization (PPO) algorithm and CPO algorithm are selected as comparison models. The improved model used in the method of the present invention models the offline reinforcement learning problem as a sequence modeling problem, and uses the Transformer network to directly generate the optimal action trajectory. As a commonly used traditional RL algorithm, the PPO algorithm performs well in system benefits, but its disadvantage is that its long-term constraint satisfaction ability is poor, and it often needs to be retrained in new scenarios. As a constrained RL algorithm, the CPO algorithm has the ability to meet long-term constraints, but its generalization ability between different scenarios is poor.
[0065] Parameter settings
[0066] Multiple typical scenario data sets of simulated drones and users are used. These data sets include communication records between different users and drones in different time periods, as well as information such as user location and channel status. The specific parameters in the scenario of simulated drones and users are as follows: the network consists of a drone and 25 information receiving points, and 15 ground users. The drone uses a frequency division multiple access model in each time slot and is associated with multiple ground users. The channel model uses the Nakagami-m channel, which flexibly describes the channel distribution from Rayleigh fading to Rice fading through the shape parameter m. The drone transmission power is set to 0.1W, the hovering power is 0.08W, and the flight power is 0.11W. The optimizer uses the Adam optimizer, the batch size is set to 5, the learning rate is 0.0001, and the discount factor is set to 0.9. The total number of experimental rounds is 3000, and 25 steps are completed in each round of the experiment.
[0067] Evaluation Metrics
[0068] Energy consumption and fairness-based system throughput are selected as evaluation indicators to measure the network performance after optimizing UAV trajectories and resource allocation.
[0069] Results Analysis
[0070] Figure 3 The cumulative energy consumption under different energy constraint threshold conditions is shown. In the UAV-assisted edge computing scenario, effectively satisfying long-term constraints is of great significance for practical deployment, such as ensuring the safe return of the UAV under limited power. The experimental results show that the energy consumption of the CADT algorithm improved by the method of the present invention is lower than the constraint threshold, and the constraint satisfaction is improved compared with the baseline model.
[0071] Figure 4 The task performance of the three algorithms in different scenarios is demonstrated. The experimental results show that although PPO has better task performance than CPO, its constraint satisfaction ability is poor; CPO has strong constraint satisfaction ability, but poor task performance. The CADT algorithm improved by the method of the present invention not only effectively meets the energy consumption constraint, but also outperforms the two algorithms in terms of performance. This shows that the method proposed in the present invention has strong adaptability and generalization in UAV-assisted edge computing, can effectively meet constraints under different constraint thresholds, and is superior to traditional methods in task performance.
[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. An adaptive UAV trajectory optimization and resource allocation method based on improved DT, characterized in that the steps include: S1. Improve the DT model and introduce cost variables into the DT input module; S2. Use the constrained reinforcement learning algorithm to generate decision actions under the constraints of different UAV battery power thresholds and combine the current state information to calculate the system throughput reward and UAV energy consumption cost corresponding to the current decision action; S3. Use the constrained reinforcement learning algorithm to collect data to pre-train the improved model. After the training, under the new battery power threshold, collect new state information data, use the pre-trained improved model to generate preliminary exploration of drone decision actions, calculate the system throughput reward and drone energy consumption cost corresponding to the current action, and combine the state, action, reward, and cost variables into a complete trajectory segment; S4. Using the trajectory segments obtained in step S3, fine-tune the improved model to optimize the improved model's drone trajectory selection, user association and band allocation decisions under the current environmental conditions, while maximizing the reward and minimizing the cost.
2. The method for adaptive UAV trajectory optimization and resource allocation based on improved DT according to claim 1, characterized in that: The status information in step S2 includes the user task queue, user processing delay and task processing time limit.
3. The method for adaptive UAV trajectory optimization and resource allocation based on improved DT according to claim 2, characterized in that: In step S2, the constraint reinforcement learning algorithm is used to generate decision actions based on the formula based on the current state information under the constraints of different drone battery power thresholds: a(t)={a n (t),n∈N}; Where N is the number of ground users and a(t) is the action at different times t.
4. The method for adaptive UAV trajectory optimization and resource allocation based on improved DT according to claim 3 is characterized in that: The reward corresponding to the current decision action is calculated in step S2, including: Calculate all end-user-based fairness indices using the following formula: Where I(t) represents the fairness index; K(·) represents the exponential function, which returns 1 when the condition is met and 0 otherwise; D n (t) represents the user task queue, δ=δ1(t) represents the task processing time limit, Indicates user processing delay; Based on I(t), the system throughput at the current moment is calculated using the formula:
5. The method for adaptive UAV trajectory optimization and resource allocation based on improved DT according to claim 4 is characterized in that: The energy consumption cost of the drone is calculated in step S2 based on the formula: In the formula, E M (t) represents the hovering energy consumption, E T (t) represents the transmission energy consumption, E C (t) represents the computational energy consumption.
6. The method for adaptive UAV trajectory optimization and resource allocation based on improved DT according to claim 5, characterized in that: The improved model pre-training in step S3 updates the parameter θ through the mean square error, and its formula is: L(θ)=[a * (t)=a(t)] 2 ; In the formula, a * (t) represents the expert action that satisfies the constraints generated by the constrained reinforcement learning algorithm.
7. The method for adaptive UAV trajectory optimization and resource allocation based on improved DT according to claim 6, characterized in that: In step S4, the LoRa lightweight method is used to fine-tune the improved model.