A trajectory optimization and computation offloading method and system based on hierarchical reinforcement learning

Abstract

This invention provides a trajectory optimization and computational unloading method and system based on hierarchical reinforcement learning, comprising: acquiring the position information of each UE and the UAV at the current time; employing a sub-target-based HRL framework, dividing the agent into an upper-layer network and a lower-layer network, wherein the upper-layer network generates k cluster centers through a clustering algorithm, and uses the k cluster centers as a sub-target space, representing the expected location of the UAV; the upper-layer network selects a sub-target non-repeatingly every time slot, using the state space and action features as input; based on the sub-targets selected by the upper-layer network, the lower-layer network adjusts the flight trajectory of the UAV in real time in each time slot to achieve rapid response and optimization to the dynamic environment, guiding the UAV to the sub-target; after the UAV moves for a preset duration in each time slot and remains stationary, the lower-layer network first determines the priority of each UE based on the distance between each UE and the UAV, and then makes a computational unloading decision. This invention can perform trajectory optimization and computational unloading.

Description

Technical Field

[0001] This invention belongs to the field of UAV-assisted mobile edge computing (UAV-assisted MEC) technology, and specifically refers to a trajectory optimization and computation offloading method and system based on hierarchical reinforcement learning. Background Technology

[0002] Mobile edge computing (MEC) refers to the use of edge servers to assist user equipment (UEs) in performing computational tasks when their computing power is limited and they struggle to complete latency-sensitive tasks while maintaining low power consumption. Unmanned aerial vehicles (UAVs) have become increasingly popular in the MEC field in recent years due to their compelling advantages such as mobility, flexibility, and limited deployment costs. In UAV-assisted MEC, the UAV operates as an aerial edge server, assisting the UE in performing computational tasks. Currently, hierarchical reinforcement learning (HRL) has made initial attempts to address the high-dimensional problems in UAV-assisted MEC.

[0003] In the MEC field, cascaded HRL methods have been initially applied to task decomposition and resource allocation. Cascaded HRL methods typically decompose the original problem into two fixed subproblems, which are usually coupled, with the result of the first subproblem influencing the decision of the second. The upper-layer network independently solves the first subproblem and passes the decision to the lower-layer network, which then solves the second subproblem based on this result. Ren et al. used cascaded HRL, with the upper and lower layers independently solving the UAV trajectory optimization and computational offloading problems. The output of the upper-layer network, i.e., the UAV's position, serves as the input to the lower-layer network. Given a fixed UAV position, the lower-layer network determines the next... The offloading strategy for UE tasks in each time slot.

[0004] The study assumes that drones are... The fixed position within each time slot does not align with real-world application scenarios. Furthermore, cascaded HRLs decompose tasks in a relatively fixed way, lacking dynamic adjustment capabilities and making them ill-suited for complex and changing environments. Therefore, how to flexibly optimize UAV trajectories and computational offloading strategies while considering communication quality has become a crucial issue requiring urgent research. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention provides a trajectory optimization and computational unloading method and system based on hierarchical reinforcement learning, the technical solution of which is as follows: On the one hand, a trajectory optimization and computational unloading method based on hierarchical reinforcement learning is provided, which includes: S1. Obtain the location information of each User Equipment (UEs) and the current UAV (UAV). The UAV operates as an airborne edge server to help the UEs perform computing tasks. S2. A hierarchical reinforcement learning (HRL) framework based on sub-targets is adopted, which divides the agent into an upper network and a lower network. The upper network analyzes the location distribution of UEs through the k-means clustering algorithm, automatically identifies dense areas of UEs, generates k cluster centers, and uses the k cluster centers as the sub-target space. The sub-target space includes k sub-targets, and the sub-targets represent the locations that the UAV is expected to reach in the future. S3. The upper-layer network takes the state space and action features as input, and every... The k non-repeating sub-targets are selected in the time slot, wherein the state space includes: UAV position and whether the sub-target is selected, and the action features include: the position and distance of k sub-targets relative to the UAV; S4. Based on the sub-target selected by the upper-layer network, the lower-layer network adjusts the flight trajectory of the UAV in real time in each time slot to achieve rapid response and optimization to the dynamic environment. The state space of the lower-layer network includes the position of the UAV and the target angle. The lower-layer network uses the state space and the sub-target as input to decide the speed and angle of the UAV flight and guide the UAV to reach the sub-target. S5. After the UAV moves for a preset time in each time slot and remains stationary, the lower-layer network first determines the priority of each UE based on the distance between each UE and the UAV, and then uses the DDQN algorithm to make a decision on the offloading of the task, including: selecting one UE from the d highest priority UEs to provide service, and deciding on the offloading method: whether the task is calculated locally by the selected UE or offloaded by the UAV.

[0006] On the other hand, a trajectory optimization and computational unloading system based on hierarchical reinforcement learning is provided, the system comprising: The acquisition module is used to acquire the location information of each user equipment (UEs) and the unmanned aerial vehicle (UAV) at the current time. The UAV runs as an airborne edge server to help the UEs perform computing tasks. The generation module is used to divide the agent into an upper network and a lower network using a sub-target-based hierarchical reinforcement learning (HRL) framework. The upper network analyzes the location distribution of UEs using the k-means clustering algorithm, automatically identifies dense UE regions, generates k cluster centers, and uses the k cluster centers as a sub-target space. The sub-target space includes k sub-targets, and each sub-target represents the location that the UAV is expected to reach in the future. The selection module is used by the upper-layer network to select parameters every [time period], taking the state space and action features as input. The k non-repeating sub-targets are selected in the time slot, wherein the state space includes: UAV position and whether the sub-target is selected, and the action features include: the position and distance of k sub-targets relative to the UAV; The adjustment module is used to adjust the flight trajectory of the UAV in real time in each time slot according to the sub-target selected by the upper network, so as to achieve rapid response and optimization to the dynamic environment. The state space of the lower network includes the position of the UAV and the target angle. The lower network uses the state space and the sub-target as input to decide the speed and angle of the UAV flight and guide the UAV to reach the sub-target. The decision module is used to determine the priority of each UE based on the distance between each UE and the UAV after the UAV moves for a preset time in each time slot and remains stationary. Then, the DDQN algorithm is used to make a decision on the offloading of the UE, including: selecting one UE from the d highest priority UEs to provide service, and deciding on the offloading method: whether the task is calculated locally by the selected UE or offloaded by the UAV.

[0007] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described trajectory optimization and computation offloading method based on hierarchical reinforcement learning.

[0008] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described trajectory optimization and computation offloading method based on hierarchical reinforcement learning.

[0009] The beneficial effects of the technical solution provided by this invention include at least the following: This invention is the first to apply the sub-target-based HRL framework to the field of UAV-assisted MEC, flexibly decomposing the trajectory optimization objective of the UAV and significantly reducing the overall computational latency of the system. Specifically: By generating UAV location sub-targets, the system macroscopically plans the location where UAVs reach densely distributed users. The agent is divided into an upper-layer network and a lower-layer network. The upper-layer network uses a clustering algorithm to identify densely distributed UE areas, takes the cluster centers of densely distributed UEs as the sub-target space, and adapts to the dynamic discrete action space by introducing action features. The lower-layer network adjusts the trajectory of the UAV in real time according to the sub-targets, and calculates the unloading strategy combined with priority to reduce the action space, further alleviating the high-dimensional problem caused by the increase in the number of UEs. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a trajectory optimization and computational unloading method based on hierarchical reinforcement learning provided in an embodiment of the present invention; Figure 2 This is a general block diagram of a trajectory optimization and computational unloading method based on hierarchical reinforcement learning provided in an embodiment of the present invention; Figure 3 This is a diagram comparing the training performance of different algorithms; Figure 4 This is a schematic diagram illustrating the relationship between system latency and the number of UEs; Figure 5 This is a schematic diagram of the movement trajectory of a UAV under different training episode numbers, where (a) is 1600 episodes, (b) is 1800 episodes, and (c) is 2000 episodes. Figure 6 yes A diagram illustrating the performance comparison under different values; Figure 7 This is a diagram illustrating the performance comparison under different distance threshold values; Figure 8 This is a block diagram of a trajectory optimization and computational unloading system based on hierarchical reinforcement learning provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0013] In reinforcement learning, an agent interacts with the environment to obtain observation data and finds the optimal policy for a given decision optimization task. The agent takes an action based on the current state and policy. The environment reacts to the agent's action and returns a reward to the agent. At the same time, the environment changes to enter the next state. The agent's ultimate goal is to find an optimal policy to maximize the cumulative reward. In the sub-goal-based hierarchical reinforcement learning (HRL) of this invention, the agent is divided into an upper network and a lower network. The upper network generates UAV position sub-goals, and the lower network adjusts the UAV's flight trajectory and performs computational unloading in real time. The hierarchical structure of HRL achieves dimensionality reduction of the state-action space. By generating sub-goals to enhance exploration, the system achieves shorter task latency and improves service quality in high-dimensional complex environments.

[0014] This invention provides a trajectory optimization and computational offloading method based on hierarchical reinforcement learning. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps: S1. Obtain the location information of each User Equipment (UEs) and the current UAV (UAV). The UAV operates as an airborne edge server to help the UEs perform computing tasks. S2. A hierarchical reinforcement learning (HRL) framework based on sub-targets is adopted, which divides the agent into an upper network and a lower network. The upper network analyzes the location distribution of UEs through the k-means clustering algorithm, automatically identifies dense areas of UEs, generates k cluster centers, and uses the k cluster centers as the sub-target space. The sub-target space includes k sub-targets, and the sub-targets represent the locations that the UAV is expected to reach in the future. Optionally, S2 specifically includes: The upper-layer network is based on the location of all UEs. The k-means algorithm is used to obtain the k cluster centers of densely distributed users: Set the number of clusters in the k-means clustering to k; Randomly set the center of the k-cluster; Iterate through the following operations until the update of any center is less than a specified threshold: Calculate the distance between each UE and the center; Assign each UE to the nearest center; Update the center of each cluster according to the newly specified UE; These k cluster centers are used as the sub-target space.

[0015] S3. The upper-layer network takes the state space and action features as input, and every... The k non-repeating sub-targets are selected in the time slot, wherein the state space includes: UAV position and whether the sub-target is selected, and the action features include: the position and distance of k sub-targets relative to the UAV; Optionally, S3 specifically includes: In time slot n, the state space of the upper-layer network is defined as follows:

[0016] in, Indicates the location of the UAV. The flag indicating whether a subtarget is selected is represented by the numbers {0,1}, where 1 indicates that the subtarget is not selected and 0 indicates that the subtarget has been selected. In the network, the subtarget is guaranteed not to be selected repeatedly in the form of a mask. The action performed by the upper-layer network is to select a sub-target from the sub-target space. The action space of the upper-layer network is defined as follows:

[0017] in, These are the cluster centers of densely distributed UEs determined by clustering algorithms; In HRL, the upper-layer network every The reward for selecting a sub-objective once per time slot is defined as follows: The cumulative external rewards from environmental feedback in each time slot: (1) in This refers to the external reward within a single time slot, i.e., the delay reward, which aligns with the ultimate goal of minimizing task processing latency. The external reward is defined as follows: (2) in Indicates the processing latency of a single task; Since the state-action space of the upper-layer network is discrete, the DDQN algorithm is chosen for sub-target selection. The target Y of the DDQN algorithm is: (4) in and These are the parameters of the Q-network and the target network in DDQN, respectively. This represents the reward the agent receives from the environment at time t+1. This represents the state of the agent at time t+1; It is a reward discount factor used to adjust for near-term and long-term effects, determining how to balance current and future rewards, and its value range is... ; a Indicates the agent's next possible action; The DDQN algorithm uses the Q value obtained directly from the current state and action of the training data (the position of the UE in the training data is randomly initialized in each round of training, and the position of the UAV is fixed at [500, 500]) and the mean square error of the target Y as the loss function to perform iterative optimization, train the neural network, and use the trained neural network to select the sub-target. Since the positions of UEs are randomly initialized in each training round, the positions of the cluster centers obtained by the k-means algorithm are constantly changing. This causes the discrete action space to keep changing. Therefore, action features are introduced into the DDQN algorithm, defining actions as learnable feature vectors. Let Q network learn To enhance network generalization, it is defined as follows:

[0018] in, This represents the position of the k cluster centers relative to the UAV. The distance between the k cluster centers and the UAV is represented by the position and distance relative to the UAV. By introducing action features, different sub-targets are represented by their positions and distances relative to the UAV in different training rounds. This is to cope with the dynamic changes in the action space caused by the different positions of the k cluster centers in each training round. DDQN can then select appropriate cluster centers as sub-targets in the dynamically changing action space.

[0019] This invention transforms the UAV trajectory optimization problem into a simpler, more easily implemented sub-target problem by selecting cluster centers with dense user distribution as sub-targets. This macroscopically optimizes the UAV trajectory and helps UAVs serve more UEs with shorter computational latency.

[0020] S4. Based on the sub-target selected by the upper-layer network, the lower-layer network adjusts the flight trajectory of the UAV in real time in each time slot to achieve rapid response and optimization to the dynamic environment. The state space of the lower-layer network includes the position of the UAV and the target angle. The lower-layer network uses the state space and the sub-target as input to decide the speed and angle of the UAV flight and guide the UAV to reach the sub-target. Optionally, S4 specifically includes: After the upper-layer network determines the sub-target, the optimization objective of the lower-layer network is to guide the agent to the sub-target. In the lower-layer network, the state space consists of the UAV's position and the target angle, defined as follows:

[0021] in, It is the angle of the target position relative to the UAV; In the lower-level network, sub-targets are achieved by controlling the flight angle and speed of the UAV; that is, the action space is the speed of the UAV's flight. and angle The definition is as follows:

[0022] Design Distance Reward Directional rewards To guide the UAV to the sub-target location, the reward for the lower-layer network is as follows: (5) in , It is a weighting factor; Since the action space of the lower-layer network is continuous, the SAC algorithm is chosen for decision-making. The SAC algorithm introduces the concept of maximum entropy based on maximizing future cumulative rewards. The purpose of adding entropy is to enhance robustness and the agent's exploratory ability. The objective function expression of the SAC algorithm is as follows: (6) in Indicating in strategy Down The distribution, This is the entropy regularization coefficient, used to control the degree of importance of entropy. This represents the entropy value. The larger the entropy value, the greater the agent's exploration of the environment, enabling the agent to find a more efficient strategy and helping to accelerate subsequent strategy learning. The Q-value of the SAC algorithm is calculated using Bellman variance improved based on entropy, and the value function is defined as follows: (7) in, The state value function is obtained by sampling from the experience replay pool D and is defined as follows: (8) It represents the expected reward in a certain state; Furthermore, the policy network in the SAC algorithm Soft-state value network Target state value network and two softQ networks They are respectively composed of , , , Parameterization: To find the optimal policy for each, stochastic gradient descent is applied to their objective functions. (9) Furthermore, the minimum value of the soft Q-value is taken from two values. and A parameterized Q-value function (similar to the form of the DDQN algorithm) helps avoid overestimating inappropriate Q values, thus improving training speed. The soft Q-value function is updated by minimizing the Bellman error. (10) The policy network is updated by minimizing the KL divergence: (11).

[0023] S5. When the UAV moves for a preset duration in each time slot and remains stationary (for example, a time slot is 7s, the UAV moves in the first 2s and remains stationary in the last 5s), the lower layer network first determines the priority of each UE based on the distance between each UE and the UAV, and then uses the DDQN algorithm to make a decision on the offload calculation, including: selecting one UE from the d highest priority UEs to provide service, and deciding on the offload method: whether the task is calculated locally by the selected UE or offloaded by the UAV.

[0024] Optionally, the state space dimension of the DDQN algorithm in S5 is 2d, consisting of the delays of local computation by d UEs and UAV offloading. The action space dimension is also 2d, which is a combination of local computation or UAV offloading actions for each UE, used to indicate which UE was selected and whether local computation or UAV offloading was selected. 0-1 offloading is adopted. The task of the selected UE is completely computed locally or completely computed by the UAV. The reward function is designed to be a negative number of the task processing delay.

[0025] For example, when the 2d-1 and 2d dimensions of the action space are {0,0}, it means that the d-th UE was not selected for service; when the 2d-1 and 2d dimensions of the action space are {1,0}, it means that the d-th UE is served and the task is computed locally; when the 2d-1 and 2d dimensions of the action space are {0,1}, it means that the d-th UE is served and the task is offloaded by UAV.

[0026] Here is a specific example: In a UAV-assisted MEC system, consider a two-dimensional square region distributed in... And assume that the UAV is at a fixed height The UAV's maximum flight speed is 50 m / s. The transmission bandwidth B is 10 MHz, and the noise power... At a reference distance of 1m, the channel power gain is set to UE's communication transmit power The UE's computing power is set to... The computing power of the UAV is set to The number of CPU cycles required for UE and UAV task execution is 1000 bits / cycle.

[0027] To ensure fairness in UE services, it is assumed that each UE can only be served once, and the service terminates after all UEs have been served. UAV can only serve one UE in each time slot.

[0028] The goal is to optimize the total computational offloading latency of all UEs in the entire system. Therefore, this embodiment of the invention uses the total processing latency of all UE tasks as the evaluation metric.

[0029] SAC and DDPG are key algorithms in reinforcement learning, and many MEC studies have been based on and improved upon them. Therefore, the embodiments of this invention mainly compare these algorithms with the proposed algorithm.

[0030] Figure 3 The overall computational latency of different algorithms at UE=60 is shown. The SAC and DDPG algorithms fail to converge in the high-dimensional state space. The HRL algorithm proposed in this embodiment of the invention rapidly and significantly reduces the system latency after a period of training, reducing the computational latency by 47 seconds compared to the SAC algorithm.

[0031] Figure 4 The trend of average time consumption with increasing number of UEs under different algorithms was evaluated. The embodiments of this invention observed that as the number of UEs increases, the overall task load of the system increases, and the system employing these offloading schemes experiences a corresponding increase in task execution latency. In contrast, the algorithm proposed in these embodiments of this invention exhibits the slowest latency escalation speed, thus highlighting its significant adaptability to different numbers of UEs.

[0032] exist Figure 5 In the analysis, we can see that the sub-targets generated by the upper layer tend to select the cluster center closest to the UAV each time. When the UAV traverses the sub-target positions, it can greedily select the sub-target that is closer. This demonstrates the rationality of the upper-layer sub-target selection.

[0033] The lower-level rewards are determined by distance. Directional rewards Composition, including distance reward Directional rewards Rewards are given for achieving sub-goals, and the reward for achieving a goal is defined as follows:

[0034] The embodiments of the present invention make ,like Figure 6 As shown, when In this way, the system can converge to the minimum delay more quickly and smoothly.

[0035] In the achievement of sub-goals, the distance threshold for determining whether a sub-goal is successful directly affects the overall performance of HRL. Figure 7 As shown, when the distance between the UAV and the sub-target is set to within 50m as the successful achievement of the sub-target, the HRL algorithm proposed in this embodiment of the invention has the best optimization effect.

[0036] like Figure 8 As shown, this embodiment of the invention also provides a trajectory optimization and computational offloading system based on hierarchical reinforcement learning, the system comprising: The acquisition module 810 is used to acquire the location information of each user equipment (UEs) and the current time of the unmanned aerial vehicle (UAV). The UAV runs as an airborne edge server to help the UEs perform computing tasks. The generation module 820 is used to adopt a sub-target-based hierarchical reinforcement learning (HRL) framework to divide the agent into an upper-layer network and a lower-layer network. The upper-layer network analyzes the location distribution of UEs using the k-means clustering algorithm, automatically identifies dense areas of UEs, generates k cluster centers, and uses the k cluster centers as a sub-target space. The sub-target space includes k sub-targets, and the sub-targets represent the locations that the UAV is expected to reach in the future. Selection module 830 is used by the upper-layer network to select, at intervals, the state space and action features as input, the selection module 830. The k non-repeating sub-targets are selected in the time slot, wherein the state space includes: UAV position and whether the sub-target is selected, and the action features include: the position and distance of k sub-targets relative to the UAV; The adjustment module 840 is used to adjust the flight trajectory of the UAV in real time in each time slot according to the sub-target selected by the upper network to achieve rapid response and optimization to the dynamic environment. The state space of the lower network includes the position of the UAV and the target angle. The lower network uses the state space and the sub-target as input to decide the speed and angle of the UAV flight and guide the UAV to reach the sub-target. The decision module 850 is used to determine the priority of each UE based on the distance between each UE and the UAV after the UAV moves for a preset time in each time slot and then uses the DDQN algorithm to make a decision on the offloading of the UE, including: selecting one UE from the d UEs with the highest priority to provide service, and deciding on the offloading method: whether the task is calculated locally by the selected UE or offloaded by the UAV.

[0037] Optionally, the generation module is specifically used for: The upper-layer network is based on the location of all UEs. The k-means algorithm is used to obtain the k cluster centers of densely distributed users: Set the number of clusters in the k-means clustering to k; Randomly set the center of the k-cluster; Iterate through the following operations until the update of any center is less than a specified threshold: Calculate the distance between each UE and the center; Assign each UE to the nearest center; Update the center of each cluster according to the newly specified UE; These k cluster centers are used as the sub-target space.

[0038] Optionally, the selection module is specifically used for: In time slot n, the state space of the upper-layer network is defined as follows:

[0039] in, Indicates the location of the UAV. The flag indicating whether a subtarget is selected is represented by the numbers {0,1}, where 1 indicates that the subtarget is not selected and 0 indicates that the subtarget has been selected. In the network, the subtarget is guaranteed not to be selected repeatedly in the form of a mask. The action performed by the upper-layer network is to select a sub-target from the sub-target space. The action space of the upper-layer network is defined as follows:

[0040] in, These are the cluster centers of densely distributed UEs determined by clustering algorithms; In HRL, the upper-layer network every The reward for selecting a sub-objective once per time slot is defined as follows: The cumulative external rewards from environmental feedback in each time slot: (1) in This refers to the external reward within a single time slot, i.e., the delay reward, which aligns with the ultimate goal of minimizing task processing latency. The external reward is defined as follows: (2) in Indicates the processing latency of a single task; Since the state-action space of the upper-layer network is discrete, the DDQN algorithm is chosen for sub-target selection. The target Y of the DDQN algorithm is: (4) in and These are the parameters of the Q-network and the target network in DDQN, respectively. This represents the reward the agent receives from the environment at time t+1. This represents the state of the agent at time t+1; It is a reward discount factor used to adjust for near-term and long-term effects, determining how to balance current and future rewards, and its value range is... ; a Indicates the agent's next possible action; The DDQN algorithm uses the Q-value obtained directly from the current state and actions of the training data and the mean square error of the target Y as the loss function to perform iterative optimization, train the neural network, and use the trained neural network to select the sub-target. Since the positions of UEs are randomly initialized in each training round, the positions of the cluster centers obtained by the k-means algorithm are constantly changing. This causes the discrete action space to keep changing. Therefore, action features are introduced into the DDQN algorithm, defining actions as learnable feature vectors. Let Q network learn To enhance network generalization, it is defined as follows:

[0041] in, This represents the position of the k cluster centers relative to the UAV. The distance between the k cluster centers and the UAV is represented by the position and distance relative to the UAV. By introducing action features, different sub-targets are represented by their positions and distances relative to the UAV in different training rounds. This is to cope with the dynamic changes in the action space caused by the different positions of the k cluster centers in each training round. DDQN can then select appropriate cluster centers as sub-targets in the dynamically changing action space.

[0042] Optionally, the adjustment module is specifically used for: After the upper-layer network determines the sub-target, the optimization objective of the lower-layer network is to guide the agent to the sub-target. In the lower-layer network, the state space consists of the UAV's position and the target angle, defined as follows:

[0043] in, It is the angle of the target position relative to the UAV; In the lower-level network, sub-targets are achieved by controlling the flight angle and speed of the UAV; that is, the action space is the speed of the UAV's flight. and angle The definition is as follows:

[0044] Design Distance Reward Directional rewards To guide the UAV to the sub-target location, the reward for the lower-layer network is as follows: (5) in , It is a weighting factor; Since the action space of the lower-layer network is continuous, the SAC algorithm is chosen for decision-making. The SAC algorithm introduces the concept of maximum entropy based on maximizing future cumulative rewards. The purpose of adding entropy is to enhance robustness and the agent's exploratory ability. The objective function expression of the SAC algorithm is as follows: (6) in Indicating in strategy Down The distribution, This is the entropy regularization coefficient, used to control the degree of importance of entropy. This represents the entropy value. The larger the entropy value, the greater the agent's exploration of the environment, enabling the agent to find a more efficient strategy and helping to accelerate subsequent strategy learning. The Q-value of the SAC algorithm is calculated using Bellman variance improved based on entropy, and the value function is defined as follows: (7) in, The state value function is obtained by sampling from the experience replay pool D and is defined as follows: (8) It represents the expected reward in a certain state; Furthermore, the policy network in the SAC algorithm Soft-state value network Target state value network and two softQ networks They are respectively composed of , , , Parameterization: To find the optimal policy for each, stochastic gradient descent is applied to their objective functions. (9) Furthermore, the minimum value of the soft Q-value is taken from two values. and A parameterized Q-value function helps avoid overestimating inappropriate Q values, thus improving training speed. The soft Q-value function is updated by minimizing the Bellman error. (10) The policy network is updated by minimizing the KL divergence: (11).

[0045] Optionally, the state space dimension of the DDQN algorithm in the decision module is 2d, consisting of the delays of local computation by d UEs and UAV offloading. The action space dimension is also 2d, which is a combination of local computation or UAV offloading actions for each UE, used to indicate which UE was selected and whether local computation or UAV offloading was selected. 0-1 offloading is adopted. The task of the selected UE is completely computed locally or completely computed by the UAV. The reward function is designed to be a negative number of the task processing delay.

[0046] The trajectory optimization and computational unloading system based on hierarchical reinforcement learning provided in this embodiment of the invention has a functional structure that corresponds to the trajectory optimization and computational unloading method based on hierarchical reinforcement learning provided in this embodiment of the invention, and will not be described again here.

[0047] Figure 9 This is a schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present invention. The electronic device 900 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 901 and one or more memories 902. The memory 902 stores at least one instruction, which is loaded and executed by the processor 901 to implement the steps of the above-described trajectory optimization and computational offloading method based on hierarchical reinforcement learning.

[0048] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to perform the aforementioned trajectory optimization and computation offloading method based on hierarchical reinforcement learning. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0049] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A trajectory optimization and computational unloading method based on hierarchical reinforcement learning, characterized in that, The method includes: S1. Obtain the location information of each User Equipment (UEs) and the current UAV (UAV). The UAV operates as an airborne edge server to help the UEs perform computing tasks. S2. A hierarchical reinforcement learning (HRL) framework based on sub-targets is adopted, which divides the agent into an upper network and a lower network. The upper network analyzes the location distribution of UEs through the k-means clustering algorithm, automatically identifies dense areas of UEs, generates k cluster centers, and uses the k cluster centers as the sub-target space. The sub-target space includes k sub-targets, and the sub-targets represent the locations that the UAV is expected to reach in the future. S3. The upper-layer network takes the state space and action features as input, and every... The k non-repeating sub-targets are selected in the time slot, wherein the state space includes: UAV position and whether the sub-target is selected, and the action features include: the position and distance of k sub-targets relative to the UAV; S4. Based on the sub-target selected by the upper-layer network, the lower-layer network adjusts the flight trajectory of the UAV in real time in each time slot to achieve rapid response and optimization to the dynamic environment. The state space of the lower-layer network includes the position of the UAV and the target angle. The lower-layer network uses the state space and the sub-target as input to decide the speed and angle of the UAV flight and guide the UAV to reach the sub-target. S5. After the UAV moves for a preset time in each time slot and remains stationary, the lower-layer network first determines the priority of each UE based on the distance between each UE and the UAV, and then uses the DDQN algorithm to make a decision on the offloading of the task, including: selecting one UE from the d highest priority UEs to provide service, and deciding on the offloading method: whether the task is calculated locally by the selected UE or offloaded by the UAV.

2. The method according to claim 1, characterized in that, S2 specifically includes: The upper-layer network is based on the location of all UEs. The k-means algorithm is used to obtain the k cluster centers of densely distributed users: Set the number of clusters in the k-means clustering to k; Randomly set the center of the k-cluster; Iterate through the following operations until the update of any center is less than a specified threshold: Calculate the distance between each UE and the center; Assign each UE to the nearest center; Update the center of each cluster according to the newly specified UE; These k cluster centers are used as the sub-target space.

3. The method according to claim 1, characterized in that, S3 specifically includes: In time slot n, the state space of the upper-layer network is defined as follows: ； in, Indicates the location of the UAV. The flag indicating whether a subtarget is selected is represented by the numbers {0,1}, where 1 indicates that the subtarget is not selected and 0 indicates that the subtarget has been selected. In the network, the subtarget is guaranteed not to be selected repeatedly in the form of a mask. The action performed by the upper-layer network is to select a sub-target from the sub-target space. The action space of the upper-layer network is defined as follows: ； in, These are the cluster centers of densely distributed UEs determined by clustering algorithms; In HRL, the upper-layer network every The reward for selecting a sub-objective once per time slot is defined as follows: The cumulative external rewards of environmental feedback in each time slot: (1) in This refers to the external reward within a single time slot, i.e., the delay reward, which aligns with the ultimate goal of minimizing task processing latency. The external reward is defined as follows: (2) in Indicates the processing latency of a single task; Since the state-action space of the upper-layer network is discrete, the DDQN algorithm is chosen for sub-target selection. The target Y of the DDQN algorithm is: (4) in and These are the parameters of the Q-network and the target network in DDQN, respectively. This represents the reward the agent receives from the environment at time t+1. This represents the state of the agent at time t+1; It is a reward discount factor used to adjust for near-term and long-term effects, determining how to balance current and future rewards, and its value range is... ; a Indicates the agent's next possible action; The DDQN algorithm uses the Q-value obtained directly from the current state and action of the training data and the mean square error of the target Y as the loss function to perform iterative optimization, train the neural network, and use the trained neural network to select the sub-target. Since the positions of UEs are randomly initialized in each training round, the positions of the cluster centers obtained by the k-means algorithm are constantly changing. This causes the discrete action space to keep changing. Therefore, action features are introduced into the DDQN algorithm, defining actions as learnable feature vectors. Let Q network learn To enhance network generalization, it is defined as follows: ； in, This represents the position of the k cluster centers relative to the UAV. The distance between the k cluster centers and the UAV is represented by the position and distance relative to the UAV. By introducing action features, different sub-targets are represented by their positions and distances relative to the UAV in different training rounds. This is to cope with the dynamic changes in the action space caused by the different positions of the k cluster centers in each training round. DDQN can then select appropriate cluster centers as sub-targets in the dynamically changing action space.

4. The method according to claim 3, characterized in that, S4 specifically includes: After the upper-layer network determines the sub-target, the optimization objective of the lower-layer network is to guide the agent to the sub-target. In the lower-layer network, the state space consists of the position of the UAV and the target angle, defined as follows: ； in, It is the angle of the target position relative to the UAV; In the lower-level network, sub-targets are achieved by controlling the flight angle and speed of the UAV; that is, the action space is the speed of the UAV's flight. and angle The definition is as follows: ； Design Distance Reward Directional rewards To guide the UAV to the sub-target location, the reward for the lower-layer network is as follows: (5) in , It is a weighting factor; Since the action space of the lower-layer network is continuous, the SAC algorithm is chosen for decision-making. The SAC algorithm introduces the concept of maximum entropy based on maximizing future cumulative rewards. The purpose of adding entropy is to enhance robustness and the agent's exploratory ability. The objective function expression of the SAC algorithm is as follows: (6) in Indicating in strategy Down The distribution, This is the entropy regularization coefficient, used to control the degree of importance of entropy. This represents the entropy value. The larger the entropy value, the greater the agent's exploration of the environment, enabling the agent to find a more efficient strategy and helping to accelerate subsequent strategy learning. The Q-value of the SAC algorithm is calculated using Bellman variance improved based on entropy, and the value function is defined as follows: (7) in, The state value function is obtained by sampling from the experience replay pool D and is defined as follows: (8) It represents the expected reward in a certain state; Furthermore, the policy network in the SAC algorithm Soft-state value network Target state value network and two softQ networks They are respectively composed of , , , Parameterization: To find the optimal policy for each, stochastic gradient descent is applied to their objective functions. (9) Furthermore, the minimum value of the soft Q-value is taken from two values. and A parameterized Q-value function helps avoid overestimating inappropriate Q values, thus improving training speed. The soft Q-value function is updated by minimizing the Bellman error. (10) The policy network is updated by minimizing the KL divergence: (11)。 5. The method according to claim 1, characterized in that, The state space dimension of the DDQN algorithm in S5 is 2d, which consists of the delays of local computation by d UEs and UAV offloading. The action space dimension is also 2d, which is a combination of local computation or UAV offloading actions for each UE. It is used to indicate which UE is selected and whether local computation or UAV offloading is selected. 0-1 offloading is adopted. The task of the selected UE is completely computed locally or completely computed by UAV. The reward function is designed to be a negative number of the task processing delay.

6. A trajectory optimization and computational unloading system based on hierarchical reinforcement learning, characterized in that, The system includes: The acquisition module is used to acquire the location information of each user equipment (UEs) and the unmanned aerial vehicle (UAV) at the current time. The UAV runs as an airborne edge server to help the UEs perform computing tasks. The generation module is used to divide the agent into an upper network and a lower network using a sub-target-based hierarchical reinforcement learning (HRL) framework. The upper network analyzes the location distribution of UEs using the k-means clustering algorithm, automatically identifies dense UE regions, generates k cluster centers, and uses the k cluster centers as a sub-target space. The sub-target space includes k sub-targets, and each sub-target represents the location that the UAV is expected to reach in the future. The selection module is used by the upper-layer network to select parameters every [time period], taking the state space and action features as input. The k non-repeating sub-targets are selected in the time slot, wherein the state space includes: UAV position and whether the sub-target is selected, and the action features include: the position and distance of k sub-targets relative to the UAV; The adjustment module is used to adjust the flight trajectory of the UAV in real time in each time slot according to the sub-target selected by the upper network, so as to achieve rapid response and optimization to the dynamic environment. The state space of the lower network includes the position of the UAV and the target angle. The lower network uses the state space and the sub-target as input to decide the speed and angle of the UAV flight and guide the UAV to reach the sub-target. The decision module is used to determine the priority of each UE based on the distance between each UE and the UAV after the UAV moves for a preset time in each time slot and remains stationary. Then, the DDQN algorithm is used to make a decision on the offloading of the UE, including: selecting one UE from the d highest priority UEs to provide service, and deciding on the offloading method: whether the task is calculated locally by the selected UE or offloaded by the UAV.

7. The system according to claim 6, characterized in that, The generation module is specifically used for: The upper-layer network is based on the location of all UEs. The k-means algorithm is used to obtain the k cluster centers of densely distributed users: Set the number of clusters in the k-means clustering to k; Randomly set the center of the k-cluster; Iterate through the following operations until the update of any center is less than a specified threshold: Calculate the distance between each UE and the center; Assign each UE to the nearest center; Update the center of each cluster according to the newly specified UE; These k cluster centers are used as the sub-target space.

8. The system according to claim 6, characterized in that, The selection module is specifically used for: In time slot n, the state space of the upper-layer network is defined as follows: ； in, Indicates the location of the UAV. The flag indicating whether a subtarget is selected is represented by the numbers {0,1}, where 1 indicates that the subtarget is not selected and 0 indicates that the subtarget has been selected. In the network, the subtarget is guaranteed not to be selected repeatedly in the form of a mask. The action performed by the upper-layer network is to select a sub-target from the sub-target space. The action space of the upper-layer network is defined as follows: ； in, These are the cluster centers of densely distributed UEs determined by clustering algorithms; In HRL, the upper-layer network every The reward for selecting a sub-objective once per time slot is defined as follows: The cumulative external rewards of environmental feedback in each time slot: (1) in This refers to the external reward within a single time slot, i.e., the delay reward, which aligns with the ultimate goal of minimizing task processing latency. The external reward is defined as follows: (2) in Indicates the processing latency of a single task; Since the state-action space of the upper-layer network is discrete, the DDQN algorithm is chosen for sub-target selection. The target Y of the DDQN algorithm is: (4) in and These are the parameters of the Q-network and the target network in DDQN, respectively. This represents the reward the agent receives from the environment at time t+1. This represents the state of the agent at time t+1; It is a reward discount factor used to adjust for near-term and long-term effects, determining how to balance current and future rewards, and its value range is... ; a Indicates the agent's next possible action; The DDQN algorithm uses the Q-value obtained directly from the current state and actions of the training data and the mean square error of the target Y as the loss function to perform iterative optimization, train the neural network, and use the trained neural network to select the sub-target. Since the positions of UEs are randomly initialized in each training round, the positions of the cluster centers obtained by the k-means algorithm are constantly changing. This causes the discrete action space to keep changing. Therefore, action features are introduced into the DDQN algorithm, defining actions as learnable feature vectors. Let Q network learn To enhance network generalization, it is defined as follows: ； in, This represents the position of the k cluster centers relative to the UAV. The distance between the k cluster centers and the UAV is represented by the position and distance relative to the UAV. By introducing action features, different sub-targets are represented by their positions and distances relative to the UAV in different training rounds. This is to cope with the dynamic changes in the action space caused by the different positions of the k cluster centers in each training round. DDQN can then select appropriate cluster centers as sub-targets in the dynamically changing action space.

9. The system according to claim 8, characterized in that, The adjustment module is specifically used for: After the upper-layer network determines the sub-target, the optimization objective of the lower-layer network is to guide the agent to the sub-target. In the lower-layer network, the state space consists of the UAV's position and the target angle, defined as follows: ； in, It is the angle of the target position relative to the UAV; In the lower-level network, sub-targets are achieved by controlling the flight angle and speed of the UAV; that is, the action space is the speed of the UAV's flight. and angle The definition is as follows: ； Design Distance Rewards Directional rewards To guide the UAV to the sub-target location, the reward for the lower-layer network is as follows: (5) in , It is a weighting factor; Since the action space of the lower-layer network is continuous, the SAC algorithm is chosen for decision-making. The SAC algorithm introduces the concept of maximum entropy based on maximizing future cumulative rewards. The purpose of adding entropy is to enhance robustness and the agent's exploratory ability. The objective function expression of the SAC algorithm is as follows: (6) in Indicating in strategy Down The distribution This is the entropy regularization coefficient, used to control the degree of importance of entropy. This represents the entropy value. The larger the entropy value, the greater the agent's exploration of the environment, enabling the agent to find a more efficient strategy and helping to accelerate subsequent strategy learning. The Q-value of the SAC algorithm is calculated using Bellman variance improved based on entropy, and the value function is defined as follows: (7) in, The state value function is obtained by sampling from the experience replay pool D and is defined as follows: (8) It represents the expected reward in a certain state; Furthermore, the policy network in the SAC algorithm Soft-state value network Target state value network and two softQ networks They are respectively composed of , , , Parameterization: To find the optimal policy for each, stochastic gradient descent is applied to their objective functions. (9) Furthermore, the minimum value of the soft Q-value is taken from two values. and A parameterized Q-value function helps avoid overestimating inappropriate Q values, thus improving training speed. The soft Q-value function is updated by minimizing the Bellman error. (10) The policy network is updated by minimizing the KL divergence: (11)。 10. The system according to claim 6, characterized in that, The state space dimension of the DDQN algorithm in the decision module is 2d, consisting of the delays of local computation by d UEs and UAV offloading. The action space dimension is also 2d, which is a combination of local computation or UAV offloading actions for each UE, used to indicate which UE was selected and whether local computation or UAV offloading was selected. 0-1 offloading is used. The task of the selected UE is completely computed locally or completely computed by the UAV. The reward function is designed to be a negative number of the task processing delay.

Images