A cooperative low-altitude unmanned aerial vehicle network assisted mobile edge computing system and an optimization method thereof

Abstract

The application belongs to the field of low-altitude economy and the field of mobile edge computing, and discloses a cooperative low-altitude unmanned aerial vehicle network assisted mobile edge computing system and an optimization method thereof, proposes an improved multi-agent deep reinforcement learning model, dynamically copes with system state uncertainty caused by high mobility by fusing space-time perception and energy optimization mechanism, and realizes collaborative minimization of unmanned aerial vehicle energy consumption and system task processing delay.

Description

Technical Field

[0001] This invention belongs to the fields of low-altitude economy and mobile edge computing, specifically relating to a collaborative low-altitude unmanned aerial vehicle (UAV) network-assisted mobile edge computing system and its optimization method. Background Technology

[0002] In recent years, the low-altitude economy has rapidly emerged as a new field integrating airspace resources, intelligent equipment, and scenario services. It encompasses diverse scenarios such as drone logistics, low-altitude transportation, and emergency rescue, creating an urgent demand for real-time data processing, low-latency response, and highly reliable services. Mobile edge computing, by offloading computing resources to the network edge, effectively shortens data transmission paths and has become a core technology supporting the low-latency, highly dynamic applications of the low-altitude economy. Against this backdrop, drones, with their high mobility and flexible deployment, have become a key carrier for mobile edge computing in low-altitude scenarios, serving as aerial edge nodes to provide computing offloading services to ground and low-altitude equipment. However, current drone-assisted mobile edge computing systems face significant challenges: drones have limited energy reserves, making continuous operation susceptible to endurance constraints; the high mobility of equipment in low-altitude environments (such as vehicle networks and low-altitude aircraft) leads to dynamic changes in system topology, resulting in uneven spatiotemporal distribution of task offloading demands; simultaneously, latency-sensitive applications (such as real-time monitoring and autonomous driving collaboration) have stringent requirements for 24 / 7 high availability, making traditional static resource scheduling strategies unsuitable for dynamic scenarios. To address the aforementioned problems, this invention proposes a collaborative framework for UAV-assisted mobile edge computing that integrates spatiotemporal awareness and energy optimization. This framework addresses the system state uncertainty caused by high mobility and ensures continuous service availability. The invention aims to provide efficient and stable computation offloading services for latency-sensitive applications in low-altitude economic scenarios, promoting the large-scale application of UAV-assisted mobile edge computing in these environments. Summary of the Invention

[0003] Addressing the significant challenges currently faced by UAV-assisted mobile edge computing systems—limited UAV energy reserves, making continuous operation susceptible to endurance constraints; high device mobility in low-altitude environments leading to dynamic changes in system topology and uneven spatiotemporal distribution of task offloading demands; and the stringent 24 / 7 high availability requirements of latency-sensitive applications, making traditional static resource scheduling strategies unsuitable for dynamic scenarios—this invention provides a collaborative low-altitude UAV network-assisted mobile edge computing system and its optimization method. It proposes an improved multi-agent deep reinforcement learning model that integrates spatiotemporal perception and energy optimization mechanisms to dynamically address system state uncertainties caused by high mobility, achieving synergistic minimization of UAV energy consumption and system task processing latency.

[0004] To achieve the above objectives, the present invention provides the following solution: A collaborative low-altitude unmanned aerial vehicle (UAV) network-assisted mobile edge computing system, the system comprising: N rotary-wing UAVs equipped with edge servers ESS and M mobile devices MDs; The collection of drones is denoted as Each drone is deployed in a specific area and equipped with a power source with a maximum energy output of [missing information]. The battery, the edge service unit with computing resources The set of mobile devices is denoted as Mobile devices are capable of moving within a specific area, and each mobile device also has local computing resources. With the transmitting antenna, its transmitting power is ; The system runtime is divided into T discrete time slots, denoted as: The duration of each time slot is Seconds; within any time slot t, the system uses a time-division multiplexing mechanism to enable each mobile device to interact with the environment sequentially; Each drone has a predetermined service range and only connects to mobile devices within the service range. Mobile devices outside the service range will perform local calculations.

[0005] Preferably, within any time slot t, the method by which the system enables each mobile device to interact with the environment sequentially through a time-division multiplexing mechanism includes: Assuming that each mobile device receives a computing task in each time slot, the mobile device can choose to execute the task locally or offload it to an edge server on a drone for processing. If it chooses to offload and the task is completed, the task will occupy the computing resources allocated to the device by the edge server until the end of the current time slot.

[0006] Preferably, in each time slot t, the mobile device m generates a computation task: this computation task is modeled as a triple. ,in Indicates the data size of the task. This indicates the number of CPU cycles required to process each bit of data. This represents the maximum end-to-end latency allowed for task completion, and the set of tasks generated by all mobile devices in this time slot is defined as follows: .

[0007] Preferably, the mobile device chooses to execute the task locally or offload it to an edge server on a drone for processing; if it chooses to offload and the task is completed, the task will occupy the computing resources allocated to the device by the edge server until the end of the current time slot. set up Indicates the ES number currently connected to device m; let This indicates whether the task is ultimately assigned to an ES for execution; in the initial phase of each time slot t, each mobile device MD will establish a connection with its nearest edge server ES: ; in Defined as the location of the drone. Defined as the location of the mobile device; The edge server (ES) will assess the characteristics of newly arriving tasks and its own remaining computing resources, and assist mobile devices in determining their compute offloading strategies. when When the mobile device m chooses to perform the computing task locally, the processing latency of the local computing task is expressed as: ; in In the time slot t Mobile devices m Local computing resources; when When this happens, the mobile device m offloads the computing task to the edge server via a wireless channel. When the device connects directly to and offloads to its nearest edge server, otherwise, if The mobile device m first transmits the task to its connected edge server, and then the server migrates the task to the target server for processing via a fixed-rate backhaul wireless channel. The transmission delay caused by this offloading process is expressed as: ; in For mobile devices m To the edge server Uplink data transmission rate, For the indicator function, the first term represents the uplink latency of the task's wireless transmission to the connected server; the second term characterizes the additional migration latency incurred when the task travels through the backhaul wireless channel. Indicates the nominal backhaul radio channel capacity. This indicates the instantaneous load on the backhaul radio channel that is occupied by other non-offloaded services due to network congestion. Once the unloaded computing tasks arrive at the target server, the server will allocate a certain computing frequency to the mobile device. The computational latency of this task on the edge server is expressed as: ; In edge computing mode, the total latency generated by a mobile device processing its computing tasks is expressed as: .

[0008] Preferably, the propulsion power energy consumption model for the rotorcraft UAV in the t-th time slot is as follows: ; in, and , where are constant coefficients, representing the blade profile power and inductive power of the UAV in hovering state, respectively; This indicates the tip velocity of the drone's rotor blades; It is the average rotor speed when the drone is hovering; and These are the drone's fuselage drag ratio and rotor solidity, respectively. and These represent air density and the area of ​​the drone's rotor disk, respectively. This refers to the flight speed of the drone.

[0009] This invention also provides an optimization method for a collaborative low-altitude unmanned aerial vehicle (UAV) network-assisted mobile edge computing system. The optimization method is used to optimize the aforementioned system and includes: The task cost of the UAV network-assisted mobile edge computing system is modeled as a weighted sum of task latency and energy consumption, and the weighted sum of task latency and energy consumption is transformed into a joint optimization problem of computation offloading and UAV trajectory. The joint optimization problem of computational unloading and UAV trajectory is modeled as a Markov decision process; The deep reinforcement learning framework D3MASAC, which integrates the dual deep Q-network D3QN and the multi-agent flexible action-evaluation algorithm MASAC, solves the Markov decision process, achieving collaborative computational offloading and UAV trajectory optimization.

[0010] Preferably, methods for modeling the task cost of a UAV network-assisted mobile edge computing system as a weighted sum of task latency and energy consumption include: The task costs of local computing and edge computing are expressed as follows: ; in and These represent the weighting coefficients for task delay costs and energy consumption costs, respectively. This represents the propulsion power energy consumption of a rotary-wing UAV in the t-th time slot. M This represents the number of mobile devices, and N represents the number of rotary-wing drones equipped with edge servers. The task completion delay of mobile device m in the t-th time slot is expressed as: ; in, Indicates the local computing task processing latency. This represents the total latency incurred by a mobile device in processing its computing tasks.

[0011] Preferably, the Markov decision process includes a state space, an action space, and a reward function; The state space is as follows: In time slot t, the system state corresponding to mobile device m is defined as follows: ; in This represents the index of the system state corresponding to mobile device m at time slot t. This state vector contains the task characteristics of the mobile device itself. Index of currently connected edge servers The location of mobile device m and the position vectors of all drones The remaining CPU computing frequency resources The remaining battery energy vector ; The system state corresponding to drone n is defined as follows: ; This state vector contains the task feature vectors of all mobile devices. The index vector of all edge servers connected to mobile devices Move the location of all mobile devices and the position vectors of all drones The remaining battery energy vector ; Action space: The action performed by the calculation unloading module is defined as follows: ; That is, to calculate the unloading strategy; In the drone trajectory optimization module, each drone agent has independent actions, defined as follows: ; in This represents the sequence number of the system state corresponding to UAV n in time slot t, including the UAV's own velocity in the x-direction. and velocity in the y-direction ; The motion vectors of all drones are defined as follows: ; in Let x be the velocity of the drone N in the x-direction and y-direction. Reward function: The calculation unloading module is in state Next action The reward function obtained is defined as follows: ; in, The maximum possible latency of a task is calculated based on task attributes. Furthermore, a penalty is applied if the latency constraint is violated. ; Each drone agent in the drone trajectory optimization module is in a state Next action The reward function obtained is defined as follows: ; This includes the sum of rewards from all mobile device tasks, as well as the negative cost of the drone's own flight energy consumption; The reward vector for all drones is defined as follows: ; in, Let be the reward function for UAV 1 in time slot t. Let be the reward function for drone N in time slot t.

[0012] Preferably, the deep reinforcement learning framework D3MASAC, based on the fusion of Duel Dual Deep Q Network D3QN and Multi-Agent Flexible Action-Evaluation Algorithm MASAC, includes: a D3QN module and a MASAC module. The D3QN module is used to generate efficient computation offloading decisions for mobile device MDs through iterative optimization, taking the current system state as input. The MASAC module is used to integrate status information and environmental feedback to output flight control actions for each UAV. The D3QN module employs two neural networks: an online network. The parameters are Used for immediate decision-making; target network The parameter is for delayed synchronization. , used for stable value estimation; Each agent in the MASAC module employs an actuator-evaluator based architecture, which includes an online actuator network. The parameters are This is used to generate probability distribution-based drone flight strategies, and two evaluator networks. The parameters are respectively This is used to estimate the Q-value, supplemented by a target evaluator network with two delays. The parameters are .

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a method for modeling the joint optimization problem of a drone-assisted mobile edge computing system as a Markov decision process, including defining a state space, action space and reward function to handle the joint optimization of computation offloading and drone trajectory; 1. A deep reinforcement learning framework (D3MASAC) based on the fusion duel dual deep Q network (D3QN) and the multi-agent flexible action-evaluation algorithm (MASAC) is used to jointly learn discrete computation offloading strategies and continuous UAV flight control strategies. 2. A dual-experience replay buffer mechanism for the computational offloading module and the UAV trajectory optimization module enables decoupled storage of experience data and independent network optimization; 3. The reward function incorporates a weighted sum of task delay cost and UAV energy consumption cost, and introduces a penalty term for violation of task delay constraints to balance system performance; 4. An integrated method for UAV propulsion energy consumption models, including thrust power modeling of rotary-wing UAVs, for accurate energy consumption estimation; 5. A multi-agent stochastic policy optimization method based on entropy regularization automatically adjusts the balance between exploration and exploitation, improving learning stability and efficiency.

[0014] This invention aims to reduce the overall latency and energy consumption of system tasks. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating the interaction process between the D3MASAC method proposed in this invention and the environment. Figure 2 This is a schematic diagram illustrating the task completion rate in an embodiment of the present invention; Figure 3 This is a schematic diagram of the reward curve in an embodiment of the present invention; Figure 4 The diagram shows the flight trajectory of the UAV in an embodiment of the present invention, wherein (a) is a schematic diagram of D3MASAC trajectory 1, (b) is a schematic diagram of D3MASAC trajectory 2, and (c) is a schematic diagram of D3MASAC trajectory 3. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1 This invention provides a collaborative low-altitude unmanned aerial vehicle (UAV) network-assisted mobile edge computing (MEC) system. In a UAV-assisted mobile edge computing (MEC) scenario, the system includes N rotary-wing UAVs equipped with edge servers (ESs) and M mobile devices (MDs). The set of UAVs is denoted as […]. Each drone is deployed in a specific area and equipped with a battery (maximum energy). Edge service units with computing resources The set of mobile devices is denoted as They can move within the area, and each device also has local computing resources. With the transmitting antenna, its transmitting power is The system runtime is divided into T discrete time slots, denoted as... The duration of each time slot is Seconds. Within any time slot t, the system uses a time-division multiplexing mechanism to enable each mobile device to interact with the environment sequentially. Assume that each mobile device receives a computational task in each time slot. At this time, the mobile device can choose to execute the task locally or offload it to an edge server on a drone for processing. If it chooses to offload and the task is completed, the task will consume the computational resources allocated to that device by the edge server until the end of the current time slot. Furthermore, each drone has a certain service range and only connects to mobile devices within its service range; devices outside the service range will perform local computation.

[0020] In each time slot t, the mobile device m generates a computation task. This task can be modeled as a triple. ,in The data size of the task is indicated in bits. This indicates the number of CPU cycles required to process each bit of data. This represents the maximum end-to-end latency allowed for task completion. The set of tasks generated by all mobile devices in this time slot is defined as follows: .

[0021] In time slot t, the wireless channel gain between edge server n and mobile device m is denoted as . , can be represented as: in This represents the channel gain at the reference distance. This represents the distance between edge server n and mobile device m. This represents the path loss index. Capture small-scale Rayleigh decay.

[0022] In time slot t, the transmit power of mobile device m is defined as Based on this, the signal-to-noise ratio of the link from mobile device m to edge server n can be expressed as: in It is the additional Gaussian white noise power.

[0023] In time slot t, the uplink data transfer rate from mobile device m to edge server n can be given by Shannon's formula: in This represents the bandwidth allocated by the system to this wireless channel.

[0024] The strategy formulated for mobile device m in time slot t can be represented by two decision variables: Let Indicates the ES number currently connected to device m; let... This indicates whether the task is ultimately assigned to an ES for execution. Specifically, at the beginning of each time slot t, each mobile device (MD) will establish a connection with its nearest edge server (ES), for example: in Defined as the location of the drone. Defined as the location of the mobile device. This server will assess the characteristics of newly arriving tasks and its own remaining computing resources, and based on this, assist the mobile device in determining its computing offload strategy. The specific offload strategy rules are as follows: When the mobile device m is unloaded in time slot t, the decision variable This means the device chooses to perform its computing tasks entirely locally. This process relies entirely on the device's own computing resources and involves no data transfer to the edge server. The specific process is described below: 1. Task Arrival. At the start of time slot t, device m generates a computational task, which can be characterized as follows: .

[0025] 2. Local resource determination. Device m is allocated a fixed CPU computing frequency based on task characteristics. (Unit: Hz) Used to perform this task.

[0026] 3. Local computation and execution. Task The data is sent to the local processor of device m, and the processor operates at a frequency... Process this task. The local computation time required to complete this task can be expressed as: in It refers to the local computing resources of mobile device m at time slot t.

[0027] When unloading decision variables At that time, the mobile device m offloads its computing tasks to the edge server for processing via a wireless channel. This represents the index of the target computing server, while This represents the index of the edge server (i.e., access point) to which the mobile device m is currently wirelessly connected. The entire process is based on... and Whether they are equal is determined in two modes: direct unloading and migration unloading. The resulting transmission latency is critical to system performance. The specific process is described below: 1. Wireless transmission from device to access point. Regardless of which server ultimately performs the calculations, the task data must first be transmitted wirelessly to the edge server currently connected to the device (indexed as...). At this point, the mobile device m is connected to the edge server. The uplink latency is: in For mobile devices m to edge servers Uplink data transmission rate.

[0028] 2. Task migration between servers (when) (Time). If the target computing server The server is not directly connected to the device. In this case, task migration needs to be performed between edge servers. The additional migration latency is: in This represents the standard wireless channel transmission rate between drones, while This indicates the instantaneous load on the backhaul radio channel that is occupied by other non-offloaded services due to network congestion.

[0029] In summary, the transmission delay caused by this unloading process can be expressed as: Right now in This is an indicator function.

[0030] Once the unloaded computing tasks arrive at the target server, the server will allocate a certain computing frequency to the mobile device. Therefore, the computational latency of this task on the edge server can be expressed as: Therefore, in edge computing mode, the total latency generated by a mobile device processing its computing tasks is represented as follows: The energy consumption of a drone mainly consists of two parts: energy used for communication and propulsion energy used to generate thrust to overcome air resistance and gravity. In practical applications, communication and computing energy consumption are typically two orders of magnitude smaller than flight energy consumption; therefore, this invention ignores the energy consumption of the communication and computing components. It is assumed that all drones in the system fly horizontally at the same altitude. In the t-th time slot, the velocity component on the horizontal plane is... and , represent the velocities in the x and y directions, respectively. Therefore, its position in the (t+1)th time slot... Updated to: in Let n be the x-axis coordinate of UAV n in time slot t+1. Let y be the y-coordinate of UAV n in time slot t+1.

[0031] At this time, the drone's flight speed is For a rotary-wing UAV, its propulsion power energy consumption in the t-th time slot can be modeled as: in, and , where are constant coefficients, representing the blade profile power and inductive power of the UAV in hovering state, respectively; This indicates the tip velocity of the drone's rotor blades; It is the average rotor speed when the drone is hovering; and These are the drone's fuselage drag ratio and rotor solidity, respectively. and These represent air density and the area of ​​the drone's rotor disk, respectively.

[0032] Example 2 This invention provides an optimization method for a collaborative low-altitude unmanned aerial vehicle (UAV) network-assisted mobile edge computing system. The optimization method is used to optimize the system described in Embodiment 1, and includes: The task cost of the UAV network-assisted mobile edge computing system is modeled as a weighted sum of task latency and energy consumption, and the weighted sum of task latency and energy consumption is transformed into a joint optimization problem of computation offloading and UAV trajectory. The joint optimization problem of computational unloading and UAV trajectory is modeled as a Markov decision process; The deep reinforcement learning framework D3MASAC, which integrates the dual deep Q-network D3QN and the multi-agent flexible action-evaluation algorithm MASAC, solves the Markov decision process, achieving collaborative computational offloading and UAV trajectory optimization.

[0033] The specific implementation process is as follows: In this invention, to balance the drone's energy consumption and mission completion delay, the system task cost is defined as the weighted sum of task delay and energy consumption. Specifically, the task cost of local computing and edge computing can be expressed as: in and These represent the weighting coefficients for task delay costs and energy consumption costs, respectively. The task completion delay of mobile device m in the t-th time slot is expressed as: Based on the above modeling, in order to solve the joint optimization problem of computational unloading and UAV trajectory, this invention proposes the following optimization problem: in .

[0034] This optimization problem belongs to the category of mixed-integer nonlinear programming. Due to its nonconvexity, direct solution involves high computational complexity. To effectively address this challenge, this invention reconstructs the original problem into a Markov Decision Process (MDP) and solves it using a deep reinforcement learning framework that combines the D3QN computational offloading module with the MASAC UAV trajectory optimization module. The specific analysis is as follows.

[0035] The joint computational unloading and UAV trajectory problem can be modeled as a sequential decision process, and the corresponding MDP consists of the following elements: State space: In time slot t, the system state corresponding to mobile device m is defined as follows: in This represents the index of the system state corresponding to mobile device m at time slot t. This state vector contains the task characteristics of the mobile device itself. Index of currently connected edge servers The location of mobile device m and the position vectors of all drones The remaining CPU computing frequency resources The remaining battery energy vector .

[0036] The system state corresponding to drone n is defined as follows: This state vector contains the task feature vectors of all mobile devices. The index vector of all edge servers connected to mobile devices Move the location of all mobile devices and the position vectors of all drones The remaining battery energy vector .

[0037] Action space: The action performed by the calculation unloading module is defined as follows: That is, to calculate the unloading strategy.

[0038] In the drone trajectory optimization module, each drone agent has independent actions, defined as follows: in This represents the sequence number of the system state corresponding to UAV n in time slot t, including the UAV's own velocity in the x-direction. and velocity in the y-direction The motion vectors of all drones are defined as follows: in Let x be the x-direction velocity and y-direction velocity of the drone N.

[0039] Reward function: The calculation unloading module is in state Next action The reward function obtained is defined as follows: in, This is the maximum possible latency for the task calculated based on task attributes. Furthermore, a penalty is applied if the latency constraint is violated. This design encourages agents to minimize task latency on mobile devices while strictly adhering to system constraints.

[0040] Each drone agent in the drone trajectory optimization module is in a state Next action The reward function obtained is defined as follows: This includes the sum of rewards from all mobile device tasks, as well as the negative cost of the drone's own flight energy consumption.

[0041] The reward vector for all drones is defined as follows: in Let be the reward function for UAV 1 in time slot t. Let be the reward function for drone N in time slot t, and so on.

[0042] Since the actions of the computational offloading module (discrete variables) and the UAV trajectory optimization module (continuous variables) influence each other, joint decision-making is required. Based on the above definition, this invention proposes a deep reinforcement learning framework (D3MASAC) that integrates Duel Dual Deep Q-Network (D3QN) and Multi-Agent Flexible Action-Evaluation Algorithm (MASAC) to achieve collaborative computational offloading and UAV trajectory optimization. D3MASAC consists of two collaborative modules: the D3QN module is responsible for learning discrete computational offloading strategies, while the MASAC module is used to learn continuous UAV flight control strategies. Specifically, D3QN takes the current system state as input and generates efficient computational offloading decisions for the mobile device (MD) through iterative optimization. Simultaneously, MASAC integrates state information and environmental feedback to output appropriate flight control actions for each UAV. The two modules jointly optimize the overall system performance through sharing state information and collaborative learning.

[0043] The core of the D3MASAC framework proposed in this invention lies in the fusion of D3QN, which handles discrete action spaces, and MASAC, which handles continuous action spaces, forming a unified, end-to-end deep reinforcement learning framework. The specific fusion process is as follows: 1. Network Responsibility Assignment and State Sharing. The system naturally decomposes the optimization problem into two types of sub-action spaces and assigns them to the network that is best suited to handle them. The starting point for fusion is state sharing, where the states of both networks are based on the same global state.

[0044] 2. Hierarchical Decision Making and Collaborative Execution. D3MASAC's execution at each time step t follows a hierarchical collaborative process: the upper-layer D3QN generates discrete unloading actions, while the lower-layer MASAC generates continuous UAV flight actions. These two actions are merged, and the system interacts with the environment simultaneously.

[0045] 3. Reward Fusion. The rewards of the lower-level MASAC include the reward signals of the upper-level D3QN, enabling fusion learning.

[0046] To mitigate Q-value overestimation and improve training stability, the D3QN module employs two neural networks: an online network. The parameters are Used for immediate decision-making; target network The parameter is for delayed synchronization. The online network processes the current state to generate computational offloading decisions. The target network provides a temporally differencing objective for Q-learning updates by synchronizing its periodic parameters with those of the online network. The duel architecture decomposes Q-value estimation into two independent streams: In the formula Estimate the value of the state. Calculate actions Advantages compared to other actions. The input is online network , The input is The advantage function, The input is The advantage function, It represents all possible actions in the decision space. This decomposition enhances the robustness of Q-value estimation by independently evaluating state quality and action-specific payoffs.

[0047] In multi-agent collaborative decision-making, environmental non-stationarity is a key challenge. Since each agent is constantly learning and updating its policy, from the perspective of a single agent, policy changes in other agents cause dynamic shifts in the environment, thus violating the stationary environment assumption upon which traditional reinforcement learning algorithms rely. This makes the training process difficult to converge and policy performance unstable. On the other hand, a completely centralized architecture requires all agents to share data in real-time without latency during execution or deployment, which is virtually impossible in real-world drone swarms due to bandwidth and latency limitations. To address this issue and promote efficient collaboration among agents, this invention employs the MASAC algorithm based on a centralized training and distributed execution paradigm.

[0048] The core idea of ​​the centralized training and distributed execution paradigm is to decouple the training process from the execution process, placing them under different information structures, thereby balancing the stability of learning with the real-time efficiency of decision-making. Specifically: Centralized training: During the training phase, the algorithm has access to global state information and the actions of all agents. Typically, a commenting network is used to estimate the global value function of joint actions. The commentator can utilize additional global information to more accurately evaluate the contribution of each agent's action and guide its policy updates. This centralized value learning helps overcome environmental non-stationarity because it implicitly models the policies of other agents during training, thus providing a stable and coordinated policy gradient signal for each agent.

[0049] Distributed execution: During the execution phase, each agent makes decisions based solely on its own local observations. Each agent possesses an independent action network that, during deployment, does not require access to information from other agents or the global state. This distributed decision-making model ensures system scalability and real-time performance, avoiding execution delays caused by communication bottlenecks or difficulties in obtaining global information.

[0050] Each agent in the MASAC module employs an action-evaluation based architecture, which includes an online action network. (Parameters are) (This is used to generate probability distribution-based drone flight strategies, and two evaluation networks) (The parameters are respectively) This is used to estimate the Q-value, supplemented by a target evaluator network with two delays. (Parameters are) Unlike deterministic policy methods, the MASAC module employs a stochastic policy gradient combined with entropy regularization. By injecting controlled randomness into the action selection and value estimation processes, it automatically balances exploration and exploitation. At its core, the agent not only pursues maximizing cumulative reward but also maximizing policy randomness. MASAC is based on an entropy-enhanced optimization objective; in standard reinforcement learning, the agent's goal is to maximize the expected cumulative reward. , As a discount factor, As a reward, MASAC adds a policy entropy term to the objective function. This forms a new maximum entropy objective: in For temperature coefficient, Let be the policy function. This combination of randomness and information-theoretic regularization prevents the algorithm from converging prematurely to a suboptimal policy while maintaining sampling efficiency through temporal difference learning.

[0051] The D3MASAC method proposed in this invention improves learning efficiency through a dual experience replay buffer mechanism. Two experience replay buffers... and The experience of unloading decisions and UAV flight strategies is stored independently to achieve decoupled neural network optimization. This invention employs a complementary exploration strategy during interaction with the environment: the D3QN module uses a decaying... The `-greedy` policy is used for task unloading, initially with... Prioritize exploration, then gradually converge to This allows the use of learned strategies; while the MASAC module is based on an online actuator network. The mean of the output and standard deviation Continuous drone flight actions are generated through parameterized Gaussian sampling. This collaborative exploration architecture reconciles the randomness in both discrete and continuous action dimensions, with collected experience partitioned into corresponding buffers for specialized policy optimization. This ensemble design ensures comprehensive exploration of the environment by progressively transitioning from random action selection to optimized policy execution, while leveraging empirical discorrelation to maintain training stability and prevent premature convergence in both decision spaces.

[0052] The interaction process between the D3MASAC method proposed in this invention and the environment is as follows: Figure 1 As shown, the specific steps are as follows: First, the environment is initialized, and the mobile devices, drone server, and neural network model are configured. , And create two independent experience replay buffer pools. and .

[0053] Then, a loop proceeds in "slots". Within each slot, the following operations are performed: 1. Dynamic environment generation: A new computing task is generated for each mobile device. It updates its own position according to the preset movement model.

[0054] 2. Drone Flight Decisions: Each drone acts as an intelligent agent, observing the current system status. Then, it is based on a specialized action network. Output a flight action It then performs this action to update its position in the air. This step aims to optimize the deployment of drones to better serve ground equipment.

[0055] 3. Mobile Device Offloading Decision: Each mobile device connects to a drone based on factors such as signal strength. For each device, the connected drone acts as its intelligent agent, observing the current state. The intelligent agent adopts -greedy exploration strategy, through The network selects a computational unloading action for the mobile device. Once the mobile device performs this action, it will immediately receive a task reward reflecting the immediate effect of that action. The local state of the system is also updated accordingly. The interactive experience at this step ( , , , It will be stored in the first experience pool. A Q-network specifically designed for training unloading decisions.

[0056] 4. Global Reward Calculation and Experience Storage: After all mobile devices have completed their offloading decisions for this time slot, the algorithm will aggregate the task rewards of all devices and calculate a global reward that reflects the overall system performance for this time slot. This global reward will serve as an evaluation of the drone swarm's flight decisions. The entire system then enters the global state for the next time slot. The flight actions of all drones within this time slot, the global rewards obtained, and the state transitions are combined into a complete experience tuple. Store in the second experience pool A strategy network specifically designed for training drone flights.

[0057] Ultimately, the algorithm outputs two types of empirical data at each step. , , , )and( These are used to train the unloading decision model and the flight control model, respectively, thereby achieving collaborative learning of UAV trajectory optimization and computational resource allocation.

[0058] The proposed D3MASAC method begins training after the experience accumulated in the experience replay buffer reaches a critical threshold. Specific details of the D3MASAC training method are explained below.

[0059] First, we examine the online networks in the D3QN and MASAC modules. , Use random parameters , Perform initialization, then copy via parameters. , , Initialize the target network , , To ensure consistency of the initial strategy, the D3QN and MASAC modules interact with the environment MT and NT times respectively during each training cycle. After each interaction, the D3QN and MASAC modules obtain empirical data and store it in an experience replay buffer. When the buffer reaches its capacity limit, the oldest stored experience will be overwritten by the newest experience.

[0060] The parameter update process of the D3QN module is as follows: it updates parameters through mini-batch random optimization from... In this process, experience replay learning is performed. Specifically, in each training iteration, a batch of experience samples is randomly sampled. And the parameters are updated by minimizing the Bellman approximation error loss function: in For a batch of experience from random sampling, for One of the lessons learned is... The input is The neural network parameters are online network , Through the target network Introduce delay strategy evaluation; in The reward decay factor for D3QN. The input is The neural network parameters are Target network , This represents the optimal action chosen by Q in the online network: To maintain the stability of the temporal difference objective during training, the target network parameters... It will periodically synchronize with the online network via hard parameter replication. This dual-Q learning architecture combines action selection (via the online network Q) with value evaluation (via the target network). This decoupling helps to alleviate the overestimation bias of the Q value.

[0061] The learning process of the MASAC module is as follows. After each interaction with the environment, each agent in the MASAC module retrieves data from the experience replay buffer. Random sampling Empirical data was collected, and the online commentator network was updated by minimizing the Bellman approximation error loss function. Parameters: in For a batch of experience from random sampling, for One of the lessons learned is... It is the kth online evaluation network for drone n. Its input is The neural network parameters are Target value The formula for calculating the truncated double-Q estimate and policy entropy is as follows: in It is the k-th target evaluation network for drone n. Its input is The neural network parameters are . , Through reparameterization techniques: The sampling action, in which This is to ensure that the policy gradient is differentiable. It is the actor network of drone n Output policy distribution It is the reward decay factor and temperature parameter of MASAC. By adjusting the entropy term The exploration and exploitation are automatically balanced relative to the expected return.

[0062] Subsequently, the online actor network is updated by minimizing the following information-theoretic regularization objective. Parameters: in It is the kth online evaluation network for drone n. Its input is The neural network parameters are . It is a drone The neural parameters of the action network. , It is the action of UAV 1 in time slot t based on experience. The drone n uses a reparameterization technique based on experience with the time slot t state. The generated action, It refers to the actions of the drone N in time slot t based on experience.

[0063] Temperature parameters The update can be achieved using the following entropy constraint adaptive rule: in The specified target entropy level is used to control the randomness of the policy.

[0064] Finally, the parameters of the target commentator network are soft-updated using Polyak averaging: In the formula To update the coefficients, ensure that the value function propagates stably between training iterations.

[0065] Example 3 like Figures 2-4 As shown, this embodiment uses a real human movement trajectory dataset for simulation experiments. The specific implementation process is as follows: Step 1: Determine the evaluation indicators This invention employs publicly available real-world human movement trajectory datasets (such as the MDT trajectory dataset) to construct a simulation environment for training and testing a reinforcement learning-based UAV-assisted mobile edge computing optimization method. Through simulation experiments, the "reward value - number of rounds" curve (i.e., the reward curve) during model training can be obtained. This curve directly reflects the learning efficiency and final performance of the reinforcement learning agent during exploration and utilization, and is a core criterion for evaluating the merits of the optimization method.

[0066] Based on the analysis of the reward curve and combined with the actual optimization objectives of the UAV-assisted mobile edge computing system, this embodiment determines the following specific evaluation indicators. These indicators closely correspond to the optimization objectives described in the invention and are used to comprehensively and quantitatively evaluate the performance of the proposed modeling and optimization methods: Task Completion Rate / Success Rate: This metric evaluates the effectiveness of optimization methods in meeting task Quality of Service (QoS) requirements. The reward function is typically designed to penalize task failures, which is reflected in the reward curve. Task Completion Rate (TCR) is defined as the proportion of tasks whose processing latency does not exceed the maximum tolerable latency in the evaluation round out of the total number of tasks. in Let m be the delay for the mobile device m to complete its task in time slot t. The maximum allowable latency for the task of mobile device m in time slot t. For the number of training rounds, For the number of mobile devices, This represents the number of time slots.

[0067] Convergence characteristics of the reward curve: This metric directly evaluates the learning efficiency and stability of the reinforcement learning optimization method itself, including: Convergence speed: The number of training steps (or rounds) required for an agent to reach near-final stable performance from an initial random policy. Fewer steps indicate higher learning efficiency and faster discovery of a better policy.

[0068] Convergence stability: The fluctuation range of the reward curve in the later stages of training. The smaller the fluctuation, the more stable the learning process and the more reliable the strategy.

[0069] Reward value: The cumulative reward the agent can obtain during the testing phase after training. This value comprehensively reflects the overall balance performance of the optimization strategy among multiple competing objectives such as latency and energy consumption; a higher value indicates a better joint optimization effect.

[0070] Drone Flight Trajectory: This metric directly evaluates the intelligence and optimization effectiveness of drone flight paths and is a unique core evaluation dimension for drone-assisted mobile edge computing systems. The quality of the trajectory directly affects communication quality, service coverage, and flight energy consumption.

[0071] Step 2: Determine the baseline comparison algorithm To objectively and comprehensively evaluate the effectiveness and superiority of the proposed reinforcement learning-based UAV-assisted mobile edge computing joint optimization method (hereinafter referred to as "the proposed method" or "D3MASAC"), it is necessary to compare it with existing typical or related baseline algorithms under the same simulation environment and evaluation metrics. The embodiments of this invention select the following baseline algorithms for comparison. These algorithms represent different optimization strategies and complexity levels, and can fully verify the comprehensive performance of the proposed method in solving joint optimization problems: Hovering Base Station Mode D3QN: The D3QN algorithm is used for computational offloading, but the drone remains statically hovering over the center of the coverage area or its initial location. Under this strategy, drone service coverage drops sharply as the device moves, leading to inconsistent communication quality at the edge devices and potentially poor mission completion rates. This baseline is used to highlight the value of drone mobility.

[0072] Standard DQN DMASAC: The DQN algorithm is used to compute the unloading, and the MASAC algorithm is used to optimize the UAV trajectory. This baseline is used to verify the effectiveness of the improvements made in the reinforcement learning algorithm layer of this invention. It is expected that the proposed customized method should outperform the above algorithms in terms of convergence speed, stability, and final performance.

[0073] Standard Multi-Agent DDPG / TD3 D3MADDPG / D3MATD3: The computational offloading adopts the D3QN algorithm, but the standard multi-agent DDPG or TD3 algorithm framework is used to optimize the UAV trajectory. This baseline is used to verify the effectiveness of the improvements made in the reinforcement learning algorithm layer of this invention. It is expected that the proposed customized method should outperform the above algorithms in terms of convergence speed, stability, and final performance.

[0074] This invention utilizes simulation experiments with a real human movement trajectory dataset to test and refine various metrics of the mobile edge computing system and the rewards for reinforcement learning. Simulation results demonstrate that, compared to baseline algorithms, the proposed method outperforms the baseline algorithm in terms of energy efficiency and real-time task processing, effectively reducing the overall task cost of the mobile edge computing system. This provides efficient and stable computational offloading services for latency-sensitive applications in low-altitude economic scenarios, promoting the large-scale application of UAV-assisted mobile edge computing in the low-altitude economy.

[0075] This invention aims to reduce the overall latency and energy consumption of the system by testing and correcting various indicators and reinforcement learning rewards of the collaborative low-altitude UAV network-assisted mobile edge computing method and system.

[0076] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A collaborative low-altitude unmanned aerial vehicle (UAV) network-assisted mobile edge computing system, characterized in that, The system includes: N rotary-wing drones equipped with edge servers (ESs) and M mobile devices (MDs); The collection of drones is denoted as Each drone is deployed in a specific area and equipped with a power source with a maximum energy output of [missing information]. The battery, the edge service unit with computing resources The set of mobile devices is denoted as Mobile devices are capable of moving within a specific area, and each mobile device also has local computing resources. With the transmitting antenna, its transmitting power is ; The system runtime is divided into T discrete time slots, denoted as: The duration of each time slot is Seconds; within any time slot t, the system uses a time-division multiplexing mechanism to enable each mobile device to interact with the environment sequentially; Each drone has a predetermined service range and only connects to mobile devices within the service range. Mobile devices outside the service range will perform local calculations.

2. The system according to claim 1, characterized in that, Within any time slot t, the system uses a time-division multiplexing mechanism to enable each mobile device to interact with the environment sequentially, including the following methods: Assuming that each mobile device receives a computing task in each time slot, the mobile device can choose to execute the task locally or offload it to an edge server on a drone for processing. If it chooses to offload and the task is completed, the task will occupy the computing resources allocated to the device by the edge server until the end of the current time slot.

3. The system according to claim 2, characterized in that, In each time slot t, the mobile device m generates a computation task: this computation task is modeled as a triple. ,in Indicates the data size of the task. This indicates the number of CPU cycles required to process each bit of data. This represents the maximum end-to-end latency allowed for task completion, and the set of tasks generated by all mobile devices in this time slot is defined as follows: .

4. The system according to claim 3, characterized in that, The mobile device may choose to execute the task locally or offload it to an edge server on a drone for processing. If it chooses to offload and the task is completed, the task will consume the computing resources allocated to the device by the edge server until the end of the current time slot. Methods include: set up Indicates the ES number currently connected to device m; let This indicates whether the task is ultimately assigned to an ES for execution; in the initial phase of each time slot t, each mobile device MD will establish a connection with its nearest edge server ES: ； in Defined as the location of the drone. Defined as the location of the mobile device; The edge server (ES) will assess the characteristics of newly arriving tasks and its own remaining computing resources, and assist mobile devices in determining their compute offloading strategies. when When the mobile device m chooses to perform the computing task locally, the processing latency of the local computing task is expressed as: ； in In the time slot t Mobile devices m Local computing resources; when When this happens, the mobile device m offloads the computing task to the edge server via a wireless channel. When the device connects directly to and offloads to its nearest edge server, otherwise, if The mobile device m first transmits the task to its connected edge server, and then the server migrates the task to the target server for processing via a fixed-rate backhaul wireless channel. The transmission delay caused by this offloading process is expressed as: ； in For mobile devices m To the edge server Uplink data transmission rate, For the indicator function, the first term represents the uplink latency of the task's wireless transmission to the connected server; the second term characterizes the additional migration latency incurred when the task travels through the backhaul wireless channel. Indicates the nominal backhaul radio channel capacity. This indicates the instantaneous load on the backhaul radio channel that is occupied by other non-offloaded services due to network congestion. Once the unloaded computing tasks arrive at the target server, the server will allocate a certain computing frequency to the mobile device. The computational latency of this task on the edge server is expressed as: ； In edge computing mode, the total latency generated by a mobile device processing its computing tasks is expressed as: 。 5. The system according to claim 1, characterized in that, The propulsion power energy consumption model for the rotorcraft UAV in the t-th time slot is as follows: ； in, and , where are constant coefficients, representing the blade profile power and inductive power of the UAV in hovering state, respectively; This indicates the tip velocity of the drone's rotor blades; It is the average rotor speed when the drone is hovering; and These are the drone's fuselage drag ratio and rotor solidity, respectively. and These represent air density and the area of ​​the drone's rotor disk, respectively. This refers to the flight speed of the drone.

6. An optimization method for a collaborative low-altitude unmanned aerial vehicle (UAV) network-assisted mobile edge computing system, the optimization method being used to optimize the system according to any one of claims 1-5, characterized in that, The optimization method includes: The task cost of the UAV network-assisted mobile edge computing system is modeled as a weighted sum of task latency and energy consumption, and the weighted sum of task latency and energy consumption is transformed into a joint optimization problem of computation offloading and UAV trajectory. The joint optimization problem of computational unloading and UAV trajectory is modeled as a Markov decision process; The deep reinforcement learning framework D3MASAC, which integrates the dual deep Q-network D3QN and the multi-agent flexible action-evaluation algorithm MASAC, solves the Markov decision process, achieving collaborative computational offloading and UAV trajectory optimization.

7. The optimization method according to claim 6, characterized in that, Methods for modeling the task cost of UAV network-assisted mobile edge computing systems as a weighted sum of task latency and energy consumption include: The task costs of local computing and edge computing are expressed as follows: ； in and These represent the weighting coefficients for task delay costs and energy consumption costs, respectively. This represents the propulsion power energy consumption of a rotary-wing UAV in the t-th time slot. M This represents the number of mobile devices, and N represents the number of rotary-wing drones equipped with edge servers. The task completion delay of mobile device m in the t-th time slot is expressed as: ； in, Indicates the local computing task processing latency. This represents the total latency incurred by a mobile device in processing its computing tasks.

8. The optimization method according to claim 7, characterized in that, Markov decision processes include a state space, an action space, and a reward function. The state space is as follows: In time slot t, the system state corresponding to mobile device m is defined as follows: ； in This represents the index of the system state corresponding to mobile device m at time slot t. This state vector contains the task characteristics of the mobile device itself. Index of currently connected edge servers The location of mobile device m and the position vectors of all drones The remaining CPU computing frequency resources The remaining battery energy vector ; The system state corresponding to drone n is defined as follows: ； This state vector contains the task feature vectors of all mobile devices. The index vector of all edge servers connected to mobile devices Move the location of all mobile devices and the position vectors of all drones The remaining battery energy vector ; Action space: The action performed by the calculation unloading module is defined as follows: ； That is, to calculate the unloading strategy; In the drone trajectory optimization module, each drone agent has independent actions, defined as follows: ； in This represents the sequence number of the system state corresponding to UAV n in time slot t, including the UAV's own velocity in the x-direction. and velocity in the y direction ; The motion vectors of all drones are defined as follows: ； in Let x be the velocity of the drone N in the x-direction and y-direction. Reward function: The calculation unloading module is in state Next action The reward function obtained is defined as follows: ； in, The maximum possible latency of a task is calculated based on task attributes. Furthermore, a penalty is applied if the latency constraint is violated. ; Each drone agent in the drone trajectory optimization module is in a state Next action The reward function obtained is defined as follows: ； This includes the sum of rewards from all mobile device tasks, as well as the negative cost of the drone's own flight energy consumption; The reward vector for all drones is defined as follows: ； in, Let be the reward function for UAV 1 in time slot t. Let be the reward function for drone N in time slot t.

9. The optimization method according to claim 6, characterized in that, The deep reinforcement learning framework D3MASAC, based on the fusion duel dual deep Q network D3QN and the multi-agent flexible action-evaluation algorithm MASAC, includes: D3QN module and MASAC module. The D3QN module is used to generate efficient computation offloading decisions for mobile device MDs through iterative optimization, taking the current system state as input. The MASAC module is used to integrate status information and environmental feedback to output flight control actions for each UAV. The D3QN module employs two neural networks: an online network. The parameters are Used for immediate decision-making; target network The parameter is for delayed synchronization. , used for stable value estimation; Each agent in the MASAC module employs an actuator-evaluator based architecture, which includes an online actuator network. The parameters are This is used to generate probability distribution-based drone flight strategies, and two evaluator networks. The parameters are respectively This is used to estimate the Q-value, supplemented by a target evaluator network with two delays. The parameters are .

Images