Edge air-ground network optimization method and system based on layered deep reinforcement learning

Abstract

The invention belongs to the technical field of wireless communication, and discloses a marginal air-ground network optimization method and system based on hierarchical deep reinforcement learning (joint optimization method and system), and the optimization of the marginal air-ground network is realized through joint optimization of a service placement strategy, UAV trajectory planning, access control, bandwidth allocation and transmission power. And efficient equipment connection and data processing services are provided for disaster area rescue, urban intelligent monitoring and other scenes. According to the method, a dual-time-scale optimization strategy is adopted: on a coarse-grained frame-level time scale, a high-level agent adopts a deep Q network (DQN) proxy, and a placement strategy of service in an MEC server is decided according to a UAV task request to ensure that resource constraints are met; on the fine-grained time slot level time scale, a low-layer agent adopts an improved depth deterministic policy gradient (IDDPG) for proxy, the UAV trajectory, TDMA-based access control, bandwidth allocation proportion and transmission power are optimized, and the real-time task requirement is met. Through the layered framework, the system can improve the task success rate, the energy efficiency and the resource scheduling fairness in a dynamic environment, and efficient and stable air-ground integrated network service is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of wireless communication, and particularly relates to an edge air-ground network optimization method and system based on hierarchical deep reinforcement learning. BACKGROUND

[0002] With the rapid development of wireless communication and Internet of Things technology, intelligent applications such as live streaming and high-density data backhaul have emerged in large-scale competition scenarios, and the requirements for computing power and response speed are increasing. Traditional ground base stations face problems such as coverage blind area and heavy data processing pressure when dealing with sudden traffic, while unmanned aerial vehicles (UAVs) can be used as mobile data collection nodes and base stations to cooperatively build an air-ground integrated network due to their flexible deployment and rapid response capabilities. Multi-access edge computing (MEC) can effectively reduce data transmission latency by setting edge nodes at the base station side and moving data processing capabilities to the network edge (such as 5G MEC controlling video backhaul latency within 15 milliseconds), and can achieve three-dimensional coverage in the air space by dynamically filling in blind spots with UAVs. Combining MEC with UAV networks can not only enhance the reliability and flexibility of event communication, but also improve network performance through dynamic resource scheduling between base stations and UAVs.

[0003] However, when applying hierarchical deep reinforcement learning (DRL) to MEC-driven air-ground integrated networks, there are still many key problems. First, in large-scale competition scenarios, the flight trajectory of the UAV, the load change of the base station, and the fluctuation of the wireless channel interact with each other, resulting in complex and variable network topology, and traditional optimization methods cannot adjust the resource allocation strategy in real time. Second, MEC server deployment needs to balance quality of service and cost: it needs to meet the needs of high-latency sensitive services such as live streaming, and avoid excessive deployment to waste resources. In addition, hierarchical DRL algorithms need to be customized for competition scenarios, such as integrating multi-source perception data for UAV trajectory optimization, addressing the problem of policy generalization in noisy observations and dynamic environments, and improving the convergence speed of existing algorithms to adapt to real-time decision-making. More importantly, in the presence of multiple UAVs, resource allocation needs to consider the service capability differences between base stations and UAVs to avoid resource tilt leading to local service quality degradation, so the algorithm design needs to add fairness constraints (such as constructing a fairness-aware reward function to balance resource allocation ratio). Currently, how to balance algorithm complexity and real-time performance, and implement joint resource optimization of base stations and UAVs, is still a core problem to be solved.

[0004] 1) Architecture fusion problem of air-ground integrated network

[0005] In Air-Ground Integrated Networks (AGINs), the complexity of network architecture and the integration requirement of mobile edge computing bring obvious challenges. The network contains multiple unmanned aerial vehicles (UAVs) and ground base stations. Due to the limitations of on-board energy and computing capacity, UAVs cannot independently handle computationally intensive or delay-sensitive tasks and need to rely on ground base stations to offload tasks to meet application requirements.

[0006] When MEC is integrated into ground base stations, although the stable computing power of base stations can alleviate the resource pressure of UAVs, it also raises new system-level optimization problems. In a dynamic task continuous access environment and a wireless channel frequently fluctuating with the movement of UAVs, achieving robust and efficient global decision-making becomes a core bottleneck. In addition, the high mobility of UAVs leads to frequent changes in network topology, making it difficult to guarantee the stability of communication links with ground base stations, and signals are easily attenuated due to distance, obstruction, etc.

[0007] 2) Data transmission and coordination in air-ground integrated networks

[0008] At the data transmission level, the communication quality between UAVs and ground base stations is directly affected by the distance. The channel power gain changes dynamically with the position of the UAV, resulting in poor stability of the transmission rate. At the same time, the total bandwidth of the base station needs to be allocated between uplink and downlink transmission. The uplink is used for UAVs to offload task data, and the downlink is used for feedback processing results. The bandwidth allocation ratio and transmission power are both limited by the hardware capabilities of the devices. These factors together make it difficult to guarantee the real-time and reliability of data transmission, and transmission delay exceeding the threshold or data loss may occur.

[0009] In terms of synchronization, system time is strictly divided into coarse-grained "frames" and fine-grained "slots". Frame-level task deployment strategies (such as which tasks are deployed to MEC servers) and slot-level real-time transmission (such as when UAVs access and how to transmit) need to be strictly time-aligned. Once the synchronization mechanism fails, it will cause the task placement and execution rhythm to be out of sync, for example, a task is deployed to the base station at the frame level, but due to the non-timely scheduling of the slot-level transmission, it ultimately cannot be completed within the delay constraint.

[0010] In terms of computing resource coordination, the processing efficiency of tasks depends on the computing power of the base station MEC server, but the characteristics of tasks (such as input data volume, processing density, and output data volume) vary significantly, and there is a natural contradiction between the matching of base station computing power and task characteristics - some tasks may have too much data, resulting in too long transmission time, while some tasks may have too high processing density, exceeding the real-time computing power of the base station. More complex is that transmission resources (bandwidth, power) and computing resources (base station computing power) are strongly coupled and constrained, for example, insufficient transmission rate will cause task data to accumulate, thereby occupying more computing resources, forming a vicious cycle, making it difficult for the two to coordinate efficiently.

[0011] 3) Network latency problem

[0012] In the scenario of air-ground integrated network enabled MEC, the time sensitivity of tasks varies significantly, which brings challenges to optimization. At the frame level, the optimization of task placement needs to consider long-term performance and reasonably allocate tasks to computing nodes to improve overall efficiency, but the decision-making period is long and it is difficult to respond to instantaneous changes. At the time slot level, the optimization of UAV deployment, flight trajectory, access control and task offloading rate needs to be adjusted in real time, which directly depends on the current system state (such as channel conditions, task arrival), and the decision-making period is short and requires fast response to meet real-time requirements. There is a conflict between the decision-making goals and constraints of different time scales: frame-level optimization may cause task delay due to not considering time-slot-level real-time state, while time-slot-level immediate adjustment may also damage the global performance of frame-level, how to balance the needs of different time sensitivities becomes a prominent problem.

[0013] 4) Network high dynamic problem

[0014] The MEC system based on air-ground integrated network faces a highly dynamic environment, making it difficult to solve the optimization problem. User task requests are random and cannot be predicted in advance, while wireless channel conditions are strongly dynamic due to factors such as UAV movement and obstruction. In addition, variables in the system are subject to their own constraints and are coupled with each other: for example, the flight trajectory of the UAV will affect the channel gain, and then be associated with the allocation of transmission power and bandwidth, the strong correlation between variables greatly increases the optimization difficulty. Therefore, traditional optimization algorithms are difficult to adapt to this highly dynamic environment, and existing deep reinforcement learning (DRL) researches are mostly limited to single-objective optimization and do not fully address the dynamic complexity of AGINs, resulting in deficiencies in multi-objective balancing and long-term performance optimization. SUMMARY

[0015] In view of the problems existing in the prior art, the present application provides a hierarchical deep reinforcement learning driven edge air-ground network optimization method and system, which focuses on the joint optimization of collaborative service placement, UAV deployment, UAV flight time, access control, bandwidth allocation and transmission power. The implementation scheme is that the method and system rely on hierarchical deep reinforcement learning, combine master-slave double-layer deep reinforcement learning network, complete the joint optimization of key parameters such as task placement, UAV trajectory planning, access control, bandwidth allocation and transmission power in air-ground integrated network, and the process includes the following steps:

[0016] S101, system initialization is performed, first the initial value setting of the core network parameters is completed: including the master Q network parameter ω, the target Q network parameter ω - , the Actor network, the Critic parameter θ, w and θ of the master and target network - = θ and w -=w, then configure the number of iterations L, the number of iterations per iteration (i.e., the number of frames K), the number of time slots per frame T, set the learning rate α1 for the DQN network, the learning rates α2 and β2 for the Critic and Actor networks, and the discount factor γ, and then determine the replay buffer size D. hi D l The mini-batch sizes M1 and M2, and the OU noise Ξ(t) for action exploration; in addition, network layout parameters, such as the number of UAVs I and other task parameters, need to be initialized.

[0017] S102, At the start of the cycle, initialize the state s of the high-level DQN agent. hi (k); Selecting a high-level action a using the ∈-greedy strategy. hi (k) Generate frame-level service placement strategy;

[0018] S103, the agent executes an action, generating temporary high-level rewards according to constraints. If the constraints are satisfied, then In each time slot within a frame, the IDPG agent performs low-level task optimization; if the condition is not met, the value is 0 and low-level optimization is not initiated.

[0019] S104, Initialize the state of the low-level IDPG agent s l (t), after probability sampling, is converted into a continuous output and then used for binary access control to execute actions. Obtain time-slot-level UAV trajectory, access control, uplink bandwidth allocation ratio, and transmission power;

[0020] S105, the agent performs an action and receives a reward. l (t), the lower-level state is updated to s l (t+1);

[0021] S106, Current experience is stored in replay buffer D l If the buffer is full, the oldest experience is discarded; from Randomly select the lower-level experience of M2 ( (for small batch sets);

[0022] S107, calculate the TD objective and error of the low-level IDPG; update the Actor network and Critic network, and synchronously update the target Actor network and Critic network;

[0023] S108 updates the parameters of the Actor and Critic networks based on small-batch experience, and updates the target network parameters simultaneously.

[0024] S109, in the last time slot of frame, after IDPG receives its reward, the higher-level DQN receives its reward r. hi(k), and update the state to the next state s hi (k+1);

[0025] S110, experience storage into D hi ; if the buffer is full, discard the oldest experience. Sample a small batch of experiences M1, input the state-action into the DQN main network, estimate the Q value;

[0026] S111, calculate the TD target and error of the high-level DQN, update the main and target DQN networks, and synchronize the target Q network parameters;

[0027] S112, repeat the iterative training, select the optimal action according to the state-action Q value, maximize the reward, and output the optimal task placement, UAV trajectory, access control, bandwidth allocation and transmission power.

[0028] Further, at the beginning of each cycle of step S102, the system state s hi of the high-level DQN agent is initialized; the state is specifically described as:

[0029] s hi (k) = {r(k)},

[0030] where r(k) = {r i,h (k)}, r i,h (k) ∈ {0, 1}, characterizing the task request of the kth frame of UAVs; an e-greedy strategy is used to select the high-level agent DQN action a hi (k), so as to obtain the service placement strategy of the frame, which can be represented as:

[0031] a hi (k) = {Δ(k)},

[0032] where Δ(k) = {δ h (k)}, δ h (k), is the service placement decision.

[0033] Further, the temporary reward reward formula in step S103 is:

[0034]

[0035] where (C1) is δ h (k) ∈ {0, 1}, represents the binary constraint of the service placement strategy, and (C2) is represents the capacity constraint of the service setting.

[0036] The high-level temporary reward is designed to determine whether it is necessary to carry out the low-level optimization. When , it indicates that the service placement meets the (C1)-(C2) constraints, that is, the placement is feasible, and at this time, the low-level optimization is carried out with the help of the IDDPG agent; when , it indicates that the task placement is not feasible, and at this time, the low-level optimization has no practical value, because the high-level solution and the low-level solution together constitute a solution to the problem , and if the high-level solution is not feasible, regardless of the state of the low-level solution, the overall solution cannot be established.

[0037] Further, in step S104, the state s l (t) of the low-level IDDPG agent is initialized, which is represented as follows:

[0038]

[0039] where Δ(k)={δ h (k)},δ h (k), represents the service placement decision, indicating whether the task h has been deployed on the base station at the start of the time slot t; u(t)={u i,x (t),u i,y (t)} is the horizontal position of the UAV i at the start of the time slot t; is used to indicate whether there is an obstruction between the UAV i and the BS; R up (t) is the uplink transmission rate between the UAV i and the BS. represents the backhaul rate between the UAV i and the base station; the state space of the low-level IDDPG agent is denoted as which is the set of all low-level states.

[0040] Through probabilistic random sampling, the continuous output value is converted into binary access control, and the IDDPG agent performs the action The action is defined as:

[0041] a l (t)={d(t),A(t),γ(t),p up (t)},

[0042] where d(t)={v i (t),θ i (t)}, is the UAV deployment strategy; A(t)={a i (t)}, where a i (t) is the access control strategy of the UAV i, γ(t), The bandwidth offloading ratio of the UAV is identified as p up (t), is the uplink transmission rate of the UAV. The action space of the IDDPG agent is represented as is the set of all feasible solutions of the task processing related variables.

[0043] Further, in step S105, the agent obtains a reward r l (t) after performing the generated action, and the low-level state transitions to the next state s l (t+1); the calculation formula of the reward reward is as follows:

[0044]

[0045] where (C3) v i (t) ∈ [0, v max ], represents the flight speed constraint of the UAV, (C4) θ i (t) ∈ [0, 2π], identifies the flight angle constraint of the UAV; a i (t) ∈ {0, 1}, is the (C5) access control decision of the base station to access the UAV i; is (C6) used to ensure that only one UAV accesses the base station in each time slot; γ i (t) ∈ [0, 1], is (C7) to represent the bandwidth allocation ratio constraint of the UAV; (C8) is represents the transmission power of UAV i and the base station. The first term in the formula refers to the fairness coefficient of the UAV when the base station accesses the UAV i; the second term refers to the final success rate of the UAV i transmitting to the base station for processing, and the third term refers to the total energy consumption of the UAV.

[0046] Further, in step S107, the main actor (Actor) network prediction is:

[0047]

[0048] The target actor (Actor) network prediction follows:

[0049]

[0050] The main critic (Critic) network prediction is:

[0051]

[0052] The target critic network prediction is:

[0053]

[0054] The TD target calculation formula is:

[0055]

[0056] The TD error derivation is:

[0057]

[0058] The Critic network is updated according to the following formula:

[0059]

[0060] Meanwhile, the Actor network is updated according to the following formula:

[0061]

[0062] Further, for step S108, according to the updated main Actor network and Critic network parameters, the formula for updating the target Actor network and Critic network parameters is as follows:

[0063] θ - ← ζ·θ + (1- ζ)·θ - ,

[0064] w - ← ζ·w + (1- ζ)·w - ,

[0065] where θ is the current policy network parameter, θ - is the target policy network parameter, w is the current value network parameter, w - is the target value network parameter, and ζ ∈ [0, 1] is a hyperparameter that needs to be manually adjusted.

[0066] Further, the calculation formula of the high-level reward in step S109 is as follows:

[0067] r hi (k) = λ1α1χ(t, k) + λ2α2r suc (t, k) - λ3α3E(t),

[0068] i.e. the objective function of the present application.

[0069] Further, the process of step S110 is as follows: store the current experience into the experience replay buffer D hi ; if the buffer is full, delete the oldest experience to make room; extract a small batch of experiences M1 from the buffer, input the state and action into the DQN main network, and obtain the estimated Q value as follows:

[0070] q hi (k)m = Q(s hi (k) m , a hi (k) m ; ω),

[0071] wherein ω is the main Q network weight;

[0072] s high (k+1) m is input into the target network, and the following is obtained:

[0073]

[0074] wherein ω - is the target Q network weight.

[0075] Further, step S111 calculates the generated TD target according to the following formula:

[0076]

[0077] and uses the calculated TD error:

[0078]

[0079] to update the main DQN network:

[0080]

[0081] wherein α1 is the learning rate; and the target DQN network is updated according to the following formula:

[0082]

[0083] wherein, is a manually adjusted hyperparameter.

[0084] Another object of the present application is to provide an edge-space network optimization method and system based on hierarchical deep reinforcement learning, which comprises:

[0085] a system initialization module, which needs to complete the setting of the deep Q network algorithm parameters, involves the main Q network parameter ω, the target Q network parameter ω - ; and simultaneously initializes the deep deterministic policy gradient algorithm parameters, including the main Actor network and Critic network parameters θ, w, and the target Actor network and Critic network parameters θ - = θ and w -w; in addition, the number of iterations L, the number of iterations (i.e., the number of frames K) of each iteration, the number of time slots T in each frame, and the DQN network learning rate a1, the learning rates a2 and b2 of the Critic network and the Actor network corresponding to the Critic network, the discount factor g, the replay buffer size D hi and D l , the size of the mini-batch M1 and M2, and the OU noise of action exploration g(t);

[0086] The configuration module is configured to set application-specific parameters, mainly including the number of unmanned aerial vehicles and other configuration items related to the task.

[0087] The agent module generates actions according to the current high-level network state at the beginning of each cycle, and selects the actions using a greedy strategy; at the same time, the agent module generates actions according to the low-level network state, and introduces exploration noise into the actions to enhance exploration.

[0088] The action execution module is responsible for implementing service placement decisions, as well as unmanned aerial vehicle deployment strategies, access control strategies, bandwidth allocation ratios, and uplink transmission power settings.

[0089] The reward acquisition module has the following functions: calculating temporary rewards according to system constraints after executing actions, and determining whether to start low-level optimization based on the temporary rewards; at the same time, calculating the benefits of system devices at the current time slot and at the end of the frame based on the current state and the executed actions, and generating corresponding rewards.

[0090] The state transition module is responsible for migrating the low-level system state from the current time to the next time.

[0091] The experience replay module is used to store low-level experience tuples, including states, actions, rewards, and next states.

[0092] The data sampling module samples small batches of samples from the experience replay library for model training.

[0093] The network update module updates the main Q network, the main Actor network, and the main Critic network using the sampled data, which includes a unit that optimizes network parameters using a gradient descent algorithm.

[0094] The parameter update module updates the parameters of all target networks in a soft update manner based on the parameters of the main network, and the core operation is to adjust the parameters of the target network using weighted averaging to gradually approximate the main network.

[0095] In combination with the above technical solutions and the technical problems solved, the technical solutions protected by the present application have the following advantages and positive effects:

[0096] The application has the following advantages in the integration of mobile edge computing (MEC) and air-ground integrated networks (AGINs) based on the characteristics of base stations as the only task processing nodes:

[0097] First, in terms of efficient processing and dynamic adaptation, the application integrates MEC into ground base stations, and all tasks are processed by the MEC server of the base station. The drone is only responsible for data collection and task offloading, without the need to bear the computing load. This architecture greatly improves the efficiency of task processing through the cooperation of drones and base stations, especially for delay-sensitive tasks. After offloading data to the base station, the drone can quickly complete the processing with the help of the stable computing power of the base station, avoiding the delay caused by the limited on-board resources of the drone. At the same time, the system uses a Time Division Multiple Access (TDMA) mechanism. The drone dynamically adjusts the communication strategy with the base station in different time slots, and flexibly adapts the transmission parameters according to the channel conditions, so that the system can quickly respond to environmental changes caused by the movement of the drone and ensure service continuity.

[0098] Second, in terms of multi-time scale optimization and resource management, the application optimizes by time layering, fully utilizing the advantages of centralized processing by base stations. At the frame level, the placement strategy of tasks in the base station is optimized to ensure the matching of base station computing power and drone task demand in the long term, avoiding resource idling. At the time slot level, the trajectory, access control, bandwidth allocation, and transmission power of the drone are adjusted in real time to make the data transmission efficiently adapt to the processing capacity of the base station. This hierarchical optimization strategy not only ensures the long-term rational allocation of base station resources, but also meets short-term transmission needs through immediate adjustments, maximizing resource utilization.

[0099] In addition, in terms of algorithm innovation and problem solving ability, the hierarchical deep reinforcement learning (DRL) framework designed by the application is specifically adapted to the "drone-base station" architecture: the high-level DQN handles the placement strategy of frame-level tasks in the base station, ensuring that the base station computing power matches the drone task demand; the low-level IDDPG adjusts the trajectory, access control, and transmission parameters of the time slot level drone to achieve efficient data transmission to the base station. Since all tasks are processed by the base station, the algorithm does not need to consider user-side variables and focuses on the cooperation of drones and base stations, greatly reducing the difficulty of solving. At the same time, the IDDPG algorithm optimizes around the constraints of base station processing (such as computing power limit, transmission resource constraint), ensuring efficient solving in dynamic environments and improving the task completion rate, energy utilization efficiency, and scheduling fairness of the system.

[0100] In summary, the application realizes significant improvement in task processing efficiency, dynamic adaptability and resource utilization by focusing on "drone-base station" cooperation, taking base station as the only processing core, combining multi-time scale optimization and hierarchical DRL algorithm, and provides a targeted solution for MEC enabling air-ground integrated network.

[0101] Secondly, the application has achieved significant technical progress in implementing an edge air-ground network optimization method and system based on hierarchical deep reinforcement learning, which has made significant progress in the following key aspects:

[0102] 1) Efficient service placement strategy

[0103] The high-level DQN agent dominates the service placement decision at the frame level, and the core is to comprehensively consider multi-dimensional factors, including the task request type, data volume and processing demand of each drone, the real-time storage capacity and computing load of the ground base station MEC server, and the dynamic change trend of the overall network. Through long-term interaction with the environment, the DQN agent can accurately evaluate the deployment value of different tasks in the base station, strictly follow the storage constraints of the MEC server, and avoid the blindness of task placement. At the same time, by dynamically adjusting the task deployment scheme, the edge computing resources are more evenly utilized, reducing the task backlog or idling caused by resource mismatch, and improving the overall operation efficiency of the system from a global perspective.

[0104] 2) Precise drone trajectory optimization scheme

[0105] The low-level IDDPG agent focuses on the time slot level drone trajectory optimization, and the decision process is closely related to the relative position of the drone and the ground base station, the real-time channel condition and the task processing demand. The agent continuously adjusts the flight speed and direction of the drone to dynamically optimize its horizontal coordinate change, thereby shortening the distance to the base station, reducing the signal blocking effect, and thus improving the channel power gain. At the same time, combined with the task queue length and transmission delay constraints, the flight rhythm is reasonably controlled to ensure that the task real-time is met, and the flight energy consumption is minimized. This fine-grained trajectory control ensures the efficiency of data transmission, laying a good foundation for subsequent access control, bandwidth allocation and other operations.

[0106] 3) Dynamic drone access control strategy

[0107] Based on the time division multiple access mechanism, the IDDPG agent realizes dynamic unmanned aerial vehicle access control decision in each time slot. The core logic is to select the optimal unmanned aerial vehicle from multiple unmanned aerial vehicles to access the base station according to the real-time system state, including the task priority of each unmanned aerial vehicle, the channel quality, the current task queue length and the like, to ensure that there is only one uplink transmission link in a single time slot, and completely avoid signal interference caused by multiple unmanned aerial vehicles accessing at the same time. This dynamic adjustment mechanism enables the access control to closely adapt to channel fluctuations and task demand changes, significantly enhances the stability of network communication, reduces transmission failures or retransmissions caused by interference, and provides key support for efficient task offloading.

[0108] 4) Innovative integration of deep reinforcement learning

[0109] The present application innovatively constructs a hierarchical DRL framework, realizing the deep integration of deep reinforcement learning and air-ground integrated network optimization. The high-level DQN agent focuses on service placement decisions at the frame level, learns for slow-changing environments, and outputs stable and globally optimal service placement schemes by evaluating the long-term value of different deployment strategies. The low-level IDDPG agent targets time slot-level fast-changing dynamics and breaks through the limitation of traditional DDPG, which is only applicable to continuous action space, by introducing a discrete action parameterization mechanism, enabling it to handle both continuous variables (such as transmission power and bandwidth allocation ratio) and discrete variables (such as binary decisions for access control). This hierarchical design enables the DRL algorithm to adapt to the mixed action characteristics of the system and meet the optimization needs of different time scales, achieving continuous optimization of long-term performance in dynamic environments through collaborative learning of master-slave networks.

[0110] 5) System stability and reliability improvement

[0111] The constraint-aware reward mechanism is introduced, which gives zero reward to actions that violate constraints, ensuring the feasibility of high-level service placement and low-level control decisions, reducing invalid operations, and enhancing system stability in high-concurrency task processing scenarios.

[0112] 6) Network autonomous learning and optimization capability

[0113] Through iterative training and experience accumulation, the master-slave network parameters are continuously updated, enabling the system to autonomously optimize the decision-making process based on real-time state, gradually improving task success rate, energy efficiency and scheduling fairness.

[0114] 7) Breakthrough of joint optimization bottleneck

[0115] To address the resource waste caused by separate optimization of service placement, unmanned aerial vehicle control and other parameters in existing technologies, the hierarchical DRL realizes joint optimization of service placement, unmanned aerial vehicle trajectory, access control, bandwidth allocation and transmission power, improving overall resource utilization efficiency.

[0116] 8) Coping with environmental complexity and dynamics

[0117] Traditional optimization methods are limited in effect due to the difficulty in adapting to the high dynamic characteristics of integrated air-ground networks, such as random task requests, time-varying channel conditions, and topology changes caused by unmanned aerial vehicle movement. The present invention effectively copes with this challenge through the strong adaptability of DRL. The DRL agent does not need to pre-set an environmental model, but dynamically adjusts the decision-making strategy by real-time perception of system states such as task request distribution, channel gain, and unmanned aerial vehicle position. This closed-loop mechanism of perception-decision-feedback enables the system to flexibly cope with various emergencies and avoids the adaptation lag caused by model hardening, maintaining high efficiency in complex and dynamic environments.

[0118] 9) Improve real-time performance and optimization efficiency

[0119] Existing optimization methods often have high computational complexity and long decision-making periods, making it difficult to meet the real-time requirements of air-ground networks. The present invention achieves a balance between efficiency and real-time performance through a hierarchical optimization strategy. The high-level DQN makes service placement decisions at the frame level, reducing the computational burden by reducing the frequency of global optimization. The low-level IDDPG makes real-time control at the time slot level to respond to instantaneous changes with finer time granularity, ensuring that the decision-making delay is controlled within the time slot duration. This mode of coarse-grained planning and fine-grained adjustment avoids the high-frequency computational pressure of global optimization while meeting real-time requirements through rapid response at the time slot level, enabling the system to quickly output optimization strategies when handling urgent tasks or responding to channel mutations, significantly improving response speed and optimization efficiency in dynamic scenarios.

[0120] 10) Innovative reward mechanism design

[0121] The present invention designs a hierarchical reward mechanism that significantly improves the relevance and efficiency of the optimization process through the synergy of high-level temporary rewards and low-level constraint-aware rewards. After generating a service placement strategy at the beginning of the frame, the high-level DQN determines whether it meets the constraints through temporary rewards, triggering the optimization process of the low-level IDDPG only when the strategy is feasible, avoiding inefficient computation at the low level due to invalid high-level decisions and saving a large amount of computing resources. The low-level IDDPG outputs corresponding rewards based on whether the actions meet the constraints, guiding the agent to focus on the feasible solution space for learning and reducing ineffective exploration. This mechanism enables the optimization process to accurately focus on effective decisions, significantly improving learning efficiency and convergence speed.

[0122] Third, the edge air-ground network optimization method and system based on hierarchical deep reinforcement learning provided by the present invention, the core of which is to use mathematical models as a key support to guide system behavior decision-making and learning evolution. To understand its technical level of action, we can start from the characteristics of these mathematical models and carry out targeted effect analysis:

[0123] 1) Calculation of rewards

[0124] The calculation of temporary rewards, low-level rewards and high-level rewards is related to system constraint conditions and core performance indicators. Among them, the high-level temporary reward is mainly used to determine whether the service deployment meets the constraint requirements; the low-level reward and the high-level final reward comprehensively consider the scheduling fairness, task completion rate and energy consumption, and with the help of the constraint perception mechanism, the feasibility and running efficiency of the decision-making process are guaranteed.

[0125] 2) Update of main Q network, main Actor network and main Critic network

[0126] By randomly extracting small batches of experience data to carry out network iteration, the gradient ascent algorithm is used to adjust the parameters of the current policy network. This process not only helps the system to learn a better decision-making strategy, but also enhances the system's dynamic adaptability to environmental changes and improves the overall response speed by using small batches of data.

[0127] 3) TD target and Critic network loss function

[0128] The calculation of the loss function is based on the time difference target and the Q value predicted by the Critic network, and the network update is completed through the gradient descent algorithm. The introduction of the TD target can balance the system learning process and reduce the instability caused by large prediction errors; and the accurate loss function calculation further optimizes the accuracy of system decision-making.

[0129] 4) Parameter update formula

[0130] The main Q network, the main Actor network and the Critic network realize policy optimization by gradually iterating the target network parameters. This gradual parameter iteration mechanism can avoid sharp changes in policy, ensure stable output of system performance, and support the system to continuously adapt to dynamic changes in the environment in the long-term learning process.

[0131] The mathematical model constructed in the application effectively improves the operation efficiency and decision-making level of the edge computing system in practical application, and strengthens the adaptability and long-term stability of the system in dynamic environment. For modern edge computing technology dealing with massive data and high-frequency interaction scenarios, it has important practical significance.

[0132] Fourthly, as the creative auxiliary evidence of the claims of the application, the expected income and commercial value after the transformation of the technical scheme are:

[0133] The layered deep reinforcement learning framework constructed by the application can achieve joint optimization of the air-ground integrated network under the MEC driving by means of the high-layer DQN bearing frame-level service deployment decision and the low-layer IDDPG responsible for time slot-level UAV control, thereby effectively improving the task success rate, energy efficiency and scheduling fairness. The framework has outstanding application potential in disaster rescue, intelligent monitoring and other scenes, and can also be extended to the fields of intelligent transportation and UAV distribution, thereby reducing the cost, improving the service quality and response efficiency, and having both economic and social benefits, and has a good market prospect. BRIEF DESCRIPTION OF DRAWINGS

[0134] In order to more accurately illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiments of the application will be briefly introduced as follows. Obviously, the drawings described below are only some embodiments of the application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0135] Figure 1 is a flowchart of an edge-air network optimization method based on layered deep reinforcement learning provided by the embodiments of the application.

[0136] Figure 2 is a scene diagram that can be applied by the embodiments of the application.

[0137] Figure 3 is an implementation flowchart of the edge-air network optimization method based on layered deep reinforcement learning provided by the embodiments of the application.

[0138] Figure 4 is a system structure diagram of the edge-air network optimization method based on layered deep reinforcement learning provided by the embodiments of the application.

[0139] Figure 5 is a comprehensive convergence performance comparison diagram provided by the embodiments of the application. DETAILED DESCRIPTION

[0140] In order to make the purpose, technical solutions and advantages of the application more clear and understandable, the application will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0141] The core innovation of the application is to construct a hierarchical deep reinforcement learning framework based on multi-time scale, aiming to optimize the mobile edge computing (MEC) driven air-ground integrated network. The framework adopts a double-layer architecture: the high-level deep Q network (DQN) agent undertakes global service deployment decision at the frame level, and determines the deployment scheme of the task on the ground base station MEC server; the low-level improved deep deterministic policy gradient (IDDPG) agent operates at the time slot level, breaks the limitation of traditional DDPG only applicable to continuous action space by introducing a discrete action parameterization mechanism, and is responsible for handling fine-grained operations such as UAV trajectory planning, access control, bandwidth allocation and transmission power adjustment. This hierarchical method can effectively adapt to the decision-making needs of different time scales in dynamic network environment, and combined with the constraint-aware reward mechanism, it not only strengthens the learning stability, but also guarantees the decision-making feasibility. Through joint optimization of service deployment, UAV control and resource allocation, the system's task success rate, energy efficiency and scheduling fairness are effectively improved, providing an efficient optimization scheme for the MEC driven air-ground integrated network.

[0142] Embodiment 1: Disaster rescue environment monitoring system

[0143] In the disaster rescue scenarios such as earthquake and flood, the MEC driven air-ground integrated network based on multi-time scale hierarchical DRL can efficiently support environment monitoring and data processing:

[0144] S101: System initialization Deploy multiple UAVs for disaster area environment perception, ground base stations (including MEC servers) are fixedly deployed in safe areas, time is divided into frames and time slots, and TDMA mechanism is adopted to avoid interference between UAVs.

[0145] S102: The high-level DQN agent decides the service placement strategy (such as priority deployment of earthquake aftershock monitoring, trapped personnel positioning, etc.) at the beginning of each frame according to the initial disaster data collected by the UAV, to ensure that key tasks are processed first.

[0146] S103-S111: The low-level IDDPG agent optimizes the UAV trajectory (adjusts the position to avoid obstacles and blind areas), access control (allocates time slots according to task urgency), bandwidth and power allocation in each time slot, to accelerate the unloading and processing of disaster data to the base station.

[0147] S112: Through iterative optimization, the system continuously adapts to the dynamic changes in the disaster area (such as signal obstruction, task addition), improves the data processing success rate and UAV endurance, and provides real-time information support for rescue command.

[0148] Embodiment 2: City intelligent monitoring system

[0149] For the intelligent monitoring scene of urban traffic hubs and key areas, this optimization method can improve the monitoring data processing efficiency and coverage flexibility:

[0150] S101: The system deploys multiple UAVs to cover key urban areas (such as commercial districts and traffic intersections), and the ground base station is responsible for centralized processing of monitoring data, initializing UAV position, communication bandwidth, etc. Parameters, time is managed in frame-time slot hierarchy.

[0151] S102: High-level DQN agents decide service placement according to monitoring requirements (such as traffic statistics and abnormal behavior identification) at the beginning of each frame, ensuring the allocation of computing resources for high-priority tasks (such as accident monitoring).

[0152] S103-S111: Low-level IDDPG agents dynamically adjust UAV flight trajectories (optimize monitoring angles), access scheduling (ensure data transmission in high-traffic areas), transmission power and bandwidth allocation at the time slot level, reducing data transmission delay.

[0153] S112: Through continuous learning optimization, the system maintains efficient data processing in traffic peak periods and emergency scenarios, improving the real-time and reliability of urban monitoring.

[0154] This embodiment is suitable for network monitoring support for large public events. By optimizing network resource allocation and task scheduling, it significantly improves the level of on-site network services and enhances user experience.

[0155] The present application significantly improves the performance and efficiency of air-ground integrated network optimization through the joint optimization framework, efficient optimization algorithm, hierarchical optimization strategy, and reward mechanism design, solves the key problems in the prior art, and provides a new effective method for air-ground integrated network optimization.

[0156] Embodiment 1

[0157] This embodiment constructs a multi-time scale optimization mechanism in a MEC and TDMA-based air-ground integrated network (AGINs), which is characterized by fine division and efficient use of the time dimension. Specifically, the system divides time into coarse-grained "frames" and fine-grained "time slots", with a total of K frames and T time slots per frame. This hierarchical division not only adapts to the time scale requirements of different decisions, but also lays the foundation for cross-scale collaborative optimization.

[0158] In terms of operation mechanism, the system adopts TDMA access mode, and each time slot is only allocated to one UAV for data transmission, which fundamentally avoids communication interference between UAVs and ensures the stability of transmission. At the same time, the task processing strictly follows the time constraint, which requires the UAV to complete the task within the allocated time slot, which strengthens the timeliness of decision-making. From the optimization dimension, the coarse-grained frame-level decision focuses on the rationality of the service placement strategy, and the fine-grained time slot-level decision deeply optimizes the UAV's trajectory planning, access control logic, bandwidth allocation ratio and transmission power adjustment, and realizes the improvement of the overall performance of the system through hierarchical cooperation. The scene initialization process covers multi-dimensional parameter configuration: deploying I UAVs (denoted as ) and one ground base station (BS), which integrates the MEC server and serves as a unified processing unit; the number of iterations, the number of frames included in each iteration, and the number of time slots in each frame are specified, providing a clear process framework for subsequent optimization.

[0159] In terms of task processing, at the beginning of each time slot, each UAV needs to complete H tasks, and their set is denoted by . A general task can be represented by a four-tuple . Here, D h is the size of the input data of the task (in bits); λ h is the processing density of the task, i.e., the number of CPU cycles required to execute each bit of input data; O h is the size of the output calculation result, which is much smaller than the size of the input data D h , and will be ignored in the following; is the maximum tolerable task processing delay of task h. This paper assumes , i.e., the task should be completed within each time slot. The controller deployed at the BS plays a key coordination role, collecting dynamic information of the UAVs in real time, including real-time location, channel gain change, task request type, etc., based on which network operation decisions and optimization are made. Since the UAVs are limited by onboard computing power and energy reserves, the data and tasks collected by the UAVs need to be offloaded to the MEC server at the BS for processing. The UAVs and the BS communicate directly through a wireless link, forming an efficient data interaction closed loop.

[0160] Embodiment 2

[0161] The dynamic nature of networks and the uncertainty of information acquisition make the problem quite challenging. To address this effectively, the problem is reconstructed as a Markov Decision Process. A hierarchical DRL algorithm is proposed to efficiently solve this problem, where a high-level DQN agent is used to obtain coarse-grained optimization for each frame, while a low-level IDDPG agent is used to obtain fine-grained optimization for each time slot. The IDDPG agent is an improvement on the traditional DDPG, treating the continuous output of access control as having a probability of 1 for the access control task, thus adapting to the mixed-integer characteristics of our short-scale optimization within each time slot.

[0162] like Figure 1 As shown, this embodiment of the invention provides a method and system for optimizing edge air-ground networks based on hierarchical deep reinforcement learning, specifically including the following steps:

[0163] S101, Execute system initialization, first complete the initial value setting of core network parameters: including the primary Q network parameter ω, the target Q network parameter ω - The Actor network of the principal and target networks, the Critic parameters θ, w, and θ - =θ and w - =w, then configure the number of iterations L, the number of iterations per iteration (i.e., the number of frames K), the number of time slots per frame T, set the learning rate α1 for the DQN network, the learning rates α2 and β2 for the Critic and Actor networks, and the discount factor γ, and then determine the replay buffer size D. hi D l The mini-batch sizes M1 and M2, and the OU noise Ξ(t) for action exploration; in addition, network layout parameters, such as the number of UAVs I and other task parameters, need to be initialized.

[0164] S102, At the start of the cycle, initialize the state s of the high-level DQN agent. hi (k); Selecting a high-level action a using the ∈-greedy strategy. hi (k) generates a frame-level service placement strategy;

[0165] S103, the agent executes an action, generating temporary high-level rewards according to constraints. If the constraints are satisfied, then In each time slot within a frame, the IDPG agent is used to perform low-level task optimization; if this condition is not met, the value is 0.

[0166] S104, Initialize the state of the low-level IDPG agent s l (t), after probability sampling, is converted into a continuous output and then used for binary access control to execute actions. Obtain time-slot-level UAV trajectory, access control, uplink bandwidth allocation ratio, and transmission power;

[0167] S105, the agent performs an action and gets a reward r l (t), the low-level state is updated to s l (t+1);

[0168] S106, the current experience is stored in the replay buffer D l If the buffer is full, the oldest experience is discarded; from M2 low-level experiences are randomly taken (as a mini-batch); for the mini-batch;

[0169] S107, the TD target and error of the low-level IDDPG are calculated; the actor network, critic network, target actor network and critic network are updated synchronously;

[0170] S108, the actor network and critic network parameters are updated based on the mini-batch experience; the target network parameters are updated synchronously;

[0171] S109, the last time slot of the frame, the high-level DQN gets a reward r hi (k), and the state is updated to the next state s hi (k+1);

[0172] S110, the experience is stored in D hi If the buffer is full, the oldest experience is discarded. A mini-batch of experiences M1 is sampled, and the DQN main network is inputted with the state-action to estimate the Q value;

[0173] S111, the TD target and error of the high-level DQN are calculated; the main and target DQN networks are updated, and the target Q network parameters are updated synchronously;

[0174] S112, the iterative training is repeated, the optimal action is selected based on the Q value of each state-action, the reward is maximized, and the optimal task placement, UAV trajectory, access control, bandwidth allocation and transmission power are outputted.

[0175] Further, at the beginning of each cycle of step S102, the system state s hi of the high-level DQN agent is initialized; the state is specifically described as:

[0176] s hi (k) = {r(k)},

[0177] wherein r(k) = {r i,h (k)}, r i,h (k) ∈ {0, 1}, is used to represent the task request of the kth frame of UAVs; the high-level agent DQN action a hi(k), to obtain the service placement strategy of the frame, and the action can be represented as:

[0178] a hi (k) = {Δ(k)},

[0179] where Δ(k) = {δ h (k)}, δ h (k), is the service placement decision.

[0180] Further, the temporary reward reward formula in step S103 is:

[0181]

[0182] where (C1) is δ h (k) ∈ {0, 1}, represents the binary constraint of the service placement strategy, and (C2) is represents the capacity constraint of the service setting.

[0183] The high-level temporary reward is designed to judge whether it is necessary to carry out low-level optimization. When , it indicates that the service placement meets the (C1)-(C2) constraint condition, that is, the placement is feasible, and the low-level optimization is carried out with the help of the IDDPG agent; when , it indicates that the task placement is not feasible, and the low-level optimization has no practical value, because the high-level solution and the low-level solution together constitute the solution of the problem , and if the high-level solution is not feasible, regardless of the state of the low-level solution, the overall solution cannot be established.

[0184] Further, in step S104, the state s l (t) of the low-level IDDPG agent is initialized, and the state is represented as follows:

[0185]

[0186] where Δ(k) = {δ h (k)}, δ h (k), represents the service placement decision, and indicates whether the task h has been deployed on the base station at the start of the time slot t; u(t) = {u i,x (t), u i,y (t)} is the horizontal position of the UAV i at the start of the time slot t; is used to indicate whether there is an obstruction between the UAV i and the BS; R up (t) is the uplink transmission rate between the UAV i and the BS. The backhaul rate of UAVi to the base station; the state space of the low-level IDDPG agent is denoted as It is the set of all low-level states.

[0187] By probabilistic random sampling, the continuous output value is converted into binary access control, and the IDDPG agent executes the action The action is defined as:

[0188] a l (t)={d(t),A(t),γ(t),p up (t)},

[0189] Where d(t)={v i (t),θ i (t)}, is the UAV deployment strategy; A(t)={a i (t)}, Where a i (t) is the access control strategy of UAVi, γ(t), Indicates the bandwidth offloading ratio of the UAV; p up (t), is the uplink transmission rate of the UAV. The action space of the IDDPG agent is represented as is the set of all feasible solutions of the task processing related variables.

[0190] Further, in step S105, the agent obtains the reward r l (t) after executing the generated action, and the low-level state is transferred to the next state s l (t+1); the calculation formula of the reward reward is as follows:

[0191]

[0192] Where (C3) v i (t)∈[0,v max ], Indicates the UAV flight speed constraint, (C4) θ i (t)∈[0,2π], Indicates the UAV flight angle constraint; a i (t)∈{0,1}, is (C5) the access control decision of the base station to access the UAV i; is (C6) used to ensure that only one UAV accesses the base station in each time slot; γ i (t)∈[0,1], is (C7) a UAV bandwidth allocation ratio constraint; (C8) is The transmission power of UAVi to the base station is represented. The first term in the formula refers to the fairness coefficient of the UAV when the base station accesses the UAV i; the second term refers to the final success rate of the UAV i transmitting to the base station, and the third term refers to the total energy consumption of the UAV.

[0193] Further, in step S107, the main Actor network is predicted as:

[0194]

[0195] The target Actor network is predicted as:

[0196]

[0197] The main Critic network is predicted as:

[0198]

[0199] The target Critic network is predicted as:

[0200]

[0201] The TD target calculation formula is:

[0202]

[0203] The TD error derivation is:

[0204]

[0205] The Critic network is updated according to the following formula:

[0206]

[0207] At the same time, the Actor network is updated as:

[0208]

[0209] Further, for step S108, according to the updated parameters of the main Actor network and the Critic network, the formula for updating the parameters of the target Actor network and the Critic network is as follows:

[0210] θ - ←ζ·θ+(1-ζ)·θ - ,

[0211] w - ←ζ·w+(1-ζ)·w - ,

[0212] where θ is the parameter of the current policy network, and θ- w are the parameters of the target policy network, w - w are the parameters of the target value network, ζ∈[0,1] is a hyperparameter that needs to be manually adjusted.

[0213] Further, the calculation formula of the high-level reward in step S109 is as follows:

[0214] r hi (k)=λ1α1χ(t,k)+λ2α2r suc (t,k)-λ3α3E(t),

[0215] That is, the objective function of the present application.

[0216] Further, the process of step S110 is as follows: storing the current experience into an experience replay buffer D hi ; if the buffer is full, deleting the oldest experience to make room; extracting experience M1 from the buffer, inputting the state and action into the DQN main network, and obtaining the estimated Q value as follows:

[0217] q hi (k) m =Q(s hi (k) m ,a hi (k) m ;ω),

[0218] wherein ω is the main Q network weight;

[0219] inputting s high (k+1) m into the target network to obtain:

[0220]

[0221] wherein ω - is the target Q network weight.

[0222] Further, step S111 calculates the generated TD target according to the following formula:

[0223]

[0224] and uses the calculated TD error:

[0225]

[0226] to update the main DQN network:

[0227]

[0228] wherein α1 is the learning rate; and the target DQN network is updated according to the following formula:

[0229]

[0230] wherein, are manually adjusted hyperparameters.

[0231] To elaborate the edge-orbit network optimization method and system of hierarchical deep reinforcement learning, the present application provides two specific application embodiments, including the key details of the implementation scheme.

[0232] Application Embodiment 1: Wireless communication optimization in remote areas

[0233] This embodiment focuses on disaster relief in remote areas, where the environment is complex and communication is easily interrupted. It is necessary to quickly realize environment perception, data collection and key task processing. By deploying multiple UAVs for disaster data collection, a ground base station integrated with a MEC server is fixed in a safe area as the core processing unit. The TDMA mechanism is used to avoid communication interference between UAVs and the base station, ensuring the efficiency of data transmission and task decision-making during the rescue process. The core steps are as follows:

[0234] 1) System initialization

[0235] In the disaster relief scenario, multiple UAVs are deployed for environment perception and data collection in the disaster area, and a ground base station integrated with a MEC server is fixed in a safe area as the core processing unit. The time dimension is divided into coarse-grained "frames" and fine-grained "slots", and the TDMA mechanism is used to ensure that only one UAV is allowed to communicate with the base station in each time slot, avoiding interference. At the same time, the Q network of the high-level DQN agent and the Actor-Critic network of the low-level IDDPG agent are initialized, and parameters such as experience replay buffer are set, laying the foundation for subsequent decision optimization.

[0236] 2) Hierarchical DRL agent and environment interaction

[0237] The high-level DQN agent decides the service placement strategy at the beginning of each frame based on the initial disaster data collected by the UAV (such as earthquake monitoring, trapped personnel positioning, etc. Task requests), i.e. which tasks are deployed in the MEC server first, to ensure that critical tasks get computing resources first. The core is the rationality of global task deployment, which does not directly participate in UAV trajectory or base station parameter adjustment.

[0238] The low-layer IDDPG agent executes fine-grained control in each time slot according to the high-layer service placement strategy and in combination with real-time disaster conditions (such as signal shielding and blind area coverage demand), and performs the following optimization: optimizing the UAV trajectory to avoid obstacles and fill in monitoring blind areas; allocating access time slots based on the task urgency through the TDMA mechanism; and dynamically adjusting the bandwidth allocation ratio and transmission power to accelerate the unloading of disaster data to the base station. Exploration noise is introduced during action generation to adapt to the dynamically changing environment in the disaster area.

[0239] 3) Action execution and reward acquisition

[0240] After the execution of the high-layer and low-layer actions, the system acquires rewards based on a constraint perception mechanism: if the action meets the constraints (such as the upper limit of the MEC server storage, the UAV speed limit, the TDMA access rules, etc.), the instantaneous reward is calculated based on the task success rate (such as whether the key disaster data is processed within the delay threshold), the UAV energy consumption (total energy consumption of flight and communication), and the resource scheduling fairness (Jain index, to avoid excessive occupation of time slots by a UAV); if the constraints are violated, the reward is 0. The system state transitions to the next state with the update of the UAV position, the addition of new tasks, and the change of channel conditions.

[0241] 4) Optimization and update

[0242] The agent updates the network based on the collected experience data (state, action, reward, next state): the high-layer DQN updates the Q network through frame-level cumulative rewards to optimize the service placement strategy; and the low-layer IDDPG updates the Actor-Critic network through time-slot-level instantaneous rewards to iteratively optimize fine-grained decisions such as trajectory and resource allocation. During the training process, the task processing success rate is improved to ensure the timeliness of rescue, and the UAV energy consumption is reduced to extend the endurance, achieving continuous monitoring and data support in the disaster area.

[0243] 5) Performance evaluation and tuning

[0244] The performance of the layered DRL algorithm is evaluated through simulation experiments or field tests. The core indicators of the communication network, such as coverage range, signal quality, and data transmission rate, are analyzed and compared with traditional methods. According to the evaluation results, the algorithm parameters are adjusted and optimized to further enhance the performance and stability of the communication network. Through the above steps, the layered DRL-based MEC-driven air-ground integrated network joint optimization method proposed in the present application can effectively cope with the challenges of wireless communication in disaster scenarios and improve the communication quality and user experience.

[0245] Application Example 2: Communication support for urban intelligent monitoring system

[0246] This embodiment is aimed at urban intelligent monitoring, which needs to monitor key areas such as traffic hubs and commercial districts in real time, process data such as traffic flow, pedestrian flow, and abnormal behavior, and cope with dynamic changes such as sudden pedestrian flow and signal obstruction in the city. Multiple UAVs are deployed to collect monitoring data, and the MEC base station in the ground core area is responsible for data processing. The TDMA mechanism is used to ensure the orderly communication between UAVs and base stations, and to ensure the efficient completion of the monitoring task. The core steps are as follows:

[0247] 1) System initialization

[0248] In the urban intelligent monitoring system, multiple UAVs are deployed to cover key areas such as traffic hubs and commercial districts, and are responsible for real-time collection of monitoring data such as traffic flow, pedestrian flow, and abnormal behavior. The ground base station integrates MEC servers and is fixedly deployed in the core area of the city as a data processing center. The time dimension is managed in frame-time slot hierarchy, and the TDMA mechanism is used to ensure that only one UAV communicates with the base station in each time slot to avoid interference. At the same time, the Q network of the high-level DQN agent and the Actor-Critic network of the low-level IDDPG agent are initialized, and parameters such as experience replay buffer and learning rate are set to complete the system startup preparation.

[0249] 2) Interaction between agent and environment

[0250] The high-level DQN agent decides the service placement strategy at the beginning of each frame based on the monitoring task requests reported by the UAV, such as accident monitoring and traffic statistics, i.e., which tasks should be deployed in the MEC server first, such as high-priority accident monitoring tasks to occupy computing resources first to ensure the priority of processing key tasks.

[0251] The low-level IDDPG agent executes fine-grained control according to the high-level service placement strategy and real-time monitoring requirements such as high-traffic area data transmission during traffic peak hours in each time slot: dynamically adjusts the UAV trajectory to optimize the monitoring view coverage of blind spots or key areas; allocates access time slots through the TDMA mechanism to ensure data transmission of high-priority tasks; adjusts the bandwidth allocation ratio and transmission power to reduce data transmission delay. Exploration noise is introduced when generating actions to adapt to dynamic urban scenarios such as sudden pedestrian flow and signal obstruction.

[0252] 3) Action execution and reward acquisition

[0253] After performing high-level and low-level actions, the system calculates the reward based on the constraint-aware mechanism: if the actions meet the constraints such as the MEC server storage upper limit, UAV speed limit, TDMA single time slot single UAV access rule, the reward is composed of three parts - task success rate such as whether the monitoring data is processed within the delay threshold, total UAV energy consumption flight and communication energy consumption, and scheduling fairness Jain index to avoid excessive occupation of time slots by a UAV; if the constraints are violated, the reward is 0. The system state is updated to the next state with the UAV position, new monitoring task joining, and channel condition changes such as channel gain fluctuations caused by occlusion.

[0254] 4) Network update and optimization

[0255] The agent iteratively optimizes the current state, performs actions, obtains rewards, and the next state based on the collected experience data: the high-level DQN updates the Q network through frame-level cumulative rewards, optimizes the service placement strategy, and improves the rationality of key task deployment; the low-level IDDPG updates the Actor-Critic network through time slot-level immediate rewards, optimizes UAV trajectory, resource allocation, and other fine-grained decisions. The optimization process focuses on improving the success rate of monitoring tasks such as real-time abnormal behavior recognition efficiency, reducing UAV energy consumption to extend endurance, and ensuring scheduling fairness.

[0256] 5) Performance evaluation and tuning

[0257] Through simulation or field testing, the system performance is evaluated: the core indicators include the success rate of monitoring tasks such as whether the data is processed within the delay threshold, the energy efficiency of UAVs per task, and the scheduling fairness of time slot occupation balance of each UAV. The results are compared with baseline schemes such as fixed deployment and random resource allocation, and the network parameters such as discount factor and exploration noise intensity are adjusted according to the evaluation results to further optimize the decision-making ability of the agent and improve the adaptability of the system in traffic peak periods and emergency scenarios.

[0258] In these two embodiments, the edge sky network optimization method and system based on hierarchical deep reinforcement learning provides an efficient, reliable, and energy-saving solution suitable for different application scenarios from urban areas to disaster-stricken areas, demonstrating its wide application potential and technical advantages.

[0259] To evaluate the performance level of the system, we selected multiple benchmark algorithms for comparison tests. The selected benchmark algorithms not only have differentiated technical characteristics and application advantages, but also have representative significance in similar problem solving scenarios. By comparing their performance data under the same experimental conditions, we can more intuitively identify the outstanding highlights of the invention and the weak links that need further optimization.

[0260] 1. Random resource: In this algorithm, resource allocation is random, including bandwidth allocation, transmission power, etc. are randomly set, without strategic optimization.

[0261] 2. Fixed deployment: In this algorithm, the service placement strategy is fixed, and the task deployment on the MEC server is pre-set, without changing according to the real-time task demand.

[0262] 3. Random access control: Since we use TDMA, each time slot can only be used for one UAV. In this algorithm, access control is randomly determined, that is, each time slot is assigned to a random UAV for task offloading.

[0263] 4. Random UAV placement: This algorithm involves randomly determining UAV placement decisions. Therefore, the UAV will fly at a random speed in a randomly selected direction.

[0264] 5. The present invention: It represents the proposed multi-time scale hierarchical DRL algorithm.

[0265] We show the convergence performance of all the above algorithms under the default parameter settings. The results show that the proposed algorithm converges faster and can reach a stable state in fewer iterations. In terms of returns and stability, the proposed algorithm performs outstandingly: not only can it obtain higher returns, but also the reward fluctuation is smaller after reaching stability, maintaining a stable state. In contrast, other algorithms converge more slowly, have lower reward levels after reaching stability, and have varying degrees of fluctuations.

[0266] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any modification, equivalent replacement and improvement within the technical range disclosed by the present application, which is within the spirit and principles of the present application, should be covered within the protection scope of the present application.

Claims

1. A method for optimizing edge-space-ground networks based on hierarchical deep reinforcement learning, characterized in that, include: Perform system initialization, set the main Q network parameters, target Q network parameters, main Actor and Critic parameters, target Actor and Critic parameters, configure the number of iterations, number of frames per iteration, number of time slots, DQN learning rate, Actor learning rate, Critic learning rate, discount factor, experience replay buffer size, mini-batch size and exploration noise, and initialize network layout parameters; At the start of the cycle, the state of the high-level DQN agent is initialized, and a frame-level service placement policy is generated through the policy. Execute high-level actions and generate temporary rewards. If the constraints are met, call the low-level IDPG agent for slot-level optimization; otherwise, the reward is 0. The low-level IDPG agent initializes the state and converts the continuous output into binary access control based on probability sampling to obtain the UAV trajectory, access control, bandwidth allocation ratio and transmission power. The lower-level agent performs actions to obtain rewards and updates its state; Store the experience in the replay buffer, sample a small batch of experience to update the Actor and Critic network parameters and synchronize the target network; At the end of the frame, the higher-level DQN receives a reward and updates its state, and the sampling experience updates the Q-network parameters; Repeatedly iterate the training until the optimal service placement, trajectory, access control, bandwidth allocation, and transmission power are output.

2. The method according to claim 1, characterized in that, The state description of the high-level DQN agent is the task request of the UAV in frame t, and the frame-level service placement action is output through the policy. The action is represented as the service placement decision matrix.

3. The method according to claim 1, characterized in that, The calculation of the high-level temporary reward satisfies the binary constraint of service placement and the capacity constraint. When the conditions are met, the reward is greater than 0 and low-level optimization is initiated; otherwise, the reward is 0.

4. A method for optimizing edge-space networks based on hierarchical deep reinforcement learning, characterized in that, include: In step S104, the state of the low-level IDPG agent is initialized. The state includes task placement decision, horizontal position of UAV at the beginning of time slot, indicator of whether there is a blockage between UAV and base station, uplink transmission rate between UAV and base station, and backlink transmission rate between UAV and base station. The IDPG agent performs actions based on the above-mentioned state, including UAV deployment strategy, access control strategy, bandwidth offload ratio, and uplink transmission power.

5. The method according to claim 4, characterized in that, The low-level reward function consists of the following three parts: the fairness coefficient when the base station accesses the UAV, the success rate of the UAV transmission task, and the total energy consumption of the UAV. The reward satisfies the constraints of flight speed, flight angle, access control, bandwidth allocation, and transmission power.

6. A method for optimizing edge-space networks based on hierarchical deep reinforcement learning, characterized in that, include: In the lower-level steps, an Actor-Critic structure is used for updates. The main Actor network predicts the output of UAV deployment and access control, while the target Actor network predicts the update. The main Critic network predicts the state action value, and the target Critic network predicts the target value. The Critic network is updated based on the TD target and TD error, and the Actor network is updated based on the policy gradient. The target network parameters are synchronized in a soft update manner.

7. The method according to claim 6, characterized in that, The target network parameters are updated using a weighted average update method, which means that the current main network parameters and the existing target network parameters are weighted and combined according to a proportional coefficient to obtain new target network parameters. The proportional coefficient is a pre-set hyperparameter used to control the following speed of the target network parameters to the main network parameters.

8. An edge air-ground network optimization system based on hierarchical deep reinforcement learning, characterized in that, include: The system initialization module is used to set the initial parameters of the Q-network, Actor network, and Critic network, and configure the number of iterations, learning rate, discount factor, experience buffer size, mini-batch size, and exploration noise. The configuration module is used to set task parameters and the number of UAVs; The agent module, comprising a high-level DQN agent and a low-level IDPG agent, is used to generate frame-level and slot-level actions; The action execution module is used to implement service placement decisions, trajectory planning, access control, bandwidth allocation, and power allocation; The reward acquisition module is used to calculate high-level and low-level rewards based on constraints. The state transition module is used to update the lower-level state; The experience replay module is used to store and sample training data; The network update module is used to update the parameters of the Q network, Actor network, and Critic network. The parameter update module is used to synchronize the target network using a soft update method based on the main network parameters.

9. The system according to claim 8, characterized in that, The agent module selects high-level actions based on a greedy strategy at the beginning of each cycle, and adds exploration noise to the low-level actions to enhance the search capability.

10. The system according to claim 8, characterized in that, The action execution module supports interface docking with the base station scheduling system to realize cross-network task placement and joint resource optimization.

Images