Layered agent-based space-air-ground caching and resource optimization method and system

Abstract

The invention discloses an air-space-ground caching and resource optimization method and system based on a hierarchical intelligent agent, and aims to construct a deep reinforcement learning architecture in which a high-layer DQN and a low-layer DDPG are coordinated for high dynamics and information uncertainty of an air-space-ground integrated network. The high-level intelligent agent generates a period-level content cache and access control strategy based on global states including a cache state, a task request, a node resource and the like, and the low-level intelligent agent executes time slot-level resource allocation, task unloading rate control and UAV deployment optimization under the constraint of the high-level strategy. The system triggers low-level optimization through a double-stage reward mechanism and constraint verification, evaluates a strategy effect based on time slot income and full-period income, and combines state perception and experience playback technologies to realize collaborative optimization of high and low-level decisions. A simulation result shows that compared with a traditional method, the method has remarkable advantages in the aspects of reducing content acquisition delay, reducing return communication overhead, improving task processing success rate and the like, and the space-air-ground MEC network resource utilization rate and user experience are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of wireless communication, and particularly relates to a space-air-ground cache and resource optimization method and system based on a hierarchical intelligent agent. BACKGROUND

[0002] With the scale deployment of space-air-ground integrated network (SAGIN), MEC nodes are embedded into satellites, UAVs and ground base stations and other access layers, giving rise to edge service forms with wide coverage, low latency and adjustable computing power near the edge. The high demand for mass differentiated content in such a heterogeneous topology brings opportunities for cache hit rate and link energy efficiency improvement, but also poses unprecedented requirements for cross-node collaborative scheduling. Single-node isolated configuration obviously cannot fully exploit the overall potential, and therefore it is necessary to comprehensively consider content placement, replacement, wireless bandwidth-power allocation and UE access mapping and other multi-dimensional strategies in a global perspective, so as to form a dynamic balance with the limited cache, computing power and spectrum resources of each node.

[0003] However, the core challenge of SAGIN lies in strong dynamics and high coupling: user requests change rapidly, channels fluctuate rapidly due to spatial occlusion, UAV maneuvering and atmospheric disturbance, resource decisions contain discrete and continuous variables, are subject to multiple constraints such as capacity, power, access, and exhibit mixed integer nonlinear complexity. Existing algorithms based on rules or single-layer optimization often suffer from dimension explosion and convergence lag, and cannot capture the global optimum in real time in the 6G scenario where communication, computing and cache are intertwined. Especially when low-orbit satellites and swarms of UAVs coexist, link competition and backhaul traffic can easily cause bottleneck congestion, leading to amplified tail peaks of task latency and increased energy consumption.

[0004] The solution lies in building a hierarchical intelligent agent framework: high-level intelligent agents focus on periodic content cache and access control games, maximizing long-term economic returns using deep Q learning; low-level intelligent agents focus on fine scheduling of bandwidth and power at the time slot level, and quickly adapt to channel transients using deterministic policy gradient. The upper and lower layers are coupled through rewards to complete the information loop, which on the one hand reduces the dimensionality and coupling of high-dimensional mixed optimization, and on the other hand takes into account both long-term performance and immediate response. This multi-agent hierarchical collaborative mode lays a methodological foundation for real-time resource orchestration of SAGIN in services such as ultra-high-definition video, emergency communication and XR rendering, and provides a feasible evolution path for wide-area edge intelligent networks in the 6G era. SUMMARY

[0005] To address the problems existing in the prior art, this invention provides a joint optimization method and system based on hierarchical multi-agent deep reinforcement learning (DRL). By collaboratively optimizing content cache placement and replacement strategies, user access control strategies, and wireless resource (bandwidth / power) allocation strategies, the method minimizes the average content acquisition latency and backhaul traffic, thereby improving network resource utilization and user service experience.

[0006] This invention is implemented as follows: It is a joint optimization method and system based on hierarchical multi-agent DRL. This system, by combining a master-slave dual-layer DRL network, achieves joint optimization of key parameters in SAGIN, such as content caching and replacement strategies, resource allocation strategies, and user access strategies, including the following steps:

[0007] To address the problems of inefficient content caching strategies, unreasonable resource allocation, fixed access control, and lack of intelligent scheduling in UAV flight path planning in existing technologies for integrated air-space-ground networks, this invention provides a joint optimization method and system based on hierarchical multi-agent DRL, which can collaboratively optimize content cache placement and replacement, joint resource allocation between UEs and UAVs, user access strategies, and UAV resource allocation strategies.

[0008] This invention employs a hierarchical master-slave DRL architecture, combining high-level discrete optimization with low-level continuous optimization mechanisms to achieve global joint intelligent optimization of the aforementioned key parameters. Specifically, it includes the following steps:

[0009] S101, System initialization, including initialization of the parameters of the main Q network and target Q network of the high-level DQN network, the main Actor and Critic networks and corresponding target network parameters of the low-level DDPG network, setting the iteration number L, the number of time slots in each iteration cycle, initializing the learning rate of the DQN network, the learning rate of the Critic and Actor networks of DDPG, the discount factor, configuring the experience replay buffer size, mini-batch size and OU noise parameters for action exploration, and initializing network layout parameters such as the number of ground mobile UEs, the number of UAVs, the content library size, the size of a single content, the bandwidth and transmission power of UAVs and satellites, and other mission parameters.

[0010] S102, the agent interacts with the environment; at the beginning of each cycle, the system state s of the high-level DQN agent is initialized. 1 Use the ∈-greedy strategy to select the high-level actions a of the DQN agent. 1 (t), thus obtaining the cache placement decision and access control strategy;

[0011] S103, the agent executes the generated action, judges whether the constraint is met to generate a temporary high-level reward indicator r DQN (t); if the constraint is met, r DQN (t) = 1, the low-level task processing related optimization is performed using the DDPG agent in each time slot of the current period, otherwise r DQN (t) = 0;

[0012] S104, initialize the state of the low-level DDPG agent The continuous output value is converted into binary access control using probability random sampling, and the DDPG agent executes the action The bandwidth resources allocated to the UAV and the satellite, and the allocation of the transmission power of the UAV and the satellite are obtained;

[0013] S105, the agent obtains the reward r j (t) after executing the generated action

[0014] S106, store the current experience to the experience replay buffer D j ; if the buffer is full, the oldest experience is deleted to make room, and a small batch of experiences M is randomly sampled for updating;

[0015] S107, calculate the TD target and TD error of the low-level DDPG agent; update the main low-level Actor network and Critic network; update the target Actor network and Critic network;

[0016] S108, update the parameters of the Actor and Critic networks based on the small batch of experiences; update the target Actor and Critic network parameters according to the updated main Actor and Critic network parameters;

[0017] S109, after the low-level of the current time slot ends, the high-level DQN agent obtains the high-level reward r DQN (t) after the DDPG agent obtains its reward, and the state is transformed to the next state s 1 (t+1);

[0018] S110, store the current experience 1 (t), a 1 (t), r DQN (t), s 1 (t+1)> to the experience replay buffer D 1 ; if the buffer is full, the oldest experience is deleted to make room; a small batch of experiences M is sampled from the experience replay buffer, the state and action are taken as the input of the DQN main network, and the estimated Q value is obtained;

[0019] S111, calculate the TD target and TD error of the high-level DQN agent, update the main DQN network and the target DQN network, and update the target Q network parameters according to the updated main Q network parameters;

[0020] S112, repeat the iterative training, select the optimal action according to the Q value of the corresponding action in each state, obtain the maximum benefit, and finally obtain the optimal content caching placement and replacement strategy, the UEs and UAV joint resource allocation strategy, and the UEs access control strategy.

[0021] Further, the S102 initializes the system state s of the high-level DQN agent at the beginning of each period 1 ; wherein the state is represented as:

[0022]

[0023] wherein represents whether UE i requests content f, and it is noted that a UE only requests one content; and represents the index of the cached content of the MEC server j at the end of time slot t-1; wherein represents the remaining available cache resources of the MEC server j; content removal indicator and represents the time-varying channel gain between the UAV and the UE; represents the path loss between the UE and the satellite; represents the horizontal distance between UE i and UAV j; represents the flight height of UAV j from the ground; represents the distance between the UAVs; the main strategy network generates an action based on the current strategy, represented as:

[0024] a 1 (t)={Λ(t),A(t)},

[0025] wherein, represents content placement; represents an access strategy.

[0026] Further, the calculation formula of the temporary reward reward in the S103 is as follows:

[0027]

[0028] The high-level temporary reward is designed as an index of whether it is necessary to perform low-level optimization. When r DQN(t) = 1, i.e., the placement and replacement of tasks satisfy the constraints (C1)-(C4), thus is feasible, we will perform the low-level optimization through the DDPG agent; when r DQN (t) = 0, i.e., the placement and replacement of tasks are infeasible, so it is meaningless to perform the low-level optimization, since the high-level solution and the low-level solution jointly constitute the solution of the problem When the high-level solution is infeasible, the whole solution is also infeasible regardless of the low-level solution.

[0029] Further, the state of the low-level DDPG agent in the S104 is initialized as where the state is represented as:

[0030]

[0031] where represents the access policy; represents the cache content of the MEC server j; where, k i,f (t) e {0, 1} represents whether the user i requests the content f, is the set of users accessing the UAV j; represents the time-varying channel gain between the UAV and the UE; represents the path loss between the UE i and the satellite, where is the set of users served or relayed by the satellite; represents the horizontal distance between the UE i and the UAV j; y j (t) represents the flight height of the UAV j from the ground; represents the distance between the UAVs; the main policy network generates the action based on the current policy, represented as:

[0032]

[0033] where, represents the allocation of bandwidth resources to the UAV and the satellite; represents the allocation of the UAV and the satellite transmission power.

[0034] Further, the DDPG agent performs the generated action in the S105: the reward function of the DDPG agent: in (P1), our goal is to minimize a target value, i.e., the long-term average weighted sum of the system content acquisition delay and the backhaul traffic. Therefore, we set the total immediate reward r(t) of the system as the inverse of the target value. Based on this consideration, the reward calculation of each DDPG agent is as follows:

[0035]

[0036] If the action satisfies all the constraints, an immediate reward will be obtained; otherwise, the system will receive a penalty as the reward, i.e., r(t) = penalty, where the penalty is a small negative constant, which will guide the agent to select a better action in the next time period.

[0037] Further, the S107 main actor network prediction is as follows:

[0038]

[0039] And the target actor network is predicted according to the following:

[0040]

[0041] The main critic network prediction is:

[0042]

[0043] And the target critic network is predicted according to the following:

[0044]

[0045] The TD target is calculated according to the following formula:

[0046]

[0047] And the TD error is obtained according to the following formula:

[0048]

[0049] The critic network is updated according to the following formula:

[0050]

[0051] The actor network is updated according to the following formula:

[0052]

[0053] Further, the S108 target actor and critic network parameter update formula according to the updated main actor and critic network parameters is as follows:

[0054] θ j - ←ζ·θ j +(1-ζ)·θ j - ,

[0055] w j - ←ζ·w j +(1-ζ)·wj - ,

[0056] where θ represents the parameters of the current policy network, θ j - represents the parameters of the target policy network, w j represents the parameters of the current value network, w j - represents the parameters of the target value network, ζ∈[0,1] is a hyperparameter that needs to be manually adjusted.

[0057] Further, the calculation formula of the high-level reward in S109 is as follows:

[0058] r DQN (t)=Profit(t),

[0059] where Profit(t) is the profit (i.e. the objective function value) of the system at the current cycle time slot, which is used to evaluate the long-term economic benefits of the high-level DQN policy (content caching, access control), and drive the agent to optimize the cycle-level decision to maximize the long-term average profit of the operator.

[0060] Further, S110: store the current experience into the experience replay buffer D 1 ; if the buffer is full, delete the oldest experience to make room; sample a small batch of experiences M1 from the experience replay buffer, use the state and action as the input of the DQN main network, and get the estimated Q value:

[0061] q 1 (t) m =Q(s 1 (t) m ,a 1 (t) m ;ω),

[0062] where ω is the weight of the main Q network;

[0063] Take s 1 (t+1) m as the input of the target network, we get:

[0064]

[0065] where ω - is the weight of the target Q network.

[0066] Further, the TD target is calculated according to the following formula in S111:

[0067]

[0068] And according to the calculated TD error:

[0069]

[0070] Update the main DQN network:

[0071]

[0072] Where alpha1 is the learning rate; update the target DQN network according to the following formula:

[0073]

[0074] Where, Is a hyperparameter that needs to be manually adjusted.

[0075] Another object of the present application is to provide a space-air-ground cache and resource optimization method and system based on hierarchical agents, the system comprising:

[0076] A system initialization module for initializing deep Q network algorithm parameters, including the parameters omega of the main Q network and the parameters omega - of the target Q network;

[0077] For initializing the parameters of the deep deterministic policy gradient algorithm, including the parameters theta and w of the main Actor and Critic networks, and the parameters theta - and w of the target Actor and Critic networks; - ;

[0078] For setting the total number of training iterations L, the length of the time slot sequence contained in each iteration (i.e. the number of time slots covered by a single round of training), and the resource scheduling sub-step size within each time slot;

[0079] For initializing the learning rate alpha1 of the DQN network, the learning rates alpha2 and beta2 corresponding to the Critic and Actor networks, the discount factor gamma, and the initialization replay buffer size D 1 and D j , the size M of the mini-batch, and the OU noise Ξ(t) for action exploration;

[0080] A configuration module for setting application-specific parameters, including task parameters such as the number of ground mobile UEs, the size of the content library, and the number of UAVs;

[0081] An agent module for generating actions based on the current high-level network state at the beginning of each cycle, with the function of using a greedy strategy for the generated actions; for generating actions based on the current low-level network state at the beginning of each cycle, with the function of adding exploration noise to the generated actions;

[0082] Action execution module, high-level action execution: used to execute content task placement and replacement strategy (corresponding to the content cache decision part in the high-level action); used to execute joint task offloading rate control of UEs and UAVs, access control strategy of UEs (corresponding to the access control decision part in the high-level action);

[0083] Low-level action execution: used to execute wireless resource allocation strategy of UEs by UAVs and access nodes, including continuous action optimization such as bandwidth and power allocation. At the same time, according to the cache and access strategy issued by the high level, make local scheduling decision at fine-grained and time slot level under the current network state, ensure that the system maintains resource utilization efficiency and low latency performance in dynamic environment.

[0084] Reward acquisition module, used to execute actions and calculate temporary rewards, the reward acquisition module is designed according to whether the constraint condition of the system is met, and a judgment module is used to judge whether to perform low-level optimization;

[0085] Reward acquisition module, used to calculate rewards according to executed actions and current state, calculate rewards according to the benefits of all devices of the system at the current time slot, and calculate rewards according to the benefits of all devices of the system at the end of the period;

[0086] State transition module, used to transfer the low-level system state from the current state to the next state;

[0087] Experience replay module, used to store experience tuples of low-level system state, action, reward and next state of each time;

[0088] Data sampling module, used to extract mini-batchM from the experience stored in the experience replay module for learning;

[0089] Network update module, used to update the main Q network, the main actor network and the main critic network according to the information obtained by the data sampling module, the module includes a parameter optimization unit for adjusting network parameters by using gradient descent method;

[0090] Parameter update module, used to update the parameters of the target Q network, the target actor network and the target critic network according to the main network parameters, the module adopts a soft update strategy, and the parameter synchronization unit of the target network parameter is the weighted average of the main network parameter;

[0091] Network update module, parameter update module, used to update the target actor network and the target critic network according to the main network parameters, the module adopts a soft update strategy, and the parameter synchronization unit of the target network parameter is the weighted average of the main network parameter.

[0092] In combination with the technical solutions and the technical problems solved above, the technical solutions to be protected by the present application have the following advantages and positive effects:

[0093] Firstly, the present application has significant advantages in the integration of edge caching and SAGIN, which are embodied as follows: Firstly, multi-agent collaborative optimization capability. The present application introduces a high-low multi-agent DRL architecture in LEO satellites and UAV nodes to realize hierarchical collaborative optimization of content caching strategy, access control and resource allocation. The high-level agent is responsible for generating caching and access strategies from a global perspective, while the low-level agent optimizes continuous actions locally at each access point, enabling the system to have end-to-end joint perception and strategy adjustment capabilities, significantly improving the efficiency and accuracy of MEC task scheduling. Secondly, fine management capability of multi-dimensional resources. The present application establishes a constraint coupling relationship among cache space, bandwidth resources and transmission power, and realizes optimal allocation under limited resources through reinforcement learning mechanism, avoiding the problems of resource waste, conflict and imbalance in traditional schemes. Especially under the conditions of scarce edge node resources and high concurrent requests, the system can still maintain stable and efficient operation. Finally, adaptive optimization capability for dynamic environment. The node position, channel state and user request distribution in the SAGIN system are highly dynamic, and the present application introduces a DRL agent with continuous learning to perceive environmental changes in real time and dynamically adjust strategies, effectively dealing with time-varying links, heterogeneous access and user behavior changes, enhancing the service continuity and stability of the system.

[0094] Secondly, the present application has achieved significant technical progress in that it realizes a space-air-ground caching and resource optimization method and system based on hierarchical agents, which has made significant progress in the following key aspects:

[0095] 1) Efficient task placement and replacement strategy. Through the high-level DQN agent, combined with task request frequency, node resource state and content popularity, intelligent placement and replacement of task content are realized. This strategy can dynamically adapt to changes in task request distribution, improve edge cache hit rate, and effectively reduce backhaul traffic and content acquisition delay.

[0096] 2) Flexible UAV deployment strategy. The present application dynamically adjusts the deployment position of UAVs through the collaborative work of high and low level agents, combined with task distribution, user location and cache hit situation. The low-level agent optimizes the resource scheduling behavior of UAVs in real time based on time slot level observation state, thereby improving the task processing efficiency of service nodes and enhancing the ability of the system to adapt to dynamic scenarios.

[0097] 3) Cooperative UAV and UE offloading strategy. According to the computing power of the UE, the task complexity and the current load state, the intelligent decision-making task is executed locally or offloaded to the UAV, realizing the reasonable sharing of task computing pressure between the end and the edge, improving the system processing throughput, reducing the energy consumption of the end device, and improving the service experience.

[0098] 4) Intelligent access control strategy. Through learning user historical behavior and link state, the DQN high-level agent can dynamically select the optimal access point to avoid network congestion and load concentration, improve link utilization and access fairness.

[0099] 5) Integration of DRL. The invention combines DQN and DDPG for discrete control and continuous control, respectively, effectively models and trains the multi-dimensional mixed decision space, realizes end-to-end optimization, and has strong generalization ability.

[0100] 6) Profit maximization optimization. The invention designs a compound reward mechanism linked to operating profits, not only focusing on resource allocation efficiency and delay performance, but also explicitly optimizing long-term economic returns, balancing service quality and cost benefits.

[0101] 7) Improvement of system stability and reliability. Through multi-agent cooperative strategy training and batch experience replay mechanism, learning jitter and non-stationarity are effectively reduced, ensuring good stability and reliability in complex dynamic environments.

[0102] 8) Network self-learning and optimization capability. Each agent can continuously train and update strategies based on environmental feedback, realizing adaptive evolution of task scheduling, cache updating and resource allocation, and the system has long-term continuous optimization capability.

[0103] 9) Breakthrough of joint optimization problem. Traditional solutions often separate cache, access and resource scheduling, the invention models multiple tasks jointly through hierarchical architecture, and realizes overall performance optimization through unified reward mechanism and cooperative training.

[0104] 10) Processing capability in complex and dynamic scenarios. In the face of highly dynamic SAGIN topology and task request changes, the learning strategy of the invention has strong generalization ability and environmental adaptability, ensuring performance robustness.

[0105] 11) Significant improvement in real-time and efficiency. Low-level agents can make quick response decisions based on local state, combined with parallel deployment mechanism, making the system have millisecond-level response capability to meet real-time optimization scenario requirements.

[0106] 12) Construction of joint optimization framework. Through master-slave structure high-low level cooperative training, a closed-loop feedback mechanism between task placement and resource configuration is established, improving overall scheduling efficiency and system coupling processing capability.

[0107] 13) Design and application of high-efficiency optimization algorithm. Improved DQN and DDPG algorithms are adopted, and target network and experience replay mechanism are introduced to significantly improve the convergence speed and learning stability of the strategy.

[0108] 14) Implementation of hierarchical optimization strategy. High-level DQN is responsible for periodic update of caching and access strategy, and low-level DDPG is responsible for real-time allocation of physical resources and task offloading rate, realizing the coordinated control of time and space dimensions.

[0109] 15) Innovation of reward mechanism design. The joint feedback mechanism of local immediate reward and global long-term benefit is introduced to guide the learning direction and strategy value evaluation of each agent, effectively accelerating the training convergence.

[0110] 16) Significant enhancement of response speed and real-time performance. All strategy decision-making processes are based on neural network forward reasoning, without complex solving process, significantly shortening the decision-making delay and meeting the online deployment requirements.

[0111] Thirdly, the space-ground-terrestrial cache and resource optimization method and system based on hierarchical agents provided by the present application use mathematical models to guide the behavior and learning process of the system. The technical effects brought by these mathematical models can be explored according to their characteristics:

[0112] 1) Calculation of rewards

[0113] The formulas for calculating temporary rewards, low-level rewards, and high-level rewards focus on maximizing the profits of all devices in the system.

[0114] Cost savings: In the long run, the improvement of energy efficiency can significantly reduce operating costs, especially in resource-intensive MEC environments.

[0115] Profit maximization: By directly associating rewards with profits, this method encourages the system to discover strategies that maximize profits, thereby generating more profits.

[0116] 1) Update of main Q network, main Actor and Critic network

[0117] Update through random sampling of small batches of experience data and use gradient ascent method to update the current policy network.

[0118] Strategy optimization: By continuously adjusting the policy network parameters, the system can learn and adopt more effective decision-making strategies.

[0119] Improved responsiveness: Using small batches of data allows the network to quickly adapt to environmental changes, enhancing the system's dynamic adjustment capabilities.

[0120] 3) Loss function of TD target and Critic network

[0121] The loss function is calculated based on the time difference TD target and the Q value predicted by the Critic network, and is updated by gradient descent method.

[0122] Learning stability: Using TD target as the basis can balance the learning process and avoid instability caused by excessive prediction error.

[0123] Performance optimization: By accurately calculating the loss function and updating the network, the accuracy and efficiency of system decision-making are improved.

[0124] 4) Parameter update formula

[0125] The method of updating the main Q network, the main Actor and Critic network parameter target network is described, which involves the current network and target network parameters.

[0126] Policy gradually approaching: By gradually updating the target network parameters, the system can smoothly transition to the new policy, preventing performance fluctuations caused by sharp changes.

[0127] Continuous learning and adaptation: This continuous parameter update mechanism ensures that the system can adapt to long-term environmental changes.

[0128] In summary, the present application realizes the whole process of reinforcement learning system from reward function design, policy network update, loss function definition to parameter update mechanism through mathematical modeling, significantly improves the system running efficiency and decision-making quality, and enhances its stability and adaptive ability in complex dynamic environment, which is suitable for modern high-density, strong dynamic space-ground-ground MEC system.

[0129] Fourth, as the creative evidence of the claims of the present application, the expected income and commercial value after the transformation of the technical scheme are:

[0130] The present application realizes the joint optimization of MEC-driven SAGIN by introducing multi-time scale hierarchical DRL, significantly improving network performance and service quality. This not only meets the growing demand for high-performance computing, but also optimizes resource utilization and reduces operating costs. In terms of commercial application, the present application can be widely used in intelligent transportation, UAV distribution, disaster emergency response and other fields, bringing significant economic and social benefits to related industries, and has broad market prospects and huge commercial value. BRIEF DESCRIPTION OF DRAWINGS

[0131] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings described below only represent some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0132] Figure 1 is a flow chart of the space-air-ground cache and resource optimization method and system based on hierarchical agents provided by the embodiments of the present application.

[0133] Figure 2 is a scene diagram that can be applied by the embodiments of the present application.

[0134] Figure 3 is a flow chart of the profit maximization research method of the space-air-ground cache and resource optimization method and system based on hierarchical agents provided by the embodiments of the present application.

[0135] Figure 4 is the total reward of the DQN network under different learning rates provided by the embodiments of the present application. DETAILED DESCRIPTION

[0136] In order to make the purpose, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0137] The core innovation of the present application is to propose a space-air-ground cache and resource optimization method and system based on hierarchical agents, which is used to solve the complex problem of joint content caching, access control and wireless resource allocation in MEC driven SAGIN. The method builds a master-slave hierarchical learning framework between LEO satellites and UAV nodes, effectively dealing with the challenges of mixed discrete and continuous variables, highly dynamic state and multi-scale decision-making in the system. The present application deploys a high-level DQN agent on the LEO satellite, which focuses on global decision tasks, including user access point selection and content caching strategy; while the low-level multiple DDPG agents are deployed on the satellite and UAV edge nodes, which focus on continuous resource optimization decisions of bandwidth and transmission power in each time slot. The high-level strategy output is used as the observation of the low-level input, so as to realize cross-layer collaborative optimization. Through this hierarchical structure, the system can optimize the linkage of cache replacement, access control and resource scheduling according to the dynamic environmental information such as user request, channel state and edge resource condition, greatly reduce the content acquisition delay and backhaul link pressure, and improve the overall system performance. In addition, the method introduces the experience replay mechanism and target network update strategy, which enhances the stability and generalization ability of policy learning. Compared with the existing single reinforcement learning or static optimization method, the present application not only significantly improves the system resource utilization efficiency, but also has good real-time adaptive ability and robustness, providing an intelligent optimization solution with practical deployment value for MEC driven SAGIN architecture.

[0138] In the current commercial deployment of space-air-ground integrated network, the heterogeneous channels and multi-level cache sites exposed by satellites, unmanned aerial vehicles and ground base stations cause the business flow to present a high time-varying characteristic. Traditional single-layer offline planning is often overwhelmed by new business peaks before the algorithm converges, resulting in a decline in cache hit rate and air interface efficiency. The present application introduces a hierarchical agent architecture, which separates content placement-access control and wireless resource fine-grained scheduling into different time scales, taking into account the global economic target and retaining the ability to respond to rapid channel fluctuations in real time, thereby suppressing link congestion and high latency caused by bandwidth cross-domain competition.

[0139] The high-level strategy network is based on a deep Q learning framework, which searches for content caching and user-UAV-satellite three-party access mode in a cycle-level scale. The state space integrates multi-dimensional features such as cache occupancy, user interest distribution, line-of-sight fading and task processing benefit, and the output action gives the cached content index and access mapping matrix. By introducing a discounted profit function instead of a pure throughput rate index, the network has an inherent preference for the long-term cash flow of the operator, which can actively compress repeated backhaul traffic and balance the content hot and cold replacement speed.

[0140] The low-level agent at the fast time slot scale adopts a deterministic policy gradient to dynamically fine-tune the bandwidth segments and power spectral density of the unmanned aerial vehicle and satellite link in a continuous action manner. The critic end carries a target network soft update mechanism to shield the distribution drift risk brought by the high-dimensional action space; the actor end maintains exploration of rare channel situations with the help of OU noise, so that the system can still approach the upper limit of the envelope capacity in the event of rain fade or obstruction. This design avoids the dimension explosion of traditional linear programming in large-scale antenna-vehicle scenarios.

[0141] The upper and lower layers are coupled through rewards to achieve cooperation: the low layer feeds the energy efficiency-delay composite score returned in each time slot to the high layer after periodic integration, which is used as the environment feedback for the next round of cache-access decision; if the high-level strategy is not feasible under physical or capacity constraints, a negative penalty is immediately applied to it and the low-level scheduling is skipped to reduce useless updates. This two-way closed loop adaptively shrinks the search space, making the joint optimization problem evolve from the original large-scale mixed integer programming to a two-level Markov process, significantly reducing the computational complexity.

[0142] The experience replay buffer uses hierarchical storage: periodic trajectories and time slot trajectories are sampled separately, and a priority replay mechanism is used to focus on high TD error samples. The soft update coefficient uses an adaptive learning rate strategy to dynamically adjust the step size according to the policy convergence rate, further suppressing gradient oscillation. Simulation results show that in a typical 6G low-orbit satellite-bee swarm unmanned aerial vehicle-ground integrated scenario, the average task delay of the proposed method is reduced by 28% compared to the existing distributed Heuristic cache scheme, and the system energy efficiency is improved by 33%.

[0143] For industrial landing, the proposed hierarchical agent framework has been verified in ultra-high-definition video backhaul, emergency communication and edge XR rendering services, and can be accessed to MEP (Multi-access Edge Platform) through software upgrade without changing the existing core network signaling. The end-to-end benefits are as follows: on the one hand, it significantly reduces the content back source pressure of the satellite-ground backhaul link and reduces the satellite transmission power expenditure during peak hours; on the other hand, it shortens the average path length from the terminal to the content, releasing additional QoS margin for latency-sensitive businesses such as vehicle networking and remote industrial control, thereby bringing direct quantifiable business value to operators and equipment manufacturers.

[0144] Embodiment 1: Remote content service acceleration system

[0145] In remote rural areas, mountainous areas and other areas with scarce communication resources, traditional infrastructure cannot meet the demand of users for multimedia, education, medical treatment and other content. Based on the hierarchical reinforcement learning structure of high-level DQN and low-level DDPG, the present application constructs a content service-oriented MEC-driven SAGIN optimization system:

[0146] S101: System initialization, deploy multiple UAVs as relay and computing nodes, LEO satellites are responsible for high-level scheduling decisions. Initialization parameters include: ground UE position distribution, UAV initial state, communication parameters, task set (task data volume, computing density, cache requirement, maximum delay), etc.

[0147] S102: High-level DQN agent perceives the global state of the system (such as cache state, task request matrix, node resources, etc.), generates content placement and replacement strategies every training period, and decides which task content should be pre-cached on UAV nodes or offloaded to LEO.

[0148] S103-S111: In each time slot, the low-level DDPG agent outputs continuous control actions based on the current state (including channel state, link load, access density, cache hit rate, etc.), and executes resource optimization strategies including user access control, task offloading rate, UAV flight trajectory, bandwidth and power allocation, etc.

[0149] S112: Both high-level and low-level strategies are learned and updated through experience replay and target network mechanisms, and are iteratively optimized. Finally, the system forms a dynamic scheduling strategy for remote area scenarios, improving the accessibility and response speed of content services, and reducing overall energy consumption and backhaul load.

[0150] This embodiment can be widely used in areas with weak network infrastructure, such as remote education, rural live broadcast, remote training, etc., to achieve efficient and sustainable information service optimization.

[0151] Embodiment 2: Intelligent emergency communication system in disaster area

[0152] In the case of natural disasters (such as earthquakes, floods) causing damage to ground communication systems, the air-ground integrated optimization method provided by the present application can quickly deploy emergency networks to ensure the emergency communication and task processing capability of users in disaster areas:

[0153] S101: System initialization, deploy emergency UAV formation and temporary base station, LEO satellite as control layer node. Build task set (including image reporting, location information, medical data, etc. Emergency tasks), and set UAV energy consumption model, channel environment, flight constraints.

[0154] S102: High-level DQN agent models the global network state in the disaster area every training period, evaluates node cache state, task urgency, link connectivity, etc. Information, output task content placement strategy and access path control scheme.

[0155] S103-S111: In each time slot, the low-layer DDPG agent perceives the local state (such as the remaining energy of the UAV, the distribution of ground users, the current load, etc.), generates access control actions, task offloading rates, power scheduling, and UAV deployment positions, ensures that high-priority tasks are executed first, the link is continuously connected, and the UAV energy consumption is controlled.

[0156] S112: The system continues to learn and optimize in the disaster evolution, forming a disaster emergency communication strategy combination with minimum delay, optimal energy consumption, and maximum task completion rate, and realizing rapid response and adaptive recovery capability.

[0157] This embodiment is especially suitable for the "golden rescue period" after a disaster, and can build an intelligent communication system covering the disaster area within a few minutes, guarantee information transmission and collaborative rescue decision-making, and has great strategic and social value.

[0158] The present application constructs a high-layer DQN and low-layer DDPG multi-time scale layered reinforcement learning structure, and fuses a joint optimization framework, an efficient algorithm, a layered collaborative mechanism, and a multi-dimensional reward function, to realize multi-parameter collaborative optimization such as task placement replacement and user access control. Through remote area content acceleration and disaster emergency communication scene verification, the resource utilization rate and task response speed are effectively improved, and the communication delay and link congestion are reduced. Compared with the traditional method, the present application has strong adaptability, stable convergence, and is easy to deploy, and provides a new path for SAGIN optimization driven by mobile MEC.

[0159] Embodiment 3

[0160] In view of the problems in the prior art, the present embodiment provides a space-ground-cache and resource optimization profit maximization research method based on layered agents, Figure 2 is a scene diagram in which the method of the present application can be applied. First, the number of ground mobile UEs I is initialized, and the set of UEs is represented as The set of UAVs is represented as The iteration number L and the sequence length (i.e., the number of time slots contained) of each iteration are set, and the training parameters are combined with the ground base station scene configuration, wherein the time slot is the basic time granularity of single-step interaction, and the cooperative optimization period of the ground base station and the space-ground node is adapted. In this scene, the UEs generate content requests based on the Zipf distribution at the beginning of each time slot, and the request matrix is represented as where k i,f (t)=1 indicates that UE i requests content f at time slot t.

[0161] The application divides the time axis into coarse-grained periods and fine-grained time slots in the SAGIN supporting MEC and TDMA: a total of T periods, each containing N time slots. The system adopts a TDMA mechanism to allocate different time slots to different UEs to avoid interference. Through the period-level content caching strategy and access control optimization (deciding UAV caching content and UE access relationship), and the time slot-level UE and UAV joint resource allocation (dynamically adjusting bandwidth and power parameters), the long-term average economic profit of the operator is maximized. UAVs and LEO satellites are both deployed with MEC servers, and base stations are integrated with high-performance cloud servers, but UEs have no direct communication link with base stations and need to be relayed through UAVs or satellites. The controller at the base station collects dynamic information such as UE request distribution, AP cache state and channel gain, and based on this, the high-level DQN agent at the satellite end is responsible for the period-level content caching and access control decision, and the low-level DDPG agent at the UAV / satellite end executes the time slot-level bandwidth and power allocation, forming a “global strategy-local execution” closed loop through the hierarchical DRL architecture, and the reward function is designed as to balance the average content acquisition delay and backhaul traffic R BH (t), maximizing the long-term economic profit.

[0162] UAVs and LEO satellites are both deployed with MEC servers as edge access points, and ground base stations are integrated with high-performance cloud servers and connected to the power grid to ensure stable energy supply. In disaster or remote areas, ground UEs have no direct communication link with base stations and need to be relayed through satellites or UAVs. The central controller at the satellite end is responsible for collecting network dynamic information, including UE request distribution, channel gain and horizontal distance, UE state, UAV / satellite cache state, remaining cache resources, available bandwidth and transmit power, AP state, and UAV flight height, UAV distance and satellite path loss, and environmental dynamic parameters. Based on this information, the controller generates a period-level content caching strategy and UE access control through a high-level DQN agent, and a low-level DDPG agent executes a time slot-level bandwidth and power allocation, forming a “global strategy-local execution” hierarchical optimization closed loop, and realizing the joint optimization of space-air-ground networks.

[0163] Embodiment 2

[0164] Considering the dynamic characteristics of the network and the uncertainty of information acquisition, the problem becomes quite tricky. In order to effectively solve it, the problem is reconstructed as a Markov decision process. A hierarchical DRL algorithm is proposed to effectively solve the problem, in which a high-level DQN agent is responsible for period-level coarse-grained optimization, and a low-level DDPG agent executes time slot-level fine-grained resource allocation. DDPG regards the continuous output of access control as the probability of access control task being 1, thus adapting to the mixed integer characteristics of our short-time scale optimization within each time slot.

[0165] As Figure 1 shown, the embodiment of the application provides a space-air-ground cache and resource optimization method and system based on hierarchical agents, specifically comprising the following steps:

[0166] S101, system initialization, including initialization of high-level DQN network main Q network and target Q network parameters, low-level DDPG network main Actor and Critic network and corresponding target network parameters, setting iteration number L, time slot number in each iteration period, initializing DQN network learning rate, DDPG Critic and Actor network learning rate, discount factor, configuring experience replay buffer size, mini-batch size and OU noise parameters for action exploration, and simultaneously initializing network layout parameters such as ground mobile UEs number, UAV number, content library size, single content size, UAV and satellite bandwidth and transmission power, etc. Task parameters.

[0167] S102, agent and environment interaction; at the beginning of each period, the system state s 1 of the high-level DQN agent is initialized 1 ; the high-level action a DQN (t) of the DQN agent is selected using the ε-greedy strategy, obtaining cache placement decision and access control strategy;

[0168] S103, the agent executes the generated action, judges whether the constraint is met to generate temporary high-level reward index r DQN (t); if the constraint is met, r DQN (t) = 0;

[0169] S104, the state of the low-level DDPG agent is initialized , the continuous output value is converted to binary access control using probability random sampling, and the DDPG agent executes the action to obtain the bandwidth resources allocated to the UAV and satellite, and the allocation of the UAV and satellite transmission power;

[0170] S105, the agent obtains the reward r j (t) after executing the generated action, and the low-level state is transformed into the next state

[0171] S106, the current experience is stored in the experience replay buffer D j ; if the buffer is full, the oldest experience is deleted to make room, and a small batch of experience M is randomly sampled for updating;

[0172] S107, calculate the TD target and TD error of the low-layer DDPG agent; update the main low-layer Actor network and Critic network; update the target Actor network and Critic network;

[0173] S108, update the parameters of the Actor and Critic networks based on the small batch of experiences; update the parameters of the target Actor and Critic networks according to the updated parameters of the main Actor and Critic networks;

[0174] S109, after the low-layer ends in the current time slot, the high-layer DQN agent obtains a high-layer reward r DQN (t) after the DDPG agent obtains its reward, and the state is transferred to the next state s 1 (t+1);

[0175] S110, store the current experience <s 1 (t), a 1 (t), r DQN (t), s 1 (t+1)> to the experience replay buffer D 1 ; if the buffer is full, delete the oldest experience to make room; sample a small batch of experiences M from the experience replay buffer, take the state and action as the input of the DQN main network, and obtain the estimated Q value;

[0176] S111, calculate the TD target and TD error of the high-layer DQN agent, update the main DQN network and the target DQN network, and update the parameters of the target Q network according to the updated parameters of the main Q network;

[0177] S112, repeat the iterative training, select the optimal action according to the Q value of the corresponding action under each state, obtain the maximum benefit, and finally obtain the optimal content caching and replacement strategy, the UEs and UAV joint resource allocation strategy, and the UEs access control strategy.

[0178] Further, the S102 initializes the system state s 1 of the high-layer DQN agent at the beginning of each period; wherein the state is represented as:

[0179]

[0180] wherein represents whether UE i requests content f, and it is noted that a UE only requests one content; and represents the index of the cached content of the MEC server j at the end of time slot t-1; wherein represents the remaining available cache resources of the MEC server j; and content removal indicator and denotes the time-varying channel gain between the UAV and the UE; denotes the path loss between the UE and the satellite; denotes the horizontal distance between the UE i and the UAV j; denotes the flying height of the UAV j; denotes the distance between the UAVs; the main policy network generates actions based on the current policy denoted as:

[0181] a 1 (t) = {Λ(t), A(t)},

[0182] wherein, denotes the content placement; denotes the access policy.

[0183] Further, the calculation formula of the temporary reward reward in S103 is as follows:

[0184]

[0185] The high-level temporary reward is designed as an index of whether it is necessary to perform low-level optimization. When r DQN (t) = 1, that is, the placement and replacement of tasks satisfy the constraint conditions (C1)-(C4), and therefore are feasible, we will perform low-level optimization through the DDPG agent; when r DQN (t) = 0, that is, the placement and replacement of tasks are infeasible, so it is meaningless to perform low-level optimization, since the high-level solution and the low-level solution jointly constitute the solution of the problem , when the high-level solution is infeasible, regardless of the low-level solution, the entire solution is also infeasible.

[0186] Further, the state of the initialized low-level DDPG agent in S104 is wherein the state is denoted as:

[0187]

[0188] wherein denotes the access policy; denotes the cached content of the MEC server j; wherein, k i,f (t) ∈ {0, 1} denotes whether the user i requests the content f, is the set of users accessing the UAV j; denotes the time-varying channel gain between the UAV and the UE; denotes the path loss between the UE i and the satellite, wherein is a set of users served or relayed by the satellite; denotes the horizontal distance between UE i and UAVj; y j (t) denotes the UAVj flight height from the ground; denotes the distance between UAVs; the main critic network generates actions based on the current policy represented as:

[0189]

[0190] where, denotes the bandwidth resource allocated to UAVs and satellites; denotes the allocation of UAV and satellite transmit power.

[0191] Further, the S105 in which the agent performs the generated action, the reward function of the DDPG agent: in (P1), our goal is to minimize a target value, i.e. the long-term average weighted sum of system content acquisition delay and backhaul traffic. Therefore, we set the total instantaneous reward r(t) of the system as the inverse of the target value. Based on this consideration, the reward calculation of each DDPG agent is as follows:

[0192]

[0193] If the action meets all the constraint conditions, it will get an immediate reward; otherwise, the system will receive a penalty as a reward, i.e. r(t) = penalty, where the penalty is a small negative constant, which will guide the agent to select better actions in the next time period.

[0194] Further, the S107 in which the main actor network predicts as follows:

[0195]

[0196] And the target actor network makes predictions according to the following:

[0197]

[0198] The main critic network predicts:

[0199]

[0200] And the target critic network makes predictions according to the following:

[0201]

[0202] The TD target is calculated according to the following formula:

[0203]

[0204] And the TD error is obtained according to the following formula:

[0205]

[0206] The critic network is updated according to the following formula:

[0207]

[0208] The actor network is updated according to the following formula:

[0209]

[0210] Further, the update formula of the target actor and critic network parameters in the S108 according to the updated main actor and critic network parameters is as follows:

[0211] θ j - ←ζ·θ j +(1-ζ)·θ j - ,

[0212] w j - ←ζ·w j +(1-ζ)·w j - ,

[0213] where θ represents the parameters of the current policy network, θ j - represent the parameters of the target policy network, w j represent the parameters of the current value network, w j - represent the parameters of the target value network, and ζ∈[0,1] is a hyperparameter that needs to be manually adjusted.

[0214] Further, the calculation formula of the high-level reward in the S109 is as follows:

[0215] r DQN (t)=Profit(t),

[0216] where Profit(t) is the profit (i.e. the objective function value) of the system at the current cycle time slot, which is used to evaluate the long-term economic benefits of the high-level DQN strategy (content caching, access control), and drive the agent to optimize the cycle-level decision to maximize the long-term average profit of the operator.

[0217] Further, the S110 stores the current experience into the experience replay buffer D 1If the buffer is full, the oldest experience is deleted to make room for new experiences; a mini-batch of experiences M1 is sampled from the experience replay buffer to input the state and action into the main DQN network to get the estimated Q value:

[0218] q 1 (t) m = Q(s 1 (t) m , a 1 (t) m ; ω),

[0219] where ω is the weight of the main Q network;

[0220] With s 1 (t+1) m as the input of the target network, we get:

[0221]

[0222] where ω - is the weight of the target Q network.

[0223] Further, the TD target is calculated according to the following formula in S111:

[0224]

[0225] And according to the calculated TD error:

[0226]

[0227] Update the main DQN network:

[0228]

[0229] where α1 is the learning rate; update the target DQN network according to the following formula:

[0230]

[0231] where, is a hyperparameter that needs to be manually adjusted.

[0232] In order to elaborate the MEC-driven SAGIN joint optimization method of hierarchical DRL, the application provides two specific application examples, including the key details of the implementation scheme.

[0233] Application Example 1: Wireless communication optimization in remote areas

[0234] 1) System initialization

[0235] In the wireless communication system in remote areas, the main Q network, the main Actor and Critic network are deployed first, corresponding to the high-level DQN agent and low-level DDPG agent respectively. Initialize network parameters, including learning rate, discount factor, experience replay buffer size, etc. At the same time, deploy multiple communication base stations and UAVs as air relay stations in remote areas to expand the communication coverage.

[0236] 2) Hierarchical DRL agent interaction with environment

[0237] In the interaction between hierarchical DRL agent and environment, the high-level DQN agent generates discrete actions such as content caching matrix and UE access control matrix based on the request distribution of ground UEs, the cache state of edge nodes and the channel dynamics to optimize global cache hit rate and reduce backhaul traffic, and balances exploration and utilization through the ε-greedy strategy. The low-level DDPG agent focuses on short-term resource fine allocation, dynamically adjusts the bandwidth and transmit power of UAV and satellite based on the high-level strategy output and real-time environment state (such as channel gain, UE request, resource remaining), and generates time-injected noise to enhance exploration ability, aiming to minimize local transmission delay and backhaul traffic under constraints, and updates the policy network through gradient descent to achieve dynamic optimization of hierarchical collaboration.

[0238] 3) Action execution and reward acquisition

[0239] After executing the actions generated by the high-level and low-level agents, the system obtains immediate rewards based on the performance changes of the communication network. The reward signal is calculated based on communication quality, signal stability, data transmission rate, etc. The system state moves to the next state, which involves changes in user location, fluctuations in channel conditions, and new communication requests, etc.

[0240] 4) Optimization and update

[0241] Based on the collected experience data (state, action, reward and next state), the high-level and low-level agents update their Q network, Actor and Critic network respectively. Through the training process of hierarchical DRL algorithm, the agent gradually learns the optimal strategy that adapts to the wireless communication environment in remote areas. The optimization process not only focuses on improving communication quality, but also pays attention to reducing the energy consumption of devices and UAVs to achieve sustainable communication services.

[0242] 5) Performance evaluation and tuning

[0243] The performance of the hierarchical DRL algorithm is evaluated through simulation experiments or field tests. Key indicators such as coverage, signal quality, and data transmission rate of the communication network are analyzed and compared with traditional methods. Based on the evaluation results, the algorithm parameters are adjusted and optimized to further improve the performance and stability of the communication network. Through the above steps, the hierarchical DRL MEC-driven SAGIN joint optimization method proposed in the present application can effectively solve the challenges faced by wireless communication in remote areas and improve communication quality and user experience.

[0244] Application Example 2: Emergency Rescue Communication Support in Disaster-Stricken Areas

[0245] 1) System initialization

[0246] Deploy the main DQN network and the DDPG network in the disaster-stricken area for emergency deployment. As high-level and low-level agents, initialize the relevant network parameters. At the same time, configure multiple emergency rescue communication equipment, including UAV-mounted communication relay stations, ground temporary communication base stations, etc.

[0247] 2) Interaction between agents and environment

[0248] Each communication device generates actions according to the real-time communication demand and channel condition of the disaster area, such as adjusting the communication frequency, power allocation, or UAV flight trajectory. The generation of actions not only considers the current communication strategy, but also considers environmental noise and uncertainty, so as to find more efficient communication support strategies in exploration.

[0249] 3) Action execution and reward acquisition

[0250] After executing the action, the immediate reward is obtained according to the stability of the communication link, the data transmission rate, and the delay, etc. key indicators. At the same time, update the system state to reflect the changes of the communication environment in the disaster area and the working state of the communication equipment.

[0251] 4) Network update and optimization

[0252] Using the collected data and rewards, the high-level DQN network and the low-level DDPG network in the hierarchical DRL algorithm are optimized collaboratively. The high-level DQN network is responsible for the formulation of long-term communication strategy, while the low-level DDPG network is responsible for the optimization of short-term communication resource allocation. Through continuous iteration and training, the algorithm gradually improves the stability and efficiency of the communication system, ensuring smooth communication in emergency rescue operations.

[0253] 5) Emergency rescue communication support

[0254] Through the layered DRL algorithm of the application, intelligent management and optimization of the emergency rescue communication system in the disaster area are realized. The algorithm focuses on improving the stability of the communication link and the data transmission efficiency, reducing the communication delay, and ensuring that the rescue team can obtain real-time disaster information and coordinate rescue operations. This not only improves the success rate of rescue operations, but also reduces the waste of rescue resources and casualties caused by poor communication. Through the above steps, the application provides an efficient and intelligent solution for the emergency rescue communication support in disaster-stricken areas, providing strong support for the successful implementation of rescue operations.

[0255] In these two embodiments, the MEC-driven SAGIN joint optimization method of layered DRL provides an efficient, reliable and energy-saving solution suitable for different application scenarios, from remote areas to disaster-stricken areas, demonstrating its broad application potential and technical advantages.

[0256] In order to comprehensively evaluate the performance of the application, it is compared with multiple benchmark algorithms. These benchmark algorithms have different characteristics and advantages, but are representative in handling similar problems. By comparing the performance of these algorithms under the same conditions, we can more clearly understand where the application excels and where it is deficient.

[0257] 1. UAV processing: The task offloading rate of all UEs is fixed at a preset value, and the UAV only receives UE offloaded data as a relay node, without involving cache strategy or resource allocation optimization.

[0258] 2. Base station processing: UEs offload all task data to UAV, and UAV forwards to base station for processing. This algorithm fixes the offloading rate of UEs to UAV at 100%, and the transmission strategy of UAV to base station is statically configured, without considering content caching or dynamic resource adjustment.

[0259] 3. DDPG: A single DDPG algorithm is used to independently optimize resource allocation (bandwidth, power), but the task placement and replacement strategy is randomly generated, without introducing global cache decisions of layered architecture. This algorithm only optimizes resource allocation within a single time slot, lacking high-level strategy guidance.

[0260] 4. Random access control: Based on the Time Division Multiple Access (TDMA) framework, each time slot is randomly assigned to a UE for task offloading, without considering UE request characteristics, channel state or cache state, and the access decision is completely randomized.

[0261] 5. Random UAV resource configuration: The resource configuration parameters of the UAV, such as bandwidth allocation and power control, are randomly generated (such as randomly allocating frequency bands and transmission power), without relying on any learning mechanism, for comparison of the necessity of dynamic resource optimization.

[0262] 6、The present application: it represents the proposed multi-time scale hierarchical DRL algorithm.

[0263] In Figure 4 the figure, we show the convergence of DQN under different learning rates. Based on three performance indicators: convergence speed, convergence stability and the reward value obtained to evaluate the convergence performance. From the three groups of data can be observed, the system reward value initially presents a sharp upward trend, and tends to be stable around the 300th training cycle - this convergence characteristics under different learning rate combinations remain consistent, indicating that the proposed algorithm has the characteristics of fast convergence.

[0264] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any modification, equivalent replacement and improvement within the technical range disclosed by the present application and within the spirit and principle of the present application shall be covered within the protection scope of the present application.

Claims

1. A method for air-space-ground caching and resource optimization based on hierarchical intelligent agents, characterized in that, Includes the following steps: a) Initialize the parameters, learning rate, discount factor, experience replay buffer size, and noise parameters of the high-level deep Q-network and the low-level deep deterministic policy gradient network, and set the number of iterations and the number of time slots included in each iteration; b) At the beginning of each iteration cycle, obtain the current high-level state of the system and use the high-level deep Q network to generate content cache placement decisions and access control strategies; c) Determine whether the high-level strategy satisfies the system constraints. If it does, call the low-level deep deterministic policy gradient network in each time slot of the cycle to generate and execute bandwidth and power allocation actions based on the current low-level state. If it does not satisfy the constraints, skip the low-level optimization. d) Calculate the low-level immediate reward and the high-level periodic reward based on the action execution results, and store the corresponding state, action, reward, and next state tuples into their respective experience replay buffers; e) Randomly extract small batches of experience from the buffer, update the main network parameters, and synchronize the target network parameters according to the soft update strategy; f) Repeat steps b) to e) until the iteration ends to obtain the optimal content caching, access control, bandwidth and power allocation strategy.

2. The method according to claim 1, characterized in that, The initialization in step a) further includes configuring network layout parameters such as the number of ground user terminals, the number of drones, the satellite and drone transmission power, the content library size, and the single content size.

3. The method according to claim 1, characterized in that, The high-level state includes the following information: content request indication for each user, cached content index and remaining cache capacity of each edge computing server, time-varying channel gain between the drone and the user, path loss between the user and the satellite, horizontal distance between the user and the drone, drone altitude, and distance between drones.

4. The method according to claim 1, characterized in that, When the high-level strategy becomes infeasible, the high-level deep Q-network is immediately penalized and enters the next cycle, while the low-level deep deterministic policy gradient network does not perform any resource allocation action.

5. The method according to claim 1, characterized in that, The low-level reward is positive when the action satisfies all constraints, otherwise it is set to a fixed negative constant to guide the low-level agent to select actionable actions in subsequent time slots.

6. A hierarchical intelligent agent-based air-space-ground caching and resource optimization system, characterized in that, include: The system initialization module is used to perform parameter and network layout initialization in step a) of claim 1; The high-level intelligent agent module is used to execute the content caching and access control decision in step b) of claim 1; A low-level intelligent agent module is used to perform the bandwidth and power allocation decision in step c) of claim 1; The action execution module is used to implement high-level and low-level actions and collect environmental feedback; The reward calculation module is used to calculate the low-level immediate reward and the high-level periodic reward in step d) of claim 1. The experience replay module is used to store and extract experience tuples; The network update module is used to update the main network and perform soft updates to the target network based on sampling experience.

7. The system according to claim 6, characterized in that, The action execution module can continuously control the bandwidth and transmission power between the user and the drone, and discretely control the placement and replacement of content cache and user access control.

8. The system according to claim 6, characterized in that, The network update module adopts a soft update strategy, setting the target network parameters as a weighted average of the main network parameters and the existing target network parameters.

9. A computer-readable storage medium having a computer program stored thereon, wherein when executed by a processor, the computer program causes the processor to perform the steps of the method according to any one of claims 1 to 5.

10. A data processing apparatus, comprising a processor and a memory, wherein the memory contains built-in instructions that, when executed by the processor, cause the apparatus to perform the method according to any one of claims 1 to 5.

Images