A space-air-ground joint optimization method and system based on layered agents
Patent Information
- Application Number
- CN202611021657.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-09
- Publication Date
- 2026-09-11
AI Technical Summary
1、缓存策略单一
Smart Images
Figure CN122742003A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of communication technology, and in particular to a method and system for joint air-space-ground optimization based on hierarchical intelligent agents. Background Technology
[0002] Space-Air-Ground Integrated Network (SAGIN), as an important direction for 6G development, integrates low-Earth orbit satellites, unmanned aerial vehicles (UAVs), and ground base stations, providing users with edge services characterized by wide coverage, low latency, and near-end computing. With the rapid growth in demand for multimedia services, emergency communications, and ultra-high-definition video, content requests in the system are characterized by high concurrency and strong dynamism. Therefore, efficient content caching and scheduling have become a key research focus.
[0003] The existing technical solutions have the following main shortcomings: 1. Limited caching strategies. Most methods employ static caching or simple replacement strategies, which struggle to adapt to rapid changes in user requests, resulting in low cache hit rates and increased backhaul communication burden.
[0004] 2. Inefficient resource allocation. Existing algorithms are mostly based on fixed bandwidth configurations and lack adaptive optimization for wireless link status and user distribution, which can easily lead to resource waste or link congestion.
[0005] 3. Rigid access control. Most studies adopt static access or single-layer optimization, lacking joint scheduling of satellites, drones, and ground nodes, resulting in inflexible user access choices and affecting overall service quality.
[0006] 4. Difficulty in handling highly dynamic environments. In scenarios where low-Earth orbit satellites and swarm drones coexist, channel conditions and user requests change rapidly. Existing rule-based or single-agent methods are prone to dimensionality explosion or convergence lag, making it difficult to obtain the global optimum in real time.
[0007] In summary, existing technologies cannot effectively solve the joint optimization problem of content caching, access control, and multi-dimensional resource scheduling. There is an urgent need to propose a new method to improve system stability and user experience while ensuring low latency and high energy efficiency. Summary of the Invention
[0008] To address the aforementioned issues, this disclosure provides a method and system for joint air-space-ground optimization based on hierarchical intelligent agents.
[0009] Firstly, a space-air-ground joint optimization method based on hierarchical intelligent agents includes: Construct an integrated air-space-ground network environment that includes low-orbit satellites, drones, and ground users, and initialize high-level intelligent agents deployed on satellites and low-level intelligent agents deployed at the edge. The high-level intelligent agent acquires global state information and generates macro-level action strategies based on the decision cycle as the time scale. The macro-level action strategies include content cache placement and replacement strategies as well as user access control strategies. The macro-action strategy output by the high-level intelligent agent is logically gated and verified. If the macro-action strategy meets the preset constraints, it is sent down to the low-level intelligent agent; otherwise, a penalty is given. Based on the macroscopic action strategy and local observation information, the low-level agent outputs and executes micro-resource control actions on a time scale, using time slots as the time scale; micro-resource control actions include bandwidth allocation and power allocation. The reward value is calculated based on the execution result of the micro-resource control action, the environmental state is updated, the experience data is stored in the experience replay buffer, and the network parameters of the high-level agent and the low-level agent are updated based on the mini-batch sampling mechanism; the iteration is repeated until the average value of the reward value tends to stabilize, and the high-level agent and the low-level agent that have been trained and converged are obtained. Optimize based on the macroscopic action strategies and microscopic resource control actions output by the high-level and low-level intelligent agents respectively.
[0010] Furthermore, the high-level intelligent agent employs a deep Q-network (DQN) to obtain global state information, specifically including: Collect data on the distribution of requests from ground users for different types of content; Get the list of currently cached content and remaining storage space for each edge node; Obtain link distance and channel status information between all nodes in the network.
[0011] Furthermore, content caching placement and replacement strategies are used to determine the deployment location of content on each edge node; The content caching placement and replacement strategy is generated as follows: the content to be cached is determined based on its popularity; if the node storage space is insufficient, an instruction to remove the content with the lowest popularity is generated.
[0012] Furthermore, user access control policies are used to determine the specific edge nodes for user access in various locations. The user access control policy is generated by generating a matrix of connection relationships between users and nodes.
[0013] Furthermore, the preset constraints for logic gating verification include: A single user can only access one edge node at a time; The total amount of cached content on edge nodes shall not exceed their maximum storage capacity.
[0014] Furthermore, the low-level agents are deployed on satellites and various drones, employing a multi-agent deep deterministic policy gradient network (MADDPG), which includes a policy network and a value network. The acquisition of local observation information specifically includes: the user access policy determined by the higher layer, the locally cached content information, the user request list of local services, and the real-time channel gain between the local system and the user.
[0015] Furthermore, the specific actions of micro-resource control include: determining the communication bandwidth and signal transmission power allocated to each user within its service range.
[0016] Furthermore, the reward value is calculated by taking the reciprocal of the weighted sum of the average content retrieval latency and the backhaul link traffic consumption.
[0017] Furthermore, the network parameters of high-level and low-level agents are updated based on a mini-batch sampling mechanism, including: A preset number of experience samples are randomly selected from the experience replay buffer to form a small batch of data; For high-level agents, the mean square error between the predicted long-term reward and the target reward is calculated as the loss function, and the network parameters are updated using the gradient descent algorithm. For low-level agents, the deterministic policy gradient algorithm is used to update the policy network parameters and minimize the error between the predicted value of the value network and the actual return. A target network soft update mechanism is adopted to synchronize the parameters of the main network to the target network according to a preset smoothing parameter ratio.
[0018] Furthermore, the environment state is updated, and the experience data is stored in the experience replay buffer, including: Update the cache status, and update the list of currently stored content and remaining space of the node according to the cache placement and replacement strategy. The physical state is updated randomly based on user requests according to the popularity distribution pattern; the drone's position and channel gain are dynamically updated based on the motion model and fading model. The experience data storage combines the current state, the macro-level action strategy and micro-level resource control action executed, the reward obtained, and the state at the next moment into a single piece of experience data, which is then stored in the experience replay buffer.
[0019] Furthermore, the time scale of the decision-making cycle is larger than the time scale of the time slot; The high-level intelligent agent maintains its policy unchanged across multiple time slots; The lower-level agent allocates resources in each time slot based on the real-time changing channel state.
[0020] Secondly, a hierarchical intelligent agent-based air-space-ground joint optimization system includes: The system comprises an initialization unit, a high-level decision-making unit, a logic verification unit, a low-level execution unit, a training and optimization unit, and an optimization execution unit. The initialization unit is used to build an integrated air-space-ground network environment that includes low-orbit satellites, drones, and ground users, and to initialize the high-level intelligent agents deployed on the satellite and the low-level intelligent agents deployed at the edge. The high-level decision-making unit is used to obtain global state information through high-level intelligent agents and generate macro-action strategies with the decision cycle as the time scale. The macro-action strategies include content cache placement and replacement strategies as well as user access control strategies. The logic verification unit is used to perform logic gating verification on the macro action policy output by the high-level intelligent agent. If the macro action policy meets the preset constraints, the macro action policy is sent down to the low-level intelligent agent; otherwise, a penalty feedback is given. The low-level execution unit is used to output and execute micro-resource control actions based on the macro-action strategies and local observation information issued by the low-level intelligent agent, with time slots as the time scale; micro-resource control actions include bandwidth allocation and power allocation. The training optimization unit is used to calculate the reward value based on the execution result of the micro-resource control action, update the environment state, store the experience data in the experience replay buffer, and update the network parameters of the high-level agent and the low-level agent based on the mini-batch sampling mechanism; repeat the iteration until the average value of the reward value tends to stabilize, and obtain the high-level agent and the low-level agent that have been trained and converged. The optimization execution unit is used to optimize scheduling based on the macro-action strategies and micro-resource control actions output by the high-level and low-level intelligent agents, respectively.
[0021] This disclosure includes at least the following beneficial effects: This disclosure effectively solves the coordination problem of content caching and resource scheduling in a highly dynamic environment through joint optimization at hierarchical spatiotemporal scales, significantly reducing content retrieval latency and backhaul link traffic, and improving system resource utilization and user experience.
[0022] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description and the accompanying drawings. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a schematic diagram of the optimization method flow according to an embodiment of the present disclosure; Figure 2 A schematic diagram of an optimized system architecture for an embodiment of this disclosure. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0026] like Figure 1 As shown, a space-air-ground joint optimization method based on hierarchical intelligent agents includes: Construct an integrated air-space-ground network environment that includes low-orbit satellites, drones, and ground users, and initialize high-level intelligent agents deployed on satellites and low-level intelligent agents deployed at the edge. The high-level intelligent agent acquires global state information and generates macro-level action strategies based on the decision cycle as the time scale. The macro-level action strategies include content cache placement and replacement strategies as well as user access control strategies. The macro-action strategy output by the high-level intelligent agent is logically gated and verified. If the macro-action strategy meets the preset constraints, it is sent down to the low-level intelligent agent; otherwise, a penalty is given. Based on the macroscopic action strategy and local observation information, the low-level agent outputs and executes micro-resource control actions on a time scale, using time slots as the time scale; micro-resource control actions include bandwidth allocation and power allocation. The reward value is calculated based on the execution result of the micro-resource control action, the environmental state is updated, the experience data is stored in the experience replay buffer, and the network parameters of the high-level agent and the low-level agent are updated based on the mini-batch sampling mechanism; the iteration is repeated until the average value of the reward value tends to stabilize, and the high-level agent and the low-level agent that have been trained and converged are obtained. Optimize based on the macroscopic action strategies and microscopic resource control actions output by the high-level and low-level intelligent agents respectively.
[0027] In practice: A two-layer architecture of high-level intelligent agents (satellite end) and low-level intelligent agents (edge end) is adopted to decompose the optimization cycle into two time scales: decision cycle and time slot, forming closed-loop control.
[0028] System initialization and parameter configuration: (1) During the system startup phase, the network environment and intelligent agents are initialized and configured: Network environment initialization: Set the number of low-Earth orbit satellites, multiple drones, and ground users; set the total size of the content library and the data size of each piece of content; set communication environment parameters, including carrier frequency, speed of light, and probability parameters for line-of-sight and non-line-of-sight transmission.
[0029] (2) Agent network initialization: High-level agent initialization: Deploy a Deep Q Network (DQN) on the satellite and initialize the weight parameters of its main network and target network (usually using a normal distribution with a mean of zero for random initialization).
[0030] Low-level agent initialization: Deploy a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) network on the satellite and each UAV to initialize their policy network (Actor), value network (Critic), and corresponding target network.
[0031] (3) Hyperparameter configuration: Set the size of the experience replay buffer, the size of the batch sampling, the discount factor (to measure the importance of future rewards), the learning rate, and the smoothing parameters for soft updates of the target network.
[0032] High-level intelligent agents generate macroscopic strategies (periodic decision-making): (1) At the beginning of each decision cycle, the high-level agent generates discrete policies based on the global state: Global state awareness: The high-level intelligent agent collects the current state of the entire network, including: the distribution of requests for different content from ground users, the list of content currently cached by each edge node, the remaining storage space of each node, and the link distance and channel state information between all network nodes.
[0033] (2) Macroscopic action generation: Based on the above states, the high-level agent outputs two types of discrete strategies: Cache placement and replacement strategy: This determines which content to place on which edge nodes. If a node's storage space is insufficient, a content removal instruction is generated based on the principle of least popular content, freeing up space for new content.
[0034] User access control strategy: Determines which edge node (satellite or drone) each ground user will connect to, and generates a connection relationship matrix between users and nodes.
[0035] (3) Logical gating judgment: The system checks whether the policy generated by the higher layer meets the basic constraints (such as a user can only access one node and the cache capacity does not exceed the limit). Only when the policy meets the constraints will the access policy be passed as input to the lower layer agent to trigger the execution of step three; otherwise, the system will give the higher layer agent a penalty reward and skip the subsequent resource allocation steps.
[0036] Low-level intelligent agents perform micro-resource allocation (time-slot level control): When the high-level strategy is feasible, the low-level agents of each edge node perform continuous resource allocation in each time slot: (1) Local observation acquisition: Each low-level agent acquires its own local observation information, including: the user access policy determined by the high level, the locally cached content information, the user request list of local services, and the real-time channel gain between the local agent and the user.
[0037] (2) Micro-action generation: Based on local observation, the low-level agent outputs continuous resource allocation actions: Bandwidth allocation: Determines the communication bandwidth allocated to each user within its service range.
[0038] Power allocation: Determines the amount of signal transmission power allocated to each user.
[0039] (3) Action execution: Each node sends data to the user based on the generated bandwidth and power values.
[0040] Environmental interaction and reward calculation: After executing the above combined strategy, the system calculates the reward based on the feedback and updates the status: (1) Real-time reward calculation: The optimization goal of the system is to minimize latency and traffic. Therefore, the reward value is designed as the reciprocal of the weighted sum of the average content retrieval latency and the traffic consumed by the backhaul link. The lower the latency and the less backhaul traffic, the greater the reward value.
[0041] (2) Environmental status update: Cache status: Update the list of currently stored content and remaining space of the node based on cache placement and replacement actions.
[0042] Physical status: User requests are refreshed randomly according to the popularity distribution pattern; UAV position and channel gain are dynamically updated according to the motion model and fading model.
[0043] (3) Experience storage: Combine the current state, the joint action performed, the reward obtained, and the state at the next moment into a piece of experience data and store it in the experience replay buffer.
[0044] Network parameter updates based on mini-batch sampling: Train the neural network using stored empirical data and optimize the strategy: (1) Mini-batch sampling: A certain number (e.g., 64) of experience samples are randomly selected from the experience replay buffer to form a mini-batch of data. The purpose of this random sampling mechanism is to break the temporal correlation between data and improve the stability of training.
[0045] (2) High-level network update: Calculate the mean square error (loss function) between the long-term return value (Q value) predicted by the high-level network and the target return value, and update the network parameters through the gradient descent algorithm so that it can more accurately evaluate the value of the macro strategy.
[0046] (3) Low-level network updates: Value network update: Minimize the error between predicted value and actual return.
[0047] Policy network update: Using a deterministic policy gradient algorithm, the policy network parameters are adjusted along the gradient direction indicated by the value network, thereby generating a bandwidth and power allocation scheme that can obtain higher rewards.
[0048] (4) Target network soft update: The soft update mechanism is adopted to synchronize the parameters of the main network to the target network at a very small proportion (smoothing parameters) to ensure smooth convergence of the training process.
[0049] Iterate through the training process until convergence: Repeat the above steps, continuously interacting with the environment and updating the network parameters. When the average reward obtained by the system tends to stabilize and no longer fluctuates drastically, the model training is considered to have converged, and the optimal joint optimization strategy is finally obtained.
[0050] The detailed process is as follows: S101: System initialization, Initialize the parameters of the primary Q-network and target Q-network of the high-level DQN network; initialize the parameters of the primary Actor network, primary Critic network, and corresponding target network of the low-level DDPG network. Set the iteration count L and the number of time slots T in each iteration cycle. Configure the DQN learning rate, the DDPG Critic learning rate, the Actor learning rate, the discount factor γ, the experience replay buffer size, the mini-batch size, and the OU noise parameters used for action exploration. Simultaneously initialize network layout parameters, including the number of ground users (UEs), the number of unmanned aerial vehicles (UAVs), the content library size, the size of a single piece of content, and the bandwidth and transmission power of UAVs and satellites.
[0051] S102: The high-level intelligent agent perceives the state and generates macroscopic actions. At the start of each decision cycle, the system state of the high-level DQN agent is initialized (including user request distribution, cache status of each node, remaining storage space, link status, etc.). Based on the current state, the high-level DQN agent selects high-level actions using the ε-greedy strategy, and outputs cache placement decisions (including cache replacement strategies) and user access control strategies.
[0052] S103: Constraint Checking and Temporary Reward Generation The system executes the action of generating high-level intelligent agents, determines whether basic constraints are met (such as unique user access, cache capacity not exceeding limits, etc.), and generates temporary high-level reward indicators accordingly (if the constraints are not met, a negative reward is given). If the constraints are met, the system enters the low-level optimization process (S104-S108) for each time slot in the current cycle; otherwise, it skips the low-level optimization and directly enters S109.
[0053] S104: Low-level agent initialization and micro-action generation Initialize the local observation state of the lower-level DDPG agent (including access policies issued by higher layers, local cache contents, service user request list, and real-time channel gain). The DDPG agent outputs continuous actions through its Actor network, including bandwidth resources allocated to each user and transmit power allocation for UAVs and satellites. Simultaneously, a probabilistic random sampling method is used to convert the continuous output values into executable binary access control (if necessary).
[0054] S105: Low-level action execution and reward acquisition The lower-level agent performs the generated bandwidth allocation and power allocation actions. The environment calculates the lower-level immediate reward based on the execution results (a weighted sum of the system's average content acquisition latency and the backhaul link's consumed traffic) and transforms the lower-level state into the state of the next time slot.
[0055] S106: Low-level experience storage, Store the current low-level experience data (state, action, reward, next state) into the experience replay buffer. If the buffer is full, delete the oldest experience to make room.
[0056] S107: Low-level TD error calculation, Randomly sample a small batch of experiences M from the experience replay buffer to compute the temporal difference (TD) objective and TD error of the low-level DDPG agent.
[0057] S108: Low-level network parameter update The calculated TD error is used to update the main low-level Actor network and Critic network; then, a soft update method is used to update the parameters of the target Actor network and Critic network based on the updated main network parameters.
[0058] S109: High-level reward acquisition and state transition After the low-level optimization of all time slots in the current cycle is completed, the DDPG agent feeds back its accumulated reward to the higher-level DQN agent to form a high-level reward. At the same time, the environmental state (buffer state, user distribution, channel conditions, etc.) is transitioned to the state of the next decision cycle.
[0059] S110: High-level experience storage, The current high-level experience data (state, high-level action, high-level reward, next state) is stored in the experience replay buffer. If the buffer is full, the oldest experience is deleted to make room. Then, a small batch of experiences is sampled from the buffer, and the state and action are used as inputs to the DQN main network to obtain the estimated Q value.
[0060] S111: Update of higher-level network parameters Calculate the TD target and TD error of the high-level DQN agent, update the main Q network parameters, and then update the target Q network parameters using a soft update method.
[0061] S112: Iterate training until convergence. Repeat steps S102 to S111 for multiple rounds of iterative training. Select the optimal action based on the Q-value of the corresponding action in each state to obtain the maximum cumulative benefit, and finally obtain the optimal: content cache placement and replacement strategy, joint resource allocation strategy between users and drones / satellites, and user access control strategy.
[0062] This disclosure employs a layered intelligent agent architecture, utilizing a deep Q-network at the high layer for content caching, content replacement, and access control, while employing a deep deterministic policy gradient at the low layer for bandwidth allocation. This approach balances global optimization with rapid local response, effectively reducing content retrieval latency and backhaul traffic, thereby improving system stability and user experience.
[0063] This disclosure employs an experience replay and target network update mechanism, which ensures stable convergence during the training process and avoids policy oscillations. This enables the system to maintain high robustness and resource utilization even in complex dynamic environments, exhibiting strong adaptability.
[0064] like Figure 2 As shown, a hierarchical intelligent agent-based air-space-ground joint optimization system includes: The system comprises an initialization unit, a high-level decision-making unit, a logic verification unit, a low-level execution unit, a training and optimization unit, and an optimization execution unit. The initialization unit is used to build an integrated air-space-ground network environment that includes low-orbit satellites, drones, and ground users, and to initialize the high-level intelligent agents deployed on the satellite and the low-level intelligent agents deployed at the edge. The high-level decision-making unit is used to obtain global state information through high-level intelligent agents and generate macro-action strategies with the decision cycle as the time scale. The macro-action strategies include content cache placement and replacement strategies as well as user access control strategies. The logic verification unit is used to perform logic gating verification on the macro action policy output by the high-level intelligent agent. If the macro action policy meets the preset constraints, the macro action policy is sent down to the low-level intelligent agent; otherwise, a penalty feedback is given. The low-level execution unit is used to output and execute micro-resource control actions based on the macro-action strategies and local observation information issued by the low-level intelligent agent, with time slots as the time scale; micro-resource control actions include bandwidth allocation and power allocation. The training optimization unit is used to calculate the reward value based on the execution result of the micro-resource control action, update the environment state, store the experience data in the experience replay buffer, and update the network parameters of the high-level agent and the low-level agent based on the mini-batch sampling mechanism; repeat the iteration until the average value of the reward value tends to stabilize, and obtain the high-level agent and the low-level agent that have been trained and converged. The optimization execution unit is used to optimize scheduling based on the macro-action strategies and micro-resource control actions output by the high-level and low-level intelligent agents, respectively.
[0065] In practice: This paper constructs an integrated air-space-ground network encompassing low-Earth orbit satellites, drones, and ground users, employing a two-layer architecture of high-level agents at the satellite end and low-level agents at the edge. The high-level agents utilize a deep Q-network to generate content caching and replacement strategies and user access control strategies based on global state and at the decision cycle scale. The low-level agents utilize a multi-agent deep deterministic policy gradient network to perform micro-resource allocation of bandwidth and power based on local observations and at the time slot scale. The system verifies the legality of the strategies through a logical gating mechanism and iteratively updates network parameters until convergence based on reward feedback and experience replay mechanisms. This disclosure effectively solves the coordination problem of content caching and resource scheduling in highly dynamic environments through hierarchical spatiotemporal joint optimization, significantly reducing content acquisition latency and backhaul link traffic, and improving system resource utilization and user experience.
[0066] This disclosure utilizes a cache placement and replacement strategy generated by a high-level intelligent agent based on content popularity and global state. This strategy enables the precise pre-deployment of popular content to the edge nodes closest to users. This proactive caching mechanism significantly improves cache hit rate and reduces the backhaul link traffic required for users to request content from the core network, thereby significantly reducing the average content retrieval latency for users.
[0067] This disclosure employs a two-layer intelligent agent architecture: a high-level (satellite-side) and a low-level (edge-side) layer. The optimization process is decomposed into two time scales: a decision cycle (macroscopic) and a time slot (microscopic). The high-level layer is responsible for long-cycle global resource scheduling, while the low-level layer is responsible for short-cycle real-time resource allocation. This layered design ensures both the optimality of the global strategy and the ability to respond quickly to instantaneous changes in channel state, effectively solving the problem that a single intelligent agent cannot simultaneously balance long-term gains and real-time control.
[0068] This disclosure employs a multi-agent deep deterministic policy gradient (MADDPG) algorithm for low-level agents, enabling dynamic adjustment of bandwidth and power allocation based on local observation information (such as real-time channel gain). Compared to traditional fixed resource allocation methods, this approach achieves on-demand allocation of wireless resources, avoiding resource waste and link congestion, and improving spectral efficiency and power utilization.
[0069] To address the highly dynamic nature of low-Earth orbit satellite and UAV network topologies, this disclosure introduces a logic gating mechanism and a target network soft update mechanism. The logic gating ensures the physical feasibility of the strategy and prevents illegal operations; the soft update mechanism guarantees smooth convergence during the training process. This enables the system to maintain high robustness in complex dynamic environments, adapt to environmental changes, and maintain stable performance.
[0070] This disclosure breaks through the limitations of traditional technologies where caching, access, and resource allocation are independent of each other. By having a high-level intelligent agent make unified decisions on user access and caching strategies, and a low-level intelligent agent cooperates in executing resource allocation, it achieves cross-layer joint optimization of the application layer, network layer, and physical layer, thereby maximizing the overall performance of the integrated air-space-ground network.
[0071] In scenario example one, in a content acceleration scenario in a remote area, several drones are deployed as relays and computing nodes, while low-Earth orbit satellites handle high-level scheduling. The high-level intelligent agent generates periodic content caching, content replacement, and access control policies based on user request distribution and cache status; the low-level intelligent agent dynamically allocates bandwidth within each time slot based on channel status and resource availability. Through layered optimization, the cache hit rate is effectively improved, and content retrieval latency and backhaul communication volume are reduced.
[0072] Scenario Example 2: In a disaster emergency communication scenario, drones and satellites are temporarily deployed as emergency network nodes. High-level agents are responsible for task content placement, content replacement, and access path control, while low-level agents perform time-slot-level bandwidth allocation to ensure real-time transmission of high-priority tasks and controlled energy consumption. This method can quickly restore communication in disaster areas, ensuring the smooth progress of rescue operations.
[0073] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A space-air-ground joint optimization method based on hierarchical agents, characterized in that, include: Construct an integrated air-space-ground network environment that includes low-orbit satellites, drones, and ground users, and initialize high-level intelligent agents deployed on satellites and low-level intelligent agents deployed at the edge. The high-level intelligent agent acquires global state information and generates macro-level action strategies based on the decision cycle as the time scale. The macro-level action strategies include content cache placement and replacement strategies as well as user access control strategies. The macro-action strategy output by the high-level intelligent agent is logically gated and verified. If the macro-action strategy meets the preset constraints, it is sent down to the low-level intelligent agent; otherwise, a penalty is given. Based on the macroscopic action strategy and local observation information, the low-level agent outputs and executes micro-resource control actions on a time scale, using time slots as the time scale; micro-resource control actions include bandwidth allocation and power allocation. The reward value is calculated based on the execution result of the micro-resource control action, the environmental state is updated, the experience data is stored in the experience replay buffer, and the network parameters of the high-level agent and the low-level agent are updated based on the mini-batch sampling mechanism; the iteration is repeated until the average value of the reward value tends to stabilize, and the high-level agent and the low-level agent that have been trained and converged are obtained. Optimize based on the macroscopic action strategies and microscopic resource control actions output by the high-level and low-level intelligent agents respectively.
2. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, The high-level intelligent agent uses a deep Q-network (DQN) to obtain global state information, specifically including: Collect data on the distribution of requests from ground users for different types of content; Get the list of currently cached content and remaining storage space for each edge node; Obtain link distance and channel status information between all nodes in the network.
3. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, Content caching placement and replacement strategies are used to determine the deployment location of content on each edge node; The content caching placement and replacement strategy is generated as follows: the content to be cached is determined based on its popularity; if the node storage space is insufficient, an instruction to remove the content with the lowest popularity is generated.
4. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, User access control policies are used to determine the specific edge nodes for user access in various locations. The user access control policy is generated by generating a matrix of connection relationships between users and nodes.
5. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, The preset constraints for logic gating verification include: A single user can only access one edge node at a time; The total amount of cached content on edge nodes shall not exceed their maximum storage capacity.
6. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, The low-level agents are deployed on satellites and various drones, and adopt the Multi-Agent Deep Deterministic Policy Gradient Network (MADDPG), which includes a policy network and a value network. The acquisition of local observation information specifically includes: the user access policy determined by the higher layer, the locally cached content information, the user request list of local services, and the real-time channel gain between the local system and the user.
7. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, The specific actions of micro-resource control include: determining the communication bandwidth and signal transmission power allocated to each user within its service range.
8. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, Calculate the reward value by taking the reciprocal of the weighted sum of average content retrieval latency and backhaul link traffic consumption.
9. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, The network parameters of high-level and low-level agents are updated based on a mini-batch sampling mechanism, including: A preset number of experience samples are randomly selected from the experience replay buffer to form a small batch of data; For high-level agents, the mean square error between the predicted long-term reward and the target reward is calculated as the loss function, and the network parameters are updated using the gradient descent algorithm. For low-level agents, the deterministic policy gradient algorithm is used to update the policy network parameters and minimize the error between the predicted value of the value network and the actual return. A target network soft update mechanism is adopted to synchronize the parameters of the main network to the target network according to a preset smoothing parameter ratio.
10. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, Update the environment state and store the experience data in the experience replay buffer, including: Update the cache status, and update the list of currently stored content and remaining space of the node according to the cache placement and replacement strategy. The physical state is updated randomly based on user requests according to the popularity distribution pattern; the drone's position and channel gain are dynamically updated based on the motion model and fading model. The experience data storage combines the current state, the macro-level action strategy and micro-level resource control action executed, the reward obtained, and the state at the next moment into a single piece of experience data, which is then stored in the experience replay buffer.
11. The air-space-ground joint optimization method based on hierarchical intelligent agents according to claim 1, characterized in that, The timescale of the decision-making cycle is larger than the timescale of the time slot; The high-level intelligent agent maintains its policy unchanged across multiple time slots; The lower-level agent allocates resources in each time slot based on the real-time changing channel state.
12. A layered-agent-based space-air-ground joint optimization system, characterized in that, include: The system comprises an initialization unit, a high-level decision-making unit, a logic verification unit, a low-level execution unit, a training and optimization unit, and an optimization execution unit. The initialization unit is used to build an integrated air-space-ground network environment that includes low-orbit satellites, drones, and ground users, and to initialize the high-level intelligent agents deployed on the satellite and the low-level intelligent agents deployed at the edge. The high-level decision-making unit is used to obtain global state information through high-level intelligent agents and generate macro-action strategies with the decision cycle as the time scale. The macro-action strategies include content cache placement and replacement strategies as well as user access control strategies. The logic verification unit is used to perform logic gating verification on the macro action policy output by the high-level intelligent agent. If the macro action policy meets the preset constraints, the macro action policy is sent down to the low-level intelligent agent; otherwise, a penalty feedback is given. The low-level execution unit is used to output and execute micro-resource control actions based on the macro-action strategies and local observation information issued by the low-level intelligent agent, with time slots as the time scale; micro-resource control actions include bandwidth allocation and power allocation. The training optimization unit is used to calculate the reward value based on the execution result of the micro-resource control action, update the environment state, store the experience data in the experience replay buffer, and update the network parameters of the high-level agent and the low-level agent based on the mini-batch sampling mechanism; repeat the iteration until the average value of the reward value tends to stabilize, and obtain the high-level agent and the low-level agent that have been trained and converged. The optimization execution unit is used to optimize scheduling based on the macro-action strategies and micro-resource control actions output by the high-level and low-level intelligent agents, respectively.