Star-ground fusion network information age optimization method based on multi-agent hierarchical deep reinforcement learning

Abstract

The application discloses a star-ground fusion network information age optimization method based on multi-agent layered deep reinforcement learning, relates to the technical field of communication, and comprises the following steps: S1, initializing network parameters; S2, initializing and setting T, Ns, F, and t=1; S3, if t%Ns=0, executing S4; otherwise, executing S5; S4, in the upper-layer DQN, the current state of each agent is selected by using a greedy strategy to select an action, a state transition based on discrete spectrum allocation is obtained, and the state transition is stored in a sample to update theta and theta'; S5, for each agent, an observation state of the lower layer is input to PPO, a state transition based on continuous power allocation is obtained, and the state transition is stored in a sample to update zeta and if t%F=0, an old policy network is updated; S6, if t

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method for optimizing the age of information in satellite-ground fusion networks based on multi-agent hierarchical deep reinforcement learning. Background Technology

[0002] With the continuous growth of global communication demands, satellite communication, due to its advantages such as being unrestricted by geographical location and having wide coverage, will play a crucial role in achieving the goal of global 6G network coverage. However, satellite communication also faces many technical challenges, including complex and variable channel environments, difficulties in modeling, and high communication latency. Information age (AoI) is a key indicator for measuring the freshness of information, defined as the time interval (ms) from information generation to receiver reception. Optimizing resource allocation schemes based on AoI is of great significance for ensuring data freshness and improving system performance.

[0003] In existing technologies, research on optimizing the communication quality of space-to-ground networks is mostly limited to optimizing traditional communication indicators, such as rate and energy efficiency, with few patent studies addressing the optimization of satellite system information age. These research directions only meet the reliability indicators of communication systems but cannot guarantee the immediacy of communication. For example, patent document CN119364423A discloses a congestion control method for space-to-ground networks based on deep reinforcement learning. By acquiring data information from multiple target space-to-ground networks and setting state space, action space, and reward function, as well as implementing state prediction, reward redistribution, and optimization strategies, it achieves congestion control for space-to-ground networks based on deep reinforcement learning. The above scheme uses a single-layer deep reinforcement learning network architecture to solve non-convex optimization problems. However, such single-layer network architectures can only handle the optimization of a single type of resource and cannot cope with the challenges of mixed action spaces brought about by jointly optimizing multiple resources. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a method for optimizing the information age of satellite-ground fusion networks based on multi-agent hierarchical deep reinforcement learning. This method overcomes the challenges of complex and dynamic satellite channels and finds a joint spectrum and power allocation scheme that can minimize the information age of the system.

[0005] The technical solution of this invention is implemented as follows:

[0006] A method for optimizing the information age of a satellite-ground fusion network based on multi-agent hierarchical deep reinforcement learning, wherein the satellite-ground fusion network includes a multi-beam low-Earth orbit (LEO) satellite, and the LEO includes M orthogonal frequency band resource blocks (RBs), with each RB occupied by a ground cell base station; the method includes the following steps:

[0007] Step S1: Initialize the network parameters θ, θ′, and correspond to the main network, target network, new policy network, old policy network, and evaluation network respectively; one RB corresponds to one agent;

[0008] In the field of wireless communication, in a communication system, an RB is the smallest unit of frequency-domain and time-domain resource allocation. An RB usually consists of 12 subcarriers in the frequency domain, each subcarrier having a bandwidth of 15 kHz and a total bandwidth of 180 kHz; in the time domain, it corresponds to one time slot (0.5 ms), containing 7 OFDM symbols (normal cyclic prefix mode) or 6 symbols (extended cyclic prefix mode).

[0009] Step S2: Initialize the experience pool and Set the total number of training rounds T, the allocation frequency Ns, and the frequency F for updating the old policy network; initialize t = 1;

[0010] Step S3: If t % Ns = 0, then execute Step S4; otherwise, execute Step S5;

[0011] Step S4: For the current state of each agent Use the greedy strategy (ε - greedy) to select an action α

[0015] ,

[0014] , , , ,

[0013] (t); m represents the m-th agent; obtain the data representing the state transition process, and store it in ; randomly select multiple samples from to update θ and θ′; α m (t) represents the spectrum allocation strategy of the m-th agent; execute Step S6;

[0012] Step S5: For each agent, input the lower-layer observation state into the PPO new policy network, and output the action probability distribution; obtain the data and store it in ; randomly select multiple samples from to update ζ and p <00​​​​​​​​​​​To adapt to the changes in different beam environments, this solution proposes a multi-agent hierarchical DQN-PPO (MHDP) algorithm. Among them, the main network θ and the target network θ′ (which can also be called the Q network and the target Q network respectively), both have the same structure but different update frequencies, and they together constitute the DQN network, that is, the deep Q network, belonging to the upper network; the new policy network The old policy network and the evaluation network constitute the proximal policy optimization PPO, belonging to the lower network. The state corresponds to the upper (upper) network, and the state corresponds to the lower (lower) network. Correspondingly, α m (t) is the action selected by the state ; p m (t) is the action selected by the state .

[0016] Since it is necessary to simultaneously optimize the mixed non-linear programming problem of the discrete spectrum allocation strategy α and the continuous power allocation strategy p, it is very difficult to solve and obtain the global optimal solution using traditional optimization algorithms. Therefore, a solution method based on hierarchical multi-agent DRL is proposed. The link from each satellite to the user (ground satellite communication device) k in beam m is regarded as an agent, and a hierarchical solution framework is adopted. The upper layer uses DQN to first optimize the resource block allocation, and the lower layer then uses the proximal policy optimization (PPO) to optimize the power allocation within each resource block based on the spectrum allocation result. Each agent independently learns the policy and optimizes the global AoI through local observation and cooperation.

[0017] As a further optimization of the above solution, there are N ground satellite communication devices; in order to reduce interference, a RB allows a maximum of K ground satellite communication devices to access, K < N; the downlink of the satellite shares the spectrum resources of the ground cellular network to provide services for the ground satellite communication devices. Ground users compete for shared RB resources based on factors such as distance, and the satellite independently allocates power for the users within each RB.

[0018] Each RB corresponds to an agent, and only observes the access and power allocation status of its所属 RB to avoid the explosion of the global state-action space.

[0019] The satellite communication devices sharing the same RB are denoted as The information sent is denoted as {W 1,m ,…,W k，m ,…,W K,m}; Denotes The information sent by the k-th satellite communication device under is the common sub-signal of W k,m , For W k，m Private sub-signals;

[0020] All public sub-signals are encoded to form a public stream, denoted as

[0021] All K sub-signals are encoded to form K private streams, and the k-th private stream is denoted as .

[0022] right The sent information is encoded and denoted as...

[0023] in, for Transmission power, for The transmission power.

[0024] The signal received by the k-th ground satellite communication device is z m It is Gaussian white noise that follows a normal distribution;

[0025] express The channel between the k-th ground satellite communication device and LEO, Among them, G s G is the transmit antenna gain of LEO. k Let f be the gain of the transmitting antenna of the k-th ground satellite communication device; c represents the speed of light, f c The carrier frequency is represented by d, and the height of LEO above the ground is represented by d. δ represents the path loss in free space; k,m This represents the Rayleigh small-scale fading distribution.

[0026] As a further optimization of the above scheme, the public flow The signal interference plus noise ratio is The calculation is as follows:

[0027]

[0028] The private flow The signal interference plus noise ratios are respectively The calculation is as follows:

[0029]

[0030] p G This represents the transmit power of the m-th ground cell base station during communication; This represents the interference channel gain of the m-th ground cell base station on the k-th satellite communication device, i.e. β k，m and Represent Rice's large-scale fading and small-scale fading, respectively; σ 2 This represents the power of Gaussian white noise.

[0031] Based on the principle of serial interference cancellation of RSMA signals, interference management is enhanced for each resource block to improve spectrum efficiency and communication quality.

[0032] As a further optimization of the above scheme, in step S4, the state... Represented as:

[0033]

[0034] In step S5, the state Represented as:

[0035]

[0036] in, For spectrum allocation factor, This indicates that in time slot t, the nth satellite communication device is assigned to the mth RB for receiving satellite signals. This indicates that the device has not been assigned and communication is not possible. Separately and Multiply to confirm whether communication has occurred;

[0037] This represents the power allocation from the m-th RB to the n-th satellite communication device in time slot t.

[0038] This invention primarily minimizes the information age by optimizing the spectrum allocation strategy α and the transmit power allocation strategy p. Therefore, the action space simultaneously includes discrete and continuous actions, denoted as a. t =

[0039] [[α1(t), p1(t)], ..., [α m (t), p m [(t)]、…、[α] M (t), p M (t)]].

[0040] As a further optimization of the above scheme, the AoI of the k-th ground satellite communication device under the m-th RB in time slot t is denoted as... The calculation process is as follows:

[0041] Once the receiving end successfully receives and correctly demodulates the signal... Keep it constant at Δt, otherwise, Increasing Δt yields

[0042] The constraints include:

[0043]

[0044] in, This represents the actual transmission rate of the public stream under the m-th RB; This indicates the maximum transmission rate allocated to the public stream; This represents the actual transmission rate of the k-th private stream under the m-th RB; Indicates the maximum transmission rate allocated to the private stream; σ N Δσ represents the interference power; Δσ represents the power threshold.

[0045] That is, interference power σ N It consists of interference power from ground base station communication and Gaussian white noise. Δσ is the power threshold required for successful Sequence Interference Cancellation (SIC), typically set to...

[0046] Only when the rates allocated to private and public streams are no less than their actual rates can the signals of private and public streams be successfully demodulated, meaning the information age does not increase.

[0047] The objective of this invention is to find suitable spectrum allocation strategy α and transmit power allocation strategy p under the constraint of maximum transmission power, thereby optimizing the information age of the satellite communication system. In other words, it aims to minimize the total information age.

[0048]

[0049] Among them, P max This is due to the limitation of the satellite's maximum transmission power.

[0050] As a further optimization of the above scheme, the reward value r m The calculation of (t) is as follows:

[0051]

[0052] The reward function needs to guide the agent to act in a direction that minimizes the information age. The larger the reward function value, the less the AoI (Aspect-Oriented Intelligence) increases. A well-designed state space should not contain too much redundant information; otherwise, the algorithm will struggle to find truly useful features, leading to convergence difficulties.

[0053] As a further optimization of the above scheme, in step S4, the first loss function is minimized using the gradient descent method to update θ and θ′;

[0054] The first loss function is expressed as:

[0055]

[0056] Among them, Q predict This represents the predicted Q value, Q target Indicates the target Q value. In the target network θ′, denoted as state Choose an action α m (t+1), making Maximum; γ is a hyperparameter representing the discount rate; r m (t+1) represents the state. Choose an action α m The reward value obtained after (t+1).

[0057] The upper-layer DQN is responsible for dynamically allocating RBs and passing the allocation results to the lower-layer PPO network. Specifically, it uses a value function-based DRL algorithm to handle the discrete RB allocation binary variable α. The core is to approximate the action-value function using a Q-neural network (QNN). It uses the Bellman equation as the training objective, reflecting the state under the current policy. Execute action α m The expected cumulative discount reward of (t).

[0058] The first loss function represents the difference between the predicted Q value and the target Q value.

[0059] The predicted Q-value represents the network θ in the current state s. t Choose action a t The expected cumulative reward is the current estimate of the Q-value. The target Q-value includes the immediate reward and the maximum Q-value after future discounts. Where γ is the discount factor.

[0060] As a further optimization of the above scheme, in step S5, importance sampling is performed on the randomly selected samples, as follows:

[0061]

[0062] Where x represents any sample collected.

[0063] In the lower-level network, since the power value is a continuous variable, DQN is no longer applicable, so PPO is adopted. The new policy network (ζ) is responsible for generating actions, while the old policy network (ζ') records historical policies for reference. However, the data distributions of the two policy networks are different at this point, so the sampled samples need to be corrected. The data generated by different policies are weighted to reflect their probability deviations. This process is called importance sampling.

[0064] Sample x from distribution ζ and calculate the mean of the function f(x) with respect to x:

[0065]

[0066] Because it is not possible to directly obtain from the distribution Sampling can only be done from another distribution. The sampled data is x, therefore a correction term needs to be added. from The difference between the two distributions can be corrected by multiplying the sampled data by this importance weight.

[0067] As a further optimization of the above scheme, step S5 also controls the update magnitude of the new policy network, that is, calculates the probability ratio of the new policy network relative to the old policy network. And the clipping is performed using a clipping function; the calculation formula is:

[0068] If the probability ratio If the probability is within the interval [1-ε, 1+ε], then the new policy is updated; otherwise, the update magnitude of the new policy network is readjusted until the probability ratio is reached. It lies within the interval [1-ε, 1+ε]; where ε is a hyperparameter representing the magnitude of the clipping.

[0069] While importance sampling solves the problem of large amounts of sampled data, to prevent overly drastic policy updates, the gap between the policy network responsible for sampling and the policy network responsible for decision-making cannot be too large. Therefore, the PPO algorithm uses a "proximal policy optimization" method to control the magnitude of policy updates. (Probability Ratio) This reflects the degree of difference before and after the policy network update. Since the new policy network was copied synchronously to the old policy network before the update, the probability ratio... It also indicates the degree of difference between the updated new policy network and the old policy network.

[0070] As a further optimization of the above scheme, in step S5, the stochastic gradient ascent method is also used to optimize the objective function; the objective function is expressed as:

[0071]

[0072] in, The corresponding function is λ is the cutoff parameter, γ is the decay factor, δ represents the timing difference error function, and min() represents taking the minimum value. Indicates will It is restricted to the range [1-ε, 1+ε].

[0073] The goal of PPO is to find a new policy that samples actions from existing policies that are better than those sampled from older policies. To achieve this, PPO defines an objective function and then uses stochastic gradient ascent to optimize this objective function. The objective function is expressed as:

[0074]

[0075] in, The advantage function, denoted as A(s, a) = Q(s, a) - V(s), where Q(s, a) is the action value function and V(s) is the state value function, represents the advantage of taking a particular action in a given state relative to the agent's average actions in that state. If the advantage function of an action is positive, the agent's probability shifts towards increasing the probability of choosing that action; conversely, if the advantage function is negative, the agent's probability shifts towards decreasing the probability of choosing that action.

[0076] However, this advantage function is calculated from data within only one time slot t, resulting in significant bias. Therefore, a generalized advantage estimation (GAE) is used to balance near-term and long-term rewards, reducing the variance of the advantage function estimate. The GAE is calculated from data across multiple time slots. The final objective function is then rewritten as:

[0077]

[0078] The update of the gradient policy is represented as:

[0079]

[0080] Where α is the step size parameter, That is, the gradient of the objective function; ultimately, the local maximum of the objective function is found through the gradient ascent method.

[0081] In step S4, the loss function corresponding to the evaluation network is the second loss function, i.e. in, Refers to the evaluation network in The state value in each state is determined, and the evaluation network is then updated using gradient descent. in, That is, the gradient of the loss function.

[0082] Compared with the prior art, the present invention achieves the following beneficial effects:

[0083] (1) This invention proposes a hierarchical deep reinforcement learning (DRL) architecture to solve mixed-integer nonlinear programming problems in satellite communication systems through a hierarchical decision-making mechanism. Specifically, the upper layer uses the DQN algorithm to handle discrete RB (resource block) allocation problems, while the lower layer uses the PPO algorithm to achieve continuous power control. The two layers work together to optimize the joint strategy of spectrum resource and power allocation. This architecture significantly reduces the information age (AoI) by decoupling the discrete and continuous decision spaces in a hierarchical manner, providing an efficient resource management solution for RSMA satellite communication systems.

[0084] (2) This invention abandons the strong dependence of traditional methods on complete CSI (Channel State Information) and proposes a dynamic power allocation decision-making method based on intelligent learning. Through the end-to-end environment interaction mechanism of deep reinforcement learning, the agent can directly learn the optimal strategy from partial observation information and adapt to the uncertainty of channel state. In addition, for large-scale user access scenarios, this scheme breaks through the bottleneck of traditional optimization algorithms in high-dimensional data processing through high-dimensional state space modeling and distributed computing, and achieves stable performance in complex scenarios.

[0085] (3) This invention constructs a multi-agent deep reinforcement learning framework, adopting a centralized learning-distributed execution (CTDE) paradigm. During the offline training phase, the satellite centrally coordinates the global information interaction and policy learning among the multiple agents; during the online deployment phase, the trained neural network is independently embedded into each agent, autonomously generating a power allocation scheme based on local observations. This framework balances training efficiency and execution real-time performance, minimizing the system's AoI through distributed autonomous decision-making, while also supporting flexible expansion under dynamic network topologies. Attached Figure Description

[0086] Figure 1 This is a schematic diagram of the network communication architecture of the satellite-ground fusion network provided in an embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of the deep learning network framework for the satellite-ground fusion network information age optimization method based on multi-agent hierarchical deep reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions and advantages of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts fall within the scope of protection of the present invention.

[0089] As Figure 1 , Figure 2 shown, this embodiment provides an information age optimization method for a space-ground integrated network based on multi-agent hierarchical deep reinforcement learning. The space-ground integrated network includes a multi-beam low-earth orbit satellite LEO. The LEO includes M orthogonal frequency band resource blocks RB, and one RB is occupied by one ground cell base station; one RB corresponds to one agent.

[0090] In this embodiment, there are N satellite communication devices on the ground; to reduce interference, one RB allows a maximum of K ground satellite communication devices to access. The K ground satellite communication devices form a ground user group, and K < N; the downlink of the satellite shares the spectrum resources of the ground cellular network to provide services for the satellite communication devices on the ground. Ground users compete for sharing RB resources based on factors such as distance, and the satellite independently allocates power to users within each RB.

[0091] Each RB corresponds to one agent, which only observes the access and power allocation status of its所属RB to avoid the explosion of the global state-action space.

[0092] The satellite communication devices sharing the same RB (i.e., a ground user group) are denoted as The information sent is denoted as {W 1,m , …, W k,m , …, W K,m}; denotes the information sent by the k-th satellite communication device at is the common sub-signal of W k，m , is the private sub-signal of W k，m ;

[0093] All the common sub-signals of

[0094] are encoded to form a common stream, denoted as

[0095] All the K sub-signals of The sent information is encoded and denoted as...

[0096] in, for Transmission power, for The transmission power.

[0097] The signal received by the k-th ground satellite communication device is z m It is Gaussian white noise that follows a normal distribution;

[0098] express The channel between the k-th ground satellite communication device and LEO, Among them, G s G is the transmit antenna gain of LEO. k Let f be the gain of the transmitting antenna of the k-th ground satellite communication device; c represents the speed of light, f c The carrier frequency is represented by d, and the height of LEO above the ground is represented by d. δ represents the path loss in free space; k，m This represents the Rayleigh small-scale fading distribution.

[0099] In this embodiment, public flow The signal interference plus noise ratio is The calculation is as follows:

[0100]

[0101] Private Stream The signal interference plus noise ratios are respectively The calculation is as follows:

[0102]

[0103] p G This represents the transmit power of the m-th ground cell base station during communication; This represents the channel gain of the m-th ground cell base station on the k-th satellite communication device, i.e. and Represent Rice's large-scale fading and small-scale fading, respectively; σ 2 This represents the power of Gaussian white noise.

[0104] Based on the principle of serial interference cancellation of RSMA signals, interference management is enhanced for each resource block to improve spectrum efficiency and communication quality.

[0105] In this embodiment, the AoI of the k-th ground satellite communication device under the m-th RB in time slot t is denoted as The calculation process is as follows:

[0106] Once the receiving end successfully receives and correctly demodulates the signal... Keep it constant at Δt, otherwise, Increasing Δt yields

[0107] The constraints include:

[0108]

[0109] in, This represents the actual transmission rate of the public stream under the m-th RB; This indicates the maximum transmission rate allocated to the public stream; This represents the actual transmission rate of the k-th private stream under the m-th RB; Indicates the maximum transmission rate allocated to the private stream; σ N Δσ represents the interference power; Δσ represents the power threshold.

[0110] That is, interference power σ N It consists of interference power from ground base station communication and Gaussian white noise. Δσ is the power threshold required for successful Sequence Interference Cancellation (SIC), typically set to...

[0111] Only when the rates allocated to private and public streams are no less than their actual rates can the signals of private and public streams be successfully demodulated, meaning the information age does not increase.

[0112] The objective of this invention is to find suitable spectrum allocation strategy α and transmit power allocation strategy p under the constraint of maximum transmission power, thereby optimizing the information age of the satellite communication system. In other words, it aims to minimize the total information age.

[0113]

[0114] Among them, P max This is due to the limitation of the satellite's maximum transmission power.

[0115] The above method includes the following steps:

[0116] Step S1: Initialize network parameters θ, θ′, and These correspond to the main network, target network, new policy network, old policy network, and evaluation network, respectively. Figure 2 The Q-network, target Q-network, new strategy, old strategy, and criticism network are included.

[0117] Step S2: Initialize the experience pool and Set the total number of training rounds T, the allocation frequency Ns, and the frequency of old policy network updates F; initialize t = 1;

[0118] Step S3: If t%Ns = 0, then proceed to step S4; otherwise, proceed to step S5.

[0119] Step S4: For the current state of each agent Using a greedy strategy (ε-greedy) to select action α m (t); m represents the m-th agent; data is obtained. Represent the state transition process and store it. middle;

[0120] In this embodiment, state Represented as:

[0121] For spectrum allocation factor, This indicates that in time slot t, the nth satellite communication device is assigned to the mth RB for receiving satellite signals. This indicates that the device has not been assigned and communication is not possible. Separately and Multiply to confirm whether communication has occurred.

[0122] from Multiple samples are randomly selected from the data, and θ and θ′ are updated; α m (t) represents the spectrum allocation strategy of the m-th agent;

[0123] In this embodiment, gradient descent is used to minimize the first loss function to update θ and θ′;

[0124] The first loss function is expressed as:

[0125]

[0126] Among them, Q predict This represents the predicted Q value, Q target Indicates the target Q value. In the target network θ′, denoted as state Choose an action α m (t+1), making Maximum; γ is a hyperparameter representing the discount rate; r m (t+1) represents the state. Choose an action α m The reward value obtained after (t+1).

[0127] The upper-layer DQN is responsible for dynamically allocating RBs and passing the allocation results to the lower-layer PPO network. Specifically, it uses a value function-based DRL algorithm to handle the discrete RB allocation binary variable α. The core is to approximate the action-value function using a Q-neural network (QNN). It uses the Bellman equation as the training objective, reflecting the state under the current policy. Execute action α m The expected cumulative discount reward of (t).

[0128] The first loss function represents the difference between the predicted Q value and the target Q value.

[0129] The predicted Q-value represents the network θ in the current state s. t Choose action a t The expected cumulative reward is the current estimate of the Q-value. The target Q-value includes the immediate reward and the maximum Q-value after future discounts. Where γ is the discount factor.

[0130] Proceed to step S6.

[0131] Step S5: For each agent, input the lower-level observation state. The PPO new policy network outputs the action probability distribution; data is obtained. And save middle;

[0132] In this embodiment, state Represented as: This represents the power allocation from the m-th RB to the n-th satellite communication device in time slot t. m (t) represents the power allocation strategy of the m-th agent.

[0133] from Randomly select multiple samples and update ζ and In this embodiment, importance sampling is performed on randomly selected samples, which is represented as:

[0134]

[0135] Where x represents any sample collected.

[0136] In the lower-level network, since the power value is a continuous variable, DQN is no longer applicable, so PPO is adopted. The new policy network (ζ) is responsible for generating actions, while the old policy network (ζ') records historical policies for reference. However, the data distributions of the two policy networks are different at this point, so the sampled samples need to be corrected. The data generated by different policies are weighted to reflect their probability deviations. This process is called importance sampling.

[0137] From the distribution Given a sample x, calculate the mean of the function f(x) with respect to x:

[0138]

[0139] Because it is not possible to directly obtain from the distribution Sampling can only be done from another distribution. The sampled data is x, therefore a correction term needs to be added. from The difference between the two distributions can be corrected by multiplying the sampled data by this importance weight.

[0140] In this embodiment, the stochastic gradient ascent method is also used to optimize the objective function; the objective function is expressed as:

[0141]

[0142] in, The corresponding function is λ is the cutoff parameter, γ is the decay factor, δ represents the timing difference error function, and min() represents taking the minimum value. Indicates will It is restricted to the range [1-ε, 1+ε].

[0143] The goal of PPO is to find a new policy that samples actions from existing policies that are better than those sampled from older policies. To achieve this, PPO defines an objective function and then uses stochastic gradient ascent to optimize this objective function. The objective function is expressed as:

[0144]

[0145] in, The advantage function, denoted as A(s, a) = Q(s, a) - V(s), where Q(s, a) is the action value function and V(s) is the state value function, represents the advantage of taking a particular action in a given state relative to the agent's average actions in that state. If the advantage function of an action is positive, the agent's probability shifts towards increasing the probability of choosing that action; conversely, if the advantage function is negative, the agent's probability shifts towards decreasing the probability of choosing that action.

[0146] However, this advantage function is calculated from data within only one time slot t, resulting in significant bias. Therefore, a generalized advantage estimation (GAE) is used to balance near-term and long-term rewards, reducing the variance of the advantage function estimate. The GAE is calculated from data across multiple time slots. The final objective function is then rewritten as:

[0147]

[0148] The update of the gradient policy is represented as:

[0149]

[0150] Where α is the step size parameter, That is, the gradient of the objective function; ultimately, the local maximum of the objective function is found through the gradient ascent method.

[0151] In step S4, the loss function corresponding to the evaluation network is the second loss function, i.e. in, Refers to the evaluation network in The state value in each state is determined, and the evaluation network is then updated using gradient descent. in, That is, the gradient of the loss function.

[0152] If t%F = 0, then update the old policy network, i.e.

[0153] In this embodiment, the update magnitude of the new policy network is also controlled, that is, the probability ratio of the new policy network relative to the old policy network is calculated. And the clipping is performed using a clipping function; the calculation formula is:

[0154]

[0155] If the probability ratio If the probability is within the interval [1-ε, 1+ε], then the new policy is updated; otherwise, the update magnitude of the new policy network is readjusted until the probability ratio is reached. It lies within the interval [1-ε, 1+ε]; where ε is a hyperparameter representing the magnitude of the clipping.

[0156] While importance sampling solves the problem of large amounts of sampled data, to prevent overly drastic policy updates, the gap between the policy network responsible for sampling and the policy network responsible for decision-making cannot be too large. Therefore, the PPO algorithm uses a "proximal policy optimization" method to control the magnitude of policy updates. (Probability Ratio) It reflects the degree of difference before and after the update of the new policy network. Since the new policy network before the update is synchronously copied to the old policy network, the probability ratio also represents the degree of difference between the updated new policy network and the old policy network.

[0157] Execute step S6.

[0158] In steps S4 and S5, the reward value r m (t) is calculated as:

[0159]

[0160] The reward function needs to have the effect of guiding the agent to act in the direction of minimizing the age of information. The larger the value of the reward function, the less the AoI increases. The design of a good state space should not contain too much redundant information, otherwise it is difficult for the algorithm to find truly useful features, resulting in difficulty in convergence.

[0161] Step S6: If t < T, then t = t + 1, and execute step S3; otherwise, end the iteration.

[0162] To adapt to the changes in different beam environments, this scheme proposes a multi-agent hierarchical DQN-PPO (MHDP) algorithm. Among them, the main network θ and the target network θ′ (which can also be called the Q network and the target Q network respectively), both have the same structure but different update frequencies, and the two together constitute the DQN network, that is, the deep Q network, belonging to the upper layer network; the new policy network the old policy network and the evaluation network constitute the proximal policy optimization PPO, belonging to the lower layer network. The state corresponds to the upper (upper) network, and the state corresponds to the lower (lower) network. Correspondingly, α m (t) is the action selected by the state ; p m (t) is the action selected by the state . This invention mainly minimizes the age of information by optimizing the spectrum allocation strategy α and the transmit power allocation strategy p, so the action space contains both discrete actions and continuous actions, expressed as

[0163] a t = [α1(t), p1(t)], …, [α m (t), p m (t)], …, [α M (t), p M (t)].

[0164] Because the problem involves a hybrid nonlinear programming problem requiring the simultaneous optimization of a discrete spectrum allocation strategy α and a continuous power allocation strategy p, traditional optimization algorithms struggle to solve it and obtain the global optimum. Therefore, a hierarchical multi-agent DRL-based solution is proposed. Each link from a satellite to a user (ground satellite communication device) k within beam m is treated as an agent, and a hierarchical framework is adopted. The upper layer uses DQN to optimize resource block allocation, while the lower layer, based on the spectrum allocation results, utilizes near-end policy optimization (PPO) to optimize power allocation within each resource block. Each agent independently learns a policy, optimizing the global AoI through local observation and collaboration.

[0165] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A method for optimizing the information age of a satellite-ground fusion network based on multi-agent hierarchical deep reinforcement learning, wherein the satellite-ground fusion network includes a multi-beam low-Earth orbit (LEO) satellite, the LEO includes M orthogonal frequency band resource blocks (RBs), and each RB is occupied by a ground cell base station; characterized in that, It includes the following steps: Step S1: Initialize network parameters θ, θ′, ζ, ζ' and These correspond to the main network, target network, new policy network, old policy network, and evaluation network, respectively; one RB corresponds to one agent. Step S2: Initialize the experience pool and Set the total number of training rounds T, the allocation frequency Ns, and the update frequency F of the old policy network; initialize t = 1; Step S3: If t%Ns = 0, then execute Step S4; otherwise, execute Step S5; Step S4: For the current state of each of the agents Using a greedy strategy to select action α m (t); m represents the m-th agent; data is obtained. And save From; Multiple samples are randomly selected from the data, and θ and θ′ are updated; α m (t) represents the spectrum allocation strategy of the m-th agent; execute step S6; Step S5: For each agent, input the lower-level observation state. The PPO new policy network outputs the action probability distribution; data is obtained. And save From; Randomly select multiple samples and update ζ and p m (t) represents the power allocation strategy of the m-th agent; If t%F = 0, then update the old policy network, i.e., ζ’ = ζ; execute Step S6; Step S6: If t < T, then t = t + 1, and execute Step S3; otherwise, end the iteration.

2. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 1, characterized in that, There are N satellite communication devices on the ground; one RB allows at most K ground satellite communication devices to access, where K < N; Satellite communication equipment sharing the same RB is denoted as The sent information is denoted as {W 1，m ,…,W k，m ,…,W K，m }; express The information sent by the kth satellite communication device; For W k，m public sub-signals, For W k，m Private sub-signals; All public sub-signals are encoded to form a public stream, denoted as All K sub-signals are encoded to form K private streams, and the k-th private stream is denoted as . right The sent information is encoded and denoted as... in, for Transmission power, for The transmission power; The signal received by the k-th ground satellite communication device is z m It is Gaussian white noise that follows a normal distribution; express The channel between the k-th ground satellite communication device and LEO, Among them, G s G is the transmit antenna gain of LEO. k Let f be the gain of the transmitting antenna of the k-th ground satellite communication device; c represents the speed of light, f c The carrier frequency is represented by d, and the height of LEO above the ground is represented by d. δ represents the path loss in free space; k，m This represents the small-scale Rayleigh fading distribution.

3. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 2, characterized in that, The public flow The signal interference plus noise ratio is The calculation is as follows: The private flow The signal interference plus noise ratios are respectively The calculation is as follows: p G This represents the transmit power of the m-th ground cell base station during communication; This represents the interference channel gain of the m-th ground cell base station on the k-th satellite communication device, i.e. β k，m and Represent Rice's large-scale fading and small-scale fading, respectively; σ 2 This represents the power of Gaussian white noise.

4. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 3, characterized in that, In step S4, the state Represented as: In step S5, the state Represented as: in, For spectrum allocation factor, This indicates that in time slot t, the nth satellite communication device is assigned to the mth RB for receiving satellite signals. This indicates that the device has not been assigned and communication is not possible. Separately and Multiply to confirm whether communication has occurred; This represents the power allocation from the m-th RB to the n-th satellite communication device in time slot t.

5. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 4, characterized in that, The AoI of the k-th ground satellite communication device under the m-th RB in time slot t is denoted as The calculation process is as follows: The constraints include: in, This represents the actual transmission rate of the public stream under the m-th RB; This indicates the maximum transmission rate allocated to the public stream; This represents the actual transmission rate of the k-th private stream under the m-th RB; Indicates the maximum transmission rate allocated to the private stream; σ N Δσ represents the interference power; Δσ represents the power threshold.

6. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 5, characterized in that, Reward value r m The calculation of (t) is as follows:

7. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 1, characterized in that, In Step S4, use the gradient descent method to minimize the first loss function to update θ and θ′; The first loss function is expressed as: Among them, Q predict This represents the predicted Q value, Q target Indicates the target Q value. In the target network θ′, denoted as , it represents the state. Choose an action α m (t+1), making Maximum; γ is a hyperparameter representing the discount rate; r m (t+1) represents the state. Choose an action α m The reward value obtained after (t+1).

8. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 1, characterized in that, In Step S5, perform importance sampling on randomly selected samples, which is expressed as: where x represents any one of the collected samples.

9. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 8, characterized in that, In step S5, the update magnitude of the new policy network is also controlled, that is, the probability ratio r of the new policy network relative to the old policy network is calculated. ζ (t), and clipped using a clipping function; the calculation formula is: If the probability ratio r ζ If (t) is within the interval [1-ε, 1+ε], then the new policy is updated; otherwise, the update magnitude of the new policy network is readjusted until the probability ratio r is reached. ζ (t) lies within the interval [1-ε, 1+ε], where ε is a hyperparameter representing the magnitude of the clipping.

10. The method for optimizing the age of satellite-ground fusion network information based on multi-agent hierarchical deep reinforcement learning according to claim 9, characterized in that, In Step S5, also use the stochastic gradient ascent method to optimize the objective function; the objective function is expressed as: in, The corresponding function is λ is the cutoff parameter, γ is the decay factor, δ represents the timing difference error function, min() represents taking the minimum value, and clip(r) ζ (t), 1-∈, 1+∈) represents r ζ (t) is restricted to [1-ε, 1+ε].

Images