A method and related apparatus for downlink spectrum sharing and beam power dynamic allocation

Abstract

The application discloses a kind of downlink spectrum sharing and beam power dynamic allocation method and related device, it is related to the technical field of satellite communication, the data queue of low-orbit satellite ground user to be served, the channel state information of high-low-orbit satellite ground user and the matching relationship of low-orbit satellite ground user and low-orbit satellite beam are obtained first in the application;Then data queue, channel information and matching relationship are input to network model, and low-orbit satellite beam power allocation result is obtained.The application updates the parameters of neural network model using proximal policy optimization algorithm in the process of model training, realizes the rapid allocation of beam power of low-orbit satellite group based on deep reinforcement learning framework, and guarantees the accuracy and timeliness of resource allocation result.The application ensures that low-orbit satellite communication system does not cause harmful interference to high-orbit satellite system when sharing the spectrum of high-orbit satellite system, and maximizes the service quality of low-orbit satellite and maintains the fairness between low-orbit satellite ground users.

Description

Technical Field

[0001] This invention belongs to the field of communications, specifically relating to a method and related apparatus for downlink spectrum sharing and dynamic beam power allocation. Background Technology

[0002] Low Earth Orbit (LEO) satellite communication systems possess key characteristics such as low transmission latency, low signal loss, and low transmission costs, bringing new developments and opportunities to satellite communications. With the increasing deployment and use of LEO satellite constellations, spectrum resources are becoming increasingly scarce. In systems where Geostationary Earth Orbit (GEO) and LEO satellites coexist, sharing the GEO system spectrum is one way to address spectrum scarcity. Spectrum sharing inevitably leads to severe interference with the GEO system. As the size, number, and heterogeneity of LEO satellite constellations increase, the interference level rises sharply. Therefore, to improve the throughput of LEO systems under GEO system interference constraints, it is urgent to explore interference mitigation techniques for spectrum sharing in GEO-LEO coexisting satellite systems.

[0003] To address interference issues, researchers have investigated isolation-based strategies. Interference is mitigated by determining appropriate rejection angles and applying techniques such as shutdown, side-looking, and progressive pitch. H. Wang et al., in "Coexistence Downlink Interference Analysis Between LEO System and GEO System in Ka Band," analyzed the impact of isolation angle strategies in LEO and GEO coexistence systems. A. Hills et al., in "Feasibility of Using Beam Steering to Mitigate Ku-Band LEO-to-GEO Interference," proposed an improved strategy that utilizes beam steering of the LEO constellation to create rejection angles of varying degrees to alleviate interference. Among these existing technologies, while shutdown techniques can completely eliminate interference to the GEO system, they sacrifice the spectral efficiency of the LEO system.

[0004] However, the aforementioned interference mitigation strategies are not well adapted to time-varying and increasingly complex wireless communication scenarios. Researchers have also proposed beam hopping (BH), adaptive power control (APC), and hybrid techniques to suppress interference and improve satellite system throughput. Regarding interference in GEO satellite networks, P.-Y. Chen et al. proposed eight BH schemes in "Coordinative Spectrum Sharing for GEO and LEO Satellite Networks." In GEO and non-GEO satellite systems, SK Sharma et al. proposed an APC technique to mitigate online interference in "Inline Interference Mitigation Techniques for Spectral Coexistence of GEO and NGEOSatellites." SK Sharma and C. Wang, in their research "Cognitive beamhopping for spectral coexistence of multibeam satellites" and "A Novel Cognitive Satellite Network With GEO and LEO Broadband Systems in the Downlink Case," respectively studied interference management using hybrid beamforming (BH) and active beamforming (APC) technologies in two-satellite systems and LEO-GEO coexistence systems. The latter, "A Novel Cognitive Satellite Network With GEO and LEO Broadband Systems in the Downlink Case," considered maximizing overall spectral efficiency in a scenario where LEO is the primary communication system and GEO is the secondary system. In GEO-LEO coexistence systems, a more general assumption is that GEO is the primary communication system and LEO is the secondary system. R. Li et al., in their research "Optimal Beam Power Control for Co-Existing Multibeam GEO and LEO Satellite System," proposed an APC scheme to jointly optimize the multibeam power of both GEO and LEO satellites to maximize LEO satellite throughput, considering only the scenario where the LEO satellite serves a single user.In "Joint multi-beam power control for LEO and GEO Spectrum-sharing networks," M. Jia et al. considered the scenario of multiple LEO ground users and proposed an APC scheme to jointly optimize the multi-beam power of GEO and LEO satellites. At the lower level, Y. Wang et al., in "A Novel Dynamic Spectrum-Sharing Method for GEO and LEOatellite Networks," developed a spectrum-aware assisted spectrum-sharing scheme to ensure that LEO satellites transmitting data do not cause harmful interference to the GEO system. WU Khan et al., in "Rate Splitting Multiple Access for Cognitive Radio GEO-LEO Co-Existing Satellite Networks," applied cognitive radio technology and fast division multiple access technology to GEO and LEO coexisting systems, simultaneously optimizing the power and subcarrier allocation of LEO satellites. In their paper "Dynamic Cooperative Spectrum Sharing in a Multi-beam LEO-GEO Co-Existing Satellite System," P. Gu et al. proposed a flexible dynamic spectrum sharing scheme that considers beam power allocation and LEO ground user scheduling within a LEO satellite constellation. However, in GEO-LEO coexistence scenarios, these APC or BH schemes only focus on the instantaneous interference when LEO and GEO and their ground users are collinear, rather than the persistent harmful interference caused by LEO passing through the GEO beam coverage area. Therefore, it is necessary to study interference mitigation schemes for continuous time-slot APC in GEO-LEO coexistence systems.

[0005] Resource allocation problems, such as beam power allocation, in satellite communication systems are often non-convex NP-hard problems with many optimization parameters. However, the aforementioned literature approximates these non-convex NP-hard problems as convex problems and then iteratively solves them. This cannot meet the timeliness requirements of satellite communication systems. Low-Earth orbit satellites, due to their small payloads, high operating speeds, and rapidly outdated optimization parameters, face even greater challenges in real-time interference management. In contrast, using machine learning-based methods to approximate the optimal solution to non-convex problems is a promising approach. For example, existing machine learning-based resource allocation literature focuses on single satellite communication systems, with limited research on resource allocation and interference management, particularly beam power allocation, in heterogeneous satellite networks.

[0006] In summary, research on machine learning-based beam power allocation and interference management in multi-layered heterogeneous satellite systems is limited. Existing beam power allocation schemes do not consider long-term system throughput, focusing only on the current optimal solution. The complexity of long-term systems increases with the dynamic changes in the wireless environment and the random arrival of data. Therefore, it is necessary to explore DRL-based long-term interference mitigation techniques to improve the long-term spectral efficiency of GEO and LEO coexisting systems. Summary of the Invention

[0007] To meet the needs of GEO-LEO coexistence systems for spectrum sharing, interference suppression, fairness for ground users, and improved spectrum efficiency, this invention provides a method and related apparatus for downlink spectrum sharing and dynamic beam power allocation. This method rapidly allocates beam power to LEO satellites while ensuring the quality of service for high-orbit satellite communication, thereby maximizing the service quality of LEO satellites and maintaining fairness among users. Based on a deep reinforcement learning framework, this invention enables rapid beam power allocation for LEO satellites, significantly improving the timeliness of resource allocation.

[0008] To achieve the above objectives, the present invention employs the following technical solution:

[0009] In a first aspect, the present invention provides a method for downlink spectrum sharing and dynamic beam power allocation, comprising the following steps:

[0010] Construct a long-term weighted sum rate model under beam power and interference threshold constraints for communication scenarios where high-Earth orbit and low-Earth orbit satellites coexist.

[0011] Construct a deep reinforcement learning framework for solving the problem of maximizing the rate of a long-term weighted sum;

[0012] The neural network in the deep reinforcement learning framework is trained using a proximal policy optimization algorithm.

[0013] The data queues, channel information, and matching relationships between LEO ground users and LEO satellite beams for each time slot are considered as states and sequentially input into a trained neural network model to obtain the LEO satellite beam power allocation scheme for each time slot.

[0014] Secondly, the present invention provides a downlink spectrum sharing and dynamic beam power allocation system, comprising:

[0015] The model building module is used to build a long-term weighted sum rate model under beam power constraints and interference threshold constraints for communication scenarios where high- and low-Earth orbit satellites coexist.

[0016] Framework building blocks are used to construct deep reinforcement learning frameworks for solving the problem of maximizing the rate of long-term weighted sums.

[0017] The model training module is used to train neural networks in a deep reinforcement learning framework using a proximal policy optimization algorithm.

[0018] The power allocation scheme calculation module is used to treat the data queues that ground users need to transmit for each time slot of LEO satellites, channel information, and the matching relationship between LEO ground users and LEO satellite beams as states, and input them sequentially into the trained neural network model to obtain the LEO satellite beam power allocation scheme for each time slot.

[0019] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The proposed solution, based on a deep reinforcement learning framework, maximizes the long-term weighted sum rate of the LEO satellite system while ensuring that the GEO satellite system is free from harmful interference. It enables real-time beam power allocation decisions for LEO satellites. Compared to the baseline solution mentioned in this invention, this solution significantly improves the timeliness of resource allocation, effectively enhances the spectral efficiency of the LEO satellite system, and reduces computational complexity. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the method of the present invention.

[0025] Figure 2 This is a schematic diagram of the system of the present invention.

[0026] Figure 3 This invention provides a deep reinforcement learning framework for solving the long-term weighted sum rate problem under beam power and interference constraints in high-Earth orbit satellite coexistence communication systems.

[0027] Figure 4 This is a diagram illustrating the training process of the neural network model of the present invention.

[0028] Figure 5 This describes the generalization performance of the present invention with respect to parameter λ.

[0029] Figure 6 The cumulative distribution of LEO ground user reception and rate over 500 consecutive time slots using the scheme of this invention and three comparative schemes is shown, where (a) is λ=10 and (b) is λ=50.

[0030] Figure 7 This is a comparison chart of the average LE0 user queue backlog in the present invention and three comparative schemes over 50 time slots (λ=10).

[0031] Figure 8 This is a comparison chart of the average LE0 user queue backlog (λ=50) of the present invention and three comparative schemes under 50 time slots.

[0032] Figure 9 This is a comparison chart of the average sum rate obtained by the present invention and three comparative schemes under different values ​​of λ. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0034] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0035] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0036] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0038] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0039] Definitions:

[0040] Low Earth Orbit (LEO) satellite communication systems aim to maximize the long-term weighted sum rate and fairness among LEO satellite ground users. This invention utilizes a deep reinforcement learning framework to achieve rapid beam power allocation for LEO satellite constellations. This invention ensures that geostationary earth orbit (GEO) satellite communication systems are not subject to harmful interference while also guaranteeing normal communication between LEO satellites in collinear regions.

[0041] Collinearity Area: In communication systems where high-Earth orbit (HEO) and low-Earth orbit (LEO) satellites coexist, HEO satellites provide real-time services to fixed-location ground users within their beam coverage area, while LEO satellites operate periodically in their orbits. As LEO satellites gradually approach the location where HEO satellites and their ground users are aligned, the interference experienced by HEO ground users gradually increases. When the interference exceeds a threshold, the LEO satellite communication system will cause harmful interference to the HEO satellite communication system. The area that causes harmful interference to the HEO satellite communication system is defined as the collinearity area. When LEO satellites operate in the collinearity area, their communication will severely affect the service quality of the HEO satellites.

[0042] The present invention will now be described in further detail with reference to the accompanying drawings:

[0043] See Figure 1 This invention discloses a method for downlink spectrum sharing and low-Earth orbit satellite resource allocation in a coexistence system, comprising the following steps:

[0044] S1 constructs a long-term weighted sum rate maximization model under beam power and interference threshold constraints for communication scenarios where high- and low-Earth orbit satellites coexist; the details are as follows:

[0045] The model for maximizing the long-term weighted sum rate is as follows:

[0046]

[0047]

[0048]

[0049]

[0050]

[0051] Where T represents the number of time slots in which a low-orbit satellite constellation passes through a collinear region; Let represent the LEO beam power allocation scheme for T time slots; P(t) represents the LEO beam power allocation scheme for time slot t, and and N represents the number of ground users for low-Earth orbit satellites and high-Earth orbit satellites, respectively; S N represents the number of LEO satellites in the LEO satellite constellation; B Indicates the number of beams per LEO satellite; t represents the time slot; k represents the index of the LEO ground user; Q k (t) represents the virtual data queue that the k-th LEO ground user needs to transmit; C k (t) represents the channel capacity of the k-th LEO ground user; R k Rk(t) represents the actual transmission rate received by the k-th LEO ground user, where Rk(t) = min(Qk(t), Ck(t)). This represents the interference factor of the LEO system on the GEO system; g i,j,g (t) represents the channel gain from the j-th beam of the i-th LEO satellite to the g-th GEO ground user; P i,j (t) represents the power of the j-th beam of the i-th LEO satellite; th This represents the tolerance threshold for interference in a GEO system; m i,j,k(t) represents the matching relationship between the j-th beam of the i-th satellite and the k-th LEO ground user; This indicates the maximum beam power of the LEO satellite.

[0052] The virtual data queue Q k The queuing dynamics of (t) satisfy the following conditions:

[0053]

[0054] Among them, S k (t) represents the transmission rate that can be provided to the user in time slot t; A k (t) represents the data rate at which the k-th LEO ground user randomly arrives in time slot t, A k (t) follows a Poisson distribution with parameter λ.

[0055] S2 constructs a deep reinforcement learning framework for solving the problem of maximizing the long-term weighted sum rate; the details are as follows:

[0056] Intelligent Agent: The intelligent agent corresponds to the Network Management Center (NMC);

[0057] Environment: The environment refers to all factors in the wireless communication environment;

[0058] State: The state s t It is the input to the neural network and the basis for the agent to execute actions; it combines the data queue q(t) of LEO satellite ground users in time slot t with the channel information. Matching relationship between LEO satellite beams and LEO ground users Model as state:

[0059]

[0060] The data queue q(t) of the LEO satellite ground user is represented as:

[0061]

[0062] The channel information Represented as:

[0063]

[0064] in, Including the channel gain vector vec(H) from the i-th LEO satellite to the LEO ground user i (t)) T ,vec(·) T This represents the matrix vectorization function, which transforms a matrix into a row vector. The interference channel gain vector vec(G) from the i-th LEO satellite to the GEO ground user. i(t)) T The interference channel gain vector g from GEO satellites to LEO ground users L (t).

[0065] Among them, H i (t) is represented as follows:

[0066]

[0067] Among them, h i,j,k (t) represents the channel gain between the j-th beam of the i-th LEO satellite and the k-th LEO ground user; G i (t) is represented as follows:

[0068]

[0069] Among them, g i,j,g (t) represents the channel gain between the j-th beam of the i-th LEO satellite and the g-th GEO ground user; the interference channel gain vector g L (t) is represented as follows:

[0070]

[0071] A sign function is used to represent the matching relationship between the j-th beam of the i-th LEO satellite and the k-th LEO ground user, h. th Indicates the channel gain threshold; This means that the j-th beam of the i-th LEO satellite covers and serves the k-th LEO satellite ground user. Otherwise, m i,j,k =0 means that the main lobe of the LEO beam does not cover ground users. Therefore, vec(M) i (t)) T M represents the matching relationship between the beam of the i-th LEO satellite and the LEO ground user, where M i (t) is represented as follows:

[0072]

[0073] Therefore, the beam matching relationship between the LEO satellite constellation and LEO ground users It is expressed as follows:

[0074]

[0075] Action: Action a t This is the output of the neural network; the agent changes the environment by performing actions. These actions are modeled as LEO beam power allocation vectors, as follows:

[0076]

[0077] in, In the current state s t Next, select action a. t Afterwards, the environment will receive the reward function and change its state from s. t Change to s t+1 ;

[0078] Reward: The reward r t The agent observes state s t Output action a t Then, the environment provides feedback to the agent; the agent then takes action a. t Afterwards, it is necessary to know whether this action satisfies or is close to the optimization objective; reward r t As an environment, action a t Feedback is for action a t The evaluation criteria include: negative rewards for actions that fail to meet beam power or interference constraints; and the reward r. t It is expressed as follows:

[0079]

[0080] Where c1 represents the preset weight, and c2 and c3 represent the penalty factors for not meeting the interference constraint and the penalty factor for not meeting the beam power constraint, respectively; I max (t) and I th Let n represent the maximum interference and the interference threshold received by the GEO ground user in time slot t, respectively. p (t) represents the action r t The number of beams that do not meet the beam power constraint; R k (t) represents the actual transmission rate received by the k-th LEO ground user; Q k (t) represents the data queue that the k-th LEO ground user needs to transmit.

[0081] The user The channel capacity C in time slot t k (t) is as follows:

[0082]

[0083] in, Let b represent the k-th LEO ground user. L ψ represents the LEO satellite beamwidth. k (t) represents the LEO ground user in time slot t. The received signal-to-noise ratio;

[0084] LEO ground users Received signal-to-noise ratio ψ k (t), as follows:

[0085]

[0086] Where, in time slot t, x i,j,k (t) indicates whether the j-th beam of the i-th LEO covers and serves the k-th LEO ground user, x i,j,k (t) = 1, indicating coverage and service; otherwise, x i,j,k (t) = 0 indicates no coverage; h i,j,k (t) represents the signal from the j-th beam of the i-th LEO satellite to the k-th LEO ground user. Channel gain between; g k (t) represents the beam from the GEO satellite to the k-th user on the LEO satellite. Channel gain between; Indicates noise power.

[0087] The satellite channel model h is as follows:

[0088]

[0089] Among them, G T G R L, A R These represent the antenna gain, free space loss, and rain attenuation for the satellite (transmitter) and the end user (receiver), respectively. The antenna gain of the transmitter or receiver can be expressed as:

[0090]

[0091] Where G0 is the maximum antenna gain when the off-axis angle is 0. J1(υ) and J3(υ) represent the first-order and third-order Bessel functions, respectively. in, This represents the off-axis angle, which is related to the antenna direction and position of the satellite and the end user. This represents the off-axis angle corresponding to a 3dB beamwidth. Since the LEO user antenna continuously tracks the LEO satellite, the off-axis angle at the receiver is 0, G. R =G0. Free space loss L = (4πd) 2 f 2 / c 2 Where d, f, and c represent the distance between the satellite and the ground user, the frequency, and the speed of light, respectively. Rain attenuation A R =ζ -1 / 2 Where ζ represents the amplitude of rain attenuation, and A R It follows a log-normal distribution.

[0092] S3 employs a proximal policy optimization algorithm to train the neural network within a deep reinforcement learning framework; specifically as follows:

[0093] Step 1: Initialize the relevant parameters in the satellite communication system and the deep reinforcement learning framework;

[0094] Step 2: The agent interacts with the environment and stores batch_size data items. t ,a t ,r t ,s t+1 [The text appears to be incomplete and contains several typos. A more accurate translation would require the full context.] t , will s t The inputs are fed into the actor and critic networks respectively. The actor network outputs the mean of the normal distribution of the beam action. A normal distribution is constructed based on the mean, and the action 'a' is sampled from it. t Execute action a t And calculate the return function r t Get the state s of the next time slot. t+1 This process is repeated, storing data in batches of size `batch_size`. t ,a t ,r t ,s t+1 ].

[0095] Step 3: Train the actor and critic neural networks;

[0096] Step 4: Repeat steps 2 and 3 until the set number of iterations is reached or the loss function converges, at which point the actor and critic network training process ends.

[0097] S3-1 Extract batch_size data [s t ,a t ,r t ,s t+1 ] Calculate the dominance estimation function according to the following formula:

[0098] A t =-V(s) t )+r t +γr t+1 +…+γ T-t-1 r T-1 +γ T-t V(s T ),t={1,2,...,T}

[0099] Wherein, V(s) t ) is to put state s tThe output value function after being input into the critic network, where γ is the discount factor;

[0100] After obtaining the advantage estimate in S3-2, the loss functions of the actor network and the critic network are calculated:

[0101]

[0102] Among them, L actor (θ) represents the loss function of the actor network, and θ represents the parameters of the actor network. p θ (a t |s t ) indicates that in state s t Choose action a t The probability of ε is the upper and lower bounds controlled by the parameters in the clip function; This represents the average value of a batch of data.

[0103]

[0104] in, This represents the loss function of the critic network. The parameters r of the critic network are represented by r. i This represents the reward for time slot i.

[0105] S3-3 uses the Adam optimizer to optimize the network parameters θ and φ of the actor network and critic network. Update, θ old ←θ,

[0106] S3-4 repeats S3-2 and S3-3 multiple times.

[0107] S4 inputs the data queues, channel information, and beam matching relationships between LEO ground users and LEO satellites for multiple time slots into the trained actor network model to obtain the beam power allocation scheme for each time slot.

[0108] like Figure 2 As shown, this embodiment of the invention provides a downlink spectrum sharing and dynamic beam power allocation system, comprising:

[0109] The model building module is used to build a long-term weighted sum rate model under beam power constraints and interference threshold constraints for communication scenarios where high- and low-Earth orbit satellites coexist.

[0110] Framework building blocks are used to construct deep reinforcement learning frameworks for solving the problem of maximizing the rate of long-term weighted sums.

[0111] The model training module is used to train neural networks in a deep reinforcement learning framework using a proximal policy optimization algorithm.

[0112] The power allocation scheme calculation module is used to treat the data queues, channel information, and matching relationship between LEO ground users and LEO satellite beams of each time slot as states, and input them sequentially into the trained neural network model to obtain the LEO satellite beam power allocation scheme for each time slot.

[0113] The principle of this invention:

[0114] A neural network model is used to achieve rapid real-time beam power allocation, modeling all factors in the satellite communication system as the environment. The network management center managing the satellites is considered as an agent, which provides a beam power allocation scheme. During the interaction between the agent and the environment, the environment first obtains the data queues that ground users on high-Earth orbit and low-Earth orbit satellites need to transmit, channel state information, and the matching relationship between low-Earth orbit ground users and low-Earth orbit satellite beams. All collected information is input as state into the neural network module of the agent to obtain the beam power allocation result. The environment calculates the reward using the existing allocation scheme and collects the state for the next time step, repeating this process. The resource allocation model is trained based on different channel samples and randomly arriving data, and a near-end strategy optimization algorithm is used to update the model parameters during training.

[0115] In the deep reinforcement learning model, the management center of the satellite network is treated as an agent. The state is modeled as the data queues that ground users need to transmit for the current LEO satellite, channel information, and the beam matching relationship between LEO ground users and LEO satellites. When making decisions, the agent considers not only the current channel conditions but also the data rate queues of each user, thus linking beam power allocation across multiple time slots. This addresses the issue of sequential coupling between multiple time slots in the long-term weighted sum rate problem. In action modeling, the beam power allocation scheme of the LEO satellite constellation is considered an action, and the neural network output is 1×N. S ·N B A row vector of dimension N, where N S N represents the number of LEO satellites in the LEO satellite constellation. B The number of beams for a LEO satellite. The reward obtained after taking an action in each time slot is defined as the weighted sum of the rate and penalty term for the current time slot. The penalty term consists of two parts: a negative reward for actions that do not meet beam power constraints or interference threshold constraints.

[0116] The framework employs the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm establishes a multivariate normal distribution by outputting the mean of the multivariate normal distribution of each dimension of the action, and then samples from the distribution to obtain the final action to be executed, thus completing the processing of the continuous action space.

[0117] T represents the length of the trajectory. The long-term weighted sum rate problem can be decomposed into the sum of weighted sum rates across multiple time slots. Therefore, the power allocation of T consecutive time slots can be viewed as a trajectory in network training. At the initial moment of network training, the initial positions of the LEO satellite constellation, the positions of the high-orbit satellites, and the positions of all users are randomly generated. In a trajectory, the GEO satellites and their ground users, as well as the LEO satellite ground users, remain stationary, and the LEO satellite constellation passes through the beam coverage area of ​​the GEO satellites at a constant speed. In any time slot of a trajectory, the amount of data randomly arriving at the LEO satellite ground users follows a Poisson distribution with parameter λ.

[0118] During training, the differences in sample data across multiple trajectories lie in the locations of LEO satellite users and the random number of LEO ground users arriving in each time slot. The locations of LEO satellite users on different trajectories are randomly generated. After extensive training to complete the mapping from state to action, the final model effectively solves the long-term weighted sum rate maximization problem with beam power and interference constraints in communication scenarios where high-Earth orbit satellites coexist, and exhibits good generalization ability for different user data transmission needs.

[0119] The specific steps are as follows:

[0120] Step 1: Modeling the long-term weighted sum rate maximization problem under beam power constraints and interference threshold constraints for communication scenarios with coexistence of high- and low-Earth orbit satellites.

[0121] In a GEO and LEO satellite coexistence system, the GEO satellite communication system is the primary communication system, and the LEO satellite communication system is the secondary communication system, sharing the frequency band of the GEO satellite system. LEO satellites can interfere with the GEO communication system when crossing the coverage area of ​​the GEO satellite beam, especially when the LEO satellite is near an area where the GEO satellite and its ground users are collinear, at which point the interference increases dramatically. In the primary-secondary communication system model, the secondary communication system should ensure that the interference to the primary communication system during information transmission remains below a given threshold.

[0122] This invention considers the following scenario: within the coverage area of ​​a single-beam GEO satellite, GEO satellites simultaneously provide services. N ground users S Multi-beam LEO satellite collaborative services One ground user, among whom and N represents the number of ground users for high-orbit and low-orbit satellites, respectively. S This indicates the number of LEO satellites in the LEO satellite constellation. N B This indicates the number of beams for each LEO satellite.

[0123] In time slot t, h i,j,k (t) represents the signal from the j-th beam of the i-th LEO satellite to the k-th user of the LEO satellite. The channel gain between, g i,j,g (t) represents the connection from the j-th beam of the i-th LEO satellite to the g-th user of the GEO satellite. The channel gain between them. Similarly, from the beam of the GEO satellite to the k-th user of the LEO satellite. The channel gain between them is expressed as g k (t). Specifically, the channel modeling between the satellite and the ground user is as follows:

[0124]

[0125] Among them, G T G R L, A R These represent the antenna gain, free space loss, and rain attenuation for the satellite (transmitter) and the end user (receiver), respectively. The antenna gain of the transmitter or receiver can be expressed as:

[0126]

[0127] Where G0 is the maximum antenna gain when the off-axis angle is 0. J1(υ) and J3(υ) represent the first-order and third-order Bessel functions, respectively. in, This represents the off-axis angle, which is related to the antenna direction and position of the satellite and the end user. This represents the off-axis angle corresponding to a 3dB beamwidth. Since the LEO user antenna continuously tracks the LEO satellite, the off-axis angle at the receiver is 0, G. R =G0. Free space loss L = (4πd) 2 f 2 / c 2 Where d, f, and c represent the distance between the satellite and the ground user, the frequency, and the speed of light, respectively. Rain attenuation A R =ζ -1 / 2 Where ζ represents the amplitude of rain attenuation, and A R It follows a log-normal distribution.

[0128] In the LEO satellite communication system, its ground users The signal-to-interference-plus-noise ratio ψ received in time slot t k(t) is represented as follows:

[0129]

[0130] Where, x i,j,k (t) indicates whether the j-th beam of the i-th LEO satellite covers and serves the user. x i,j,k (t) = 1, indicating coverage and service; otherwise, x i,j,k (t) = 0 indicates that it is not covered. As an interference factor for the LEO system, Indicates ground users Received noise power. P i,j (t) represents the power of the j-th beam of the i-th LEO satellite; P G (t) represents the beam power of the LEO satellite, which is assumed in this paper. in, This indicates the maximum beam power of the LEO satellite.

[0131] Therefore, in time slot t, the user The channel capacity can be expressed as C k (t):

[0132]

[0133] Among them, b L This indicates the LEO satellite beam bandwidth.

[0134] GEO users Received interference from the LEO system It can be represented as:

[0135]

[0136] in, This represents the interference factor of the LEO system on the GEO system. The following constraints must be met:

[0137]

[0138] Among them, I th This represents the interference threshold that the GEO system can tolerate.

[0139] Simply maximizing reception and rate for ground users in a LEO communication system leads to a lack of fairness among the multiple users served. Therefore, to ensure fairness among users, this invention addresses the optimization problem for each LEO ground user... Introduce a virtual data queue Q k (t), Q k(t) The queuing dynamics satisfy the following conditions:

[0140]

[0141] Among them, S k (t) represents the transmission rate that can be provided to the user in time slot t. A k (t) represents the user in time slot t. The rate of data arrival at random, A k (t) follows a Poisson distribution with parameter λ.

[0142] This invention considers the interference process of LEO satellite constellations passing near GEO satellites and their ground users for a period of time. The invention dynamically allocates LEO satellite beam power, aiming to ensure that the interference requirements for GEO satellite ground users are always met during this period, while maximizing the data transmission rate of the LEO satellites. Therefore, the modeling of the problem of maximizing the long-term weighted sum and rate maximization under beam power constraints and interference constraints for communication scenarios with coexisting high- and low-Earth orbit satellites is as follows:

[0143]

[0144] Where T represents the number of time slots in which a low-orbit satellite constellation passes through a collinear region; Let represent the LEO beam power allocation scheme for T time slots; P(t) represents the LEO beam power allocation scheme for time slot t, and t represents the time slot; k represents the index of the LEO ground user; Q k (t) represents the virtual data queue that the k-th LEO ground user needs to transmit; C k (t) represents the channel capacity of the k-th LEO ground user; R k (t) represents the actual transmission rate received by the k-th LEO ground user, R k (t)=min(Q k (t),C k (t)); This represents the interference factor of the LEO system on the GEO system; g i,j,g (t) represents the channel gain from the j-th beam of the i-th LEO satellite to the g-th GEO ground user; P i,j (t) represents the power of the j-th beam of the i-th LEO satellite; m i,j,k (t) represents the matching relationship between the j-th beam of the i-th satellite and the k-th LEO ground user; th This represents the interference threshold that the GEO system can tolerate.

[0145] Step 2: Construct a deep reinforcement learning framework for solving the problem of maximizing the long-term weighted sum rate.

[0146] This invention allocates beam power for LEO satellites across consecutive time slots to maximize their long-term weighted sum rate. The invention employs a near-end policy optimization (PPO) algorithm, suitable for solving continuous variables, to obtain beam power allocation. Markov decision processes (MDPs) are a general framework for modeling sequential decision problems. The definitions of key elements in this reinforcement learning framework—agent, environment, state, action, and reward—are as follows:

[0147] Intelligent Agent: The intelligent agent is the central element in the reinforcement learning framework, equivalent to the "brain" of a human. The intelligent agent acquires state information and makes decisions by observing the environment. Since all satellites are managed by the network management center (NMC), the NMC can be considered as an intelligent agent in this invention.

[0148] Environment: In reinforcement learning, all content that interacts with the agent is called the environment. The agent obtains its current state from the environment and performs actions to change the environment. In this invention, the environment refers to all factors in the satellite communication system.

[0149] State: State s t This is the input to the neural network and the basis for the agent's actions. In this invention, to solve the long-term weighted sum rate maximization problem, the agent considers not only the channel conditions of the current time slot but also the data queues of each user when making decisions, thus correlating the beam power allocation across multiple time slots. This addresses the problem of sequential coupling between multiple time slots in the long-term weighted sum rate problem. Therefore, the data queues q(t) of LEO satellite ground users in time slot t, and the channel information... Matching relationship between LEO satellite beams and LEO ground users Model as state:

[0150]

[0151] The data queue q(t) for LEO satellite ground users is represented as:

[0152]

[0153] Channel information Represented as:

[0154]

[0155] in, Including the channel gain vector vec(H) from the i-th LEO satellite to the LEO ground user i (t)) T,vec(·) T This represents the matrix vectorization function, which transforms a matrix into a row vector. The interference channel gain vector vec(G) from the i-th LEO satellite to the GEO ground user. i (t)) T The interference channel gain vector g from GEO satellites to LEO ground users L (t).

[0156] Among them, H i (t) is represented as follows:

[0157]

[0158] Among them, h i,j,k (t) represents the channel gain between the j-th beam of the i-th LEO satellite and the k-th LEO ground user; G i (t) is represented as follows:

[0159]

[0160] Among them, g i,j,g (t) represents the channel gain between the j-th beam of the i-th LEO satellite and the g-th GEO ground user; the interference channel gain vector g L (t) is represented as follows:

[0161]

[0162] A sign function is used to represent the matching relationship between the j-th beam of the i-th LEO satellite and the k-th LEO ground user, h. th Indicates the channel gain threshold; This means that the j-th beam of the i-th LEO satellite covers and serves the k-th LEO satellite ground user. Otherwise, m i,j,k =0 means that the main lobe of the LEO beam does not cover ground users. Therefore, vec(M) i (t)) T M represents the matching relationship between the beam of the i-th LEO satellite and the LEO ground user, where M i (t) is represented as follows:

[0163]

[0164] Therefore, the beam matching relationship between the LEO satellite constellation and LEO ground users It is expressed as follows:

[0165]

[0166] Action: Action at This is the output of the neural network; the agent changes the environment by performing actions. These actions are modeled as LEO beam power allocation vectors, as follows:

[0167]

[0168] in, In the current state s t Next, select action a. t Afterwards, the environment will receive the reward function and change its state from s. t Change to s t+1 ;

[0169] Reward: The reward r t The agent observes state s t Output action a t Then, the environment provides feedback to the agent; the agent then takes action a. t Afterwards, it is necessary to know whether this action satisfies or is close to the optimization objective; reward r t As an environment, action a t Feedback is for action a t The evaluation criteria include: negative rewards for actions that fail to meet beam power or interference constraints; and the reward r. t It is expressed as follows:

[0170]

[0171] Where c1 represents the preset weight, and c2 and c3 represent the penalty factors for not meeting the interference constraint and the penalty factor for not meeting the beam power constraint, respectively; I max (t) represents the maximum interference received by the GEO ground user in time slot t. I th Let n represent the interference threshold of the GEO system, respectively. p (t) represents the action r t The number of beams that do not meet the beam power constraint; R k (t) represents the actual transmission rate received by the k-th LEO ground user; Q k (t) represents the virtual data queue that the k-th LEO ground user needs to transmit.

[0172] Step 3: The training process of the neural network in the reinforcement learning framework for solving the long-term weighted sum rate problem under beam power constraints and interference constraints in communication scenarios where high- and low-orbit satellites coexist.

[0173] The optimization problem shown in Equation (6) is a mixed-integer nonlinear programming problem that is difficult to obtain the optimal solution in polynomial time. This problem involves continuous beam power decision-making processes across multiple consecutive time slots, making the solution even more complex. However, deep reinforcement learning frameworks can approximate the optimal solution quickly and accurately. Proximal Policy Optimization (PPO) is considered one of the most advanced methods in deep reinforcement learning. Based on an actor-critic network architecture, it can solve problems in both continuous and discrete action spaces. In the PPO algorithm, the probability distribution of the policy is parameterized, allowing the agent to directly learn the policy. Regarding convergence, the PPO algorithm limits the range of policy changes through a clip function, resulting in faster and better convergence than natural gradient and trust domain policy optimization.

[0174] Therefore, this invention proposes a deep reinforcement learning architecture based on the PPO algorithm to solve the optimization problem shown in equation (6). This architecture can provide an intelligent and real-time beam power allocation scheme. The main implementation process of the beam power allocation algorithm (DRL-based beam power allocation, drlBPA) based on the deep reinforcement learning framework proposed in this scheme is shown in Algorithm 1:

[0175] Algorithm 1: Beampower Allocation Algorithm Based on DRL

[0176]

[0177]

[0178] The specific execution process is as follows:

[0179] Step 1: Initialize the relevant parameters in the satellite communication system and PPO algorithm.

[0180] Table 1 Satellite Channel System Parameters

[0181]

[0182] Initialize the relevant parameters of the satellite system, including but not limited to the frequency, receiver noise temperature, noise bandwidth, antenna efficiency, high-orbit satellite antenna diameter, low-orbit satellite antenna diameter, 3dB angle, rain attenuation mean and standard deviation, low-orbit satellite operating altitude, low-orbit satellite beam count, and low-orbit satellite maximum power, as shown in Tables 1 and 2. (In rain attenuation A...) R =ζ -1 / 2 In the middle, ζ dB =20log(ζ), μ and σ represent the mean and variance of rain attenuation, respectively, which are related to the receiver's location, the frequency and polarization direction of the receiving user, and the angle relative to the satellite.

[0183] Table 2 Simulation parameters of high- and low-Earth orbit satellite systems

[0184]

[0185] The parameters for initializing the DRL model are shown in Table 3 below. These parameters include at least one of the following: discount factor, learning rate, number of training epochs, trajectory length, update interval, number of updates, and pruning factor. Table 4 below shows the layer structure and number of nodes in each layer of the actor and critic neural networks. Both the actor and critic neural networks consist of an input layer, a fully connected layer, and an output layer, with the tanh function used as the activation function.

[0186] Table 3 DRL Algorithm Simulation Parameter Settings

[0187]

[0188] Table 4 Network Structure Parameters

[0189]

[0190] Step 2: The agent interacts with the environment and stores a batch of data [s] in the experience replay pool. t ,a t ,r t ,s t+1 [The text appears to be incomplete and contains several typos. A more accurate translation would require the full context.] t , will s t The inputs are fed into the actor and critic networks respectively. The actor network outputs the mean of the normal distribution of the beam action. A normal distribution is constructed based on the mean, and the action 'a' is sampled from it. t Execute action a t And calculate the return function r t Get the state s of the next time slot. t+1 This process is repeated, storing data in batches of size `batch_size`. t ,a t ,r t ,s t+1 ].

[0191] Step 3: Train the actor and critic neural networks based on the state data of at least one trajectory (i.e., batch_size = T);

[0192] Step 4: Repeat steps 2 and 3 until the set number of iterations is reached or the loss function converges, at which point the actor and critic network training process ends.

[0193] S3-1 Extract batch_size data from the experience replay pool [s t ,a t ,r t ,s t+1 ] Calculate the dominance estimation function according to the following formula:

[0194] A t =-V(s) t )+r t +γr t+1 +…+γ T-t-1 r T-1 +γ T-t V(s T ),t={1,2,...,T} (10)

[0195] Wherein, V(s) t ) is to put state s t The output value function after being input into the critic network, where γ is the discount factor;

[0196] After obtaining the advantage estimate in S3-2, the loss functions of the actor network and the critic network are calculated:

[0197]

[0198] Among them, L actor (θ) represents the loss function of the actor network, and θ represents the parameters of the actor network. p θ (a t |s t ) indicates that in state s t Choose action a t The probability of ε is the upper and lower bounds controlled by the parameters in the clip function; This represents the average value of a batch of data.

[0199]

[0200] in, This represents the loss function of the critic network. The parameters r of the critic network are represented by r. i This represents the reward for time slot i.

[0201] S3-3 uses the Adam optimizer to optimize the network parameters θ and φ of the actor network and critic network. Update, θold ←θ,

[0202] S3-4 repeats S3-2 and S3-3 multiple times.

[0203] Step 3: In a communication scenario where high-Earth orbit (HEO) and low-Earth orbit (LEO) satellites coexist, the deep reinforcement learning framework works as follows: The data queues to be transmitted by each user in time slot t, channel information, and the matching relationship between the LEO satellite beam and the LEO ground user are input into the actor network. The output of the actor network is the beam power allocation scheme for time slot t. Multiple consecutive states of a trajectory are sequentially input into the actor network, which then outputs the optimal beam power allocation scheme for each time slot in real time, ultimately optimizing the long-term weighted sum rate. It is important to note that in practical applications, the critic network is no longer operational.

[0204] Example

[0205] The following analysis is conducted using this embodiment: To simplify the evaluation scenario, this embodiment considers a high-Earth orbit (GEO) and low-Earth orbit (LEO) coexistence communication system consisting of one GEO satellite and a constellation of four LEO satellites. The LEO satellite constellation shares the spectrum of the GEO satellite communication system and transmits information. The GEO satellite has a single beam and simultaneously serves five GEO ground users within its coverage area; the LEO satellite constellation consists of four LEO satellites with seven beams each, collaboratively serving ten LEO ground users.

[0206] The method provided in this embodiment of the invention is used to analyze the optimization problem of LEO beam power. To verify the beneficial effects of this embodiment, the traditional iterative method is compared with the actor-critic DRL algorithm in the same scenario.

[0207] The system parameters are set as follows:

[0208] Initialize the relevant parameters of the satellite system, including but not limited to the frequency, receiver noise temperature, noise bandwidth, antenna efficiency, high-orbit satellite antenna diameter, low-orbit satellite antenna diameter, 3dB angle, average and standard deviation of rain attenuation, low-orbit satellite operating altitude, low-orbit satellite beam count, and low-orbit satellite maximum power as shown in Tables 5 and 6.

[0209] Table 5 Satellite Channel System Parameters

[0210]

[0211]

[0212] Table 6 Simulation parameters of high- and low-Earth orbit satellite systems

[0213]

[0214] The parameters for initializing the DRL model are shown in Table 7 below. These provide hyperparameters for training the actor and critic networks, including at least one of the following: the number of layers and nodes in each layer of the actor and critic neural networks, discount factor, learning rate, number of training epochs, trajectory length, update interval, number of updates, and pruning factor. During network training, the state consists of the user channel state information and the number of queues of user demand data. T=50 sets of data are randomly generated in each epoch, and a complete training process considers 6000 batches. After multiple experiments, the learning rate of the network was consistently set to 10. -5 .

[0215] Table 7 DRL Algorithm Simulation Parameter Settings

[0216]

[0217] Both the actor and critic neural networks are constructed, each consisting of an input layer, a fully connected layer, and an output layer. The network structure is shown in Table 8 below: <Input dimension, 256, 128, Output dimension>. The tanh function is used as the activation function after each hidden layer to add a non-linear relationship to the data.

[0218] Table 8 Network Structure Parameters

[0219]

[0220] Three baseline schemes were considered as comparison options.

[0221] (1) Fractional optimization-based scheme (FO): The FO scheme is an improvement on the CBSS-MQR scheme mentioned by P.Gu et al. in "Dynamic Cooperative Spectrum Sharing in a Multi-beam LEO-GEO Co-Existing Satellite System". In this scheme, the problem of maximizing the weighted sum rate is transformed into three sub-problems (beam and user allocation (BUA), beam power control (BPC), and time proportional allocation (TPA)). This scheme can only handle instantaneous beam power allocation. However, optimization problem (6) requires solving the power allocation of multiple beams in continuous time slots. Therefore, optimization problem (6) is decomposed into a series of sub-optimization problems:

[0222]

[0223] The FO scheme decomposes the optimization problem shown in equation (6) into a series of static sub-optimization problems, and then uses the CBSS-MQR scheme to solve the sub-problems (13).

[0224] (2) Improved FO-based scheme (IFO): IFO is an improved version of FO. When the proportion of remaining time is not zero and there are still users who are not fully served, the FO scheme will lead to wasted time resources and low spectral efficiency. Therefore, an IFO scheme that makes full use of time resources was developed. Specifically, in IFO, as long as there is a proportion of remaining time and users who are not fully served, the BUA algorithm (grouping all users who are not fully served) and the BPC algorithm will be executed sequentially. The IFO algorithm is described in detail in Algorithm 2.

[0225] Algorithm 2: An improved fractional optimization-based algorithm

[0226]

[0227]

[0228] (3) The DNN Approximate IFO Scheme (dnnAifo): The IFO scheme makes full use of time resources and improves the system throughput. However, the IFO scheme has high computational complexity. Therefore, in order to better meet the real-time requirements of satellite communication, the dnnAifo scheme is proposed. It uses a BPC-DNN based on a DNN model to replace the BPC algorithm based on iterative optimization to solve the beam power allocation problem. The DNN model is trained using the solution of the BPC algorithm as a label.

[0229] Compared to Algorithm 2, the dnnAifo algorithm is identical in all steps except for line 7. The only difference is that line 7 of Algorithm 2, which is the "BPC algorithm" step, is replaced with the "BPC-DNN" step: to obtain the d-th... i Channel gain vector of each user group The input to the DNN network is The output is user group d i Power allocation scheme ".in, It is expressed as follows:

[0230]

[0231] Where, d i Indicates the dthi Indicator vectors for each user group. If k∈d i d i [k] = 1; otherwise, d i [k] = 0.

[0232] Specifically, the DNN model consists of one input layer, three hidden layers, and one output layer. The hidden layer has 400 neurons. The activation functions for the hidden and output layers are the tanh function and the clipped ReLU function, respectively. The training dataset is preprocessed to ensure a 18:1:1 ratio of training, validation, and test sets. The optimizer is rmsprop, and the maximum number of iterations is 600. The minimum batch size is 128, and the initial learning rate is 0.01, which is halved every 70 iterations. The DNNAifo scheme can quickly obtain power allocation results, with speed performance approaching that of the IFO scheme.

[0233] Figure 4 The training process, showing the gradual increase and convergence of cumulative reward for λ=10, 20, and λ=50, is plotted. λ represents the mean of the Poisson distribution followed by the LEO ground user arrival data in each time slot. Each point on the curve represents the average cumulative reward of 4 trajectories (a batch of training data). Figure 4 It can be observed that after extensive training, the increase and convergence of the cumulative reward indicates that the neural network model has been well trained, demonstrating the convergence of the proposed drlBPA algorithm. During the learning process, the cumulative reward exhibits a drop-off-like pattern, because in the early stages of training, the model is still in the exploratory phase and is not suitable for the newly introduced training set, leading to a sharp decrease in reward. The model converges in approximately 5000 iterations, indicating that it has gained experience in maximizing the cumulative reward. The network model was tested on 10,000 scenarios, and its feasibility rate was 100%.

[0234] Figure 5 This relates to the generalization performance of the present invention with respect to parameter λ. To evaluate the generalization performance of the model in the drlBPA scheme with respect to λ, such as... Figure 5 As shown, the average and speed of six trained models were tested and compared under different λ values. The number of randomly arriving LEO ground users in the training datasets used to train these six models follows the parameters λ = 10, 20, ..., 60 Mbps. Figure 5 In the dataset, the horizontal axis λ ranges from 10Mbps to 90Mbps, with intervals of 10Mbps. The nine points on the horizontal axis correspond to nine test datasets. These datasets are identical except for the amount of data arriving randomly from LEO ground users. From... Figure 5It can be seen that when the six models were tested on different test datasets, the average sum rate was approximately the same, with the model with λ = 20 Mbps showing the highest average sum rate. Since the amount of randomly arriving LEO users in the training dataset follows a Poisson distribution with a certain λ, and the model is still applicable to test sets with other Poisson distributions of λ, the drlBPA scheme model exhibits good generalization ability and robustness to λ. This generalization ability indicates that the agent has learned a good policy that can be applied to situations where λ changes dynamically.

[0235] Therefore, the following simulation experiments mainly test the performance of various aspects of the network model with λ=20. The sum and speed obtained by different algorithms are compared under scenarios with smaller and larger λ.

[0236] Figure 6 The diagram shows the downlink and rate cumulative distribution function graphs of the proposed drlBPA and three comparative methods in the dynamic solution of beam power allocation problems when the drlBPA method has converged, in scenarios with λ = 10 Mbps and λ = 50 Mbps. Figure 6 Each curve in the graph is plotted based on the average and rate obtained from 100 implementations. Figure 6 It can be seen that when λ is small, except for the FO scheme, the other three schemes achieve essentially the same throughput, all higher than the FO scheme. When λ is large, the proposed drlBPA scheme can achieve higher throughput. Compared with the comparative schemes, the drlBPA scheme proposed in this invention can effectively improve the throughput of the LEO satellite system. The performance improvement is even more significant when the amount of data arriving from LEO users is large.

[0237] When λ is small, the data rate of random arrivals for LEO satellite ground users is low. The drlBPA and IFO schemes obtain a receiving channel capacity greater than the data rate of random arrivals, allowing all user data to be transmitted within a single time slot. Therefore, when λ is small, the actual transmission rates obtained by the drlBPA and IFO schemes are the same (both equivalent to the data queue Q). The dnnAifo scheme obtains data tags based on the IFO scheme. Once the network model is trained, the dnnAifo scheme can quickly achieve a transmission rate close to that of the IFO scheme. However, when λ is large, the data rate of random arrivals for LEO satellite ground users is high, and the actual transmission rate of each scheme depends on the receiving channel capacity; the larger the channel capacity, the higher the actual transmission rate.

[0238] Figure 7 and Figure 8 The average LEO user queue backlog over 50 consecutive time slots was compared between the dnnAifo scheme and three other comparative schemes. Figure 7As shown, when λ = 10 Mbps, only the FO scheme exhibits a high average user queue backlog, demonstrating instability. Figure 8 In the case of λ = 50 Mbps, all schemes exhibit relatively large average user backlogs, demonstrating instability. However, the proposed drlBPA scheme and the three comparative schemes show unstable queue backlogs of 4, 9, 6, and 6 LEO users, respectively. The maximum LEO user queue backlog of the drlBPA scheme is approximately 1200, which is less than the queue backlogs of the three comparative schemes. In comparison, the proposed drlBPA scheme demonstrates better system stability.

[0239] like Figure 9 The figure shows a comparison of the average and rate of the proposed drlBPA scheme and three comparative schemes as the Poisson distribution parameter increases. From... Figure 9 It can be seen that the average sum rate of the proposed drlBPA scheme is superior to the three comparative schemes. This is because extensive training allows the agent to fully explore the beam power space and learn a better mapping between the state and the beam power policy.

[0240] exist Figure 9 In the above, when λ ≤ 20 Mbps, the average sum rate is the same for all schemes except the FO scheme. This is because, in this case, when λ is small, the channel capacity received by each LEO ground user is much greater than the amount of randomly arriving data. Except for FO, all other methods can complete the transmission of queued backlog data. The FO scheme allocates a time proportion to each user group only once per time slot. When λ is small, only a small portion of time needs to be allocated to each user group to ensure that at least one LEO user completes transmission, which wastes time resources and cannot guarantee that all users' queued backlog data will be transmitted. Therefore, the average sum rate obtained by the FO scheme is lower than the other three schemes. Furthermore, compared to the FO scheme, which does not fully utilize time resources, the IFO scheme sacrifices computational complexity for improved spectral efficiency. When the remaining time proportion of a time slot is sufficient, the IFO scheme continues to allocate time to the user group with the largest weighted sum rate that has not yet completed transmission until all users have completed transmission, while the dnnAifo scheme only reduces computational complexity compared to the IFO scheme.

[0241] Furthermore, in Figure 9 In the case where λ≥130Mbps, the average sum rate of the three comparison schemes is the same because, when λ is large, LEO users... The number of randomly arriving data A k (t) is much larger than the channel capacity C of LEO users. k(t). The three comparison schemes can only complete one time allocation of the user group within a time slot, with the remaining time allocation being zero. Therefore, the actual transmission rates of the three comparison schemes are all limited by the channel capacity and are the same.

[0242] Table 9 shows the execution time (ms) for resource allocation under four different schemes.

[0243] λ = 10 0.133460 125.210 640.330 290.420 λ = 50 0.130034 132.081 387.562 209.812 λ = 90 0.123437 120.470 205.570 138.821 λ = 130 0.133476 93.5250 148.801 86.0400

[0244] The computational complexity of the proposed framework was evaluated. Table 9 lists the average computation time of the proposed drlBPA scheme and three comparative methods under a 10,000-run experimental environment. The results show that, due to the offline training of the neural network, the proposed method has the shortest execution time (approximately 0.130 ms), which is about one-thousandth of the execution time of the three comparative schemes. The results demonstrate that the proposed drlBPA has a significant advantage in runtime and can meet the requirements of real-time BPA in satellite communication systems.

[0245] Furthermore, Table 9 shows that the IFO scheme has the longest average runtime. Compared to the IFO scheme, the dnnAifo scheme has a shorter computation time. However, the improvement in computation time is limited. This is because the dnnAifo scheme only uses a DNN model with lower computational complexity to replace the iterative process of solving the BPC algorithm. Moreover, compared to FO, the dnnAifo scheme has a longer computation time when λ≤90Mbps, but a shorter computation time when λ=130Mbps. This is because in time slot t, when S... k (t)≥Q k When (t), time resources are not fully utilized. In the dnnAifo scheme, BUA and BPC-DNN (for all user groups) are executed multiple times, while in the FO scheme, BUA and BPC (for all user groups) are executed only once. The dnnAifo scheme sacrifices computation time to achieve a higher LEO system speed. When S k (t)<Q k When (t), in the dnnAifo scheme, BUA and BPC-DNN (for all user groups) are executed only once, so the remaining time proportion is zero. In this case, the time advantage of dnnAifo based on the network model becomes apparent.

[0246] A computer device is provided according to an embodiment of the present invention. This computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0247] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0248] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory.

[0249] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0250] The memory can be used to store the computer program and / or module, and the processor implements various functions of the computer device by running or executing the computer program and / or module stored in the memory, and by calling the data stored in the memory.

[0251] If the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0252] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for downlink spectrum sharing and dynamic beam power allocation, characterized in that, Includes the following steps: A long-term weighted sum rate maximization model is constructed under beam power and interference threshold constraints for communication scenarios with coexistence of high- and low-Earth orbit satellites. The specific method is as follows: The model for maximizing the long-term weighted sum rate is as follows: in, T This indicates the number of time slots that a low-Earth orbit satellite constellation passes through near collinear regions; express T LEO beam power allocation scheme for each time slot; Indicates time slot The LEO beam power allocation scheme, and ; and These represent the number of ground users for low-Earth orbit satellites and high-Earth orbit satellites, respectively. Indicates the number of LEO satellites in the LEO satellite constellation; This indicates the number of beams for each LEO satellite; t Indicates a time slot; k Index representing LEO ground users; Indicates the first Each LEO ground user needs to transmit a virtual data queue; Indicates the first Channel capacity for each LEO ground user; Indicates the first The actual transmission rate received by each LEO ground user ; This represents the interference factor of the LEO system on the GEO system; Indicates from the first The first LEO satellite The beam to the first Channel gain for each GEO ground user; Indicates the first The first LEO satellite The power of each beam; This represents the interference threshold that the GEO system can tolerate. Indicates the first The first satellite The beam and the first Matching relationships among LEO ground users; This indicates the maximum beam power of the LEO satellite; Construct a deep reinforcement learning framework for solving the problem of maximizing the rate of a long-term weighted sum; The neural network in the deep reinforcement learning framework is trained using a proximal policy optimization algorithm. The data queues, channel information, and matching relationships between LEO ground users and LEO satellite beams for each time slot are considered as states and sequentially input into a trained neural network model to obtain the LEO satellite beam power allocation scheme for each time slot.

2. The downlink spectrum sharing and dynamic beam power allocation method according to claim 1, characterized in that, The virtual data queue The queuing dynamics satisfy the following conditions: in, Indicates in time slot t The transmission rate that can be provided to users; Indicates in time slot t No. The data rate at which each LEO ground user arrives randomly Obtain the parameter as λ The Poisson distribution.

3. The downlink spectrum sharing and dynamic beam power allocation method according to claim 1, characterized in that, The deep reinforcement learning framework for solving the problem of maximizing the long-term weighted sum rate includes: Intelligent Agent: The intelligent agent corresponds to the Network Management Center (NMC); Environment: The environment refers to all factors in a satellite communication system; State: The state It is the input to the neural network and the basis for the agent to execute actions; it divides time slots LEO satellite ground user data queue Channel information Matching relationship between LEO satellite beams and LEO ground users Model as state: The data queue of LEO satellite ground users Represented as: The channel information Represented as: in, Including the i Channel gain vector from LEO satellite to LEO ground user , This represents a matrix vectorization function that transforms a matrix into a row vector; the... i Interference channel gain vector from LEO satellites to GEO ground users Interference channel gain vector from GEO satellite to LEO ground users ; in, It is expressed as follows: in, Indicates the first i The first LEO satellite j The beam to the first k Channel gain between LEO ground users; It is expressed as follows: in, Indicates the first i The first LEO satellite j The beam to the first Channel gain between GEO ground users; interference channel gain vector It is expressed as follows: Using symbolic functions to indicate the first i The first LEO satellite The beam and the first Matching relationships between LEO ground users. Indicates the channel gain threshold; Meaning the first i The first LEO satellite j The beam covers and serves the first k LEO satellite ground users ;otherwise, This means that the main lobe of the LEO beam does not cover ground users. ;therefore, Indicates the first i The matching relationship between the beams of LEO satellites and LEO ground users, among which, It is expressed as follows: Therefore, the beam matching relationship between the LEO satellite constellation and LEO ground users It is expressed as follows: Action: The action described This is the output of the neural network; the agent changes the environment by performing actions. These actions are modeled as LEO beam power allocation vectors, as follows: in, In the current state Next, select an action. Afterwards, the environment will receive the reward function and change its state from Change to ; Rewards: The rewards mentioned r t It is the intelligent agent that observes the state Output action Then, the environment provides feedback to the agent; the agent then takes action. Afterwards, it is necessary to know whether this action meets or is close to the optimization goal; reward. As an environment to action Feedback is the action The evaluation criteria include: negative rewards for actions that fail to meet beam power or interference constraints; and the rewards are... It is expressed as follows: in, c 1 represents the preset weight. c 2 and c 3 represents the penalty factor for not satisfying the interference constraint and the penalty factor for not satisfying the beam power constraint, respectively; and They represent time slots respectively t Maximum interference and interference threshold received by GEO ground users Indicates action The number of beams that do not meet the beam power constraint; Indicates the first The actual transmission rate received by each LEO ground user; Indicates the first A virtual data queue that needs to be transmitted for each LEO ground user.

4. The downlink spectrum sharing and dynamic beam power allocation method according to claim 3, characterized in that, The user In the time slot Channel capacity as follows: in, Indicates the first LEO ground users, Indicates the LEO satellite beam bandwidth. Indicates in time slot LEO ground users The received signal-to-noise ratio; LEO ground users Received signal-to-noise ratio ,as follows: Among them, in time slots t , Indicates the first i The first LEO j Does the beam cover and serve the first...? LEO ground users, , indicates coverage and service; otherwise. Indicates that it is not covered; Representing the satellite channel model, Indicates the first i The first LEO satellite j The beam to the first k LEO ground users Channel gain between; This indicates the beam distance from the GEO satellite to the LEO satellite. k individual users Channel gain between; Indicates noise power; The satellite channel model as follows: in, , , and These represent the antenna gain for satellite users, antenna gain for end users, free space loss, and rain attenuation, respectively; the antenna gain at the transmitting or receiving end is expressed as: in, This represents the maximum antenna gain when the off-axis angle is 0. and Let these represent the first-order and third-order Bessel functions, respectively. ,in, This indicates the off-axis angle, which is related to the antenna direction and position of the satellite and the end user; This represents the off-axis angle corresponding to a 3dB beamwidth; since the LEO user antenna continuously tracks the LEO satellite, the off-axis angle at the receiver is 0. Free space loss ,in, , and These represent the distance between the satellite and the ground user, the frequency, and the speed of light, respectively; rain attenuation. ,in, The amplitude representing rain attenuation, It follows a log-normal distribution.

5. The downlink spectrum sharing and dynamic beam power allocation method according to claim 4, characterized in that, The method of training the neural network in the deep reinforcement learning framework using the proximal policy optimization algorithm includes: Step 1: Initialize the relevant parameters in the satellite communication system and the deep reinforcement learning framework; Step 2: The agent interacts with the environment and stores the data. Data sets ,in, Indicates the size of a batch; specifically, in a time slot. The agent observes the environment to obtain the current state. ,Will The beam action is fed into the actor and critic networks respectively. The actor network outputs the mean of the normal distribution of the beam action. A normal distribution is constructed based on the mean, and the action is sampled from it. ; Perform actions And calculate the return function. Get the state of the next time slot. This process is repeated, with a storage size of [value missing]. Group data ; Step 3: Train the actor and critic neural networks; Step 4: Repeat steps 2 and 3 until the set number of iterations is reached or the loss function converges, at which point the actor and critic network training process ends.

6. The downlink spectrum sharing and dynamic beam power allocation method according to claim 5, characterized in that, Step 3 is as follows: S3-1 Extraction data Calculate the dominance estimation function according to the following formula: in, It is the state The output value function after being input into the critic network. It is a discount factor; After obtaining the advantage estimate in S3-2, the loss functions of the actor network and the critic network are calculated: in, This represents the loss function of the actor network. The parameters representing the actor network, , Indicates the state Select action The probability, ε for Parameters that control the upper and lower bounds in a function; This represents the average value of a batch of data. in, This represents the loss function of the critic network. These represent the parameters of the critic network. Indicates time slot The reward; S3-3 uses the Adam optimizer to optimize the network parameters of the actor and critic networks. and Update , ; S3-4 repeats S3-2 and S3-3 multiple times.

7. A downlink spectrum sharing and dynamic beam power allocation system for implementing the method of claim 1, characterized in that, include: The model building module is used to build a long-term weighted sum rate model under beam power constraints and interference threshold constraints for communication scenarios where high- and low-Earth orbit satellites coexist. Framework building blocks are used to construct deep reinforcement learning frameworks for solving the problem of maximizing the rate of long-term weighted sums. The model training module is used to train neural networks in a deep reinforcement learning framework using a proximal policy optimization algorithm. The power allocation scheme calculation module is used to treat the data queues, channel information, and matching relationship between LEO ground users and LEO satellite beams of each time slot as states, and input them sequentially into the trained neural network model to obtain the LEO satellite beam power allocation scheme for each time slot.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images