Resource allocation method for multi-beam satellite communication system

Abstract

The invention relates to a multi-beam satellite communication system resource allocation method, and belongs to the technical field of wireless communication. The method comprises the following steps: S1, modeling a satellite communication system model; s2, determining a user clustering strategy; s3, modeling a beam illumination variable and a power distribution variable; s4, modeling a user rate model; s5, modeling a user service model and a satellite queue model; s6, modeling a user cluster data volume to be transmitted and a system cost function; s7, modeling a system resource allocation constraint condition; s8, modeling a system state, an action and an award; s9, constructing and training a stage strategy gradient PPG network; and S10, determining a system resource allocation strategy by using the trained PPG network. According to the invention, through joint optimization of beam illumination, beam power distribution and in-cluster beam forming strategies, the system accumulated reward maximization is realized.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, relates to the field of satellite communication technology, and specifically relates to a resource allocation method for a multi-beam satellite communication system. Background Art

[0002] In recent years, multi-beam low-orbit satellite networks have become an effective supplement to ground-based cellular networks due to their ability to provide global coverage, meet diverse business needs, and resist major natural disasters. In order to improve the resource utilization and overall system performance of multi-beam low-orbit satellite networks, beam lighting optimization and wireless resource allocation are effective solutions. By allocating different time-frequency resources and power resources to users, different business needs of users can be maximized. However, due to the dynamic changes in network topology caused by satellite movement, the uneven geographical distribution of user business needs, and the contradiction between growing user needs and limited resources, the beam lighting and resource allocation problems of multi-beam low-orbit satellite systems face challenges. How to design efficient beam lighting, beam power allocation, and intra-cluster beamforming strategies to achieve system performance improvement has become an important research topic.

[0003] Existing research has considered the resource allocation problem of multi-beam satellite communication systems, but fewer studies have considered the inability to serve all users simultaneously and the optimization of the average performance of the system in scenarios with multiple multi-beam low-orbit satellites, resulting in limited performance of resource allocation schemes. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a resource allocation method for a multi-beam satellite communication system. For a communication system including multiple multi-beam low-orbit satellites and multiple ground users, the cumulative reward of the modeling system is used as the optimization target, and the beam illumination, beam power allocation and intra-cluster beamforming strategy are optimized to improve the overall performance of the system.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-beam satellite communication system resource allocation method specifically comprises the following steps:

[0007] S1: Modeling satellite communication system model;

[0008] S2: Determine user clustering strategy;

[0009] S3: Modeling beam illumination variables and power allocation variables;

[0010] S4: Modeling user rate model;

[0011] S5: Modeling user traffic model and satellite queue model;

[0012] S6: Modeling the amount of data to be transmitted by the user cluster and the system cost function;

[0013] S7: Modeling system resource allocation constraints;

[0014] S8: Modeling system states, actions, and rewards;

[0015] S9: Build and train a phased policy gradient PPG network;

[0016] S10: Determine a system resource allocation strategy using the trained PPG network.

[0017] Further, in step S1, a satellite communication system model is built, specifically including: the system includes multiple multi-beam low-orbit satellites and multiple ground users, let M represent the number of low-orbit satellites, S m represents the mth satellite, each satellite is equipped with a uniform planar array of size N s =N x ×N y , each satellite can generate N beams simultaneously, let K represent the number of ground users, U k represents the kth user, each user is a single-antenna user; let P tot Represents the total power of a single satellite, P max represents the maximum transmission power of a single satellite beam; divide the system time into T consecutive equal-length time slots, with a time slot length of τ; let q m (t) = [x m (t),y m (t),H m (t)] T Indicates time slot S m The coordinates of q k =[x k ,y k ,0] T Indicates U k Let B represent the total bandwidth of the satellite beam, assuming that all beams of a single satellite occupy N time-frequency resource blocks in an orthogonal manner;

[0018] Modeling the satellite channel model, specifically including: let h m,k (t) represents the time slot S m with U k The channel gain of the link between where g m,k (t),υ m,k , τ m,k and ν m,k are t time slots S m To U k The complex gain, Doppler shift, propagation delay and array response vector of the link, fc is the carrier frequency; g m,k (t) can be modeled as where δ m,k (t) represents the time slot S m and U k The Rice fading factor between c and d m,k (t) respectively represent S m The transmit antenna gain, U k The receiving antenna gain, the speed of light and the time slot S m with U k The distance between m,k Can be modeled as in, and Represent the array response vectors in the x-axis and y-axis directions respectively, represents the Kronecker product.

[0019] Further, in step S2, the user clustering strategy is determined, which specifically includes: max represents the coverage radius of the satellite beam. The mean shift algorithm is used to design the user clustering strategy. The specific steps are as follows:

[0020] (1) Initialization: Let Φ represent the set of unclustered users, that is, Φ = {U k ,1≤k≤K},C i represents the user set of the i-th cluster, 1≤i<K, let i=1;

[0021] (2) Select the initial center point: Randomly select an unclustered user and use its position as the initial center point. For example, select U k ∈Φ, q k As C i The initial center point of

[0022] (3) Determine the initial cluster members: Calculate the number of unclustered users and If the distance Then U k′ Add to C i , that is, C i =C i ∪{U k′}, let L i Represents C i Number of users in

[0023] (4) Update the cluster center: Let M i Represents C i The mean shift vector of can be modeled as: in is the Gaussian kernel function, σ is the Gaussian kernel function parameter; based on Update C i The center point of

[0024] (5) Update cluster members: Repeat steps (3) and (4) until M i =0; Update C i , delete C i Zhongyu The distance is greater than R max , that is, if C i =C i \{U k′}; Update Φ, delete the clustered users, if U k′ ∈C i , then Φ=Φ\{U k′};

[0025] (6) Determine whether the initial clustering algorithm is terminated: Determine whether there are unclustered U k′ ,like Then let i=i+1 and return to step (2). Otherwise, execute step (7) and record the total number of user clusters as I.

[0026] (7) Calculate the clustering strategy evaluation function: Determine the clustering performance evaluation function based on the intra-cluster and inter-cluster distances, let S i Represents C i The dispersion is defined as C i The average distance from all users to the center point can be modeled as make Represents C i With C j The distance between them can be modeled as Let F i,j C i With C j The similarity measure between can be modeled as Let F i Represents C i The evaluation function can be modeled as Let F represent the clustering strategy evaluation function, which can be modeled as

[0027] (8) Determine the update conditions of clustering strategy: Let F th is the predefined evaluation function threshold, if F≤F th , the algorithm terminates, the current clustering result is the final result, and the user clustering strategy β is output i,k ∈{0,1}, that is, if U k ∈C i , then β i,k =1, otherwise βi,k =0; if F>F th , let σ=ασ, 0<α<1, return to step (1).

[0028] Further, in step S3, the beam illumination variables are modeled, specifically including: modeling α m,n,i (t)∈{0,1} is the time slot S m The beam n and C i The illumination variable between t and S m The beam n illuminates C i , then α m,n,i (t)=1, otherwise α m,n,i (t) = 0;

[0029] Modeling power allocation variables, including: Modeling time slot S m The beam power allocation vector is P m (t) = [p m,1 (t),p m,2 (t),…p m,N (t)] T , where p m,n (t) represents the time slot S m The transmit power of beam n.

[0030] Further, in step S4, a user rate model is modeled, specifically including: dividing the message sent by the satellite to the user into a public part and a private part, uniformly encoding the public part of all user messages in each cluster into a public stream, and independently encoding the private part of each user message into a user private stream; the satellite respectively designs precoding for the public stream and multiple private streams of the users in the cluster, and multiplexes and transmits them; after the users in the cluster receive the message from the satellite, they decode the public stream and apply the continuous interference cancellation SIC technology, after removing the public stream from the received signal, each user then respectively decodes its corresponding private stream;

[0031] Order i,c (t) represents the time slot C i The public stream of all user messages in s i,k (t) represents the time slot C i User U k The private stream of Indicates time slot S m With C i The beamforming matrix during communication, where w m,i,c (t) and w m,i,k (t) are public flow and U k Precoding vector of private stream; t time slot S m Send to C i The signal can be modeled as tTime slot Ci Inside U k Received from S m The signal can be modeled as in represents additive white Gaussian noise;

[0032] Let γ m,i,k,c (t) and γ m,i,k (t) represents the time slot C i Inside U k Received from S m The signal-to-interference-noise ratios of the public and private flows in the signal are modeled as and Let R m,i,c (t) represents the time slot S m to C i The public flow transmission rate can be modeled as Let R m,i,k (t) represents the time slot S m to C i Inside U k The private flow transmission rate can be modeled as R m,i,k (t) = Blog2(1+γ m,i,k (t)); Indicates time slot S m to C i Inside U k The sum rate can be modeled as

[0033] Further, in step S5, a user service model is modeled, specifically including: assuming that the user service flow arrives randomly and dynamically, and the service volume arriving at each user in each time slot follows a Poisson distribution; let A k (t) represents U k The traffic volume arriving at time slot t is expected to be E[A k (t)]=λ k τ, where λ k Indicates U k Average arrival rate of business flows;

[0034] The satellite queue model is built, which includes: each satellite is equipped with a user data cache server, which can store the data streams randomly arriving from each user in the corresponding queue. max Indicates the maximum capacity of the satellite buffer; Q m,k (t) represents the end of time slot t m Cache U k The queue length of the data flow can be modeled as:

[0035]

[0036] Further, in step S6, the amount of data to be transmitted by the user cluster is modeled, which specifically includes: m,i (t) represents the time slot S m to C i The amount of data to be transmitted can be modeled as Let r(t) represent the system cost function of time slot t, which can be modeled as

[0037] Further, in step S7, the modeling system resource allocation constraint conditions are specifically including:

[0038] (1) Beam illumination constraints

[0039] In any time slot, a single beam of each satellite can only serve one user cluster at most, so:

[0040]

[0041] In any time slot, each user cluster is served by at most one beam of one satellite, so:

[0042]

[0043] (2) Beam Transmit Power Constraint

[0044] The total transmission power of multiple beams of each satellite in any time slot cannot exceed the total transmission power of the satellite, so:

[0045]

[0046] There is a maximum transmit power limit for a single beam in any time slot, so:

[0047]

[0048] The power that can be allocated to the data stream of each user in the cluster cannot exceed the transmit power of the beam, so:

[0049]

[0050] In order to successfully implement SIC while decoding the public stream at the receiving end, the public stream signal strength should be greater than the private stream and noise, so:

[0051]

[0052] where θ th is the minimum power difference between the public flow and the private flow and the noise.

[0053] Further, in step S8, the system state, action and reward are modeled, specifically including: defining the global state space of time slot t as s t ={s t,m|1≤m≤M}, where s t,m = {Q m,k (t),h m,k (t)1≤k≤K} represents the time slot S m The state of t time slot joint action space is defined as a t ={α m,n,i (t),p m,n (t),W m,i (t)1≤m≤M,1≤n≤N,1≤i≤I}, including beam illumination, beam power allocation and intra-cluster beamforming strategy; let r(t) be the system reward function for time slot t.

[0054] Further, in step S9, a phased policy gradient PPG network is constructed and trained, specifically including: the PPG network includes a policy network and a value network, and θ and φ are respectively represented by the parameters of the policy network and the value network; the training process of the PPG network is alternately performed by a policy training phase and a knowledge distillation-assisted training phase, wherein the policy training phase uses a proximal policy optimization PPO algorithm to train the agent, and the knowledge distillation-assisted training phase can send useful information of the value network to the policy network; initialize the policy network and value network parameters and the experience replay cache pool Given an initial state s t , the agent follows the output of the policy network π(a t |s t θ t ) Perform action a t , and get reward r after interacting with the environment t , the system transfers to the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 )Deposit Each time the network parameters are updated, Extract training samples from the network and update the parameters of the policy network and the value network alternately;

[0055] Define the loss function of the policy network as in Indicates that the current strategy is different from the previous strategy in state s t Take action a t The probability ratio, π(a t |s t θ t ) indicates that under the current strategy, in state s t Take action a t The probability of π(a t |s t θ old ) indicates that under the previous strategy, in state st Take action a t The probability of A(t) = δ t +(γλ)δ t+1 +…+(γλ) X-t+1 δ X-1 represents the advantage function of time slot t, δ t =r t +γV φ (s t+1 )-V φ (s t ) represents the time difference error, V φ (s t ) indicates state s t The value function of , γ∈(0,1) is the discount factor, λ is a constant, X is the trajectory length; ε∈(0,1) is the truncation coefficient, clip(μ t (θ t ),1-ε,1+ε) is the truncation function, which means that μ t (θ t ) is limited to [1-ε, 1+ε] to ensure that the policy update amplitude is not too large; the stochastic gradient ascent method is used to update the parameters θ of the policy network, and its update formula can be modeled as θ t+1 =θ t +η·▽ θ L(θ t ), where ▽ θ L(θ t ) represents L(θ t ) is the gradient of the parameter θ, and η is the learning rate;

[0056] The loss function of the value network is defined as in is the target value function of time slot t; the parameter φ of the value network is updated by stochastic gradient descent method, and its update formula can be modeled as φ t+1 =φ t -η·▽ φ L(φ t );

[0057] In the knowledge distillation assisted training phase, the loss function of the strategy network is defined as Where L(θ t ) aux For the auxiliary loss part, by using the auxiliary value function V of the policy network θ (s t ) to learn useful information of the value network, which can be modeled as L(θ t ) joint The second term is the behavior cloning loss part, β clone is the cloning parameter, π(·st θ old ) is the strategy before the auxiliary phase begins, π(·s t θ t ) is the current strategy, KL(π(·s t θ old ),π(·s t θ t )) is used to calculate π(·s t θ old ) and π(·s t θ t ) between them; the stochastic gradient descent method is used to update the parameters θ of the policy network and the parameters φ of the value network respectively, and the update formulas can be modeled as θ t+1 =θ t -η·▽ θ L(θ t ) joint and φ t+1 =φ t -η·▽ φ L(φ t ).

[0058] Further, in step S10, the trained PPG network is used to determine the beam illumination, beam power allocation and intra-cluster beamforming strategy, which specifically includes: optimizing and determining the resource allocation strategy with the goal of maximizing the system cumulative reward under the condition of satisfying the beam illumination and beam transmission power constraints, that is:

[0059]

[0060] in and They are optimal beam illumination, beam power allocation and intra-cluster beamforming strategy respectively.

[0061] The beneficial effect of the present invention is that the method of the present invention can maximize the system cumulative reward and improve the overall performance of the system based on beam illumination, beam power allocation and intra-cluster beamforming strategies while ensuring the communication needs of different users.

[0062] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0064] Figure 1 This is a schematic diagram of a multi-beam satellite communication system scenario;

[0065] Figure 2 The figure is a flow chart of a method for allocating resources in a multi-beam satellite communication system according to the present invention. DETAILED DESCRIPTION

[0066] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0067] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0068] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0069] See also Figure 1-2 , Figure 1 It is a schematic diagram of a multi-beam satellite communication system scenario. Figure 1 As shown in the figure, the communication system includes multiple multi-beam low-orbit satellites and multiple ground users. The system cumulative reward can be maximized by jointly designing the optimal beam illumination, beam power allocation and intra-cluster beamforming strategy.

[0070] Figure 2 FIG. 1 is a flow chart of a method for allocating resources in a multi-beam satellite communication system according to the present invention. Figure 2 As shown, the method specifically comprises the following steps:

[0071] Step 1: Model the satellite communication system.

[0072] The system includes multiple multi-beam low-orbit satellites and multiple ground users. Let M represent the number of low-orbit satellites, S m represents the mth satellite, each satellite is equipped with a uniform planar array of size N s =N x ×N y , each satellite can generate N beams simultaneously, let K represent the number of ground users, U k represents the kth user, each user is a single-antenna user; let P tot Represents the total power of a single satellite, P max represents the maximum transmission power of a single satellite beam; divide the system time into T consecutive equal-length time slots, with a time slot length of τ; let q m (t) = [x m (t),y m (t),H m (t)] T Indicates time slot S m The coordinates of q k =[x k ,y k ,0] T Indicates U k Let B represent the total bandwidth of the satellite beam, assuming that all beams of a single satellite occupy N time-frequency resource blocks in an orthogonal manner;

[0073] Modeling the satellite channel model, specifically including: let h m,k (t) represents the time slot S m with U k The channel gain of the link between where g m,k (t),υ m,k , τ m,k and ν m,k are t time slots S m To U k The complex gain, Doppler shift, propagation delay and array response vector of the link, f c is the carrier frequency; g m,k (t) can be modeled as where δ m,k (t) represents the time slot S m and U k The Rice fading factor between c and d m,k (t) respectively represent S m The transmit antenna gain, U kThe receiving antenna gain, the speed of light and the time slot S m with U k The distance between m,k Can be modeled as in, and Represent the array response vectors in the x-axis and y-axis directions respectively, represents the Kronecker product.

[0074] Step 2: Determine the user clustering strategy.

[0075] Let R max represents the coverage radius of the satellite beam. The mean shift algorithm is used to design the user clustering strategy. The specific steps are as follows:

[0076] (1) Initialization: Let Φ represent the set of unclustered users, that is, Φ = {U k ,1≤k≤K},C i represents the user set of the i-th cluster, 1≤i<K, let i=1;

[0077] (2) Select the initial center point: Randomly select an unclustered user and use its position as the initial center point. For example, select U k ∈Φ, q k As C i The initial center point of

[0078] (3) Determine the initial cluster members: Calculate the number of unclustered users and If the distance Then U k′ Add to C i , that is, C i =C i ∪{U k′}, let L i Represents C i Number of users in

[0079] (4) Update the cluster center: Let M i Represents C i The mean shift vector of can be modeled as: in is the Gaussian kernel function, σ is the Gaussian kernel function parameter; based on Update C i The center point of

[0080] (5) Update cluster members: Repeat steps (3) and (4) until M i =0; Update C i , delete C i Zhongyu The distance is greater than Rmax , that is, if C i =C i \{U k′}; Update Φ, delete the clustered users, if U k′ ∈C i , then Φ=Φ\{U k′};

[0081] (6) Determine whether the initial clustering algorithm is terminated: Determine whether there are unclustered U k′ ,like Then let i=i+1 and return to step (2). Otherwise, execute step (7) and record the total number of user clusters as I.

[0082] (7) Calculate the clustering strategy evaluation function: Determine the clustering performance evaluation function based on the intra-cluster and inter-cluster distances, let S i Represents C i The dispersion is defined as C i The average distance from all users to the center point can be modeled as make Represents C i With C j The distance between them can be modeled as Let F i,j C i With C j The similarity measure between can be modeled as Let F i Represents C i The evaluation function can be modeled as Let F represent the clustering strategy evaluation function, which can be modeled as

[0083] (8) Determine the update conditions of clustering strategy: Let F th is the predefined evaluation function threshold, if F≤F th , the algorithm terminates, the current clustering result is the final result, and the user clustering strategy β is output i,k ∈{0,1}, that is, if U k ∈C i , then β i,k =1, otherwise β i,k =0; if F>F th , let σ=ασ, 0<α<1, return to step (1).

[0084] Step 3: Model beam illumination variables and power allocation variables.

[0085] Modeling α m,n,i (t)∈{0,1} is the time slot S m The beam n and Ci The illumination variable between t and S m The beam n illuminates C i , then α m,n,i (t)=1, otherwise α m,n,i (t) = 0;

[0086] Modeling power allocation variables, including: Modeling time slot S m The beam power allocation vector is P m (t) = [p m,1 (t),p m,2 (t),…p m,N (t)] T , where p m,n (t) represents the time slot S m The transmit power of beam n.

[0087] Step 4: Model the user rate model.

[0088] The messages sent by the satellite to the user are divided into a public part and a private part. The public part of all user messages in each cluster is uniformly encoded into a public stream, and the private part of each user message is independently encoded into a user private stream. The satellite designs precoding for the public stream and multiple private streams of the users in the cluster respectively, and multiplexes them for transmission. After receiving the message from the satellite, the users in the cluster decode the public stream and apply the continuous interference cancellation SIC technology. After removing the public stream from the received signal, each user decodes its corresponding private stream respectively.

[0089] Order i,c (t) represents the time slot C i The public stream of all user messages in s i,k (t) represents the time slot C i User U k The private stream of Indicates time slot S m With C i The beamforming matrix during communication, where w m,i,c (t) and w m,i,k (t) are public flow and U k Precoding vector of private stream; t time slot S m Send to C i The signal can be modeled as tTime slot C i Inside U k Received from S m The signal can be modeled as in represents additive white Gaussian noise;

[0090] Let γ m,i,k,c (t) and γm,i,k (t) represents the time slot C i Inside U k Received from S m The signal-to-interference-noise ratios of the public and private flows in the signal are modeled as and Let R m,i,c (t) represents the time slot S m to C i The public flow transmission rate can be modeled as Let R m,i,k (t) represents the time slot S m to C i Inside U k The private flow transmission rate can be modeled as R m,i,k (t) = Blog2(1+γ m,i,k (t)); Indicates time slot S m to C i Inside U k The sum rate can be modeled as

[0091] Step 5: Model the user service model and satellite queue model.

[0092] Assume that user traffic arrives randomly and dynamically, and the traffic arriving at each user in each time slot follows a Poisson distribution; let A k (t) represents U k The traffic volume arriving at time slot t is expected to be E[A k (t)]=λ k τ, where λ k Indicates U k Average arrival rate of business flows;

[0093] The satellite queue model is built, which includes: each satellite is equipped with a user data cache server, which can store the data streams randomly arriving from each user in the corresponding queue. max Indicates the maximum capacity of the satellite buffer; Q m,k (t) represents the end of time slot t m Cache U k The queue length of the data flow can be modeled as:

[0094]

[0095] Step 6: Model the amount of data to be transmitted by the user cluster and the system cost function.

[0096] Let O m,i (t) represents the time slot S m to C i The amount of data to be transmitted can be modeled as Let r(t) represent the system cost function of time slot t, which can be modeled as

[0097] Step 7: Model system resource allocation constraints.

[0098] (1) Beam illumination constraints

[0099] In any time slot, a single beam of each satellite can only serve one user cluster at most, so:

[0100]

[0101] In any time slot, each user cluster is served by at most one beam of one satellite, so:

[0102]

[0103] (2) Beam Transmit Power Constraint

[0104] The total transmission power of multiple beams of each satellite in any time slot cannot exceed the total transmission power of the satellite, so:

[0105]

[0106] There is a maximum transmit power limit for a single beam in any time slot, so:

[0107]

[0108] The power that can be allocated to the data stream of each user in the cluster cannot exceed the transmit power of the beam, so:

[0109]

[0110] In order to successfully implement SIC while decoding the public stream at the receiving end, the public stream signal strength should be greater than the private stream and noise, so:

[0111]

[0112] where θ th is the minimum power difference between the public flow and the private flow and the noise.

[0113] Step 8: Model system states, actions, and rewards.

[0114] Define the global state space of time slot t as s t ={s t,m |1≤m≤M}, where s t,m = {Q m,k (t),h m,k (t)1≤k≤K} represents the time slot S mThe state of t time slot joint action space is defined as a t ={α m,n,i (t),p m,n (t),W m,i (t)1≤m≤M,1≤n≤N,1≤i≤I}, including beam illumination, beam power allocation and intra-cluster beamforming strategy; let r(t) be the system reward function for time slot t.

[0115] Step 9: Build and train the phased policy gradient PPG network.

[0116] The PPG network includes a policy network and a value network. Let θ and φ represent the parameters of the policy network and the value network respectively. The training process of the PPG network is carried out alternately by the policy training phase and the knowledge distillation-assisted training phase. The policy training phase uses the proximal policy optimization PPO algorithm to train the agent. The knowledge distillation-assisted training phase can send useful information of the value network to the policy network. Initialize the policy network and value network parameters and the experience replay cache pool. Given an initial state s t , the agent follows the output of the policy network π(a t |s t θ t ) Perform action a t , and get reward r after interacting with the environment t , the system transfers to the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 )Deposit Each time the network parameters are updated, Extract training samples from the network and update the parameters of the policy network and the value network alternately;

[0117] Define the loss function of the policy network as in Indicates that the current strategy is different from the previous strategy in state s t Take action a t The probability ratio, π(a t |s t θ t ) indicates that under the current strategy, in state s t Take action a t The probability of π(a t |s t θ old ) indicates that under the previous strategy, in state s t Take action a t The probability of A(t) = δ t +(γλ)δ t+1+…+(γλ) X-t+1 δ X-1 represents the advantage function of time slot t, δ t =r t +γV φ (s t+1 )-V φ (s t ) represents the time difference error, V φ (s t ) indicates state s t The value function of , γ∈(0,1) is the discount factor, λ is a constant, X is the trajectory length; ε∈(0,1) is the truncation coefficient, clip(μ t (θ t ),1-ε,1+ε) is the truncation function, which means that μ t (θ t ) is limited to [1-ε, 1+ε] to ensure that the policy update amplitude is not too large; the stochastic gradient ascent method is used to update the parameters θ of the policy network, and its update formula can be modeled as θ t+1 =θ t +η·▽ θ L(θ t ), where ▽ θ L(θ t ) represents L(θ t ) is the gradient of the parameter θ, and η is the learning rate;

[0118] The loss function of the value network is defined as in is the target value function of time slot t; the parameter φ of the value network is updated by stochastic gradient descent method, and its update formula can be modeled as φ t+1 =φ t -η·▽ φ L(φ t );

[0119] In the knowledge distillation assisted training phase, the loss function of the strategy network is defined as Where L(θ t ) aux For the auxiliary loss part, by using the auxiliary value function V of the policy network θ (s t ) to learn useful information of the value network, which can be modeled as L(θ t ) joint The second term is the behavior cloning loss part, β clone is the cloning parameter, π(·s t θ old ) is the strategy before the auxiliary phase begins, π(·s t θ t) is the current strategy, KL(π(·s t θ old ),π(·s t θ t )) is used to calculate π(·s t θ old ) and π(·s t θ t ) between them; the stochastic gradient descent method is used to update the parameters θ of the policy network and the parameters φ of the value network respectively, and the update formulas can be modeled as θ t+1 =θ t -η·▽ θ L(θ t ) joint and φ t+1 =φ t -η·▽ φ L(φ t ).

[0120] Step 10: Use the trained PPG network to determine the system resource allocation strategy.

[0121] Under the conditions of satisfying the constraints of beam illumination and beam transmission power, the resource allocation strategy is optimized to maximize the system cumulative reward, namely:

[0122]

[0123] in and They are optimal beam illumination, beam power allocation and intra-cluster beamforming strategy respectively.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-beam satellite communication system resource allocation method, characterized in that: The method specifically comprises the following steps: S1: Modeling satellite communication system model; S2: Determine user clustering strategy; S3: Modeling beam illumination variables and power allocation variables; S4: Modeling user rate model; S5: Modeling user traffic model and satellite queue model; S6: Modeling the amount of data to be transmitted by the user cluster and the system cost function; S7: Modeling system resource allocation constraints; S8: Modeling system states, actions, and rewards; S9: Build and train a phased policy gradient PPG network; S10: Determine a system resource allocation strategy using the trained PPG network.

2. A multi-beam satellite communication system resource allocation method according to claim 1, characterized in that: In step S1, the satellite communication system model is specifically constructed as follows: the system includes a plurality of multi-beam low-orbit satellites and a plurality of ground users, and M is used to represent the number of low-orbit satellites, S m represents the mth satellite, each satellite is equipped with a uniform planar array of size N s =N x ×N y , each satellite can generate N beams simultaneously, let K represent the number of ground users, U k represents the kth user, each user is a single-antenna user; let P tot Represents the total power of a single satellite, P max represents the maximum transmission power of a single satellite beam; divide the system time into T consecutive equal-length time slots, with a time slot length of τ; let q m (t) = [x m (t),y m (t),H m (t)] T Indicates time slot S m The coordinates of q k =[x k ,y k ,0] T Indicates U k Let B represent the total bandwidth of the satellite beam, assuming that all beams of a single satellite occupy N time-frequency resource blocks in an orthogonal manner; Modeling the satellite channel model, specifically including: let h m,k (t) represents the time slot S m with U k The channel gain of the link between where g m,k (t),υ m,k , τ m,k and ν m,k are t time slots S m To U k The complex gain, Doppler shift, propagation delay and array response vector of the link, f c is the carrier frequency; g m,k (t) can be modeled as where δ m,k (t) represents the time slot S m and U k The Rice fading factor between c and d m,k (t) respectively represent S m The transmit antenna gain, U k The receiving antenna gain, the speed of light and the time slot S m with U k The distance between m,k Can be modeled as in, and Represent the array response vectors in the x-axis and y-axis directions respectively, represents the Kronecker product.

3. The method for allocating resources of a multi-beam satellite communication system according to claim 1, characterized in that: In step S2, the user clustering strategy is determined, specifically including: setting R max represents the coverage radius of the satellite beam. The mean shift algorithm is used to design the user clustering strategy. The specific steps are as follows: (1) Initialization: Let Φ represent the set of unclustered users, that is, Φ = {U k ,1≤k≤K},C i represents the user set of the i-th cluster, 1≤i<K, let i=1; (2) Select the initial center point: Randomly select an unclustered user and use its position as the initial center point. For example, select U k ∈Φ, q k As C i The initial center point of (3) Determine the initial cluster members: Calculate the number of unclustered users and If the distance Then U k′ Add to C i , that is, C i =C i ∪{U k′ }, let L i Represents C i Number of users in (4) Update the cluster center: Let M i Represents C i The mean shift vector of can be modeled as: in is the Gaussian kernel function, σ is the Gaussian kernel function parameter; based on Update C i The center point of (5) Update cluster members: Repeat steps (3) and (4) until M i =0; Update C i , delete C i Zhongyu The distance is greater than R max , that is, if C i =C i \{U k′ }; Update Φ, delete the clustered users, if U k′ ∈C i , then Φ=Φ\{U k′ }; (6) Determine whether the initial clustering algorithm is terminated: Determine whether there are unclustered U k′ ,like Then let i=i+1 and return to step (2). Otherwise, execute step (7) and record the total number of user clusters as I. (7) Calculate the clustering strategy evaluation function: Determine the clustering performance evaluation function based on the intra-cluster and inter-cluster distances, let S i Represents C i The dispersion is defined as C i The average distance from all users to the center point can be modeled as make Represents C i With C j The distance between them can be modeled as Let F i,j C i With C j The similarity measure between can be modeled as Let F i Represents C i The evaluation function can be modeled as Let F represent the clustering strategy evaluation function, which can be modeled as (8) Determine the update conditions of clustering strategy: Let F th is the predefined evaluation function threshold, if F≤F th , the algorithm terminates, the current clustering result is the final result, and the user clustering strategy β is output i,k ∈{0,1}, that is, if U k ∈C i , then β i,k =1, otherwise β i,k =0; if F>F th , let σ=ασ, 0<α<1, return to step (1).

4. The method for allocating resources of a multi-beam satellite communication system according to claim 1, characterized in that: In step S3, the modeling of beam illumination variables specifically includes: modeling α m,n,i (t)∈{0,1} is the time slot S m The beam n and C i The illumination variable between t and S m The beam n illuminates C i , then α m,n,i (t)=1, otherwise α m,n,i (t) = 0; Modeling power allocation variables, including: Modeling time slot S m The beam power allocation vector is P m (t) = [p m,1 (t),p m,2 (t),…p m,N (t)] T , where p m,n (t) represents the time slot S m The transmit power of beam n.

5. The method for allocating resources of a multi-beam satellite communication system according to claim 1, characterized in that: In step S4, the modeling of the user rate model specifically includes: dividing the message sent by the satellite to the user into a public part and a private part, uniformly encoding the public part of all user messages in each cluster into a public stream, and independently encoding the private part of each user message into a user private stream; the satellite respectively designs precoding for the public stream and multiple private streams of the users in the cluster, and multiplexes and transmits them; after the users in the cluster receive the message from the satellite, they decode the public stream and apply the continuous interference cancellation SIC technology, after removing the public stream from the received signal, each user then respectively decodes its corresponding private stream; Order i,c (t) represents the time slot C i The public stream of all user messages in s i,k (t) represents the time slot C i User U k The private stream of Indicates time slot S m With C i The beamforming matrix during communication, where w m,i,c (t) and w m,i,k (t) are public flow and U k Precoding vector of private stream; t time slot S m Send to C i The signal can be modeled as tTime slot C i Inside U k Received from S m The signal can be modeled as in represents additive white Gaussian noise; Let γ m,i,k,c (t) and γ m,i,k (t) represents the time slot C i Inside U k Received from S m The signal-to-interference-noise ratios of the public and private flows in the signal are modeled as and Let R m,i,c (t) represents the time slot S m to C i The public flow transmission rate can be modeled as Let R m,i,k (t) represents the time slot S m to C i Inside U k The private flow transmission rate can be modeled as R m,i,k (t) = Blog2(1+γ m,i,k (t)); Indicates time slot S m to C i Inside U k The sum rate can be modeled as 6. A multi-beam satellite communication system resource allocation method according to claim 1, characterized in that: In step S5, the user service model is modeled, specifically including: assuming that the user service flow arrives randomly and dynamically, and the service volume arriving at each user in each time slot follows a Poisson distribution; let A k (t) represents U k The traffic volume arriving at time slot t is expected to be E[A k (t)] = λ k τ, where λ k Indicates U k Average arrival rate of business flows; The satellite queue model is built, which includes: each satellite is equipped with a user data cache server, which can store the data streams randomly arriving from each user in the corresponding queue. max Indicates the maximum capacity of the satellite buffer; Q m,k (t) represents the end of time slot t m Cache U k The queue length of the data flow can be modeled as:

7. The method for allocating resources of a multi-beam satellite communication system according to claim 1, characterized in that: In step S6, the modeling of the amount of data to be transmitted by the user cluster specifically includes: m,i (t) represents the time slot S m to C i The amount of data to be transmitted can be modeled as Let r(t) represent the system cost function of time slot t, which can be modeled as 8. The method for allocating resources of a multi-beam satellite communication system according to claim 1, characterized in that: In step S7, the modeling system resource allocation constraints specifically include: (1) Beam illumination constraints In any time slot, a single beam of each satellite can only serve one user cluster at most, so: In any time slot, each user cluster is served by at most one beam of one satellite, so: (2) Beam Transmit Power Constraint The total transmission power of multiple beams of each satellite in any time slot cannot exceed the total transmission power of the satellite, so: There is a maximum transmit power limit for a single beam in any time slot, so: The power that can be allocated to the data stream of each user in the cluster cannot exceed the transmit power of the beam, so: In order to successfully implement SIC while decoding the public stream at the receiving end, the public stream signal strength should be greater than the private stream and noise, so: where θ th is the minimum power difference between the public flow and the private flow and the noise.

9. A multi-beam satellite communication system resource allocation method according to claim 1, characterized in that: In step S8, the modeling system state, action and reward specifically includes: defining the global state space of time slot t as s t ={s t,m |1≤m≤M}, where s t,m = {Q m,k (t),h m,k (t)1≤k≤K} represents the time slot S m The state of t time slot joint action space is defined as a t ={α m,n,i (t),p m,n (t),W m,i (t)1≤m≤M,1≤n≤N,1≤i≤I}, including beam illumination, beam power allocation and intra-cluster beamforming strategy; let r(t) be the system reward function for time slot t.

10. A multi-beam satellite communication system resource allocation method according to claim 1, characterized in that: In step S9, the construction and training of the phased policy gradient PPG network specifically includes: the PPG network includes a policy network and a value network, and θ and φ represent the parameters of the policy network and the value network respectively; the training process of the PPG network is alternately performed by a policy training phase and a knowledge distillation-assisted training phase, wherein the policy training phase uses a proximal policy optimization PPO algorithm to train the agent, and the knowledge distillation-assisted training phase can send useful information of the value network to the policy network; initialize the policy network and value network parameters and the experience replay cache pool Given an initial state s t , the agent follows the output of the policy network π(a t |s t θ t ) Execute action a t , and get reward r after interacting with the environment t , the system transfers to the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 )Deposit Each time the network parameters are updated, Extract training samples from the network and update the parameters of the policy network and the value network alternately; Define the loss function of the policy network as in Indicates that the current strategy is different from the previous strategy in state s t Take action a t The probability ratio, π(a t |s t θ t ) indicates that under the current strategy, in state s t Take action a t The probability of π(a t |s t θ old ) indicates that under the previous strategy, in state s t Take action a t The probability of A(t) = δ t +(γλ)δ t+1 +…+(γλ) X-t+1 δ X-1 represents the advantage function of time slot t, δ t =r t +γV φ (s t+1 )-V φ (s t ) represents the time difference error, V φ (s t ) indicates state s t The value function of , γ∈(0,1) is the discount factor, λ is a constant, X is the trajectory length; ε∈(0,1) is the truncation coefficient, clip(μ t (θ t ),1-ε,1+ε) is the truncation function, which means that μ t (θ t ) is limited to [1-ε, 1+ε] to ensure that the policy update amplitude is not too large; the stochastic gradient ascent method is used to update the parameters θ of the policy network, and its update formula can be modeled as follows: in represents L(θ t ) is the gradient of the parameter θ, and η is the learning rate; The loss function of the value network is defined as Where V t tar is the target value function of time slot t; the parameter φ of the value network is updated using the stochastic gradient descent method, and its update formula can be modeled as In the knowledge distillation assisted training phase, the loss function of the strategy network is defined as Where L(θ t ) aux For the auxiliary loss part, by using the auxiliary value function V of the policy network θ (s t ) to learn useful information of the value network, which can be modeled as L(θ t ) joint The second term is the behavior cloning loss part, β clone is the cloning parameter, π(·s t θ old ) is the strategy before the auxiliary phase begins, π(·s t θ t ) is the current strategy, KL(π(·s t θ old ),π(·s t θ t )) is used to calculate π(·s t θ old ) and π(·s t θ t ) between them; the stochastic gradient descent method is used to update the parameters θ of the policy network and φ of the value network respectively, and the update formulas can be modeled as and 11. A multi-beam satellite communication system resource allocation method according to claim 1, characterized in that: In step S10, the beam illumination, beam power allocation and intra-cluster beamforming strategy are determined by using the trained PPG network, specifically including: optimizing and determining the resource allocation strategy with the goal of maximizing the system cumulative reward under the condition of satisfying the beam illumination and beam transmission power constraints, that is: in and They are optimal beam illumination, beam power allocation and intra-cluster beamforming strategy respectively.

Images