Multi-beam satellite communication system resource allocation method based on multi-agent A3C algorithm

Abstract

The present application relates to a kind of multi-beam satellite communication system resource allocation method based on multi-agent A3C algorithm, belong to multi-beam satellite communication technical field.The method includes the following steps: S1: modeling satellite communication system model and ground service model;S2: modeling beam association variable and power allocation variable;S3: modeling satellite channel;S4: determine satellite load balancing scheme S5: modeling system throughput;S6: modeling system utility function;S7: modeling system resource allocation constraint condition;S8: modeling system state, action and reward;S9: construct and train multi-agent A3C network, determine system beam hopping scheme and resource allocation strategy based on multi-agent A3C algorithm.The present application maximizes system average return by optimizing beam association and power allocation strategy jointly, to improve system comprehensive performance.

Description

Technical Field

[0001] This invention belongs to the field of multi-beam satellite communication technology and relates to a resource allocation method for multi-beam satellite communication systems based on the multi-agent A3C (Asynchronous Advantage Actor-Critic) algorithm. Background Technology

[0002] In recent years, satellite communication systems, as a supplement to terrestrial communication network infrastructure, have been widely used due to their advantages such as strong global coverage, long communication distance, resistance to major natural disasters, and the ability to provide fixed and mobile communication services. To improve the utilization of spectrum resources and transmission performance of satellite systems, multi-beam satellites can be employed, using multiple high-gain narrow beams to enhance user coverage and achieve spectrum reuse gain. However, considering the diverse ground requirements of multi-beam satellite communication systems and the dynamic nature and spatiotemporal unevenness of satellite communication system topologies, designing efficient beam association and power allocation strategies to ensure quality of service and improve system performance in ground areas has become an important research topic.

[0003] Existing research has considered the resource allocation problem of multi-beam satellite communication systems, but few studies have addressed the situation where multiple multi-beam low-Earth orbit satellites cannot simultaneously serve all ground cells and have uneven service demands, and have researched resource allocation strategies based on system average performance optimization. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a resource allocation method for a multi-beam satellite communication system based on the multi-agent A3C algorithm. For a multi-beam satellite communication system containing N multi-beam low-orbit satellites and M ground cells, the average system return is modeled as the optimization objective, and beam association and power allocation strategies are implemented to improve the overall system performance.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A resource allocation method for a multi-beam satellite communication system based on the multi-agent A3C algorithm, the method comprising the following steps:

[0007] S1: Establish a satellite communication system model and a ground service model. The established satellite communication system model includes several multi-beam low-orbit satellites, and the established ground service model includes several ground cells.

[0008] S2: Modeling beam correlation variable scheme and power allocation variable scheme;

[0009] S3: Modeling satellite channels;

[0010] S4: Determine the satellite load balancing scheme;

[0011] S5: Modeling system throughput;

[0012] S6: Modeling the system utility function;

[0013] S7: Constraints on resource allocation in the modeling system;

[0014] S8: Model system states, actions, and rewards;

[0015] S9: Construct and train a multi-agent A3C network, and determine the system beam hopping scheme and resource allocation strategy based on the multi-agent A3C algorithm.

[0016] Furthermore, S1, establishing the satellite communication system model and the terrestrial service model, specifically includes:

[0017] Let N represent the number of low-Earth orbit satellites, and S n Let M represent the nth satellite, assuming each satellite simultaneously generates K beams, and let M represent the number of ground cells, C m Let P represent the m-th cell; tot P represents the total power of a single satellite. max Let B represent the maximum power of a single satellite beam, and let B represent the total bandwidth of the satellite system. Assume that all beams use the satellite system bandwidth with full frequency reuse. Let the total system time be T. tot It consists of I hopping beam periods, with one period lasting T. Then T tot =IT, the ground area served by the satellite system remains unchanged within each cycle, and T is divided into time slots of equal length; assuming a single satellite can cover M s A cell, the same cell can be covered by multiple satellites, let Θ i,n Let |Θ| represent the set of cells covered by the nth satellite in the i-th beam hopping period, then we have |Θ| i,n |=M s , 1≤i≤I, 1≤n≤N;

[0018] Assuming each cell has a virtual aggregation device to aggregate the cell's communication traffic needs, the service needs of the ground cells are divided into L levels, let r t,l This represents the communication request rate corresponding to a ground service demand level of l in time slot t, modeled as r. t,l Let be a normally distributed random variable with a mean of μ. l The variance is Let b i,m,l ∈{0,1} represents the i-th hopping beam period cell C m The business demand identifier, if cell C during this period m If the business requirement is level l, then b i,m,l =1, otherwise, b i,m,l=0, each cell selects only the primary service requirement in each cycle, that is make Indicates cell C in time slot t. m The communication request rate, then

[0019]

[0020] Furthermore, in S2, the modeling beam correlation variable scheme and power allocation variable scheme specifically include:

[0021] Modeling t-timeslot satellite S n The beam-hopping correlation variable scheme is as follows:

[0022] A t,n =[α t,n,1 ,α t,n,2 ,...,α t,n,k ,...,α t,n,K (1)

[0023] α t,n,k =[α t,n,k,1 ,α t,n,k,2 ,...,α t,n,k,m ,...,α t,n,k,M ] T (2)

[0024] Where, α t,n,k Indicates the satellite S in time slot t n The illumination vector of beam k, α t,n,k,m For time slot satellite S n Beam k and cell C m The related variable, α t,n,k,m ∈{0,1}, if satellite S in time slot t n The beam k illuminates the cell C m , then α t,n,k,m =1, otherwise α t,n,k,m =0, 0<t≤T tot ;

[0025] Modeling t-timeslot satellite S n The power allocation variable scheme is as follows:

[0026] P t,n =[p t,n,1 ,p t,n,2 ,...,p t,n,k ,...,p t,n,K (3)

[0027] p t,n,k =[p t,n,k,1 ,p t,n,k,2 ,...,p t,n,k,m,...,p t,n,k,M ] T (4)

[0028] Where, p t,n,k Indicates the time slot S of satellite t n The power allocation vector of beam k, p t,n,k,m Indicates the satellite S in time slot t n The k-th beam illuminates cell C m The corresponding transmission power at time α t,n,k,m =0, then p t,n,k,m =0.

[0029] Furthermore, in S3, modeling the satellite channel specifically includes:

[0030] Let h t,n,k,m Indicates the satellite S in time slot t n With Community C m The channel gain of beam k occupied by the link between them, h t,n,k,m The model is represented as:

[0031]

[0032] in, L represents the transmit antenna gain. t,n,m For time slot satellite S n With Community C m Path loss of direct links, Indicates cell C in time slot t. m The receiving antenna gain is given by A, where A represents the total transmission attenuation.

[0033] The model is represented as:

[0034]

[0035] in, The maximum gain of the transmitting antenna is given by η, where η represents the antenna efficiency and θ is the maximum gain of the transmitting antenna. 3dB The angle corresponding to a 3dB beamwidth, J1 and J3 are the first and third order Bessel functions, respectively, u t,n,k,m =2.07123sin(θ) t,n,k,m ) / sin(θ 3dB ), where θ t,n,k,m Indicates the time slot S of satellite t n The center point of beam k and cell C m The off-axis angle of the receiving antenna of the focusing device;

[0036] L t,n,m The model is represented as:

[0037]

[0038] Where c is the speed of light, d t,n,m For time slot satellite S n To Community C m The distance to the aggregation device, where f is the carrier frequency;

[0039] Modeling A is represented as:

[0040] A = A r +A c (8)

[0041]

[0042]

[0043] Among them, A r = represents rain attenuation, e represents the probability of rainfall, and the values ​​of variables a1 and b1 depend on the antenna's frequency and polarization; A c For cloud and fog attenuation, θ t,n,m For community C m Convergence device to satellite S n The elevation angle is given by a2, where a2 is the columnar water content and b2 is the attenuation coefficient.

[0044] Furthermore, in step S4, the satellite load balancing scheme is determined, specifically including:

[0045] Let Φ i,n Represents the i-th hop beam period satellite S n The collection of service communities, D i,n Represents the i-th hop beam period satellite S n The load, Represents the i-th hopping beam cycle cell C m To meet the average traffic demand, inter-satellite load balancing is performed by selecting access satellites for cells with overlapping coverage by multiple satellites.

[0046] For the i-th hopping beam cycle, the inter-satellite load balancing scheme is determined, which specifically includes the following steps:

[0047] S41: Initialize the set of satellite-serving cells as the set of cells covered only by each satellite within this period, represented as:

[0048]

[0049] Then, S n The initial load is represented as:

[0050]

[0051] S42: Order Ψ iLet be the set of cells covered by multiple satellites within the i-th beam hopping period, denoted as:

[0052]

[0053] For set Ψ i The community is divided into three groups according to the traffic demand of each community. i,m Sort in descending order, let 1≤j0≤M′,M′=|Ψ i |, j = 1;

[0054] S43: In the coverage area Of all the satellites, select the one with the lowest throughput load. Then select satellite As a community The service satellite, namely

[0055] S44: Update satellites The load, i.e.: Let j = j + 1;

[0056] S45: If j > M′, the algorithm ends; otherwise, return to S43 to select again.

[0057] Furthermore, in S5, modeling system throughput specifically includes:

[0058] Let R t,m Indicates that the t-slot system is cell C. m The provided throughput is modeled as follows:

[0059]

[0060] Where, γ t,n,k,m For time slot satellite S n Use beam k to transmit data to cell C m The corresponding signal-to-interference-plus-noise ratio at the receiving end;

[0061] γ t,n,k,m The model is represented as:

[0062]

[0063] Where, k B T is the Boltzmann constant. N For noise temperature, I t,n,k,m 、I′ t,n,k,m These represent interference within a satellite and interference between satellites, respectively.

[0064] I t,n,k,m The model is represented as:

[0065]

[0066] Where, α t,n,k′,m ∈{0,1} is the cell beam illumination variable, if the satellite S in time slot t n The k′-th beam illuminates cell C m , then α t,n,k′,m =1, otherwise α t,n,k′,m =0; p t,n,k′,m′ Indicates the satellite S in time slot t n The k′-th beam illuminates cell C m′ The corresponding transmit power at time h; t,n,k′,m Indicates the satellite S in time slot t n With Community C m The link between them occupies the channel gain of beam k′;

[0067] I' t,n,k,m The model is represented as:

[0068]

[0069] Where, α t,n,k′,m ∈{0,1} is the cell beam illumination variable, if the satellite S in time slot t n The k′-th beam illuminates cell C m , then α t,n,k′,m =1, otherwise α t,n,k′,m =0; p t,n,k′,m′ Indicates the satellite S in time slot t n The k′-th beam illuminates cell C m′ The corresponding transmit power at time h; t,n,k′,m Indicates the satellite S in time slot t n With Community C m The link between them occupies the channel gain of beam k′.

[0070] Furthermore, in S6, the utility function is modeled, specifically including:

[0071] Let the system utility function for time slot t be:

[0072]

[0073] Where λ1 and λ2 are weighting coefficients, δ t The fairness index for the t-slot system;

[0074] δ t The model is as follows:

[0075]

[0076] in, Indicates cell C during the i-th beam hopping periodm The number of time slots that are irradiated.

[0077] Furthermore, in S7, the modeling system resource allocation constraints specifically include: beam illumination constraints and beam transmission power constraints;

[0078] The beam illumination constraints for resource allocation in the modeling system are as follows:

[0079] If at any given time, the beams of any satellite cannot illuminate cells outside their service range, then:

[0080] α t,n,k,m =0, if

[0081] If a single satellite beam in each time slot can only serve one cell at most, then:

[0082]

[0083] If each cell in each time slot is served by at most one beam from one satellite, then:

[0084]

[0085] The beam transmit power constraint for resource allocation in the modeling system is as follows:

[0086] If there is a maximum transmit power limit for a single beam, then:

[0087]

[0088] If the sum of the transmission power of all beams of the satellite cannot exceed the total transmission power of the satellite, then:

[0089]

[0090] Furthermore, in S8, the system state, actions, and rewards are modeled, specifically including:

[0091] Define the global state space for time slot t as follows:

[0092] s t ={s t,1 ,s t,2 ,...,s t,N} (20)

[0093] in, Indicates the satellite S in time slot t n The state defines the joint action space a for time slot t. t ={a t,1 ,a t,2 ,…a t,N}, where at,n ={A t,n ,P t,n} represents satellite S in time slot t. n The actions, including beam correlation and power allocation strategies, model the system's reward function r in time slot t. t =U t .

[0094] Furthermore, in S9, a multi-agent A3C network is constructed and trained, and the system beam-hopping scheme and resource allocation strategy are determined based on the multi-agent A3C algorithm, specifically including:

[0095] Consider N low-Earth orbit satellites as N agents, each agent containing a global actor-critic network and multiple parallel actor-critic networks; for agent n, initialize its global actor network and global critic network parameters as θ. n ω n Initialize its multi-threaded actor network π n Commentator Network V n The parameters are θ′ n ,ω′ n Synchronization parameter θ′ n =θ n , ω′ n =ω n Initialize system state s t The agent n, based on its state s t,n Select action a t,n =π n (a t,n |s t,n ;θ′ n ), to obtain value V n (s t,n ;ω′ n ) and global reward r t and transition to the next state s t+1,n Define an advantage function to evaluate the advantage of the current state-action pair. The advantage function is expressed as:

[0096]

[0097] Where, γ i With γ k Let γ be the power of i and the power of k, respectively; update the actor network using the loss function, which is expressed as:

[0098] L π (θ′ n ) = logπ(a t,n |s t,n ;θ′ n )A(s t,n ,at,n )+cH(π(s t,n ;θ′ n )) (twenty two)

[0099] Wherein, H(π(s) t,n ;θ′ n )) is the policy entropy function to avoid premature convergence, and c is the entropy coefficient used to control the regularization strength of entropy. The cumulative gradient is obtained based on the loss function and is expressed as:

[0100]

[0101] in, This indicates the calculation of x with respect to parameter θ′. n The derivative; using the loss function L V (ω′ n )=(A(s t,n ,a t,n )) 2 Update the critic network and obtain the cumulative gradient based on the loss function, expressed as:

[0102]

[0103] in, This indicates the calculation of x with respect to ω′. n The partial derivative of ; update the global parameters, expressed as:

[0104]

[0105] Where, Δθ n The cumulative gradient of the loss function is η, where η is the learning rate and ε is a small positive number. g = αg + (1-α)Δθ 2 It estimates the gradient, where α is the momentum coefficient;

[0106] Under the constraints of transmitting power and beam correlation of each satellite beam, and with the goal of maximizing the system's average return, the resource allocation strategy is optimized and determined as follows:

[0107]

[0108] in and These are the optimal beam association strategy and the power allocation strategy.

[0109] The beneficial effects of this invention are as follows:

[0110] This invention can effectively ensure that, under conditions of uneven spatiotemporal distribution of communication services, the average return of the system is maximized based on beam association and power allocation strategies, thereby improving the overall performance of the system.

[0111] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0112] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0113] Figure 1 This is a schematic diagram of a multi-beam satellite communication system scenario.

[0114] Figure 2 This is a flowchart illustrating the resource allocation method for a multi-beam satellite communication system based on the multi-agent A3C algorithm. Detailed Implementation

[0115] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0116] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0117] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0118] Please see Figures 1-2 This is a resource allocation method for multi-beam satellite communication systems based on the multi-agent A3C algorithm.

[0119] like Figure 1 As shown, the communication system comprises N multi-beam low-orbit satellites and M ground cells. By jointly designing the optimal beam association and power allocation strategy, the average return of the system can be maximized.

[0120] Figure 2 This is a flowchart illustrating the resource allocation method for a multi-beam satellite communication system based on the multi-agent A3C algorithm, as shown below. Figure 2 As shown, the method specifically includes the following steps:

[0121] Step 1: Modeling the satellite communication system and terrestrial services

[0122] The system comprises multiple multi-beam low-Earth orbit satellites and multiple ground cells, where N represents the number of low-Earth orbit satellites and S... n Let M represent the nth satellite, and assume that each satellite can generate K beams simultaneously. Let M represent the number of ground cells, and C... m Let P represent the m-th cell; tot P represents the total power of a single satellite. max Let B represent the maximum power of a single satellite beam, and let B represent the total bandwidth of the satellite system. Assume that all beams use the satellite system bandwidth with full frequency reuse. Let the total system time be T. tot It consists of I hopping beam periods, with one period having a time of T, then T tot =IT, the ground area served by the satellite system remains unchanged within each cycle, and T is divided into time slots of equal length; assuming a single satellite can cover M s A cell, the same cell can be covered by multiple satellites, let Θ i,n Let |Θ| represent the set of cells covered by the nth satellite in the i-th beam hopping period, then we have |Θ| i,n |=M s , 1≤i≤I, 1≤n≤N;

[0123] Assuming each cell has a virtual aggregation device to aggregate the cell's communication traffic needs, the service needs of the ground cells can be divided into L levels, let r t,l This represents the communication request rate corresponding to a ground service demand level of l in time slot t, modeled as r. t,l Let be a normally distributed random variable with a mean of μ. l The variance is Let b i,m,l ∈{0,1} represents the i-th hopping beam period cell C m The business demand identifier, if cell C during this period m If the business requirement is level l, then b i,m,l=1, otherwise, b i,m,l =0, each cell selects only the primary service requirement in each cycle, that is make Indicates cell C in time slot t. m The communication request rate, then

[0124] Step 2: Modeling beam correlation variables and power allocation variables

[0125] Modeling t-timeslot satellite S n The hopping beam scheme is: A t,n =[α t,n,1 ,α t,n,2 ,...,α t,n,k ,...,α t,n,K ], where α t,n,k Indicates the time slot S of satellite t n The illumination vector of beam k, α t,n,k =[α t,n,k,1 ,α t,n,k,2 ,...,α t,n,k,m ,...,α t,n,k,M ] T α t,n,k,m For time slot satellite S n beam k and cell C m The related variable, α t,n,k,m ∈{0,1}, if satellite S in time slot t n The beam k illuminates the cell C m , then α t,n,k,m =1, otherwise α t,n,k,m =0, 0<t≤T tot ;

[0126] Modeling t-timeslot satellite S n The power allocation scheme is P t,n =[p t,n,1 ,p t,n,2 ,...,p t,n,k ,...,p t,n,K ], where p t,n,k Indicates the time slot S of satellite t n The power allocation vector of beam k, p t,n,k =[p t,n,k,1 ,p t,n,k,2 ,...,p t,n,k,m ,...,p t,n,k,M ] T , where p t,n,k,m Indicates the time slot S of satellite t n The k-th beam illuminates cell C m The corresponding transmission power at time α t,n,k,m =0, then pt,n,k,m =0.

[0127] Step 3: Modeling Satellite Channels

[0128] Let h t,n,k,m Indicates the time slot S of satellite t n With Community C m The channel gain of beam k occupied by the link between them, h t,n,k,m Modeling as in The transmit antenna gain can be modeled as... ,in The maximum gain of the transmitting antenna is given by η, where η represents the antenna efficiency and θ is the maximum gain of the transmitting antenna. 3dB The angle corresponding to a 3dB beamwidth, J1 and J3 are the first and third order Bessel functions, respectively, u t,n,k,m =2.07123sin(θ) t,n,k,m ) / sin(θ 3dB ), where θ t,n,k,m Indicates the time slot S of satellite t n The center point of beam k and cell C m The off-axis angle of the receiving antenna of the focusing device. Indicates cell C in time slot t. m The receiving antenna gain;

[0129] L t,n,m For time slot satellite S n With Community C m Path loss of direct links, Where c is the speed of light, d t,n,m For time slot satellite S n To Community C m The distance to the aggregation device, f is the carrier frequency; A represents the total transmission attenuation, A = A r +A c ,in For rain attenuation, e represents the probability of rainfall, and the values ​​of variables a1 and b1 depend on the antenna's frequency and polarization; A c For cloud and fog attenuation, the model is as follows Where θ t,n,m For community C m Convergence device to satellite S n The elevation angle is given by a2, where a2 is the columnar water content and b2 is the attenuation coefficient.

[0130] Step 4: Determine the satellite load balancing scheme

[0131] Let Φ i,n Represents the i-th hop beam period satellite S n The collection of service communities, D i,n Represents the i-th hop beam period satellite Sn The load, Represents the i-th hopping beam cycle cell C m To determine the average traffic demand, inter-satellite load balancing is performed by selecting access satellites for cells with overlapping coverage from multiple satellites. For the i-th beam hopping cycle, the inter-satellite load balancing scheme is as follows:

[0132] a) Initialize the set of satellite serving cells as the set of cells covered only by each satellite within that period, i.e. Then S n Initial load is

[0133] b) Let Ψ i Let i be the set of cells covered by multiple satellites during the i-th beam hopping period. For set Ψ i The community is divided into three groups according to the traffic demand of each community. i,m Sort in descending order, let 1≤j0≤M′,M′=|Ψ i |, j = 1;

[0134] c) In the coverage area Of all the satellites, select the one with the lowest throughput load. Then select satellite As a community The service satellite, namely

[0135] d) Update satellites The load, i.e.: Let j = j + 1;

[0136] e) If j > M′, the algorithm ends; otherwise, proceed to step c).

[0137] Step 5: Model system throughput

[0138] Let R t,m This indicates that the t-slot system can be a cell C. m The provided throughput is modeled as Where γ t,n,k,m For time slot satellite S n Use beam k to transmit data to cell C m The corresponding receiver signal-to-interference-plus-noise ratio is modeled as follows: Where k B T is the Boltzmann constant. N For noise temperature, I t,n,k,m 、I′ t,n,k,m Interference within and between satellites is represented respectively, and modeled as follows: Where, α t,n,k′,m∈{0,1} is the cell beam illumination variable, if the satellite S in time slot t n The k′-th beam illuminates cell C m , then α t,n,k′,m =1, otherwise α t,n,k′,m =0; p t,n,k′,m′ Indicates the satellite S in time slot t n The k′-th beam illuminates cell C m′ The corresponding transmit power at time h; t,n,k′,m Indicates the satellite S in time slot t n With Community C m The link between them occupies the channel gain of beam k′;

[0139] Where, α t,n′,k′,m′ ∈{0,1} is the cell beam illumination variable, if the satellite S in time slot t n′ The k′-th beam illuminates cell C m′ , then α t,n′,k′,m′ =1, otherwise α t,n′,k′,m′ =0; p t,n′,k′,m′ Indicates the satellite S in time slot t n′ The k′-th beam illuminates cell C m′ The corresponding transmit power at time h; t,n′,k′,m Indicates the satellite S in time slot t n′ With Community C m The link between them occupies the channel gain of beam k′.

[0140] Step Six: Model the System Utility Function

[0141] Let the system utility function of time slot t be Where λ1 and λ2 are weighting coefficients, δ t The fairness index of the t-slot system is modeled as follows: in Indicates cell C during the i-th beam hopping period m The number of time slots that are irradiated.

[0142] Step 7: Modeling System Resource Allocation Constraints

[0143] 1) Beam illumination constraint

[0144] If at any given time, the beams of any satellite cannot illuminate cells outside their service range, then:

[0145] α t,n,k,m =0, if

[0146] If a single satellite beam in each time slot can serve at most one cell, then:

[0147]

[0148] If each cell in each time slot is served by at most one beam from one satellite, then:

[0149]

[0150] 2) Beam transmit power constraint

[0151] If there is a maximum transmit power limit for a single beam, then:

[0152]

[0153] If the sum of the transmission power of all beams of the satellite cannot exceed the total transmission power of the satellite, then:

[0154]

[0155] Step 8: Model the system state, actions, and rewards

[0156] Define the global state space s for time slot t. t ={s t,1 ,s t,2 ,...,s t,N},in Indicates the satellite S in time slot t n The state defines the joint action space a for time slot t. t ={a t,1 ,a t,2 ,…a t,N}, where a t,n ={A t,n ,P t,n} represents satellite S in time slot t. n The actions, including beam correlation and power allocation strategies, model the system's reward function r in time slot t. t =U t .

[0157] Step 9: Construct and train a multi-agent A3C network, and determine the system resource allocation strategy based on the multi-agent A3C algorithm.

[0158] Consider N low-Earth orbit satellites as N agents, each agent containing a global actor-critic network and multiple parallel actor-critic networks; for agent n, initialize its global actor network and global critic network parameters as θ. n ω n Initialize its multi-threaded actor network π n Commentator Network V n The parameters are θ′ n ,ω′ n Synchronization parameter θ′n =θ n , ω′ n =ω n Initialize system state s t The agent n, based on its state s t,n Select action a t,n =π n (a t,n |s t,n ;θ′ n ), to obtain value V n (s t,n ;ω′ n ) and global reward r t and transition to the next state s t+1,n Define the advantage function Used to evaluate the advantage of the current state-action pair, where γ i With γ k Let i and k represent the state γ raised to the power of γ, respectively.

[0159] Using loss function L π (θ′ n ) = logπ(a t,n |s t,n ;θ′ n )A(s t,n ,a t,n )+cH(π(s t,n ;θ′ n Update the actor network, where H(π(s)) t,n ;θ′ n )) is the policy entropy function to avoid premature convergence, and c is the entropy coefficient used to control the strength of entropy regularization. The cumulative gradient is obtained based on the loss function. in, This indicates the calculation of x with respect to parameter θ′. n The derivative; using the loss function L V (ω′ n )=(A(s t,n ,a t,n )) 2 Update the commentator network and obtain the cumulative gradient based on the loss function: in, This indicates the calculation of x with respect to ω′. n The partial derivatives are then used to update the global parameters, specifically: Where Δθ n It is the cumulative gradient of the loss function, η is the learning rate, ε is a small positive number, and g = αg + (1-α)Δθ 2 It estimates the gradient, where α is the momentum coefficient.

[0160] Under the constraints of satellite beam transmit power and beam correlation, and with the objective of maximizing the system's average return, the resource allocation strategy is optimized and determined as follows:

[0161]

[0162] in, and These are the optimal beam association strategy and the power allocation strategy.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A resource allocation method for a multi-beam satellite communication system based on the multi-agent A3C algorithm, characterized in that: The method includes the following steps: S1: Establish a satellite communication system model and a ground service model. The established satellite communication system model includes several multi-beam low-orbit satellites, and the established ground service model includes several ground cells. S2: Modeling beam correlation variable scheme and power allocation variable scheme; S3: Modeling satellite channels; S4: Determine the satellite load balancing scheme; S5: Modeling system throughput; S6: Modeling the system utility function; S7: Constraints on resource allocation in the modeling system; S8: Model system states, actions, and rewards; S9: Construct and train a multi-agent A3C network, and determine the system beam hopping scheme and resource allocation strategy based on the multi-agent A3C algorithm; S1, establishing the satellite communication system model and the ground service model, specifically includes: make Indicates the number of low-orbit satellites. Indicates the first There are 10 satellites, assuming each satellite generates simultaneously. A beam, making Indicates the number of ground cells. Indicates the first One community; order This represents the total power of a single satellite. The maximum power of a single satellite beam. Let represent the total bandwidth of the satellite system, assuming all beams utilize the satellite system bandwidth with full frequency reuse; let the total system time be . ,Depend on It consists of several hop beam cycles, with one cycle time being [time value missing]. ,but The ground area served by the satellite system remains unchanged during each cycle. Divided into time slots of equal length; assuming a single satellite can cover In some residential areas, the same area is covered by multiple satellites, making... Indicates the first The first hop beam cycle A collection of satellite-covered cells, then , , ; Assuming each cell has a virtual aggregation device to aggregate the cell's communication traffic needs, the service needs of the ground cells are divided into... Level, Order express t The demand for time-slot ground services is Modeling the communication request rate corresponding to the level Let be a normally distributed random variable with a mean of . The variance is ,make Indicates the first One beam-hopping periodic cell The business demand identifier, if the cell during this period Business requirements are Level, then ,on the contrary, Each community selects only the primary business requirement in each cycle, i.e. , ;make express t Time slot cell The communication request rate, then , ; In S2, the modeling beam correlation variable scheme and power allocation variable scheme specifically include: Modeling t Time slot satellite The beam-hopping correlation variable scheme is as follows: in, express t Time slot satellite Beam The illumination vector, for t Time slot satellite beam With the community Related variables, ,like t Time slot satellite beam Illuminate the community ,but ,on the contrary , ; Modeling t Time slot satellite The power allocation variable scheme is as follows: in, express t Time slot satellite Beam The power allocation vector, express t Time slot satellite The beam illumination cell The corresponding transmission power at that time, if ,but ; In S3, modeling the satellite channel specifically includes: make express t Time slot satellite With the community The link between them occupies the beam. Channel gain, The model is represented as: in, Indicates the transmit antenna gain. for t Time slot satellite With the community Path loss of direct links, express t Time slot cell The receiving antenna gain, Indicates the total transmission attenuation; The model is represented as: in, This represents the maximum gain of the transmitting antenna. Indicates antenna efficiency. The angle corresponding to a 3dB beamwidth. J 1, J 3 represent first-order and third-order Bessel functions, respectively. ,in express t Time slot satellite beam The center point and the community The off-axis angle of the receiving antenna of the focusing device; The model is represented as: in, At the speed of light, for t Time slot satellite Arrive at the community Distance of the convergence device For carrier frequency; The model is represented as: in, Due to the decline of rain, Represents the probability of rainfall, variable and The value depends on the antenna's frequency and polarization; To reduce cloud cover, For the community Convergence devices to satellite The angle of elevation, This is columnar water content. It is the attenuation coefficient; In step S4, the satellite load balancing scheme is determined, specifically including: make Indicates the first One hop beam cycle satellite A collection of services for the community. Indicates the first One hop beam cycle satellite The load, Indicates the first One beam-hopping periodic cell To meet the average traffic demand, inter-satellite load balancing is performed by selecting access satellites for cells with overlapping coverage by multiple satellites. Regarding the first Determining the inter-satellite load balancing scheme for each hop beam cycle includes the following steps: S41: Initialize the set of satellite-serving cells as the set of cells covered only by each satellite within this period, represented as: but, The initial load is represented as: S42: Order For the first The set of cells covered by multiple satellites within a beam hopping period is represented as: For sets The communities are categorized according to their traffic demand. Sort in descending order, let , , , , ; S43: In the coverage area Of all the satellites, select the one with the lowest throughput load. Then select satellite As a community The service satellite, namely ; S44: Update satellites The load, i.e.: ,make ; S45: If If the result is positive, the algorithm ends; otherwise, return to S43 to select again. In S5, the system throughput is modeled, specifically including: make express t Time slot system for cells The provided throughput is modeled as follows: in, for t Time slot satellite Occupied Beam Transmit data to the cell The corresponding signal-to-interference-plus-noise ratio at the receiving end; The model is represented as: in, Boltzmann's constant, For noise temperature, , These represent interference within a satellite and interference between satellites, respectively. The model is represented as: in, Illuminate the variable for the cell beam, if t Time slot satellite The A beam illuminates the community ,but ,on the contrary ; express t Time slot satellite The beam illumination cell The corresponding transmission power at that time; express t Time slot satellite With the community The link between them occupies the beam. Channel gain; The model is represented as: in, Illuminate the variable for the cell beam, if t Time slot satellite The A beam illuminates the community ,but ,on the contrary ; express t Time slot satellite The beam illumination cell The corresponding transmission power at that time; express t Time slot satellite With the community The link between them occupies the beam. Channel gain; In S6, the utility function is modeled, specifically including: make t The utility function of the time-slot system is: in, and These are the weighting coefficients. for t Fairness index of time-slot system; The model is as follows: in, Indicates the first Cell within a single beam hopping cycle The number of time slots that are irradiated; In S7, the modeling system resource allocation constraints specifically include: beam illumination constraints and beam transmission power constraints; The beam illumination constraints for resource allocation in the modeling system are as follows: If at any given time, the beams of any satellite cannot illuminate cells outside their service range, then: ,like , , If a single satellite beam in each time slot can only serve one cell at most, then: ， ， If each cell in each time slot is served by at most one beam from one satellite, then: ， ； The beam transmit power constraint for resource allocation in the modeling system is as follows: If there is a maximum transmit power limit for a single beam, then: ， ， If the sum of the transmission power of all beams of the satellite cannot exceed the total transmission power of the satellite, then: ， In S8, the system state, actions, and rewards are modeled, specifically including: definition t The time-slot global state space is represented as: in, express t Time slot satellite State, definition t Joint action space of time slots ,in express t Time slot satellite The actions, including beam correlation and power allocation strategies, are modeled by the system. t Time slot reward function ; In step S9, a multi-agent A3C network is constructed and trained. Based on the multi-agent A3C algorithm, the system beam hopping scheme and resource allocation strategy are determined, specifically including: Will N Each low-orbit satellite is considered as N There are *n* agents, each containing a global actor-critic network and multiple parallel actor-critic networks; for each agent... n Initialize its global actor network and global critic network parameters as follows: , Initialize its multi-threaded actor network Critics Network The parameters are respectively , Synchronization parameters , Initialize system state intelligent agent n According to its state Select Action to obtain value and global rewards and transition to the next state. Define an advantage function to evaluate the advantage of the current state-action pair. The advantage function is expressed as: in, and Representing states respectively of power and The power of this; update the actor network using the loss function, which is expressed as: in, The policy entropy function is used to avoid premature convergence. c The entropy coefficient is used to control the regularization strength of the entropy. It is derived from the cumulative gradient obtained from the loss function and is expressed as: in, Indicates calculation x For parameters The derivative; using the loss function Update the critic network and obtain the cumulative gradient based on the loss function, expressed as: in, Indicates calculation x right The partial derivative of ; update the global parameters, expressed as: in, It is the cumulative gradient of the loss function. It's the learning rate. For smaller positive numbers, It estimates the gradient, where The momentum coefficient; Under the constraints of transmitting power and beam correlation of each satellite beam, and with the goal of maximizing the system's average return, the resource allocation strategy is optimized and determined as follows: in and These are the optimal beam association strategy and the power allocation strategy.

Images