A multi-beam dynamic power allocation method for 5G satellite-to-ground links based on reinforcement learning

Abstract

The present invention discloses a 5G satellite-to-ground link multi-beam dynamic power allocation method based on reinforcement learning. The method builds a multi-beam satellite communication system model, obtains the satellite's on-board resource cache information according to the different channel states of the uplink and downlink during the communication process, takes maximizing the user transmission rate as the optimization goal, and simultaneously considers multiple constraints such as power, cache, and rate to give the objective function of the satellite system resource allocation problem. Based on the Q learning method, the multi-beam satellite is designed as an intelligent agent, the action is power selection, and the environment is the channel state. In the process of interaction between the intelligent agent and the channel environment, the Q value is continuously updated to accumulate learning experience, and the optimal solution of the proposed optimization problem is gradually found. The method has good convergence and system performance, is an efficient and reliable dynamic resource allocation method, can adapt to complex and changeable communication environments, and is particularly suitable for application in resource-constrained multi-beam satellite communication systems.

Description

Technical Field

[0001] The present invention relates to a satellite Internet technology, and in particular to a power allocation method in satellite Internet, and more specifically to a 5G satellite-to-ground link multi-beam dynamic power allocation method based on reinforcement learning. Background Art

[0002] With the development of aerospace technology and breakthroughs in key technologies, modern communication networks must not only provide ubiquitous high-quality communication services, but also meet the growing information acquisition needs of all users. As an important part of the 5G non-terrestrial communication network, satellite communication has unique advantages over traditional terrestrial communication networks in terms of high reliability, high scalability, large coverage, global coverage, and simple networking. Therefore, satellite communication systems are widely used. Therefore, satellite communication systems integrated with ground network systems are the best solution to meet users' multi-service needs and high service quality. However, in satellite communication systems, there are problems such as limited on-board computing resources and storage resources and high-speed movement of satellites. Therefore, satellite systems require relatively special resource management methods. At the same time, the faster the communication satellite moves over a coverage area, the more difficult it is to achieve the goal of satellite resource management. More importantly, most satellite communication systems divide the earth's coverage area into multiple cells, which makes satellite resource management more complicated.

[0003] In the integrated space-ground-air system, satellite Internet has attracted more and more attention. However, the problems caused by limited frequency bands in satellite communication systems have become more and more prominent. Nowadays, how to achieve efficient and reasonable resource allocation under the constraints of limited computing and storage resources on satellites has become a key research issue. The emergence of multi-beam antenna (MBA) satellite communication system has become the key to effectively solving the problem of satellite resource allocation. Through multi-beam antennas or phased array antennas, mutual communication between nodes in multiple different areas can be achieved. At the same time, due to the advantages of beam isolation and frequency reuse, multi-beam satellite communication technology has been widely used in integrated space-ground-air network systems. Therefore, the use of multi-beam technology in satellite communication systems can not only significantly improve the spectrum efficiency of the system, but also increase the system capacity of satellite communication. However, limited spectrum resources have always been a key factor restricting the development of satellite communication technology.

[0004] At the same time, reinforcement learning technology, which benefits from animal learning theory and conditioned reflex theory, has also developed rapidly. In the complex communication environment of satellites, optimizing the resource allocation of satellite systems by using reinforcement learning technology has become a hot topic of research. The use of reinforcement learning technology can enable intelligent agents to learn and accumulate experience in the process of continuous interaction with the environment, and maximize the reward benefits, thereby improving the decision-making optimization ability of satellite resource allocation. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of how to use reinforcement learning technology to dynamically allocate resources under the constraint of limited resources, and to provide a satellite resource allocation method based on Q learning (Q-learning based Satellite Resource Allocation, QSRA). Through the continuous interaction of the intelligent agent with the complex communication environment, the state information of the satellite communication system is obtained, and then the dynamic allocation of power resources of the satellite system is realized, and the spectrum utilization and system capacity of the system are improved. In order to achieve this purpose, the steps adopted by the present invention are:

[0006] Step 1: Establish a multi-beam satellite communication system model and provide an uplink channel link model and a downlink channel link model;

[0007] Step 2: Determine the objective function of satellite communication system resource allocation that maximizes the user's transmission rate under the condition of satisfying the on-board buffer constraint of the multi-beam satellite communication system;

[0008] Step 3: The multi-beam satellite agent continuously observes the communication environment, obtains the allocation status of carriers and users and the status of the satellite's onboard resource cache, and completes the selection of the power allocation action based on the Q table at this time;

[0009] Step 4: The communication environment updates the status of carrier allocation, served users, and on-board cache resources based on the execution results of the action, and evaluates the reward benefits of the execution results of this power allocation action, and feeds it back to the multi-beam satellite agent;

[0010] Step 5: The multi-beam satellite agent updates the corresponding Q value in the Q table according to the reward benefits fed back by the communication environment, the updated environmental state, and the update rule of the value function in the Q learning algorithm, so as to continuously update and optimize the agent's action selection strategy, and finally maximize the agent's reward benefits and find the optimal solution to the optimization problem.

[0011] Furthermore, the step 1 includes the following specific steps:

[0012] Step 1-1: In a multi-beam satellite communication system, the signal received by the satellite node from the ground user through the uplink can be expressed as:

[0013]

[0014] Among them, n 0 represents the zero-mean additive white Gaussian noise in the satellite-to-ground link, p u represents the transmission power of the ground user, x s Represents the corresponding signal, H s represents the channel from the satellite node to the ground user, which can be represented by the Gilbert-Elliott channel model;

[0015] Step 1-2: Assuming that the communication channel switches between two states, one is G (good) state and the other is B (bad) state, the channel state X(k) at time slot k can be obtained:

[0016]

[0017] Where γ(k) represents the signal-to-noise ratio (SNR) at time slot k, γ th represents the threshold value of γ(k);

[0018] Step 1-3: Assuming that the channel state is G at time slot k, the probability of still being in the G state at time slot k+1 is expressed as 1-q, and the probability of switching states is expressed as q. Similarly, if the channel state is B at time slot k, the probability of still being in the B state at time slot k+1 is expressed as 1-r, and the probability of switching states is expressed as r, which can be expressed as:

[0019]

[0020] Step 1-4: The transmission rate of the uplink channel link can be expressed as:

[0021]

[0022] Among them, w s is the link bandwidth of the uplink in the satellite-to-ground link, h s is the channel coefficient of the uplink in the satellite-to-ground link, and N is the noise power in the satellite-to-ground link;

[0023] Step 1-5: Similarly, the downlink model can be derived. In a multi-beam satellite communication system, the signal received by the ground user from the satellite node through the downlink can be expressed as:

[0024]

[0025] Among them, n 0 represents the zero-mean additive white Gaussian noise in the satellite-to-ground link, p s represents the transmission power of the communication satellite node, x uRepresents the corresponding signal, then the transmission rate of the downlink channel link can be expressed as:

[0026]

[0027] Among them, w u represents the link bandwidth of the downlink in the satellite-to-ground link, h u represents the channel coefficient of the downlink in the satellite-to-ground link, and N represents the noise power in the satellite-to-ground link.

[0028] Furthermore, the step 2 includes the following specific steps:

[0029] Step 2-1: Assume that the total system downlink power in the multi-beam satellite communication system is P total , the total number of beams in the satellite system is N total , then the power of each beam can be expressed as:

[0030]

[0031] Step 2-2: The total number of carriers in each beam is N carrier Expressed as: the power of each carrier in the multi-beam satellite communication system can be obtained:

[0032]

[0033] Where M = N total ·N carrier Indicates the total number of carriers in the system;

[0034] Step 2-3: Assume that the total bandwidth in the satellite communication system is B total , the bandwidth of each carrier can be obtained:

[0035]

[0036] It can be concluded that the maximum transmission rate of the multi-beam satellite communication system is:

[0037] C i =B m det(log 2 (I M +SINR i )) (10)

[0038] Wherein, m=1, 2, 3, ..., M represents the mth carrier in the multi-beam satellite communication system, i=1, 2, 3, ..., N represents the ith ground user, I M represents the M-order unit matrix, SINR i represents the signal-to-interference ratio matrix of ground user i, which can be obtained by the following formula:

[0039]

[0040] Among them, U i represents the useful power received by ground user i, R i represents the noise power of the satellite communication system, which is composed of co-channel interference and additive Gaussian white noise, that is,

[0041] Step 2-4: Assume that the ground user node G i With G j The amount of on-board resource cache generated during the communication process is According to equations (4) and (6), the on-board resource cache in the multi-beam satellite communication system can be obtained:

[0042]

[0043] Step 2-5: Therefore, the total on-board resource buffer in the satellite communication system can be obtained from formula (12):

[0044]

[0045] Among them, N up represents the total number of uplinks in the satellite communication system, N down represents the total number of downlinks in the satellite communication system;

[0046] Step 2-6: Obviously, assuming that the threshold value of the on-board resource buffer in the multi-beam satellite communication system is Φ th , the amount of on-board resource cache Φ(t) at time t should satisfy:

[0047] Φ(t)≤Φ th ; (14)

[0048] Step 2-7: Further, at time t, in order to maximize the system capacity of the satellite-to-ground link and ensure the user's service quality, the resource optimization problem of the satellite communication system is modeled as:

[0049]

[0050] in, represents the transmission rate of ground user u on the kth carrier on the mth beam in the satellite communication system at time t, represents the transmission rate of ground user u at time t, R th Indicates the minimum threshold of the ground user transmission rate. Indicates the allocation of the kth carrier of the terrestrial user u on the mth beam at time t (if the terrestrial user u is allocated to this subcarrier, then Otherwise it is 0), Φt represents the resource cache on the satellite at time t, P th Indicates the threshold value of the total power of the satellite communication system, Indicates the threshold value of a single beam power.

[0051] Furthermore, the step 3 includes the following specific steps:

[0052] Step 3-1: Define the state space S. The state s is essentially an abstract representation of the environment, which provides a basis for the action selection of the multi-beam satellite agent. According to the system model of the multi-beam satellite system in the above steps, the carrier allocation matrix W of the satellite system can be obtained:

[0053]

[0054] Among them, w n Indicates the carrier allocation status of the nth beam, which is a 1×N carrier The column vector of , from the above formula, we can get the carrier allocation of each beam. At the same time, as mentioned above, the resource cache at time t is To express;

[0055] Step 3-2: Define the allocated carrier matrix W, the served user set U, and the satellite resource cache Φ at time t as a state space, which can be expressed as follows:

[0056]

[0057] Step 3-3: Define the transmission power allocation as the agent's optional action set space A(s t ), considering the practical and training limitations, the power selection adopts a discrete power range;

[0058] Step 3-4: When the agent selects an action to perform, it is in the current action set space A(s t ) is performed according to the ε-greedy algorithm, that is, the multi-beam satellite agent has a probability of 1-ε to select the action that maximizes its Q value to execute, and has a probability of ε to randomly select an action in the feasible action set space to execute. The action selection strategy of the multi-beam satellite agent can be expressed as:

[0059]

[0060] Step 3-5: The agent's exploration process of the environment is a process of increasing cognition and experience. Therefore, at the beginning, the value of its exploration probability ε is generally large. However, as the agent's cognition of the environment and learning experience gradually increase, the exploration probability ε will continue to decrease, so that the probability 1-ε continues to increase, that is, there is a greater probability to choose an action that can obtain a larger Q value, thereby avoiding the agent from falling into a local optimal solution, so as to continuously optimize the exploration process of the multi-beam satellite agent and finally obtain the global optimal solution. The attenuation formula of its exploration probability can be expressed as:

[0061] ε=ε min +(ε max -ε min )·exp(-h·t) (19)

[0062] Where h represents the attenuation factor of the exploration probability ε, ε max represents the maximum value of ε, ε min It is obvious from formula (19) that as the agent's cognition of the environment increases, its exploration probability also decreases.

[0063] Furthermore, the step 4 includes the following specific steps:

[0064] Step 4-1: Define the reward benefit r obtained from the feedback of the environment after the multi-beam satellite agent performs the corresponding action and acts on the communication environment. The reward benefit feedback includes positive feedback benefit and negative feedback benefit, which is a reflection of the evaluation of the action result performed by the agent by the current state of the environment. Usually, the reward benefit is designed as a variable related to the performance of the entire communication system. In this resource allocation method, the increment Δδ of the transmission rate of the multi-beam satellite system is designed as the reward benefit of the agent, which can be expressed as:

[0065] Δδ=δ t+1 -δ t (20)

[0066] in, It represents the total transmission rate of the multi-beam satellite system at time t. It can be seen from formula (20) that the reward benefit is a variable positively correlated with the performance of the satellite system;

[0067] Step 4-2: The reward income can be expressed as follows:

[0068]

[0069] Among them, r g >r d , and r g 、r dThe value range of is a closed interval from 0 to 1. It can be seen from formula (21) that when the transmission rate increment is greater than 0, the reward feedback obtained by the agent will be greater, that is, r t =r g , otherwise, the benefits obtained by the agent will be reduced.

[0070] Furthermore, the step 5 includes the following specific steps:

[0071] Step 5-1: After the multi-beam satellite agent performs an action and acts on the communication environment, the reward gain from the environmental feedback is r, and the updated environmental state is s′;

[0072] Step 5-2: Define the discount factor γ to represent the attenuation coefficient of the benefits during the learning process. Generally, its value range is between 0 and 1.

[0073] Step 5-3: The multi-beam satellite agent updates the corresponding Q value in the Q table according to the reward r, the updated environment state s′, and the update rule of the value function in the Q learning algorithm, that is,

[0074] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′,a′)-Q(s,a)]; (22)

[0075] Step 5-4: Let s←s′, determine whether s is a terminal state, and determine whether the Q value table converges. If it converges, the training ends. Otherwise, repeat the above steps 4 to 6. Finally, the optimal allocation strategy is obtained as shown in the following formula:

[0076] π(s)=arg max a∈A Q(s, a). (23) BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a diagram of the algorithm architecture model based on Q-learning described in the present invention;

[0078] Figure 2 It is the algorithm flow chart of the present invention;

[0079] Figure 3 It is a model diagram of a multi-beam satellite system;

[0080] Figure 4 This is a simulation result diagram showing the impact of different learning efficiencies on the QSRA method;

[0081] Figure 5 This is a comparison chart of the convergence speed of the QSRA method under different business requirements;

[0082] Figure 6This is a simulation diagram comparing the cache optimization before and after the QSRA method; DETAILED DESCRIPTION

[0083] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0084] The purpose of this invention is to solve the problem of how to use reinforcement learning technology to perform dynamic resource allocation under the constraint of limited resources. A satellite resource allocation method based on Q learning is given. The architecture model diagram is shown in the attached figure. Figure 1 As shown in the flowchart attached Figure 2 As shown. Specifically including the following steps:

[0085] Step 1: Establish a multi-beam satellite communication system model and provide an uplink channel link model and a downlink channel link model;

[0086] Step 2: Determine the objective function of satellite communication system resource allocation that maximizes the user's transmission rate under the condition of satisfying the on-board buffer constraint of the multi-beam satellite communication system;

[0087] Step 3: The multi-beam satellite agent continuously observes the communication environment, obtains the allocation status of carriers and users and the status of the satellite's onboard resource cache, and completes the selection of the power allocation action based on the Q table at this time;

[0088] Step 4: The communication environment updates the status of carrier allocation, served users, and on-board cache resources based on the execution results of the action, and evaluates the reward benefits of the execution results of this power allocation action, and feeds it back to the multi-beam satellite agent;

[0089] Step 5: The multi-beam satellite agent updates the corresponding Q value in the Q table according to the reward benefits fed back by the communication environment, the updated environmental state, and the update rule of the value function in the Q learning algorithm, so as to continuously update and optimize the agent's action selection strategy, and finally maximize the agent's reward benefits and find the optimal solution to the optimization problem.

[0090] Furthermore, the above step 1 includes the following specific steps:

[0091] Step 1-1: Within the service coverage of the multi-beam satellite communication system, each beam can cover and serve multiple ground terminal devices (such as mobile terminals, vehicle terminals), and in the multi-beam satellite communication process, there are i (0≤i≤N up ) uplinks and j(0≤j≤N down) downlinks. The satellite communication system adopts time division multiplexing and frequency division multiplexing, in which the uplink model and the downlink model are both modeled as additive white Gaussian noise channel models. At the same time, there will be a certain degree of co-channel interference (CCI) between different beams.

[0092] At the same time, a requesting user can only occupy one subcarrier in the satellite communication system, and each subcarrier can only be allocated to one user. When two ground user nodes communicate through a satellite network, they may be in the same beam or between two different beams, and the communication may be between multiple different types of ground user terminals. Considering that there may be differences in the channel states of the uplink and downlink, in some cases, the downlink channel conditions are worse than the uplink channel conditions, which will generate cache resources on the satellite. However, the size of the satellite is limited, and the computing and storage resources on the satellite are very precious. Therefore, it is necessary to dynamically, efficiently and reasonably allocate the resources on the satellite, not only to ensure the user's service quality, but also to improve the system performance of the satellite communication system. The multi-beam satellite system model is shown in the figure below. Figure 3 shown.

[0093] In order to analyze the cache resources generated by the multi-beam satellite communication system during the communication service process, it is necessary to give the uplink channel link model and the downlink channel link model of the multi-beam satellite communication system. In the channel model, "satellite" generally refers to a low-orbit satellite or a geostationary satellite. Similarly, "ground users" correspond to users of low-orbit satellites or users of geostationary satellites.

[0094] In a multi-beam satellite communication system, the signal received by the satellite node from the ground user through the uplink can be expressed as:

[0095]

[0096] Among them, n 0 represents the zero-mean additive white Gaussian noise in the satellite-to-ground link, p u represents the transmission power of the ground user, x s Represents the corresponding signal, H s represents the channel from the satellite node to the ground user, which can be represented by the Gilbert-Elliott channel model;

[0097] Step 1-2: Assuming that the communication channel switches between two states, one is G (good) state and the other is B (bad) state, the channel state X(k) at time slot k can be obtained:

[0098]

[0099] Where γ(k) represents the signal-to-noise ratio (SNR) at time slot k, γ th represents the threshold value of γ(k);

[0100] Step 1-3: Assuming that the channel state is G at time slot k, the probability of still being in the G state at time slot k+1 is expressed as 1-q, and the probability of switching states is expressed as q. Similarly, if the channel state is B at time slot k, the probability of still being in the B state at time slot k+1 is expressed as 1-r, and the probability of switching states is expressed as r, which can be expressed as:

[0101]

[0102] Step 1-4: The transmission rate of the uplink channel link can be expressed as:

[0103]

[0104] Among them, w s is the link bandwidth of the uplink in the satellite-to-ground link, h s is the channel coefficient of the uplink in the satellite-to-ground link, and N is the noise power in the satellite-to-ground link;

[0105] Step 1-5: Similarly, the downlink model can be derived. In a multi-beam satellite communication system, the signal received by the ground user from the satellite node through the downlink can be expressed as:

[0106]

[0107] Among them, n 0 represents the zero-mean additive white Gaussian noise in the satellite-to-ground link, p s represents the transmission power of the communication satellite node, x u Represents the corresponding signal, then the transmission rate of the downlink channel link can be expressed as:

[0108]

[0109] Among them, w u represents the link bandwidth of the downlink in the satellite-to-ground link, h u represents the channel coefficient of the downlink in the satellite-to-ground link, and N represents the noise power in the satellite-to-ground link.

[0110] Furthermore, the above step 2 includes the following specific steps:

[0111] Step 2-1: Assume that the total system downlink power in the multi-beam satellite communication system is P total , the total number of beams in the satellite system is N total , then the power of each beam can be expressed as:

[0112]

[0113] Step 2-2: The total number of carriers in each beam is N carrier Expressed as: the power of each carrier in the multi-beam satellite communication system can be obtained:

[0114]

[0115] Where M = N total ·N carrier Indicates the total number of carriers in the system;

[0116] Step 2-3: Assume that the total bandwidth in the satellite communication system is B total , the bandwidth of each carrier can be obtained:

[0117]

[0118] It can be concluded that the maximum transmission rate of the multi-beam satellite communication system is:

[0119] C i =B m det(log 2 (I M +SINR i )) (10)

[0120] Wherein, m=1, 2, 3, ..., M represents the mth carrier in the multi-beam satellite communication system, i=1, 2, 3, ..., N represents the ith ground user, I M represents the M-order unit matrix, SINR i represents the signal-to-interference ratio matrix of ground user i, which can be obtained by the following formula:

[0121]

[0122] Among them, U i represents the useful power received by ground user i, R i represents the noise power of the satellite communication system, which is composed of co-channel interference and additive Gaussian white noise, that is,

[0123] Step 2-4: Assume that the ground user node G i With G j The amount of on-board resource cache generated during the communication process is According to equations (4) and (6), the on-board resource cache in the multi-beam satellite communication system can be obtained:

[0124]

[0125] Step 2-5: Therefore, the total on-board resource buffer in the satellite communication system can be obtained from formula (12):

[0126]

[0127] Among them, N up represents the total number of uplinks in the satellite communication system, N down represents the total number of downlinks in the satellite communication system;

[0128] Step 2-6: Obviously, assuming that the threshold value of the on-board resource buffer in the multi-beam satellite communication system is Φ th , the amount of on-board resource cache Φ(t) at time t should satisfy:

[0129] Φ(t)≤Φ th ; (14)

[0130] Step 2-7: Further, at time t, in order to maximize the system capacity of the satellite-to-ground link and ensure the user's service quality, the resource optimization problem of the satellite communication system is modeled as:

[0131]

[0132] In the above formula, represents the transmission rate of ground user u on the kth carrier on the mth beam in the satellite communication system at time t, represents the transmission rate of ground user u at time t, R th Indicates the minimum threshold of the ground user transmission rate. Indicates the allocation of the kth carrier of the terrestrial user u on the mth beam at time t (if the terrestrial user u is allocated to this subcarrier, then Otherwise, it is 0), Φt represents the resource cache on the satellite at time t, P th Indicates the threshold value of the total power of the satellite communication system, Indicates the threshold value of a single beam power.

[0133] The constraints of the above optimization problem are further described as follows:

[0134] C1: At any time, the user's transmission rate must be greater than or equal to the minimum transmission rate threshold;

[0135] C2, C3: At any time, any user is assigned at most one subcarrier, and each subcarrier can be assigned to at most one user;

[0136] C4: At any time, the amount of on-board resource cache must be less than the threshold value;

[0137] C5: total system power constraint;

[0138] C6: At any time, the power of each beam must be less than the single beam power constraint;

[0139] C7: Ensure that the single beam power is greater than or equal to 0 at any time.

[0140] Furthermore, the step 3 includes the following specific steps:

[0141] Step 3-1: Define the state space S. The state s is essentially an abstract representation of the environment, which provides a basis for the action selection of the multi-beam satellite agent. According to the system model of the multi-beam satellite system in the above steps, the carrier allocation matrix W of the satellite system can be obtained:

[0142]

[0143] Among them, w n Indicates the carrier allocation status of the nth beam, which is a 1×N carrier The column vector of , from the above formula, we can get the carrier allocation of each beam. At the same time, as mentioned above, the resource cache at time t is To express;

[0144] Step 3-2: Define the allocated carrier matrix W, the served user set U, and the satellite resource cache Φ at time t as a state space, which can be expressed as follows:

[0145]

[0146] Step 3-3: Define the transmission power allocation as the agent's optional action set space A(s t ), considering the practical and training limitations, the power selection adopts a discrete power range;

[0147] Step 3-4: When the agent selects an action to perform, it is in the current action set space A(s t ) is performed according to the ε-greedy algorithm, that is, the multi-beam satellite agent has a probability of 1-ε to select the action that maximizes its Q value to execute, and has a probability of ε to randomly select an action in the feasible action set space to execute. The action selection strategy of the multi-beam satellite agent can be expressed as:

[0148]

[0149] Step 3-5: The agent's exploration process of the environment is a process of increasing cognition and experience. Therefore, at the beginning, the value of its exploration probability ε is generally large. However, as the agent's cognition of the environment and learning experience gradually increase, the exploration probability ε will continue to decrease, so that the probability 1-ε continues to increase, that is, there is a greater probability to choose an action that can obtain a larger Q value, thereby avoiding the agent from falling into a local optimal solution, so as to continuously optimize the exploration process of the multi-beam satellite agent and finally obtain the global optimal solution. The attenuation formula of its exploration probability can be expressed as:

[0150] ε=ε min +(ε max -ε min )·exp(-h·t) (19)

[0151] Where h represents the attenuation factor of the exploration probability ε, ε max represents the maximum value of ε, ε min It is obvious from formula (19) that as the agent's cognition of the environment increases, its exploration probability also decreases.

[0152] Furthermore, the step 4 includes the following specific steps:

[0153] Step 4-1: Define the reward benefit r obtained from the feedback of the environment after the multi-beam satellite agent performs the corresponding action and acts on the communication environment. The reward benefit feedback includes positive feedback benefit and negative feedback benefit, which is a reflection of the evaluation of the action result performed by the agent by the current state of the environment. Usually, the reward benefit is designed as a variable related to the performance of the entire communication system. In this resource allocation method, the increment Δδ of the transmission rate of the multi-beam satellite system is designed as the reward benefit of the agent, which can be expressed as:

[0154] Δδ=δ t+1 -δ t (20)

[0155] in, It represents the total transmission rate of the multi-beam satellite system at time t. It can be seen from formula (20) that the reward benefit is a variable positively correlated with the performance of the satellite system;

[0156] Step 4-2: The reward income can be expressed as follows:

[0157]

[0158] Among them, r g >r d , and r g , r dThe value range of is a closed interval from 0 to 1. From formula (21), it can be seen that when the transmission rate increment is greater than 0, the reward feedback obtained by the agent will be greater, that is, r t =r g , otherwise, the benefits obtained by the agent will be reduced.

[0159] Furthermore, the step 5 includes the following specific steps:

[0160] Step 5-1: After the multi-beam satellite agent performs an action and acts on the communication environment, the reward gain from the environmental feedback is r, and the updated environmental state is s′;

[0161] Step 5-2: Define the discount factor γ to represent the attenuation coefficient of the benefits during the learning process. Generally, its value range is between 0 and 1.

[0162] Step 5-3: The multi-beam satellite agent updates the corresponding Q value in the Q table according to the reward r, the updated environment state s′, and the update rule of the value function in the Q learning algorithm, that is,

[0163] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′,a′)-Q(s,a)]; (22)

[0164] Step 5-4: Let s←s′, determine whether s is a terminal state, and determine whether the Q value table converges. If it converges, the training ends. Otherwise, repeat the above steps 4 to 6. Finally, the optimal allocation strategy is obtained as shown in the following formula:

[0165] π(s)=arg max a∈A Q(s, a). (23)

[0166] Finally, the pseudo code of the multi-beam satellite system resource allocation method based on Q learning is shown in Table 1.

[0167] Table 1 Pseudo code of 5G satellite-to-ground link multi-beam dynamic power allocation method based on Q learning

[0168]

[0169] In order to verify the effectiveness of the resource allocation method based on reinforcement learning, this paper uses Pycharm to build a simulation scenario of a multi-beam satellite communication system. The simulation scenario adopts a multi-beam satellite communication system, and its simulation parameters are shown in Table 2. At the beginning of the simulation, 100 ground user terminals are randomly distributed, and the exploration probability of Q learning is initialized according to formula (19), and it decreases as the agent explores the environment and accumulates experience.

[0170] Table 2 Simulation parameter design

[0171]

[0172] After the multi-beam satellite system is initialized, the agent starts to complete the learning process according to the Q learning algorithm, maintaining and updating the Q table. Table 3 gives the relevant parameters of Q learning.

[0173] Table 3 Q-learning related parameters

[0174]

[0175] In PyCharm, Python language is used to simulate the Q value change of the QSRA method under different learning efficiencies α. The learning efficiencies α are set to 0.1, 0.4, and 0.7, the discount factor γ is set to 0.9, and the attenuation factor h is set to 0.02. The simulation results are shown in the attached figure. Figure 4 As shown, it can be seen that when the learning efficiency of Q learning is 0.7, the convergence speed of the QSRA method is the fastest, and convergence can be obtained in about 500 times, while when the learning efficiency is 0.1, it takes about 5000 iterations, which is significantly slower than the convergence speed when the learning efficiency α=0.7.

[0176] In order to compare the convergence performance of the QSRA method under the conditions of multiple different communication service requests, Figure 5 The curve of the Q value of the QSRA method changing with the number of iterations for three different service requests is given, where the learning efficiency is set to 0.7, the discount factor is set to 0.9, and the attenuation factor of the exploration probability ε is 0.02. The results show that under the three different communication service conditions randomly generated by the ground terminal users, the QSRA method basically converges after about 284 iterations, which further illustrates that the proposed QSRA method has stable convergence.

[0177] Attached Figure 6 The on-board resource cache amount of the QSRA method is compared when considering cache resource constraints and when not considering cache resource constraints. The cache threshold value when considering cache constraints is set to 10Mbps. It can be clearly seen from the figure that when the number of iterations reaches about 500, the optimized QSRA method can converge quickly and reduce to below the threshold value of 10Mbps, and maintain at around 9.94Mbps. When the cache constraint is not considered, the number of iterations of the QSRA method reaches more than 2000 times before it begins to stabilize, the convergence speed is slow, and the cache is still around 16Mbps, and the occupied cache resources are significantly more than after the QSRA method is optimized. Therefore, after the QSRA method optimizes the on-board cache resources, it can significantly reduce the size of the satellite's on-board resource cache and reduce the satellite's resource storage burden.

[0178] comprehensive Figure 4-Figure 6 It can be seen that the Q-learning-based resource allocation method proposed in the present invention has good performance both in terms of the convergence of the method itself and in terms of system performance. It is an efficient and reliable dynamic resource allocation method. At the same time, it can adapt to complex and changeable communication environments and is particularly suitable for application in resource-constrained multi-beam satellite communication systems.

[0179] The contents not described in detail in the present application belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A 5G satellite-to-ground link multi-beam dynamic power allocation method based on reinforcement learning, specifically comprising the following steps: Step 1: Establish a multi-beam satellite communication system model, and give the uplink channel link model and the downlink channel link model, and obtain the transmission rate of the uplink channel link as where w s is the link bandwidth of the uplink in the satellite-to-ground link, h s is the channel coefficient of the uplink in the satellite-to-ground link, N is the noise power in the satellite-to-ground link, and p u represents the transmission power of the ground user. Similarly, the transmission speed of the downlink channel link is where w u is the link bandwidth of the downlink in the satellite-to-ground link, h u is the channel coefficient of the downlink in the satellite-to-ground link, N is the noise power in the satellite-to-ground link, and p s Indicates the transmission power of the communication satellite node; Step 2: Determine the objective function of satellite communication system resource allocation that maximizes the user's transmission rate under the condition of satisfying the on-board cache constraints of the multi-beam satellite communication system. At time t, the resource optimization problem of the satellite communication system is modeled as: Among them, N total Represents the total number of beams in the satellite system, N carrier represents the total number of carriers in each beam, represents the transmission rate of ground user u on the kth carrier on the mth beam in the satellite communication system at time t, represents the transmission rate of ground user u at time t, R th Indicates the minimum threshold of the ground user transmission rate. represents the allocation of the kth carrier of the ground user u on the mth beam at time t. If the ground user u is allocated to the kth carrier on the mth beam, then Otherwise it is 0, Φ t represents the resource cache on the satellite at time t, Φ th represents the threshold value of the on-board resource buffer in the multi-beam satellite communication system, P th Indicates the threshold value of the total power of the satellite communication system, Indicates the threshold value of a single beam power; Step 3: The multi-beam satellite agent continuously observes the communication environment, obtains the allocation status of carriers and users, and the status of the satellite's onboard resource cache. The carrier allocation matrix is ​​expressed as w n The carrier allocation status of the nth beam is represented by defining the allocated carrier matrix W, the served user set U, and the satellite resource cache Φ at time t as the state space, which is expressed as Then the power allocation action is selected according to the Q table at this time; Step 4: The communication environment updates the status of carrier allocation, served users, and on-board cache resources based on the execution results of the action, and evaluates the reward benefits of the execution results of this power allocation action, and feeds it back to the multi-beam satellite agent, which specifically includes the following steps: Step 4-1: After the multi-beam satellite agent performs the corresponding action and acts on the communication environment, the reward benefit obtained from the environmental feedback is r. The reward benefit feedback results include positive feedback benefits and negative feedback benefits, which is a reflection of the current state of the environment's evaluation of the results of the actions performed by the agent. In this resource allocation method, the increment Δδ of the multi-beam satellite system transmission rate is designed as the reward benefit of the agent, which is expressed as: Δδ=δ t+1 -d t in, represents the total transmission rate of the multi-beam satellite system at time t; Step 4-2: The reward income is: Among them, r g >r d , and r g 、r d The value range of is a closed interval of 0 to 1. When the transmission rate increment is greater than 0, the reward feedback obtained by the agent will be greater, that is, r t =r g , otherwise, the benefits obtained by the agent will be reduced; Step 5: The multi-beam satellite agent updates the corresponding Q value in the Q table according to the reward benefits fed back by the communication environment, the updated environmental state, and the update rule of the value function in the Q learning algorithm, so as to continuously update and optimize the agent's action selection strategy, and finally maximize the agent's reward benefits and find the optimal solution to the optimization problem.

Images