Intelligent resource allocation method and device in low-orbit satellite communication

Abstract

The application provides an intelligent resource allocation method and device in low-orbit satellite communication, and relates to the technical field of satellite communication.The method comprises the following steps: initializing parameters in a satellite communication system, receiving an access request of a user; constructing a Markov decision process for a resource allocation optimization problem in the satellite communication system to obtain a reward function; constructing a deep Q network, training and updating the deep Q network according to the reward function; receiving a user request state, outputting a final resource strategy, and completing intelligent resource allocation in low-orbit satellite communication.The method can effectively allocate resources for a dynamic satellite communication system, improve the energy efficiency and spectrum efficiency of the satellite communication system, optimize the throughput and power performance of a multi-beam satellite system, and achieve the purpose of multi-objective optimization of satellite resource allocation.

Description

Technical Field

[0001] This invention relates to the field of satellite communication technology, and in particular to an intelligent resource allocation method and apparatus for low-Earth orbit satellite communication. Background Technology

[0002] With the advent of the information age, satellite communication has become a crucial communication method. Compared with traditional mobile communication methods, satellite communication has advantages such as wide coverage, long communication distance, and large communication capacity, making it an important supplement to terrestrial mobile communication.

[0003] In multi-beam satellite communication systems, each satellite carries multiple beams, covering different areas of the ground. Effective resource allocation can significantly improve the performance of the satellite communication system. Traditional resource allocation schemes are fixed and not suitable for the highly complex and dynamic low-Earth orbit satellite communication systems. Summary of the Invention

[0004] To address the problem of resource allocation in the dynamically changing satellite environment that cannot be solved by existing technologies, this invention proposes an intelligent resource allocation method and device for low-Earth orbit satellite communication.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] On the one hand, an intelligent resource allocation method for low-Earth orbit satellite communication is provided. This method is applied to electronic devices and includes the following steps:

[0007] S1: Initialize the parameters in the satellite communication system and receive user access requests;

[0008] S2: The resource allocation optimization problem in the satellite communication system is constructed as a Markov decision process to obtain the reward function;

[0009] S3: Construct a deep Q-network, and train and update the deep Q-network according to the reward function;

[0010] S4: Receive user request status, output final resource strategy, and complete intelligent resource allocation in low-Earth orbit satellite communication.

[0011] Optionally, in step S1, the parameters of the satellite communication system are initialized, including:

[0012] The initial total number of users is U set ={u|u=1,2,3…,U}; Initialize the low-orbit satellite state-action value function Q(s,a); where s represents the state and a represents the action; Set the discount coefficient λ, set the depth Q network parameters, and set the target depth Q network parameters.

[0013] Optionally, in step S1, receiving the user's access request includes:

[0014] User u t When time t arrives and an access request is sent, if there is an idle carrier in the beam the user is trying to access, the user is allowed to access; otherwise, the user waits and proceeds to the next time step until successful access is achieved.

[0015] Optionally, in step S2, the resource allocation optimization problem in the satellite communication system is constructed as a Markov decision process to obtain the reward function, including:

[0016] S21: Calculate the state variable s in the Markov decision process according to the following formula (1):

[0017] s={W t U t ,u t} (1)

[0018] Among them, W t U represents the satellite carrier allocation matrix at time t. t u represents the set of users that have already been served. t This indicates a newly arrived user who needs service;

[0019] S22: Input the state variable s into the deep Q network, and output M Q values ​​corresponding to different carriers. Then, calculate the action value a in the Markov decision process according to the following formula (2):

[0020] a = {m, m ∈ M} (2)

[0021] Where m represents the m-th Q-value;

[0022] S23: Define the reward r in the Markov decision process as the judgment index of the optimization objective. Then, construct the reward function according to the following formula (3), that is, the increment ΔF of the optimization objective, and use it to judge the increase or decrease of the reward r:

[0023] ΔF=F t+1 -F t (3)

[0024] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

[0025] Optionally, in step S23, constructing the incremental ΔF of the optimization objective includes:

[0026] S231: Calculate the total power P of the satellite communication system according to the following formula (3-1):

[0027]

[0028] Where p0 is the power loss of the satellite platform itself, p n This represents the power of the nth beam;

[0029] S232: Calculate the energy efficiency EE of the satellite communication system under ideal conditions according to the following formula (3-2):

[0030]

[0031] Among them, R i This represents the maximum speed of user i;

[0032] S234: Calculate the spectral efficiency SE of the satellite communication system according to the following formula (3-3):

[0033]

[0034] Where W represents the total bandwidth of the carrier;

[0035] S235: Based on the spectral efficiency SE, calculate the optimization objective F of the satellite communication system as shown in the following formula (3-4):

[0036] F=ω1a1SE+ω2a2SE (3-4)

[0037] Where a1 and a2 represent the normalization parameters in the formula, i.e., a1+a2=1, and ω1 and ω2 represent the weight parameters of the satellite communication system; according to the optimization objective F, the increment of the optimization objective is obtained as follows (3):

[0038] ΔF=F t+1 -F t (3)

[0039] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

[0040] Optionally, in step S3, a deep Q-network is constructed, and the deep Q-network is trained and updated according to the reward function, including:

[0041] S31: Based on the obtained state s and action value a, configure the action-value network, i.e., reconfigure the deep Q network;

[0042] S32: Construct the deep Q-network using a convolutional neural network, and perform reinforcement learning on the deep Q-network according to the reward function;

[0043] S33: Calculate the error function, and use the calculation result to perform reverse training and update the deep Q network after reinforcement learning through the stochastic gradient descent (SGD) method to correct the network parameters.

[0044] Optionally, in step S33, the error function is calculated, and the calculation result is used to back-train and update the trained deep Q-network using the stochastic gradient descent (SGD) method to correct the network parameters, including:

[0045] S331: Calculate the error function L(θ) according to the following formula (4):

[0046] L(θ)=E[y i -Q(s,a;θ)] (4)

[0047] Among them, y i This represents the target Q-value that the deep Q-network aims to achieve.

[0048] S332: The calculation results are used to back-train and update the trained deep Q-network using the stochastic gradient descent (SGD) method. After a certain number of steps, the deep Q-network is copied to the target Q-network, and the parameters θ of the target Q-network are updated using the parameters of the Q-network. - , that is, θ - =θ.

[0049] Optionally, step S33 further includes: using an experience replay mechanism to establish an experience pool during the reinforcement learning process, putting data into the experience pool, and randomly selecting data from the experience pool to perform reverse training and update of the neural network when the data in the experience pool reaches a certain amount.

[0050] Optionally, in step S4, the user request status is received, the final resource strategy is output, and intelligent resource allocation in low-Earth orbit satellite communication is completed, including:

[0051] S41: Receive user request status. If the user leaves the system, delete the carrier used during the online time and enter the next status time. If the user has not left the system, continue to perform resource allocation operation.

[0052] S42: Outputs the final strategy, resource allocation results, and optimization results, completing intelligent resource allocation in low-orbit satellite communication.

[0053] On the one hand, an intelligent resource allocation device for low-Earth orbit satellite communication is provided. This device is applied to electronic equipment and includes:

[0054] The initialization module is used to initialize the parameters in the satellite communication system and receive user access requests.

[0055] An optimization target construction module is used to construct the resource allocation optimization problem in the satellite communication system as a Markov decision process to obtain the reward function;

[0056] The training and update module is used to construct a deep Q-network and train and update the deep Q-network according to the reward function.

[0057] The resource allocation module is used to receive user request status, output the final resource strategy, and complete intelligent resource allocation in low-Earth orbit satellite communication.

[0058] On the one hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the aforementioned intelligent resource allocation method in low-Earth orbit satellite communication.

[0059] On the one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned intelligent resource allocation method in low-Earth orbit satellite communication.

[0060] The above-described technical solutions of the embodiments of the present invention have at least the following beneficial effects:

[0061] In the above-described scheme, this invention proposes an intelligent resource allocation method based on reinforcement learning. This algorithm models the resource allocation optimization problem in satellite communication systems as a Markov decision process and utilizes the interactive learning between the satellite agent and the environment to complete resource allocation and optimization. The method provided by this invention can effectively allocate resources in dynamic satellite communication systems, improve the energy efficiency and spectral efficiency of satellite communication systems, optimize the throughput and power performance of multi-beam satellite systems, and achieve the goal of multi-objective optimization of satellite resource allocation. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of an intelligent resource allocation method in low-Earth orbit satellite communication provided by an embodiment of the present invention;

[0064] Figure 2 This is a flowchart of an intelligent resource allocation method in low-Earth orbit satellite communication provided by an embodiment of the present invention;

[0065] Figure 3 This is a multi-beam satellite system model diagram of an intelligent resource allocation method in low-Earth orbit satellite communication provided by an embodiment of the present invention;

[0066] Figure 4 This is a diagram illustrating the architecture of an intelligent resource allocation method in low-Earth orbit satellite communication, as provided in an embodiment of the present invention.

[0067] Figure 5 This is a block diagram of an intelligent resource allocation device in low-Earth orbit satellite communication provided by an embodiment of the invention;

[0068] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0070] This invention provides an intelligent resource allocation method for low-Earth orbit satellite communication. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart shown illustrates an intelligent resource allocation method in low-Earth orbit satellite communication. The processing flow of this method may include the following steps:

[0071] S101: Initialize the parameters in the satellite communication system and receive user access requests;

[0072] S102: The resource allocation optimization problem in the satellite communication system is constructed as a Markov decision process to obtain the reward function;

[0073] S103: Construct a deep Q-network, and train and update the deep Q-network according to the reward function;

[0074] S104: Receive user request status, output final resource strategy, and complete intelligent resource allocation in low-Earth orbit satellite communication.

[0075] Optionally, in step S101, the parameters of the satellite communication system are initialized, including:

[0076] The initial total number of users is U set ={u|u=1,2,3…,U}; Initialize the low-orbit satellite state-action value function Q(s,a); where s represents the state and a represents the action; Set the discount coefficient λ, set the depth Q network parameters, and set the target depth Q network parameters.

[0077] Optionally, in step S101, receiving the user's access request includes:

[0078] User u t When time t arrives and an access request is sent, if there is an idle carrier in the beam the user is trying to access, the user is allowed to access; otherwise, the user waits and proceeds to the next time step until successful access is achieved.

[0079] Optionally, in step S102, the resource allocation optimization problem in the satellite communication system is constructed as a Markov decision process to obtain the reward function, including:

[0080] S121: Calculate the state variable s in the Markov decision process according to the following formula (1):

[0081] s={W t U t ,u t} (1)

[0082] Among them, W t U represents the satellite carrier allocation matrix at time t. t u represents the set of users that have already been served. t This indicates a newly arrived user who needs service;

[0083] S122: Input the state variable s into the deep Q network, and output M Q values ​​corresponding to different carriers. Then, calculate the action value a in the Markov decision process according to the following formula (2):

[0084] a = {m, m ∈ M} (2)

[0085] Where m represents the m-th Q-value;

[0086] S123: Define the reward r in the Markov decision process as the judgment index of the optimization objective. Then, construct the reward function according to the following formula (3), that is, the increment ΔF of the optimization objective, and use it to judge the increase or decrease of the reward r:

[0087] ΔF=F t+1 -F t (3)

[0088] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

[0089] Optionally, in step S123, constructing the incremental ΔF of the optimization objective includes:

[0090] S1231: Calculate the total power P of the satellite communication system according to the following formula (3-1):

[0091]

[0092] Where p0 is the power loss of the satellite platform itself, p n This represents the power of the nth beam;

[0093] S1232: Calculate the energy efficiency EE of the satellite communication system under ideal conditions according to the following formula (3-2):

[0094]

[0095] Among them, R i This represents the maximum speed of user i;

[0096] S1234: Calculate the spectral efficiency SE of the satellite communication system according to the following formula (3-3):

[0097]

[0098] Where W represents the total bandwidth of the carrier;

[0099] S1235: Based on the spectral efficiency SE, calculate the optimization objective F of the satellite communication system as shown in the following formula (3-4):

[0100] F=ω1a1SE+ω2a2SE (3-4)

[0101] Where a1 and a2 represent the normalization parameters in the formula, i.e., a1+a2=1, and ω1 and ω2 represent the weight parameters of the satellite communication system; according to the optimization objective F, the increment of the optimization objective is obtained as follows (3):

[0102] ΔF=F t+1 -F t (3)

[0103] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

[0104] Optionally, in step S103, constructing a deep Q-network and training and updating the deep Q-network according to the reward function includes:

[0105] S131: Based on the obtained state s and action value a, configure the action-value network, i.e., the deep Q-network;

[0106] S132: Construct the deep Q-network using a convolutional neural network, and perform reinforcement learning on the deep Q-network according to the reward function;

[0107] S133: Calculate the error function, and use the calculation result to perform reverse training and update the deep Q network after reinforcement learning through the stochastic gradient descent (SGD) method to correct the network parameters.

[0108] Optionally, in step S133, the error function is calculated, and the calculation result is used to back-train and update the trained deep Q-network using the stochastic gradient descent (SGD) method to correct the network parameters, including:

[0109] S1331: Calculate the error function L(θ) according to the following formula (4):

[0110] L(θ)=E[y i -Q(s,a;θ)] (4)

[0111] Among them, y i This represents the target Q-value that the deep Q-network aims to achieve.

[0112] S1332: The calculation results are used to back-train and update the trained deep Q-network using the stochastic gradient descent (SGD) method. After a certain number of steps, the deep Q-network is copied to the target Q-network, and the parameters θ of the target Q-network are updated using the parameters of the Q-network. - , that is, θ - =θ.

[0113] Optionally, step S133 further includes: using an experience replay mechanism to establish an experience pool during the reinforcement learning process, putting data into the experience pool, and randomly selecting data from the experience pool to perform reverse training and update of the neural network when the data in the experience pool reaches a certain amount.

[0114] Optionally, in step S104, the user request status is received, the final resource strategy is output, and intelligent resource allocation in low-Earth orbit satellite communication is completed, including:

[0115] S141: Receive user request status. If the user leaves the system, delete the carrier used during the online time and enter the next status time. If the user has not left the system, continue to perform resource allocation operation.

[0116] S142: Outputs the final strategy, resource allocation results, and optimization results, completing intelligent resource allocation in low-orbit satellite communication.

[0117] This invention proposes an intelligent resource allocation method based on reinforcement learning. The algorithm models the resource allocation optimization problem in satellite communication systems as a Markov decision process and utilizes the interactive learning between the satellite agent and the environment to complete resource allocation and optimization. The method provided by this invention can effectively allocate resources in dynamic satellite communication systems, improve the energy efficiency and spectral efficiency of satellite communication systems, optimize the throughput and power performance of multi-beam satellite systems, and achieve the goal of multi-objective optimization of satellite resource allocation.

[0118] This invention provides an intelligent resource allocation method for low-Earth orbit satellite communication. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 2 The flowchart shown illustrates an intelligent resource allocation method in low-Earth orbit satellite communication. The processing flow of this method may include the following steps:

[0119] S201: Initialize the parameters in the satellite communication system and receive user access requests;

[0120] In one feasible implementation, initializing parameters in the satellite communication system includes:

[0121] The initial total number of users is U set ={u|u=1,2,3…,U}; Initialize the low-orbit satellite state-action value function Q(s,a); where s represents the state and a represents the action; Set the discount coefficient λ, set the depth Q network parameters, and set the target depth Q network parameters.

[0122] In one feasible implementation, receiving a user's access request includes:

[0123] User u t When time t arrives and an access request is sent, if there is an idle carrier in the beam the user is trying to access, the user is allowed to access; otherwise, the user waits and proceeds to the next time step until successful access is achieved.

[0124] like Figure 3 The proposed multi-beam satellite system model is as follows: Figure 4 As shown, this is the framework of the proposed intelligent resource allocation method in low-Earth orbit satellite communication. This method treats the satellite as an intelligent agent, models the satellite communication channel environment as an interactive environment, uses convolutional neural networks to extract features of the channel power allocation state and beam user service request state in the environment, and outputs a resource allocation scheme through a deep Q network to allocate carrier resources to service requesting users. Based on the feedback benefits given by the environment, the channel power allocation strategy, i.e., the deep Q network, is trained, optimized and updated, thereby maximizing the energy efficiency and spectral efficiency of the satellite.

[0125] S202: Calculate the state variable s in the Markov decision process according to the following formula (1):

[0126] s={W t U t ,u t} (1)

[0127] Among them, W t U represents the satellite carrier allocation matrix at time t. t u represents the set of users that have already been served. t This indicates a newly arrived user who needs service;

[0128] In this embodiment of the invention, a Markov Decision Process (MDP) is a mathematical model of sequential decision-making used to simulate stochastic policies and rewards achievable by an agent in an environment where the system state has Markov properties. The MDP is constructed based on a set of interactive objects, namely the agent and the environment, and its elements include state, action, and reward. The resource allocation problem is modeled as a Markov Decision Process, the satellite is modeled as the agent, and the channel environment is modeled as the interactive environment. Corresponding states s, actions a, and rewards r need to be established. Therefore, the state is defined as the current carrier allocation state of the system and new service request information; the action is defined as the carrier allocated to the service request; and the reward is defined as the gain of the optimization objective.

[0129] In one feasible implementation, the state is a representation of the environment and must contain key information about the environment. For the satellite's agent to optimize energy efficiency and spectral efficiency, the state quantity should include the satellite carrier allocation matrix at time t, the set of already served users, and newly arrived users requiring service.

[0130] The satellite communication system forms N beams on the ground via an onboard multi-beam transmitter, with the number being N. set = {n|n=1,2,3…,N}. The total bandwidth of the system is W, and the bandwidth of each carrier is W. m Then the maximum number of carriers M c For W / W m Since carriers are dynamically allocated, the bandwidth of each beam depends on the number of carriers allocated to that beam. A larger number of carriers indicates that the beam has a larger bandwidth. Therefore, the satellite carrier allocation matrix is ​​as follows:

[0131]

[0132] Where each element v in the matrix m,n The value is either 0 or 1, where 1 represents that the m-th carrier is assigned to the n-th beam, and 0 represents that the m-th carrier is not assigned to a beam. At different times t, V t It is also constantly changing.

[0133] S203: Input the state variable s into the deep Q network, and output M Q values ​​corresponding to different carriers. Then, calculate the action value a in the Markov decision process according to the following formula (2):

[0134] a = {m, m ∈ M} (2)

[0135] Where m represents the m-th Q-value.

[0136] In one feasible implementation, the action is the agent's output to the environment; in this method, it involves assigning a corresponding carrier to the user. After the agent's state variables are input into the Q-network, it outputs M Q-values ​​corresponding to different carriers. A greedy algorithm is used to select a Q-value, which corresponds to the selection of the appropriate carrier.

[0137] S204: Define the reward r in the Markov decision process as the judgment index of the optimization objective. Then, construct the reward function according to the following formula (3), that is, the increment ΔF of the optimization objective, and use it to judge the increase or decrease of the reward r:

[0138] ΔF=F t+1 -F t (3)

[0139] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system.n .

[0140] In this embodiment of the invention, the resource allocation optimization problem in the satellite communication system is modeled as a Markov decision process, the satellite is modeled as an agent, the satellite communication channel environment is modeled as an interactive environment, and energy efficiency and spectral efficiency are used as optimization objectives. Resource allocation and optimization are completed by the interactive learning between the satellite agent and the environment.

[0141] In one feasible implementation, the reward is the feedback from the environment during the interaction between the agent and the environment. It is an evaluation of the agent's selected action and a criterion for judging whether the optimization goal has been achieved. The goal of this method is to optimize energy efficiency and spectral efficiency. Based on this, this method uses the increment of the optimization goal to measure the system performance.

[0142] In one feasible implementation, the incremental ΔF of the optimization objective is constructed, including:

[0143] Assume the power of each carrier is a fixed value p. c Then the power of each beam can be expressed as:

[0144]

[0145] The total power P of the satellite communication system can then be calculated using the following formula (3-1):

[0146]

[0147] Where p0 is the power loss of the satellite platform itself, p n This represents the power of the nth beam;

[0148] Assuming the system is in an ideal state, i.e., the user can achieve the maximum data rate, the energy efficiency EE of the satellite communication system under ideal conditions can be calculated using the following formula (3-2):

[0149]

[0150] Among them, R i Indicates the maximum rate of user i:

[0151] R i =W m det(1b(I M +SINR i (4)

[0152] Among them, SINR i It is the signal-to-interference-plus-noise ratio (SIR) matrix for user i, taking into account co-frequency interference between beams and Gaussian white noise.

[0153] The spectral efficiency SE of a satellite communication system is calculated using the following formula (3-3):

[0154]

[0155] Where W represents the total bandwidth of the carrier;

[0156] Based on the spectral efficiency SE, calculate the optimization objective F of the satellite communication system as shown in the following formula (3-4):

[0157] F=ω1a1SE+ω2a2SE (3-4)

[0158] Where a1 and a2 represent the normalization parameters in the formula, i.e., a1+a2=1, and ω1 and ω2 represent the weight parameters of the satellite communication system; according to the optimization objective F, the increment of the optimization objective is obtained as follows (3):

[0159] ΔF=F t+1 -F t (3)

[0160] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

[0161] S205: Based on the obtained state s and action value a, configure the action-value network, i.e., the deep Q-network.

[0162] In one feasible implementation, an action-value network, or Q-network, is configured based on the states and actions in the established MDP. Deep reinforcement learning completes the mapping from state to action through a deep Q-network. The deep Q-network extracts features from the environmental state and outputs the Q-values ​​corresponding to all actions; that is, the output of the neural network is Q(s,a). The agent selects the appropriate action to execute based on the Q-values. Deep reinforcement learning often uses convolutional neural networks to build deep Q-networks. The Q-network is trained, and the error function is calculated. The calculation results are then used to back-train and update the Q-network using the stochastic gradient descent (SGD) method to correct the network parameters. This action-value network uses a neural network (including deep neural networks), with the input being the state value s_t and the output being the M real values ​​corresponding to each action.

[0163] S206: Construct the deep Q-network using a convolutional neural network, and perform reinforcement learning on the deep Q-network according to the reward function.

[0164] S207: Calculate the error function, and use the calculation result to perform reverse training and update the deep Q network after reinforcement learning through the stochastic gradient descent (SGD) method to correct the network parameters.

[0165] In one feasible implementation, the error function L(θ) is calculated according to the following formula (4):

[0166] L(θ)=E[y i -Q(s,a;θ)] (4)

[0167] Among them, y i This represents the target Q-value that the deep Q-network aims to achieve; y is calculated according to the following formula. i

[0168]

[0169] The calculation results are used to back-train and update the trained deep Q-network using the stochastic gradient descent (SGD) method. After a certain number of steps, the deep Q-network is copied to the target Q-network, and the parameters θ of the target Q-network are updated using the parameters of the Q-network. - , that is, θ - =θ.

[0170] In one feasible implementation, this invention employs an experience replay mechanism. During reinforcement learning, an experience pool is established, and data is placed into it. When the experience pool reaches a certain size, data is randomly selected from the pool for reverse training and updating of the neural network. Both experience replay and target Q-network techniques are used to make the training and updating of the deep Q-network more stable.

[0171] In one feasible implementation, during the training process, since the data generated by the interaction between the agent and the environment has a certain correlation, using this data to train the network will make the network less stable. Therefore, in order to break the correlation between the data, an experience replay mechanism is adopted. That is, during the reinforcement learning process, an experience pool is established, and data is put into the experience pool. When the data in the experience pool reaches a certain amount, data is randomly selected from the experience pool for neural network training.

[0172] In this embodiment of the invention, the training, optimization and updating of the deep Q network are carried out by using convolutional neural networks to extract features of carrier allocation status and user service request status in a multi-beam environment, and by outputting a resource allocation scheme through the deep Q network to allocate carrier resources to users requesting services. The training and updating of the network are completed by using rewards.

[0173] S208: Receive user request status. If the user leaves the system, delete the carrier used during the online time and enter the next status time. If the user has not left the system, continue to perform resource allocation operation.

[0174] S209: Outputs the final strategy, resource allocation results, and optimization results to complete intelligent resource allocation in low-orbit satellite communication.

[0175] This invention proposes an intelligent resource allocation method based on reinforcement learning. The proposed method employs a dynamic resource allocation approach using deep reinforcement learning. Comparison with traditional fixed resource allocation schemes and traditional dynamic channel allocation schemes shows that the proposed method effectively improves the energy efficiency and spectral efficiency of satellite communication systems, and effectively optimizes the throughput and power performance of multi-beam satellite systems. The algorithm models the resource allocation optimization problem in satellite communication systems as a Markov decision process and utilizes the interactive learning between the satellite agent and the environment to complete resource allocation and optimization. The method provided by this invention can effectively allocate resources in dynamic satellite communication systems, improve the energy efficiency and spectral efficiency of satellite communication systems, and optimize the throughput and power performance of multi-beam satellite systems, achieving the goal of multi-objective optimization of satellite resource allocation.

[0176] Figure 5 A block diagram of an intelligent resource allocation device in low-Earth orbit satellite communication is shown according to an exemplary embodiment. (Refer to...) Figure 5 The device 300 includes:

[0177] The initialization module 310 is used to initialize the parameters in the satellite communication system and receive user access requests.

[0178] The optimization objective construction module 320 is used to construct the resource allocation optimization problem in the satellite communication system into a Markov decision process to obtain the reward function;

[0179] The training and update module 330 is used to construct a deep Q-network and train and update the deep Q-network according to the reward function.

[0180] The resource allocation module 340 is used to receive user request status, output the final resource strategy, and complete intelligent resource allocation in low-Earth orbit satellite communication.

[0181] Optionally, the initialization module 310 is used to initialize parameters in the satellite communication system, including:

[0182] The initial total number of users is U set={u|u=1,2,3…,U}; Initialize the low-orbit satellite state-action value function Q(s,a); where s represents the state and a represents the action; Set the discount coefficient λ, set the depth Q network parameters, and set the target depth Q network parameters.

[0183] Optionally, the initialization module 310 is used by user u t When time t arrives and an access request is sent, if there is an idle carrier in the beam the user is trying to access, the user is allowed to access; otherwise, the user waits and proceeds to the next time step until successful access is achieved.

[0184] Optionally, the optimization objective building module 320 is used to calculate the state variable s in the Markov decision process according to the following formula (1):

[0185] s={W t U t ,u t} (1)

[0186] Among them, W t U represents the satellite carrier allocation matrix at time t. t u represents the set of users that have already been served. t This indicates a newly arrived user who needs service;

[0187] The state variable s is input into the deep Q network, and M Q values ​​corresponding to different carriers are output. Then, the action value a in the Markov decision process is calculated according to the following formula (2):

[0188] a = {m, m ∈ M} (2)

[0189] Where m represents the m-th Q-value;

[0190] If the reward r in the Markov decision process is defined as the criterion for judging the optimization objective, then the reward function is constructed according to the following formula (3), which is the increment ΔF of the optimization objective, and the increase or decrease of the reward r is judged accordingly:

[0191] ΔF=F t+1 -F t (3)

[0192] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

[0193] Optionally, the target construction module 320 is optimized to calculate the total power P of the satellite communication system according to the following formula (3-1):

[0194]

[0195] Where p0 is the power loss of the satellite platform itself, p n This represents the power of the nth beam;

[0196] The energy efficiency EE of the satellite communication system under ideal conditions is calculated using the following formula (3-2):

[0197]

[0198] Among them, R i This represents the maximum speed of user i;

[0199] The spectral efficiency SE of a satellite communication system is calculated using the following formula (3-3):

[0200]

[0201] Where W represents the total bandwidth of the carrier;

[0202] Based on the spectral efficiency SE, calculate the optimization objective F of the satellite communication system as shown in the following formula (3-4):

[0203] F=ω1a1SE+ω2a2SE (3-4)

[0204] Where a1 and a2 represent the normalization parameters in the formula, i.e., a1+a2=1, and ω1 and ω2 represent the weight parameters of the satellite communication system; according to the optimization objective F, the increment of the optimization objective is obtained as follows (3):

[0205] ΔF=F t+1 -F t (3)

[0206] Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

[0207] Optionally, the training update module 330 is used to configure the action-value network, i.e., the deep Q-network, based on the obtained state s and action value a;

[0208] The deep Q-network is constructed using a convolutional neural network, and reinforcement learning is performed on the deep Q-network based on the reward function.

[0209] The error function is calculated, and the calculation result is used to back-train and update the deep Q-network after reinforcement learning through the stochastic gradient descent (SGD) method to correct the network parameters.

[0210] Optionally, the resource allocation module 330 is used to calculate the error function L(θ) according to the following formula (4):

[0211] L(θ)=E[y i -Q(s,a;θ)] (4)

[0212] Among them, y i This represents the target Q-value that the deep Q-network aims to achieve.

[0213] The calculation results are used to back-train and update the trained deep Q-network using the stochastic gradient descent (SGD) method. After a certain number of steps, the deep Q-network is copied to the target Q-network, and the parameters θ of the target Q-network are updated using the parameters of the Q-network. - , that is, θ - =θ.

[0214] Optionally, the training update module 330 is also used to employ an experience replay mechanism to establish an experience pool during the reinforcement learning process, put data into the experience pool, and when the data in the experience pool reaches a certain amount, randomly select data from the experience pool for reverse training and update of the neural network.

[0215] Optionally, the resource allocation module 340 is used to receive user request status. If the user leaves the system, the carrier used during the user's online time is deleted, and the system enters the next status time. If the user does not leave the system, the resource allocation operation continues.

[0216] Output the final strategy, resource allocation results, and optimization results to complete intelligent resource allocation in low-orbit satellite communication.

[0217] This invention proposes an intelligent resource allocation method based on reinforcement learning. The algorithm models the resource allocation optimization problem in satellite communication systems as a Markov decision process and utilizes the interactive learning between the satellite agent and the environment to complete resource allocation and optimization. The method provided by this invention can effectively allocate resources in dynamic satellite communication systems, improve the energy efficiency and spectral efficiency of satellite communication systems, optimize the throughput and power performance of multi-beam satellite systems, and achieve the goal of multi-objective optimization of satellite resource allocation.

[0218] Figure 6This invention provides a schematic diagram of the structure of an electronic device 400. The electronic device 400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 401 and one or more memories 402. The memory 402 stores at least one instruction, which is loaded and executed by the processor 401 to implement the steps of the intelligent resource allocation method in low-Earth orbit satellite communication described below:

[0219] S1: Initialize the parameters in the satellite communication system and receive user access requests;

[0220] S2: The resource allocation optimization problem in the satellite communication system is constructed as a Markov decision process to obtain the reward function;

[0221] S3: Construct a deep Q-network, and train and update the deep Q-network according to the reward function;

[0222] S4: Receive user request status, output final resource strategy, and complete intelligent resource allocation in low-Earth orbit satellite communication.

[0223] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the intelligent resource allocation method in low-Earth orbit satellite communication described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0224] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0225] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent resource allocation method for low-Earth orbit satellite communication, characterized in that, Includes the following steps: S1: Initialize the parameters in the satellite communication system and receive user access requests; S2: The resource allocation optimization problem in the satellite communication system is constructed as a Markov decision process to obtain the reward function; In step S2, the resource allocation optimization problem in the satellite communication system is constructed as a Markov decision process to obtain the reward function, including: S21: Calculate the state variable s in the Markov decision process according to the following formula (1): s={W t ,U t ,u t } (1) Among them, W t U represents the satellite carrier allocation matrix at time t. t u represents the set of users that have already been served. t This indicates a newly arrived user who needs service; S22: Input the state variable s into the deep Q network, and output M Q values ​​corresponding to different carriers. Then, calculate the action value a in the Markov decision process according to the following formula (2): a = {m, m ∈ M} (2) Where m represents the m-th Q-value; S23: Define the reward r in the Markov decision process as the judgment index of the optimization objective. Then, construct the reward function according to the following formula (3), that is, the increment ΔF of the optimization objective, and use it to judge the increase or decrease of the reward r: ΔF=F t+1 -F t (3) Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n ; S3: Construct a deep Q-network, and train and update the deep Q-network according to the reward function; In step S3, a deep Q-network is constructed, and the deep Q-network is trained and updated according to the reward function, including: S31: Based on the obtained state s and action value a, configure the action-value network, i.e., reconfigure the deep Q network; S32: Construct the deep Q-network using a convolutional neural network, and perform reinforcement learning on the deep Q-network according to the reward function; S33: Calculate the error function, and use the calculation result to back-train and update the deep Q network after reinforcement learning through the stochastic gradient descent (SGD) method to correct the network parameters; In step S33, the error function is calculated, and the calculation result is used to perform reverse training and update on the trained deep Q-network using the stochastic gradient descent (SGD) method to correct the network parameters, including: S331: Calculate the error function L(θ) according to the following formula (4): L(θ)=E[y i -Q(s,a;θ)] (4) Among them, y i This represents the target Q-value that the deep Q-network aims to achieve. S332: The calculation results are used to back-train and update the trained deep Q-network using the stochastic gradient descent (SGD) method. After a certain number of steps, the deep Q-network is copied to the target Q-network, and the parameters θ of the target Q-network are updated using the parameters of the Q-network. - , that is, θ - =θ; S4: Receive user request status, output final resource strategy, and complete intelligent resource allocation in low-Earth orbit satellite communication.

2. The method according to claim 1, characterized in that, In step S1, the parameters of the satellite communication system are initialized, including: The initial total number of users is U set ={u|u=1,2,3…,U}; Initialize the low-orbit satellite state-action value function Q(s,a); where s represents the state and a represents the action; Set the discount coefficient λ, set the depth Q network parameters, and set the target depth Q network parameters.

3. The method according to claim 2, characterized in that, In step S1, receiving the user's access request includes: User u t When time t arrives and an access request is sent, if there is an idle carrier in the beam the user is trying to access, the user is allowed to access; otherwise, the user waits and proceeds to the next time step until successful access is achieved.

4. The method according to claim 3, characterized in that, In step S23, constructing the increment ΔF of the optimization objective includes: S231: Calculate the total power P of the satellite communication system according to the following formula (3-1): Where p0 is the power loss of the satellite platform itself, p n This represents the power of the nth beam; S232: Calculate the energy efficiency EE of the satellite communication system under ideal conditions according to the following formula (3-2): Among them, R i This represents the maximum speed of user i; S234: Calculate the spectral efficiency SE of the satellite communication system according to the following formula (3-3): Where W represents the total bandwidth of the carrier; S235: Based on the spectral efficiency SE, calculate the optimization objective F of the satellite communication system as shown in the following formula (3-4): F=ω1a1ω+ω2a2SE (3-4) Where a1 and a2 represent the normalization parameters in the formula, i.e., a1+a2=1, and ω1 and ω2 represent the weight parameters of the satellite communication system; according to the optimization objective F, the increment of the optimization objective is obtained as follows (3): ΔF=F t+1 -F t (3) Among them, F t+1 F represents the optimization objective at time t+1; t Let represent the optimization objective at time t; when the increment of the optimization objective is positive, it indicates that the reward r of the satellite communication system has increased, and therefore positive feedback r is given to the satellite communication system. n When the increment of the optimization objective is negative, it indicates that the reward r of the satellite communication system has decreased, and therefore negative feedback r is given to the satellite communication system. n .

5. The method according to claim 4, characterized in that, Step S33 further includes: using an experience replay mechanism, establishing an experience pool during the reinforcement learning process, putting data into the experience pool, and randomly selecting data from the experience pool to perform reverse training and update of the neural network when the data in the experience pool reaches a certain quantity.

6. The method according to claim 1, characterized in that, In step S4, the user request status is received, the final resource strategy is output, and intelligent resource allocation in low-Earth orbit satellite communication is completed, including: S41: Receive user request status. If the user leaves the system, delete the carrier used during the online time and enter the next status time. If the user has not left the system, continue to perform resource allocation operation. S42: Outputs the final strategy, resource allocation results, and optimization results, completing intelligent resource allocation in low-orbit satellite communication.

7. An intelligent resource allocation device for low-Earth orbit satellite communication, characterized in that, The apparatus is suitable for the method according to any one of claims 1-6, and the apparatus comprises: The initialization module is used to initialize the parameters in the satellite communication system and receive user access requests. An optimization target construction module is used to construct the resource allocation optimization problem in the satellite communication system into a Markov decision process to obtain the reward function; The training and update module is used to construct a deep Q-network and train and update the deep Q-network according to the reward function. The resource allocation module is used to receive user request status, output the final resource strategy, and complete intelligent resource allocation in low-Earth orbit satellite communication.

Images