Dynamic resource allocation method for low-orbit satellites

By subdividing communication time and calculating weighted energy efficiency in satellite communication systems, and optimizing resource allocation with deep reinforcement learning algorithm (Dueling DQN), the problem of resource allocation mismatch in multi-beam satellite systems is solved, and system efficiency and performance are improved. Ensure a stable training process.

CN119010984APending Publication Date: 2024-11-22CHONGQING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410951652.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In traditional satellite communication systems, the multi-beam satellite resource allocation method has the problem of mismatch between terminal requirements and allocated capacity, resulting in a decline in user service quality and network performance, which reduces overall efficiency and performance.

Method used

Using the dynamic resource allocation method of low-orbit satellites, the communication time is subdivided into time slots by initializing the multi-beam satellite scene model, and the user's transmission power and signal-to-noise interference ratio in the beam are calculated, and the weighted sum of energy efficiency and spectral efficiency is based on the weighting and conversion of energy efficiency and spectrum efficiency. For a single-objective optimization problem, a deep reinforcement learning algorithm (Dueling DQN) is used to solve the optimal resource allocation strategy.

Benefits of technology

The relative balance between energy efficiency and spectral efficiency is achieved, the overall efficiency, performance, stability and reliability of satellite communication systems are improved, and the unstable training process caused by multi-objective optimization in traditional methods is avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119010984A_ABST
    Figure CN119010984A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of satellite communication, and particularly relates to a low-orbit satellite dynamic resource allocation method, which comprises the following steps of: solving a weighted sum of energy efficiency EE and spectral efficiency SE, and converting a multi-target optimization problem into a single-target optimization problem according to the weighted sum of the energy efficiency EE and the spectral efficiency SE; and solving a single target optimization problem through deep reinforcement learning to obtain an optimal resource allocation strategy. According to the method, compromise operation is carried out on the EE and the SE, when maximum energy efficiency and spectrum efficiency cannot be achieved at the same time, relative balance can be achieved by solving the weighted sum of the EE and the SE, the multi-target optimization problem can be converted into the single-target optimization problem, and the overall efficiency and performance of a satellite communication system are greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of satellite communications, and in particular relates to a method for dynamic resource allocation of a low-orbit satellite. Background Art

[0002] In today's digital age, the development of communication technology plays a vital role in all areas of human society. Satellite communication, as a widely used communication method, plays an irreplaceable role in providing global coverage and strong connectivity. Especially in the fields of ocean, aviation, and emergency communications, satellite communication can achieve seamless coverage and effective connection, solving the communication needs in areas and environments that traditional ground networks cannot cover. However, with the continuous development of communication technology and the diversification of application scenarios, satellite communication systems are facing more and more challenges. Traditional resource allocation methods often have some defects, especially in multi-beam satellite systems. In these systems, due to the uneven distribution of satellite terminals, the business requirements between different beams vary significantly. Traditional resource allocation methods usually adopt fixed allocation schemes, but this scheme often leads to a mismatch between terminal demand and allocated capacity. For example, some beams may observe demand that exceeds the available beam capacity, while other beams may have idle capacity. This mismatch affects user quality of service (QoS) and satellite network performance, reducing the overall efficiency and performance of satellite communication systems. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a low-orbit satellite dynamic resource allocation method, comprising:

[0004] S1: Initialize the low-orbit multi-beam satellite scene model;

[0005] S2: In the low-orbit multi-beam satellite scenario model, the satellite's communication mobility time is subdivided into multiple time slots, and the satellite's channel parameters do not change in each time slot. Each beam can use N orthogonal subcarriers, and each subcarrier can only be used by a single user. When a beam is accessed by a user at time t, the beam is activated in the form of transmission power P, and each user can connect to a beam at time t.

[0006] S3: Calculate the user's transmission power on the subcarriers in the beam and the link signal-to-noise-interference ratio SINR;

[0007] S4: Calculate the throughput when the user is connected to the subcarrier according to the transmission power and the link signal-to-noise-interference ratio SINR, and calculate the overall throughput of the low-orbit multi-beam satellite scenario model based on the throughput when the user is connected to the subcarrier;

[0008] S5: Calculate the energy efficiency EE and spectrum efficiency SE of the low-orbit multi-beam satellite scenario model based on the transmission power and link signal-to-noise-interference ratio SINR;

[0009] S6: by calculating the weighted sum of energy efficiency EE and spectrum efficiency SE, the multi-objective optimization problem is converted into a single-objective optimization problem according to the weighted sum of energy efficiency EE and spectrum efficiency SE;

[0010] S7: Define a deep reinforcement learning algorithm to solve the single-objective optimization problem to obtain the optimal resource allocation strategy.

[0011] Beneficial effects of the present invention:

[0012] The present invention achieves a compromise operation on EE and SE. When it is impossible to simultaneously achieve maximum energy efficiency and spectrum efficiency, a relative balance can be achieved by calculating the weighted sum of the two, and the multi-objective optimization problem can be converted into a single-objective optimization problem, thereby greatly improving the overall efficiency and performance of the satellite communication system.

[0013] The present invention can alleviate the problem of over-estimation by defining a deep learning algorithm (Dueling DQN) to decompose the Q value into two parts: state value and action advantage. This operation can learn the Q value more stably and improve the reliability of training. Traditional DQN is easily affected by variance when learning the Q values ​​of different actions, which may lead to an unstable training process. The state value part of Dueling DQN is easier to learn, thus helping to improve the stability of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A low-orbit multi-beam satellite model diagram of a low-orbit satellite dynamic resource allocation method of the present invention;

[0015] Figure 2 This is a Dueling DQN satellite resource allocation framework diagram of a low-orbit satellite dynamic resource allocation method of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The present invention provides a low-orbit satellite dynamic resource allocation method, comprising:

[0018] S1: Initialize the low-orbit multi-beam satellite scene model;

[0019] S2: In the low-orbit multi-beam satellite scenario model, the satellite's communication mobility time is subdivided into multiple time slots, and the satellite's channel parameters do not change in each time slot. Each beam can use N orthogonal subcarriers, and each subcarrier can only be used by a single user. When a beam is accessed by a user at time t, the beam is activated in the form of transmission power P, and each user can connect to a beam at time t.

[0020] S3: Calculate the user's transmission power on the subcarriers in the beam and the link signal-to-noise-interference ratio SINR;

[0021] S4: Calculate the throughput when the user is connected to the subcarrier according to the transmission power and the link signal-to-noise-interference ratio SINR, and calculate the overall throughput of the low-orbit multi-beam satellite scenario model based on the throughput when the user is connected to the subcarrier;

[0022] S5: Calculate the energy efficiency EE and spectrum efficiency SE of the low-orbit multi-beam satellite scenario model based on the transmission power and link signal-to-noise-interference ratio SINR;

[0023] S6: by calculating the weighted sum of energy efficiency EE and spectrum efficiency SE, the multi-objective optimization problem is converted into a single-objective optimization problem according to the weighted sum of energy efficiency EE and spectrum efficiency SE;

[0024] S7: Define a deep reinforcement learning algorithm to solve the single-objective optimization problem to obtain the optimal resource allocation strategy.

[0025] The low-orbit multi-beam satellite scenario considered by the present invention is as follows: Figure 1 As shown in Figure 1, in order to improve network coverage performance, LEO satellites can generate B beams, which are represented by B = {b|b = 1, 2, ..., B}. The user set is defined as U = {u|u = 1, 2, ..., U}, where users are randomly distributed within the coverage of the beam. The number of users accessing beam b in time slot t is expressed as is the number of users accessing beam b at time t, and U is the number of users covered by all beams, that is, the total number of users accessing. In order to simplify the model complexity without reducing the model effectiveness, the present invention chooses to adopt the multi-beam scenario in OFDM mode. In this mode, different beams will cause different degrees of co-channel interference, but users in the same beam will not interfere with each other.

[0026] It is assumed that the information control center carried by the satellite can collect beam information including transmission power and link signal-to-noise-interference ratio (SNIR). Each beam can use N orthogonal subcarriers, and each subcarrier can only be used by a single user. When a beam is accessed by a user at time t, the beam is activated in the form of transmission power P, and each user can be connected to a beam at time t. The free space loss model is used as the channel model of the present invention.

[0027] In the model considered in the present invention, the satellite's communication mobility time is subdivided into multiple time slots, and the satellite's channel parameters do not change in each time slot. In the tth time slot, the channel gain of the nth subcarrier in beam b for the signal experienced by user u is:

[0028]

[0029] G b,u represents the beam gain between user u and beam b, and its expression is:

[0030]

[0031] G max is the maximum gain of the receiving antenna, u=2.70123sinθ b,u / sinθ 3db ,θ 3db The angle corresponding to the 3dB beamwidth, θ b,u represents the angle between the beam center and the user terminal, J1 and J3 represent the first-order and third-order Bessel functions of the first kind, respectively. The expression of L is:

[0032] L=4πd / λ 2

[0033] L represents the propagation loss in vacuum. Free space propagation loss refers to the propagation loss caused by the energy diffusion phenomenon due to the increase of propagation distance. The loss is measured in db, π is pi, d is the propagation distance, and λ is the wavelength. The transmission power in the nth subcarrier in the bth beam is expressed as In other words, user u is in the bth ′ The transmission power of the nth subcarrier in a beam is The useful signal power of the nth subcarrier in the bth beam is:

[0034]

[0035] The interference signal power of the nth subcarrier in beams other than beam b is:

[0036]

[0037] The SINR of the nth subcarrier of user u in beam b is expressed as

[0038]

[0039] The present invention adopts full frequency reuse, mainly to deal with the interference caused by other beams. is a binary variable indicating whether terminal device u is connected to beam b at time slot t:

[0040]

[0041] α is the variance of the additive white Gaussian noise. At time t, the throughput of the user connected to the subcarrier is expressed as follows:

[0042]

[0043] Therefore, the throughput of the entire system is:

[0044]

[0045] The present invention defines the energy efficiency (EE) in the LEO system as the ratio of the throughput to the power consumption of the entire system. In time slot t, EE is expressed as:

[0046]

[0047] The power consumption p sum =p a +p c , p a is the transmission power of the satellite system, p c Expressed as constant circuit power consumption. The spectrum efficiency (SE) in the LEO system is defined as the sum of the rates of the entire system, that is:

[0048]

[0049] When the spectrum efficiency is optimized unilaterally in the LEO system, the energy efficiency will decrease. The two cannot increase at the same time, and they may even conflict with each other. On the contrary, increasing the energy efficiency may lead to a decrease in the spectrum efficiency. Therefore, a compromise between EE and SE can change this situation to a certain extent. When it is impossible to maximize energy efficiency and spectrum efficiency at the same time, finding the weighted sum of the two can achieve a relative balance and transform the multi-objective optimization problem into a single-objective optimization problem.

[0050] Since the units of energy efficiency and spectrum efficiency are bits / Joule and bits / s / Hz respectively, it is not advisable to directly add energy efficiency and spectrum efficiency. Therefore, the ω weight factor is used, B / P sum Units used to identify two items:

[0051]

[0052] The optimization problem can be formulated as follows:

[0053]

[0054] Among them, P max is the maximum transmission power of the satellite. The sum of the total power of each beam cannot be greater than the maximum power of the satellite. min is the minimum transmission power of the subcarrier.

[0055] In the present invention, in addition to EE, SE is also comprehensively considered, so the resource allocation problem becomes an NP-hard problem and it is difficult to get the optimal solution. In order to balance the EE and SE of the system, the resource allocation problem in the LEO scenario can be expressed as MDP. The present invention adopts a deep reinforcement learning method to solve this problem.

[0056] Reinforcement learning regards learning as a heuristic evaluation process. When facing the environment, the agent needs to choose an action. After this action is received by the environment, it will cause a change in state and receive reward feedback from the environment. Based on this reward feedback and the current state of the environment, the agent will choose the next action. The principle is to increase the probability of reward in order to obtain the optimal strategy. Therefore, in reinforcement learning, environment, state and reward are three crucial elements. The present invention defines the state as: Among them, M b represents the number of users in the beam, Indicates the power level of the beam;

[0057] The power level is divided into a finite number of discrete levels, and the discrete rules are defined as: when When, among them, p0=p max , p Y =0; when Y is larger, the calculation error is smaller and the level division is finer, where Y represents power dispersion, Indicates the current power level. Indicates the power corresponding to the current power level, N indicates the number of subcarriers, represents the transmission power of the nth subcarrier in the bth beam to user u, p Y Indicates the power corresponding to the maximum discreteness, P max Indicates the maximum transmission power of the satellite;

[0058] The action space is defined as: Among them, n b represents the nth subcarrier of the bth beam, Indicates the power adjustment value of the subcarrier, indicating the bth action selection, a 3N Indicates the 3Nth action selection;

[0059] There are N subcarriers in total. The power adjustment of each subcarrier includes three options: unchanged, increase, and decrease. Therefore, there are 3N action options in total. When the total capacity of users served by a subcarrier increases, When the total capacity decreases, If the overall capacity does not change, then The reward is a feedback signal that guides the agent to choose the best action based on the results of its choice. Given the η(t) maximization problem defined in this invention, such as Figure 2 As shown, Dueling DQN consists of a main network and a target network. The main network is used to perform actual power adjustments, while the target network is used to estimate the target Q value and help stabilize the training process. By obtaining state information feedback from the satellite communication environment, the observed state information is input into Dueling DQN for training the learning model. In Dueling DQN, the Q function is usually decomposed into a state function and an advantage function. This approach allows the network to simultaneously estimate the value of the state and the advantage of each action, enhancing our understanding of how each action affects the value of the state. This can be expressed as follows:

[0060] Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α)

[0061] Where V(s; θ, β) is the state function, which outputs a scalar value. A(s, a; θ, α) is the advantage function, which outputs a vector with a length equal to the size of the action space. θ represents the parameters of the network convolutional layer, and α and β are the parameters of the fully connected layers of the two branches.

[0062] However, Q(s,a;θ,α,β) is only an approximation of the exact Q function. When an exact Q function is given, V(s;θ,β) and A(s,a;θ,α) are not unique. In other words, when a constant is added to the function V and subtracted from the function A, the same Q value is still obtained. To prevent this from happening, Dueling-DQN forces the sum of the output vectors of the advantage function to be set to zero. Therefore, the q function can be expressed as:

[0063] Q(s,a;θ,α,β)=V(s;θ,β)+(A(s,a;θ,α)-argmax A(s,a′;θ,α))

[0064] In order to create a simple replacement module and simplify the architecture, an average approach is used to replace the maximum operation. That is:

[0065]

[0066] Initially, we initialize the parameters α, β, θ, and specify the size of the experience replay buffer. The optimization process consists of multiple training episodes, each of which consists of a series of time steps. At each time step, we observe the state of the environment from the satellite, choose an action based on the ε-greedy policy, obtain the global reward, and send the four-tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer ψ. We then extract small batches of experience from the experience replay buffer and use the Adam optimization method to calculate the loss and update (s t ,a t ,r t ,s t+1 )New Q value network. To maintain stability, the parameters of the target network need to be updated regularly. As training progresses, the exploration probability ε is gradually reduced to better utilize the learned strategy.

[0067] In order to verify the performance of the proposed method, the present invention simulates the proposed method in Python 3.9, torch 1.12.1 version. The system simulation parameters and Dueling DQN parameter settings are shown in Table 1:

[0068] Table 1 System simulation parameters

[0069]

[0070]

[0071] Traditional DQN algorithms have the problem of overestimation when estimating Q values, that is, they overestimate the value of certain actions. Dueling DQN can alleviate the problem of overestimation by decomposing Q values ​​into two parts: state value and action advantage. This operation can learn Q values ​​more stably and improve the reliability of training. Traditional DQN is susceptible to variance when learning Q values ​​for different actions, which can lead to an unstable training process. Dueling DQN's state value part is easier to learn, so it helps to improve the stability of training.

[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic resource allocation of low-orbit satellites, characterized in that: include: S1: Initialize the low-orbit multi-beam satellite scene model; S2: In the low-orbit multi-beam satellite scenario model, the satellite's communication mobility time is subdivided into multiple time slots, and the satellite's channel parameters do not change in each time slot. Each beam can use N orthogonal subcarriers, and each subcarrier can only be used by a single user. When a beam is accessed by a user at time t, the beam is activated in the form of transmission power P, and each user can connect to a beam at time t. S3: Calculate the user's transmission power on the subcarriers in the beam and the link signal-to-noise-interference ratio SINR; S4: Calculate the throughput when the user is connected to the subcarrier according to the transmission power and the link signal-to-noise-interference ratio SINR, and calculate the overall throughput of the low-orbit multi-beam satellite scenario model based on the throughput when the user is connected to the subcarrier; S5: Calculate the energy efficiency EE and spectrum efficiency SE of the low-orbit multi-beam satellite scenario model based on the transmission power and link signal-to-noise-interference ratio SINR; S6: by calculating the weighted sum of energy efficiency EE and spectrum efficiency SE, the multi-objective optimization problem is converted into a single-objective optimization problem according to the weighted sum of energy efficiency EE and spectrum efficiency SE; S7: Define a deep reinforcement learning algorithm to solve the single-objective optimization problem to obtain the optimal resource allocation strategy.

2. A low-orbit satellite dynamic resource allocation method according to claim 1, characterized in that: Initialize the low-orbit multi-beam satellite scenario, including: The beam set generated by the LEO satellite through the OFDM mode is denoted as B = {b|b = 1, 2, ...}, and the user set is defined as U = {u|u = 1, 2, ...}, where users are randomly distributed within the coverage of the beam. The number of users accessing beam b in time slot t is expressed as represents the number of users accessing beam b at time t; In the OFDM mode, different beams will cause co-channel interference to varying degrees, but users in the same beam will not interfere with each other.

3. A low-orbit satellite dynamic resource allocation method according to claim 1, characterized in that: Calculate the user's transmission power on the subcarriers in the beam and the link signal-to-noise-interference ratio SINR, including: in, represents the SINR of the nth subcarrier of user u in beam b in time slot t, A binary variable indicating whether user u is connected to beam b at time slot t, represent the bth and bth ′ The channel gain of the nth subcarrier in the beam for user u is, represent the bth and bth ′ The transmission power of the nth subcarrier in the beam to user u, τ 2 represents the variance of additive white Gaussian noise.

4. A method for dynamic resource allocation of low-orbit satellites according to claim 1, characterized in that: The throughput when the user is connected to the subcarrier is calculated according to the transmission power and the link signal-to-noise-interference ratio SINR, and the overall throughput of the low-orbit multi-beam satellite scenario model is calculated based on the throughput when the user is connected to the subcarrier, including: Throughput when users are connected to subcarriers: Overall throughput of the low-orbit multi-beam satellite scenario model in, represents the throughput when the user is connected to the nth subcarrier in beam b in time slot t, B represents the beam set, N represents the number of subcarriers, and U represents the user set. represents the SINR of the nth subcarrier of user u in beam b in time slot t; R t Indicates the overall throughput.

5. A low-orbit satellite dynamic resource allocation method according to claim 1, characterized in that: The energy efficiency EE and spectrum efficiency SE of the low-orbit multi-beam satellite scenario model are calculated based on the transmission power and link signal-to-noise-interference ratio SINR, including: Energy Efficiency EE: Spectral efficiency SE: in, represents the energy efficiency EE in time slot t, represents the spectrum efficiency SE in time slot t, R t represents the overall throughput, O sum Represents the power consumption of the entire satellite system.

6. A method for dynamic resource allocation of low-orbit satellites according to claim 1, characterized in that: Find the weighted sum of energy efficiency EE and spectrum efficiency SE, including: Where η(t) represents the weighted sum of energy efficiency EE and spectrum efficiency SE in time slot t, represents the energy efficiency EE in time slot t, represents the spectrum efficiency SE in time slot t, B represents the beam set, ω represents the weight factor, P sum Represents the power consumption of the entire satellite system.

7. A method for dynamic resource allocation of low-orbit satellites according to claim 1, characterized in that: According to the weighted sum of energy efficiency EE and spectrum efficiency SE, the multi-objective optimization problem is transformed into a single-objective optimization problem, including: Where η(t) represents the weighted sum of energy efficiency EE and spectrum efficiency SE in time slot t, represents the transmission power of the nth subcarrier in the bth beam to user u, p min Indicates the minimum transmission power of the subcarrier, P max represents the maximum transmission power of the satellite, Represents any subcarrier and any beam.

8. A method for dynamic resource allocation of low-orbit satellites according to claim 1, characterized in that: By defining a deep reinforcement learning algorithm to solve a single-objective optimization problem, we can obtain the optimal resource allocation strategy, including: By defining the number of users and the power level of the beam as the state space of deep reinforcement learning, the increase or decrease of power is defined as the action space; The deep reinforcement learning algorithm includes: a main network and a target network, wherein the main network is used to perform actual power adjustment, and the target network is used to estimate the target Q value; Initialize the parameters of the neural network and specify the replay buffer size. The optimization process includes multiple training segments, each of which consists of a series of time steps. At each time step, the environment state is observed from the satellite, an action is selected based on the ε-greedy strategy, the global reward is obtained, and the quadruple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer ψ, small batches of experience are extracted from the experience replay buffer, and the Adam optimization method is used to calculate the loss and update the Q value network. In order to maintain stability, the parameters of the target network need to be updated at a certain frequency. As the training proceeds, the exploration probability ε gradually decreases to better utilize the learned strategy. The optimal resource allocation strategy is obtained through continuous strategy optimization iteration.

9. A method for dynamic resource allocation of low-orbit satellites according to claim 8, characterized in that: By defining the number of users and the power level of the beam as the state space of deep reinforcement learning, the increase or decrease of power is defined as the action space, including: Define the state as: Among them, M b represents the number of users in the beam, Indicates the power level of the beam; The power level is divided into a finite number of discrete levels, and the discrete rules are defined as: when When, among them, p0=p max , p Y =0; when Y is larger, the calculation error is smaller and the level division is finer, where Y represents power dispersion, Indicates the current power level. Indicates the power corresponding to the current power level, N indicates the number of subcarriers, represents the transmission power of the nth subcarrier in the bth beam to user u, p Y Indicates the power corresponding to the maximum discreteness, P max Indicates the maximum transmission power of the satellite; The action space is defined as: Among them, n b represents the nth subcarrier of the bth beam, Indicates the power adjustment value of the subcarrier, indicating the bth action selection, a 3N Indicates the 3Nth action selection; There are N subcarriers in total. The power adjustment of each subcarrier includes three options: unchanged, increase, and decrease. Therefore, there are 3N action options in total. When the total capacity of users served by a subcarrier increases, When the total capacity decreases, If the overall capacity does not change, then Rewards are feedback signals that guide the agent to choose the best action based on the consequences of its choices.

Citation Information

Cited By

  • Load balancing and resource allocation method based on satellite-ground cooperation

    CN122457122A