A low earth orbit satellite communication link optimization method based on deep reinforcement learning

Abstract

The application relates to the technical field of satellite communication, and particularly discloses a low-orbit satellite communication link optimization method based on deep reinforcement learning. The method is aimed at the high-dynamic and resource-limited communication environment of a low-orbit satellite network, a multi-dimensional state space containing channel state information, an interference map, queue state and historical performance indexes is constructed, a hybrid deep reinforcement learning architecture combining near-end strategy optimization and a deep Q network is innovatively designed, continuous adjustment of transmitting power and beamforming weight and discrete decision of modulation and coding scheme and retransmission mechanism are respectively processed, and joint closed-loop optimization of physical layer and link layer parameters is realized; simulation results show that, compared with traditional fixed strategies and adaptive modulation and coding technology, the method can intelligently balance multiple targets such as throughput, bit error rate, time delay and energy consumption under rapidly changing channel conditions, realizes significant performance gains such as more than 28.5% improvement in throughput, 93.9% reduction in bit error rate and 28.3% saving in transmitting power, and provides an effective adaptive optimization solution for constructing an efficient and reliable low-orbit satellite communication system.

Description

Technical Field

[0001] This invention belongs to the field of satellite communication, specifically relating to a method for optimizing low-Earth orbit satellite communication links based on deep reinforcement learning. Background Technology

[0002] Low Earth Orbit (LEO) satellite networks (such as Starlink, OneWeb, StarNet, and Qianfan) are characterized by highly dynamic topologies, prolonged satellite-to-ground propagation time, and limited resources. Under dynamic channel conditions, limited spectrum resources, and complex interference environments, satellite communication links urgently require intelligent parameter adjustment strategies to improve communication system performance. Therefore, researching intelligent optimization methods for LEO satellite communication links is of great significance.

[0003] Traditional optimization methods for satellite communication links are mostly based on static models or empirical rules, such as fixed modulation and coding schemes (MCS) (see reference [1]) and heuristic power control (see reference [2]), which are difficult to adapt to the complex satellite communication environment that changes in real time. Thanks to the development of artificial intelligence, the application of intelligent methods such as deep reinforcement learning in the field of wireless communication has gradually become a research hotspot, see reference [3]; for example, Udeme C et al. used reinforcement learning for dynamic spectrum allocation, which can effectively predict spectrum changes and determine the optimal transmission frequency, see reference [4]. Hongguang Li et al. combined DLSTM to process time-varying channels and established a new CQI prediction model. This reduced the communication bit error rate, see reference [5].

[0004] However, existing research still has shortcomings. First, there is insufficient support for hybrid action space, as satellite links require the joint implementation of discrete and continuous actions, and existing methods are mostly limited to a single type. Second, the granularity of channel representation is coarse, making it difficult for traditional fully connected networks to capture the spatiotemporal correlation of CSI. Finally, energy efficiency optimization is lacking, with most studies failing to consider multi-objective trade-offs under power consumption constraints. Therefore, existing methods have many limitations in practical applications and their applicability needs further improvement. Summary of the Invention

[0005] This invention proposes a deep reinforcement learning-based method for optimizing low-Earth orbit (LEO) satellite communication links, aiming to solve the link adaptation challenge in the highly dynamic and resource-constrained environment of LEO satellite networks. By jointly optimizing transmit power, modulation and coding scheme (MCS), beamforming parameters, and retransmission mechanism, link adaptation in dynamic environments is achieved, effectively improving the throughput of satellite communication links and reducing the communication error rate. This provides a new approach for efficient resource management and dynamic environment adaptation in satellite communication systems.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for optimizing low-Earth orbit satellite communication links based on deep reinforcement learning, the specific steps of which are as follows:

[0008] Step 1: Consider the Doppler frequency shift and its time-varying characteristics caused by the high-speed motion of the satellite, and combine it with atmospheric effects to construct a channel dynamic scenario for low-orbit communication satellites in space transmission.

[0009] Step 2: Based on the link performance evaluation criteria and in conjunction with the performance indicators of the communication link, design a weighted comprehensive reward mechanism; among which, key performance indicators include throughput, bit error rate, transmission latency, and power consumption.

[0010] Step 3: Define the agent's action as the satellite transmitting signals to the ground according to the designed radio frequency parameters and modulation and coding scheme in the channel dynamic scenario constructed in Step 1; and design a deep reinforcement learning DRL model, with the environment state as the input and the corresponding agent action as the output; wherein, the DRL model includes a PPO branch and a DQN branch, the PPO branch is used to adjust the transmit power and beamforming weights, and the DQN branch is used to select the modulation and coding scheme and the number of retransmissions;

[0011] Step 4: The DRL model is trained offline using a large number of randomly constructed channel scenarios. During the training process, the agent obtains the state from the environment, outputs actions through the DRL model, and then the environment executes the actions to calculate the new state and reward, update the DRL model, and optimize the low-Earth orbit satellite communication link using the output of the DRL model in the corresponding design scenario.

[0012] Furthermore, the performance metrics of the communication link in step 2 include:

[0013] (1) Throughput norm :

[0014] Use effective throughput, i.e., the rate of successfully received bits:

[0015]

[0016] In the formula, For channel bandwidth, Here, BLER is the modulation order, and BLER is the block error rate, which is the probability that a data block is received incorrectly. This represents the system's maximum theoretical throughput.

[0017] (2) Bit error rate (BER):

[0018] Theoretical BER for QPSK modulation:

[0019]

[0020] For M-QAM modulation:

[0021]

[0022] Actual simulation BER:

[0023]

[0024] In the formula, Bit signal-to-noise ratio (SNR) is the ultimate representation of the link budget. and These are the complementary error function and the Gaussian Q function, respectively.

[0025] (3) Total transmission delay include:

[0026]

[0027] In the formula, The propagation delay is determined by the distance between the satellite and the ground. ,in This is the instantaneous distance between Earth and space. The speed of light; Transmission delay, related to data packet length and symbol rate related, ; To handle latency, including encoding, modulation, demodulation, and decoding time, it can be modeled as a fixed value or a random distribution; Queuing delay, which occurs in on-board routers when multiple data streams compete for the same output port, is related to traffic load and scheduling algorithm, and can be estimated using M / M / 1 or queuing theory models.

[0028] (4) Power efficiency P consumed :

[0029]

[0030] In the formula, For transmission power, For power amplifier efficiency, This represents the actual power consumption of the power amplifier. This refers to static power consumption;

[0031] By employing a linear weighted sum method, the above multiple indicators are merged into a single scalar reward value, transforming the multi-objective optimization problem into a single-objective problem:

[0032]

[0033] In the formula, In time step The instant reward received; The normalized throughput metric, i.e. ; The bit error rate metric after penalty, i.e. The aim is to amplify the impact of BER and give it a reasonable weight in the reward function; The normalized latency metric, i.e. D max This is the maximum latency that the system can tolerate; The normalized power consumption metric, i.e. ,in This is the maximum permissible transmission power; These are the weighting coefficients.

[0034] Furthermore, step 3 specifically includes the following processes:

[0035] (1) The actions of the agent Defined as a composite action containing continuous and discrete components: ;

[0036] For continuous action space Used for fine-tuning radio frequency parameters, defined as:

[0037]

[0038] In the formula, The transmit power adjustment is a normalized continuous value, typically ranging from [−1, 1], where P is the actual transmit power. t We obtain this through linear mapping:

[0039]

[0040] In the formula, For beamforming weights, i.e., phase offset, This is the maximum permissible transmission power; It is the minimum permissible transmission power; This refers to the number of elements in a phased array antenna.

[0041] For discrete action space Used to make category selection decisions, defined as:

[0042]

[0043] Wherein, MCS Index is the modulation and coding scheme index, which is a discrete value. The corresponding index and modulation and coding relationship can be set. Each index corresponds to a predefined combination of modulation order and coding rate; Retry Count is the maximum number of retransmissions;

[0044] (2) Design a deep reinforcement learning (DRL) model, including a feature extraction network, a PPO branch, and a DQN branch, with the original state as the input. ;

[0045] The feature extraction network uses a CNN-LSTM hybrid structure, with the CNN part used to extract the original state. The local features in the image contain spatial structure information about the state; the LSTM part is used to capture the original state. Long-term dependencies in time series; extracted feature vectors Simultaneously transmitted to both the PPO branch and the DQN branch;

[0046] The PPO branch contains an Actor network and a Critic network:

[0047] Actor Network The input is the feature vector. The output is a continuous action. The mean and variance of are defined in the state The probability distribution of taking continuous actions is determined; the goal is to learn the optimal continuous control strategy.

[0048] Critic Network The input is the feature vector. The output is the state value. That is, from the state Initially, it represents the expected cumulative reward that can be obtained in the future; the goal is to evaluate the quality of the current state to guide the updating of the Actor network.

[0049] The PPO branch ensures the stability of policy updates through its pruned objective function:

[0050]

[0051] in, This represents the probability ratio between the old and new strategies. The advantage function estimate represents the superiority or inferiority of a particular action relative to the average level, and is calculated from the Critic network and the actual reward. , It is a hyperparameter; In time Instant rewards received This represents the state value output by the Critic network;

[0052] The DQN branch contains a Q network. It is used to handle discrete decision problems, with the input being a feature vector. The output is a Q-value vector, where each element corresponds to a discrete action combination. The Q-value is the expected long-term cumulative reward; the decision is to select the discrete action with the largest Q-value using a greedy or ε-greedy strategy:

[0053]

[0054] In the formula, It represents a discrete action chosen randomly. Represents the current state The action with the highest Q value;

[0055] The DQN branch updates the network using temporal difference error, with the goal of minimizing the following loss function:

[0056]

[0057]

[0058] in, That is, the target Q value; Indicates the discount factor. ; That is, the target Q-network, its parameters Regularly from the main network This was copied and is used for stable training. To predict the Q value; Represents the set of all weights and biases in a deep Q-network; Indicates the next state Below, all possible actions The highest Q value.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] This invention presents a method for optimizing low-Earth orbit satellite communication links based on deep reinforcement learning. It adopts a dual-model architecture of near-end policy optimization (PPO) and deep Q-network (DQN) to handle continuous and discrete action spaces respectively, and combines convolutional neural networks (CNN) and long short-term memory networks (LSTM) to extract spatiotemporal channel features, thereby achieving optimization of low-Earth orbit satellite communication links. It has the advantages of high throughput, low bit error rate, and strong adaptability. Attached Figure Description

[0061] Figure 1 This is a flowchart of a low-Earth orbit satellite communication link optimization method based on deep reinforcement learning according to the present invention.

[0062] Figure 2 This is a diagram of the channel dynamic characteristics model of the communication link of the present invention.

[0063] Figure 3 This is a schematic diagram illustrating the Doppler frequency offset principle of satellite communication carrier in this invention.

[0064] Figure 4 This is a diagram of the hybrid DRL algorithm architecture for satellite communication links in this invention. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings and examples.

[0066] This invention presents a low-Earth orbit satellite communication link optimization method based on deep reinforcement learning, abbreviated as DRL-OSCL. The core of this method lies in constructing a deep reinforcement learning agent with a hybrid action space to achieve joint closed-loop optimization of transmit power, modulation and coding scheme, beamforming parameters and retransmission strategy. This breaks through the bottleneck of suboptimal traditional modular optimization strategies and significantly improves link throughput, reliability and energy efficiency.

[0067] like Figure 1 As shown, the present invention includes the following steps:

[0068] Step 1: Establish a dynamic characteristic model of the communication link based on the channel characteristics of low-Earth orbit satellite communication.

[0069] The channel dynamics of low-Earth orbit communication satellites in space transmission are quite complex, such as... Figure 2 As shown. Based on electromagnetic wave propagation theory, the energy of an electromagnetic wave gradually disperses on the in-phase plane during propagation in space. This means that the energy loss of electromagnetic waves becomes more severe as the propagation distance increases. Generally, when calculating propagation loss in free space, only the direct path is considered. When the transmitting end uses an ideal point source antenna, the signal power will be uniformly distributed within a spherical region, and the received signal power can be expressed as:

[0070]

[0071] in, Indicates signal transmission power. Indicates the signal received power. Indicates the antenna gain at the transmitting end. Indicates the antenna gain at the receiving end. For the operating wavelength, Let be the propagation distance. When the gain of both the transmitting and receiving antennas is 1, the free space loss is as shown in the following formula:

[0072]

[0073] Doppler frequency shift is related to the carrier frequency, satellite speed, and user movement speed. The rate of change of Doppler frequency shift describes how quickly the Doppler frequency shift changes over time. Figure 3 As shown. In low-Earth orbit satellite communication systems, the satellite moves at a continuous high speed, and under the same conditions, the corresponding Doppler frequency offset and rate of change of Doppler frequency offset are also greater than those of ground systems. The formula for calculating Doppler frequency offset is as follows:

[0074]

[0075] in, It is the carrier frequency. It is the relative velocity between the satellite and the ground. It is the angle between the direction of movement of the ground terminal and the satellite-to-ground link. It's the speed of light. In scenarios where low-Earth orbit satellites move at high speeds, when the satellite approaches the ground station at its minimum access elevation angle, the velocity vector component along the satellite-to-ground link is at its maximum, and the corresponding Doppler frequency shift is also at its maximum. Conversely, when the satellite's elevation angle with the ground station is 90°, the velocity vector component along the satellite-to-ground link is zero, and the corresponding Doppler frequency shift is also zero. Typically, the Doppler frequency shift of the received signal can reach the order of tens to hundreds of kHz.

[0076] Due to the scattering and absorption of electromagnetic waves by rainfall, when satellite signals pass through a rainy area, some of the energy is scattered or absorbed by raindrops. Especially in communication frequency bands above 10 GHz, the impact of rainfall attenuation on communication quality cannot be ignored.

[0077] The latitude of the known ground station The corresponding rain layer height can be calculated. .

[0078]

[0079] When the satellite enters the rain zone, the angle between its tilt path and the ground is... slanted path length It can be represented as:

[0080]

[0081] Based on the length of the inclined path of the satellite signal through the rain area Determine its horizontal projection distance Size:

[0082]

[0083] Based on the length of the inclined path of the satellite signal through the rain area Determine its horizontal projection distance The average annual rainfall reduction exceeding 0.01% can be expressed as:

[0084]

[0085]

[0086]

[0087]

[0088] in, The decay rate per unit time; This indicates a high correction factor.

[0089] Gas absorption loss Primarily caused by oxygen and water vapor, this can be calculated using the ITU-RP.676-11 model and is related to frequency and elevation angle. In the 10-30 GHz band, the typical value is 0.1-1 dB.

[0090] Tropospheric scintillation loss Rapid fluctuations in signal amplitude caused by atmospheric turbulence can be estimated using the model in ITU-RP.618-10 Recommendation.

[0091] Ultimately, the channel can be modeled as a time-varying complex gain. For a flat fading channel:

[0092]

[0093] in, This represents the initial random phase shift; . Other losses can be used to add subsequent dynamic factors. It is a stochastic process characterizing small-scale fading, such as Ricean fading:

[0094]

[0095] here, It is a complex Gaussian random process. This is the Rice factor, representing the ratio of the power of the direct path component to the power of the multipath scattering component. In satellite communications, when the elevation angle is high, The factor is relatively large.

[0096] Construction of the CSI matrix: In actual simulation, the channel needs to be discretized and sampled. The channel state information of the low-Earth orbit network communication system is represented as a complex matrix. ,in: Number of receiving antennas. Number of transmitting antennas. : Number of subcarriers (for OFDM).

[0097] Each element in the matrix All follow the aforementioned integrated channel model, which includes information on path loss, shadowing, Doppler phase shift, and small-scale fading. This high-dimensional, time-varying CSI matrix is ​​precisely the core environmental state observed by the DRL agent.

[0098] Step 2: Establish a performance index model for the communication link based on the link performance evaluation criteria;

[0099] (1) Normalized throughput

[0100] Throughput is a core metric for measuring the efficiency of system data transmission. We use effective throughput, which is the rate of successfully received bits, rather than the theoretical peak value at the physical layer.

[0101]

[0102] in, Channel bandwidth, in Hz. : Modulation order (e.g., M=4 in QPSK, M=16 in 16QAM). BLER: Block error rate, the probability that a data block (e.g., a codeword) is received incorrectly. It is related to the bit error rate (BER), but reflects the actual transmission failure situation after forward error correction coding more accurately. The system's maximum theoretical throughput, used for normalization, for example... After normalization, the index ranges between [0,1], making it easier to integrate with other indicators.

[0103] (2) Bit Error Rate (BER)

[0104] Bit error rate (BER) is a fundamental indicator for measuring communication reliability. It is used to calculate the theoretical BER, and can also be used to calculate the actual BER using the Monte Carlo method.

[0105] Theoretical BER (for AWGN channel approximation), for QPSK modulation:

[0106]

[0107] For M-QAM modulation (M≥16):

[0108]

[0109] Actual simulation BER:

[0110]

[0111] in, Bit signal-to-noise ratio (SNR) is the ultimate reflection of the link budget and is related to received power, noise power, etc. and Complementary error function and Gaussian Q function. These are commonly used in reward functions. This amplifies the advantages of a low BER and prevents its value from being overwhelmed by other metrics due to its small size.

[0112] (3) Transmission delay

[0113] Latency is a key factor in evaluating the quality of service, especially crucial for applications with high real-time requirements. Total transmission latency Mainly includes:

[0114]

[0115] in, The propagation delay is determined by the distance between the satellite and the ground. ,in This is the instantaneous distance between Earth and space. This is the speed of light; for low-Earth orbit satellites, this value varies between a few milliseconds and tens of milliseconds. Transmission delay, and data packet length and symbol rate related, ; Processing latency, including encoding, modulation, demodulation, and decoding time, can be modeled as a fixed value or a random distribution; Queuing delay, in on-board routers, occurs when multiple data streams compete for the same output port. It is related to traffic load and scheduling algorithm and can be estimated using M / M / 1 or more complex queuing theory models.

[0116] (4) Power consumption efficiency

[0117] Power efficiency is directly related to the energy sustainability and lifespan of a satellite. We focus on the power consumption of the communication subsystem, especially the transmit power.

[0118]

[0119] in, Transmit power, in watts (W). This is a key variable that the DRL agent can directly optimize. Power amplifier efficiency. The efficiency of a typical spaceborne traveling wave tube amplifier or solid-state power amplifier is between 30% and 60%, i.e. ; This represents the actual power consumption of the power amplifier. Static power consumption. This includes the power required for the normal operation of baseband devices such as modems and digital signal processors.

[0120] The key to guiding the behavior of a DRL agent lies in integrating the above multiple indicators into a single scalar reward value. A linear weighted sum method is used to transform the multi-objective optimization problem into a single-objective problem.

[0121]

[0122] in, : at time step The instant reward received; Normalized throughput metric, i.e. ; The bit error rate metric after penalty, i.e. The goal is to amplify the impact of BER (Percentage of Return) and give it a reasonable weight in the reward function. Normalized latency metric, i.e. , where is the maximum latency that the system can tolerate; Normalized power consumption metrics, i.e. ,in This is the maximum permissible transmission power; Weighting coefficients. These coefficients determine the agent's preference for different performance metrics. Here, to achieve high throughput and high reliability, while also having certain constraints on latency and power consumption, the following configuration is adopted:

[0123] .

[0124] Step 3: Design the continuous and discrete action space parameters in reinforcement learning, and establish an algorithm architecture based on deep reinforcement learning (DRL);

[0125] (1) Design of mixed motion space

[0126] Actions of the intelligent agent It is defined as a composite action containing continuous and discrete components: .

[0127] For continuous action space Used for fine-tuning radio frequency parameters, defined as:

[0128]

[0129] in, Transmit power adjustment. This is a normalized continuous value, typically ranging from [−1, 1]. The actual transmit power is obtained through a linear mapping:

[0130]

[0131] Beamforming weights (phase offset). This is the maximum permissible transmission power; It is the minimum permissible transmission power; This represents the number of elements in a phased array antenna. For antennas with... The phased array antenna has individual elements. The intelligent agent directly outputs the phase adjustment amount (normalized) of each element to dynamically shape the beam, align it with the target user, and suppress interference.

[0132] For discrete action space Used to make category selection decisions, defined as:

[0133]

[0134] MCS Index: Modulation and Coding Scheme Index; this is a discrete value that allows setting corresponding indexes and modulation / coding relationships, such as 0-QPSK, coding rate 1 / 2; 1-16QAM, coding rate 3 / 4; 2-64QAM, coding rate 5 / 6. Each index corresponds to a predefined combination of modulation order and coding rate.

[0135] Retry Count: Maximum number of retransmissions. In protocols based on automatic retransmission requests, the agent can select the maximum number of retransmissions for link-layer packets, such as 0 (no retransmission), 1, or 2. This allows the agent to make trade-offs between latency and reliability.

[0136] (2) Hybrid DRL algorithm architecture: PPO+DQN

[0137] To address the problem of mixed action space, this invention designs a decomposed hybrid architecture, where PPO is responsible for handling continuous actions and DQN is responsible for handling discrete actions.

[0138] First, the original state (Including CSI matrix, historical BER, etc.) The features are processed through a shared feature extraction network, which employs a CNN-LSTM hybrid structure. The CNN part extracts local features with spatial structure, such as the CSI matrix. The LSTM part captures long-term dependencies in the time series, such as historical BER and queue states. The extracted feature vectors are then processed. It is transmitted simultaneously to both the PPO and DQN branches. For example... Figure 4 As shown.

[0139] Secondly, the PPO algorithm branch contains an Actor network and a Critic network.

[0140] Actor Network The input is a feature vector. Output is continuous action. The mean and variance. It is defined in the state. The probability distribution of taking consecutive actions is given. The goal is to learn the optimal continuous control strategy.

[0141] Critic Network The input is a feature vector. The output is the state value. That is, from the state Initially, it represents the expected cumulative reward to be obtained in the future. The goal is to evaluate the quality of the current state to guide the updates of the Actor network.

[0142] The PPO algorithm ensures the stability of policy updates through its pruning objective function:

[0143]

[0144] in, The probability ratio between the old and new strategies. The advantage function estimate represents the merits of a particular action relative to the average level. It is typically calculated using a Critic network and the actual reward. , It is a hyperparameter; In time Instant rewards received This represents the state value output by the Critic network.

[0145] The DQN branch contains a Q network. It is used to handle discrete decision problems. The input is a feature vector. The output is a Q-value vector, where each element corresponds to a discrete action combination. The Q-value (expected long-term cumulative reward). Decision-making involves selecting the discrete action that maximizes the Q-value using a greedy or ε-greedy strategy:

[0146]

[0147] In the formula, It represents a discrete action chosen randomly. Represents the current state The action with the highest Q value.

[0148] DQN updates the network using temporal difference error, with the goal of minimizing the following loss function:

[0149]

[0150]

[0151] in, That is, the target Q value; Indicates the discount factor. ; That is, the target Q-network, its parameters Regularly from the main network It was copied and is used for stable training. To predict the Q value; This represents the set of all weights and biases in a depth Q network. In time The instant reward received; Indicates the next state Below, all possible actions The highest Q value.

[0152] The final collaborative training and decision-making process is as follows:

[0153] a. Forward propagation: At each time step, the state... Feature vectors are obtained through a feature extraction network. .

[0154] b. Action generation: The PPO-Actor network samples a continuous action based on the current policy. The DQN-Q network selects the discrete action with the highest Q value. .

[0155] c. Environmental interaction: Executing compound actions Earn rewards from the environment (low-Earth orbit satellite communication link simulator). and the next state .

[0156] d. Experience storage: Transferring samples Store in the experience replay buffer.

[0157] e. Network Update: Sample a batch of data from the buffer. Update the DQN branch: Calculate the Q-value loss. And backpropagate. Update the PPO branch: calculate the advantage function using the sampled data. Then, by maximizing the pruning objective function The Actor network is updated by minimizing the error of the value function, and the Critic network is updated accordingly.

[0158] Step 4: Establish a random channel scenario DRL model for training, and then use this method to optimize the low-Earth orbit satellite communication link in the corresponding design scenario.

[0159] (1) Design and parameterization of random training environment

[0160] To ensure that the trained DRL agent possesses strong generalization ability, a highly randomized dynamic channel simulation environment was constructed. The core parameters of the environment and the randomization range are shown in Table 1.

[0161] Table 1

[0162] Initial elevation angle 10°～80° At the start of each training round, the satellite has a random elevation angle relative to the ground station. orbital height 950~1000km Simulate the minute orbital differences between different low-Earth orbit satellites. Rainfall rate 0~50mm / h (exponential distribution) Simulates various weather conditions ranging from sunny to torrential rain, with most of the time being light rain or no rain. Number of interference sources 0–4 (Poisson distribution) Simulate a dynamic co-frequency interference environment. Interference source power -20 to 0 dBm (uniform distribution) The transmission power of each interference source. Rice K factor 5–15 dB (uniform distribution) Simulate different multipath environments, from paths with some scattering to strong direct sunlight. Packet arrival rate 0.1–1.0 Mbps (uniform distribution) Simulate changing network traffic load.

[0163] Environment state transition function: The environment is updated at each step according to the following formula.

[0164] Satellite position update: ;

[0165] in, express The satellite's position vector at any given time; express The satellite's position vector at a given time. This is the position update function.

[0166] Channel update: ,in It is the channel correlation coefficient. It's a random perturbation. This simulates the time-varying correlation of the channel.

[0167] Business update: At each time step, based on arrival rate Randomly generated data packets are added to the queue.

[0168] (2) DRL training process and hyperparameters

[0169] The training method combines federated learning and centralized training, and the training hyperparameters are shown in Table 2.

[0170] Table 2

[0171] PPO learning rate <![CDATA[3×10 -4 ]]> DQN learning rate <![CDATA[1×10 -3 ]]> Discount factor 0.99 Experience replay buffer size <![CDATA[1×10 6 ]]> Batch size 128 Federal aggregation cycle <![CDATA[Every 10 4 step]]>

[0172] The DRL training process is as follows:

[0173] a. Initialization: Randomly initialize the DRL network parameters for all satellites.

[0174] b. Episode Loop: Each training episode simulates a complete satellite overhead process (approximately 10-15 minutes of simulation time).

[0175] c. Step-by-step loop: Observation is performed at each time step (e.g., 10ms): The agent obtains the state from the environment. The agent then outputs actions through the PPO and DQN networks. The environment then performs actions and calculates the new state. and rewards ; Empirical tuples Store the data in an experience replay buffer; then periodically sample from the buffer and update the network parameters using the PPO pruning objective function and the DQN temporal difference error.

[0176] (3) Simulation results and comparative analysis

[0177] After training, the performance was evaluated in a fixed test scenario (orbital altitude 975km, initial elevation angle 30°, heavy rain conditions - rainfall rate 25mm / h, 2 medium-power interference sources) and compared with two existing methods.

[0178] Comparing two traditional methods: one is a fixed strategy, employing a conservative fixed configuration with a transmit power of 40 dBm, an MCS of 16QAM 3 / 4, and a fixed beam, which is the benchmark scheme for many traditional satellite systems. The other is adaptive modulation and coding, a widely studied adaptive method that dynamically adjusts the modulation and coding scheme based on the instantaneous signal-to-noise ratio (SNR) fed back from the receiver, using a preset SNR-MCS mapping table, but with a fixed transmit power.

[0179] The performance comparison results are shown in Table 3.

[0180] Table 3

[0181] Average throughput (Mbps) 58.2 67.5 74.8 +28.5% Average Bit Error Rate (BER) <![CDATA[8.5×10 -5 ]]> <![CDATA[2.1×10 -5 ]]> <![CDATA[5.2×10 -6 ]]> -93.9% Average transmission delay (ms) 22.3 18.9 16.1 -27.8% Average transmit power (dBm) 40.0 (fixed) 40.0 (fixed) 28.7 -28.3%

[0182] As shown in Table 3, compared to a fixed strategy, this invention dynamically adjusts parameters through the DRL algorithm, maintaining a low BER under poor channel conditions by reducing MCS and using intelligent power control; while under good channel conditions, it increases MCS and optimizes beamforming to maximize throughput. However, the fixed strategy has lower implementation complexity and computational overhead, while this invention requires certain on-board computing resources to run neural network inference. Compared to adaptive modulation and coding, since adaptive modulation and coding only adjusts MCS, while this invention jointly optimizes power and MCS, in the test scenario, the DRL agent not only reduces MCS but also simultaneously reduces power when the channel is poor. Furthermore, adaptive modulation and coding is a passive response based on the current instantaneous SNR, while the DRL agent, through learning, can sense that interference is about to increase and take robust configurations in advance, avoiding sudden spikes in BER. This invention also optimizes parameters such as beamforming, further improving signal quality and anti-interference capability. However, the adaptive modulation and coding algorithm is simple, easy to understand and deploy, while the decision logic of the DRL model in this invention is not as intuitive as the lookup table method, requiring a more complex training and verification process.

[0183] Overall, the simulation results fully verify the effectiveness of the deep reinforcement learning-based link optimization method proposed in this invention. Compared with two typical existing technologies, this invention, through intelligent cross-layer parameter joint decision-making, can simultaneously improve throughput, reliability, latency, and energy efficiency in highly dynamic and non-stationary low-Earth orbit satellite channels, demonstrating strong environmental adaptability and overall performance advantages, and providing key technical support for next-generation intelligent satellite communication systems.

Claims

1. A method for optimizing low-Earth orbit satellite communication links based on deep reinforcement learning, characterized in that, The specific steps are as follows: Step 1: Consider the Doppler frequency shift and its time-varying characteristics caused by the high-speed motion of the satellite, and combine it with atmospheric effects to construct a channel dynamic scenario for low-orbit communication satellites in space transmission. Step 2: Based on the link performance evaluation criteria and in conjunction with the performance indicators of the communication link, design a weighted comprehensive reward mechanism; among which, key performance indicators include throughput, bit error rate, transmission latency, and power consumption. Step 3: Define the agent's action as the satellite transmitting signals to the ground according to the designed radio frequency parameters and modulation and coding scheme in the channel dynamic scenario constructed in Step 1; and design a deep reinforcement learning DRL model, with the environment state as the input and the corresponding agent action as the output; wherein, the DRL model includes a PPO branch and a DQN branch, the PPO branch is used to adjust the transmit power and beamforming weights, and the DQN branch is used to select the modulation and coding scheme and the number of retransmissions; Step 4: The DRL model is trained offline using a large number of randomly constructed channel scenarios. During the training process, the agent obtains the state from the environment, outputs actions through the DRL model, and then the environment executes the actions to calculate the new state and reward, update the DRL model, and optimize the low-Earth orbit satellite communication link using the output of the DRL model in the corresponding design scenario.

2. The method for optimizing low-Earth orbit satellite communication links based on deep reinforcement learning according to claim 1, characterized in that, The performance metrics of the communication link in step 2 include: (1) Throughput norm : Use effective throughput, i.e., the rate of successfully received bits: In the formula, For channel bandwidth, Here, BLER is the modulation order, and BLER is the block error rate, which is the probability that a data block is received incorrectly. This represents the system's maximum theoretical throughput. (2) Bit error rate (BER): Theoretical BER for QPSK modulation: For M-QAM modulation: Actual simulation BER: In the formula, Bit signal-to-noise ratio (SNR) is the ultimate representation of the link budget. and These are the complementary error function and the Gaussian Q function, respectively. (3) Total transmission delay include: In the formula, The propagation delay is determined by the distance between the satellite and the ground. ,in This is the instantaneous distance between Earth and space. The speed of light; Transmission delay, related to data packet length and symbol rate related, ; To handle latency, including encoding, modulation, demodulation, and decoding time, it can be modeled as a fixed value or a random distribution; Queuing delay, which occurs in on-board routers when multiple data streams compete for the same output port, is related to traffic load and scheduling algorithm, and can be estimated using M / M / 1 or queuing theory models. (4) Power efficiency P consumed : In the formula, For transmission power, For power amplifier efficiency, This represents the actual power consumption of the power amplifier. This refers to static power consumption. By employing a linear weighted sum method, the above multiple indicators are merged into a single scalar reward value, transforming the multi-objective optimization problem into a single-objective problem: In the formula, In time step The instant reward received; The normalized throughput metric, i.e. ; The bit error rate metric after penalty, i.e. The aim is to amplify the impact of BER and give it a reasonable weight in the reward function; The normalized latency metric, i.e. D max This is the maximum latency that the system can tolerate; The normalized power consumption metric, i.e. ,in This is the maximum permissible transmission power; These are the weighting coefficients.

3. The method for optimizing low-Earth orbit satellite communication links based on deep reinforcement learning according to claim 1, characterized in that, Step 3 includes the following specific steps: (1) The actions of the agent Defined as a composite action containing continuous and discrete components: ; For continuous action space Used for fine-tuning radio frequency parameters, defined as: In the formula, The transmit power adjustment is a normalized continuous value, typically ranging from [−1, 1], where P is the actual transmit power. t We obtain this through linear mapping: In the formula, For beamforming weights, i.e., phase offset, This is the maximum permissible transmission power; It is the minimum permissible transmission power; This refers to the number of elements in a phased array antenna. For discrete action space Used to make category selection decisions, defined as: Wherein, MCS Index is the modulation and coding scheme index, which is a discrete value. The corresponding index and modulation and coding relationship can be set. Each index corresponds to a predefined combination of modulation order and coding rate; Retry Count is the maximum number of retransmissions; (2) Design a deep reinforcement learning (DRL) model, including a feature extraction network, a PPO branch, and a DQN branch, with the original state as the input. ; The feature extraction network uses a CNN-LSTM hybrid structure, with the CNN part used to extract the original state. The local features in the image contain spatial structure information about the state; the LSTM part is used to capture the original state. Long-term dependencies in time series; extracted feature vectors Simultaneously transmitted to both the PPO branch and the DQN branch; The PPO branch contains an Actor network and a Critic network: Actor Network The input is the feature vector. The output is a continuous action. The mean and variance of are defined in the state The probability distribution of taking continuous actions is determined; the goal is to learn the optimal continuous control strategy. Critic Network The input is the feature vector. The output is the state value. That is, from the state Initially, it represents the expected cumulative reward that can be obtained in the future; the goal is to evaluate the quality of the current state to guide the updating of the Actor network. The PPO branch ensures the stability of policy updates through its pruned objective function: in, This represents the probability ratio between the old and new strategies. The advantage function estimate represents the superiority or inferiority of a particular action relative to the average level, and is calculated from the Critic network and the actual reward. , It is a hyperparameter; In time Instant rewards received This represents the state value output by the Critic network; The DQN branch contains a Q network. It is used to handle discrete decision problems, with the input being a feature vector. The output is a Q-value vector, where each element corresponds to a discrete action combination. The Q-value is the expected long-term cumulative reward; the decision is to select the discrete action with the largest Q-value using a greedy or ε-greedy strategy: In the formula, It represents a discrete action chosen randomly. Represents the current state The action with the highest Q value; The DQN branch updates the network using temporal difference error, with the goal of minimizing the following loss function: in, That is, the target Q value; Indicates the discount factor. ; That is, the target Q-network, its parameters Regularly from the main network This was copied and is used for stable training. To predict the Q value; Represents the set of all weights and biases in a deep Q-network; Indicates the next state Below, all possible actions The highest Q value.

Images