LEO satellite access switching method based on multi-agent deep recurrent Q network
The MA-DRQN algorithm addresses the challenges of LEO satellite access by modeling dynamic environments and resource competition, enhancing access success rates and throughput in LEO satellite networks through a distributed decision framework.
Patent Information
- Application Number
- CN202310887785.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-07-19
AI Technical Summary
Traditional LEO satellite access algorithms face challenges in high mobility, frequent satellite switching, excessive signaling resource overhead, reduced communication quality, and low access success rates due to complex and dynamic wireless environments, especially in mega constellations, with existing DRL algorithms failing to adapt to time-varying environmental states and ignoring resource competition and dynamic changes.
A LEO satellite access switching method based on a multi-agent deep recurrent Q-network (MA-DRQN) that models shadow Rice fading channels, constructs a multi-objective dynamic optimization problem, and uses a distributed decision center training framework to integrate resource competition factors, enabling efficient access and switching decisions through a DRQN network with LSTM layers.
The MA-DRQN algorithm improves access success rates, reduces switching frequency, and enhances wireless resource utilization efficiency, optimizing long-term throughput and communication quality in LEO satellite networks.
Smart Images

Figure CN117040588B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of LEO satellite communications, and in particular to a LEO satellite access switching method based on a multi-agent deep cyclic Q network. Background Art
[0002] Low Earth Orbit (LEO) satellite communications have the advantages of wide coverage, low latency, and large capacity, and are considered an important part of the future 6G integrated space-ground network. Constrained by high mobility and limited payload, in order to provide seamless global high-speed coverage, LEO satellite networks are evolving towards mega constellations. This results in LEO satellite access showing the characteristics of multi-satellite overlapping coverage and frequent cell switching. Traditional access solutions are difficult to capture and track the dynamic changes of the wireless environment, resulting in low wireless resource utilization efficiency and high access failure probability during LEO satellite access.
[0003] Reinforcement Learning (RL) is an important branch of machine learning. Its core is to enable intelligent agents to take actions to interact with the environment and obtain the optimal strategy through continuous risk-taking and trial and error. RL is often used in decision-making problems in low-dimensional, small-scale discrete environmental states and discrete action space scenarios, but it is difficult to play a role in high-dimensional, large-scale continuous environmental states and action space scenarios. Based on this challenge, the Deep Reinforcement Learning (DRL) algorithm combines RL with Deep Learning (DL), fully integrating DL's ability to extract low-dimensional features from high-dimensional data with RL's decision-making ability, and has shown excellent performance in complex continuous decision-making problems.
[0004] Traditional LEO satellite access algorithms mainly include: access algorithms based on a single decision-making objective, access algorithms based on multi-attribute weighted decision-making, and access scheme optimization algorithms based on game theory or deep learning. The problems exposed by traditional LEO satellite access algorithms in this scenario are:
[0005] ① Due to the high mobility of LEO satellites, user equipment (UE) needs to frequently switch satellites, resulting in excessive signaling resource overhead, reduced communication quality, reduced system throughput, and a high probability of communication interruption;
[0006] ② As the future construction direction of LEO satellites is to build giant constellations, user equipment terminals may be densely covered by multiple satellites at the same time. In a complex and dynamic wireless environment, it is necessary to select an optimal access satellite, which involves multiple access control factors, such as the remaining visible time of the satellite, the carrier-to-noise ratio of the uplink, the interference-to-noise ratio, the number of available channels, etc. There may be conflicts or trade-offs between these factors, and it is difficult to obtain a comprehensive access control strategy, which makes it difficult for user equipment (UE) to select the optimal access option, resulting in low access success rate and reduced system throughput.
[0007] ③ Due to the high mobility of LEO satellite nodes and the overlapping coverage of giant constellations, the wireless environment observed by user equipment (UE) is complex and dynamic. Traditional target decision algorithms or access solutions based on game theory and deep learning are highly dependent on the data of the wireless environment and are difficult to cope with the ever-changing and complex LEO satellite access scenarios, resulting in the intervention strategy failing to converge to the optimal state, resulting in reduced system throughput, high switching frequency, and low access success rate.
[0008] The DRL-based low-orbit satellite access handover decision algorithm, for example, the IEEE paper "Deep Reinforcement Learning-based Satellite Handover Scheme for Satellite Communications" proposed a DRL algorithm based on a convolutional neural network (CNN), which is used to solve the handover decision problem of a single user in a LEO satellite network. The algorithm uses multi-angle observation data as input, comprehensively considers factors such as LEO satellite coverage information, remaining service time, carrier-to-noise ratio, interference-to-noise ratio, satellite channel resource information, etc., constructs a dual-objective optimization problem of minimizing the probability of handover failure and the number of handovers, and designs a CNN-based DQN algorithm to solve the problem. However, some problems are exposed in the problems that the present invention hopes to solve:
[0009] ① It is not possible to effectively extract the time-varying characteristics of the environmental state caused by the high-speed movement of LEO satellites;
[0010] ② The existing solution of using base stations or satellite nodes to make DRL access decisions will require frequent relearning of environmental parameters, because each access or switch of user equipment (UE) will cause the base station or satellite node to observe the change of environmental status, which will lead to a sharp increase in computing resource overhead, high time cost, and inability to adapt to delay-sensitive services;
[0011] ③ The existing algorithms do not consider the competitive impact of massive user equipment (UE) on the limited resources of LEO satellites;
[0012] ④ The existing algorithm does not consider the reality of dense multiple coverage of user equipment (UE) by LEO giant constellations, which simplifies the problem to be solved. When faced with the problem that the present invention is intended to solve, it will lead to problems such as failure to converge to the optimal intervention strategy, decreased system throughput, and low access success rate;
[0013] ⑤ Existing DRL-based LEO satellite access decision algorithms often only consider a single objective, such as spectrum efficiency, energy efficiency or user satisfaction, while ignoring the trade-offs and coordination among multiple objectives;
[0014] ⑥ Existing DRL-based LEO satellite access decision algorithms often rely on static or simplified network models, while ignoring the dynamic changes and uncertainties in low-orbit satellite networks.
[0015] Therefore, it is necessary to design an intelligent access decision algorithm that can quickly adapt to the highly dynamic environmental characteristics of LEO satellites, so as to obtain higher access success rate, lower switching overhead and higher network throughput in the LEO satellite access problem and improve the overall performance of the network. Summary of the invention
[0016] The present invention is aimed at the multi-user competitive access problem scenario in the dense coverage scenario of a highly dynamic LEO satellite giant constellation. In view of the problem that traditional access solutions are difficult to capture and track the dynamic changes of the wireless environment, resulting in low wireless resource utilization efficiency and high access failure probability in the satellite access process, the present invention aims to maximize the long-term system throughput and reduce the switching frequency, comprehensively considers the access and switching problems of wireless resource management, and designs a LEO satellite access switching algorithm based on a multi-agent deep recurrent Q-network (Multi-agent Deep Recurrent Q-Network, MA-DRQN).
[0017] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0018] The LEO satellite access switching method based on a multi-agent deep cycle Q network of the present invention comprises the following steps:
[0019] Step S1: construct a shadow Rice fading channel model;
[0020] Step S2: constructing a transmission rate model;
[0021] Step S3: Modeling a multi-objective dynamic optimization problem;
[0022] Step S4: construct a LEO satellite access switching method framework based on a multi-agent deep cyclic Q network;
[0023] Step S5: constructing a Markov decision process model;
[0024] Step S6: DRQN network design;
[0025] Step S7: Design of MA-DRQN access switching decision algorithm.
[0026] Furthermore, the specific steps of step one are as follows:
[0027] The satellite-to-ground link channel is modeled as a shadowed Rice fading channel model, namely the SR model, which is a channel model that combines a Rice fading channel with fluctuating LoS path parameters. First, assume that the user equipment UE in time slot t i Receive LEO satellite SAT j The transmitted signal power is:
[0028]
[0029] Where P t j Indicates LEO satellite SAT j Transmitting power, g i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The channel gain between the two channels is expressed as g i,j (t)=|H i,j (t) 2 , H i,j (t) includes free space loss, rain attenuation and SR fading factors, so H i,j (t) is expressed as:
[0030] H i,j (t) = A i,j (t)W i,j (t)(2)
[0031] Where A i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The large-scale fading caused by the path loss between and rain attenuation is:
[0032]
[0033] Where λ represents the carrier wavelength, G r The square of represents the receiving antenna gain of the user equipment terminal, s i,j (t) represents the t-time slot user equipment UEi and LEO satellite SAT j The rain attenuation index between ξ and d represents the antenna gain of the LEO satellite transmitter. i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The geometric distance between
[0034] Where W i,j (t) represents the small-scale fading part of the satellite-to-ground link, which is modeled as SR fading and expressed as SR(Ω,b0,m). Its LoS component is modeled as Nakagami-m distribution, and the scattered component obeys Rayleigh distribution. Ω represents the average power of the LoS component signal, 2b0 represents the average power of the scattered component signal, and m is the Nakagami parameter. In summary, the user equipment UE that has undergone SR fading i and LEO satellite SAT j Signal power gain value The cumulative distribution function calculation formula is:
[0035]
[0036] Where (m) n =Γ(m+n) / Γ(m) is the Pochhammer symbol, Γ(·) represents the gamma function, n is the n below the summation formula, ranging from 0 to ∞, n! is the mathematical representation of the factorial of n, and γ(·,·) represents the incomplete gamma function.
[0037] Furthermore, the specific steps of step 2 are as follows:
[0038] Assume that the user equipment UE in time slot t i If a LEO satellite is connected, the interference signal-to-noise ratio calculation formula is:
[0039]
[0040] In the formula and Represents LEO satellite SAT j and LEO satellite SAT k The transmission frequency, g i,j (t) represents the t-time slot user equipment UE i and the connected communication LEO satellite SAT j The channel gain between i,k (t) represents the t-time slot user equipment UE i and other unconnected interfering LEO satellites SAT k The channel gain between 2 User equipment UE iNoise power;
[0041] If the LEO satellite SAT j A total of L j (t) user equipment, then according to Shannon's formula, we can get the UE of time slot t i Received LEO satellite SAT j The data transfer rate is:
[0042]
[0043] Where B is the total bandwidth, L j (t) represents the time slot t of the LEO satellite SAT j Number of connected user devices, SINR i,j (t) represents the t-time slot user equipment UE i Interference signal-to-noise ratio when accessing a LEO satellite.
[0044] Furthermore, the specific steps of step three are as follows:
[0045] To optimize the long-term throughput, the long-term throughput is modeled as the sum of the data transmission rates in T time slots from 0 to T-1:
[0046]
[0047] Where x i,j (t) represents the t-time slot user equipment UE i Whether to access LEO satellite SAT j , x i,j (t) is a binary variable, taking the value 0 or 1. If x i,j (t) = 1 means t time slot user equipment UE i Request access to LEO satellite SAT j , if x i,j (t) = 0 means t time slot user equipment UE i Not to LEO satellite SAT j Send access request; Formula (9) represents the user equipment UE in the same time slot i Only J candidate LEO satellites can be accessed SAT = {SAT0, SAT1, ..., SAT J-1} 1 LEO satellite in; c i,j (t) represents the t-time slot user equipment UE i Received LEO satellite SAT j Data transmission rate; User equipment set U = {UE0, UE1, ..., UE M-1}, Represents any user equipment UE in the user equipment set Ui ;
[0048] Based on formula (8), the switching cost HOCost is introduced i,j (t) is used as a negative reward and extended to the scenario of multi-user device access. Then the optimization goal of each user device is:
[0049]
[0050]
[0051] In the formula, c i,j (t) represents the t-time slot user equipment UE i Received LEO satellite SAT j Data transmission rate, t time slot LEO satellite SAT j The number of connected user devices L j (t) should not exceed LEO satellite SAT j The upper load limit L j,max ; Among them, the switching cost HOCost i,j The calculation formula of (t) is:
[0052]
[0053] Where C is a man-made constant that represents the size of the switching overhead value, x i,j (t) represents the t-time slot user equipment UE i Whether to access LEO satellite SAT j , x i,j (t-1) represents the user equipment UE in the time slot t-1 i Whether to access LEO satellite SAT j .
[0054] Furthermore, the specific steps of step 4 are as follows:
[0055] The LEO satellite access switching method based on multi-agent deep recurrent Q network, namely the MA-DRQN access switching decision algorithm, adopts a framework of distributed decision center training, including:
[0056] ① Distributed user equipment terminal: Each user equipment terminal contains a DRQN network with the same structure. As a distributed user equipment terminal, it independently observes the environmental status and makes access switching decisions, and stores and uploads experience tuples;
[0057] ② Central training node deployed in the backhaul network: The central training node contains a DRQN network with the same structure as the user equipment and a capacity of N PThe central experience pool P stores the fused experience tuples; the central training node aggregates the local experience tuples uploaded by each user device through the LEO satellite, calculates the resource competition factor of each user device, forms a fused experience tuple after fusion processing, and performs learning and network parameter θ update. After the network parameter θ is updated, the network parameter copy The network parameter copy is transmitted back to the connected user equipment through each LEO satellite access node, and the user equipment uses the network parameter copy Update your network parameters
[0058] ③LEO satellite access node: responsible for accepting or rejecting access requests from user devices, and responsible for uploading and downloading data transmitted by user devices and central training nodes.
[0059] Furthermore, the specific operation steps of step five are as follows:
[0060] Step S5.1: Construct state space s i (t):
[0061] (i) t-1 time slot user equipment UE i The received signal strength transmitted by each LEO satellite in the LEO satellite set SAT g i,j (t-1) represents the user equipment UE in the time slot t-1 i and LEO satellite SAT j The channel gain of the satellite-to-ground link between
[0062] (ii) t time slot user equipment UE i The received signal strength transmitted by each LEO satellite in the LEO satellite set SAT g i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The channel gain between
[0063] (iii) Load of each LEO satellite in time slot t-1, and the number of user devices L accessing in time slot t-1 j (t-1),
[0064] In summary, the user equipment UE i The multi-angle state space required to be observed in time slot t is:
[0065]
[0066] In the formula They represent SAT in the LEO satellite set SAT in time slot t respectively. j,j=0,1,…,J-1 transmission power, g i,0 (t-1),g i,1 (t-1),…,g i,J-1 (t-1) represents the SAT in the LEO satellite set SAT in the time slot t-1 j ,j=0,1,…,J-1 to user equipment UE i The channel gain value of the satellite-to-ground link between i,0 (t),g i,1 (t),…,g i,J-1 (t) represents the SAT in the LEO satellite set SAT in time slot t j ,j=0,1,…,J-1 to user equipment UE i The channel gain values of the satellite-to-ground link between L0(t-1), L1(t-1),…, L J-1 (t-1) represents the SAT in the LEO satellite set SAT in the time slot t-1 j ,j=0,1,…,J-1The number of user devices connected, s i (t) contains 3J elements in total. The signal strengths transmitted by each LEO satellite in (i) and (ii) are directly measured by the user equipment terminal. The load conditions of each LEO satellite in (iii) are sent by the LEO satellite to the central training node in the time slot and then fed back to each user equipment terminal after aggregation.
[0067] Step S5.2: Construct action space a i (t);
[0068] User Equipment UE i The action in time slot t is to select a LEO satellite that provides coverage above it, so the action space a in time slot t is i (t) is:
[0069] a i (t) = {x i,0 (t),x i,1 (t),…,x i,J-1 (t)}(16)
[0070] Where x i,0 (t),x i,1 (t),…,x i,J-1 (t) represents the t time slot user equipment UE i Whether to access LEO satellite SAT j ,j=0,1,…,J-1,x i,j (t) is a binary variable, taking the value 0 or 1. If x i,j (t) = 1 means t time slot user equipment UE i Request access to LEO satellite SATj , if x i,j (t) = 0 means t time slot user equipment UE i Not to LEO satellite SAT j Sending access request;
[0071] Step S5.3: Constructing the initial reward function
[0072] User Equipment UE i At time slot t, according to the observed state space s i (t) After taking an action, a certain reward will be obtained. By designing the initial reward function, the user equipment UE i Learn and optimize the strategy in the set direction. The initial reward function designed is:
[0073]
[0074] Where a i (t) represents the action space of time slot t, a i (t-1) represents the action space of time slot t-1, c i,j (t) represents the t-time slot user equipment UE i Received LEO satellite SAT j Data transfer rate, HOCost i,j (t) represents the switching overhead;
[0075] Step S5.4: Construct the fusion reward function r i (t);
[0076] Based on the initial reward function of each user equipment, the part affecting other user equipment is added. Suppose the user equipment UE i Access to LEO satellite SAT at time slot t j , let Ω j (t) represents the time slot t minus the user equipment UE i Other access LEO satellite SAT j The user equipment set is UE i After taking action in time slot t, due to resource competition, the user equipment set Ω j (t) The impact on user equipment is:
[0077]
[0078] Where ω k (t) represents the user equipment set Ω in time slot t j (t) User equipment UE k Actual data transfer rate achieved:
[0079]
[0080] Where P t j Indicates LEO satellite SAT j The transmission power, P t n Indicates the number of LEO satellites in the time slot t except SAT j Transmitting power of other interfering satellites, g k,j (t) represents the time slot t of the LEO satellite SAT j To the user device set Ω j (t) User equipment UE k The channel gain of the satellite-to-ground link between k,n (t) represents the number of LEO satellites in the time slot t except SAT j Other interference satellites to user equipment set Ω j (t) User equipment UE k The channel gain of the satellite-to-ground link, ω k-i (t) represents the assumed user equipment UE i No access to LEO satellite SAT j User device set Ω j (t) User equipment UE k Achievable data transfer rates:
[0081]
[0082] In summary, the fusion reward function that integrates the initial reward function of each user device and the user device resource competition factor is given as:
[0083]
[0084] In the formula, β represents the impact factor (in percentage form) of resource competition factors between user devices, which can be flexibly adjusted according to needs.
[0085] Further, in step six, the DRQN network includes:
[0086] ① DRQN network input: Each user equipment UE in time slot t i The input of the DRQN network is a set of length T seq The state sequence S i,seq (t) = {s i (t),s i (t-1),…,s i (tT seq +1)},s i (t),s i (t-1),…,si (tT seq +1) respectively represent the user equipment UE in time slot t and the previous time slots i The state space of the user equipment UE can be stored in the memory of the user equipment without changing the hardware design, including the current time slot user equipment UE i The observed state s i (t) and the past T stored in the user device cache seq -1 time slot User Equipment UE i The observed state, the input data size is T seq ×3J matrix;
[0087] ② DRQN network output: output data is user equipment UE i In the action space a at time slot t i (t);
[0088] ③DRQN network structure design: The DRQN network structure includes: input layer, fully connected layer and output layer; the input layer is a 2-layer LSTM network; the number of fully connected layers is 3, which is used to process the temporal characteristics of the input layer output, and the activation functions of the three fully connected layers all use the Leaky_ReLU function; the activation function of the output layer uses the softmax function.
[0089] Furthermore, the specific operation steps of step seven are as follows:
[0090] Step S7.1: MA-DRQN access handover decision algorithm structure;
[0091] The MA-DRQN access switching decision algorithm structure includes:
[0092] ① Distributed user equipment intelligent terminal: Each user equipment independently performs status observation and access decision-making; each user equipment is modeled as an intelligent agent with independent decision-making ability, that is, a DRQN network with the same network structure as the central training node is built for each user equipment. i Network parameters θ i is a copy of the parameters θ trained by the central training node, and each agent has a size of N P The experience playback pool P; each user equipment UE in time slot t i The input data of the network is a set of length T seq The time sequence state sequence S i,seq (t) = {s i (t),s i (t-1),…,s i (tT seq +1)},s i (t),si (t-1),…,s i (tT seq +1) respectively represent the user equipment UE in time slot t and the previous time slots i The state space of the user equipment UE can be stored in the memory of the user equipment without changing the hardware design, including the current time slot user equipment UE i The observed state s i (t) and the past T stored in the user device cache seq -1 time slot User Equipment UE i The observed state, the network output is the user equipment UE i Action decision at time slot t; User equipment UE i According to the action decision, an access request is sent to the corresponding LEO satellite using the uplink UL. After access and connection establishment, each user device uploads the local experience tuple to the LEO satellite through the uplink UL and then forwards it to the central training node of the backhaul network for storage and utilization;
[0093] ② The central training node deployed in the backhaul network is responsible for aggregating and integrating the independent access data of each user device, calculating the resource competition factor of the user device, training and updating the network parameters, and transmitting the updated parameters back to the user device; when a certain amount of experience is stored in the experience pool of the central training node for experience playback, a group of experiences is randomly selected for training and network parameter update, and the updated parameter copies are The LEO satellite node transmits the data back to each user device through the downlink DL and updates the original parameters of the user device network.
[0094] ③LEO satellite access node: responsible for uploading and downloading data from user devices and training nodes;
[0095] Step S7.2: MA-DRQN access handover decision algorithm timing logic;
[0096] The MA-DRQN access switching decision algorithm is divided into two stages: the initialization stage and the training stage. The initialization stage is responsible for: (i) each intelligent agent and the training node initialize the DRQN network with the same structure, (ii) each intelligent agent uses the RSS algorithm to make access decisions and store experience tuples; the training stage is responsible for: (i) making access and switching decisions according to the ε-greedy strategy, and (ii) updating network parameters.
[0097] The beneficial effects of the present invention are:
[0098] In the multi-user competitive access problem in the scenario of dense coverage of a high-dynamic LEO satellite giant constellation, the traditional access solution is difficult to capture and track the dynamic changes of the wireless environment, resulting in low wireless resource utilization efficiency and high access failure probability in the LEO satellite access process. The present invention designs a LEO satellite access switching method based on a multi-agent deep loop Q network. The present invention models the resource competition factors for the access of massive user equipment UE, designs a multi-agent DRL architecture based on user equipment UE decision-making, and designs a DRQN network including a long short-term memory (LSTM) network to extract the hidden features of the time-varying environmental state caused by the high-speed movement of LEO satellites, assist the intelligent agent in making accurate decisions, improve the access success rate, system throughput, and improve the efficiency of wireless resource utilization. The present invention solves the access control and switching decision-making problems of LEO satellite networks, which is of great significance for realizing a 6G space-ground integrated network. BRIEF DESCRIPTION OF THE DRAWINGS
[0099] Figure 1 The present invention is a framework for the MA-DRQN access switching decision algorithm.
[0100] Figure 2 The DRQN network structure designed by the present invention.
[0101] Figure 3 This is the MA-DRQN access decision solution of the present invention.
[0102] Figure 4 This is a timing logic diagram of the MA-DRQN access switching decision algorithm constructed by the present invention. DETAILED DESCRIPTION
[0103] The purpose of the present invention is to provide a LEO satellite access switching decision algorithm based on a multi-agent deep recurrent Q network (Multi-agent DeepRecurrent Q-Network, MA-DRQN). The algorithm can effectively utilize the historical information and current state of the user equipment terminal, utilize the characteristics of the temporal environment state, and integrate the resource competition factors of each UE. Through the method of deep reinforcement learning, it can realize adaptive, intelligent and efficient satellite access switching decision-making, thereby improving the communication quality and network performance of the user equipment terminal.
[0104] See also Figures 1 to 4 To illustrate, the LEO satellite access switching method based on a multi-agent deep cycle Q network of the present invention mainly includes the following steps:
[0105] Step S1: construct a shadow Rice fading channel model;
[0106] The environment and electromagnetic environment of the satellite-to-ground link are complex and easily affected by multiple factors, such as long propagation distance, rain attenuation, and small-scale fading during propagation. In practical applications, each factor should be considered. In terms of small-scale fading, the present invention models the satellite-to-ground link channel as a shadowed-Rician (SR) fading channel model. The shadowed-Rician fading channel model (hereinafter referred to as the SR model) is a channel model that combines a Rice fading channel with a fluctuating LoS (Line of Sight) path parameter. Compared with the commonly used Rayleigh fading channel, the SR model is a more accurate channel model than the Rayleigh distribution in the satellite-to-ground link scenario, and the SR model has a smaller amount of calculation, which is more suitable for user devices with limited computing power and reduces time overhead.
[0107] In the present invention, the t time slot user equipment UE i Receive LEO satellite SAT j The transmitted signal power is:
[0108]
[0109] Where P t j Indicates LEO satellite SAT j Transmitting power, g i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j For the convenience of the following description, the channel gain is written in square form here, and there is g i,j (t)=|H i,j (t)| 2 .
[0110] Considering the long transmission distance of the satellite-to-ground link and the complex electromagnetic and climatic conditions, the H i,j (t) includes free space loss, rain attenuation and SR fading factors, and H i,j (t) is expressed as:
[0111] H i,j (t) = A i,j (t)W i,j (t)(2)
[0112] Where A i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The large-scale fading caused by the path loss between and rain attenuation is:
[0113]
[0114] Where λ represents the carrier wavelength, G r The square of represents the receiving antenna gain of the user equipment terminal, s i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The rain attenuation index between ξ and d represents the antenna gain of the LEO satellite transmitter. i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The geometric distance between them.
[0115] Where W i,j (t) represents the small-scale fading part of the satellite-to-ground link, which is modeled as SR fading and expressed as SR(Ω,b0,m). Its LoS component is modeled as Nakagami-m distribution, and the scattered component obeys Rayleigh distribution. Among its parameters, Ω is the average power of the LoS component signal, 2b0 represents the average power of the scattered component signal, and m is the Nakagami parameter. In summary, the user equipment UE that has undergone SR fading can be i and LEO satellite SAT j Signal power gain value The cumulative distribution function of is written as:
[0116]
[0117] Where (m) n =Γ(m+n) / Γ(m) is the Pochhammer symbol, Γ(·) represents the gamma function, n is the n below the summation formula, ranging from 0 to ∞, n! is the mathematical representation of the factorial of n, and γ(·,·) represents the Lower Incomplete Gamma Function.
[0118] Step S2: constructing a transmission rate model;
[0119] In the LEO satellite multi-coverage network, user equipment UE i The interference signals received from multiple LEO satellites are a challenge to deal with a large number of interferences in LEO satellite networks. Therefore, the present invention first writes the calculation formula of Signal-to-Interference-plus-Noise ratio (SINR), assuming that the user equipment UE in time slot t i If a LEO satellite is connected, the interference signal-to-noise ratio should be written as:
[0120]
[0121] In the formula and Represents LEO satellite SAT j and LEO satellite SAT k The transmission frequency, g i,j (t) represents the t-time slot user equipment UE i and the connected communication LEO satellite SAT j The channel gain between i,k (t) represents the t-time slot user equipment UE i and other unconnected interfering LEO satellites SAT k The channel gain between 2 User equipment UE i Noise power.
[0122] In the present invention, the LEO satellite uses TDMA to deal with the access of multiple user devices. The LEO satellite needs to allocate resources such as channel bandwidth according to its own load. If the LEO satellite SAT j A total of L j (t) user equipment, then according to Shannon's formula, we can get the user equipment UE in time slot t i Received LEO satellite SAT j The data transfer rate is:
[0123]
[0124] Where B is the total bandwidth, L j (t) represents the time slot t of the LEO satellite SAT j Number of connected user devices, SINR i,j (t) represents the t-time slot user equipment UE i Interference signal-to-noise ratio when accessing a LEO satellite.
[0125] Step S3: Modeling a multi-objective dynamic optimization problem;
[0126] The core goal of the algorithm of the present invention is to improve the long-term throughput of the multi-user equipment access system under the dense coverage of the high-mobility LEO satellite giant constellation and reduce the switching frequency and access failure rate. Therefore, modeling is performed for the multi-objective dynamic optimization problem respectively.
[0127] First, for the goal of optimizing long-term throughput, each user equipment needs to maximize the long-term cumulative throughput. The long-term throughput is modeled as the comprehensive data transmission rate in T time slots from 0 to T-1. The optimization problem of each user equipment can be modeled as:
[0128]
[0129] Where x i,j(t) represents the t-time slot user equipment UE i Whether to access LEO satellite SAT j , x i,j (t) is a binary variable that can only take the value 0 or 1. i,j (t) = 1 means t time slot user equipment UE i Request access to LEO satellite SAT j , if x i,j (t) = 0 means t time slot user equipment UE i Not to LEO satellite SAT j Send access request. Formula (9) represents the user equipment UE in the same time slot i Only J candidate LEO satellites can be accessed SAT = {SAT0, SAT1, ..., SAT J-1} 1 LEO satellite in; c i,j (t) represents the t-time slot user equipment UE i Received LEO satellite SAT j Data transmission rate; User equipment set U = {UE0, UE1, ..., UE M-1}, Represents any user equipment UE in the user equipment set U i .
[0130] Furthermore, due to the high mobility of LEO satellites, if only the optimization is performed according to the goal of formula (7), the user equipment will frequently switch when seeking a higher instantaneous throughput in each time slot to achieve the goal, which will cause huge switching signaling overhead. Therefore, the present invention introduces the switching overhead HOCost based on formula (7) i,j (t) is used as a negative reward and extended to the scenario of multi-user device access. The optimization goal is transformed into seeking a balance between the high long-term throughput and low switching frequency of the entire system to achieve the desired goal. The optimization problem of each user device can be modeled as:
[0131]
[0132]
[0133] Where c i,j (t) represents the t-time slot user equipment UE i Received LEO satellite SAT j Data transmission rate, t time slot LEO satellite SAT j The number of connected user devices L j (t) should not exceed LEO satellite SAT j The upper load limit L j,max , switching cost HOCost i,jThe calculation formula of (t) is:
[0134]
[0135] Where C is a man-made constant that represents the size of the switching overhead value, x i,j (t) represents the t-time slot user equipment UE i Whether to access LEO satellite SAT j , x i,j (t-1) represents the user equipment UE in the time slot t-1 i Whether to access LEO satellite SAT j .
[0136] Step S4: constructing a LEO satellite access switching algorithm based on a multi-agent deep recurrent Q-network (MA-DRQN) (hereinafter referred to as MA-DRQN access switching decision algorithm) framework;
[0137] The MA-DRQN access switching decision algorithm constructed by the present invention adopts a distributed decision center training framework, which mainly includes three parts: distributed user equipment terminals, central training nodes and LEO satellite access nodes. Figure 1 shown.
[0138] ① Distributed user equipment terminal: Each user equipment terminal contains a DRQN network with the same structure. As a distributed user equipment terminal, it independently observes the environmental status and makes access switching decisions, and stores and uploads experience tuples.
[0139] ② Central training node deployed in the backhaul network: The central training node also contains a DRQN network with the same structure as the user equipment and a capacity of N P The central experience pool P stores the fused experience tuples. The central training node can aggregate the local experience tuples uploaded by each user device through the LEO satellite, calculate the resource competition factor of each user device, form a fused experience tuple after fusion processing, and perform learning and network parameter θ update. After the network parameter θ is updated, the network parameter copy The network parameter copy is transmitted back to the connected user equipment through each LEO satellite access node, and the user equipment uses the network parameter copy Update your network parameters
[0140] ③LEO satellite access node: responsible for accepting or rejecting access requests from user devices, and responsible for uploading and downloading data transmitted by user devices and central training nodes.
[0141] Step S5: constructing a Markov Decision Process (MDP) model;
[0142] This part constructs the required Markov decision process (MDP) model (hereinafter referred to as MDP model) for the multi-user access control and switching problems of the multi-coverage, highly dynamic LEO satellite network. First, the state space, action space and reward function are constructed. In particular, since the MA-DRQN access switching decision algorithm constructed by the present invention takes multiple user devices as intelligent agents and adopts a distributed decision center training framework, the reward function will be designed in two parts, namely, the initial reward that does not consider the impact of access decisions between user devices and the fusion reward that considers the mutual impact between user devices. The specific operation steps are as follows:
[0143] Step S5.1: Construct state space s i (t):
[0144] In order to enable each user equipment to make the best access and switching decisions among multiple high-speed mobile LEO satellites providing coverage, and taking into account the perception and computing capabilities of the user equipment terminal and the time cost, the user equipment UE is designed i The multi-angle state space that needs to be observed in time slot t includes:
[0145] (i) t-1 time slot user equipment UE i The received signal strength transmitted by each LEO satellite in the LEO satellite set SAT g i,j (t-1) represents the user equipment UE in the time slot t-1 i and LEO satellite SAT j The channel gain of the satellite-to-ground link between
[0146] (ii) t time slot user equipment UE i The received signal strength transmitted by each LEO satellite in the LEO satellite set SAT g i,j (t) represents the t-time slot user equipment UE i and LEO satellite SAT j The channel gain between
[0147] (iii) Load of each LEO satellite in the t-1 time slot. In the present invention, the number of user devices accessed in the t-1 time slot is L j (t-1),
[0148] In summary, the user equipment UE i The multi-angle state space required to be observed in time slot t can be written as:
[0149]
[0150] In the formula They represent SAT in the LEO satellite set SAT in time slot t respectively. j ,j=0,1,…,J-1 transmission power, g i,0 (t-1),g i,1 (t-1),…,g i,J-1 (t-1) represents the SAT in the LEO satellite set SAT in the time slot t-1 j ,j=0,1,…,J-1 to user equipment UE i The channel gain value of the satellite-to-ground link between i,0 (t),g i,1 (t),…,g i,J-1 (t) represents the SAT in the LEO satellite set SAT in time slot t j ,j=0,1,…,J-1 to user equipment UE i The channel gain values of the satellite-to-ground link between L0(t-1), L1(t-1),…, L J-1 (t-1) represents the SAT in the LEO satellite set SAT in the time slot t-1 j ,j=0,1,…,J-1The number of user devices connected, s i (t) contains 3J elements in total, among which (i) and (ii) can be directly measured by the user equipment terminal, and the load conditions of each LEO satellite in (iii) can be sent by the LEO satellite to the central training node within the time slot and then fed back to each user equipment terminal after aggregation.
[0151] Step S5.2: Construct action space a i (t);
[0152] User equipment UE in the algorithm of the present invention i The action in time slot t is to select a LEO satellite that provides coverage above it, so the action space a in time slot t is i (t) can be written as:
[0153] a i (t) = {x i,0 (t),x i,1 (t),…,x i,J-1 (t)}(16)
[0154] Where x i,0 (t),x i,1 (t),…,x i,J-1 (t) represents the t time slot user equipment UE i Whether to access LEO satellite SATj ,j=0,1,…,J-1,x i,j (t) is a binary variable, taking the value 0 or 1. If x i,j (t) = 1 means t time slot user equipment UE i Request access to LEO satellite SAT j , if x i,j (t) = 0 means t time slot user equipment UE i Not to LEO satellite SAT j Sending access request;
[0155] Step S5.3: Constructing the initial reward function
[0156] According to the optimization target of the present invention, the instant initial reward function of the algorithm of the present invention is set. The initial reward function is the initial reward of each user device, and does not consider the impact of the user device itself on other user devices after making an access decision. The fusion reward considering the mutual impact between user devices will be derived in step S5.4. User equipment UE i At time slot t, according to the observed state space s i (t) After taking an action, a certain reward will be obtained. By designing the initial reward function, the agent can learn and optimize the strategy in the set direction. The present invention intends to increase the long-term throughput of the system and reduce the switching overhead. Therefore, the designed initial reward function is:
[0157]
[0158] Where a i (t) represents the action space of time slot t, a i (t-1) represents the action space of time slot t-1, c i,j (t) represents the t-time slot user equipment UE i Received LEO satellite SAT j Data transfer rate, HOCost i,j (t) represents the switching overhead;
[0159] Step S5.4: Construct the fusion reward function r i (t);
[0160] The multi-user device access problem studied in the present invention is simultaneous rather than sequential, that is, each user device needs to make an access or switching decision in the same time slot t. Since the access request of each user device will occupy the wireless resources such as the channel of the LEO satellite in the time slot t, the access or switching decision of one user device will affect the transmission rate obtained by other user devices accessing the LEO satellite, resulting in resource competition. Therefore, the present invention designs a fusion reward function, which adds the part affecting other user devices on the basis of the initial reward function of each user device. Suppose the user device UE i Access to LEO satellite SAT at time slot t j , let Ω j (t) represents the time slot t minus the user equipment UE i Other access LEO satellite SAT j The user equipment set is UE i After taking action in time slot t, due to resource competition, the user equipment set Ω j The impact of user equipment within (t) can be modeled as:
[0161]
[0162] Where ω k (t) represents the user equipment set Ω in time slot t j (t) User equipment UE k The actual data transmission rate can be written as:
[0163]
[0164] Where P t j Indicates LEO satellite SAT j The transmission power, P t n Indicates the number of LEO satellites in the time slot t except SAT j Transmitting power of other interfering satellites, g k,j (t) represents the time slot t of the LEO satellite SAT j To the user device set Ω j (t) User equipment UE k The channel gain of the satellite-to-ground link between k,n (t) represents the number of LEO satellites in the time slot t except SAT j Other interference satellites to user equipment set Ω j (t) User equipment UE k The channel gain of the satellite-to-ground link, ω k-i (t) represents the assumed user equipment UE i No access to LEO satellite SATj User device set Ω j (t) User equipment UE k The data transfer rate that can be obtained can be written as:
[0165]
[0166] At this point, the fusion reward function that integrates the initial reward functions of each user device and the user device resource competition factor can be given:
[0167]
[0168] Where β represents the impact factor (in percentage form) of resource competition factors between user devices, which can be flexibly adjusted according to demand.
[0169] Due to the impact of computing on other user devices It is necessary to use the summary information of each user device, so the calculation of the fusion reward function is handed over to the central training node. After the central training node completes the training, it can be used for parameter training of the MA-DRQN network proposed by the present invention.
[0170] So far, the MDP model for the access control and switching problems of multiple user devices in a LEO satellite network with multiple coverage and high dynamics has been listed. Traditional optimization methods such as the Lyapunov equation method require a large amount of environmental information. The complex information is difficult to be quickly and reliably obtained by the user device or the central training node, and it is difficult to ensure the feasibility and immediacy of the access and switching decisions, and it is difficult to deal with the access and switching decision-making problems in the LEO satellite communication scenario. The method based on deep reinforcement learning (DRL) can quickly make decisions and optimize in time based on concise observation results. Therefore, the present invention solves the above problems based on DRL. The present invention subdivides the MDP model reward function into an initial reward function and a fusion reward function. The initial reward function is directly calculated by each user device UE, and the fusion reward function is obtained by the central training node based on the initial reward of each user device UE and considering the resource competition factor of each user device UE. The two parts of the reward function jointly guide the user device UE to improve the utilization efficiency of the wireless resources of the LEO satellite network while obtaining a higher access success rate and system throughput.
[0171] Step S6: DRQN network design;
[0172] It is difficult to construct a mathematical model for state transition in the present invention, because the state of the next time slot is not only affected by the current action, but also by multiple factors, such as the relative motion between the LEO satellite and the user equipment and access requests from other user equipment. Therefore, the present invention designs a deep recurrent Q network (DRQN) to implement a time series feature extraction network to fit the Q value and estimate the state transition probability hidden in the environment.
[0173] The DRQN network in the present invention is designed based on the LSTM network. Considering the state space s designed by the present invention i (t) mainly includes the received signal power data of two consecutive time slots, the current time slot t and the previous time slot t-1, and the load conditions of each LEO satellite in the previous time slot t-1. The received signal power data is related to the spatial distance between the LEO satellite and the user equipment and the load conditions of the LEO satellite. The spatial distance between the LEO satellite and the user equipment is mainly determined by the motion trajectory of the LEO satellite. Although the orbit and trajectory of the LEO satellite are pre-set, the timing characteristics hidden therein are unknown to the user equipment agent.
[0174] Therefore, the present invention designs a DRQN network with the ability to extract time series features based on a multi-layer LSTM network, which effectively extracts the hidden features of the time-varying environmental state caused by the high mobility of LEO satellites, and assists the user equipment UE agent in making access decisions. Compared with the traditional DQN algorithm, the access strategy is more accurate, the system throughput is greatly improved, and the computing power and delay-sensitive business requirements are taken into account. The DRQN network structure designed by the present invention is as follows: Figure 2 As shown, the DRQN network structure mainly includes the following aspects:
[0175] ① DRQN network input: In order to extract the hidden temporal features in the environment state, the network input state should also be temporal. Therefore, each user equipment UE is designed to have a temporal i The input of the DRQN network is a set of length T seq The state sequence S i,seq (t) = {s i (t),s i (t-1),…,s i (tT seq +1)},s i (t),s i (t-1),…,s i (tT seq +1) respectively represent the user equipment UE in time slot t and the previous time slots i The state space of the user equipment UE can be stored in the memory of the user equipment without changing the hardware design, including the current time slot user equipment UE i The observed state si (t) and the past T stored in the user device cache seq -1 time slot User Equipment UE i The observed state, the input data size is T seq ×3J matrix.
[0176] ② DRQN network output: output data is user equipment UE i In the action space a at time slot t i (t).
[0177] ③DRQN network structure design: Considering that the input data of the network is T seq ×3J matrix, so the input layer of the DRQN network is designed to be a 2-layer LSTM network to process matrix data with time series characteristics. The reason for not building a network with more layers is to reduce the learning time cost as much as possible to cope with the inherent long delay of LEO satellite communication and adapt to more delay-sensitive services. Since the LSTM network output corresponds to the data of each time slot of the input, in order to obtain the final action decision output, three fully connected layers are arranged to process the time characteristics of the LSTM output. The activation function of the three fully connected layers uses the Leaky_ReLU function, which can deal with the problem of neuron death caused by the use of the ReLU function, making learning more efficient. Finally, the output layer is set. The output layer activation function uses the softmax function. The output layer outputs the action value function value of the state and action, and selects the satellite access with the maximum action value function value.
[0178] Step S7: MA-DRQN access handover decision algorithm design;
[0179] Step S7.1: MA-DRQN access handover decision algorithm structure;
[0180] The MA-DRQN access switching decision algorithm of the present invention adopts a distributed decision center training architecture, such as Figure 3 As shown in the figure, UE represents user equipment and SAT represents LEO satellite, the algorithm consists of three parts:
[0181] ① Distributed user equipment intelligent terminal: Each user equipment independently performs status observation and access decision-making. Each user equipment is modeled as an intelligent agent with independent decision-making ability, that is, a DRQN network with the same network structure as the central training node is built for each user equipment. i Network parameters θ i is a copy of the parameters θ trained by the central training node, and each agent has a size of N P The experience playback pool P. In time slot t, each user equipment UE i The input data of the network is a set of length Tseq The time sequence state sequence S i,seq (t) = {s i (t),s i (t-1),…,s i (tT seq +1)},s i (t),s i (t-1),…,s i (tT seq +1) respectively represent the user equipment UE in time slot t and the previous time slots i The state space of the user equipment UE can be stored in the memory of the user equipment without changing the hardware design, including the current time slot user equipment UE i The observed state s i (t) and the past T stored in the user device cache seq -1 time slot User Equipment UE i The observed state, the network output is the user equipment UE i Action decision at time slot t; User equipment UE i According to the action decision, an access request is sent to the corresponding LEO satellite using the uplink (UL). After access and connection establishment, each user device uploads the local experience tuple to the LEO satellite through the uplink (UL) and then forwards it to the central training node of the backhaul network for storage and utilization;
[0182] ② The central training node deployed in the backhaul network is responsible for aggregating and integrating the independent access data of each user device, calculating the user device resource competition factor, training and updating the network parameters, and transmitting the updated parameters back to the user device. When the experience pool of the central training node stores a certain amount of experience for experience playback, a set of experiences (experience0(t), experience1(t), ..., experience M-1 (t)) Perform training and update network parameters, and update the parameter copies The data is transmitted back to each user device through the downlink (DL) via the LEO satellite node and the original parameters of the user device network are updated.
[0183] ③LEO satellite access node: responsible for uploading and downloading data from user devices and training nodes.
[0184] Step S7.2: MA-DRQN access handover decision algorithm timing logic;
[0185] The MA-DRQN access switching decision algorithm of the present invention can be divided into two stages: initialization stage and training stage. Figure 4As shown in the figure, the initialization phase is mainly responsible for: (i) each agent and the training node initialize the DRQN network with the same structure; (ii) each agent uses the RSS algorithm to make access decisions and store experience tuples. The training phase is mainly responsible for: (i) making access and switching decisions according to the ε-greedy strategy; (ii) updating network parameters.
[0186] (1) Initialization phase: The core goal is to accumulate experience tuples for use as training data sets in the training phase. First, each user device and the central training node initialize a DRQN network with the same structure to support the network training of the central training node and the direct update of network parameters of each user device terminal. Each time slot in the initialization phase can be further divided into four steps:
[0187] ① Each user equipment observes the environment and obtains the current time slot user equipment UE i The observed state s i (t), if it is not the initial time slot, then it is the partial experience tuple of the previous time slot t-1 Complete experience tuple They represent the user equipment UE in the t-1 time slot respectively i The multi-angle state space, action space and initial reward function required for observation;
[0188] ② Each user device makes access decisions according to the RSS algorithm;
[0189] ③Each user device obtains the initial reward function
[0190] ④Merge the incomplete experience tuples of the current time slot
[0191] The above four steps are repeated in each time slot until the last time slot of the initialization phase. After completing the above steps, each user device uploads the local experience tuple to the central training node through the connected LEO satellite. The central training node processes the fused experience and stores it centrally.
[0192] (2) During the training phase, each time slot still needs to be divided into four steps:
[0193] ① Each user equipment independently observes the environment and obtains the current time slot user equipment UE i The observed state s i (t) is the incomplete experience tuple with the previous time slot t-1 Complete experience tuple
[0194] ② Each user device makes access and switching decisions according to the ε-greedy strategy and obtains initial rewards Composing incomplete experience tuples
[0195] ③Each user device sends the local experience tuple Upload to the central training node via the connected LEO satellite;
[0196] ④ The central training node calculates and stores the fusion experience, and randomly extracts a sequence experience tuple with a sample batch size of N from the experience memory for learning and network parameter update. After every L time slots, a copy of the current parameters is The data is transmitted back to each user device through the downlink (DL) via the LEO satellite node and the original parameters of the user device network are updated. Until the end.
[0197] The present invention utilizes direct competition and indirect collaboration among massive user equipment UE to guide access strategy convergence, models the competition of massive user equipment UE for LEO satellite resources and incorporates the access strategy reference index, and effectively improves the utilization efficiency of LEO satellite network resources after optimization by MA-DRQN access switching decision algorithm.
[0198] The present invention performs distributed access decision-making based on user equipment agents and designs a multi-agent DRL algorithm according to a central training distributed decision (CTDE) architecture. Compared with traditional base station-based access decision schemes, the agent learning computing resource overhead of the present invention is smaller.
[0199] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A LEO satellite access switching method based on a multi-agent deep cyclic Q network, characterized in that: The following steps are involved: Step S1: construct a shadow Rice fading channel model; The satellite-to-ground link channel is modeled as a shadowed Rice fading channel model, namely the SR model, which is a channel model that combines a Rice fading channel with fluctuating LoS path parameters. First, assume that the user equipment UE in time slot t i Receive LEO satellite SAT j The transmitted signal power is: In the formula Indicates LEO satellite SAT j Transmitting power, g i,j (t) represents the time slot t of the user equipment UE i and LEO satellite SAT j The channel gain between the two channels is expressed as g i,j (t)=|H i,j (t) 2 , H i,j (t) includes free space loss, rain attenuation and SR fading factors, so H i,j (t) is expressed as: H i,j (t)=A i,j (t)W i,j (t)(2) Where A i,j (t) represents the time slot t of the user equipment UE i and LEO satellite SAT j The large-scale fading caused by the path loss between and rain attenuation is: Where λ represents the carrier wavelength, G r The square of represents the receiving antenna gain of the user equipment terminal, s i,j (t) represents the time slot t of the user equipment UE i and LEO satellite SAT j The rain attenuation index between ξ and d represents the antenna gain of the LEO satellite transmitter. i,j (t) represents the time slot t of the user equipment UE i and LEO satellite SAT j The geometric distance between Where W i,j (t) represents the small-scale fading part of the satellite-to-ground link, which is modeled as SR fading and expressed as SR(Ω,b0,m). Its LoS component is modeled as Nakagami-m distribution, and the scattered component obeys Rayleigh distribution. Ω represents the average power of the LoS component signal, 2b0 represents the average power of the scattered component signal, and m is the Nakagami parameter. In summary, the user equipment UE that has undergone SR fading i and LEO satellite SAT j Signal power gain value The cumulative distribution function calculation formula is: Where (m) n =Γ(m+n) / Γ(m) is the Pochhammer symbol, Γ(·) represents the gamma function, n is the n below the summation formula, ranging from 0 to ∞, n! is the mathematical representation of the factorial of n, and γ(·,·) represents the incomplete gamma function; Step S2: constructing a transmission rate model; Step S3: Modeling a multi-objective dynamic optimization problem; Step S4: construct a LEO satellite access switching method framework based on a multi-agent deep cyclic Q network; The LEO satellite access switching method based on multi-agent deep recurrent Q network, namely the MA-DRQN access switching decision algorithm, adopts a framework of distributed decision center training, including: ① Distributed user equipment terminal: Each user equipment terminal contains a DRQN network with the same structure. As a distributed user equipment terminal, it independently observes the environmental status and makes access switching decisions, and stores and uploads experience tuples; ② Central training node deployed in the backhaul network: The central training node contains a DRQN network with the same structure as the user equipment and a capacity of N P The central experience pool P stores the fused experience tuples; the central training node aggregates the local experience tuples uploaded by each user device through the LEO satellite, calculates the resource competition factor of each user device, forms a fused experience tuple after fusion processing, and performs learning and network parameter θ update. After the network parameter θ is updated, the network parameter copy The network parameter copy is transmitted back to the connected user equipment through each LEO satellite access node, and the user equipment uses the network parameter copy Update your network parameters ③LEO satellite access node: responsible for accepting or rejecting access requests from user devices, and responsible for uploading and downloading data transmitted by user devices and central training nodes; Step S5: constructing a Markov decision process model; Step S5.1: Construct state space s i (t): (i) t-1 time slot user equipment UE i The received signal strength transmitted by each LEO satellite in the LEO satellite set SAT g i,j (t-1) represents the user equipment UE in the time slot t-1 i and LEO satellite SAT j The channel gain of the satellite-to-ground link between (ii) t time slot user equipment UE i The received signal strength transmitted by each LEO satellite in the LEO satellite set SAT g i,j (t) represents the time slot t of the user equipment UE i and LEO satellite SAT j The channel gain between (iii) Load of each LEO satellite in the t-1 time slot and the number of user devices accessing the t-1 time slot In summary, the user equipment UE i The multi-angle state space required to be observed in time slot t is: In the formula They represent SAT in the LEO satellite set SAT in time slot t respectively. j ,j=0,1,…,J-1 transmission power, g i,0 (t-1),g i,1 (t-1),…,g i,J-1 (t-1) represents the SAT in the LEO satellite set SAT in time slot t-1 j ,j=0,1,…,J-1 to user equipment UE i The channel gain value of the satellite-to-ground link between i,0 (t),g i,1 (t),…,g i,J-1 (t) represents the SAT in the LEO satellite set SAT in time slot t j ,j=0,1,…,J-1 to user equipment UE i The channel gain value of the satellite-to-ground link between L0(t-1), L1(t-1),…, L J-1 (t-1) represents the SAT in the LEO satellite set SAT in time slot t-1 j ,j=0,1,…,J-1The number of user devices connected, s i (t) contains 3J elements in total. The signal strengths transmitted by each LEO satellite in (i) and (ii) are directly measured by the user equipment terminal. The load conditions of each LEO satellite in (iii) are sent by the LEO satellite to the central training node in the time slot and then fed back to each user equipment terminal after aggregation. Step S5.2: Construct action space a i (t); User Equipment UE i The action in time slot t is to select a LEO satellite that provides coverage above it, so the action space a in time slot t is i (t) is: a i (t)={x i,0 (t),x i,1 (t),…,x i,J-1 (t)}(16) Where x i,0 (t),x i,1 (t),…,x i,J-1 (t) represents the t time slot user equipment UE i Whether to access LEO satellite SAT j ,j=0,1,…,J-1,x i,j (t) is a binary variable, taking the value 0 or 1. If x i,j (t) = 1 means t time slot user equipment UE i Request access to LEO satellite SAT j , if x i,j (t) = 0 means t time slot user equipment UE i Not to LEO satellite SAT j Sending access request; Step S5.3: Constructing the initial reward function User Equipment UE i At time slot t, according to the observed state space s i (t) After taking an action, a certain reward will be obtained. By designing the initial reward function, the user equipment UE i Learn and optimize the strategy in the set direction. The initial reward function designed is: Where a i (t) represents the action space of time slot t, a i (t-1) represents the action space of time slot t-1, c i,j (t) represents the time slot t of the user equipment UE i Received LEO satellite SAT j Data transfer rate, HOCost i,j (t) represents the switching overhead; Step S5.4: Construct the fusion reward function r i (t); Based on the initial reward function of each user equipment, the part affecting other user equipment is added. Suppose the user equipment UE i Access to LEO satellite SAT at time slot t j , let Ω j (t) represents the time slot t minus the user equipment UE i Other access LEO satellite SAT j The user equipment set is UE i After taking action in time slot t, due to resource competition, the user equipment set Ω j (t) The impact on user equipment is: Where ω k (t) represents the user equipment set Ω in time slot t j (t) User equipment UE k Actual data transfer rate achieved: In the formula Indicates LEO satellite SAT j The transmission power, Indicates the number of LEO satellites in the time slot t except SAT j Transmitting power of other interfering satellites, g k,j (t) represents the time slot t of the LEO satellite SAT j To the user device set Ω j (t) User equipment UE k The channel gain of the satellite-to-ground link between k,n (t) represents the number of LEO satellites in the time slot t except SAT j Other interference satellites to user equipment set Ω j (t) User equipment UE k The channel gain of the satellite-to-ground link, ω k-i (t) represents the assumed user equipment UE i No access to LEO satellite SAT j User device set Ω j (t) User equipment UE k Achievable data transfer rates: In summary, the fusion reward function that integrates the initial reward function of each user device and the user device resource competition factor is given as: Where β represents the impact factor of resource competition between user devices; Step S6: DRQN network design; The DRQN network includes: ① DRQN network input: Each user equipment UE in time slot t i The input of the DRQN network is a set of length T seq The state sequence S i,seq (t) = {s i (t),s i (t-1),…,s i (tT seq +1)}, s i (t),s i (t-1),…,s i (tT seq +1) respectively represent the user equipment UE in time slot t and the previous time slots i The state space of the user equipment UE can be stored in the memory of the user equipment without changing the hardware design, including the current time slot user equipment UE i The observed state s i (t) and the past T stored in the user device cache seq -1 time slot User Equipment UE i The observed state, the input data size is T seq ×3J matrix; ② DRQN network output: output data is user equipment UE i In the action space a at time slot t i (t); ③DRQN network structure design: The DRQN network structure includes: input layer, fully connected layer and output layer; the input layer is a 2-layer LSTM network; the number of fully connected layers is 3, which is used to process the temporal characteristics of the input layer output, and the activation functions of the three fully connected layers all use the Leaky_ReLU function; the activation function of the output layer uses the softmax function; Step S7: MA-DRQN access handover decision algorithm design; Step S7.1: MA-DRQN access handover decision algorithm structure; The MA-DRQN access switching decision algorithm structure includes: ① Distributed user equipment intelligent terminal: Each user equipment independently performs status observation and access decision-making; each user equipment is modeled as an intelligent agent with independent decision-making ability, that is, a DRQN network with the same network structure as the central training node is built for each user equipment. i Network parameters θ i is a copy of the parameters θ trained by the central training node, and each agent has a size of N P The experience playback pool P; each user equipment UE in time slot t i The input data of the network is a set of length T seq The time sequence state sequence S i,seq (t) = {s i (t),s i (t-1),…,s i (tT seq +1)},s i (t),s i (t-1),…,s i (tT seq +1) respectively represent the user equipment UE in time slot t and the previous time slots i The state space of the user equipment UE can be stored in the memory of the user equipment without changing the hardware design, including the current time slot user equipment UE i The observed state s i (t) and the past T stored in the user device cache seq -1 time slot User Equipment UE i The observed state, the network output is the user equipment UE i Action decision at time slot t; User equipment UE i According to the action decision, an access request is sent to the corresponding LEO satellite using the uplink UL. After access and connection establishment, each user device uploads the local experience tuple to the LEO satellite through the uplink UL and then forwards it to the central training node of the backhaul network for storage and utilization; ② The central training node deployed in the backhaul network is responsible for aggregating and integrating the independent access data of each user device, calculating the resource competition factor of the user device, training and updating the network parameters, and transmitting the updated parameters back to the user device; when a certain amount of experience is stored in the experience pool of the central training node for experience playback, a group of experiences is randomly selected for training and network parameter update, and the updated parameter copies are The LEO satellite node transmits the data back to each user device through the downlink DL and updates the original parameters of the user device network. ③LEO satellite access node: responsible for uploading and downloading data from user devices and training nodes; Step S7.2: MA-DRQN access handover decision algorithm timing logic; The MA-DRQN access switching decision algorithm is divided into two stages: the initialization stage and the training stage. The initialization stage is responsible for: (i) each intelligent agent and the training node initialize the DRQN network with the same structure, (ii) each intelligent agent uses the RSS algorithm to make access decisions and store experience tuples; the training stage is responsible for: (i) making access and switching decisions according to the ε-greedy strategy, and (ii) updating network parameters.
2. The LEO satellite access switching method based on a multi-agent deep cycle Q network according to claim 1 is characterized in that: The specific steps of step 2 are as follows: Assume that the user equipment UE in time slot t i If a LEO satellite is connected, the interference signal-to-noise ratio calculation formula is: In the formula and Represents LEO satellite SAT j and LEO satellite SAT k The transmission frequency, g i,j (t) represents the time slot t of the user equipment UE i and the connected communication LEO satellite SAT j The channel gain between i,k (t) represents the time slot t of the user equipment UE i and other unconnected interfering LEO satellites SAT k The channel gain between 2 User equipment UE i Noise power; If the LEO satellite SAT j A total of L j (t) user equipment, then according to Shannon's formula, we can get the user equipment UE in time slot t i Received LEO satellite SAT j The data transfer rate is: Where B is the total bandwidth, L j (t) represents the time slot t of the LEO satellite SAT j Number of connected user devices, SINR i,j (t) represents the time slot t of the user equipment UE i Interference signal-to-noise ratio when accessing a LEO satellite.
3. The LEO satellite access switching method based on a multi-agent deep cycle Q network according to claim 2 is characterized in that: The specific steps of step three are as follows: To optimize the long-term throughput, the long-term throughput is modeled as the sum of the data transmission rates in T time slots from 0 to T-1: Where x i,j (t) represents the time slot t of the user equipment UE i Whether to access LEO satellite SAT j , x i,j (t) is a binary variable, taking the value 0 or 1. If x i,j (t) = 1 means t time slot user equipment UE i Request access to LEO satellite SAT j , if x i,j (t) = 0 means t time slot user equipment UE i Not to LEO satellite SAT j Send access request; Formula (9) represents the user equipment UE in the same time slot i Only J candidate LEO satellites can be accessed SAT = {SAT0, SAT1, ..., SAT J-1 } 1 LEO satellite in; c i,j (t) represents the time slot t of the user equipment UE i Received LEO satellite SAT j Data transmission rate; User equipment set U = {UE0, UE1, ..., UE M-1 }, Represents any user equipment UE in the user equipment set U i ; Based on formula (8), the switching cost HOCost is introduced i,j (t) is used as a negative reward and extended to the scenario of multi-user device access. Then the optimization goal of each user device is: Where c i,j (t) represents the time slot t of the user equipment UE i Received LEO satellite SAT j Data transmission rate, t time slot LEO satellite SAT j The number of connected user devices L j (t) should not exceed LEO satellite SAT j The upper load limit L j,max ; Among them, the switching cost HOCost i,j The calculation formula of (t) is: Where C is a man-made constant that represents the size of the switching overhead value, x i,j (t) represents the time slot t of the user equipment UE i Whether to access LEO satellite SAT j , x i,j (t-1) represents the user equipment UE in the time slot t-1 i Whether to access LEO satellite SAT j .
