An efficient ground station reselection method based on multi-dimensional spatiotemporal data and DQN

Abstract

The present invention relates to an efficient ground station reselection method based on multi-dimensional spatiotemporal data and DQN, the method comprising the following steps: building a distributed decision center training framework, including a communication LEO satellite node, a meteorological LEO satellite node and a central training node, the communication LEO satellite node and the central training node both containing a sequential DQN network with the same structure; using the communication LEO satellite as an intelligent agent in the DQN, using the time slot as the minimum time unit, each satellite in each time slot independently performs environmental state observation by its own sequential DQN network and outputs the optimal ground station selection result under the current time slot, and each satellite performs ground station connection according to the optimal ground station selection result. The present invention is based on the DQN algorithm, using the communication LEO satellite as the DQN intelligent agent, using spatiotemporal multi-dimensional data to assist the communication LEO satellite in selecting a ground station, effectively reducing the number of attempts to select and switch ground stations, improving signal service quality, and reducing signaling overhead.

Description

Technical Field

[0001] The present invention relates to the field of satellite communication technology, and in particular to an efficient ground station reselection method based on multi-dimensional spatiotemporal data and DQN. Background Art

[0002] With the development of modern satellite manufacturing technology, launch technology and network technology, large-scale constellation communication networks have become a reality. High-dynamic communication satellite communication systems, especially low-orbit (LEO) satellite communication networks, are easily affected by the satellite-to-ground link environment. Therefore, when the low-orbit satellite communication network encounters a complex dynamic satellite-to-ground link environment, it will cause channel attenuation, resulting in disconnection with the ground station, and the ground station needs to be reselected and switched.

[0003] The invention patent with the publication number of CN111835394A discloses a system and method for multi-ground station collaboration to resist satellite channel attenuation. The system and method can detect the satellite-to-ground channel conditions in real time and select the best modulation and coding scheme that meets the service QoS requirements according to the current situation. When the channel capacity declines seriously, a multi-site collaboration scheme can be adopted. By changing the inter-satellite routing, the remaining low-priority load data is diverted and forwarded to the collaborative ground site, and then sent to the destination node via the ground site network, which not only guarantees the QoS and delay requirements of the high-priority data service, but also guarantees the overall data transmission capacity of the system. Although the multi-ground site collaboration system and method for resisting satellite channel attenuation can resist satellite-to-ground channel attenuation and improve the system data transmission capacity, it still relies on RSS data when selecting the ground station, and the basis for ground station selection (based on the data type) is relatively single. Although the method can obtain better access results at the current moment, it is impossible to predict which ground stations will be better choices in the future based on these data. There is a disadvantage of insufficient high dynamic characteristics of the electromagnetic environment of the satellite-to-ground link of the LEO satellite network, which may lead to the problem of many reselections of ground stations in the future. Summary of the invention

[0004] Aiming at the problem that the low earth orbit (LEO) satellite communication network encounters the influence of complex dynamic satellite-to-ground link environment, resulting in channel attenuation and disconnection with the ground station, and the need to reselect and switch the ground station multiple times, the present invention proposes an efficient ground station reselection method based on multi-dimensional spatiotemporal data and DQN.

[0005] To solve the above problems, the present invention adopts the following technical solutions:

[0006] An efficient ground station reselection method based on multi-dimensional spatiotemporal data and DQN includes the following steps:

[0007] Step 1: Building a distributed decision center training framework, the distributed decision center training framework includes multiple communication LEO satellite nodes, at least one meteorological LEO satellite node and a central training node, each of the communication LEO satellite nodes and the central training node contains a sequential DQN network with the same structure;

[0008] Step 2: Take the communication LEO satellite as the intelligent agent in DQN, and the time slot as the minimum time unit. In each time slot, each satellite independently observes the environmental state through its own sequential DQN network and outputs the optimal ground station selection result in the current time slot. Each satellite connects to the ground station according to the optimal ground station selection result in its current time slot.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] The efficient ground station reselection method proposed in the present invention is based on the DQN algorithm, which models the communication LEO satellite as a DQN decision-making agent, and uses spatiotemporal multi-dimensional data such as RSS and meteorological data to assist the communication LEO satellite in ground station selection, so as to make a more accurate and efficient strategy that can reduce the number of ground station selections, effectively reduce the number of ground station re-access attempts caused by the poor satellite-to-ground link channel conditions caused by bad weather, improve signal service quality, and reduce signaling overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Schematic diagram of the interaction process between the agent and the environment;

[0012] Figure 2 Schematic diagram of the distributed decision center training framework;

[0013] Figure 3 A structural diagram of a sequential DQN network in an embodiment of the present invention;

[0014] Figure 4 A timing logic diagram of the method proposed in the present invention. DETAILED DESCRIPTION

[0015] The following is an introduction to the principles of reinforcement learning and deep reinforcement learning.

[0016] The two basic components of reinforcement learning (RL) are the environment and the agent. The problem that RL hopes to solve is how to maximize the reward of the agent in the process of interacting with the environment. Figure 1As shown in the figure, in the interaction process, the agent learns a policy by continuously interacting with the environment, and the quality of the policy depends on the cumulative rewards obtained after long-term execution of the policy. Therefore, in the RL task, the purpose of learning is to find a strategy that maximizes the long-term cumulative rewards.

[0017] RL tasks are usually described using a Markov decision process (MDP) model. The MDP model must include<S,A,P,R,π> , where the elements represent the state space S, action space A, state transfer function P, reward function R and strategy π, as follows:

[0018] (1) State space S: This part should contain the set of all states in the environment. The state at time t is represented by s(t), s(t)∈S;

[0019] (2) Action space A: This part should contain the set of all actions that the agent can choose in all states of the environment. The action at time t is represented by a(t), a(t)∈A;

[0020] (3) State transition function P: This part represents the probability that the agent takes action a(t) in the current state s(t) to transfer the environment state to s(t+1). This probability generally depends on the environment itself.

[0021] (4) Reward function R: This part represents the reward value that the environment gives back to the agent after the agent selects an action in the current environment state. By setting the reward function, the agent can learn and develop in the direction of the optimization goal;

[0022] (5) Strategy π: The strategy can be expressed as a probability distribution function related to the state-action pair, which is a mapping from state to action. For example, the probability of taking action a according to strategy π in state s can be expressed as:

[0023] π(a|s)=p(A=a|S=s), (1)

[0024] The agent continuously takes actions according to the observed state of the environment from t to t+T-1, and transfers the environment to the next state. This process can be recorded as

[0025] <s(t),a(t),r(t),s(t+1),…,s(t+T-1),a(t+T-1),r(t+T-1)> , (2)

[0026] Through this interactive process, experience tuples are continuously accumulated<s(t),a(t),r(t),s(t+1)> , and then the agent can continuously modify the policy parameters through learning according to the guidance of the reward function R to achieve the optimal policy π * To maximize the long-term cumulative discounted value function:

[0027]

[0028] Where γ represents the discount factor, which is used to obtain more rewards in the shortest possible time; r(s(t), a(t)) represents the immediate reward obtained by the agent when taking a(t) at time t.

[0029] Next, we will introduce Q learning and DQN. The traditional Q learning algorithm is an offline algorithm based on solving the value function. The action value function Q is updated by the agent constantly interacting with the environment. π (s,a) to obtain the optimal strategy π * , action value function Q π (s,a) can be expressed as:

[0030]

[0031] Where p(s(t+1)|s(t),a(t)) represents the state transition probability of the environment state transferring to s(t+1) after taking action a(t) in state s(t).

[0032] In Q-learning, by constructing a Q π (s,a) is a table of elements (Q-table) and Q(s,a) is repeatedly updated through the temporal difference method. The iterative process can be expressed as:

[0033]

[0034] Where α is the learning rate. The optimal Q-table is finally obtained through repeated iterations. When in use, the network selects actions by looking up the table. However, in actual scenarios such as LEO satellite access control, the state and action space are very complex, which causes the size of the Q-table to become huge. It is not feasible to find the optimal strategy through a huge Q-table.

[0035] The DQN algorithm uses a deep neural network to extract features from high-dimensional environmental state information to address the above problems. The DQN algorithm uses a deep neural network (Q network) as a function Q π (s, a; θ) to approximate Q in Q learning π (s,a), where θ is the parameter of the deep neural network in DQN. Then the original Q learning Qπ The (s,a) value update process can be replaced by iterating the parameters θ through gradient descent.

[0036] This architecture requires two Q networks with the same structure, namely the current Q network and the target Q network. The current Q network is used for action decision-making and model parameter update, and the target Q network is used to calculate the target Q value. The parameters of the target Q network do not need to be iterated in each round, but are directly copied from the current Q network after a certain period of time to achieve delayed update, reduce the correlation between the target Q value and the current Q value, and thus ensure the stability of training.

[0037] When training the current Q network, a large amount of independent data is needed. Since the MDP model has Markov properties, the correlation between state changes is strong. In order to eliminate the correlation between data, DQN uses the experience replay technology to replay the experience tuples obtained from each round of interaction between the agent and the environment.<s(t),a(t),r(t),s(t+1)> Stored in an experience memory, after accumulating a certain cardinality, a part of the experience tuples are randomly extracted and input into the current Q network for training, and the parameters θ are updated by back propagation. The correlation between experiences is broken by random sampling to obtain better training results. According to the above dual network architecture, the following mean square error loss function can be constructed to facilitate parameter update:

[0038] L(θ)=E[(y DQN -Q π (s(t),a(t);θ)) 2 ], (6)

[0039] in The Q value output by the target network. Through the Stochastic Gradient Descent (SGD) strategy, the loss value can be continuously reduced after a period of training, and the optimal parameter θ of the Q network can be obtained after convergence. * .

[0040] The ε-greedy strategy is adopted in the action decision part of DQN, which can be expressed as:

[0041]

[0042] Here, ε is a manually set hyperparameter used to change the “risk-taking” level of the agent.

[0043] At the current moment, the DQN selects the action according to the current state and strategy π, the reward obtained, and the state at the next moment to form an experience tuple, which is stored in the experience replay pool for the next moment DQN to perform network learning and parameter update. This cycle is repeated to finally obtain the optimal network parameters and intelligent agent strategy.

[0044] The purpose of the present invention is to provide an efficient ground station reselection method based on multi-dimensional spatiotemporal data and DQN, and to provide a DQN-based compensation mechanism for dynamic coordination loss of multiple ground stations, so as to cope with the channel attenuation and ground station connection interruption caused by the high dynamic communication LEO satellite network encountering the complex dynamic satellite-to-ground link environment, thereby improving the system throughput. The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0045] The embodiment of the present invention provides a method for efficiently reselecting a ground station based on multi-dimensional spatiotemporal data and DQN, the method comprising the following steps:

[0046] Step 1: Build a distributed decision center training framework, such as Figure 2 As shown, the distributed decision center training framework includes multiple communication LEO satellite nodes, at least one meteorological LEO satellite node and a central training node, wherein each communication LEO satellite node and the central training node contain a sequential DQN network with the same structure.

[0047] Still see Figure 2 The distributed decision center training framework includes N communication LEO satellite nodes, namely, communication LEO satellite-1, communication LEO satellite-2, ..., communication LEO satellite-N. There are M ground stations in the coverage area of ​​the meteorological LEO satellite, namely, ground station-1, ground station-2, ..., ground station-M. The compensation mechanism provided in this embodiment adopts the framework of distributed decision center training, which specifically includes:

[0048] ① Communication LEO satellite node: Each communication LEO satellite node contains a sequential DQN network with the same structure, namely the DRQN ​​network. As a distributed communication LEO satellite node, it independently observes the environmental status and makes access switching decisions, and stores and uploads experience tuples;

[0049] ② Meteorological LEO satellite node: observes meteorological data within the coverage area; LEO satellites serve as ground user equipment (UE) access nodes; two types of LEO satellites serve as nodes of the LEO satellite network of this system;

[0050] ③ Central training node deployed in the backhaul network: The central training node contains a DRQN ​​network with the same structure as the user equipment and a capacity of N PThe central experience pool P is used to store fused experience tuples; the central training node aggregates the local experience tuples uploaded by each communication LEO satellite node through the LEO satellite, calculates the resource competition factor of each communication LEO satellite node, forms a fused experience tuple after fusion processing, and performs learning and network parameter θ update. After the network parameter θ is updated, the network parameter copy The communication LEO satellite nodes use the network parameter copies to Update your network parameters

[0051] ④ Onboard Software Defined Network (SDN) controller: used to obtain the bandwidth capacity of the satellite-to-ground channel and the amount of load data required by the system, change the remaining data routing, and forward the data that the channel cannot accommodate to the remaining ground stations with remaining bandwidth;

[0052] ⑤ Ground SDN controller: used to obtain the routing topology between ground sites and receive the remaining data routing table, and forward the changed data flow from the ground network to the destination site according to the data flow destination node.

[0053] The sequential DQN network structure of the communication LEO satellite and the central training node is as follows: Figure 3 As shown. Considering that the input data of the network is a matrix, that is, the input state sequence state(t), state(t-1), ..., state(tT seq +1), where T seq Represents the length of the temporal state input to the DQN network in this embodiment. Its specific value can be manually defined. Therefore, the input layer of the DQN network in this embodiment is a 2-layer LSTM network. The 2-layer LSTM network is used to process the matrix data with temporal characteristics. The reason for not building a network with more layers is to reduce the learning time cost as much as possible to cope with the longer delay inherent in LEO satellite communications and adapt to more delay-sensitive services. Since the LSTM network output corresponds to the data of each time slot of the input, in order to obtain the final action decision output, three fully connected layers are arranged to process the temporal characteristics of the LSTM output. The activation function of the three fully connected layers uses the Leaky_ReLU function, which can deal with the problem of neuron death caused by the use of the ReLU function, making learning more efficient. Finally, the output layer is set. The output layer activation function uses the softmax function. The output state of the output layer is the Q of the action. π (s,a).

[0054] Step 2: Take the communication LEO satellite as the intelligent agent in DQN, and the time slot as the minimum time unit. In each time slot, each satellite independently observes the environmental state through its own sequential DQN network and outputs the optimal ground station selection result in the current time slot. Each satellite connects to the ground station according to the optimal ground station selection result in its current time slot.

[0055] Furthermore, step 2 specifically includes the following steps:

[0056] Step 2.1: The communication LEO satellite knows the geographical location covered by its own beam through GNSS and ephemeris data, and sends a request for meteorological data through the intersatellite link or the meteorological LEO satellite on the ground that is turned to the area. The meteorological LEO satellite sends the atmospheric data of the past D consecutive observations to the communication LEO satellite;

[0057] Step 2.2: Each communicating LEO satellite detects the connection status with the connected ground station at each time slot t, and observes the current time slot state space at each time slot t. The state space is the collection of the RSS data of the current time slot t, the RSS data of the previous t-1 time slot, and the meteorological data of the previous t-1 time slot. Since the number of ground stations is relatively fixed, the size of the state space does not need to be adjusted too much, which makes it easy to design the DRQN ​​network;

[0058] Step 2.3: In 1~N P In each time slot, each communicating LEO satellite node selects the ground station that provides the maximum signal power RSS value to connect to, because the number of experience tuples must first reach N P Only then can DQN training be performed, and then the ground station selection strategy can be selected according to the DQN algorithm;

[0059] Step 2.4: In 1~N P In each time slot, each communicating LEO satellite node obtains the reward for this time slot after making the action of selecting the ground station to connect to. The reward function is defined as follows:

[0060]

[0061] Where SAT={SAT0,SAT1,…,SAT J-1} represents the communication LEO satellite set: x i,j (t) represents the t-time slot user equipment UE i Whether to access LEO satellite SAT j , x i,j (t) is a binary variable, taking the value 0 or 1. If x i,j (t) = 1 means t time slot user equipment UE i Request access to LEO satellite SAT j , if x i,j (t) = 0 means t time slot user equipment UEi Not to LEO satellite SAT j Send access request; the user equipment set is U = {UE0, UE1, ..., UE M-1},UE i represents any user equipment in the user equipment set U; c i,j (t) represents the t-time slot user equipment UE i Received communication LEO satellite SAT j The data transmission rate of i (t) = {x i,0 (t),x i,1 (t),…,x i,J-1 (t)} represents the action space of time slot t; C(t) represents the switching loss value at time t, which can be set manually according to actual needs.

[0062] The reward function is a piecewise reward function. The first line of the formula indicates that if the ground station selected in the current time slot is different from that in the previous time slot, a positive value C(t) is subtracted from the basic reward, which encourages the agent to avoid frequent switching of ground stations as much as possible after training.

[0063] Step 2.5: In the next t+1 time slot, the communicating LEO satellite agent observes the state and forms an experience tuple, which is stored in the local experience pool;

[0064] Step 2.6: In N P Time slot, each communicating LEO satellite node sends local N P The experience tuples are uploaded to the central training node deployed in the backhaul network. The central training node stores the above experience tuples and randomly extracts BATCH experience tuples for DQN network training and updates network parameters. When the number of experience tuples reaches N P After that, the initialization phase is completed and the training phase begins. Figure 4 The following is a schematic diagram of timing logic, which is generally divided into the initialization phase and the training phase. Each time slot in the initialization phase can be divided into the following Figure 4 The five sub-steps shown in Figure 2 show that each time slot in the training phase can be divided into Figure 4 The 4 sub-steps shown, where A = N P +1 indicates a positive integer. The number of time slots in the initialization phase needs to be greater than T seq +A, that is, the number of experience tuples accumulated in the initialization phase should be at least greater than the data T input to the DQN network seq , experience pool size N P The sum of these three.

[0065] Step 2.7: Thereafter, in each time slot, each communicating LEO satellite obtains the observation status of the current time slot, connects according to the optimal ground station selection result output by the DQN network, obtains rewards, and uploads experience tuples to the central training node; every L time slots, the central training node returns the updated DQN parameters θ of the time slot to each communicating LEO satellite node.

[0066] The efficient ground station reselection method proposed in this embodiment is based on the DQN algorithm, which models the communication LEO satellite as a DQN decision-making agent, and uses spatiotemporal multi-dimensional data such as RSS and meteorological data to assist the communication LEO satellite in ground station selection, so as to make a more accurate and efficient strategy that can reduce the number of ground station selections, effectively reduce the number of ground station re-access attempts caused by the bad satellite-to-ground link channel conditions caused by bad weather, improve signal service quality, and reduce signaling overhead.

[0067] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An efficient ground station reselection method based on multi-dimensional spatiotemporal data and DQN, characterized in that: The following steps are involved: Step 1: Building a distributed decision center training framework, the distributed decision center training framework includes multiple communication LEO satellite nodes, at least one meteorological LEO satellite node and a central training node, each of the communication LEO satellite nodes and the central training node contains a sequential DQN network with the same structure; Step 2: Take the communication LEO satellite as the intelligent agent in DQN, and the time slot as the minimum time unit. In each time slot, each satellite independently observes the environmental state through its own sequential DQN network and outputs the optimal ground station selection result in the current time slot. Each satellite connects to the ground station according to the optimal ground station selection result in its current time slot. Step 2 includes the following steps: Step 2.1: The communication LEO satellite obtains the geographical location covered by its own beam through GNSS and ephemeris data, and sends a request for obtaining meteorological data through an intersatellite link or a meteorological LEO satellite on the ground that is turned to the location. The meteorological LEO satellite sends the atmospheric data of the past D consecutive observations to the communication LEO satellite; Step 2.2: Each communicating LEO satellite detects the connection status with the connected ground station at each time slot t, and observes the current time slot state space at each time slot t, wherein the state space is a collection of the RSS data of the current time slot t, the RSS data of the previous t-1 time slot, and the meteorological data of the previous t-1 time slot; Step 2.3: In 1~N P In the time slot, each communicating LEO satellite node selects the ground station that provides the maximum signal power RSS value for connection; Step 2.4: In 1~N P In each time slot, each communicating LEO satellite node obtains the reward for this time slot after making the action of selecting the ground station to connect to. The reward function is defined as follows: Where SAT={SAT0,SAT1,…,SAT J-1 } represents the communication LEO satellite set; x i,j (t) indicates whether the ground station i is connected to the communication LEO satellite SAT in time slot t j , x i,j (t) is a binary variable, taking the value 0 or 1. If x i,j (t) = 1 means that the ground station i requests to access the communication LEO satellite SAT in time slot t j , if x i,j (t) = 0 means that the ground station i did not send any signal to the communication LEO satellite SAT in time slot t. j Send access request; c i,j (t) represents the communication LEO satellite SAT received by ground station i in time slot t j The data transmission rate of i (t) = {x i,0 (t),x i,1 (t),…,x i,J-1 (t)} represents the action space of time slot t; C(t) represents the switching loss value at time t; Step 2.5: In the next t+1 time slot, the communicating LEO satellite agent observes the state and forms an experience tuple, which is stored in the local experience pool; Step 2.6: In N P Time slot, each communicating LEO satellite node sends local N P The experience tuples are uploaded to the central training node deployed in the backhaul network. The central training node stores the above experience tuples and randomly extracts BATCH experience tuples for DQN network training and updates network parameters. Step 2.7: Thereafter, in each time slot, each communicating LEO satellite obtains the observation status of the current time slot, connects according to the optimal ground station selection result output by the DQN network, obtains rewards, and uploads experience tuples to the central training node; every L time slots, the central training node returns the updated DQN parameters θ of the time slot to each communicating LEO satellite node.

2. According to claim 1, a method for efficient ground station reselection based on multi-dimensional spatiotemporal data and DQN is characterized in that: The central training node also includes a capacity of N P The central experience pool P is used to store fused experience tuples; the central training node aggregates the local experience tuples uploaded by each communication LEO satellite node through the LEO satellite, calculates the resource competition factor of each communication LEO satellite node, forms a fused experience tuple after fusion processing, and performs learning and network parameter θ update. After the update, the network parameter copy The network parameters are sent back to each communication LEO satellite node, and each communication LEO satellite node uses a copy of the network parameters Update your network parameters 3. The method for efficient ground station reselection based on multi-dimensional spatiotemporal data and DQN according to claim 1, characterized in that: The distributed decision center training framework also includes: The onboard software-defined network controller is used to obtain the bandwidth capacity of the satellite-to-ground channel and the amount of load data required by the system, change the remaining data routing, and forward the data that the channel cannot accommodate to the remaining ground stations with remaining bandwidth; The ground software-defined network controller is used to obtain the routing topology between ground sites and receive the remaining data routing table, and forward the changed routed data stream from the ground network to the destination site according to the destination node of the data stream.

4. The method for efficient ground station reselection based on multi-dimensional spatiotemporal data and DQN according to claim 1, characterized in that: The sequential DQN network includes an input layer, three fully connected layers and an output layer. The input layer is a 2-layer LSTM network. The activation function of the three fully connected layers adopts the Leaky_ReLU function, and the activation function of the output layer adopts the softmax function.

Images