A satellite-to-ground tracking and control link planning method based on reinforcement learning in low-orbit constellations

Through the deep Q network algorithm based on reinforcement learning, the star-ground measurement and control link planning of low-orbit constellations is optimized, and the problems of low efficiency and poor results in giant constellations and multi-ground station environments are solved, and efficient and flexible link planning is achieved. High antenna resource utilization rate.

CN115865168BActive Publication Date: 2024-11-22ZHENGZHOU UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202211455593.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-21
Publication Date
2024-11-22
Estimated Expiration
2042-11-21

AI Technical Summary

Technical Problem

The existing low-orbit constellation star-ground measurement and control link planning methods are inefficient and poor in the environment of giant constellations and multi-ground measurement and control stations, making it difficult to meet the problems of high complexity resource competition and excessive dimensions.

Method used

A deep Q network algorithm based on reinforcement learning is adopted to build a multi-objective optimization model, and through the interaction between the agent and the environment, the number of star-ground link switching times is optimized, the utilization rate of antenna resources is improved, and fast, flexible and efficient link planning is achieved.

Benefits of technology

Complete the large-scale constellation star-ground measurement and control link planning in a short period of time, reduce the number of link handover times and overhead, improve the utilization rate of antenna resources, and improve system stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115865168B_ABST
    Figure CN115865168B_ABST
Patent Text Reader

Abstract

The present invention proposes a satellite-to-ground measurement and control link planning method based on reinforcement learning in a low-orbit constellation, including: considering the satellite-to-ground link planning problem of a low-orbit giant constellation, satisfying the satellite-to-ground visible constraints, satellite link quantity constraints, antenna resource constraints and variable feasible domain restrictions, and establishing a multi-objective optimization model that maximizes resource utilization and minimizes the number of switching times; decoupling the link planning problem of the entire operation cycle of the constellation into the link planning problem within each discrete time slice, using the deep Q network algorithm in deep reinforcement learning to solve the multi-objective optimization model, completing the optimal giant constellation satellite-to-ground link planning decision, and the intelligent agent selects the best satellite to establish a link according to the visible satellite state at each allocation to complete the satellite-to-ground measurement and control link planning. The present invention uses reinforcement learning to realize the interaction between the intelligent agent and the environment, optimizes the number of switching times of the satellite-to-ground link during the constellation measurement and control process, improves the utilization rate of the antenna resources of the ground station, and realizes fast, flexible and efficient planning of the satellite-to-ground link.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of low-orbit satellite mobile communications, and in particular to a satellite-to-ground measurement and control link planning method based on reinforcement learning in a low-orbit constellation. When a ground center is used to measure and control the low-orbit constellation, deep reinforcement learning is used to perform satellite-to-ground link planning and allocation. Background Art

[0002] Satellite mobile communications are basically not restricted by geographical conditions. They can cover oceans, deserts and some areas with extreme geographical conditions. They complement the advantages of the current hot 5G communications, forming a communication network pattern that integrates the sky and the earth. They will also play an important role in the future 6G communications and gradually become an indispensable part of the communications field. Compared with high-orbit satellites, low-orbit satellites have the characteristics of short transmission delay, low transmission loss and high bandwidth, which are more advantageous in user communications. Starting with Iridium, low-orbit satellite communication systems have gradually moved towards large-scale, giant and multi-functional integration. Technology pioneers represented by the US SpaceX company have taken the lead in fully launching the construction of space-based Internet. The number of giant low-orbit constellations planned to be deployed worldwide is as high as 18 or more. my country is also promoting the planning and deployment of low-orbit giant constellations.

[0003] Satellite measurement and control technology tracks the orbit and position of satellites in low-orbit giant constellations, measures and monitors their operating attitude and the status of onboard equipment, and corrects and controls abnormal satellite working conditions. Satellite measurement and control technology is an important means for low-orbit giant constellations to maintain normal operation, increase reliability, and extend their operating life. Low-orbit constellation measurement and control is the control of the entire life cycle of satellites in orbit. There are many measurement and control targets, long measurement and control time, and many stages. A large number of satellites will also make the measurement process cumbersome. At the same time, the control efficiency requirements are high. The existing measurement and control modes and facilities cannot meet the supply and demand balance. Satellite tracking and control requires the establishment of a corresponding feeder link between the satellite and the ground tracking and control station so that the ground station can capture the operating status of the satellite. However, due to the high-speed movement of low-orbit satellites, the satellite-to-ground visibility window is short and changes rapidly. The maximum connection time with the ground tracking and control station is only more than ten minutes, and the link switching is frequent. Too much switching will lead to excessive signaling overhead and interruption rate. The tracking and control switching management of satellites in low-orbit constellations is very important. Therefore, it is necessary to design a flexible and efficient satellite-to-ground tracking and control link planning method to uniformly plan or select the ground tracking and control centers connected to all satellites in the constellation at different time periods during the entire operating cycle, so that the constellation tracking and control process has less overhead and is more stable.

[0004] Most of the existing low-orbit constellation satellite-to-ground tracking and control link planning methods consider small low-orbit constellations and fewer ground tracking and control station resources, and mostly use heuristic algorithms. However, the number of satellites in low-orbit giant constellations has increased exponentially, and there are more ground station resources, which will lead to more intense resource competition and more complex link planning and management. The performance of traditional heuristic algorithms is greatly challenged, and it is difficult to solve the problem of excessive dimensionality caused by the increase in satellites and tracking and control stations, which leads to a significant decline in planning effect.

[0005] Therefore, a new satellite-to-ground tracking and control link planning method for giant constellations is urgently needed to solve the above problems. Summary of the invention

[0006] In view of the technical problems that the existing satellite-to-ground link planning methods are inefficient, poor in effect, low in performance and poor in robustness when facing large-scale constellations and ground tracking and control stations, the present invention proposes a satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation. A learning environment is constructed according to the satellite-to-ground visible window, and reinforcement learning is used to realize the interaction between the intelligent agent and the environment, and a link planning model is trained, so as to optimize the number of switching times of the satellite-to-ground link in the process of constellation tracking and control, improve the utilization rate of ground station antenna resources, and realize fast, flexible and efficient planning of the satellite-to-ground link.

[0007] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a satellite-to-ground measurement and control link planning method based on reinforcement learning in a low-orbit constellation, characterized in that it includes:

[0008] Considering the satellite-to-ground link planning problem of low-orbit giant constellations, a multi-objective optimization model is established to maximize resource utilization and minimize the number of switching times while satisfying the satellite-to-ground visibility constraints, satellite link quantity constraints, antenna resource constraints, and variable feasible domain constraints.

[0009] The link planning problem of the entire operation cycle of the constellation is decoupled into the link planning problem within each discrete time slice. The deep Q-network algorithm in deep reinforcement learning is used to solve the multi-objective optimization model to complete the optimal giant constellation satellite-to-ground link planning decision. The intelligent agent makes the best action according to the visible satellite status at each allocation, selects the best satellite to establish the link, and thus completes the satellite-to-ground measurement and control link planning.

[0010] Preferably, the satellite-to-ground link planning problem includes a constellation operation cycle, multiple satellites, and multiple ground tracking and control stations. The entire link planning cycle is T divided into K time slices, and each time slice t k The size of each is T / K, and the time slice set is {t k |k=1,2,3,...,K}; Assume that the number of ground tracking and control stations is M, and the set of ground tracking and control stations G={g i |i=1,2,3,...,M},for each ground control station g i, assuming that it has L antenna resources, that is, it can establish tracking and control links with L satellites at the same time; the antenna set is An = {a j |j=1, 2, 3, ..., L}, a l represents the lth antenna; suppose the entire low-orbit constellation consists of N satellites, and the satellite set is defined as S = {s l |l=1,2,3,...,N},s l represents the lth satellite;

[0011] In each time slice t k In the example above, use the variable Indicates satellites l and ground control stationg i Whether it is always in the visible state, 0 means invisible, 1 means visible; ground control station g i In time slice t k The visible satellite sequence in where l∈[1,N].

[0012] Preferably, use the variable Indicates that in the planning process, at time slice t k Satellites l Is it connected to the ground control station g i Antenna a j Establish a connection, 0 means no connection is established, 1 means the connection is established; When planning the link, the following constraints need to be met: 1) Satellite-to-ground visibility constraint: If the satellite s l With ground control station i To establish a connection between antennas, they need to be visible to each other, that is, Where i∈[1, M], l∈[1, N]; 2) Satellite link quantity constraint: A satellite can only establish a connection with one antenna of a ground station in a time slice tk, that is, where i∈[1,M], l∈[1,N], k∈[1,K]; 3) Antenna resource constraint: A ground station can establish connections with at most L satellites in a time slice, that is, it needs to satisfy where k∈[1,K].

[0013] Preferably, the multi-objective optimization model is:

[0014] Maximize the antenna resources allocated to each ground tracking and control station: Among them, RU i Indicates the ground station g i The ratio of the allocated antennas to the total number of antennas;

[0015] At the same time, the number of link switches between different ground tracking and control stations within each satellite movement cycle is minimized, and the total number of constellation switches is minimized: Among them, SW i Represents ground station g i The number of switching of all antennas within the entire satellite-to-ground link planning period T.

[0016] Preferably, the method for decoupling the link planning problem is: first, in each time slice, the mth ground control station g m All antennas are initialized to an idle state, and then the link planning scheme within each time slice is studied according to the satellite-ground visible relationship in each time slice; the deep Q network algorithm in the deep reinforcement learning utilizes the powerful perception ability of deep learning to process high-dimensional data to extract features combined with the decision-making ability of reinforcement learning to explore interaction, so as to realize efficient and optimal planning of satellite-ground measurement and control links.

[0017] Preferably, the deep Q network algorithm in the deep reinforcement learning includes three elements: state space, action space, and reward function;

[0018] Among them, the state space: in time slice t k In the ground control station g i The visible satellite sequence is The visible satellite sequence length is LV, the state sequence In the set F(s l ) represents the state value of the corresponding satellite in the visible satellite sequence, F(s l )∈[0,5], 0 means that the satellite is invisible to the ground station, 1 means that the satellite has established a connection with other ground tracking and control stations, 2 means that the satellite was not connected to the ground tracking and control station in the previous time slice, and the satellite is visible to other ground tracking and control stations at the same time, 3 means that the satellite was not connected to the ground tracking and control station in the previous time slice, and the satellite is not visible to other ground tracking and control stations, 4 means that the satellite was connected to the ground tracking and control station in the previous time slice, and the satellite is visible to other ground tracking and control stations at the same time, 5 means that the satellite was connected to the ground tracking and control station in the previous time slice, and the satellite is not visible to other ground tracking and control stations;

[0019] Action space: In time slice t k For each ground control station g i , action space where [s1, s LV ] represents the visible satellite sequence of the ground tracking and control station, represents the filling satellite sequence of the ground tracking and control station; after the algorithm calculation and result screening, the corresponding satellite will be allocated to each antenna resource of the ground station, and finally the action set A = {A1, A2, A3, ..., A L}, Aj ∈Action indicates the assignment to the ground control station g i Antenna a j The visible satellites, the set length is equal to the number of antenna resources of each satellite;

[0020] The reward function is:

[0021]

[0022] Among them, R is the basic reward value, P is the penalty factor, and the satellite state value F(s l ) The higher the value corresponding to it, the better the state is, and the higher the reward value can be obtained by selecting this satellite; when the satellite state value F(s l ) is 0 or 1, Reward is set to -50. When the value is 0, the satellite-to-ground visibility constraint is not met. When the value is 1, the satellite link quantity constraint is not met.

[0023] Preferably, the length of the state sequence State is uniformly defined as the maximum length LV of all visible satellite sequences of the ground tracking and control stations. max , for visible satellite sequence length LV less than the maximum length LV max The state sequence of satellite states is filled, that is, F(s i ′), i∈[1, LV_max-LV], F(s i ′) is uniformly set to 0.

[0024] Preferably, the deep Q network algorithm in the deep reinforcement learning obtains the visible satellite sequence state information of each ground tracking and control station in each time slice, generates the corresponding action Q value sequence through the Q network, and generates the final action set to obtain rewards and update the model after filtering and screening the actions. The implementation method is:

[0025] Step 1: Initialize the weights and bias parameters of each layer of neurons in the evaluation network and target network in DQN;

[0026] Step 2: The agent obtains the current environment state information, including time slice t k Ground tracking and control station i Visible satellite sequence and the ground control station gi at the previous time slice t k-1 With satellite The connection status of satellites is given according to the definition of satellite state values ​​in the state space. l A state value F(s l ), and based on this, construct the state sequence of visible satellites As network input features;

[0027] Step 3: Input the state sequence State of the visible satellite into the evaluation network, and the evaluation network extracts features and outputs the Q value sequence corresponding to the action space The g-greedy strategy is used for action selection, that is, the satellite with the largest Q value is assigned to the antenna with probability ε, a satellite is randomly selected from the action space with probability 1-ε and assigned to the antenna, and the reward function is used to calculate the action a. select The reward Reward1 obtained is the action a in the state sequence State select Corresponding satellites select The state value F(s select ) is updated to 1, indicating that the satellite is selected, and a new satellite state sequence State′ is generated;

[0028] Step 4: Set the initial satellite state sequence State and the selected action a select、 The reward Reward1 and the new satellite state sequence State′ form a four-tuple (State, a select , Rewardl, State′) is stored in the buffer pool as a link planning experience; after completing all link planning in a time slice, the experience in the buffer pool is extracted, and the parameters of the evaluation network are updated by calculating the loss function and the gradient descent method;

[0029] Step 5: Repeat the training process of steps 2-4 until the evaluation network and the target network converge;

[0030] Step 6: Use the converged evaluation network model to plan and allocate the satellite-to-ground tracking and control links of the low-orbit giant constellation to obtain the final planning results.

[0031] Preferably, the evaluation network and the target network are a network with three fully connected layers, the first layer has 1024 neurons, the second layer has 512 neurons, the third layer has 256 neurons, the dimension of the input layer is the dimension of the state space, and the dimension of the output layer is the dimension of the action space; before training, the parameters of each layer are initialized, the weights of each layer are initialized to the values ​​generated in the normal distribution with a mean of 0 and a standard deviation of 0.3, and the bias items of each layer are initialized to a constant of 0.1; the activation functions used in the evaluation network and the target network are both Sigmoid functions;

[0032] The experience pool collects the link planning experience (State, a) obtained from each interaction between the agent and the environment. select, Reward, State′) are all stored in the replay memory unit. The capacity of the experience pool is 500. When the evaluation network needs to be updated, a batch of experiences is randomly extracted from the experience pool for parameter update, and a batch is set to 100 experiences. When 500 experiences are accumulated in the experience pool, the deep Q network algorithm starts the learning process and uses the data in the experience pool to update the parameters of the evaluation network after reorganization.

[0033] Preferably, the loss function is the mean square error between the target Q value and the current Q value, which is:

[0034] L(θ)=E[(Q target -Q(State, a select ;θ)) 2 ];

[0035] And Q target = Reward + γmax a′ Q(State′, a select ′;θ);

[0036] In the formula, θ is the parameter of the evaluation network, State is the current environment state, that is, the initial visible satellite state sequence; a select is the action taken in the current environment state State, that is, the satellite selected from the action space; Reward is the reward for taking action a select The immediate reward obtained; γ is the reward decay factor; State′ is the action a select The new state obtained after the transfer is the new visible satellite state sequence; a select ′ is the action taken in the new state s′. The agent updates the network parameters using the gradient descent method. After 200 learnings, the evaluation network parameters are assigned to the target network.

[0037] Compared with the prior art, the invention has the following beneficial effects: the invention first obtains the visible satellite sequence of the ground tracking and control station corresponding to the time slice according to the satellite-ground visible window, constructs the satellite state sequence according to the global information, uses the advantage of deep neural network in processing high-dimensional data to input the state sequence into the deep neural network output Q value sequence, generates an action set according to the number of antenna resources, matches the corresponding reward function according to the action, and stores it in the experience pool. After the experience in the experience pool reaches a certain number, randomly extracts experience training, calculates the loss function, completes the evaluation network parameter update, and assigns the evaluation network parameters to the target network after learning a certain number of times. The invention can complete the satellite-ground tracking and control link planning of large-scale constellations in a relatively short time, and has a lower number of link switching times, thereby having a lower switching overhead and interruption rate. In terms of antenna resource utilization, this method can also reach a very high level. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 Application scenario diagram for the planning of satellite-to-ground tracking and control links for low-orbit constellations.

[0040] Figure 2 It is the overall flow chart of the present invention.

[0041] Figure 3 It is a flowchart of the core part of the DQN algorithm in the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 2 As shown in the figure, a satellite-to-ground link planning method based on reinforcement learning in a low-orbit constellation is proposed, with the goal of maximizing the utilization of ground antenna resources and minimizing the number of satellite-to-ground measurement and control link switching. Considering the optimization of long-term performance, an intelligent planning and scheduling algorithm based on deep reinforcement learning is proposed, which specifically includes the following steps:

[0044] S1: Considering the satellite-to-ground link planning problem of low-orbit giant constellations, a multi-objective optimization model that maximizes resource utilization and minimizes the number of switching times is established while satisfying the satellite-to-ground visibility constraints, satellite link quantity constraints, antenna resource constraints, and variable feasible domain constraints.

[0045] The satellite-to-ground link planning problem includes the constellation operation cycle, multiple satellites, and multiple ground tracking and control stations. The entire link planning cycle is T, which is divided into K time slices. Each time slice t k The size of each is T / K, and the time slice set is {t k |k=1,2,3,...,K}. Figure 1 The figure shows the application scenario of the satellite-to-ground tracking and control link planning of the low-orbit constellation of the present invention, which includes multiple low-orbit satellites, multiple ground tracking and control stations, and the feeder link between the low-orbit satellites and the ground tracking and control stations. Assuming that the number of ground tracking and control stations is M, the ground tracking and control station set G is defined as {g m|m=1,2,3,...,M},for each ground control station g m , assuming that it has L antenna resources, that is, it can establish tracking and control links with L satellites at the same time, and define the antenna set as An = {a l |l=1,2,3,...,L},a l represents the lth antenna. Assuming that the entire low-orbit constellation consists of N satellites, the satellite set is defined as S = {s n |n=1,2,3,...,N},s j represents the jth satellite.

[0046] In each time slice t k In the example above, use the variable Indicates satellites i and ground control stationg j Whether it is always in the visible state, 0 means invisible, 1 means visible. j In time slice t k The visible satellite sequence within can be defined as where j∈[1,N].

[0047] Using variables To represent the time slice t in the planning process k Satellites j Is it connected to the ground control station g i Antenna a l Establish a connection, 0 means no connection is established, 1 means established connection. When planning the link, the following constraints need to be met: 1) Satellite-to-ground visibility constraint: If the satellite s j With ground control station i To establish a connection between antennas, they need to be visible to each other, that is, Where i∈[1,M], j∈[1,N]. 2) Constraint on the number of satellite links: A satellite can k A connection can only be established with one antenna of a ground station at most. Where i∈[1,M], j∈[1,N], k∈[1,K]. 3) Antenna resource constraint: A ground station can establish connections with at most L satellites in a time slice, that is, it needs to satisfy where k∈[1,K].

[0048] A multi-objective optimization model is established to maximize resource utilization and minimize the number of switching times, that is, to maximize the antenna resources allocated to each ground tracking and control station, which can be expressed as RU i Indicates the ground station g iThe ratio of the allocated antennas to the total number of antennas. At the same time, the number of link switches between different ground tracking and control stations in each satellite movement cycle is minimized, and the total number of switching times of the constellation is minimized, which can be expressed as Among them, SW i Represents ground station g i The number of switching of all antennas within the entire satellite-to-ground link planning period T.

[0049] S2: In order to more effectively solve the satellite-to-ground link planning problem of low-orbit giant constellations, the link planning problem of the entire constellation operation cycle is decoupled into the link planning problem within each discrete time slice. In each time slice, the ground station g i All antennas are initialized to an idle state, and then the link planning scheme in each time slice is studied according to the satellite-ground visible relationship in each time slice. The deep Q network algorithm in deep reinforcement learning is used to solve the multi-objective optimization model proposed in step S1 to complete the optimal giant constellation satellite-ground link planning decision. The intelligent agent can make the best action according to the visible satellite state at each allocation, select the best satellite to establish a link, and thus complete the satellite-ground measurement and control link planning.

[0050] The link planning algorithm used is the deep Q network (DQN) algorithm in deep reinforcement learning. It uses the powerful perception ability of deep learning to process high-dimensional data to extract features and the decision-making ability of reinforcement learning to explore interactions to achieve efficient and optimal planning of satellite-to-ground measurement and control links. The algorithm requires three elements: state space, action space, and reward function, which are defined as follows:

[0051] State space: At time slice t k In the ground control station g i The visible satellite sequence is The visible satellite sequence length is LV, defined as In the set F(s l ) represents the state value of the corresponding satellite in the visible satellite sequence, F(s l )∈[0,5], 0 means that the satellite is invisible to the ground station, 1 means that the satellite has established a connection with other ground tracking and control stations, 2 means that the satellite was not connected to the ground tracking and control station in the previous time slice, and the satellite is visible to other ground tracking and control stations at the same time, 3 means that the satellite was not connected to the ground tracking and control station in the previous time slice, and the satellite is not visible to other ground tracking and control stations, 4 means that the satellite was connected to the ground tracking and control station in the previous time slice, and the satellite is visible to other ground tracking and control stations at the same time, 5 means that the satellite was connected to the ground tracking and control station in the previous time slice, and the satellite is not visible to other ground tracking and control stations. Since the length of the visible satellite sequence of each ground tracking and control station is different, in order to facilitate the neural network training in the algorithm and reduce the complexity of the model, the length of the state sequence State is uniformly defined as the maximum length LV of the visible satellite sequence of all ground tracking and control stations.max , for visible satellite sequence length LV less than the maximum length LV max The state sequence of satellite states, we fill in the satellite states, that is, F(s i ′), i∈[1, LV_max-LV], F(s i ′) is uniformly set to 0.

[0052] Action space: In time slice t k For each ground control station g i , define the action space where [s1, s LV ] represents the visible satellite sequence of the ground tracking and control station, It represents the filling satellite sequence of the ground tracking and control station, which is consistent with the filling satellite defined in the state space. After the algorithm calculation and result screening, the corresponding satellite will be allocated to each antenna resource of the ground station, and finally the action set A = {A1, A2, A3, ..., A L}, A j ∈Action indicates the assignment to ground station g i Antenna a j The set length is equal to the number of antenna resources of each satellite.

[0053] Reward function: The state value F(s) of the satellite in the visible satellite sequence is defined in the state space. l ), since selecting a satellite that was connected to the ground control station in the previous time slice can reduce the number of switching times, selecting a satellite that is not visible to other ground control stations in this time slice can increase the allocation success rate and ultimately improve the antenna resource utilization. Therefore, the reward function is specifically defined as follows:

[0054]

[0055] Among them, R is the basic reward value, which defines the upper and lower limits of the reward for selecting a satellite that meets the link building constraints. The basic reward value R is taken as 20, P is the penalty factor, and P is taken as -5. The satellite status value F(s l ) The higher the value corresponding to it, the better the state is, and the higher the reward value can be obtained by selecting this satellite. l ) takes the value of 0 or 1, and Reward is set to -50. When the value is 0, the satellite-to-ground visibility constraint is not met. When the value is 1, the satellite link quantity constraint is not met. In these two cases, it is considered that the connection cannot be established, so selecting this satellite will get a very low reward value. By setting this reward function, ground stations can be encouraged to select better satellites to improve antenna resource utilization and reduce the number of switching times.

[0056] Combination Figure 2As shown in the figure, the DQN algorithm used obtains the visible satellite sequence state information of each ground tracking station in each time slice, generates the corresponding action Q value sequence through the Q network in the algorithm, and then generates the final action set to obtain rewards and update the model after filtering and screening the actions. The specific steps include the following:

[0057] Step 1: Initialize the weights and bias parameters of each layer of neurons in the evaluation network and target network in DQN to accelerate the convergence speed of the evaluation network training and improve the convergence effect.

[0058] Step 2: The agent obtains the current environment state information, including the time slice t k Medium Ground Station i Visible satellite sequence and ground station g i In the previous time slice t k-1 With satellites l The connection status, According to the definition of satellite state value in state space, satellite s in the visible satellite sequence is assigned l A state value F(s l ), and based on this, construct the visible satellite state sequence As network input features.

[0059] Step 3: Input the visible satellite state sequence State into the evaluation network, and the evaluation network extracts features and outputs the Q value sequence corresponding to the action space It reflects the value of each action in the action space. The ε-greedy strategy is used for action selection, that is, the satellite with the largest Q value is assigned to the antenna with a probability of ε, and a satellite is randomly selected from the action space with a probability of 1-ε and assigned to the antenna. The reward function is used to calculate the probability of taking action a. select The reward obtained is Reward, which changes the action a in State select Corresponding satellites select The state value F(s select ) is updated to 1, indicating that the satellite has been selected, and a new satellite state sequence State′ is generated.

[0060] Step 4: Initial satellite state sequence State, selected action a select、 The reward obtained and the new satellite state sequence State′ form a four-tuple (State, a select, Reward, State′) is stored in the buffer pool as a link planning experience. After completing all link planning in a time slice, the experience in the buffer pool is extracted, and the parameters of the evaluation network are updated by calculating the loss function and the gradient descent method.

[0061] Step 5: Repeat the training process of steps 2-4 until the evaluation network and the target network converge.

[0062] Step 6: Use the converged evaluation network model to plan and allocate the satellite-to-ground tracking and control links of the low-orbit giant constellation to obtain the final planning results.

[0063] Figure 3 The schematic diagram of the flow chart of the core part of the DQN algorithm adopted by the present invention, firstly, the intelligent agent obtains the satellite state sequence State by interacting with the environment and inputs it into the evaluation network, the DQN algorithm uses the structure of dual neural networks, namely the evaluation network and the target network, the network structure and initial parameters of the two networks are the same, and the difference is that the evaluation network generates the Q value corresponding to the current network parameter, and the network parameters are updated once for each learning during training, the role of the target network is to increase the stability of the algorithm, with the continuous updating of the network parameters, the target Q value is also constantly updated, which will cause the training to be easy to oscillate and diverge, so using a target network to fix the target Q value can increase the stability of learning, the target network generates the target Q value, but it is not updated every time learning, but after a certain number of learning times, the parameters of the evaluation network are directly assigned to the target parameters. The use of a dual network structure can speed up the convergence speed of the evaluation network and improve the stability of the algorithm.

[0064] In the present invention, both the evaluation network and the target network are defined as a network with three fully connected layers, the first layer has 1024 neurons, the second layer has 512 neurons, and the third layer has 256 neurons. The dimension of the input layer is the dimension of the state space, and the dimension of the output layer is the dimension of the action space. In order to make the evaluation network model converge faster and more effective, each layer is parameter initialized before training, and the weight of each layer is initialized to a value generated from a normal distribution with a mean of 0 and a standard deviation of 0.3. The bias term of each layer is initialized to a constant of 0.1. The weight initialization can activate the output of an overly large or undersized value of the function when the evaluation network and the target network are forward-transferred, so as to avoid gradient explosion or disappearance. The activation functions used in the evaluation network and the target network are both Sigmoid functions, which are monotonically continuous and can control the output value within the interval of [0, 1], so that the data is not easy to diverge during the transmission process of the two networks.

[0065] The environmental state information is evaluated through the network to generate a Q value sequence corresponding to the action space. The DQN algorithm uses the ε-greedy strategy when selecting actions. The traditional greedy strategy will directly select the action with the largest Q value, which will cause the agent to be unable to explore the environment more and easily fall into the local optimal solution. The ε-greedy strategy randomly selects actions from the Q value sequence with a probability of ε, and selects the action with the largest Q value with a probability of 1-ε. This allows the agent to have a more comprehensive understanding of the environmental information, so that it can accumulate more experience to optimize the model to obtain the maximum reward value.

[0066] The DQN algorithm introduces an experience pool, which is the link planning experience (State, a select , Reward, State′) are stored in the replay memory unit, and the capacity of the experience pool is set to 500. When the evaluation network needs to be updated, a batch of experience is randomly extracted from the experience pool for parameter update, and a batch is set to 100 experiences. This can avoid the influence of the timing correlation of the input data state, so that each historical experience can be used to improve the training effect.

[0067] When 500 experiences are accumulated in the experience pool, the algorithm can start the learning process, that is, use the data in the experience pool to update the parameters of the evaluation network after reorganization. The loss function of DQN is expressed as the mean square error between the target Q value and the current Q value. The specific definition of the loss function is as follows:

[0068] L(θ)=E[(Q target -Q(State, a select ;θ)) 2 ];

[0069] in:

[0070] Q target = Reward + γmax a′ Q(State′, a select ′;θ);

[0071] Where θ is the parameter of the evaluation network, State is the current environment state, i.e., the initial visible satellite state sequence; a select is the action taken in the current environment state State, that is, the satellite selected from the action space; Reward is the reward for taking action a select The immediate reward obtained; γ is the reward decay factor; used to calculate the long-term benefits of the action; State ′ is the time taken to take action a select The new state obtained after the transfer is the new visible satellite state sequence; a select′ is the action taken in the new state s′. The agent updates the network parameters using the gradient descent method. After 200 learnings, the evaluation network parameters are assigned to the target network.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation, characterized in that: include: Considering the satellite-to-ground link planning problem of low-orbit giant constellations, a multi-objective optimization model is established to maximize resource utilization and minimize the number of switching times while satisfying the satellite-to-ground visibility constraints, satellite link quantity constraints, antenna resource constraints, and variable feasible domain constraints. The link planning problem of the entire operation cycle of the constellation is decoupled into the link planning problem within each discrete time slice. The deep Q network algorithm in deep reinforcement learning is used to solve the multi-objective optimization model to complete the optimal giant constellation satellite-to-ground link planning decision. The intelligent agent makes the best action according to the visible satellite status at each allocation, selects the best satellite to establish the link, and thus completes the satellite-to-ground measurement and control link planning. The deep Q network algorithm in the deep reinforcement learning obtains the visible satellite sequence state information of each ground tracking and control station in each time slice, generates the corresponding action Q value sequence through the Q network, and generates the final action set to obtain rewards and update the model after filtering and screening the actions. The implementation method is: Step 1: Initialize the weights and bias parameters of each layer of neurons in the evaluation network and target network in DQN; Step 2: The agent obtains the current environment state information, including time slice t k Ground tracking and control station i Visible satellite sequences And ground control station i In the previous time slice t k-1 With satellite The connection status of the satellite is assigned to the satellite s in the visible satellite sequence according to the definition of the satellite state value in the state space. l A state value F(s l ), and based on this, construct the state sequence of visible satellites As the network input features; among them, LV max is the maximum length of the visible satellite sequence Step 3: Input the state sequence State of the visible satellite into the evaluation network, and the evaluation network extracts features and outputs the Q value sequence corresponding to the action space The ε-greedy strategy is used for action selection, that is, the satellite with the largest Q value is assigned to the antenna with probability ε, a satellite is randomly selected from the action space with probability 1-ε and assigned to the antenna, and the reward function is used to calculate the action a. select The reward Reward1 obtained is the action a in the state sequence State select Corresponding satellites select The state value F(s select ) is updated to 1, indicating that the satellite is selected, and a new satellite state sequence State' is generated; Step 4: Set the initial satellite state sequence State and the selected action a select , the reward Reward1 and the new satellite state sequence State' form a four-tuple (State, a select , Reward1, State') is stored in the buffer pool as a link planning experience; after completing all link planning in a time slice, the experience in the buffer pool is extracted, and the parameters of the evaluation network are updated by calculating the loss function and the gradient descent method; Step 5: Repeat the training process of steps 2-4 until the evaluation network and the target network converge; Step 6: Use the converged evaluation network model to plan and allocate the satellite-to-ground tracking and control links of the low-orbit giant constellation to obtain the final planning results.

2. The satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation according to claim 1, characterized in that: The satellite-to-ground link planning problem includes a constellation operation cycle, multiple satellites, and multiple ground tracking and control stations. The entire link planning cycle is T, which is divided into K time slices. Each time slice t k The size of each is T / K, and the time slice set is {t k |k=1,2,3,...,K}; Assume that the number of ground tracking and control stations is M, and the set of ground tracking and control stations G={g i |i=1,2,3,...,M},for each ground control station g i , assuming that it has L antenna resources, that is, it can establish tracking and control links with N satellites at the same time; the antenna set is An = {a j |j=1, 2, 3, ..., L}, a j represents the jth antenna; suppose the entire low-orbit constellation consists of N satellites, and the satellite set is defined as S = {s l |l=1,2,3,...,N},s l represents the lth satellite; In each time slice t k In the example above, use the variable Indicates satellites l and ground control stationg i Whether it is always in the visible state, 0 means invisible, 1 means visible; ground control station g i In time slice t k The visible satellite sequence in where l∈[1,N].

3. The satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation according to claim 2, characterized in that: Using variables Indicates that in the planning process, at time slice t k Satellites l Is it connected to the ground control station g i Antenna a j Establish a connection, 0 means no connection is established, 1 means the connection is established; When planning the link, the following constraints need to be met: 1) Satellite-to-ground visibility constraint: If the satellite s l With ground control station i To establish a connection between antennas, they need to be visible to each other, that is, Where i∈[1, M], l∈[1, N]; 2) Satellite link quantity constraint: A satellite can only establish a connection with one antenna of a ground station in a time slice tk, that is, where i∈[1,M], l∈[1,N], k∈[1,K]; 3) Antenna resource constraint: A ground station can establish connections with at most L satellites in a time slice, that is, it needs to satisfy where k∈[1,K].

4. The satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation according to claim 3, characterized in that: The multi-objective optimization model is: Make the antenna resources allocated to each ground tracking and control station the most: Among them, RU i Indicates the ground station g i The ratio of the allocated antennas to the total number of antennas; At the same time, the number of link switches between different ground tracking and control stations within each satellite movement cycle is minimized, and the total number of constellation switches is minimized: Among them, SW i Represents ground station g i The number of switching of all antennas within the entire satellite-to-ground link planning period T.

5. The satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation according to any one of claims 2 to 4, characterized in that: The method for decoupling the link planning problem is as follows: First, the mth ground control station g m All antennas are initialized to an idle state, and then the link planning scheme within each time slice is studied according to the satellite-ground visible relationship in each time slice; the deep Q network algorithm in the deep reinforcement learning utilizes the powerful perception ability of deep learning to process high-dimensional data to extract features combined with the decision-making ability of reinforcement learning to explore interaction, so as to realize efficient and optimal planning of satellite-ground measurement and control links.

6. The satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation according to claim 5, characterized in that: The deep Q network algorithm in deep reinforcement learning includes three elements: state space, action space, and reward function; Among them, the state space: in time slice t k In the ground control station g i The visible satellite sequence is The visible satellite sequence length is LV, and the state sequence State={F(s1), F(s2), ..., F(s LV ), F(s1'), F(s2'), ..., In the set F(s l ) represents the state value of the corresponding satellite in the visible satellite sequence, F(s l )∈[0,5], 0 means that the satellite is invisible to the ground station, 1 means that the satellite has established a connection with other ground tracking and control stations, 2 means that the satellite was not connected to the ground tracking and control station in the previous time slice, and the satellite is visible to other ground tracking and control stations at the same time, 3 means that the satellite was not connected to the ground tracking and control station in the previous time slice, and the satellite is not visible to other ground tracking and control stations, 4 means that the satellite was connected to the ground tracking and control station in the previous time slice, and the satellite is visible to other ground tracking and control stations at the same time, 5 means that the satellite was connected to the ground tracking and control station in the previous time slice, and the satellite is not visible to other ground tracking and control stations; Action space: In time slice t k For each ground control station g i , action space where [s1, s LV ] represents the visible satellite sequence of the ground tracking and control station, represents the filling satellite sequence of the ground tracking and control station; after the algorithm calculation and result screening, the corresponding satellite will be allocated to each antenna resource of the ground station, and finally the action set A = {A1, A2, A3, ..., A L }, A j ∈Action indicates the assignment to the ground control station g i Antenna a j The visible satellites, the set length is equal to the number of antenna resources of each satellite; The reward function is: Among them, R is the basic reward value, P is the penalty factor, and the satellite state value F(s l ) The higher the value corresponding to it, the better the state is, and the higher the reward value can be obtained by selecting this satellite; when the satellite state value F(s l ) is 0 or 1, Reward is set to -50. When the value is 0, the satellite-to-ground visibility constraint is not met. When the value is 1, the satellite link quantity constraint is not met.

7. The satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation according to claim 6, characterized in that: The length of the state sequence State is uniformly defined as the maximum length LV of the visible satellite sequence of all ground tracking and control stations. max , for visible satellite sequence length LV less than the maximum length LV max The state sequence of satellite states is filled, that is, F(s i '), i∈[1, LV_max-LV], F(s i ') is uniformly set to 0.

8. The method for planning satellite-to-ground tracking and control links in a low-orbit constellation based on reinforcement learning according to any one of claims 1, 6 or 7, characterized in that: The evaluation network and the target network are a network with three fully connected layers, the first layer has 1024 neurons, the second layer has 512 neurons, the third layer has 256 neurons, the dimension of the input layer is the dimension of the state space, and the dimension of the output layer is the dimension of the action space; before training, the parameters of each layer are initialized, the weights of each layer are initialized to the values ​​generated in the normal distribution with a mean of 0 and a standard deviation of 0.3, and the bias items of each layer are initialized to a constant of 0.1; the activation functions used in the evaluation network and the target network are both Sigmoid functions; The experience pool collects the link planning experience (State, a) obtained from each interaction between the agent and the environment. select , Reward, State′) are all stored in the replay memory unit. The capacity of the experience pool is 500. When the evaluation network needs to be updated, a batch of experiences is randomly extracted from the experience pool for parameter update, and a batch is set to 100 experiences. When 500 experiences are accumulated in the experience pool, the deep Q network algorithm starts the learning process and uses the data in the experience pool to update the parameters of the evaluation network after reorganization.

9. The satellite-to-ground tracking and control link planning method based on reinforcement learning in a low-orbit constellation according to claim 8, characterized in that: The loss function is the mean square error between the target Q value and the current Q value, which is: L(θ)=E[(Q target -Q(State,a select ;θ)) 2 ]; And Q target = Reward + γ max a , Q(State′, a select ′; θ); In the formula, θ is the parameter of the evaluation network, State is the current environment state, that is, the initial visible satellite state sequence; a select is the action taken in the current environment state State, that is, the satellite selected from the action space; Reward is the reward for taking action a select The immediate rewards received; γ is the reward decay factor; State′ is the action a select The new state obtained after the transfer is the new visible satellite state sequence; a select ′ is the action taken in the new state s′; the agent uses the gradient descent method to update the network parameters, and after 200 learning times, the evaluation network parameters are assigned to the target network.

Citation Information

Patent Citations

  • Construction and maintenance of satellite-to-ground and inter-satellite laser communication network based on ants colony algorithm

    AU2020102041A4

  • Low-orbit Internet constellation satellite-ground link planning method based on constraint satisfaction

    CN113612525A

  • 5G satellite-ground link multi-beam dynamic power allocation method based on reinforcement learning

    CN114599099A