Satellite pursuit and evasion method and system based on track action prediction

Abstract

This invention discloses a reinforcement learning-based satellite pursuit and escape method and system based on orbital maneuver prediction. The method constructs a SAC policy network, designs a suitable loss function for backpropagation, and trains the initial policy of the model. A curiosity mechanism is introduced into the SAC algorithm, using the error between the prediction results and the actual values ​​from a fully connected network as a reward signal to encourage the agent to explore the action space. An input vector format and dataset for Transformer training are constructed, and the input vectors are positionally encoded upon input. A suitable loss function is designed for backpropagation of the Transformer. The Transformer's output is used as the input to the SAC algorithm, and the SAC observations are reconstructed, forming an integrated decision network. This invention solves the game problem of satellite pursuit and escape using pulse maneuvers.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence satellite orbit pursuit and escape application, specifically involving a reinforcement learning satellite pursuit and escape method and system based on orbital motion prediction. Background Technology

[0002] Satellites, when orbiting in space, are subject to perturbations from various celestial bodies, rendering the simple two-body motion mathematical model used to describe satellite motion inapplicable. In addition to the gravitational pull from Earth's oblateness, the gravitational effects of other celestial bodies must also be considered, making the mathematical model of satellite motion extremely complex. In the field of satellite motion modeling, traditional methods construct mathematical expressions based on physical models and solve the models through optimization techniques. However, this traditional approach has two significant drawbacks: firstly, it typically relies on heuristic algorithms such as particle swarm optimization, which consume considerable time during iterative solutions, making it difficult to meet the high real-time requirements of satellite game scenarios; secondly, traditional methods only focus on the two-body model and do not incorporate the perturbation effects of other celestial bodies into the calculation, resulting in limited model accuracy. In recent years, deep learning, as one of the core technologies in the field of artificial intelligence, has experienced rapid development. Its core mechanism is to simulate the operating mode of neurons in the human brain, accurately extracting key features from complex data. Reinforcement learning, as an important branch of deep learning, has gradually become a key component of task decision-making algorithms. The core idea of ​​this technology is to allow the agent to continuously interact with the environment and learn through trial and error, obtaining the optimal decision-making strategy through iterative optimization. This method does not rely on specific mathematical models, effectively addressing the complexities of satellite motion models and meeting the stringent real-time requirements of satellite decision-making scenarios. In recent years, some scholars have applied reinforcement learning to orbital pursuit and escape, but these methods have not considered that the escape satellite's strategy changes during its evolution, leading to significant variance in the pursuer's strategy and making convergence difficult. Summary of the Invention

[0003] The purpose of this invention is to provide a reinforcement learning-based satellite pursuit and escape method and system based on orbital motion prediction, which solves the game problem of satellites using pulse maneuvers to change orbits and pursue each other, and realizes that the tracking satellite continuously approaches the escape satellite during the game process, making more accurate decisions.

[0004] The technical solution to achieve the purpose of this invention is as follows:

[0005] A reinforcement learning-based satellite escape tracking method based on orbital motion prediction includes:

[0006] Step S1: Design the state observations and rewards of the two parties in a single-agent game as inputs and environmental feedback values ​​for the agents;

[0007] Step S2: Using state observations as input, construct and train the initial SAC policy network, construct the loss function, and introduce a curiosity mechanism as an external signal input to the reward.

[0008] Step S3: Construct the Transformer model input vector based on the pursuer's observation and the escaper's strategy, encode the vector position, and generate a sequence as the Transformer model's dataset;

[0009] Step S4: Train the Transformer model using the constructed dataset and design a loss function for backpropagation;

[0010] Step S5: Using the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstruct the input state observation of the SAC policy network, build an integrated decision network, train it together, design a loss function for backpropagation, and obtain an iterative training model.

[0011] Step S6: Use the trained integrated decision network to obtain the escaped satellite's orbit change amount, and execute the orbit change action based on the orbit change amount of the pursuing satellite.

[0012] Furthermore, the state observation includes the position and velocity of the reference satellite in the Earth's inertial coordinate system, the motion description of the pursuing satellite in the reference coordinate system, the motion description of the escaping satellite in the reference coordinate system, and the escaping satellite's escape maneuvers.

[0013] Furthermore, the curiosity mechanism employs a four-layer multilayer perceptron network to predict the features of the next state based on the current state features and the action chosen by the agent, calculates the error between the predicted value and the actual value, and converts the prediction error into an intrinsic reward. Meanwhile, the prediction error is used as the loss value to update the curiosity network.

[0014] Furthermore, the reward for the pursuer, who incorporates a curiosity mechanism, is:

[0015] ;

[0016] in It is the distance between satellites after time t following the execution of the corresponding action. This refers to the distance between satellites after time t if the action has not been taken. This indicates the reward for the pursuing party successfully catching up with the fleeing party. As a reward for curiosity, Indicates weight;

[0017] The reward for the fleeing party is the opposite of the reward for the pursuing party.

[0018] Furthermore, the aforementioned for:

[0019] .

[0020] Furthermore, the SAC policy network introduces entropy regularization into the Actor-Critic framework, and simultaneously employs a dual-Critic network and a target network soft update mechanism.

[0021] Furthermore, the loss function in step S2 includes the Actor loss function and the Critic network loss function, wherein,

[0022] The Actor loss function is:

[0023] ;

[0024] in, Indicates the state From experience replay pool D, action The expectation obtained from the sampling process of the current Actor policy. Indicates the state Next action The value of the action, These are the parameters of the Critic network. This is the temperature coefficient (hyperparameter), used to balance the policy's reward and entropy. The current Actor's strategy is to choose an action in state s. The probability of;

[0025] The loss function of the Critic network is:

[0026] ;

[0027] ;

[0028] ;

[0029] in, Value function network The loss, These are the parameters of the value function network. For the current Critic network, the current state Next action Value prediction; For state Execute action Even if there is a reward later, The discount factor is set to 0.99. The next state Next, strategy Sampling action Seeking expectations, Take two Q networks ( and The minimum value output. It is the SAC entropy regularization term. It is in state Choose below The probability, It is the temperature coefficient. Adding this entropy regularization term can encourage agents to explore, achieving the dual goals of maximizing rewards and maximizing entropy. for Network loss, and The calculations are similar, but the difference lies in... use As a future revenue predictor, for the current state Next action The value prediction is then performed. Finally, the minimum of the two losses is taken to reduce the overestimation problem.

[0030] Furthermore, the Transformer model incorporates all Value projection parameter matrices from the two multi-head self-attention mechanisms in the encoder, the multi-head self-attention mechanism in the decoder, and the masked self-attention mechanism. The data is uniformly replaced with a fixed identity matrix and does not participate in the model training process; at the same time, a sliding window is set to eliminate old data in the Transformer model's dataset that does not conform to the data distribution.

[0031] Furthermore, the loss function of the Transformer model is:

[0032] ;

[0033] The result predicted by the Transformer, and This is real trajectory data.

[0034] A reinforcement learning-based satellite escape tracking system based on orbital motion prediction includes:

[0035] The state observation and reward design unit designs the state observation and reward of both sides in a single-agent game as input to the agent and feedback to the environment.

[0036] The SAC policy network training unit takes state observations as input, constructs and trains the initial SAC policy network, builds the loss function, and introduces a curiosity mechanism as an external signal input to the reward.

[0037] The dataset construction unit constructs input vectors for the Transformer model based on the pursuer's observations and the escaper's strategy, encodes the vectors at their positions, and generates sequences as the dataset for the Transformer model.

[0038] The Transformer model training unit uses the constructed dataset to train the Transformer model and designs a loss function for backpropagation.

[0039] The integrated decision network construction unit uses the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstructs the SAC input state observation, builds the integrated decision network, and trains it together. A loss function is designed for backpropagation to obtain an iterative training model.

[0040] The output unit uses a trained integrated decision network to obtain the escaped satellite's orbit change amount, and then executes the orbit change action based on the orbit change amount of the pursuing satellite.

[0041] Compared with existing technologies, the beneficial effects of this invention are as follows: First, it fully utilizes the spatiotemporal characteristics of orbital motion, enabling the pursuer to predict the escapee's actions and adopt more effective pursuit strategies, thereby increasing the model's pursuit success rate and accelerating the model's convergence speed; Second, by using the SAC algorithm and incorporating a curiosity mechanism into the reward, it encourages the agent to explore action spaces that have never been explored before, avoiding the model from getting trapped in local optima; Third, by using a lightweight Transformer, it accelerates the satellite's inference speed and meets the real-time requirements of satellite pursuit. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0043] Figure 1 This is a flowchart of a reinforcement learning-based satellite pursuit method based on Transformer orbital motion prediction.

[0044] Figure 2 A flowchart for calculating curiosity rewards.

[0045] Figure 3 This is the input and output design diagram for the Transformer model.

[0046] Figure 4 Lightweighting of the multi-head attention mechanism for Transformer models.

[0047] Figure 5A diagram illustrating the process of collecting and updating data for the Transformer.

[0048] Figure 6 This is a general framework diagram of a reinforcement learning-based satellite pursuit method based on Transformer orbital motion prediction.

[0049] Figure 7 This is a flowchart of the reinforcement learning satellite pursuit method based on Transformer orbital motion prediction.

[0050] Figure 8 This is a local situation simulation diagram based on the algorithm of this invention. Detailed Implementation

[0051] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments represent only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0052] To more clearly describe the structure and function of the present invention, the following description is provided in conjunction with the accompanying drawings and specific embodiments.

[0053] The satellite pursuit scenario refers to satellites from different camps changing their orbits through orbital maneuvers. The pursuing satellite chases the fleeing satellite, while the fleeing satellite will also adopt certain strategies to move away from the pursuing satellite.

[0054] First, we model the satellite pursuit process, which can be viewed as a Markov decision problem. From a temporal perspective, it consists of three parts: past, present, and future. A Markov problem is one where the future state depends solely on the decisions made in the present, and is independent of past decisions. The four main elements in a Markov decision process are: S (state), A (action), P (state transition), and R (reward function). The agent's goal is to find a policy that, given the current state S, takes action A, and under the probability distribution of state transition P, maximizes the expected value of future R. P (state transition) is provided by the environment; we first design the agent's S, A, and R.

[0055] For state observation S, the present invention first sets a reference satellite in orbit. The reference satellite does not perform any actions and only operates in a fixed orbit. The significance of setting this satellite is to provide a reference for other satellites so as to establish a reference coordinate system based on this satellite to describe the local motion of the satellite.

[0056] The state input of the pursuing satellite consists of four parts: the first part is the position and velocity of the reference satellite in the Earth's inertial coordinate system; the second part is the motion description of the pursuing satellite in the reference coordinate system; the third part is the motion description of the escaping satellite in the reference coordinate system; and the fourth part is the escape action of the escaping satellite, which is predicted and filled in by the Transformer. The reason for this design is to separate the satellite's motion relative to the Earth and relative to the reference satellite, so that the satellite can focus more on the local motion between satellites and improve the satellite's learning efficiency in the pursuit. The state input of the escaping satellite is similar to that of the pursuing satellite: the first part is the description of the reference satellite in the inertial coordinate system; the second part is the description of the escaping satellite in the reference coordinate system; and the third part is the description of the pursuing satellite in the reference coordinate system.

[0057] Because the first and third parts of the observations are in different coordinate systems, the data distribution described by these different coordinate systems differs significantly. The data scale described in the inertial coordinate system is approximately in the range of [-42157, 42157], while the data scale described in the reference coordinate system is approximately in the range of [-200, 200]. If this uneven distribution is not addressed, the agent may not learn effective data. Therefore, before formally inputting the data into the agent, fuzzy normalization of the observations is necessary. This invention employs z-score normalization. First, a large amount of satellite data is collected, and the mean and variance of each quantity are statistically analyzed. Then, the distribution of each dimension is standardized, with the mean constrained to 0 and the variance constrained to 1.

[0058] For action A, the satellite's orbital maneuvering can be categorized into two types: continuous low-thrust maneuvering and high-thrust pulse maneuvering. This invention targets orbital satellites and is suitable for using high-thrust pulse maneuvers to instantaneously change the satellite's velocity and complete its orbital transfer. Therefore, the satellite's maneuvering space is... , which are the velocity increments along the x, y, and z axes in the inertial coordinate system, respectively.

[0059] Regarding the reward R, since the goal of this invention is to bring the pursuer closer to the escaper, the distance between the pursuer and the escaper must be as short as possible. However, the escaper's goal is to increase the distance between itself and the pursuer. Therefore, this invention must use distance as part of the reward.

[0060] However, in satellite tracking, due to the complex coupling of satellite motion, focusing solely on the current distance to the satellite to generate rewards is insufficient to achieve convergence for the satellite agent. This invention, based on traditional orbital relative kinematics, introduces the CW equation and incorporates it into the reward function design to predict the future state of the satellite. The specific implementation method is as follows:

[0061]

[0062] Rewrite the CW equations in the form of state transitions, This is the initial state. The state transition matrix is ​​used to deduce the state after time t. The specific form of the state transition matrix is ​​as follows:

[0063]

[0064] Based on the above state transition equations, the reward function can be designed as follows:

[0065]

[0066] in It is the distance between satellites after time t following the execution of the corresponding action. This refers to the distance between satellites after time t if the action had not been taken. It indicates whether the distance between satellites will decrease in the future after the action is taken. If it decreases, a positive reward is given; otherwise, a negative reward is given. This means that if the pursuing party successfully catches up with the fleeing party, it signifies mission completion and a large reward will be given, as detailed below:

[0067]

[0068] The reward function of the escaping satellite is the opposite of that of the pursuing satellite, and the game between the escaping and pursuing satellites constitutes a zero-sum game.

[0069] SAC (Soft Actor-Critic) is an offline policy reinforcement learning algorithm suitable for continuous action spaces. Its core idea revolves around "maximum entropy reinforcement learning": by introducing entropy regularization into the traditional Actor-Critic framework, the algorithm aims not only to maximize the cumulative environment reward but also to encourage the randomness of the policy (the higher the entropy, the stronger the exploration), thereby balancing exploration and exploitation and avoiding premature entrapment in local optima. At the same time, it adopts a dual-Critic network and a soft update mechanism for the target network to effectively alleviate the overestimation problem of the value function and improve training stability. Combined with the experience replay pool to reuse historical data, it has high sample efficiency and performs well in complex continuous control tasks.

[0070] The loss function of the SAC algorithm is shown below:

[0071]

[0072] Critic uses a double Q-network, and its loss function is:

[0073]

[0074] In satellite game simulations, the satellite has a large action space, and relying solely on the SAC algorithm can easily lead the model into local optima. Therefore, this invention introduces a curiosity mechanism to incorporate exploration into the reward function, rewarding the agent when it performs an action it has never performed before. The reward.

[0075] The curiosity mechanism is a core technology that simulates the human instinct to "actively explore the unknown." Its core value lies in solving a key bottleneck in traditional learning reinforcement learning (RL)—the exploration-exploitation trade-off: preventing the agent from getting trapped in local optima due to excessive "exploitation of known high-yield strategies," while simultaneously driving it to actively explore unexplored areas of the environment, ultimately achieving a globally optimal learning objective. It fills the learning motivation gap when "external rewards are sparse or absent" by constructing "intrinsic rewards," allowing the agent to continue exploring and learning even in scenarios without explicit feedback. This invention sets up a four-layer multilayer perceptron network that predicts the features of the next state based on the current state features and the agent's chosen action, and calculates the error between the predicted and actual values. The larger the error, the stronger the curiosity. The prediction error is converted into intrinsic rewards, forming the agent's final reward signal.

[0076] like Figure 2 As shown, the current state is known. and current state The two are then input into the curiosity network and the orbital environment simulation, respectively, to obtain the predicted state for the next moment. and the actual state The calculated prediction error is used as a reward signal and input into reinforcement learning. Furthermore, this prediction error is used as the loss value to update the curiosity network.

[0077]

[0078] Since the escape agent and the pursuer agent are trained together, the pursuer satellite cannot fully predict the escape actions of the escaper satellite during the strategy evolution process. Therefore, it is necessary to design a method that can predict the next action of the escaper satellite based on the current and past actions. The pursuer satellite can receive the prediction result and make better pursuit actions.

[0079] The present invention intends to adopt as follows Figure 3The Transformer model shown is an algorithm for predicting escape actions. It is a deep learning model architecture based on the self-attention mechanism. It gets rid of the limitation of traditional recurrent neural networks (RNN) and long short-term memory networks (LSTM) on "long sequence dependencies" and greatly improves training efficiency through parallel computing.

[0080] However, the traditional Transformer has the problem of excessive parameters. On the one hand, it may lead to an excessively long training cycle, and on the other hand, it does not meet the real-time requirements of satellites. Therefore, a lightweight design is carried out on the basis of the traditional Transformer.

[0081] The Transformer was first applied in the field of natural language processing to solve the problem of focusing key information in long sequences of data. Its core lies in the multi-head attention mechanism, whose fundamental mathematical essence is a weighted summation of the input information. The specific attention calculation process includes the following three steps:

[0082] Relevance metric: Calculate the dot product of the query vector Q and all key vectors K. The magnitude of the dot product reflects the similarity or projection length of the two vectors in the feature space, which physically corresponds to the state correlation between agents.

[0083] Normalization and weight allocation: To prevent the gradient from vanishing due to excessively large dot product values, we first divide by the scaling factor. ( (where the key vector dimension is used), and then the relevance values ​​are mapped to a probability distribution, i.e., attention weights, using the Softmax function. :

[0084]

[0085] in, This ensures that the sum of the weights of all neighbors is 1. A high weight means that the neighboring satellite has extremely high tactical value for the current decision (such as high threat or high need for cooperation).

[0086] Information aggregation utilizes calculated weights The value vector V is weighted and summed to generate a collaborative representation vector that contains global context information.

[0087] A single attention head can only focus on the features of the input sequence in a specific subspace, making it difficult to capture complex heterogeneous game relationships simultaneously. For example, an attention head may only focus on proximity in terms of "geometric distance" while ignoring threats in terms of velocity vectors.

[0088] To enrich the semantic hierarchy of the representation, this study employs a multi-head attention mechanism. This mechanism linearly projects the input features into h different subspaces, computes attention in parallel, and finally concatenates and fuses the results. The calculation formula is as follows:

[0089]

[0090]

[0091] in, Let be the projection matrices of the i-th head. This is the final output linear transformation matrix.

[0092] like Figure 4 As shown, to achieve model lightweighting to adapt to training and execution efficiency requirements, the core modification of the original satellite trajectory prediction Transformer architecture is as follows: all Value(V) projection parameter matrices within the two multi-head self-attention mechanisms in the encoder, the multi-head self-attention mechanism in the decoder, and the masked self-attention mechanism are modified. The identity matrix is ​​uniformly replaced and no longer participates in the model training process. The remaining modules (including the embedding layer, positional encoding, query and key projection matrices, fully connected feature extraction layer, physical loss function, etc.) retain their original design, while significantly reducing... While learning parameters and improving training and inference speed, the model retains its core modeling capabilities for the physical characteristics and temporal dependencies of satellite trajectories.

[0093] This invention will determine the satellite state at time t. And the actions of the escaping satellite at the corresponding moment. As input to the Transformer, historical data from a game is used to derive the prediction result for the next move. as well as This allows them to obtain information about the escapee's next move.

[0094] The Transformer model collects data through interaction with the environment. Because the escape strategy constantly changes, the distribution of the Transformer's dataset also changes continuously. Therefore, a "sliding window" is needed to discard old data to prevent the old and new data from having different distributions, which could lead to convergence difficulties for the Transformer. This is achieved by taking multiple batches of data, calculating the loss, and performing backpropagation to update the network weights. Figure 5 As shown.

[0095] As reinforcement learning interacts with the environment, the Transformer's dataset discards outdated data and collects new data, ensuring that the Transformer's predictions remain accurate despite changes in the escaper's policy. The Transformer's loss function is:

[0096]

[0097] The result predicted by the Transformer, and This is real trajectory data.

[0098] The present invention has designed the SAC algorithm to control the pursuing and escaping satellites in a game, and designed the Transformer method to predict the actions of the escaping satellite. Next, the present invention will combine these two methods into a unified framework for collaborative iterative training, as detailed below. Figure 6 The Transformer's input is used to create the pursuer's observations, which are then fed into the pursuer's SAC (Strategic Action Center) to play against the escaper's SAC. During training, the SAC algorithm, combined with a curiosity mechanism, exhibits strong exploratory behavior in the action space, thus exploring better strategies based on the reward function. As the escaper's SAC continuously evolves, the Transformer continuously collects new data and discards old data to adapt to the evolution of the escaper's strategy, and performs backpropagation. As the escaper's SAC algorithm eventually converges, the Transformer also converges, and the pursuer's SAC evolves into a better strategy based on the Transformer's output.

[0099] Finally, in practical applications, the obtained state is first input into the Transformer, which predicts the enemy satellite's actions. Then, the actions are input into the SAC as part of the observation. The SAC outputs the specific orbit change amount, causing the satellite to perform the orbit change action.

[0100] The above completes all the designs of this invention, such as Figure 7 As shown. Based on the above design process, combined with Figure 1 The present invention proposes a reinforcement learning-based satellite pursuit and escape method based on Transformer orbital motion prediction, comprising the following steps:

[0101] S1. Design a single-agent game where the states of both sides are observed and rewards are used as inputs to the agents and feedback values ​​from the environment.

[0102] S2. Using state observations as input, construct and train an initial SAC policy network, construct a loss function, and introduce a curiosity mechanism as an external signal input to the reward. By using the reinforcement learning SAC algorithm and introducing the curiosity mechanism, the state space can be fully explored to obtain a better policy.

[0103] S3. Construct Transformer input vectors based on the pursuer's observations and the escaper's strategy, encode the vectors at positions, and generate sequences as Transformer datasets.

[0104] S4. Construct a dataset, train the Transformer, and design a loss function for backpropagation; use the fuzzy normalized vector as the input to the Transformer and encode it with input embedding; create a dataset based on the environment; based on the real feedback from the environment, this embodiment uses the MSE reward function to ensure that the Transformer prediction model can converge normally.

[0105] S5. Using the trained Transformer output as part of the SAC input observation, reconstruct the SAC input state observation;

[0106] S6. Based on the designed Transformer and SAC, construct an integrated decision network, train it together, design a loss function for backpropagation, and iteratively train the model.

[0107] In this example, the curiosity mechanism is first introduced to address the problem of insufficient exploration by the agent due to the large space of satellite action. Secondly, after the algorithm is trained, it can predict the escaper's actions based on the escaper's observations. Combined with the SAC algorithm, the pursuing satellite can react in advance, achieving a better pursuit effect.

[0108] In this embodiment, an environment needs to be constructed as a data source for reinforcement learning. The environment in this invention uses the Runge-Kutta method to extrapolate the satellite's position, returning the satellite state every 10 minutes and using reinforcement learning to make decisions. The maximum allowed mission time is 10 hours. Maneuvers are performed in a circular orbit at an altitude of 42,157 kilometers.

[0109] In this embodiment, the model is developed based on Python 3.8 and the accompanying PyTorch 2.0.0 as a deep learning development package, and adopts a single-agent reinforcement learning decision-making method based on the SAC algorithm.

[0110] In this embodiment, in step S1, the dimension of the returned observation is 21, the action space is 3, and the amplitude limit of the satellite engine on the three axes is 2m / s.

[0111] In this embodiment, in step S2, the maximum number of data entries in the experience cache array is 1e6, the learning rate of the Actor is 3e-4, the learning rate of the Critic is 1e-4, and the learning rate of the Curiosity Network is 1e-4.

[0112] In this embodiment, in step S3, the Transformer dataset window size is 100 rounds.

[0113] In this embodiment, in step S4, the input of the Transformer is 21-dimensional, and the learning rate of the network is 1e-4.

[0114] In this embodiment, in step S5, an observation interface is provided to receive the prediction results of the Transformer, which are 3D.

[0115] In this embodiment, in step S6, the above content is integrated into a framework, and the commonly used Adam optimizer is used for parameter backpropagation. The training is carried out for 20,000 rounds, with 120 steps per round and a 10-minute interval between each step.

[0116] This embodiment also provides a reinforcement learning-based satellite escape tracking system based on orbital motion prediction, including:

[0117] The state observation and reward design unit designs the state observation and reward of both sides in a single-agent game as input to the agent and feedback to the environment.

[0118] The SAC policy network training unit takes state observations as input, constructs and trains the initial SAC policy network, builds the loss function, and introduces a curiosity mechanism as an external signal input to the reward.

[0119] The dataset construction unit constructs input vectors for the Transformer model based on the pursuer's observations and the escaper's strategy, encodes the vectors at their positions, and generates sequences as the dataset for the Transformer model.

[0120] The Transformer model training unit uses the constructed dataset to train the Transformer model and designs a loss function for backpropagation.

[0121] The integrated decision network construction unit uses the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstructs the SAC input state observation, builds the integrated decision network, and trains it together. A loss function is designed for backpropagation to obtain an iterative training model.

[0122] The output unit uses a trained integrated decision network to obtain the escaped satellite's orbit change amount, and then executes the orbit change action based on the orbit change amount of the pursuing satellite.

[0123] Finally, to verify the rationality and effectiveness of the above algorithm framework, this invention simulates the algorithm, selecting several examples such as... Figure 8 The local situation diagram shown is a 3D plot of the local situation on the left, where r0 represents the pursuing satellite and b0 represents the escaping satellite. The right side shows the distance change curves between the pursuing satellite r0 and the escaping satellite. The escaping satellite uses the SAC algorithm, while the pursuing satellite uses a pursuit-escape algorithm based on Transformer-based orbital motion prediction combined with reinforcement learning. It can be seen that the escaping satellite has been adopting an escape strategy to try to evade the pursuit of the pursuing satellite. However, because the pursuing satellite can predict the actions of the escaping satellite, it takes preemptive measures and quickly catches up with the escaping satellite.

[0124] The innovation of this invention lies in the design of a lightweight Transformer model to predict enemy actions in order to meet the real-time requirements of satellite tracking, based on traditional deep reinforcement learning. This makes the actions of the pursuer more predictable. Furthermore, to address the problem of excessive action space, this invention introduces a curiosity mechanism into the traditional SAC algorithm to increase the exploration capability of the SAC algorithm.

[0125] In this invention, terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily indicate an actual relationship or order between them. Furthermore, the use of terms such as "comprising," "including," or other similar terms implies non-exclusive inclusion, meaning that a process, method, article, or apparatus that covers a series of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to the process, method, article, or apparatus.

[0126] Although embodiments of the present invention have been shown and described, those skilled in the art will recognize that various changes, modifications, substitutions, and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A reinforcement learning-based satellite escape tracking method based on orbital motion prediction, characterized in that, include: Step S1: Design the state observations and rewards of the two parties in a single-agent game as inputs and environmental feedback values ​​for the agents; Step S2: Using state observations as input, construct and train the initial SAC policy network, construct the loss function, and introduce a curiosity mechanism as an external signal input to the reward. Step S3: Construct the Transformer model input vector based on the pursuer's observation and the escaper's strategy, encode the vector position, and generate a sequence as the Transformer model's dataset; Step S4: Train the Transformer model using the constructed dataset and design a loss function for backpropagation; Step S5: Using the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstruct the input state observation of the SAC policy network, build an integrated decision network, train it together, design a loss function for backpropagation, and obtain an iterative training model. Step S6: Use the trained integrated decision network to obtain the escaped satellite's orbit change amount, and execute the orbit change action based on the orbit change amount of the pursuing satellite.

2. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The state observations include the position and velocity of the reference satellite in the Earth's inertial coordinate system, the motion description of the pursuing satellite in the reference coordinate system, the motion description of the escaping satellite in the reference coordinate system, and the escaping satellite's escape maneuvers.

3. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The curiosity mechanism employs a four-layer multilayer perceptron network to predict the features of the next state based on the current state features and the agent's chosen action. It then calculates the error between the predicted and actual values ​​and converts this prediction error into an intrinsic reward. Meanwhile, the prediction error is used as the loss value to update the curiosity network.

4. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 3, characterized in that, The reward for the pursuer who incorporates a curiosity mechanism is: ； in It is the distance between satellites after time t following the execution of the corresponding action. This refers to the distance between satellites after time t if the action has not been taken. This indicates the reward for the pursuing party successfully catching up with the fleeing party. As a reward for curiosity, Indicates weight; The reward for the fleeing party is the opposite of the reward for the pursuing party.

5. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 4, characterized in that, The for: 。 6. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The SAC policy network introduces entropy regularization into the Actor-Critic framework, and employs a dual-Critic network and a target network soft update mechanism.

7. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 6, characterized in that, The loss function in step S2 includes the Actor loss function and the Critic network loss function, where, The Actor loss function is: ； in, Indicates the state From experience replay pool D, action The expectation obtained from the sampling process of the current Actor policy. Indicates the state Next action The value of the action, These are the parameters of the Critic network. For temperature coefficient, Select an action in state s The probability of; The loss function of the Critic network is: ； ； ； in, Value function network The loss, For the parameters of the value function network, For the current Critic network, the current state Next action Value prediction; For state Execute action Even if there is a reward later, As a discount factor, The next state Next, strategy Sampling action Seeking expectations, Pick and The minimum value of the network output. It is the SAC entropy regularization term. It is in state Choose below The probability, It is the temperature coefficient. for Network loss.

8. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The Transformer model incorporates all Value projection parameter matrices from the two multi-head self-attention mechanisms in the encoder, the multi-head self-attention mechanism in the decoder, and the masked self-attention mechanism. The data is uniformly replaced with a fixed identity matrix and does not participate in the model training process; at the same time, a sliding window is set to eliminate old data in the Transformer model's dataset that does not conform to the data distribution.

9. A reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 8, characterized in that, The loss function of the Transformer model is: ； The result predicted by the Transformer, and This is real trajectory data.

10. A reinforcement learning satellite escape tracking system implementing the method of any one of claims 1-9, characterized in that, include: The state observation and reward design unit designs the state observation and reward of both sides in a single-agent game as input to the agent and feedback to the environment. The SAC policy network training unit takes state observations as input, constructs and trains the initial SAC policy network, builds the loss function, and introduces a curiosity mechanism as an external signal input to the reward. The dataset construction unit constructs input vectors for the Transformer model based on the pursuer's observations and the escaper's strategy, encodes the vectors at their positions, and generates sequences as the dataset for the Transformer model. The Transformer model training unit uses the constructed dataset to train the Transformer model and designs a loss function for backpropagation. The integrated decision network construction unit uses the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstructs the SAC input state observation, builds the integrated decision network, and trains it together. A loss function is designed for backpropagation to obtain an iterative training model. The output unit uses a trained integrated decision network to obtain the escaped satellite's orbit change amount, and then executes the orbit change action based on the orbit change amount of the pursuing satellite.

Images