Reinforced learning satellite pursuit method and system based on orbit action pre-judgment

Abstract

The invention discloses a reinforcement learning satellite pursuit method and system based on orbit action prejudgment, and the method comprises the steps: constructing an SAC strategy network, designing a proper loss function for back propagation, and training an initial strategy of a model; a curiosity mechanism is introduced into the SAC algorithm, and an error between a full-connection network prediction result and an actual error is used as a reward signal to encourage an intelligent agent to explore an action space; the method comprises the following steps: constructing an input vector format and a data set for Transform training, performing position coding on an input vector during input, and designing a loss function to perform back propagation on the Transform; the output of the Transform is intercepted as the input of the SAC algorithm, SAC observation is reconstructed, and the two form an integrated decision network. According to the method, the game problem of performing orbital transfer pursuit by using pulse maneuver between satellites is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence satellite orbit pursuit and evasion application, and particularly relates to a reinforcement learning satellite pursuit and evasion method and system based on orbit action prediction. BACKGROUND

[0002] When a satellite runs in the space, it will be affected by the perturbation of various celestial bodies. At this time, the simple two-body motion mathematical model used to describe the satellite motion is no longer applicable. In addition to the gravity produced by the Earth's oblateness, the gravity of other celestial bodies also needs to be considered, which makes the mathematical model of satellite motion extremely complex. In the field of satellite motion modeling, the traditional method constructs a mathematical expression based on a physical model and solves the model through optimization means. However, this traditional scheme has two obvious defects: on the one hand, it usually relies on heuristic algorithms such as particle swarm optimization algorithm, which consumes a certain amount of time in the iterative solution process, making it difficult to meet the demand for high real-time performance in the satellite game scenario; on the other hand, the traditional method only designs around the two-body model and does not include the perturbation effects of other celestial bodies in the calculation category, resulting in limited model accuracy. In recent years, as one of the core technologies in the field of artificial intelligence, deep learning has developed rapidly. Its core mechanism is to simulate the operation mode of human brain neurons to accurately extract key features from complex data. Reinforcement learning, as an important branch of deep learning, has gradually become a key component of the task decision algorithm system. The core idea of this technology is to let the intelligent agent continuously interact with the environment through trial and error, and obtain the optimal decision strategy through continuous iteration and optimization. This method does not rely on specific mathematical models, and can not only effectively solve the complex problem of satellite motion model, but also meet the stringent real-time requirements of the satellite decision-making scenario. In recent years, some scholars have applied reinforcement learning to orbit pursuit and evasion, but these methods do not take into account the fact that the escape satellite's strategy is also changing in the process of strategy evolution, which will cause a large variance in the pursuit strategy, making it difficult to converge. SUMMARY

[0003] The purpose of the present application is to provide a reinforcement learning satellite pursuit and evasion method and system based on orbit action prediction, which solves the game problem of using impulse maneuvers for orbit transfer and evasion between satellites, and realizes the continuous approach of the tracking satellite to the target of the escaping satellite in the game process, making more accurate decisions.

[0004] The technical solution for achieving the purpose of the present application is as follows:

[0005] A reinforcement learning satellite pursuit and evasion method based on orbit action prediction, comprising:

[0006] Step S1, design single-agent game state observation and reward as agent input and environment feedback value;

[0007] Step S2, taking the state observation as input, constructing and training the initial SAC policy network, constructing the loss function, and introducing the curiosity mechanism as an external signal input into the reward;

[0008] Step S3, constructing a Transformer model input vector based on the pursuit party observation and the escape party strategy, performing position encoding on the vector, and generating a sequence as a data set for the Transformer model;

[0009] Step S4, training the Transformer model using the constructed data set, and designing a loss function for backpropagation;

[0010] Step S5, taking the output of the trained Transformer model as part of the input observation for the SAC policy network, reconstructing the input state observation for the SAC policy network, constructing an integrated decision-making network, and jointly training, designing a loss function for backpropagation, and obtaining an iterative training model;

[0011] Step S6, using the trained integrated decision-making network to obtain the escape party's orbit change amount, and executing orbit change actions based on the orbit change amount for the pursuit party satellite.

[0012] Further, the state observation includes the position and velocity of the reference satellite in the Earth inertial coordinate system, the motion description of the pursuit party satellite in the reference coordinate system, the motion description of the escape party satellite in the reference coordinate system, and the escape action of the escape party satellite.

[0013] Further, the curiosity mechanism is: using a four-layer multi-layer perception network, based on the current state features and the action selected by the agent, predicting the features of the next state, and calculating the error between the predicted value and the true value, converting the prediction error into an intrinsic reward , and using the prediction error as the loss value to update the curiosity network.

[0014] Further, the pursuit party reward with the curiosity mechanism is:

[0015] ;

[0016] wherein is the distance between the satellites after t time after the corresponding action is made, is the distance between the satellites after t time without making the action, represents the reward for the pursuit party successfully catching up with the escape party, is the reward of the curiosity mechanism, represents the weight;

[0017] The reward of the escape party is the opposite of the reward of the pursuit party.

[0018] Further, the SAC policy network introduces entropy regularization in the Actor-Critic framework, and adopts a double Critic network and a target network soft update mechanism. Further, the SAC policy network introduces entropy regularization in the Actor-Critic framework, and adopts a double Critic network and a target network soft update mechanism.

[0019] .

[0020] Further, the SAC policy network introduces entropy regularization in the Actor-Critic framework, and adopts a double Critic network and a target network soft update mechanism.

[0021] Further, the loss function in step S2 includes an Actor loss function and a Critic network loss function, wherein,

[0022] The Actor loss function is:

[0023] ; wherein, represents the state from the experience replay pool D, the action sampled from the current Actor policy, represents the action value of performing the action in the state , is the parameter of the Critic network, is a temperature coefficient (hyperparameter) for balancing the return of the policy and the entropy of the policy, is the current Actor policy, i.e., the probability of selecting the action in the state s;

[0024] The Critic network loss function is:

[0025] ;

[0026] ;

[0027] ;

[0028] wherein, is the loss of the value function network , is the parameter of the value function network. is the value prediction of the current Critic network for performing the action in the current state ; is the immediate reward after performing the action in the state , is a discount factor taking 0.99, is the next state Policy Sampled action Expectation, Take the minimum value of the outputs of two Q networks And , the entropy regularization term of SAC is the probability of selecting in state is the temperature coefficient, and adding this entropy regularization term can encourage the agent to explore, achieving the dual goal of maximizing rewards + maximizing entropy. For network loss, and The calculation is similar, except that uses as a future reward estimator to predict the value of performing action in the current state Finally, take the minimum value between the two losses to reduce overestimation problems.

[0029] Further, all Value projection parameter matrices in the two multi-head self-attention mechanisms in the encoder, the multi-head self-attention mechanism in the decoder, and the masked self-attention mechanism in the Transformer model are uniformly replaced with fixed identity matrices, which do not participate in the model training process; at the same time, a sliding window is set to eliminate old data in the data set of the Transformer model that does not conform to the data distribution.

[0030] Further, the loss function of the Transformer model is:

[0031] ;

[0032] is the result predicted by the Transformer, and is the real trajectory data.

[0033] A reinforcement learning satellite pursuit and evasion system based on orbit action prediction, comprising:

[0034] State observation and reward design unit, design the state observation and reward of both sides of the single-agent game as the input of the agent and the environmental feedback value;

[0035] SAC policy network training unit, taking state observation as input, constructing and training the initial SAC policy network, constructing the loss function, and introducing the curiosity mechanism as an external signal input into the reward;

[0036] ​​​A data set construction unit constructs a Transformer model input vector based on the pursuit side observation and the escape side strategy, encodes the vector, and generates a sequence as a data set of the Transformer model;

[0037] A Transformer model training unit trains the Transformer model using the constructed data set, and designs a loss function for back propagation;

[0038] An integrated decision network construction unit uses the output of the trained Transformer model as part of the SAC strategy network input observation, reconstructs the SAC input state observation, constructs an integrated decision network, and jointly trains the integrated decision network, designs a loss function for back propagation, and obtains an iterative training model;

[0039] An output unit uses the trained integrated decision network to obtain the escape side orbit change amount, and the pursuit side satellite performs an orbit change action based on the orbit change amount.

[0040] Compared with the prior art, the beneficial effects of the present application are: 1. The spatiotemporal characteristics of orbital motion are fully utilized, so that the pursuit side can predict the action of the escape side, take a more effective pursuit strategy, increase the pursuit success rate of the model, and speed up the convergence speed of the model; 2. The SAC algorithm is used, and the curiosity mechanism is added to the reward to encourage the agent to explore the unexplored action space and avoid the model from falling into local optimum; 3. The lightweight Transformer is used to speed up the reasoning speed of the satellite and meet the real-time requirements of the satellite pursuit. BRIEF DESCRIPTION OF DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 It is a flowchart of the satellite pursuit and escape method based on the Transformer orbital action prediction reinforcement learning.

[0043] Figure 2 It is a calculation flowchart for calculating the curiosity reward.

[0044] Figure 3 It is an input and output design diagram of the Transformer model.

[0045] Figure 4 It is a lightweight multi-head attention mechanism of the Transformer model.

[0046] Figure 5Process diagram for collecting data and updating for the Transformer.

[0047] Figure 6 Overall framework diagram for the satellite pursuit and evasion method based on the reinforcement learning of the action prediction of the Transformer orbit.

[0048] Figure 7 Algorithm flowchart of the satellite pursuit and evasion method based on the reinforcement learning of the action prediction of the Transformer orbit.

[0049] Figure 8 Local situation simulation diagram based on the algorithm of the present application. DETAILED DESCRIPTION

[0050] The technical solutions of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0051] In order to more clearly describe the structure and function of the present application, the following further describes in combination with the drawings and specific embodiments.

[0052] The satellite pursuit and evasion scenario refers to the satellites of different camps changing orbits through orbit transfer actions, the pursuit satellite chasing the escape satellite, and the escape satellite also taking certain strategies to move away from the pursuit satellite.

[0053] First, the satellite pursuit and evasion is modeled. The satellite pursuit and evasion process can be regarded as a Markov decision problem. From the time perspective, it is divided into past, present and future. The Markov problem refers to the future state being only related to the decision made at present, and not related to the past decision. The four main elements in the Markov decision process are: S (state), A (action), P (state transition), and R (reward function). The goal of the agent is to find a strategy so that in the current state S, the action A is made, and under the probability distribution of P state transition, the expected future R is always the maximum. P (state transition) is given by the environment, and S, A and R of the agent are designed first.

[0054] For state observation S, the present application first sets a reference satellite on the orbit. The reference satellite does not perform actions, but only runs on a fixed orbit. The significance of setting this satellite is to provide a reference for other satellites, so as to establish a reference coordinate system based on the satellite to describe the local motion of the satellite.

[0055] The state input of the pursuit satellite is composed of four parts: the first part is the position and velocity of the reference satellite in the earth inertial coordinate system, the second part is the motion description of the pursuit satellite in the reference coordinate system, the third part is the motion description of the escape satellite in the reference coordinate system, and the fourth part is the escape action of the escape satellite filled in by the Transformer prediction. The reason for such design is to separate the motion of the satellite relative to the earth and relative to the reference satellite, so that the satellite pays more attention to the local motion between satellites, improves the learning efficiency of the satellite to the pursuit, and the state input of the escape satellite is similar to that of the pursuit, the first part is the description of the reference satellite in the inertial coordinate system, the second part is the description of the escape satellite in the reference coordinate system, and the third part is the description of the pursuit satellite in the reference coordinate system.

[0056] Because the first part of the observation and the second and third parts of the observation are in different coordinate systems, the data distribution described by different coordinate systems is very different, among which the data scale described in the inertial coordinate system is about [-42157, 42157], and the data scale described in the reference coordinate system is about [-200, 200]. If this uneven distribution is not processed, it may cause the agent to learn ineffective data. Therefore, before being formally input to the agent, the observation needs to be fuzzy normalized. The present application adopts the z-score standardization method, first collects a large amount of satellite data, calculates the mean and variance between each quantity, and standardizes the distribution of each dimension, and constrains the mean to 0 and the variance to 1.

[0057] For action A, the orbit maneuvering method of the satellite is divided into two kinds, one is small thrust continuous orbiting, and the other is large thrust pulse orbiting. The orbiting object of the present application is the orbit satellite, which is suitable for using large thrust pulse orbiting to instantaneously change the speed of the satellite and complete the orbit transfer of the satellite. Therefore, the action space of the satellite is , which are the x, y and z axis velocity increments in the inertial coordinate system respectively.

[0058] For the reward R, since the goal of the present application is to make the pursuit approach the escape, the distance between the pursuit and the escape is required to be as short as possible, and the goal of the escape is to increase the distance from the pursuit, so the present application necessarily uses distance as part of the reward.

[0059] But in satellite pursuit, because the motion of the satellite is complex and coupled, only paying attention to the current distance of the satellite is not enough to make the satellite agent converge. The present application introduces the CW equation according to the traditional relative orbit kinematics, adds the CW equation to predict the future state of the satellite in the reward function design, and the specific making method is as follows:

[0060]

[0061] Rewrite the CW equations in the form of state transitions, This is the initial state. The state transition matrix is ​​used to deduce the state after time t. The specific form of the state transition matrix is ​​as follows:

[0062]

[0063] Based on the above state transition equations, the reward function can be designed as follows:

[0064]

[0065] in It is the distance between satellites after time t following the execution of the corresponding action. This refers to the distance between satellites after time t if the action had not been taken. It indicates whether the distance between satellites will decrease in the future after the action is taken. If it decreases, a positive reward is given; otherwise, a negative reward is given. This means that if the pursuing party successfully catches up with the fleeing party, it signifies mission completion and a large reward will be given, as detailed below:

[0066]

[0067] The reward function of the escaping satellite is the opposite of that of the pursuing satellite, and the game between the escaping and pursuing satellites constitutes a zero-sum game.

[0068] SAC (Soft Actor-Critic) is an offline policy reinforcement learning algorithm suitable for continuous action spaces. Its core idea revolves around "maximum entropy reinforcement learning": by introducing entropy regularization into the traditional Actor-Critic framework, the algorithm aims not only to maximize the cumulative environment reward but also to encourage the randomness of the policy (the higher the entropy, the stronger the exploration), thereby balancing exploration and exploitation and avoiding premature entrapment in local optima. At the same time, it adopts a dual-Critic network and a soft update mechanism for the target network to effectively alleviate the overestimation problem of the value function and improve training stability. Combined with the experience replay pool to reuse historical data, it has high sample efficiency and performs well in complex continuous control tasks.

[0069] The loss function of the SAC algorithm is shown below:

[0070]

[0071] Critic uses a double Q-network, and its loss function is:

[0072]

[0073] In the satellite game task, the action space of the satellite is large, and only relying on the SAC algorithm is easy to make the model fall into a local optimum, therefore, the curiosity mechanism is introduced, and the purpose is to add exploration into the reward function, and rewards are given to the agent when the agent makes an action that has never been made before.

[0074] The curiosity mechanism is the core technology of simulating the human "active exploration of unknown" instinct, and its core value lies in solving the key bottleneck of traditional RL - exploration-exploitation trade-off: avoiding the agent from falling into a local optimum due to excessive "exploitation of known high-yield strategies", and driving it to actively explore the uncontacted areas in the environment, and finally realizing the global optimal learning goal. It fills the learning motivation gap when the external reward is sparse or missing by constructing "intrinsic reward", so that the agent can still continue to explore and learn in the scene without explicit feedback. The present application sets up a four-layer multi-layer perception network, based on the current state features and the action selected by the agent, predicts the features of the next state, and calculates the error between the predicted value and the true value, the greater the error, the stronger the curiosity. The prediction error is converted into intrinsic reward to form the final reward signal of the agent.

[0075] As shown in Figure 2 , the current state and the current state are known, which are input into the curiosity network and the orbit environment deduction respectively, to obtain the predicted state and the real state at the next moment, and the calculated prediction error is input into the reinforcement learning as a reward signal. And the prediction error is taken as the loss value to update the curiosity network:

[0076]

[0077] Since the escape agent and the pursuit agent are trained together, the pursuit satellite cannot completely predict the escape action of the escape satellite in the process of strategy evolution, so a method is needed to predict the escape action of the escape satellite in the next step according to the present and past actions, and the pursuit satellite can receive the prediction result and make better pursuit actions.

[0078] The present application intends to adopt as Figure 3 ​The illustrated Transformer model is an algorithm for escape action prediction, which is a deep learning model architecture based on self-attention mechanism. It breaks away from the limitations of traditional recurrent neural network (RNN) and long short-term memory network (LSTM) on "long sequence dependence", and greatly improves the training efficiency through parallel computing.

[0079] However, the traditional Transformer has the problem of too large parameter quantity, which may lead to too long training period on one hand, and does not meet the real-time requirements of satellite on the other hand. Therefore, lightweight design is made on the basis of traditional Transformer.

[0080] Transformer was first applied in natural language processing field, aiming to solve the problem of focusing on key information in long sequence data. The main core is the multi-head attention mechanism, and its core mathematical essence is to perform weighted summation on input information. The specific attention calculation process includes the following three steps:

[0081] Correlation measurement: calculate the dot product of query vector Q and all key vectors K. The size of dot product reflects the similarity or projection length of two vectors in feature space, which corresponds to the state correlation between agents in physics.

[0082] Normalization and weight distribution: in order to prevent the gradient from disappearing due to the large value of dot product, first divide by the scaling factor ( The dimension of key vector), and then map the correlation value to a probability distribution, i.e. attention weight :

[0083]

[0084] Among them, Ensure that the sum of weights of all neighbors is 1. High weight means that the neighbor satellite has a very high tactical value (such as high threat or high cooperation demand) for the current decision.

[0085] Information aggregation uses the calculated weight To perform weighted summation on value vector V to generate cooperation representation vector containing global context information.

[0086] Single attention head (Single Head) can only focus on the features of input sequence in a certain subspace, and it is difficult to capture complex heterogeneous game relationship at the same time. For example, a certain attention head may only focus on the proximity in "geometric distance", while ignoring the threat in velocity vector.

[0087] To enrich the semantic hierarchy of the representation, this study employs a multi-head attention mechanism. This mechanism linearly projects the input features into h different subspaces, computes attention in parallel, and finally concatenates and fuses the results. The calculation formula is as follows:

[0088]

[0089]

[0090] in, Let be the projection matrices of the i-th head. This is the final output linear transformation matrix.

[0091] like Figure 4 As shown, to achieve model lightweighting to adapt to training and execution efficiency requirements, the core modification of the original satellite trajectory prediction Transformer architecture is as follows: all Value(V) projection parameter matrices within the two multi-head self-attention mechanisms in the encoder, the multi-head self-attention mechanism in the decoder, and the masked self-attention mechanism are modified. The identity matrix is ​​uniformly replaced and no longer participates in the model training process. The remaining modules (including the embedding layer, positional encoding, query and key projection matrices, fully connected feature extraction layer, physical loss function, etc.) retain their original design, while significantly reducing... While learning parameters and improving training and inference speed, the model retains its core modeling capabilities for the physical characteristics and temporal dependencies of satellite trajectories.

[0092] This invention will determine the satellite state at time t. And the actions of the escaping satellite at the corresponding moment. As input to the Transformer, historical data from a game is used to derive the prediction result for the next move. as well as This allows them to obtain information about the escapee's next move.

[0093] The Transformer model collects data through interaction with the environment. Because the escape strategy constantly changes, the distribution of the Transformer's dataset also changes continuously. Therefore, a "sliding window" is needed to discard old data to prevent the old and new data from having different distributions, which could lead to convergence difficulties for the Transformer. This is achieved by taking multiple batches of data, calculating the loss, and performing backpropagation to update the network weights. Figure 5 As shown.

[0094] As the reinforcement learning interacts with the environment, the dataset of the Transformer discards outdated data, collects new data, achieves the result predicted by the Transformer, and does not cause inaccurate prediction due to the change of the escape strategy. The loss function of the Transformer is:

[0095]

[0096] The result predicted by the Transformer is The real trajectory data.

[0097] The above application designs the SAC algorithm to control the pursuit satellite and the escape satellite to play the game, and designs the Transformer method to predict the action of the escape satellite. Next, the application combines the two methods together to form an overall framework, and iteratively trains in cooperation. The specific framework is as follows Figure 6 The input of the Transformer is used to make the pursuit satellite observation, which is input into the pursuit SAC and used to play with the escape SAC. During the training process, since the SAC algorithm combines the curiosity mechanism in the action space and has strong exploration, a better strategy can be explored according to the reward function. In this process, since the escape SAC is constantly evolving, the Transformer also constantly collects new data and discards old data to adapt to the evolution of the escape strategy, and back propagation. Since the escape SAC algorithm constantly converges, the Transformer will also constantly converge, and the pursuit SAC will also evolve a better strategy according to the output of the Transformer.

[0098] Finally, in the actual application process, the state obtained first is input into the Transformer, the action of the enemy satellite is predicted by the Transformer, and then the action is input into the SAC as part of the observation. The SAC outputs the specific orbit change amount, so that the satellite executes the orbit change action.

[0099] The above completes all the designs of the application, as shown in Figure 7 Based on the above design process, combined with Figure 1 The application provides a reinforcement learning satellite pursuit and escape method based on Transformer orbit action prediction, which includes the following steps:

[0100] S1, design the state observation and reward of single-agent game as the input of the agent and the feedback value of the environment;

[0101] S2, take the state observation as input, build and train the initial SAC policy network, build loss, and introduce curiosity mechanism as external signal input into reward; use reinforcement learning SAC algorithm, and introduce curiosity mechanism, which can fully explore the state space and obtain better strategy;

[0102] S3, build a Transformer input vector based on the pursuit party observation and the escape party strategy, position encode the vector, and generate a sequence as the Transformer dataset;

[0103] S4, build a good dataset, train the Transformer, design a loss function for back propagation; use the fuzzy normalized vector as the input of the Transformer, and perform input embedding coding, and make a dataset according to the environment; according to the real feedback of the environment, the embodiment uses the MSE reward function, which can ensure that the Transformer prediction model can converge normally;

[0104] S5, take the trained Transformer output as part of the SAC input observation, and rebuild the SAC input state observation;

[0105] S6, based on the designed Transformer and SAC, build an integrated decision-making network, jointly train, design loss for back propagation, and iteratively train the model.

[0106] In this example, the curiosity mechanism is first introduced to solve the problem of insufficient exploration of the satellite action space by the agent, and secondly, the algorithm can predict the action of the escape party after training, and the pursuit party satellite can react in advance to achieve better pursuit effect.

[0107] In this embodiment, an environment needs to be built as the data source for reinforcement learning. The environment in the present application uses the Runge-Kutta method to deduce the position of the satellite, returns the satellite state every 10 minutes and uses reinforcement learning to make decisions, and the longest allowed time for the task is 10 hours. The maneuver is performed on a circular orbit with an orbital height of 42157 kilometers.

[0108] In this embodiment, the model development is based on python3.8, and the corresponding pytorch2.0.0 is used as the deep learning development package, and a single-agent reinforcement learning decision-making method based on SAC algorithm is adopted.

[0109] In this embodiment, in step S1, the dimension of the observation returned each time (the time of one decision-making cycle is 10 minutes) is 21, the action space is 3, and the amplitude of the satellite engine on the three axes is 2 m / s.

[0110] In the step S2, the maximum number of data of the experience cache array is 1e6, the learning rate of the Actor is 3e-4, the learning rate of the Critic is 1e-4, and the learning rate of the curiosity network is 1e-4.

[0111] In the step S3, the window size of the Transformer dataset is 100 rounds.

[0112] In the step S4, the input of the Transformer is 21 dimensions, and the learning rate of the network is 1e-4.

[0113] In the step S5, the observation interface is reserved to receive the prediction results of the Transformer, and the dimension is 3.

[0114] In the step S6, the above contents are integrated into a framework, the commonly used Adam optimizer is used for parameter back propagation, 20000 rounds of training are performed, 120 steps per round, and 10 minutes per step.

[0115] The embodiment also provides a reinforcement learning satellite pursuit and evasion system based on track action prediction, which comprises:

[0116] A state observation and reward design unit is configured to design state observation and reward of both sides of a single-agent game as an input of an agent and an environmental feedback value.

[0117] A SAC policy network training unit is configured to take the state observation as an input, construct and train an initial SAC policy network, construct a loss function, and introduce a curiosity mechanism as an external signal input into the reward.

[0118] A dataset construction unit is configured to construct a Transformer model input vector based on the observation of the pursuit side and the strategy of the escape side, perform position encoding on the vector, and generate a sequence as a dataset of the Transformer model.

[0119] A Transformer model training unit is configured to train the Transformer model by using the constructed dataset and perform back propagation by using a designed loss function.

[0120] An integrated decision network construction unit is configured to take the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstruct the input state observation of the SAC, construct an integrated decision network, and jointly train the integrated decision network, perform back propagation by using a designed loss function, and obtain an iterative training model.

[0121] An output unit obtains the orbit changing amount of the escaping party by using the integrated decision network, and performs orbit changing action based on the orbit changing amount of the pursuit party.

[0122] Finally, in order to verify that the above-mentioned algorithm framework is reasonable and effective, the present application simulates the algorithm, selects several as Figure 8 The left side is a three-dimensional dot plot of the local situation, r0 represents the pursuit satellite, b0 represents the escaping satellite, and the right side is a distance change curve of the pursuit satellite r0 and the escaping satellite, the escaping satellite adopts the SAC algorithm, and the pursuit satellite adopts the pursuit and escape algorithm based on the orbit action prediction of the Transformer combined with reinforcement learning, it can be seen that the escaping satellite is always taking escape strategy to avoid the pursuit of the pursuit satellite, but since the pursuit satellite can predict the action of the escaping satellite, it takes a strategy in advance, and soon pursues the escaping satellite.

[0123] The innovation of the present application is that on the basis of using traditional deep reinforcement learning, in order to meet the real-time requirements of satellite pursuit, a lightweight Transformer model is designed to predict the action of the enemy, so that the action of the pursuit party is more predictable, and for the problem of too large action space, the present application introduces a curiosity mechanism in the traditional SAC algorithm to increase the exploration ability of the SAC algorithm.

[0124] In the present application, the terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily indicate the actual relationship or sequence between them. In addition, the use of "include", "contain" or other similar terms means non-exclusive inclusion, that is, the process, method, article or equipment that covers a series of elements not only includes the explicitly listed elements, but also includes other elements not explicitly listed or inherent to the process, method, article or equipment.

[0125] Although the embodiments of the present application have been shown and described, various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application for those of ordinary skill in the art. The scope of the present application is defined by the appended claims.

Claims

1. A reinforcement learning-based satellite escape tracking method based on orbital motion prediction, characterized in that, include: Step S1: Design the state observations and rewards of the two sides in a single-agent game as inputs and environmental feedback values ​​for the agents; Step S2: Using state observations as input, construct and train the initial SAC policy network, construct the loss function, and introduce a curiosity mechanism as an external signal input to the reward. Step S3: Construct the Transformer model input vector based on the pursuer's observation and the escaper's strategy, encode the vector position, and generate a sequence as the Transformer model's dataset; Step S4: Train the Transformer model using the constructed dataset and design a loss function for backpropagation; Step S5: Using the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstruct the input state observation of the SAC policy network, build an integrated decision network, train it together, design a loss function for backpropagation, and obtain an iterative training model. Step S6: Use the trained integrated decision network to obtain the escaped satellite's orbit change amount, and execute the orbit change action based on the orbit change amount of the pursuing satellite.

2. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The state observations include the position and velocity of the reference satellite in the Earth's inertial coordinate system, the motion description of the pursuing satellite in the reference coordinate system, the motion description of the escaping satellite in the reference coordinate system, and the escaping satellite's escape maneuvers.

3. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The curiosity mechanism employs a four-layer multilayer perceptron network to predict the features of the next state based on the current state features and the agent's chosen action. It then calculates the error between the predicted and actual values ​​and converts this prediction error into an intrinsic reward. Meanwhile, the prediction error is used as the loss value to update the curiosity network.

4. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 3, characterized in that, The reward for the pursuer who incorporates the curiosity mechanism is: ； in It is the distance between satellites after time t following the execution of the corresponding action. This refers to the distance between satellites after time t if the action has not been taken. This indicates the reward for the pursuing party successfully catching up with the fleeing party. As a reward for curiosity, Indicates weight; The reward for the fleeing party is the opposite of the reward for the pursuing party.

5. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 4, characterized in that, The for: 。 6. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The SAC policy network introduces entropy regularization into the Actor-Critic framework, and employs a dual-Critic network and a target network soft update mechanism.

7. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 6, characterized in that, The loss function in step S2 includes the Actor loss function and the Critic network loss function, where, The Actor loss function is: ； in, Indicates the state From the experience replay pool D, action The expectation obtained from the sampling process of the current Actor policy. Indicates the state Next action The value of the action, These are the parameters of the Critic network. For temperature coefficient, Select an action in state s The probability of; The loss function of the Critic network is: ； ； ； in, Value function network The loss, For the parameters of the value function network, For the current Critic network, the current state Next action Value prediction; For state Execute action Even if there is a reward later, As a discount factor, The next state Next, strategy Sampling action Seeking expectations, Pick and The minimum value of the network output. It is the SAC entropy regularization term. It is in state Choose below The probability, It is the temperature coefficient. for Network loss.

8. The reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 1, characterized in that, The Transformer model incorporates all Value projection parameter matrices from the two multi-head self-attention mechanisms in the encoder, the multi-head self-attention mechanism in the decoder, and the masked self-attention mechanism. The data is uniformly replaced with a fixed identity matrix and does not participate in the model training process; at the same time, a sliding window is set to eliminate old data in the Transformer model's dataset that does not conform to the data distribution.

9. A reinforcement learning-based satellite pursuit method based on orbital motion prediction according to claim 8, characterized in that, The loss function of the Transformer model is: ； The result predicted by the Transformer, and This is real trajectory data.

10. A reinforcement learning satellite escape tracking system implementing the method of any one of claims 1-9, characterized in that, include: The state observation and reward design unit designs the state observation and reward of both sides in a single-agent game as input to the agent and feedback to the environment. The SAC policy network training unit takes state observations as input, constructs and trains the initial SAC policy network, builds the loss function, and introduces a curiosity mechanism as an external signal input to the reward. The dataset construction unit constructs input vectors for the Transformer model based on the pursuer's observations and the escaper's strategy, encodes the vectors at their positions, and generates sequences as the dataset for the Transformer model. The Transformer model training unit uses the constructed dataset to train the Transformer model and designs a loss function for backpropagation. The integrated decision network construction unit uses the output of the trained Transformer model as part of the input observation of the SAC policy network, reconstructs the SAC input state observation, builds the integrated decision network, and trains it together. A loss function is designed for backpropagation to obtain an iterative training model. The output unit uses a trained integrated decision network to obtain the escaped satellite's orbit change amount, and then executes the orbit change action based on the orbit change amount of the pursuing satellite.

Images