Satellite autonomous rendezvous control method based on adaptive time sequence hidden space learning

Abstract

The invention discloses a satellite autonomous rendezvous control method based on adaptive time sequence hidden space learning, and relates to the technical field of deep reinforcement learning and spacecraft control. According to the method, a satellite long-distance rendezvous task is modeled into a partially observable Markov decision process, and the core innovation is that the decision time step length is converted from a fixed hyper-parameter to a key decision variable which can be autonomously optimized by an intelligent agent. A continuous step size scaling factor is introduced by expanding an action space, so that end-to-end collaborative optimization of a control strategy and a decision time sequence is realized; designing a self-adaptive time sequence decision-making mechanism based on hidden state perception, and enabling an intelligent agent to dynamically adjust decision-making frequency according to a task stage; and constructing a multi-target composite reward function containing step length efficiency reward and punishment, and guiding the intelligent agent to intelligently balance the control precision and task time consumption under multiple constraints. Experiments prove that the method is obviously superior to a fixed step length baseline algorithm in task success rate, fuel economy and time efficiency, shows strong generalization ability, and provides an efficient and robust solution for intelligent autonomous control of spacecrafts.

Description

Technical Field

[0001] This invention relates to the fields of deep reinforcement learning technology and spacecraft control technology, specifically to a satellite autonomous rendezvous control method based on adaptive temporal latent space learning. Background Technology

[0002] Autonomous satellite rendezvous control is of great significance in modern space missions. It refers to the process of controlling a tracking satellite to approach a target satellite until a predetermined relative distance is reached and specific observation constraints are met. In recent years, deep reinforcement learning algorithms have provided new ideas for overcoming the limitations of manually designed control parameters and realizing intelligent control strategies. However, existing deep reinforcement learning methods still face key challenges when applied to long-distance satellite rendezvous missions: most methods use a fixed decision time step, which cannot adapt to the dynamic requirements of control frequency at different stages of the rendezvous mission. This invention proposes an autonomous satellite rendezvous control method based on adaptive temporal latent space learning. It models the long-distance satellite rendezvous mission as a partially observable Markov decision process. By transforming the decision time step from a fixed hyperparameter into a key decision variable that the agent can autonomously optimize, it achieves synergistic optimization of the control strategy and the decision time sequence, thereby deriving an efficient, robust, and highly generalizable autonomous rendezvous control strategy. Summary of the Invention

[0003] This invention designs a satellite autonomous rendezvous control method based on adaptive temporal latent space learning. In long-distance rendezvous, the initial distance between two satellites is over 200 kilometers. The goal is to use a single, stable control strategy to guide the tracking satellite towards the target satellite, ultimately reducing the distance between the two satellites to within 10 kilometers of the effective payload range, while simultaneously meeting the illumination observation condition of a solar angle of no less than 15 degrees, and ensuring that the fuel consumption throughout the process does not exceed a preset budget. This is achieved by modeling the long-distance satellite rendezvous mission as a partially observable Markov decision process and training the adaptive temporal control strategy for the rendezvous scenario using a model-based deep reinforcement learning framework. The main components of this method include: modeling the partially observable Markov decision process for long-distance satellite rendezvous, environmental dynamics learning based on a latent space world model, and an adaptive temporal decision mechanism.

[0004] The satellite autonomous rendezvous control method based on adaptive temporal latent space learning includes the following steps: Step 1: Model the long-distance satellite rendezvous mission as a partially observable Markov decision process, defining the state space, observation space, action space, reward function, and state transition function; Step 2: Construct a world model based on a cyclic state-space model, including an encoder, decoder, dynamic predictor, reward predictor, continuity predictor, and sequence model, to learn environmental dynamics and generate imagined trajectories in the latent space; Step 3: Design an adaptive timing decision-making mechanism, using the decision time step as an optimizable decision variable for the agent, and introduce a step scaling factor by expanding the action space to achieve coordinated optimization of control commands and decision timing; Step 4: Construct a multi-objective composite reward function, which includes distance convergence reward, solar angle maintenance reward, fuel penalty, boundary safety penalty, step size efficiency reward, distance change reward, constraint flag reward, and terminal reward, to guide the agent to optimize control performance under multiple constraints; Step 5: Employ a model-based reinforcement learning framework, alternately perform environment interaction, world model learning, and model-based policy training, and optimize the policy network and value network end-to-end until an autonomous rendezvous control policy that meets the task requirements is obtained.

[0005] In step 2, the sequence model of the world model calculates the current deterministic hidden state based on the hidden state, velocity pulse and physical time interval of the previous moment, and explicitly uses the time interval as an input condition, so that the model learns the precise dynamic process of the state evolving with a specific control quantity and a specific duration.

[0006] In step 3, the adaptive temporal decision-making mechanism expands the action space into a 4-dimensional continuous vector, including a three-dimensional velocity pulse and a step-size scaling factor. The actual decision step size is obtained by linear mapping and pruning of the step-size scaling factor, enabling the agent to continuously adjust the decision interval within a preset boundary.

[0007] In step 4, the step size efficiency reward is calculated based on the deviation between the normalized actual step size and the ideal step size. The ideal step size is dynamically determined according to the current relative distance. At the same time, a time efficiency reward is introduced to encourage the use of large step sizes in long-distance segments to improve task efficiency.

[0008] In step 5, the model-based policy training phase uses a world model to generate imagined trajectories, calculates the target value based on reward, updates the policy network through policy gradients, and updates the value network through classification cross-entropy loss. During the training process, there is no real interaction with the environment.

[0009] In step 1, the observation space includes the relative position and velocity in the LVLH coordinate system, the solar angle, the solar direction vector, the distance constraint flag, the angle constraint flag, and the time scale factor, providing the agent with some observable environmental information.

[0010] In step 4, the distance change reward is calculated based on the relative distance change between two adjacent steps. When the distance decreases, a positive reward is given with a coefficient related to the initial distance. When the distance increases, a negative penalty is given with a penalty coefficient greater than the benchmark reward coefficient when approaching, making the agent more sensitive to the behavior of moving away.

[0011] The present invention also includes a control system comprising the above-described satellite autonomous rendezvous control method based on adaptive temporal latent space learning. The control system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the above-described satellite autonomous rendezvous control method.

[0012] The present invention also includes a computer-readable storage medium containing the above-described satellite autonomous rendezvous control method based on adaptive temporal latent space learning, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements the above-described satellite autonomous rendezvous control method.

[0013] This method models the partially observable Markov decision process for long-distance satellite rendezvous as follows: make and These represent the position and velocity vectors of the target and tracking stars in the EME2000 Earth inertial coordinate system, respectively. Since the absolute state of the tracking star cannot be completely observed, an observation space is introduced. ,Will Observations at time The model is as follows:

[0014] in The position and velocity of the tracking star relative to the target star in the LVLH coordinate system; The angle between the sun and the sun; This is the unit vector of the sun's direction in the LVLH coordinate system; These are the distance constraint and angle constraint flags; It is the ratio of the time already consumed to the maximum allowed task time.

[0015] The core innovation of this invention lies in the motion space. The design: The decision time interval is treated as an optimizable object for the agent. Moment value Modeled as a 4-dimensional continuous vector:

[0016] in This indicates a normalized velocity pulse command applied along the three axes of the LVLH coordinate system; This is the step size scaling factor, used to dynamically determine the duration of the current action; the actual simulated step size. Calculated using the following mapping:

[0017] in Based on the reference value for the step size, and For scaling factor boundaries, and These are the upper and lower limits of the physical step size. This is a truncation function.

[0018] Will The reward value at time step is modeled as a multi-objective composite function:

[0019] The components are defined as follows: Distance to convergence reward : Guide the tracking star into and maintain a relative distance Within the effective load range Inside.

[0020]

[0021] Sun angle retention reward Encourage the angle of the sun. Greater than the minimum threshold .

[0022]

[0023] Fuel Penalty Optimize fuel efficiency. This is the ratio of cumulative fuel consumption to total fuel budget. In the formula... This is an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise, meaning that a double penalty will be applied when the condition is true.

[0024]

[0025] Border security penalties To prevent the tracking satellite from escaping beyond a safe range, among which For the maximum permissible relative distance, .

[0026]

[0027] Step length efficiency reward Balancing control precision with time efficiency The current step size after regularization. This represents the optimal step size corresponding to the current distance.

[0028]

[0029] Distance change reward Provides intensive approach behavior feedback. .

[0030]

[0031] Constraint Mark Reward

[0032] Terminal rewards :

[0033] The adaptive temporal latent space learning (ATLSL) algorithm framework involved in this invention is further described below, which includes three main modules: a temporal adaptive world model (TASSM), an Actor network, and a Critic network. The specific training method includes the following stages: Phase 1: Environmental Interaction Phase Initialize the time-adaptive world model (Including encoder, decoder, dynamic predictor, reward predictor, continuous predictor, sequence model), policy network and value network The parameters are used to initialize the experience replay buffer. Based on the current policy network Interacting with the environment Step 1: Collect trajectory data Store in the playback pool ,in This is a Boolean value indicating whether the round has terminated. The sequence model is based on the deterministic and stochastic hidden states from the previous time step. Velocity pulse and physical time interval Calculate the current deterministic hidden state:

[0034] The encoder will observe and deterministic hidden states Encoded as a random hidden state:

[0035] Phase 2: World Model Learning Phase From the replay pool The number of samples drawn is The trajectory segments, each segment length Fixed-policy network and value network The parameters, based on the loss function Training the world model :

[0036] in To stop the gradient operator, the specific forms of each loss term are as follows:

[0037] Phase 3: Model-based policy training phase Fixed World Model Parameters from the playback pool Starting from each initial state sampled, the current policy network... To guide, unfold a fixed step size. The imagined trajectory. Based on the imagined trajectory, according to the loss function... Training Policy Network :

[0038] in Return As a discount factor, This is the entropy coefficient.

[0039] According to the loss function Training Value Network :

[0040] in This is a two-hot encoding function.

[0041] Phases 1 through 3 are executed in a loop until the policy network is reached. It can reliably output successful long-distance rendezvous control strategies. Attached Figure Description

[0042] Figure 1 A schematic diagram of the world model structure; Figure 2 This is a training flowchart.

Claims

1. A satellite autonomous rendezvous control method based on adaptive temporal latent space learning, characterized in that, Includes the following steps: Step 1: Model the long-distance satellite rendezvous mission as a partially observable Markov decision process, defining the state space, observation space, action space, reward function, and state transition function; Step 2: Construct a world model based on a cyclic state-space model, including an encoder, decoder, dynamic predictor, reward predictor, continuity predictor, and sequence model, to learn environmental dynamics and generate imagined trajectories in the latent space; Step 3: Design an adaptive timing decision-making mechanism, using the decision time step as an optimizable decision variable for the agent, and introduce a step scaling factor by expanding the action space to achieve coordinated optimization of control commands and decision timing; Step 4: Construct a multi-objective composite reward function, which includes distance convergence reward, solar angle maintenance reward, fuel penalty, boundary safety penalty, step size efficiency reward, distance change reward, constraint flag reward, and terminal reward, to guide the agent to optimize control performance under multiple constraints; Step 5: Employ a model-based reinforcement learning framework, alternately perform environment interaction, world model learning, and model-based policy training, and optimize the policy network and value network end-to-end until an autonomous rendezvous control policy that meets the task requirements is obtained.

2. The satellite autonomous rendezvous control method based on adaptive temporal latent space learning according to claim 1, characterized in that, In step 2, the sequence model of the world model calculates the current deterministic hidden state based on the hidden state, velocity impulse and physical time interval of the previous moment, and explicitly uses the time interval as an input condition, so that the model learns the precise dynamic process of the state evolving with a specific control quantity and a specific duration.

3. The satellite autonomous rendezvous control method based on adaptive temporal latent space learning according to claim 1, characterized in that, In step 3, the adaptive temporal decision-making mechanism expands the action space into a 4-dimensional continuous vector, including a three-dimensional velocity pulse and a step-size scaling factor. The actual decision step size is obtained by linear mapping and pruning of the step-size scaling factor, enabling the agent to continuously adjust the decision interval within a preset boundary.

4. The satellite autonomous rendezvous control method based on adaptive temporal latent space learning according to claim 1, characterized in that, In step 4, the step length efficiency reward is calculated based on the deviation between the normalized actual step length and the ideal step length. The ideal step length is dynamically determined according to the current relative distance. At the same time, a time efficiency reward is introduced to encourage the use of large step lengths in long-distance segments to improve task efficiency.

5. The satellite autonomous rendezvous control method based on adaptive temporal latent space learning according to claim 1, characterized in that, In step 5, the model-based policy training phase uses a world model to generate imagined trajectories, based on... - The reward calculation targets the value, updates the policy network through policy gradient, and updates the value network through classification cross-entropy loss. During training, there is no real interaction with the environment.

6. The satellite autonomous rendezvous control method based on adaptive temporal latent space learning according to claim 1, characterized in that, The observation space in step 1 includes the relative position and velocity in the LVLH coordinate system, the solar angle, the solar direction vector, the distance constraint flag, the angle constraint flag, and the time scale factor, providing the agent with some observable environmental information.

7. The satellite autonomous rendezvous control method based on adaptive temporal latent space learning according to claim 1, characterized in that, In step 4, the distance change reward is calculated based on the relative distance change between two adjacent steps. When the distance is shortened, a positive reward is given with a coefficient related to the initial distance. When the distance is increased, a negative penalty is given with a penalty coefficient greater than the benchmark reward coefficient when approaching, making the agent more sensitive to the behavior of moving away.

8. A control system comprising the satellite autonomous rendezvous control method based on adaptive temporal latent space learning as described in any one of claims 1-7, characterized in that, The control system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the satellite autonomous rendezvous control method as described in any one of claims 1 to 7.

9. A computer-readable storage medium comprising the satellite autonomous rendezvous control method based on adaptive temporal latent space learning as described in any one of claims 1-7, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the satellite autonomous rendezvous control method as described in any one of claims 1 to 7.

Images