Satellite long-distance rendezvous and docking control method based on model deep reinforcement learning

Abstract

The invention relates to a satellite long-distance rendezvous and docking control method based on model deep reinforcement learning, which is characterized in that the initial distance of two satellites in a long-distance rendezvous and docking task scene is more than 100km, and a single stable control strategy is expected to be used for controlling a working satellite to approach a target satellite; finally, the distance between the two satellites is reduced to be within 5 m, the final relative speed is not higher than 0.5 m / s, the remote rendezvous and docking task of the satellites is modeled into a partially observable Markov decision process, expert data and a world model are utilized, a control strategy of the rendezvous and docking scene is trained through a model deep reinforcement learning algorithm, and the rendezvous and docking task of the satellites is obtained. On the basis, the time for training to obtain an effective strategy is shortened, and the success rate of strategy control rendezvous and docking is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning, and in particular to a satellite long-distance rendezvous and docking control method based on model-based deep reinforcement learning. Background Art

[0002] Satellite rendezvous and docking control is of great significance in contemporary aerospace. Rendezvous and docking refers to the process of controlling the working satellite to approach the target satellite until it reaches a certain relative distance and speed limit. Deep reinforcement learning algorithms can break through the limitations of artificially designed control parameters and realize intelligent control strategies. However, when the working satellite and the target satellite are initially far apart, the deep reinforcement learning algorithm needs to improve the efficiency of exploring the universe, otherwise it is impossible to explore a feasible control strategy. The present invention proposes a satellite long-distance rendezvous and docking control method based on model-based deep reinforcement learning, which models the satellite long-distance rendezvous and docking task as a partially observable Markov decision process, and efficiently explores through a model-based deep reinforcement learning algorithm to derive an effective control strategy. Summary of the invention

[0003] The present invention designs a satellite long-distance rendezvous and docking control method based on model-based deep reinforcement learning. It is characterized in that: the initial distance between the two satellites for long-distance rendezvous and docking is more than 100km, and it is expected to use a single stable control strategy to control the working satellite to approach the target satellite, and finally reduce the distance between the two satellites to within 5m, and the final relative speed is not higher than 0.5m / s, by modeling the satellite long-distance rendezvous and docking task as a partially observable Markov decision process, and using expert data and world models, the control strategy of the above rendezvous and docking scenario is trained by a model-based deep reinforcement learning algorithm. The main components of this method include: the construction of a partially observable Markov decision process for satellite long-distance rendezvous and docking, and the training of the rendezvous and docking control strategy based on model-based deep reinforcement learning.

[0004] The partially observable Markov decision process for long-distance satellite rendezvous and docking in this method is modeled as:

[0005] Let x target,t ,v target,t and x chaser,t ,v chaser,t Represent the coordinates and velocity vectors of the target satellite and the working satellite in the EME2000 Earth inertial coordinate system, respectively. Let x LVLH,t ,v LVLH,t represents the relative position and velocity vector of the working satellite relative to the target satellite in the LVLH orbit coordinate system with the target satellite as the coordinate origin. The observation value at time t is modeled as:

[0006] o t =(x target,t ,vtarget,t ,x chaser,t ,v chaser,t ,x LVLH,t ,v LVLH,t )

[0007] Let × be the Cartesian product. The working star can accelerate or decelerate on the x, y, and z axes or keep the current speed unchanged. The speed of each axis changes by 0.5 m / s each time. The action value at time t is modeled as a discrete action:

[0008] a t ∈A={-1,0,1}×{-1,0,1}×{-1,0,1}

[0009] Δv t =0.5a t m / s

[0010] Model the reward value at time t as r t =R final +R distance +R velocity +R collision , where R final To terminate the reward, R distance is the distance reward, R speed is the speed reward, R collision The collision reward is calculated as follows:

[0011]

[0012] The following further describes the rendezvous and docking control strategy training based on model-based deep reinforcement learning involved in this patent:

[0013] Using the world model neural network w θ 、Strategy neural network π μ and the value neural network V σ , where θ, μ, σ are network parameters; the world model neural network w θ It includes six modules: encoder ec, decoder dc, reward predictor rp, dynamic predictor dp, termination predictor cp and sequence model sq; the training method includes the following stages:

[0014] Phase 1: Based on the existing expert strategy, the replay pool is pre-filled. The data stored in the replay pool includes o t ,r t ,c t ,a t , where c t Used to indicate whether the round ends at time t;

[0015] Phase 2: Iteratively train the world model neural network w θ、Strategy neural network π μ and the value neural network V σ , as follows:

[0016] 1) Based on the current policy neural network π μ , interact with the environment to obtain data and add it to the playback pool;

[0017] 2) Extract T amount of data from the replay pool, let KL represent the calculation of Kullback-Leibler divergence, sg represent no gradient propagation, and train the world model neural network w according to the loss function L(θ) θ :

[0018]

[0019] z t ~ec(z t |h t ,o t )

[0020] h t =sq(h t-1 ,z t-1 ,a t-1 )

[0021] 3) Using the world model neural network w θ Generate a length of H T The imagined trajectory;

[0022] 4) Based on the imagined trajectory, according to the loss function L actor (μ) Training strategy neural network π μ :

[0023]

[0024] 5) Let twohot be the two-hot encoding function, sign be the sign function, and train the value neural network V based on the imaginary trajectory according to the loss function σ :

[0025]

[0026] symlog(x)=sign(x)ln(|x|+1)

[0027] 6) Loop from 1) to 5) until the policy neural network π μ Capable of completing successful long-distance rendezvous and docking control.

Claims

1. A satellite long-distance rendezvous and docking control method based on model-based deep reinforcement learning, characterized in that: The method includes modeling of partially observable Markov decision processes for long-distance satellite rendezvous and docking; training of rendezvous and docking control strategies based on model-based deep reinforcement learning; The partially observable Markov decision process for long-distance satellite rendezvous and docking is modeled as: Let x target,t ,v target,t and x chaser,t ,v chaser,t Represent the coordinates and velocity vectors of the target satellite and the working satellite in the EME2000 Earth inertial coordinate system, respectively. Let x LVLH,t ,v LVLH,t represents the relative position and velocity vector of the working satellite relative to the target satellite in the LVLH orbit coordinate system with the target satellite as the coordinate origin. The observation value at time t is modeled as: o t =(x target,t ,v target,t ,x chaser,t ,v chaser,t ,x LVLH,t ,v LVLH,t ) Let × be the Cartesian product. The working star can accelerate or decelerate on the x, y, and z axes or keep the current speed unchanged. The speed of each axis changes by 0.5 m / s each time. The action value at time t is modeled as a discrete action: a t ∈A={-1,0,1}×{-1,0,1}×{-1,0,1} Δv t =0.5a t m / s Model the reward value at time t as r t =R final +R distance +R velocity +R collision , where R final To terminate the reward, R distance is the distance reward, R speed is the speed reward, R collision The collision reward is calculated as follows: The rendezvous and docking control strategy training based on model-based deep reinforcement learning is: Using the world model neural network w θ 、Strategy neural network π μ and the value neural network V σ , where θ, μ, σ are network parameters; the world model neural network w θ It includes six modules: encoder ec, decoder dc, reward predictor rp, dynamic predictor dp, termination predictor cp and sequence model sq; control strategy training includes the following stages: Phase 1: Based on the existing expert strategy, the replay pool is pre-filled. The data stored in the replay pool includes o t ,r t ,c t ,a t , where c t Used to indicate whether the round ends at time t; Phase 2: Iteratively train the world model neural network w θ 、Strategy neural network π μ and the value neural network V σ , as follows: 1) Based on the current policy neural network π μ , interact with the environment to obtain data and add it to the playback pool; 2) Extract T amount of data from the replay pool, let KL represent the calculation of Kullback-Leibler divergence, sg represent no gradient propagation, and train the world model neural network w according to the loss function L(θ) θ : z t ~ec(z t |h t ,o t ) h t =sq(h t-1 ,z t-1 ,a t-1 ) 3) Using the world model neural network w θ Generate a length of H T The imagined trajectory; 4) Based on the imagined trajectory, according to the loss function L actor (μ) Training strategy neural network π μ : 5) Let twohot be the two-hot encoding function, sign be the sign function, and train the value neural network V based on the imaginary trajectory according to the loss function σ : symlog(x)=sign(x)ln(|x|+1) 6) Loop from 1) to 5) until the policy neural network π μ Capable of completing successful long-distance rendezvous and docking control.

Images