Non-cooperative spacecraft active tracking method based on deep reinforcement learning
By using Transformer network and deep deterministic strategy gradient method in non-cooperative spacecraft active tracking algorithms, the problem of insufficient algorithm robustness and tracking accuracy in the existing technology is solved, and accurate and robust tracking and timing of spacecraft is achieved. Information processing improves tracking effect and algorithm robustness.
Patent Information
- Application Number
- CN202410966496.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-18
AI Technical Summary
The existing non-cooperative spacecraft active tracking algorithm based on deep reinforcement learning cannot effectively integrate spacecraft dynamics models and satellite orbit dynamics, cannot improve the robustness and tracking accuracy of the algorithm, and it is difficult to extract timing-related information from the training samples.
Using Transformer-based actor network and critic network, combined with deep deterministic strategy gradient method, learn the optimal active tracking strategy through an end-to-end way, process timing information and improve the robustness of the algorithm, and design a novel loss function to optimize Control policy.
Accurate and robust tracking of non-cooperative spacecraft is achieved, complex model construction and parameter adjustment in traditional control theory are avoided, and can deal with external perturbations and sensor observation interference, which significantly improves the convergence and tracking effect of the active tracking algorithm.
Smart Images

Figure CN119002255A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a non-cooperative spacecraft active tracking method based on deep reinforcement learning and belongs to the field of aerospace. Background Art
[0002] At present, the active tracking problem of spacecraft is mainly solved by traditional control methods, such as PID control, H∞ control, sliding mode control, etc. However, traditional control methods often rely on a more accurate control object model to solve the active tracking problem of spacecraft, and need to design a stable control law through complex and tedious mathematical derivation and carefully adjust the parameters of the controller to ensure the effectiveness of the control algorithm. In fact, the attitude and orbit motion information of non-cooperative spacecraft is often inaccurate. At the same time, there are many uncertain high-intensity external disturbances in the relative motion dynamics model of the chasing spacecraft and the target spacecraft, which poses a great challenge to the current active tracking method of spacecraft. In recent years, the vigorous development of deep reinforcement learning algorithms has provided a novel perspective for solving the active tracking problem of non-cooperative spacecraft.
[0003] The deep reinforcement learning algorithm mainly interacts with the environment through the agent, and learns the optimal control strategy based on the environmental rewards obtained. Its advantage is that it does not need to model the environment, and only needs to ensure that the dynamic model of the agent itself is accurate. The agent continuously updates the strategy through iterative interaction with the environment to obtain better strategy parameters. Its advantage is that when facing nonlinear, non-static, and non-deterministic environments, it can still optimize the control strategy of the agent in a large number of online / offline iterations, thereby obtaining better control effects. In addition, the deep reinforcement learning algorithm can add different external disturbances during the training process, so that the agent can learn a more robust action strategy and make corresponding adjustments in real time according to changes in the environment. Summary of the invention
[0004] The purpose of the present invention is to solve the defects of the existing non-cooperative spacecraft active tracking algorithm based on deep reinforcement learning, such as the inability to integrate the spacecraft dynamics model and the satellite orbit dynamics, the inability to effectively improve the robustness of the algorithm while ensuring the tracking accuracy, and the inability to effectively extract the timing-related information about the target from the training samples. A non-cooperative spacecraft active tracking method based on deep reinforcement learning is proposed.
[0005] A non-cooperative spacecraft active tracking method based on deep reinforcement learning. The specific process is as follows:
[0006] Step 1: Based on the basic parameters of the target spacecraft and the chasing spacecraft, the coordinate systems of the target spacecraft and the chasing spacecraft, and the universal gravitational force exerted on the target spacecraft and the chasing spacecraft, the universal gravitational force exerted on the target spacecraft is converted into the position and velocity of the target spacecraft, and the universal gravitational force exerted on the chasing spacecraft is converted into the position and velocity of the chasing spacecraft;
[0007] Step 2: Set the state space and action space, build the Transformer-based actor network and critic network, as well as the loss function of the actor network and the loss function of the critic network;
[0008] Step 3: Construct a reward function and obtain the trained actor network and critic network;
[0009] Step 4: Obtain the difference between the position of the target spacecraft and the chasing spacecraft and the expected position, as well as the speed difference between the target spacecraft and the chasing spacecraft, and input them into the trained actor network, which outputs the action of the current time step.
[0010] The beneficial effects of the present invention are:
[0011] The present invention provides an end-to-end non-cooperative spacecraft active tracking algorithm based on deep reinforcement learning, which integrates spacecraft dynamics model and satellite orbit dynamics, can process timing information and improve algorithm robustness.
[0012] The present invention proposes a non-cooperative spacecraft active tracking algorithm based on deep reinforcement learning. The algorithm takes the spacecraft pose information with estimated noise as input, combines the relative orbital motion dynamics model of the target spacecraft and the chasing spacecraft, and learns the approximately optimal active tracking strategy in an end-to-end manner through a deep deterministic policy gradient method.
[0013] This paper proposes an active visual tracker based on unsupervised deep reinforcement learning, which realizes accurate and robust tracking of non-cooperative spacecraft. The tracker realizes optimal control of the second-order control system, avoiding the complex model construction and parameter adjustment in traditional control theory. In addition, deep reinforcement learning itself has good robustness to state data containing disturbances, and can cope with possible sensor observation interference.
[0014] The present invention designs a novel Transformer-based actor network and critic network, which effectively extracts high-level semantic information and temporal connections from the sequence input state, implicitly represents the coupled motion between the target spacecraft and the chasing spacecraft, greatly improves the convergence of the active tracking algorithm and optimizes the tracking effect;
[0015] The present invention improves the loss functions of the actor network and the critic network, so that they can learn the optimal control strategy better, faster and more stably according to the input sequence state, and improves the tracking effect of the active tracking algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A diagram of the actor network and critic network structure for deep reinforcement learning networks;
[0017] Figure 2 It is a flow chart of the algorithm proposed in the present invention;
[0018] Figure 3 It is a schematic diagram of the final effect of the present invention;
[0019] Figure 4 This is a control effect diagram of the PID control algorithm in the model of the present invention. DETAILED DESCRIPTION
[0020] Specific implementation method 1: This implementation method is a non-cooperative spacecraft active tracking method based on deep reinforcement learning. The specific process is as follows:
[0021] Step 1: Based on the basic parameters of the target spacecraft and the chasing spacecraft, the coordinate systems of the target spacecraft and the chasing spacecraft, and the universal gravitational force exerted on the target spacecraft and the chasing spacecraft, the universal gravitational force exerted on the target spacecraft is converted into the position and velocity of the target spacecraft, and the universal gravitational force exerted on the chasing spacecraft is converted into the position and velocity of the chasing spacecraft;
[0022] Step 2: Set the state space and action space, build the Transformer-based actor network and critic network, as well as the loss function of the actor network and the loss function of the critic network;
[0023] The loss function enables the actor network and the critic network to establish the temporal relationship between the current input state and the historical input state;
[0024] Step 3: Construct a reward function and obtain the trained actor network and critic network;
[0025] Step 4: Obtain the difference between the position of the target spacecraft and the chasing spacecraft and the expected position, as well as the speed difference between the target spacecraft and the chasing spacecraft, and input them into the trained actor network. The trained actor network outputs the action of the current time step, which is the control instruction for the chasing spacecraft force.
[0026] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that: in step 1, based on the basic parameters of the target spacecraft and the chasing spacecraft, the coordinate systems of the target spacecraft and the chasing spacecraft, and the universal gravitational force exerted on the target spacecraft and the chasing spacecraft, the universal gravitational force exerted on the target spacecraft is converted into the position and speed of the target spacecraft, and the universal gravitational force exerted on the chasing spacecraft is converted into the position and speed of the chasing spacecraft;
[0027] The specific process is:
[0028] Step 11: Define the basic parameters of the target spacecraft and the chasing spacecraft. The specific process is as follows:
[0029] The basic parameters of the target spacecraft are: the target spacecraft's own mass, the target spacecraft's maximum thrust, and the target spacecraft's adjustment frequency; its own weight is 100kg, its maximum thrust is 4N, and its adjustment frequency is 10Hz;
[0030] The basic parameters of the chasing spacecraft are: the chasing spacecraft's own mass, the chasing spacecraft's maximum thrust, and the chasing spacecraft's adjustment frequency; its own weight is 100kg, its maximum thrust is 4N, and its adjustment frequency is 10Hz;
[0031] Step 12: Set the coordinate systems of the target spacecraft and the chasing spacecraft. The specific process is as follows:
[0032] The coordinate system of the target spacecraft and the chasing spacecraft uses the geocentric inertial coordinate system globally, and only considers the two-dimensional motion of the target spacecraft and the chasing spacecraft, without considering the position transformation, velocity transformation and thrust output on the z-axis. The origin of the geocentric inertial coordinate system is the center of the earth.
[0033] The position information of the chasing spacecraft and the target spacecraft in the present invention is the coordinates of the chasing spacecraft and the target spacecraft in the geocentric inertial coordinate system;
[0034] Step 13: Define the gravitational force on the target spacecraft and the chasing spacecraft. The specific process is as follows:
[0035] The gravitational force F on the target spacecraft g,tar and the gravitational force F on the chasing spacecraft g,c As shown below:
[0036]
[0037] r c =r tar +c,c∈[-150,150]
[0038] in,
[0039] r tar is the initial orbit radius of the target spacecraft, rc To chase the orbital radius of the spacecraft;
[0040] G is the gravitational constant, which is 6.67×10 -11 N·m 2 / kg 2 ;
[0041] M is the mass of the Earth, which is 5.977×10 24 kg;
[0042] m tar is the target spacecraft's own weight, m c To pursue the dead weight of the spacecraft;
[0043] c is a random variable used to adjust the initial position of the chasing spacecraft to increase the robustness of the algorithm;
[0044] Step 14, converting the gravitational force on the target spacecraft into the position and velocity of the target spacecraft;
[0045] Convert the gravitational force on the chasing spacecraft into the position and velocity of the chasing spacecraft;
[0046] The specific process is:
[0047] Step 141, convert the gravitational force on the target spacecraft into the position and velocity of the target spacecraft; the specific conversion is shown in the following formula:
[0048]
[0049] Where · represents matrix dot product, P t,tar represents the position of the target spacecraft in the geocentric inertial coordinate system at time t, V t,tar represents the velocity of the target spacecraft in the geocentric inertial coordinate system at time t, P t+1,tar represents the position of the target spacecraft in the geocentric inertial coordinate system at time t+1, V t+1,tar represents the speed of the target spacecraft in the geocentric inertial coordinate system at time t+1; Δt is the adjustment frequency; A is the spacecraft information conversion matrix at time t;
[0050] The target spacecraft is not controlled by the deep reinforcement learning network, so there is no F when calculating the position and velocity changes of the target spacecraft. RL item;
[0051] Represents the unit vector of the target spacecraft at its current position, which is used to decompose the gravitational force into its component forces; the specific form is shown in the following formula:
[0052]
[0053] Among them, r2,tar represents the vector position of the target spacecraft in the geocentric inertial coordinate system, and r1 is the vector position of the center of the earth in the geocentric inertial coordinate system;
[0054] Step 142, convert the gravitational force on the chasing spacecraft into the position and velocity of the chasing spacecraft; the specific conversion is shown in the following formula:
[0055]
[0056] Among them, F RL represents the action output by the actor network (the actor network outputs the action, and the output of the critic network is only used in the update process to guide the update process of the actor network and the critic network), P t,c represents the position of the chasing spacecraft in the geocentric inertial coordinate system at time t, V t,c represents the speed of the chasing spacecraft in the geocentric inertial coordinate system at time t, P t+1,c represents the position of the chasing spacecraft in the geocentric inertial coordinate system at time t+1, V t+1,c represents the speed of the chasing spacecraft in the geocentric inertial coordinate system at time t+1; Δt is the adjustment frequency; A is the spacecraft information conversion matrix at time t;
[0057] It is the unit vector of the chasing spacecraft at its current position, which is used to decompose the gravitational force into its components; the specific form is shown in the following formula:
[0058]
[0059] where r 2,c represents the vector position of the chasing spacecraft in the geocentric inertial coordinate system, and r1 is the vector position of the center of the earth in the geocentric inertial coordinate system;
[0060] The other steps and parameters are the same as those in the first embodiment.
[0061] Specific implementation method three: This implementation method is different from specific implementation method one or two in that: the specific form of the spacecraft information transposition matrix A at time t is as follows:
[0062]
[0063] The other steps and parameters are the same as those in the first or second embodiment.
[0064] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that: in step 2, the state space and action space are set, and the actor network and critic network based on Transformer are constructed, as well as the loss function of the actor network and the loss function of the critic network; and the actor network and the critic network are enabled to establish a temporal relationship between the current input state and the historical input state through the loss function; and the following steps are included:
[0065] Step 21, setting the state space;
[0066] The state space contains the difference between the position of the target spacecraft and the chasing spacecraft and the expected position, as well as the velocity difference between the target spacecraft and the chasing spacecraft;
[0067] The state space is a continuous state space;
[0068] The state space is a dictionary, which includes a two-dimensional vector representing the position difference and a two-dimensional vector representing the speed difference. In the present invention, n consecutive training samples are integrated and input into the deep reinforcement learning network as a training sample for training. Therefore, the specific form of the training sample in the present invention is [B×4×n]. In the present invention, n is defined as 10, B is the batch, which is set to 64 in the present invention, so the input state should be [s t-9 ,s t-8 ,...,s t ];
[0069] Step 22: Set the action space;
[0070] The action space is defined as a two-dimensional vector, and the range of the two-dimensional vector is specified between -4 and 4;
[0071] The action space is a continuous action space: a = [a x ,a y ],a x ∈[-4,4],a y ∈[-4,4];
[0072] Among them, a is the action, a x is the motion applied to the chasing spacecraft on the x-axis of the geocentric inertial coordinate system, a y is the action applied to the chasing spacecraft on the y-axis of the geocentric inertial coordinate system;
[0073] Step 23: Construct actor network and critic network;
[0074] The actor network includes: Embedding layer, Positional Encoding layer, first encoding layer (EncoderLayer), second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first Dimension Reduce layer, fourth FC layer, second DimensionReduce layer, fifth FC layer, output layer;
[0075] The FC layer is a fully connected layer;
[0076] The specific processing process of the actor network is:
[0077] Input the current state data into the Embedding layer, Positional Encoding layer, the first encoding layer (Encoder Layer), the second encoding layer, the third encoding layer, the first FC layer, the second FC layer, the third FC layer, the first Dimension Reduce layer, the fourth FC layer, the second Dimension Reduce layer, the fifth FC layer, and the output layer in sequence. The output layer outputs the action in the current state (the input is the current state, and the output is the action in the current state);
[0078] The critic network includes: Embedding layer, Positional Encoding layer, first encoding layer (Encoder Layer), second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first Dimension Reduce layer, fourth FC layer, second Dimension Reduce layer, Concatenate layer, fifth FC layer, output layer;
[0079] The specific processing process of the critic network is:
[0080] The current state data is sequentially input into the Embedding layer, Positional Encoding layer, the first encoding layer (Encoder Layer), the second encoding layer, the third encoding layer, the first FC layer, the second FC layer, the third FC layer, the first DimensionReduce layer, the fourth FC layer, the second DimensionReduce layer. The feature data output by the second Dimension Reduce layer is combined with the current action corresponding to the current state and then sequentially input into the fifth FC layer and the output layer. The output layer outputs the current state and the evaluation value data of the current action (the critic network input is the current state action pair (s t ,a t ), the output is (s t ,a t)’s evaluation q value);
[0081] The input in the present invention has strong temporal continuity. In order to better extract the temporal information in the input, position coding is first used to define the sequence of the input information.
[0082] The Transformer encoder is used to extract the temporal features in the continuous input sequence, and the network output dimension is satisfied through the fully connected layer and the dimensionality reduction layer. The actor network and the critic network structure of the present invention are as follows: Figure 1 As shown;
[0083] First, the input data is mapped to a higher dimension using the Embedding Layer. In the present invention, the dimension of the Embedding Layer is set to 128 bits. The Positional Encoding Layer is used to define the sequence of the input data to show its temporal nature. The three-layer Encoder Layer is used to extract various features of the input information, especially the temporal information. The three-layer FC Layer is used to reduce the last dimension of the data dimension to 1 dimension, and then the DimensionReduce Layer is used to reduce the dimension of the data, and the fourth FC Layer is used to further reduce the last dimension of the data to 1 dimension.
[0084] In the actor network, the fifth FC layer is used to reduce the data to an output dimension suitable for the environment, while in the critic network, it needs to be combined with the input batch actions and directly give the output for weighted averaging;
[0085] Step 24: construct a target actor network and a target critic network;
[0086] The target actor network includes: Embedding layer, Positional Encoding layer, first encoding layer (Encoder Layer), second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first Dimension Reduce layer, fourth FC layer, second Dimension Reduce layer, fifth FC layer, output layer;
[0087] The specific processing process of the target actor network is:
[0088] The next moment state s t+1The data is sequentially input into the Embedding layer, Positional Encoding layer, the first encoding layer (Encoder Layer), the second encoding layer, the third encoding layer, the first FC layer, the second FC layer, the third FC layer, the first Dimension Reduce layer, the fourth FC layer, the second Dimension Reduce layer, the fifth FC layer, and the output layer. The output layer outputs the next state s t+1 Next action a t+1 data;
[0089] The target critic network includes: Embedding layer, Positional Encoding layer, first encoding layer (Encoder Layer), second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first Dimension Reduce layer, fourth FC layer, second Dimension Reduce layer, Concatenate layer, fifth FC layer, output layer;
[0090] The specific processing process of the target critic network is:
[0091] The next moment state s t+1 The data is sequentially input into the Embedding layer, Positional Encoding layer, the first encoding layer (Encoder Layer), the second encoding layer, the third encoding layer, the first FC layer, the second FC layer, the third FC layer, the first Dimension Reduce layer, the fourth FC layer, the second Dimension Reduce layer. The second Dimension Reduce layer outputs the feature data and the next moment action corresponding to the next moment state input into the Concatenate layer (combined), and then sequentially input into the fifth FC layer and the output layer. The output layer outputs the state-action pair (s t+1 ,a t+1 )’s target value data;
[0092] Step 25: Set the loss function of the actor network; the specific process is:
[0093] The loss function of the actor network in the present invention first calculates the action in the current state by the actor network, which is the action without adding noise at this time, and calculates the cumulative expected return of the current state and action by the critic network. Since the gradient ascent is difficult to implement, the cumulative expected return is negative, and the obtained batch of actor network negative cumulative expectations are weighted averaged, and higher weights are given to steps with larger gradients, so as to obtain the loss function of the actor network;
[0094] The expression of the loss function of the actor network is:
[0095]
[0096] Among them, J μ is the loss function of the actor network, Q is the critic network, μ is the actor network, θ is the network parameter, and θ μ are the parameters of the actor network, θ Q is the parameter of the critic network, ω t is the weight, t is the time step, B is the batch size, which is set to 64 in the present invention, s t is the current state;
[0097] The loss function of the actor network is obtained as follows:
[0098] During the network update process, the dimension of the Q value obtained by the critic network is B×1. The cumulative expected return is obtained based on the Q value obtained by the critic network, and the cumulative expected returns are sorted, that is:
[0099] q qt =-Q(s t ,μ(s t |θ μ )|θ Q )
[0100] q q1 <q q2 <...<q q64
[0101] Among them, q qt To accumulate expected returns;
[0102] The input of the critic network is the current state-action pair (s t ,a t ), but this a t Not obtained during the experience collection process t Instead, a is retrieved through the actor network input state t , the output of the critic network is the evaluation value q of the current state and action t ;
[0103] (Here q q1 It only represents the largest Q value, not the first item in a batch of Q values, the same below) Since the gradient descent method is usually used to update network parameters in programming, the Q value here is negative, and the smaller the Q value, the higher the weight;
[0104] Different weights are assigned to Q values according to the order of cumulative expected returns from small to large:
[0105] ω q1 >ω q2>...>ω q64
[0106]
[0107] Among them, ω qt is the weight value corresponding to the Q value;
[0108] Based on q qt and ω qt Construct the loss function for the actor network:
[0109]
[0110] Step 26: Set the loss function of the critic network; the specific process is:
[0111] In the present invention, in addition to the actor and critic networks, target actor and target critic networks are also included. These two networks are mainly used to calculate the expected value of the state-action pair and compare it with the actual evaluation value. The obtained time series difference error is calculated by weighted average. A higher time series difference error will be given a higher weight, and its loss is solved according to the obtained result. The target actor network has the same structure as the actor network, and the target critic network has the same structure as the critic network.
[0112] The loss function of the critic network is expressed as:
[0113]
[0114] Among them J Q is the loss function of the critic network; r t is the reward value for executing the current action in the current state (obtained by the reward function calculation in step 33); done is to determine whether the current state is the end state; Q' is the target critic network; μ' is the target actor network; γ is a decay function approaching 1, which reduces the impact of the uncertainty of future estimated values on the target value; s t+1 is the state at time t+1; a t is the action at the current moment; θ μ' are the parameters of the target actor network; θ Q' are the parameters of the target critic network;
[0115] The remaining symbols are the same as in the previous step.
[0116] Target actor network μ'(s t+1 |θ μ' ) Calculate the action a in the current state t+1 ;
[0117] The loss function of the critic network is obtained as follows:
[0118] First, the critic network is used to calculate the current state-action pair (s t ,a t )’s evaluation value (the critic network input is (s t ,a t ), the output is q t ):
[0119] q t =Q(s t ,a t |θ Q )
[0120] Among them, q t is the evaluation value at the current moment, Q is the critic network, s t is the current state, a t is the action at the current moment, θ Q are the parameters of the critic network;
[0121] Through the target actor network, the action at the next moment is obtained (the target actor network input s t+1 , output a t+1 ):
[0122] a t+1 =μ'(s t+1 |θ μ' )
[0123] Among them, a t+1 is the action at the next moment predicted by the target actor network, μ' is the target actor network, s t+1 is the state of the next moment (before training begins, the collected (s t ,a t ,r t ,s t+1 ,D) Store it in an experience replay pool, and continue to add it during the subsequent training process. When the upper limit is reached, the first one will be removed and the latest one will be added, and so on. See step 34 for details. ,θ μ' are the parameters of the target actor network;
[0124] Through the target critic network, the state-action pair (s) is calculated by combining the next moment action, the next moment state, the reward obtained at the current step, and whether it is the terminal state. t+1 ,a t+1 ) target value:
[0125] y t =r t +γ(1-done)Q'(s t+1 ,a t+1 |θ Q ')
[0126] In the formula, y t is a state-action pair (s t+1 ,a t+1 ) target value, r t is the current state t Execute action a t The reward value (calculated by the reward function in step 33), Q' is the target critic network, θ Q' are the parameters of the target critic network;
[0127] The input of the target critic network is the current state-action pair (s t ,a t ), but this a t Not obtained during the experience collection process t Instead, a is retrieved through the actor network input state t , the output of the target critic network is the evaluation value q of the current state and action t ;
[0128] After calculating the target value and the evaluation value, subtract the target value from the evaluation value:
[0129] L t =y t -q t
[0130] The obtained L t The dimension is B×1. Through the weighted average algorithm described in step 24, the loss function of the critic network is finally obtained:
[0131]
[0132] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0133] Specific implementation method five: This implementation method is different from any one of specific implementation methods one to four in that: in step 3, a reward function is constructed to obtain a trained actor network, a critic network, a target actor network, and a target critic network;
[0134] Active tracking strategy learning based on deep reinforcement learning, through interaction with the environment, continuously obtains training samples and adjusts network parameters; obtains the active tracking network of non-cooperative spacecraft. The input data of this algorithm in the training and application process are the same, which are the difference between the position of the target spacecraft and the chasing spacecraft and the expected position, as well as the speed difference between the target spacecraft and the chasing spacecraft. Then the output is the control command of the chasing spacecraft force to complete the active tracking of the target spacecraft;
[0135] The following steps are involved:
[0136] Step 31: Initialize the initial positions of the target spacecraft and the chasing spacecraft; the specific process is as follows:
[0137] The initial position of the target spacecraft is in an orbit 7500 km from the center of the earth, which is the initial orbit radius r of the target spacecraft mentioned in step 13;
[0138] The position of the chasing spacecraft is on and inside a circle with a radius of 1 km and the target spacecraft as the center;
[0139] Step 32: Initialize the initial speeds of the target spacecraft and the chasing spacecraft. The specific process is as follows:
[0140] The initial velocity of the target spacecraft is set according to the following formula:
[0141]
[0142] Among them, v tar is the initial velocity of the target spacecraft at the initial position when it is subject to gravity; r tar is the initial orbit radius of the target spacecraft, F g,tar is the gravitational force on the target spacecraft, m tar is the target spacecraft’s own weight, X tar ,Y tar ,Z tar is the position coordinate of the target spacecraft in the geocentric inertial coordinate system; is the velocity unit vector of the target spacecraft at its current position, · is the dot product, is the velocity of the target spacecraft, v x,tar is the velocity of the target spacecraft on the x-axis of the geocentric inertial coordinate system, v x,tar To take a negative value, v y,tar is the velocity of the target spacecraft on the y-axis of the geocentric inertial coordinate system, v z,tar is the speed of the target spacecraft on the z-axis of the geocentric inertial coordinate system;
[0143] The initial velocity of the chasing spacecraft is set according to the following formula:
[0144]
[0145] Among them, v c is the initial velocity of the chasing spacecraft at the initial position under the gravitational force; r c is the orbit radius of the chasing spacecraft, F g,c To chase the gravitational force on the spacecraft, m c To pursue the spacecraft's own weight, X c ,Y c ,Z c The position coordinates of the chasing spacecraft in the geocentric inertial coordinate system; is the velocity unit vector of the chasing spacecraft at its current position, · is the dot product, is the speed of the chasing spacecraft, v x,c is the speed of the chasing spacecraft on the x-axis of the geocentric inertial coordinate system, v x,c Take a negative value, v y,c is the speed of the chasing spacecraft on the y-axis of the geocentric inertial coordinate system, v z,c The speed of the chasing spacecraft on the z-axis in the geocentric inertial coordinate system;
[0146] Step 33: Define the reward function.
[0147] Step 34: Set the experience replay pool parameters, the hyperparameters of the critic network and the actor network;
[0148] Step 35: Action selection strategy;
[0149] Step 36: Extract n consecutive training samples (n groups (s) from the experience replay pool t ,a t ,r t ,s t+1 ,D)) as a set of training samples, in the present invention, n is set to 10;
[0150] In the present invention, when updating the network parameters, the current training data is the training samples from time t-9 to time t, and the next training data will be the data from time t-8 to time t+1;
[0151] Step 37: Use the critic network to predict the action value of the current time step (the critic network input is the state and action of the current time step, and the critic network output is the predicted action value of the current time step);
[0152] Update the actor network parameters and the target actor network parameters;
[0153] Step 38: Use the actor network to predict the action of the current time step (the actor network input is the state of the current time step, and the actor network output is the predicted action of the current time step), and use the critic network to estimate the action value of the next time step. Finally, according to the temporal difference error (L t =y t -q t ,y t =r t +γ(1-done)Q'(s t+1 ,a t+1 |θ Q' )), update the critic network parameters and the target critic network parameters;
[0154] Step 39: Repeat steps 33 to 38 until convergence to obtain the trained actor network and critic network.
[0155] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0156] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that: the reward function is defined in step 33; the specific process is:
[0157] The reward function will be defined based on the relative position accuracy of the tracking. The closer the relative position of the chasing spacecraft and the target spacecraft is to the expected relative position, the smaller the corresponding position reward penalty will be. The more tracking steps there are, the higher the tracking reward will be.
[0158] Let the position coordinates of the chasing spacecraft in the geocentric inertial coordinate system (X c ,Y c ,Z c )=P c ;
[0159] Let the position coordinates of the target spacecraft in the geocentric inertial coordinate system (X tar ,Y tar ,Z tar )=P t ;
[0160] The reward function is as follows:
[0161] r=-(||P c -P t ||-P e )×f p
[0162]
[0163] Among them, P e is the expected relative position between the chasing spacecraft and the target spacecraft, f p is the adjustment coefficient of the reward function, which is set to 0.01 in the present invention; c is a constant term; r is the reward value;
[0164] The reward function is mainly calculated at each step during the network simulation process. Initialization is to give the agent an initial value so that it has an initial state in the subsequent simulation process. Initialization does not participate in the reward function calculation. The calculation starts after the first action is executed in the initialization state.
[0165] In the present invention, the chasing spacecraft needs to simultaneously complete the tasks of chasing and keeping the relative distance unchanged, so the expected distance of the chasing spacecraft relative to the target spacecraft is set to 300.
[0166] In order to encourage the algorithm to improve the tracking time, a constant reward term c is added. If the deviation between the relative distance between the two and the expected distance is less than 500, a constant term is added to the reward, otherwise no constant term is added. In the present invention, the reward constant term is set to 1.
[0167] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.
[0168] Specific implementation method 7: This implementation method is different from any one of the specific implementation methods 1 to 6 in that: in step 34, the hyper parameters of the experience replay pool, the critic network and the actor network are set; the specific process is:
[0169] The total number of training episodes is 2000, the maximum step length of each episode is 1000 steps, the random perturbation of the action conforms to the Gaussian distribution, the initial standard deviation is 0.7, the final standard deviation is 0.1, the mean is 0, and the reward decay coefficient is 0.95;
[0170] The experience replay pool initially collects 10,000 sets of samples and then starts training. The total capacity is 50,000. The experience replay pool adopts a first-in, first-out strategy. t ,a t ,r t ,s t+1 ,D) as a set of samples;
[0171] Among them, D is the marker of whether this scene is terminated;
[0172] In the present invention, if the deviation between the relative distance between the chasing spacecraft and the target spacecraft and the expected distance exceeds 500 for 10 consecutive times, it is judged that the current scene has failed and the next scene will be restarted. If there is a step in the middle where the deviation is less than 500, the counting will be restarted.
[0173] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0174] Specific implementation eight: This implementation differs from any one of the specific implementations one to seven in that: the action selection strategy in step 35; the specific process is:
[0175] The action selection strategy is divided into two cases;
[0176] When the number of samples in the experience replay pool is less than or equal to 10, the random action generation network outputs the action (randomly selecting actions from the action space), and the random action generation network parameters are shared with the actor network parameters;
[0177] The random action generation network has the same structure as the actor network. At the beginning of each scene, the parameters are copied from the actor network. After more than 10 experiences, the random action generation network will no longer be used until the next scene begins.
[0178] When the number of samples in the experience replay pool is greater than 10, the action output by the actor network is used, and the input of the actor network is a continuous state sequence;
[0179] At the beginning of retraining and at the beginning of each scene, there will be no continuous sequence state of length n, so there is also a random action generation network in the present invention, which will randomly select actions from the same action space, and the network parameters will be shared with the actor network, and the input dimension will be changed through a fully connected layer so as to adapt to the actor network parameters;
[0180] In the experience collection phase, when the total amount of data in the experience playback pool is less than n or the total number of steps per scene is less than n, the random action generation network will select an action from the action space for the chasing spacecraft to execute based on the initialized network parameters. In the training phase, when the total number of steps in the first scene is less than n, the random action generation network will still be used to generate random actions. When the first scene ends, the actor network parameters will be synchronously updated to the random action generation network, so that the random action generation network will generate more reliable action outputs in the next scene. Different network parameter dimensions will change the input dimension through the fully connected layer.
[0181] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.
[0182] Specific implementation nine: This implementation differs from any one of specific implementations one to eight in that: in step 37, the critic network is used to predict the action value of the current time step (the critic network input is the state and action of the current time step, and the critic network output is the predicted action value of the current time step);
[0183] Update the actor network parameters and the target actor network parameters;
[0184] The specific process is:
[0185] As described in step 24, the critic network is used to analyze the current state-action pair (s t ,a t ) is evaluated, and the evaluation value is negative. By comparing the obtained evaluation values, the evaluation values are assigned a weight of 1 according to their size, and this is used as the loss of the actor network (the expression of the loss function of the actor network is: The actor network is updated via gradient descent.
[0186] Based on the loss function of the actor network, the back-propagation algorithm is used to update the actor network parameters to minimize the loss function;
[0187] For the parameter update of the target actor network, a new parameter update strategy will be used. The specific update method is shown in the following formula:
[0188] P μ' =(1-τ)×μ+τ×μ'
[0189] Among them, P μ' is the network parameter of the target actor network, τ is the parameter ratio of the definition actor network and the target actor network, μ is the actor network, and μ' is the target actor network;
[0190] In the present invention, τ is not a fixed value, because in the early stage of training, the total number of steps is small, the network is extremely unstable, and the target actor network needs to be updated in time to better calculate the target value. Therefore, in the present invention, τ is defined as follows:
[0191]
[0192] Where T is the total number of steps;
[0193] That is, τ will be very small in the early stage, and the target actor network will be basically the same as the actor network. As the total number of steps T continues to increase, τ will gradually increase, and the maximum value will not exceed 0.5.
[0194] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0195] Specific implementation method 10: This implementation method is different from the specific implementation methods 1 to 9 in that: in step 38, the actor network is used to predict the action of the current time step (the actor network input is the state of the current time step, and the actor network output is the predicted action of the current time step), and the critic network is used to estimate the action value of the next time step. Finally, according to the temporal difference error (L t =y t -q t ,y t =r t +γ(1-done)Q'(s t+1 ,a t+1 |θ Q' )), update the critic network parameters and the target critic network parameters;
[0196] The specific process is:
[0197] Based on the loss function of the critic network, the back propagation algorithm is used to update the critic network parameters to minimize the loss function;
[0198] For the parameter update of the target critic network, a new parameter update strategy will be used. The specific update method is shown in the following formula:
[0199] P Q' =(1-τ′)×Q+τ′×Q'
[0200] Among them, PQ' is the network parameter of the target critic network, τ′ is the parameter ratio of the critic network and the target critic network, Q is the critic network, and Q′ is the target critic network;
[0201] In the present invention, τ′ is not a constant value, because in the early stage of training, the total number of steps is small, the network is extremely unstable, and the target critic network needs to be updated in time to better calculate the target value. Therefore, in the present invention, τ′ is defined as follows:
[0202]
[0203] Where T is the total number of steps;
[0204] That is, τ′ will be very small in the early stage, and the target critic network will be basically consistent with the critic network. As the total number of steps T continues to increase, τ′ will gradually increase, and the maximum value will not exceed 0.5;
[0205] As described in step 26, the input dimension is first changed through the critic network, reducing the input dimension from B×4×n to B×4, and concatenated with the input batch action (dimension is B×2) to become B×6 input, and then reduced to B×1 dimension through the fully connected layer, and the target value is calculated through the target actor network and the target critic network, and compared with the current state action pair (s t ,a t )’s evaluation values are compared, the loss value of the critic network is calculated by the same weighted average method, and the critic network parameters are updated by the back propagation algorithm to minimize the loss function;
[0206] Based on the loss function of the critic network, the back propagation algorithm is used to update the critic network parameters to minimize the loss function;
[0207] The target critic network parameter update is consistent with step 37.
[0208] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.
[0209] Evaluate the performance of the deep reinforcement learning active tracker; this includes the following steps:
[0210] 1. In this invention, the average screen tracking length is first used as a main performance indicator of the deep reinforcement learning active tracker, and the obtained network model is applied in the test process. During the test, the flight trajectory of the target spacecraft will be changed to verify the robustness of the obtained network model;
[0211] The present invention is divided into a training phase and a testing phase. The training will be suspended every 20 scenes, and the non-cooperative spacecraft chasing and maintaining relative distance stability tasks in different environments will be tested according to the existing network parameters. Different scenarios are used to test whether the obtained strategy network is only effective for a single scenario and lacks robustness.
[0212] The average tracking length will be tested in 10 scenes in various scenarios, and the tracking steps of the 10 scenes in various scenarios will be averaged to obtain the average tracking length. The average tracking length will evaluate whether the obtained strategy can effectively keep the target within a specific range. Although it may not be guaranteed to be at the expected distance, it can still be kept within an acceptable distance.
[0213] 2. The present invention also uses the average screen reward as part of the evaluation index. If the average screen tracking length is high but the average screen reward is low, it means that the obtained network model can only keep tracking but cannot keep the relative position close to the expected relative position;
[0214] The average episode reward is basically the same as 1, but the average of all rewards obtained in each episode in each scene is taken to verify whether the obtained strategy can ensure that the tracking distance is as close as possible to the expected distance while ensuring that the tracking distance is within an acceptable range. The higher the average episode reward, the higher the tracking accuracy.
[0215] 3. The present invention also takes the speed of reaching the expected relative position as part of the evaluation index. Compared with the traditional PID control algorithm, if the expected position can be reached in a shorter number of steps, it means that the proposed deep reinforcement learning active tracker will be able to reach the expected relative position while consuming less energy;
[0216] Since the target is a spacecraft, the most ideal situation is to consume the least resources to reach the designated position in space. Therefore, a threshold will be set. When the error between the relative distance and the expected distance is less than a certain range, it is considered that the chasing spacecraft has reached the designated position. After that, the size of the output action will be further monitored. The smaller the action, the less energy is used. This evaluation standard can be used as a supplementary evaluation standard.
[0217] The following examples are used to verify the beneficial effects of the present invention:
[0218] Embodiment 1:
[0219] Combination Figure 1This embodiment is described. In this embodiment, an end-to-end non-cooperative spacecraft active tracking algorithm based on robust deep reinforcement learning is involved in this embodiment. The collected training samples are preprocessed, and feature extraction and function fitting of the input samples are realized through the Transformer encoder and the fully connected layer. The temporal motion information of the target spacecraft is estimated, and an approximate optimal tracking strategy is given through a deep deterministic policy gradient algorithm to achieve the pursuit of the non-cooperative target spacecraft and keep the relative distance unchanged.
[0220] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. A non-cooperative spacecraft active tracking method based on deep reinforcement learning, characterized by: The specific process of the method is: Step 1: Based on the basic parameters of the target spacecraft and the chasing spacecraft, the coordinate systems of the target spacecraft and the chasing spacecraft, and the universal gravitational force exerted on the target spacecraft and the chasing spacecraft, the universal gravitational force exerted on the target spacecraft is converted into the position and velocity of the target spacecraft, and the universal gravitational force exerted on the chasing spacecraft is converted into the position and velocity of the chasing spacecraft; Step 2: Set the state space and action space, build the Transformer-based actor network and critic network, as well as the loss function of the actor network and the loss function of the critic network; Step 3: Construct a reward function and obtain the trained actor network and critic network; Step 4: Obtain the difference between the position of the target spacecraft and the chasing spacecraft and the expected position, as well as the speed difference between the target spacecraft and the chasing spacecraft, and input them into the trained actor network, which outputs the action of the current time step.
2. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 1, characterized in that: In step 1, based on the basic parameters of the target spacecraft and the chasing spacecraft, the coordinate systems of the target spacecraft and the chasing spacecraft, and the universal gravitational force exerted on the target spacecraft and the chasing spacecraft, the universal gravitational force exerted on the target spacecraft is converted into the position and velocity of the target spacecraft, and the universal gravitational force exerted on the chasing spacecraft is converted into the position and velocity of the chasing spacecraft; The specific process is: Step 11: Define the basic parameters of the target spacecraft and the chasing spacecraft. The specific process is as follows: The basic parameters of the target spacecraft are: the target spacecraft's own mass, the target spacecraft's maximum thrust, and the target spacecraft's adjustment frequency; The basic parameters of the chasing spacecraft are: the chasing spacecraft's own mass, the chasing spacecraft's maximum thrust, and the chasing spacecraft's adjustment frequency; Step 12: Set the coordinate systems of the target spacecraft and the chasing spacecraft. The specific process is as follows: The coordinate system of the target spacecraft and the chasing spacecraft uses the geocentric inertial coordinate system, and only considers the two-dimensional motion of the target spacecraft and the chasing spacecraft, without considering the position transformation, velocity transformation and thrust output on the z-axis. The origin of the geocentric inertial coordinate system is the center of the earth. Step 13: Define the gravitational force on the target spacecraft and the chasing spacecraft. The specific process is as follows: The gravitational force F on the target spacecraft g,tar and the gravitational force F on the chasing spacecraft g,c As shown below: r c =r tar +c,c∈[-150,150] in, r tar is the initial orbit radius of the target spacecraft, r c To chase the orbital radius of the spacecraft; G is the gravitational constant, which is 6.67×10 -11 N·m 2 / kg 2 ; M is the mass of the Earth, which is 5.977×10 24 kg; m tar is the target spacecraft's own weight, m c To pursue the dead weight of the spacecraft; c is a random variable; Step 14, converting the gravitational force on the target spacecraft into the position and velocity of the target spacecraft; Convert the gravitational force on the chasing spacecraft into the position and velocity of the chasing spacecraft; The specific process is: Step 141, convert the gravitational force on the target spacecraft into the position and velocity of the target spacecraft; the specific conversion is shown in the following formula: Where · represents matrix dot product, P t,tar represents the position of the target spacecraft in the geocentric inertial coordinate system at time t, V t,tar represents the velocity of the target spacecraft in the geocentric inertial coordinate system at time t, P t+1,tar represents the position of the target spacecraft in the geocentric inertial coordinate system at time t+1, V t+1,tar represents the speed of the target spacecraft in the geocentric inertial coordinate system at time t+1; Δt is the adjustment frequency; A is the spacecraft information conversion matrix at time t; Represents the unit vector of the target spacecraft at its current position. The specific form is shown in the following formula: Among them, r 2,tar represents the vector position of the target spacecraft in the geocentric inertial coordinate system, and r1 is the vector position of the center of the earth in the geocentric inertial coordinate system; Step 142, convert the gravitational force on the chasing spacecraft into the position and velocity of the chasing spacecraft; the specific conversion is shown in the following formula: Among them, F RL represents the action output by the actor network, P t,c represents the position of the chasing spacecraft in the geocentric inertial coordinate system at time t, V t,c represents the speed of the chasing spacecraft in the geocentric inertial coordinate system at time t, P t+1,c represents the position of the chasing spacecraft in the geocentric inertial coordinate system at time t+1, V t+1,c represents the speed of the chasing spacecraft in the geocentric inertial coordinate system at time t+1; Δt is the adjustment frequency; A is the spacecraft information conversion matrix at time t; is the unit vector of the chasing spacecraft at its current position; the specific form is shown in the following formula: where r 2,c represents the vector position of the chasing spacecraft in the geocentric inertial coordinate system, and r1 is the vector position of the center of the earth in the geocentric inertial coordinate system.
3. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 2, characterized in that: The specific form of the spacecraft information transposition matrix A at time t is as follows:
4. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 3 is characterized in that: In the step 2, the state space and the action space are set, and the actor network and the critic network based on Transformer are constructed, as well as the loss function of the actor network and the loss function of the critic network; the following steps are included: Step 21, setting the state space; The state space contains the difference between the position of the target spacecraft and the chasing spacecraft and the expected position, as well as the velocity difference between the target spacecraft and the chasing spacecraft; The state space is a continuous state space; Step 22: Set the action space; The action space is defined as a two-dimensional vector; The action space is a continuous action space: a = [a x ,a y ],a x ∈[-4,4],a y ∈[-4,4]; Among them, a is the action, a x is the motion applied to the chasing spacecraft on the x-axis of the geocentric inertial coordinate system, a y is the action applied to the chasing spacecraft on the y-axis of the geocentric inertial coordinate system; Step 23: Construct actor network and critic network; The actor network includes: Embedding layer, Positional Encoding layer, first encoding layer, second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first Dimension Reduce layer, fourth FC layer, second Dimension Reduce layer, fifth FC layer, output layer; The FC layer is a fully connected layer; The specific processing process of the actor network is: The current state data is sequentially input into the Embedding layer, Positional Encoding layer, first encoding layer, second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first DimensionReduce layer, fourth FC layer, second DimensionReduce layer, fifth FC layer, and output layer. The output layer outputs the action under the current state. The critic network includes: Embedding layer, Positional Encoding layer, first encoding layer, second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first DimensionReduce layer, fourth FC layer, second DimensionReduce layer, Concatenate layer, fifth FC layer, output layer; The specific processing process of the critic network is: The current state data is sequentially input into the Embedding layer, Positional Encoding layer, first coding layer, second coding layer, third coding layer, first FC layer, second FC layer, third FC layer, first DimensionReduce layer, fourth FC layer, and second DimensionReduce layer. The feature data output by the second DimensionReduce layer and the current action corresponding to the current state are input into the Concatenate layer and then sequentially input into the fifth FC layer and the output layer. The output layer outputs the current state and the evaluation value data of the current action. Step 24: construct a target actor network and a target critic network; The target actor network includes: Embedding layer, Positional Encoding layer, first encoding layer, second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first DimensionReduce layer, fourth FC layer, second DimensionReduce layer, fifth FC layer, output layer; The specific processing process of the target actor network is: The next moment state s t+1 The data is sequentially input into the Embedding layer, Positional Encoding layer, first encoding layer, second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first DimensionReduce layer, fourth FC layer, second DimensionReduce layer, fifth FC layer, and output layer. The output layer outputs the next state s t+1 Next action a t+1 data; The target critic network includes: Embedding layer, Positional Encoding layer, first encoding layer, second encoding layer, third encoding layer, first FC layer, second FC layer, third FC layer, first DimensionReduce layer, fourth FC layer, second DimensionReduce layer, Concatenate layer, fifth FC layer, output layer; The specific processing process of the target critic network is: The next moment state s t+1 The data is sequentially input into the Embedding layer, Positional Encoding layer, first coding layer, second coding layer, third coding layer, first FC layer, second FC layer, third FC layer, first DimensionReduce layer, fourth FC layer, second DimensionReduce layer. The second Dimension Reduce layer outputs the feature data and the next moment action corresponding to the next moment state, which is then input into the Concatenate layer and then into the fifth FC layer and output layer. The output layer outputs the state-action pair (s t+1 ,a t+1 )’s target value data; Step 25: Set the loss function of the actor network; the specific process is: The expression of the loss function of the actor network is: Among them, J μ is the loss function of the actor network, Q is the critic network, μ is the actor network, θ is the network parameter, and θ μ are the parameters of the actor network, θ Q is the parameter of the critic network, ω t is the weight, t is the time step, B is the batch size, s t is the current state; The loss function of the actor network is obtained as follows: The dimension of the Q value obtained by the critic network is B×1. The cumulative expected return is obtained based on the Q value obtained by the critic network, and the cumulative expected returns are sorted, that is: q qt =-Q(s t ,μ(s t |θ μ )|θ Q ) q q1 <q q2 <...<q q64 Among them, q qt To accumulate expected returns; Different weights are assigned to Q values according to the order of cumulative expected returns from small to large: oh q1 >oh q2 >...>oh q64 Among them, ω qt is the weight value corresponding to the Q value; Based on q qt and ω qt Construct the loss function for the actor network: Step 26: Set the loss function of the critic network; the specific process is: The loss function of the critic network is expressed as: Among them J Q is the loss function of the critic network; r t is the reward value for executing the current action in the current state; done is to determine whether the current state is the end state; Q' is the target critic network; μ' is the target actor network; γ is the decay function; s t+1 is the state at time t+1; a t is the action at the current moment; θ μ' are the parameters of the target actor network; θ Q' are the parameters of the target critic network; The loss function of the critic network is obtained as follows: First, the critic network is used to calculate the current state-action pair (s t ,a t )’s evaluation value: q t =Q(s t ,a t |θ Q ) Among them, q t is the evaluation value at the current moment, Q is the critic network, s t is the current state, a t is the action at the current moment, θ Q are the parameters of the critic network; Through the target actor network, get the action at the next moment: a t+1 =μ'(s t+1 |θ μ' ) Among them, a t+1 is the action at the next moment predicted by the target actor network, μ' is the target actor network, s t+1 is the state at the next moment, θ μ' are the parameters of the target actor network; Through the target critic network, the state-action pair (s) is calculated by combining the next moment action, the next moment state, the reward obtained at the current step, and whether it is the terminal state. t+1 ,a t+1 ) target value: y t =r t +γ(1-done)Q'(s t+1 ,a t+1 |θ Q ') In the formula, y t is a state-action pair (s t+1 ,a t+1 ) target value, r t is the current state t Execute action a t The reward value of Q' is the target critic network, θ Q ' are the parameters of the target critic network; After calculating the target value and the evaluation value, subtract the target value from the evaluation value: L t =y t -q t The obtained L t The dimension is B×1, and the loss function of the critic network is finally obtained:
5. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 4, characterized in that: In the step 3, a reward function is constructed to obtain a trained actor network, a critic network, a target actor network, and a target critic network; The following steps are involved: Step 31: Initialize the initial positions of the target spacecraft and the chasing spacecraft; the specific process is as follows: The target spacecraft’s initial position is in an orbit 7500 km from the Earth’s center; The position of the chasing spacecraft is on and inside a circle with a radius of 1 km and the target spacecraft as the center; Step 32: Initialize the initial speeds of the target spacecraft and the chasing spacecraft. The specific process is as follows: The initial velocity of the target spacecraft is set according to the following formula: Among them, v tar is the initial velocity of the target spacecraft at the initial position when it is subject to gravity; r tar is the initial orbit radius of the target spacecraft, F g,tar is the gravitational force on the target spacecraft, m tar is the target spacecraft’s own weight, X tar ,Y tar ,Z tar is the position coordinate of the target spacecraft in the geocentric inertial coordinate system; is the velocity unit vector of the target spacecraft at its current position, · is the dot product, is the velocity of the target spacecraft, v x,tar is the velocity of the target spacecraft on the x-axis of the geocentric inertial coordinate system, v x,tar To take a negative value, v y,tar is the velocity of the target spacecraft on the y-axis of the geocentric inertial coordinate system, v z,tar is the speed of the target spacecraft on the z-axis of the geocentric inertial coordinate system; The initial velocity of the chasing spacecraft is set according to the following formula: Among them, v c is the initial velocity of the chasing spacecraft at the initial position under the gravitational force; r c is the orbit radius of the chasing spacecraft, F g,c To chase the gravitational force on the spacecraft, m c To pursue the spacecraft's own weight, X c ,Y c ,Z c The position coordinates of the chasing spacecraft in the geocentric inertial coordinate system; is the velocity unit vector of the chasing spacecraft at its current position, · is the dot product, is the speed of the chasing spacecraft, v x,c is the speed of the chasing spacecraft on the x-axis of the geocentric inertial coordinate system, v x,c Take a negative value, v y,c is the speed of the chasing spacecraft on the y-axis of the geocentric inertial coordinate system, v z,c The speed of the chasing spacecraft on the z-axis in the geocentric inertial coordinate system; Step 33: Define the reward function. Step 34: Set the experience replay pool parameters, the hyperparameters of the critic network and the actor network; Step 35: Action selection strategy; Step 36: extract n consecutive training samples from the experience replay pool as a group of training samples; Step 37: Use the critic network to predict the action value of the current time step; Update the actor network parameters and the target actor network parameters; Step 38: Use the actor network to predict the action of the current time step, and use the critic network to estimate the action value of the next time step. Finally, update the critic network parameters and the target critic network parameters according to the temporal difference error. Step 39: Repeat steps 33 to 38 until convergence to obtain the trained actor network and critic network.
6. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 5, characterized in that: The reward function is defined in step 33; the specific process is: Let the position coordinates of the chasing spacecraft in the geocentric inertial coordinate system (X c ,Y c ,Z c )=P c ; Let the position coordinates of the target spacecraft in the geocentric inertial coordinate system (X tar ,Y tar ,Z tar )=P t ; The reward function is as follows: r=-(||P c -P t ||-P e )×f p Among them, P e is the expected relative position between the chasing spacecraft and the target spacecraft, f p is the adjustment coefficient of the reward function; c is the constant term; r is the reward value.
7. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 6, characterized in that: In step 34, the experience replay pool parameters, the hyper parameters of the critic network and the actor network are set; the specific process is: The total number of training episodes is 2000, the maximum step length of each episode is 1000 steps, the random perturbation of the action conforms to the Gaussian distribution, the initial standard deviation is 0.7, the final standard deviation is 0.1, the mean is 0, and the reward decay coefficient is 0.95; The experience replay pool initially collects 10,000 sets of samples and then starts training. The total capacity is 50,000. The experience replay pool adopts a first-in, first-out strategy. t ,a t ,r t ,s t+1 ,D) as a set of samples; Among them, D is a marker to indicate whether this episode is terminated.
8. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 7, characterized in that: The action selection strategy in step 35 is as follows: The action selection strategy is divided into two cases; When the number of samples in the experience replay pool is less than or equal to 10, the random action generation network outputs the action, and the random action generation network parameters are shared with the actor network parameters; The random action generation network has the same structure as the actor network. At the beginning of each scene, the parameters are copied from the actor network. After more than 10 experiences, the random action generation network will no longer be used until the next scene begins. When the number of samples in the experience replay pool is greater than 10, the action output by the actor network is used, and the input of the actor network is a continuous state sequence.
9. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 8, characterized in that: In step 37, the critic network is used to predict the action value of the current time step; Update the actor network parameters and the target actor network parameters; The specific process is: Based on the loss function of the actor network, the back-propagation algorithm is used to update the actor network parameters to minimize the loss function; For the parameter update of the target actor network, the specific update method is as follows: P μ' =(1-τ)×μ+τ×μ' Among them, P μ' is the network parameter of the target actor network, τ is the parameter ratio of the actor network to the target actor network, μ is the actor network, and μ' is the target actor network; The definition of τ is as follows: Among them, T is the total number of steps.
10. The non-cooperative spacecraft active tracking method based on deep reinforcement learning according to claim 9, characterized in that: In step 38, the actor network is used to predict the action of the current time step, and the critic network is used to estimate the action value of the next time step. Finally, the critic network parameters and the target critic network parameters are updated according to the temporal difference error. The specific process is: Based on the loss function of the critic network, the back propagation algorithm is used to update the critic network parameters to minimize the loss function; For the parameter update of the target critic network, the specific update method is as follows: P Q' =(1-τ′)×Q+τ′×Q' Among them, P Q' is the network parameter of the target critic network, τ′ is the parameter ratio of the critic network and the target critic network, Q is the critic network, and Q′ is the target critic network; The definition of τ′ is as follows: Among them, T is the total number of steps.
Citation Information
Patent Citations
Autonomous avoidance maneuvering method for multiple interceptors by spacecraft based on reinforcement learning
CN112001120A
Spacecraft multi-space debris collision avoidance autonomous decision-making method based on near-end strategy optimization
CN116125811A
Asteroid flexible detector intelligent control method based on deep reinforcement learning SAC algorithm
CN116400589A
Pulse type track pursuit barrier cooperative game intelligent decision control method
CN116991067A
Multi-spacecraft hunting and pursuit game decision-making method based on group independent learning strategy
CN118052290A
Cited By
Structural damage detection method based on theory-assisted multi-agent
CN119862733A
A structural damage detection method based on theory-assisted multi-agent
CN119862733B
Reinforced learning satellite pursuit method and system based on orbit action pre-judgment
CN121525779A
Satellite pursuit and evasion method and system based on track action prediction
CN121525779B
