Interplanetary orbit transfer method based on hidden state and reinforcement learning

Abstract

The invention discloses an interplanetary orbit transfer method based on a hidden state and reinforcement learning, and relates to the technical field of spacecraft control. According to the method, by establishing the sequence hidden variable model, representation learning can be explicitly carried out on the uncertain environment, and hidden information hidden under observation of the uncertain environment is extracted. And a reinforcement learning controller is formed through the actor network and the commentator network. A sequence hidden variable model and a reinforcement learning controller are adopted to construct a reinforcement learning algorithm framework, so that the training of an intelligent agent in an uncertain environment is accelerated, and the processing capability of the intelligent agent on uncertainty is improved. Besides, according to the embodiment of the invention, the next state is predicted based on the current observation and expected operation, and the quality of the predicted next state is brought into the reward structure, so that the real-time reward can capture the validity of the current and subsequent strategies at the same time, and the learning efficiency of the algorithm is further improved.

Description

Technical Field

[0001] The present invention relates to the field of spacecraft control technology, and in particular to an interplanetary orbit transfer method based on hidden state and reinforcement learning. Background Art

[0002] In recent years, small and micro spacecraft equipped with low-thrust electric propulsion have gradually become a research hotspot for deep space exploration due to their low fuel consumption, high mission flexibility, and strong scalability. The design of low-thrust interplanetary orbit transfer is one of the basic guarantees for the success of small and micro spacecraft in deep space exploration missions. The low-thrust interplanetary orbit transfer design aims to use the low-thrust propulsion system for long-term continuous propulsion to achieve efficient orbit transfer.

[0003] In practical applications, due to the limited onboard resources of small and micro spacecraft, there is a high requirement for lightweight computing. At the same time, spacecraft will be affected by various external interferences and internal uncertainties in interplanetary space, which will affect the accuracy and reliability of orbit transfer.

[0004] In view of the space environment with multi-source uncertainty, a low-thrust interplanetary trajectory robust design method based on meta-reinforcement learning is proposed. The adaptability and robustness to complex environments and uncertainties are achieved through continuous interaction between the agent and the environment, and good real-time performance is achieved with the help of the pre-trained characteristics of artificial intelligence methods. However, the agent does not understand uncertainty by relying solely on the characteristics of continuous interaction between reinforcement learning and the environment. In addition, the key performance indicators of the traditional method include terminal fuel consumption and errors in terminal position and velocity, which means that its effect evaluation is only performed at the end of the mission, resulting in an extremely sparse reward structure.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method for interplanetary orbit transfer based on hidden state and reinforcement learning is provided, aiming to solve the problem that the robust design method of interplanetary trajectory based on reinforcement learning in the prior art only relies on the characteristics of continuous interaction between reinforcement learning and the environment for learning, the intelligent agent does not understand uncertainty, and the reward structure is sparse, which leads to the problem of difficulty in learning of the intelligent agent.

[0007] The technical solution adopted by the present invention to solve the problem is as follows: An embodiment of the present invention provides an interplanetary orbit transfer method based on hidden state and reinforcement learning, the method comprising: Acquiring observation data of the spacecraft; wherein the observation data is determined based on state data and observation uncertainty; and the state data is calculated based on a state transfer matrix constructed based on a preset orbit transfer mission; Inputting the observed data into a preset sequence latent variable model to obtain latent variables; Inputting the hidden variables and the historical trajectory records of the spacecraft into the reinforcement learning controller to obtain the execution action of the spacecraft; Predicting next state data of the spacecraft according to the observation data and the execution action, calculating a reward value according to the state data and the next state data, and causing the reinforcement learning controller to update the orbit transfer strategy based on the reward value; Wherein, the reinforcement learning controller comprises: A critic network, used for estimating a state value according to an input latent variable and a self-value estimation function, and updating the track transfer strategy according to the state value; The actor network is used to calculate the execution action of the spacecraft according to the historical trajectory record and the orbit transfer strategy.

[0008] Beneficial effects of the present invention: The embodiments of the present invention can explicitly perform representation learning on uncertain environments and extract hidden information hidden under observations of uncertain environments by establishing a sequential latent variable model. A reinforcement learning controller is formed through an actor network and a critic network. The sequential latent variable model and the reinforcement learning controller are used to construct a reinforcement learning algorithm framework, thereby accelerating the training of the agent in an uncertain environment and improving the agent's ability to handle uncertainty. In addition, the embodiments of the present invention will also predict the next state based on the current observation and expected operation, and incorporate the quality of the predicted next state into the reward structure, so that the immediate reward can simultaneously capture the effectiveness of the current and subsequent strategies, thereby improving the learning efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 It is a flowchart of an interplanetary orbit transfer method based on hidden state and reinforcement learning provided by an embodiment of the present invention.

[0011] Figure 2 It is a structural diagram of the sequence latent variable model provided by an embodiment of the present invention.

[0012] Figure 3 It is a schematic diagram of a framework for random hidden state proximal strategy optimization provided by an embodiment of the present invention.

[0013] Figure 4It is a comparison diagram of the reward convergence effect before and after the automatic weight adjustment provided by the embodiment of the present invention.

[0014] Figure 5 It is a distribution diagram of the size of the velocity increment applied by the pulse thrust provided by an embodiment of the present invention.

[0015] Figure 6 This is a diagram of orbital transfer results generated by Monte Carlo simulation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention discloses an interplanetary orbit transfer method based on hidden state and reinforcement learning. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0018] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0019] In view of the above-mentioned defects of the prior art, the present invention provides an interplanetary orbit transfer method based on hidden state and reinforcement learning, such as Figure 1 As shown, the method specifically comprises the following steps: Step S100, obtaining observation data of the spacecraft; wherein the observation data is determined based on state data and observation uncertainty; the state data is calculated based on a state transfer matrix constructed based on a preset orbit transfer mission; Step S200, inputting the observed data into a preset sequence latent variable model to obtain latent variables; Step S300, inputting the hidden variables and the historical trajectory records of the spacecraft into the reinforcement learning controller to obtain the execution action of the spacecraft; Step S400, predicting the next state data of the spacecraft according to the observation data and the execution action, calculating a reward value according to the state data and the next state data, and enabling the reinforcement learning controller to update the orbit transfer strategy based on the reward value; Wherein, the reinforcement learning controller comprises: A critic network, used for estimating a state value according to an input latent variable and a self-value estimation function, and updating the track transfer strategy according to the state value; The actor network is used to calculate the execution action of the spacecraft according to the historical trajectory record and the orbit transfer strategy.

[0020] In general, considering that the environment has various uncertainties, the system cannot directly obtain the state and observation errors caused by these uncertainties. This embodiment regards the uncertain environment as a partially observable Markov decision process, and proposes a reinforcement learning framework (which can be called a random hidden state proximal policy optimization algorithm framework) that can handle uncertainty and partial observability by integrating the sequence latent variable model with the proximal policy optimization algorithm. The algorithm flow of this embodiment is mainly divided into three steps. The first step is problem modeling, that is, constructing a state transfer matrix based on the preset trajectory transfer task, and constructing an expression for the observed data. The second step is to construct a reinforcement learning algorithm framework that handles uncertainty and partial observability, that is, first using a sequence latent variable model (such as Figure 2 (as shown) performs representation learning on the track transfer model containing uncertainty, autonomously extracts hidden state information that can characterize the system state and predicts the future state of the system, and improves the ability to handle uncertainty. Secondly, a random hidden state proximal policy optimization reinforcement learning framework is established by combining the sequence latent variable model and the proximal policy optimization technology. The learned hidden variable information is used instead of the uncertain state information to design the track transfer strategy, reducing the impact of system uncertainty on the overall control. The third step is reward design. The sequence latent variable model in this embodiment can not only promote the extraction of hidden information in the uncertain environment, but also predict the next state based on the current observations and expected operations. Incorporating data related to the predicted future state into the calculation of the reward value at each step can alleviate the sparsity of the reward structure, thereby improving the future expected effect of the algorithm, data efficiency, and model learning speed.

[0021] Specifically, the reinforcement learning algorithm framework is as follows Figure 3As shown, it includes a sequence latent variable model and a reinforcement learning controller composed of an actor network and a critic network. That is, the sequence latent variable model and the reinforcement learning controller together form an intelligent agent. The sequence latent variable model explicitly represents the learning of the uncertain environment and extracts the hidden information hidden under the observation of the uncertain environment, thereby accelerating the training of the intelligent agent in the uncertain environment and improving the ability of the intelligent agent to handle uncertainty. In this embodiment, the latent variables learned by the sequence latent variable model are used as the input of the critic network, and the current and historical environment trajectories are used as the input of the actor network, so as to solve the problem of insufficient decision information caused by uncertainty. In addition, unlike the traditional reward design, which is designed only according to the current state of the environment and the actions taken, it only reflects the quality of the current action. In the training process of this embodiment, the sequence latent variable model and the current strategy are comprehensively considered and used to predict the next state, and the quality of the predicted state is quantified and incorporated into the reward structure, so that the immediate reward can capture the effectiveness of the current and subsequent strategies at the same time, thereby improving the learning efficiency of the algorithm.

[0022] In one implementation, the state transfer matrix is: , formula (1); in, , They represent the position of the spacecraft at the kth time step and the k+1th time step respectively; , They represent the velocities of the spacecraft at the kth time step and the k+1th time step respectively; , They represent the mass of the spacecraft at the kth time step and the k+1th time step respectively; and represents the Lagrangian coefficient of the kth time step; and represents the derivative of the Lagrange coefficient at the kth time step; vector is the output of the reinforcement learning controller, representing the speed change obtained by impulse inference ; represents the equivalent exhaust velocity of the propeller; represents the position deviation caused by state uncertainty at the k+1th time step; represents the speed deviation caused by state uncertainty at the k+1th time step; The observation data is in vector form, and the calculation formula of the observation vector is: , formula (2); in, represents the observation vector; Indicates time, , represents the total time step; Represents matrix or vector transpose; represents the position deviation caused by observation uncertainty at the kth time step; represents the velocity deviation caused by observation uncertainty at the kth time step.

[0023] Specifically, the derivation process of the calculation formula of the above state transfer matrix and observation vector is as follows: First, the problem is modeled: In a low-thrust interplanetary rendezvous mission, a spacecraft using a low-thrust electric propulsion device starts from a given initial position, transfers to the target orbit at a specified time through a low-thrust orbital maneuver, and reaches the same speed as the target planet. Assuming that the spacecraft is only affected by the gravity of the sun, the spacecraft model can be approximated as a two-body model. Considering the electric thrust and space perturbations of the spacecraft, its initial kinematic and dynamic models can be described as: , formula (3); , formula (4); , formula (5); in, and They represent the position vector and velocity vector pointing from the center of mass of the central celestial body to the spacecraft respectively; Indicates the distance from the central celestial body to the spacecraft; represents the standard gravitational parameter, and denote the mass of the central celestial body and the mass of the spacecraft, respectively. is the gravitational constant; is the thrust amplitude of the electric propulsion system; is the direction vector of the thrust; is the disturbance acceleration experienced by the spacecraft; is the gravitational acceleration at sea level; is the specific impulse of the electric propulsion system.

[0024] The low-thrust orbit is discretized and approximated as Pulse Due to the maximum thrust limit of the low-thrust thruster, the Pulse of time step Has a maximum magnitude limit.

[0025] Maximum pulse value and The pulse value of the time step The definitions are as follows: , formula (6); , formula (7); In the formula, It represents the maximum thrust amplitude that the spacecraft engine can provide; represents the total transfer time, The spacecraft is at time The quality of time.

[0026] The initial kinematic and dynamic models can be further rewritten as a discrete Markov decision process as follows: , formula (8); , formula (9); in, It is a stochastic discrete dynamic model that describes the transition of environmental states; is a function representing a policy mapping; and Respectively and The system status at the time of the step, which includes the position of the spacecraft ,speed and quality ;vector is the output of the reinforcement learning controller, representing the speed change obtained by the impulse thrust ; represents the observation vector of the spacecraft, which contains All information and current time in , .

[0027] Based on the initial kinematic and dynamic models, the initial state transition matrix of the Markov decision process is established: , formula (10); Actions are defined as: , formula (11); in, and Indicates Lagrangian coefficients for time steps; The formula for the change in mass is given by the Tsiolkovsky equation; is the equivalent exhaust velocity of the propeller.

[0028] In addition, in order to accurately describe the state transition process, the present invention considers two different types of uncertainty: 1. State uncertainty: uncertainty caused by external perturbations and unmodeled dynamics; 2. Observation uncertainty: uncertainty caused by sensor errors and communication failures.

[0029] time (k=0,1,2,…, ) is expressed as additive Gaussian noise on position and velocity. Specifically: , formula (12); , formula (13); in, Indicates state uncertainty The covariance matrix of represents the position deviation caused by state uncertainty at the kth time step, represents the velocity deviation caused by state uncertainty at the kth time step; Represents observation uncertainty The covariance matrix of represents the position deviation caused by observation uncertainty at the kth time step, represents the velocity deviation caused by observation uncertainty at the kth time step; , , , represents the standard deviation of position and velocity in state uncertainty and observation uncertainty; The dimension is The identity matrix of The dimension is The zero matrix of .

[0030] Therefore, the initial state conversion matrix established above, that is, formula (10), can be rewritten as formula (1). And the observation vector It can be expanded into formula (2).

[0031] In one implementation, the sequence latent variable model consists of an action model, a latent variable, an observation model, and latent variable dynamics, wherein the latent variable dynamics is used to establish a connection between the observation model, the action model, and the latent variables; the objective function of the sequence latent variable model is established based on the equivalence principle of maximizing the probability of observed data and maximizing the lower bound of evidence.

[0032] Specifically, Figure 2 As shown, the structure of the sequence latent variable model in this embodiment mainly includes four parts: action, latent variable, observation model and latent variable dynamics. Figure 2 middle is the hidden variable at the kth time step, which summarizes the current and historical information, including position, velocity, mass, time, and the state transition probability to the next state. represents the observation model, which includes the sum of state information and the deviation caused by uncertainty. In order to highlight the ability of the sequence latent variable model to represent an environment with uncertainty, Figure 2 The deviation caused by uncertainty is extracted separately. is the time step The action taken when the latent variable model is established. The latent variable dynamics is to establish the connection between the observation model, the action model and the latent variable. In order to ensure that the established latent variable model accurately represents the environment, the probability of each observation data from the training set must be maximized in the entire generation process. Since the marginalization probability of the latent variable is difficult to calculate directly, this embodiment derives the evidence lower bound of the probability in a manner similar to amortized variational inference. Maximizing the probability of the observed data is equivalent to maximizing the evidence lower bound, and then the objective function of the sequence latent variable model is obtained.

[0033] In one implementation, the latent variable dynamics includes a generative model and an inference model; The generative model is used to sequentially generate the next latent variable and the prior distribution of each latent variable according to the previous latent variable information; The inference model is used to calculate the posterior distribution of the current latent variable based on subsequent latent variable information and observation data.

[0034] Specifically, Figure 2 As shown in Figure 1, the solid arrows represent the generative model and the dotted arrows represent the inference model. The generative model generates the next latent variable in sequence based on the previous information, including the prior distribution of each latent variable. This includes the initial prior distribution and , the prior distribution and ( ), and decoder In the generative model, the initial distribution It is defined as a multivariate standard normal distribution, that is, , while the other distributions are parameterized by In contrast, the inference model uses the subsequent latent variable information and observation information to infer the probability of the current latent variable, that is, the posterior distribution of each latent variable. The inference model includes the initial posterior distribution and , the posterior distribution and To simplify the subsequent steps in the inference model, assume The posterior distribution of is the same as its prior distribution. The posterior distribution of Represented by the parameter Parameterization of neural networks.

[0035] Furthermore, the probability of each observation from the training set is maximized during the generation process. . Derive the evidential lower bound for this probability: , formula (14); in, is the sequence length of a single sequence latent variable model; represents the relative entropy of two distributions.

[0036] Furthermore, maximizing the probability of observed data is equivalent to maximizing the lower bound of evidence, so the objective function of the sequence latent variable model is: , formula (15); in, represents the objective function of the sequence latent variable model; represents the sequence length of a single sequence latent variable model; represents the relative entropy of two distributions; Represents the probability of each observation data; represents the lower bound of evidence; represents the observation vector, is the hidden variable at the kth time step, and They represent the first layer latent variables and the second layer latent variables at the kth time step respectively; represents the execution action at the kth time step; Represents the trajectory of the function obtained by sampling from the probability distribution q expected value.

[0037] In one implementation, the overall objective function of the reinforcement learning controller is composed of the objective formula of the actor network, the mean square value function error term calculated based on the value function, and the entropy term calculated based on the historical trajectory record.

[0038] Specifically, Figure 3 As shown, the latent variable obtained by the sequence latent variable model is used as the input of the evaluation network to estimate the value function of the state, expressed as At the same time, historical observations and decision sequences are directly used as input to the actor network to output actions, expressed as ,in is a hyperparameter that controls the length of the sequence latent variable model. The original formula of the proximal policy optimization algorithm is modified accordingly to obtain the target formula of the actor network. The overall objective function of the reinforcement learning controller (i.e., the proximal policy optimization algorithm part) is then formed by combining the mean square function error term and the entropy term.

[0039] In one implementation, the overall objective function of the reinforcement learning controller is: , formula (16); in, represents the overall objective function of the reinforcement learning controller; Represent the objective formula of the actor network; represents the mean square value function error term; represents the entropy term; Represents the trainable parameters of the neural network corresponding to the reinforcement learning controller; , represents the hyperparameters that control the relative importance of the three objectives; The objective formula of the actor network is: , formula (17); in, Represent the objective formula of the actor network; Indicates that along the strategy The sampled trajectory expectations; represents the total time step; represents the importance weight, represents the old strategy used when sampling data, Indicates the execution of an action. Indicates the system status; Indicates that The value range is limited to The function of Indicates The importance weight of each time step; Represents the hyperparameter that controls the scope of pruning; represents the generalized advantage estimate; The calculation formula of the mean square value function error term is: , formula (18); in, represents the mean square value function error term; represents a latent variable; Indicated by the parameter The self-value estimation function established by the neural network; Reward discount factor Power; Indicates The environment reward for each time step; The calculation formula of the entropy term is: , formula (19); in, represents the entropy term; Indicates time steps and previous The trajectory of the step, .

[0040] In one implementation, the generalized advantage estimate (generalized advantage estimate, GAE) as an advantage function The unbiased estimate of is calculated as follows: , formula (20); in, It is an adjustable hyperparameter in GAE; is the reward discount factor.

[0041] In one implementation, calculating a reward value according to the state data and the next state data includes: Acquiring a reference trajectory of the spacecraft, wherein the reference trajectory is an optimal trajectory obtained based on a deterministic environment; Calculate a reward item according to the next state data and the reference trajectory; Acquire target state data corresponding to the target planet, calculate a final position error and a final velocity error of the spacecraft according to the state data and the target state data, and calculate a penalty term according to the final position error and the final velocity error respectively; Calculate a single-step fuel loss of the spacecraft according to the state data, and calculate a penalty term according to the single-step fuel loss; Calculate a penalty term according to the state data and a preset maximum thrust threshold; Calculating a penalty term based on the state data and a preset forbidden area; Calculate a position deviation and a speed deviation according to the state data and the reference trajectory, and calculate a penalty term according to the position deviation and the speed deviation respectively; The reward value is calculated according to the reward item and all the penalty items.

[0042] Specifically, this embodiment uses a reward function to calculate the reward value. The reward value, as environmental feedback, can be used to update the track transfer strategy, and specifically can be used to update the network parameters of the critic network. The reward function is composed of multiple data items, and the types of data items are divided into reward items and penalty items. The reward function specifically includes the following data items: Reward for predicted next state: The next state is predicted based on the current observation and expected action through the sequence latent variable model, and a reward term is generated from the predicted next state and the reference trajectory to alleviate the sparsity of the reward structure. The value of the reference trajectory is determined based on the trajectory generated by the agent trained in a deterministic environment.

[0043] Penalty terms for the final position error and final velocity error, which represent the difference between the final position and velocity of the spacecraft and the final position and velocity of the target planet. Including the terminal position and velocity errors as penalties in the calculation process of the reward value can effectively reduce the final position error and final velocity error.

[0044] Penalty term for single-step fuel consumption: Small and micro spacecraft used for deep space exploration usually lack the ability to carry a large amount of fuel or refuel during the mission. Therefore, in the design of orbit transfer missions, it is crucial to minimize fuel consumption while ensuring that the terminal error remains as small as possible. To this end, this embodiment introduces a penalty term related to fuel consumption, which penalizes the fuel consumption at each time step, so that the spacecraft can save as much fuel as possible.

[0045] Penalty term for maximum thrust threshold: This term reflects the constraint related to the maximum thrust limit. Spacecraft equipped with low-thrust propulsion systems are subject to maximum thrust limits. If a decision causes the thrust to exceed this maximum limit, it is considered invalid and the decision is executed with the maximum allowed thrust. In order to reduce the occurrence of such events, this embodiment introduces a penalty term for decisions that exceed the maximum thrust threshold.

[0046] Penalty item for forbidden area: The forbidden area refers to a specific area that is prohibited from entry. This item reflects the penalty for entering the specific area. In order to improve the efficiency of spacecraft detection and avoid excessive exploration in invalid areas, this embodiment establishes a forbidden area for the spacecraft that is significantly deviated from the optimal trajectory. A large constant penalty is imposed on the spacecraft entering this area to limit excessive exploration in areas that are significantly deviated from the orbit.

[0047] Penalty terms for position deviation and velocity deviation: This embodiment uses reference trajectories to speed up agent exploration. In order to improve the training speed of the agent in an uncertain environment, this embodiment uses the optimal trajectory derived from a deterministic environment as a reference trajectory. Penalties are generated based on the deviation between the current position of the spacecraft and the reference trajectory to guide the agent's exploration. In practical applications, in order to avoid over-reliance on the reference trajectory leading to suboptimal solutions, the weight assigned to the penalty term established based on the reference trajectory constraint should not be too large.

[0048] In one implementation, calculating the reward value according to the reward item and all the penalty items includes: Calculating the reward value according to a weighted combination of the reward item and all the penalty items; The method for adjusting the weights corresponding to the reward items and the penalty items includes: Establishing a reward function according to the reward term and all the penalty terms, wherein the penalty term established based on the final position error and the final velocity error includes a constraint value that changes with the training process; Using two sets of reward functions with different weights to perform training respectively; Determining the training potentials corresponding to the reward items and the penalty items respectively according to the relative sizes of the reward items and the penalty items in the training results and the difference between the training results of the two groups of reward functions; The weight is adjusted according to the training potential of the reward item and each of the penalty items.

[0049] Specifically, each data item in the reward function in this embodiment is weighted, and the weight requirements of each data item are different to reflect the relative importance of each data item. If the weight setting is not correctly calibrated, it may cause lower-weighted targets to be obscured by higher-weighted targets. This imbalance will slow down the optimization of lower-priority targets and have an adverse impact on overall training efficiency. To solve this problem, this embodiment will Constraints are incorporated into the design of the reward function. Constraints can segment the training process, thereby reducing the sensitivity to the weights of the reward function. Second, given that the reward function has multiple components, The effectiveness of constraints is limited. This embodiment also designs a self-tuning method for reward weights, so as to realize the autonomous design of appropriate reward weights. The way to realize weight self-tuning in this embodiment is to use two sets of different reward functions for training respectively. The weights are adjusted autonomously according to the relative size of each data item in the training results and the training potential of each data item measured by the difference between the two training results, so as to balance the relative importance and optimization potential between each data item.

[0050] In one implementation, the reward function for calculating the reward value is: , formula (21); in, Indicates the reward value; Represents the weight of each term in the reward function; Represents a constraint that reduces the sensitivity of the reward function weights; Indicates the penalty item corresponding to the forbidden area; represents the penalty term corresponding to the maximum thrust threshold; represents the penalty term corresponding to the single-step fuel loss; represents the final position error; represents the final velocity error; Represents the penalty term corresponding to the position deviation; represents the penalty term corresponding to the speed deviation; Indicates the reward item corresponding to the next state data.

[0051] Specifically, predict the next state based on the current observation and expected operation , , , They represent the predicted position, velocity, and mass of the spacecraft in the next state. The reward term related to the predicted state in the reward function is defined as follows: , formula (22); in, is the current residual mass of the spacecraft, and is the next time step The position and velocity of the reference trajectory.

[0052] The specific definitions of the penalty terms for the final position error and final velocity error are as follows: , formula (23); , formula (24); in , represent the terminal position and velocity errors, respectively, which are only defined at the end of the transfer (i.e. when Here and denote the final position and velocity of the spacecraft at the end of the mission, respectively, and and Corresponds to the position and velocity of the target planet at the end of the rendezvous mission. and Represent the current time step and the total time step respectively.

[0053] The specific definition of the penalty term for single-step fuel loss is as follows: , formula (25); in represents the final remaining mass of the spacecraft at the end of its mission; Indicated in The residual mass of the spacecraft at each time step.

[0054] The specific definition of the penalty term of the maximum thrust threshold is as follows: , formula (26).

[0055] The specific definitions of the penalty items for entering the prohibited area are as follows: , formula (27); in, and The upper and lower limits of the prohibited area.

[0056] The specific definitions of the penalty terms for position deviation and velocity deviation established based on the reference trajectory constraint are as follows: , formula (28); , formula (29); in and is the reference trajectory at time step The position and speed at time.

[0057] Therefore, by integrating formulas (22)-(29), we get the complete reward function, namely formula (21).

[0058] In order to facilitate understanding of the technical solution of this embodiment, the overall algorithm flow is described by taking the Earth-Mars orbit transfer task as an example: In the Earth-Mars orbit transfer mission, a spacecraft using a low-thrust electric propulsion device, given an initial position , initial velocity , initial mass Through low-thrust orbital maneuvers, transfer to the target orbit at the specified time , and reach the same speed as the target planet .

[0059] Constructing the kinematic and dynamic models of the spacecraft, we can obtain formula (3)-formula (5), where: represents the standard gravitational parameter, is the specific impulse of the electric propulsion system.

[0060] After discretization of the low thrust orbit, the maximum pulse value and The pulse value of the time step Defined as formula (6) and formula (7) where, =0.5 represents the maximum thrust amplitude that the spacecraft engine can provide, =348.79days represents the total transfer time, Take it as 40.

[0061] Construct the calculation formula of the state transfer matrix and the observation vector, that is, obtain formula (1) and formula (2), where the formula used in the construction process is , , , represents the standard deviation of position and velocity in state uncertainty and observation uncertainty, and their values ​​are respectively =1km, =0.05km / s, =1km, =0.05km / s.

[0062] Through the sequence latent variable model and reinforcement learning controller, a random hidden state proximal policy optimization framework is established, in which the neural network describing the hidden state dynamics is set as follows: the hidden state dimension is set to , , all distributions consist of an input layer of their respective input dimensions, two fully connected hidden layers with 64 nodes, and an output layer of their respective output dimensions. The learning rate of this part is set to the initial value starts to decrease linearly throughout the training process.

[0063] The lower bound of evidence is derived by formula (14). is the sequence length of a single sequence latent variable model, =4.

[0064] Construct the target formula of the actor network, that is, get formula (17), where Represents the hyperparameter that controls the pruning range, with an initial value of =0.3, and decreases linearly as the training process progresses. The generalized advantage estimate (GAE) is calculated by the following formula: : , formula (30); in, is an adjustable hyperparameter in GAE, taking =0.99; is the reward discount factor, =0.9999.

[0065] The calculation formulas for the mean square value function error term and entropy term are constructed, and formulas (18) and (19) are obtained, and then the overall objective function of the reinforcement learning controller is constructed, and formula (16) is obtained. , represents the hyperparameter that controls the relative importance of the three objectives. =0.5, =4.75x10 -8 The settings of the actor network and the critic network are an input layer of the corresponding input dimension, two fully connected hidden layers with 64 nodes, and an output layer of the corresponding dimension. The learning rate is set to the initial value =2.5x10 -4 starts to decrease linearly throughout the training process.

[0066] Construct the reward function, that is, get formula (21), where formula (21) The value is set to: , formula (31); in, Indicates the current training step number, Indicates the total number of training steps, which is set to 10 here 8 .

[0067] Secondly, the penalty term for the forbidden area, that is, formula (27), and For the upper and lower bounds of the prohibited area, =0.8 AU and =1.68 AU .

[0068] In addition, two different reward functions are used for training respectively, and the initial weights are taken and Perform self-tuning of reward weights.

[0069] Finally, the weight adjustment result is obtained . Figure 4 The comparison of training curves using initial weights and adjusted weights is shown. It can be seen that the reward curve obtained using adjusted weights is significantly better than the reward curve generated using initial weights. At step , the agent trained with the adjusted weights has almost converged, while the agent trained with the initial weights still shows considerable fluctuations and has not yet converged. During the training step, both curves seem to have converged. However, the locally enlarged curve shows that the agent using the adjusted weights obtains a higher and more stable reward, while the reward curve using the initial weights still has a small fluctuation. In short, the above experimental data proves that the automatic reward weight adjustment method of this embodiment can effectively make the agent converge quickly and obtain better convergence results.

[0070] The present invention has at least the following four beneficial effects, which can be combined arbitrarily: 1. The present invention explicitly performs representation learning on uncertain environments through a sequence latent variable model, extracts hidden information hidden under observations of uncertain environments, thereby accelerating the training of intelligent agents in uncertain environments and improving the ability of intelligent agents to handle uncertainty.

[0071] 2. By combining the sequential latent variable model with the reinforcement learning framework of the proximal policy optimization algorithm, the present invention enables the intelligent agent to learn effective strategies in continuous interaction with the environment without the need for a large amount of prior knowledge and precise models, thereby improving its adaptability to complex environments and uncertainties and enhancing the robustness of the track transfer design. 3. Previous low-thrust orbit design methods based on reinforcement learning may have a sparse reward problem when designing reward functions, resulting in low learning efficiency. The adaptive hidden state reinforcement learning method disclosed in the present invention combines the future state predicted by the sequence hidden variable model and other relevant constraints in the orbit transfer process to design a dense reward function, which can more effectively guide the agent to learn, improve learning efficiency, and accelerate convergence to a better orbit transfer strategy. The introduced reward items related to the predicted state enable the agent to make decisions not only based on the current immediate rewards, but also take into account the rewards that may be obtained in the future, thereby helping the agent to make more forward-looking and reasonable decisions and avoid short-sighted behavior. 4. Previous reinforcement learning methods may lack flexibility in determining the weight of the reward function, and cannot adapt well to different task scenarios and environmental changes or require a lot of engineering experience in the design process. The adaptive hidden state reinforcement learning method disclosed in the present invention adopts a reward function weight self-tuning algorithm, which can automatically adjust the reward function weight according to environmental feedback, improve the flexibility and adaptability of the method, optimize the track transfer design effect, and further improve the training efficiency.

[0072] In order to prove the above technical effects, the present invention trains the neural network and environment established in the previous steps, and performs Monte Carlo simulation verification on the trained model. Figure 5 The velocity increment distribution of the designed trajectory at each pulse point is shown. Figure 6The obtained robust trajectory and Monte Carlo trajectory are displayed, where one line depicts the robust trajectory generated by the strategy, the arrow of the robust trajectory represents the thrust vector at each time step, the direction of the arrow reflects the thrust direction, and its length represents the thrust magnitude. The other two lines depict the motion trajectories of Mars and Earth during the rendezvous mission, and another line represents the trajectory generated by 1,000 Monte Carlo simulations to evaluate its performance in handling uncertainty. To improve the display, the error between the Monte Carlo simulation trajectory and the optimal robust trajectory (i.e., the robust trajectory generated by the strategy) is amplified five times. Figure 6 It can be seen that the low-thrust orbit transfer design method proposed in the present invention can realize robust orbit transfer in an uncertain environment. In the uncertain environment, the independently designed trajectory curve has only a small error in the middle part, and ultimately the orbit transfer goal can be achieved.

[0073] Possible technical variations of the present invention are: The sequence latent variable model used in the present invention can be replaced by a variational autoencoder (VAE) to perform representation learning on uncertain environments. Compared with the sequence latent variable model used in the present invention, the variational autoencoder can achieve representation learning of data by mapping the input data to a latent space and encoding and decoding the data in the latent space. However, the sequence latent variable model focuses more on modeling the dynamic information in the sequence data and can better capture the dependencies in the time series. When processing sequence data, the variational autoencoder may require additional processing steps or model structures to consider the information of the time dimension. In addition, the prediction generation ability of the variational autoencoder is weak, which will lead to inaccurate prediction states in the scheme, thereby affecting model training.

[0074] In summary, the present invention discloses an interplanetary orbit transfer method based on hidden state and reinforcement learning, which relates to the field of spacecraft control technology. By establishing a sequential latent variable model, the present invention can explicitly perform representation learning on an uncertain environment and extract hidden information hidden under observations of the uncertain environment. And a reinforcement learning controller is formed by an actor network and a critic network. A reinforcement learning algorithm framework is constructed using a sequential latent variable model and a reinforcement learning controller, thereby accelerating the training of the intelligent agent in an uncertain environment and improving the intelligent agent's ability to handle uncertainty. In addition, an embodiment of the present invention will also predict the next state based on the current observation and expected operation, and incorporate the quality of the predicted next state into the reward structure, so that the immediate reward can simultaneously capture the effectiveness of the current and subsequent strategies, thereby improving the learning efficiency of the algorithm.

[0075] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An interplanetary orbit transfer method based on hidden state and reinforcement learning, characterized in that: The method comprises: Acquiring observation data of the spacecraft; wherein the observation data is determined based on state data and observation uncertainty; and the state data is calculated based on a state transfer matrix constructed based on a preset orbit transfer mission; Inputting the observed data into a preset sequence latent variable model to obtain latent variables; Inputting the hidden variables and the historical trajectory records of the spacecraft into the reinforcement learning controller to obtain the execution action of the spacecraft; Predicting next state data of the spacecraft according to the observation data and the execution action, calculating a reward value according to the state data and the next state data, and causing the reinforcement learning controller to update the orbit transfer strategy based on the reward value; Wherein, the reinforcement learning controller comprises: A critic network, used for estimating a state value according to an input latent variable and a self-value estimation function, and updating the track transfer strategy according to the state value; The actor network is used to calculate the execution action of the spacecraft according to the historical trajectory record and the orbit transfer strategy.

2. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 1, characterized in that: The state transfer matrix is: ； in, , They represent the position of the spacecraft at the kth time step and the k+1th time step respectively; , They represent the velocities of the spacecraft at the kth time step and the k+1th time step respectively; , They represent the mass of the spacecraft at the kth time step and the k+1th time step respectively; and represents the Lagrangian coefficient of the kth time step; and represents the derivative of the Lagrange coefficient at the kth time step; vector is the output of the reinforcement learning controller, representing the speed change obtained by impulse inference ; represents the equivalent exhaust velocity of the propeller; represents the position deviation caused by state uncertainty at the k+1th time step; represents the speed deviation caused by state uncertainty at the k+1th time step; The observation data is in vector form, and the calculation formula of the observation vector is: ； in, represents the observation vector; Indicates time, , represents the total time step; Represents matrix or vector transpose; represents the position deviation caused by observation uncertainty at the kth time step; represents the velocity deviation caused by observation uncertainty at the kth time step.

3. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 1, characterized in that: The sequence latent variable model is composed of an action model, a latent variable, an observation model and latent variable dynamics, and the latent variable dynamics is used to establish a connection between the observation model, the action model and the latent variable; The objective function of the sequence latent variable model is established based on the equivalence principle of maximizing the probability of observed data and maximizing the lower bound of evidence.

4. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 3, characterized in that: The latent variable dynamics includes a generative model and an inference model; The generative model is used to sequentially generate the next latent variable and the prior distribution of each latent variable according to the previous latent variable information; The inference model is used to calculate the posterior distribution of the current latent variable based on subsequent latent variable information and observation data.

5. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 4, characterized in that: The objective function of the sequence latent variable model is: ； in, represents the objective function of the sequence latent variable model; represents the sequence length of a single sequence latent variable model; represents the relative entropy of two distributions; Represents the probability of each observation data; represents the lower bound of evidence; represents the observation vector; is the hidden variable at the kth time step, and They represent the first layer latent variables and the second layer latent variables at the kth time step respectively; represents the execution action at the kth time step; Represents the trajectory of the function obtained by sampling from the probability distribution q expected value.

6. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 1, characterized in that: The overall objective function of the reinforcement learning controller consists of the objective formula of the actor network, the mean squared value function error term based on the value estimation of the critic network, and the entropy term of the decision distribution generated by the policy function.

7. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 6, characterized in that: The overall objective function of the reinforcement learning controller is: ； in, represents the overall objective function of the reinforcement learning controller; Represent the objective formula of the actor network; represents the mean square value function error term; represents the entropy term; Represents the trainable parameters of the neural network corresponding to the reinforcement learning controller; , represents a hyperparameter; The objective formula of the actor network is: ； in, Represent the objective formula of the actor network; Indicates that along the strategy The sampled trajectory expectations; represents the total time step; represents the importance weight, represents the old strategy used when sampling data, Indicates the execution of an action. Indicates the system status; Indicates that The value range is limited to The function of Indicates The importance weight of each time step; Represents the hyperparameter that controls the scope of pruning; represents the generalized advantage estimate; The calculation formula of the mean square value function error term is: ； in, represents the mean square value function error term; represents a latent variable; Indicated by the parameter The self-value estimation function established by the neural network; Reward discount factor Power; Indicates The environment reward for each time step; The calculation formula of the entropy term is: ； in, represents the entropy term; Indicates time steps and previous The trajectory of the steps.

8. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 1, characterized in that: Calculating a reward value according to the state data and the next state data includes: Acquiring a reference trajectory of the spacecraft, wherein the reference trajectory is an optimal trajectory obtained based on a deterministic environment; Calculate a reward item according to the next state data and the reference trajectory; Acquire target state data corresponding to the target planet, calculate a final position error and a final velocity error of the spacecraft according to the state data and the target state data, and calculate a penalty term according to the final position error and the final velocity error respectively; Calculate a single-step fuel loss of the spacecraft according to the state data, and calculate a penalty term according to the single-step fuel loss; Calculate a penalty term according to the state data and a preset maximum thrust threshold; Calculating a penalty term based on the state data and a preset forbidden area; Calculate a position deviation and a speed deviation according to the state data and the reference trajectory, and calculate a penalty term according to the position deviation and the speed deviation respectively; The reward value is calculated according to the reward item and all the penalty items.

9. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 8, characterized in that: Calculating the reward value according to the reward item and all the penalty items includes: Calculating the reward value according to a weighted combination of the reward item and all the penalty items; The method for adjusting the weights corresponding to the reward items and the penalty items includes: Establishing a reward function according to the reward term and all the penalty terms, wherein the penalty term established based on the final position error and the final velocity error includes a constraint value that changes with the training process; Using two sets of reward functions with different weights to perform training respectively; Determining the training potentials corresponding to the reward items and the penalty items respectively according to the relative sizes of the reward items and the penalty items in the training results and the difference between the training results of the two groups of reward functions; The weight is adjusted according to the training potential of the reward item and each of the penalty items.

10. The interplanetary orbit transfer method based on hidden state and reinforcement learning according to claim 8, characterized in that: The reward function for calculating the reward value is: ； in, Represents the reward value; Represents the weight of each term in the reward function; Represents constraints to reduce the sensitivity of the reward function weights; Indicates the penalty item corresponding to the forbidden area; represents the penalty term corresponding to the maximum thrust threshold; represents the penalty term corresponding to the single-step fuel loss; represents the final position error; represents the final velocity error; Represents the penalty term corresponding to the position deviation; represents the penalty term corresponding to the speed deviation; Indicates the reward item corresponding to the next state data.

Images