Satellite Exploration Control System and Method Based on Deep Reinforcement Learning

Abstract

This invention discloses a satellite exploration control system and method based on deep reinforcement learning. The system includes a reinforcement learning module and an environment module. The environment module includes an action module, an initialization module, a state update module, a basic environment module, and a reward calculation module. The reinforcement learning module calls the initialization module, which initializes the basic environment. After initialization, the initialization module returns the current state. The reinforcement learning module then outputs an action to the environment module after obtaining the current state. The action module in the environment module captures and processes the action, which is then passed to the state update module, which calls the basic environment module to update the current environment state. The reward calculation module then obtains all state and action information, provides the reward for the current learning round, and passes the reward and the updated state to the reinforcement learning module. The reinforcement learning module combines all the current information to form a four-tuple and trains it. Then, it outputs the next action based on the next state.

Description

Technical Field

[0001] This invention belongs to the field of aircraft navigation, guidance and control, and specifically relates to a navigation control system and method based on deep reinforcement learning. Background Technology

[0002] Satellite exploration and control is a challenging task, requiring high-precision dynamic adjustments in complex space environments. In the past, due to limitations in satellite computing resources and energy, efficient and complex computing devices were not carried. However, with the development of semiconductors and the aerospace industry today, carrying a computing terminal capable of training and simulation has become a reality. Traditional satellite control methods rely on predictions and calculations from various observation devices at ground centers, modifying the parameters of automatic control methods based on the calculation results. While simple, stable, and highly accurate, these methods suffer from high latency and complex parameter tuning, failing to achieve real-time control and posing significant safety risks to satellites. If control commands are not accurately transmitted due to certain factors, irreparable losses may occur. Furthermore, with increasingly complex space missions, conventional attitude control is no longer sufficient, especially the PID algorithm most commonly used in satellite attitude control. Although it consumes fewer computing resources, it requires complex adjustments to detailed parameters in both hardware and software, making it difficult to adapt to the demands of dynamic and complex missions.

[0003] Traditional control methods may struggle to handle these changes and uncertainties. Deep reinforcement learning combines the advantages of deep learning and reinforcement learning, enabling it to learn and make autonomous decisions. This allows it to achieve optimization goals in complex and dynamic environments, adapting and optimizing control strategies through continuous trial and error, thus achieving more precise and effective satellite exploration control. Specifically, deep reinforcement learning can be applied to optimize satellite orbit control, attitude control, and energy management. For example, by training satellites with deep reinforcement learning, they can autonomously select optimal thruster activity, adjust attitude, and optimize energy consumption. This can significantly improve satellite detection efficiency and accuracy, enabling better scientific research and commercial applications. Therefore, researching deep reinforcement learning for satellite exploration control has significant meaning and application value.

[0004] In summary, deep reinforcement learning may have the following advantages over traditional automatic control algorithms in satellite control problems:

[0005] (1) Autonomous learning: Deep reinforcement learning can gradually optimize control strategies through autonomous learning. Compared with traditional automatic control algorithms, deep reinforcement learning can update strategies in real time according to changes in the environment, thereby improving the robustness and adaptability of control.

[0006] (2) Better adaptability to complex and uncertain environments: During satellite control, the environment changes are very complex, including various physical influences and attitude adjustments. Traditional automatic control algorithms are often designed based on specific models and assumptions, making it difficult to cope with complex and uncertain environmental changes. Deep reinforcement learning, through exploratory learning, can adaptively adjust control strategies to cope with these changes.

[0007] (3) Suitable for nonlinear and high-dimensional problems: Satellite control problems usually involve nonlinear and high-dimensional state spaces, including various state information such as satellite attitude, thrust vector, and spatial position. Traditional automatic control algorithms have difficulty processing this high-dimensional information, while deep reinforcement learning can effectively process high-dimensional information through deep neural networks, thereby improving control accuracy and efficiency.

[0008] Therefore, deep reinforcement learning offers better adaptability and flexibility in satellite control problems, enabling more efficient and precise control strategies, making it an important option for solving such problems. However, current research on this topic is limited. Based on this, this invention proposes a satellite exploration control system and method based on deep reinforcement learning. Summary of the Invention

[0009] The purpose of this invention is to solve the problem of satellite exploration control and ensure its effectiveness. It provides a satellite exploration control system and method based on deep reinforcement learning. The exploration control operation method of this invention involves ground-based deployment and training. After training, the model is deployed to the satellite for execution. The satellite transmits space data to the model, automatically realizing the control function and ensuring adaptive adjustment and control of the satellite.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] An exploration control algorithm system based on deep reinforcement learning is proposed. Its system architecture consists of a deep reinforcement learning module and an environment module. The environment module includes an action module, an initialization module, a state update module, a basic environment module, and a reward calculation module. The reinforcement learning module calls the initialization module, which initializes the basic environment. After initialization, the initialization module returns the current state s. Upon obtaining the current state s, the reinforcement learning module outputs an action 'a' to the environment module, where 'a' is a triplet array representing the satellite's three-axis acceleration action. The action module within the environment module captures and processes the actions, which are then passed to the state update module for further processing. The basic environment module handles the current environment state update, and the reward calculation module obtains all state and action information and gives the reward r for the current learning round. The reward r and the updated state s′ are then passed to the reinforcement learning module. The reinforcement learning module combines the current state s, the action a taken by the agent in the current state, the reward r for the action output by the agent, and the state s′ after the agent completes the action to form a quadruple (s,a,r,s′). After one round of neural network update, the quadruple is input into the network, and the next action a′ is output according to the next state s′, thus completing the Markov decision process closed loop.

[0012] Preferably, the deep reinforcement learning module includes a TD3 network submodule, a composite noise submodule, and a network supplement module, which can solve machine learning-related problems.

[0013] Further preferred, the TD3 network submodule includes an Actor network, a Critic network, and an experience replay pool;

[0014] Composite noise module: includes exploration noise and strategy noise; noise is a function method, which can be understood as a small module;

[0015] The network add-on modules include an LSTM network submodule and a RunningMeanStd normalization function submodule. LSTM (Long Short-Term Memory) is a special type of Recurrent Neural Network (RNN) primarily used for processing sequential data and effectively addressing the vanishing and exploding gradient problems inherent in RNNs. LSTM networks utilize gating mechanisms to control input, output, and forgetting of information, thus better capturing long-term dependencies and identifying correlations and patterns in time series data, making it suitable for reinforcement learning. The RunningMeanStd normalization function is a normalization method that maps data to a certain range through preprocessing. This method normalizes the data by maintaining a sliding window to calculate the mean and standard deviation.

[0016] Preferably, deep reinforcement learning is a branch of machine learning that aims to solve the problem of maximizing the expected cumulative reward by an agent through trial and error in its interaction with the environment. Deep reinforcement learning typically involves an agent, an environment, and a set of actions. The agent takes a series of actions to interact with the environment, which provides feedback. The agent uses this feedback to improve its action selection strategy to obtain a greater reward. The core of deep reinforcement learning is learning based on reward signals. The agent needs to try different actions to maximize its long-term gains (cumulative reward) and can continuously learn and improve its decision-making strategy in the interaction, ultimately achieving the task objective. The control strategy of the deep reinforcement learning module adopts the Advanced-TD3 algorithm. The specific steps of the Advanced-TD3 algorithm are as follows:

[0017] S1. At the start of training, initialize the parameters by loading the hyperparameters for the Actor network, Critic network, experience replay pool, and LSTM network.

[0018] S2. Perform several rounds of random sampling with the environment module to expand the amount of experience in the experience pool;

[0019] S3. Interact with the environment under the intervention of compound noise, and obtain a tuple consisting of the current state, action, reward and next state from the environment module; where the current state includes the observation value of the environment and the previous action, and the previous state is the observation value and action of the previous step.

[0020] S4. Use the RunningMeanStd normalization method to normalize the current state;

[0021] S5. The input includes the current state and several previous states after being processed by the LSTM network (forming a time series). At the same time, after receiving the input values, the LSTM network models the feature vector sequence and learns the time dependencies between the state sequences.

[0022] S6. Use reinforcement learning to select the next action from the output of the LSTM network to maximize the expected reward, while updating the parameters of the Actor network and the Critic network.

[0023] The detailed algorithm pseudocode is shown below:

[0024]

[0025]

[0026] Preferably, the environment module is used to solve the problem of near-Earth space environment simulation. The core environment's underlying physics functions are all built internally by VPython, and the environment required for reinforcement learning is constructed using internal library functions. VPython is an extension of the Python language, implemented based on OpenGL, providing a series of functions and classes, supporting integration with other Python libraries, and is primarily used for 3D computer graphics, suitable for the development of physical simulations. Based on VPython, 3D animations can be easily created for physical simulation, visualization, and teaching.

[0027] Preferably, before the reinforcement learning module learns the environment, the trajectory exploration control problem needs to be transformed into a Markov decision process. A Markov decision process mainly consists of three key elements: State, Action, and Reward. The detailed composition of these three elements will be described below.

[0028] (1) State space

[0029] The satellite's motion environment uses a geocentric inertial coordinate system. The x-axis points to the vernal equinox, is a line within the Earth's equatorial plane intersecting the Sun-Earth plane, and points towards 0 degrees ecliptic longitude. The y-axis is perpendicular to the x-axis and points towards 90 degrees ecliptic longitude, i.e., east. The z-axis coincides with the Earth's rotation axis and points towards the North Pole. Based on this, the state space mainly consists of binary arrays, representing the satellite's current position and velocity. At the current time t, the state space is structured as follows:

[0030] S t =[[x t ,y t ,z t ],[v xt ,v yt ,x zt (1)

[0031] Here [x t ,y t ,z t [v] represents the satellite's current position across its three spatial axes. xt ,v yt ,x zt [This represents the magnitude of the satellite's three-axis velocity in its current state.]

[0032] (2) Action space

[0033] The satellite is given a reasonable initial velocity during initialization at its initial point. If there are no external forces, only the gravitational force will act on it. Simultaneously, the motion is defined as acceleration along three axes, with the specific formulas as follows:

[0034] A t =[a xt ,a yt ,a zt a∈[-c,c] (2)

[0035] Here, the magnitude of acceleration 'a' is given by the three axes, with each axis having its corresponding acceleration. The velocity range is within [-c, c], where c represents the maximum acceleration.

[0036] (3) Rewards

[0037] The reward function is a crucial component of reinforcement learning algorithms, playing a vital role in the learning process. In this invention, on the one hand, it considers the agent's safe and reliable operation, meaning it cannot collide or exceed its speed, preventing an eccentricity greater than 1 and loss of control. On the other hand, it aims to enable the agent to find the target point in a sparse environment and learn obstacle avoidance behavior before reaching the target. In all rewards, this invention sets a weighting coefficient to ensure that rewards across different dimensions tend to be as smooth as possible under proper guidance.

[0038] Therefore, the reward is divided into a reward part and a penalty part. The penalty part includes collision penalty and eccentricity penalty to ensure that when a satellite collision occurs or the satellite's eccentricity is greater than 1, both are penalized sufficiently under certain weights while ensuring the smoothness of the reward. The reward part is divided into an incentive module and a one-time reward when the target is reached.

[0039] In the penalty section, the collision penalty r cp And biased punishment r ep Once set as the trigger condition, training is immediately terminated, all current rewards are calculated, and penalties are assigned. ε is the collision penalty coefficient, and φ is the eccentricity coefficient, detailed in the following formula:

[0040] R P =εr cp +φr ep (3)

[0041] In normal training tasks, the satellite target is pre-assigned to the training environment based on the actual situation and stored in the reward. The deep reinforcement learning agent uses the coordinates of the satellite itself and the coordinates of the target point as a reference, and the difference is used as a negative reward. At the same time, in order to improve training efficiency and encourage the agent to get as close to the target point as possible in the early exploration process, an additional incentive reward function is set. If the target is being explored, the reward r in this round is higher than the reward r in the previous round. - If the difference is higher, the reward will be multiplied by a coefficient and added to the current round's reward *r*. The distance difference reward formula is as follows:

[0042] R A =αrx +βr y +δr z (4)

[0043]

[0044] Where, r x r y r z The reward is the distance difference between the current position's geocentric inertial coordinates and the target point's coordinates at time i. α, β, and δ are the weights on the three-axis reward, used to balance the smoothness of the reward itself. The distance reward is itself a negative reward. On the other hand, the specific formula for the activation function is as follows:

[0045]

[0046] in, This is the balance coefficient of the incentive function, also used to ensure the smoothness of the reward. In the incentive function, if the reward of the current state is greater than the reward of the previous state, a positive reward is given to the current state; conversely, if the reward of the current state is lower than the reward of the previous state, a negative reward is given as a reminder. All the formulas can be summarized into a single reward function, as follows:

[0047]

[0048]

[0049] This invention also discloses a satellite exploration control method based on deep reinforcement learning, which comprises the following steps:

[0050] S1. Initialize the basic environment. After initialization, obtain the current state s and output action a, where a is a ternary array representing the satellite's three-axis acceleration action.

[0051] S2. Capture and process actions, update the current environment state, obtain all state and action information, give the reward r for the current learning round, and combine the reward r with the updated state s′;

[0052] S3. Combine all the current information to form a quadruple (s, a, r, s′) and train it. Then, output the next action a' based on the next state s′.

[0053] This invention proposes a satellite exploration control system and method based on deep reinforcement learning, which has the following technical advantages: The use of deep reinforcement learning algorithms ensures the satellite's adaptability in control, guaranteeing the safety of satellite control; simultaneously, it proposes the Advanced-TD3 algorithm, ensuring normal training execution during satellite control missions and improving the algorithm's robustness and efficiency. Compared to the traditional TD3 algorithm, this algorithm incorporates a composite noise method, a RunningMeanStd normalization method, and LSTM-based state prediction.

[0054] Regarding the composite noise method, policy noise and exploration noise are added to the algorithm. Policy noise is a random noise added during the policy update step, which enables Advanced-TD3 to better evaluate the quality of the policy and reduce the bias in policy evaluation. Exploration noise is a dynamically decaying noise added during action output, which makes it easier for Advanced-TD3 to explore unexplored regions in the early stages of training, improving the algorithm's exploration efficiency and sampling diversity.

[0055] Regarding the RunningMeanStd normalization method, the environment module contains a large amount of data with varying dimensions. For example, a satellite in outer space can travel tens of millions of meters away from the Earth's center, but its speed generally does not exceed 10,000. The data dimensions are clearly not on the same order of magnitude. If this state is used as input to a reinforcement learning network, gradient explosion or vanishing gradients may occur, preventing normal training. Therefore, it is necessary to normalize the input data. An empirical step size n is incorporated into the normalization method, and all normalized data is stored separately as network experience, ensuring correct normalization during experience replay.

[0056] Regarding LSTM-based state prediction methods, in space exploration problems, satellite orbital states are influenced by various factors such as gravity, space disturbances, network noise, and exploration noise, which exhibit complex long-term dependencies. This can prevent the agent from achieving optimal decisions over extended periods, primarily because the TD3 algorithm lacks the ability to analyze noise fluctuations and space disturbances within the state. Therefore, improving agent training efficiency requires the agent to capture long-term dependencies. Since satellite orbital state changes are a temporal process, the agent needs to handle temporal tasks. Traditional TD3 algorithms only consider the impact of the current state and actions on subsequent rewards, neglecting long-term dependencies, potentially leading to slow learning and unstable convergence. LSTM networks can help agents better handle temporal tasks and long-term dependencies, improving performance and efficiency. Specifically, in satellite control problems, the agent needs to determine the appropriate control strategy based on the current satellite state to achieve the desired orbital state. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings in the embodiments will be briefly described below.

[0058] Figure 1 This is a schematic diagram of a satellite control application scenario for the satellite exploration control system based on deep reinforcement learning, as described in this invention.

[0059] Figure 2 This is a system block diagram of the satellite exploration control system based on deep reinforcement learning according to the present invention.

[0060] Figure 3 This diagram illustrates the Actor network module and Critic network module after fusing LSTM in the satellite exploration control system based on deep reinforcement learning, as described in this invention.

[0061] Figure 4 This is a structural diagram of the Advanced-TD3 algorithm involved in the satellite exploration control system based on deep reinforcement learning in this invention.

[0062] Figure 5 This is a schematic diagram of the VPython simulation environment involved in the satellite exploration control system based on deep reinforcement learning of this invention.

[0063] Figure 6 This is a performance comparison chart between the present invention and existing algorithms. Detailed Implementation

[0064] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0065] This invention focuses on the problem that traditional control methods cannot cope with changes and uncertainties. It utilizes deep reinforcement learning, which combines the advantages of deep learning and reinforcement learning, to enable satellite control systems to learn and make autonomous decisions. This allows them to achieve optimization goals in complex and dynamic environments. Through continuous trial and error and learning, they can adapt and optimize control strategies, thereby achieving more precise and effective satellite exploration control.

[0066] Specifically, this invention constructs a satellite exploration and control system based on deep reinforcement learning, utilizing deep reinforcement learning to optimize aspects such as satellite orbit control, attitude control, and energy management. For example, by training with deep reinforcement learning, the satellite can autonomously select the optimal thruster pulse mode while simultaneously optimizing energy consumption. This significantly improves the satellite's detection efficiency and accuracy, enabling better applications.

[0067] Furthermore, this invention also relates to a control algorithm based on Advanced-TD3, which has better adaptability and flexibility, and can achieve more efficient and accurate control strategies, thereby achieving the control objectives.

[0068] The following preferred embodiment provides a satellite exploration control system based on deep reinforcement learning for satellite exploration control problems.

[0069] This embodiment provides a satellite exploration and control system based on deep reinforcement learning, such as... Figure 1 As shown, after modeling the environment in space observed by satellites and ground observation equipment, the data center of the ground observation center sends detailed data to the server for simulation training. After the network converges, the optimal path and its control strategy are obtained and the network model is saved. Finally, the model is uploaded to the satellite for deployment, so that the satellite can use the optimal ignition method to complete the control from the optimal path.

[0070] like Figure 2 The diagram shown is a system block diagram of this embodiment. To implement the satellite exploration control strategy based on deep reinforcement learning, this embodiment considers dividing the entire strategy into multiple modules, including a reinforcement learning module and an environment module. The deep reinforcement learning module of this embodiment includes:

[0071] D3 Network Submodule: Includes Actor Network, Critic Network, and Experience Replay Pool;

[0072] Composite noise module: includes exploration noise and strategy noise;

[0073] Network add-on module: includes LSTM network submodule and RunningMeanStd normalization function submodule.

[0074] The environment module also includes an action module, an initialization module, a state update module, a basic environment module, and a reward calculation module.

[0075] When training begins, the reinforcement learning module first calls the initialization module to initialize the basic environment. After initialization, the initialization module returns a current state s. The reinforcement learning module then outputs an action 'a', which is a triplet array representing the satellite's three-axis acceleration action, and sends it to the environment module. The action module in the environment module captures and processes the action, which is then passed to the state update module, which calls the basic environment module to update the current environment state. Finally, the reward calculation module obtains all state and action information and outputs the reward 'r' for this round of learning. It then passes the reward 'r' and the updated state 's' to the reinforcement learning module. The reinforcement learning module combines all the current information to form a quadruple (s, a, r, s'), which is then passed to the reinforcement learning module for training. After training, the next action 'a' is output based on the next state 's', thus completing the Markov decision process loop.

[0076] like Figure 3 The diagram shows the Actor and Critic network modules of the satellite exploration control system based on deep reinforcement learning fused with LSTM, as provided in this embodiment. It demonstrates how to add an LSTM network to the Actor and Critic networks of the TD3 algorithm to process temporal information in the continuous action space. In the Actor network, the LSTM outputs of the current state and historical states can be concatenated as input to the Actor network, thereby capturing the correlation between the current and historical states. Similarly, in the Critic network, the LSTM outputs of the current state and historical states can be concatenated as input to the Critic network, thus better estimating the Q-value of the current state. Assume the state at time t is s. t The output of the Actor network is a t The output of the Critic network is Q(s) t ,a tIn this network, the parameters of the LSTM network and the TD3 algorithm are optimized together. The network parameters are updated through backpropagation to minimize the loss function, enabling the agent to learn a better policy. Ultimately, in this way, the LSTM network and the TD3 algorithm can jointly handle temporal tasks in a continuous action space and capture long-term dependencies in the time series, improving the agent's performance and efficiency.

[0077] like Figure 4 The diagram shown illustrates the Advanced-TD3 algorithm architecture used in the deep reinforcement learning-based satellite exploration control system of this embodiment. The specific steps are as follows:

[0078] The S1.Advanced-TD3 algorithm initializes parameters at the start of training, loading and initializing hyperparameters for the Actor module, Critic module, experience pool, and LSTM network.

[0079] S2. Perform several rounds of random sampling with the environment module to expand the amount of experience in the experience pool.

[0080] The S3.Advanced-TD3 algorithm interacts with the environment amidst complex noise, retrieving a tuple from the environment module consisting of the current state, action, reward, and next state. The current state includes the environmental observations and the previous action, while the previous state contains the observations and action from the previous step.

[0081] S4. The current state is normalized using the RunningMeanStd normalization method.

[0082] The input to the S5.Advanced-TD3 algorithm network includes the current state and several previous states after being processed by the LSTM network (recurrent neural network) (forming a time series). At the same time, after receiving the input values, the LSTM network models the feature vector sequence and learns the time dependencies between the state sequences.

[0083] S6. Use reinforcement learning to select the next action from the output of the LSTM network to maximize the expected reward, while updating the network parameters of the Actor and Critic modules to accumulate and maximize the reward in subsequent training.

[0084] In step S3, the composite noise consists of target policy noise and adaptive target exploration noise. Target policy noise is a random noise added during the policy update step, which allows Advanced-TD3 to better evaluate the quality of the policy and reduce policy evaluation bias. Adaptive target exploration noise is a dynamically decaying noise added during action output, making it easier for Advanced-TD3 to explore unexplored regions during early training, improving algorithm exploration efficiency and sampling diversity. The formulas for target policy noise and adaptive target policy noise are as follows:

[0085] a target =clip((μ target (s t )+ò a ),-c,c)

[0086] ò a =rand(-e,e) (9)

[0087] a = clip((μ(s) t )+ò b )-c,c)

[0088]

[0089] Among them, μ target μ and μ represent the policy functions in the actor-target network and the actor network, respectively, representing the actions and states s. t The input to either policy function will produce an output action. a and ò b Representing target policy noise and adaptive target exploration noise, their values ​​are determined differently: ò a It is a value randomly selected from the upper and lower bounds (-e, e) defined during algorithm initialization, while ò b The decay function is determined during algorithm initialization. and noise lower bound σ min The maximum value is taken from the given values. In this embodiment, the two actions and two noises are added together and converged to the upper and lower bounds of the actions (-c, c) to obtain the final action a. target and a.

[0090] In step S4, RunningMeanStd normalization normalizes the data by maintaining a sliding window to calculate the mean and standard deviation. Specifically, for each element in the input data, the algorithm updates the mean and standard deviation and then uses these values ​​to normalize the element. Z-score, on the other hand, normalizes the data by calculating the mean and standard deviation of the samples. RunningMeanStd normalization scales the data to the interval between 0 and 1, even if the input data has negative values. Z-score scales the data to a standard normal distribution centered at 0, i.e., a mean of 0 and a standard deviation of 1. Because RunningMeanStd normalization uses a sliding window to calculate the mean and standard deviation, it is a relatively stable normalization method for streaming and real-time data. The formula for RunningMeanStd normalization can be expressed as follows: Let the current input be x, the current time step be t, and the mean of the previous time step be μ. t-1 The standard deviation of the previous time step is σ. t-1 Then the mean μ at the current time step t and standard deviation σ t The calculation formula is as follows:

[0091]

[0092]

[0093] The above formula allows for continuous updating of the mean and standard deviation, thus performing RunningMeanStd normalization on the input data. Specifically, for each input x, its standardized value can be calculated:

[0094]

[0095] Once the satellite's position and velocity are obtained through interaction with the environment, they are then input into RunningMeanStd for normalization. The normalization result can then be used to ensure the normal updating of the network gradient during reinforcement learning network training.

[0096] like Figure 5The diagram shows the simulation environment of the VPython-based satellite exploration control system of this invention, which utilizes deep reinforcement learning. In this embodiment, all necessary low-level physical functions are built internally by VPython, and the environment required for reinforcement learning is constructed using internal library functions. VPython is an extension of the Python language, implemented based on OpenGL. It provides a series of functions and classes, supports integration with other Python libraries, and is primarily used for 3D computer graphics, making it suitable for physical simulation development. Based on VPython, 3D animations can be easily created for physical simulation, visualization, and teaching purposes.

[0097] A satellite simulation environment built using VPython can be used to simulate the motion and control processes of satellites, as well as the control effects of the control system. It mainly includes the following aspects:

[0098] (1) Satellite motion model: The process of a satellite moving in its orbit can be described by kinematic and dynamic equations. The physics library in VPython can be used to easily implement a satellite motion model, including orbital parameters, motion state, etc.

[0099] (2) Control System Model: The satellite control process relies on a control system, which can be implemented by writing Python code. In the VPython environment, the controller model can be embedded into the simulation environment to achieve real-time satellite control.

[0100] (3) Simulation Environment: Using VPython's 3D graphics capabilities, a simulation environment can be created, including a celestial coordinate system, satellite orbits, etc. In the simulation environment, the satellite control process can be simulated by controlling the motion state of the satellite model, and real-time visualization can be displayed.

[0101] (4) Data Analysis: Using VPython's data visualization capabilities, the motion state and control data during the simulation process can be visualized. By analyzing the simulation data, the performance of the control system and optimization schemes can be evaluated.

[0102] like Figure 6As shown, compared with existing contrastive algorithms based on reinforcement learning (standard TD3 algorithm and standard DDPG algorithm), the algorithm of this invention is superior to other contrastive algorithms based on reinforcement learning in both training speed and training stability. Firstly, regarding the speed of algorithm exploration, the Advanced-TD3 algorithm explores rewards much faster in the early stages, reaching the target point within approximately 2000 training rounds, while the standard TD3 algorithm reaches it within approximately 3000 rounds and the standard DDPG algorithm within approximately 5000 rounds. This directly demonstrates that composite noise significantly improves the agent's exploration ability, and compared to random number methods, it also demonstrates stronger exploration capabilities. Secondly, in terms of algorithm control, the Advanced-TD3 algorithm exhibits significantly higher stability during convergence, without significant fluctuations, while the other two reinforcement learning comparison algorithms show some moderate fluctuations. This proves that the Advanced-TD3 algorithm is more stable during convergence than the other two reinforcement learning algorithms. Finally, regarding optimal algorithm control, the Advanced-TD3 algorithm can still guarantee the possibility of exploring targets with higher rewards during training, while the other two reinforcement learning comparison algorithms have a lower probability of exploring better trajectories for further training. In summary, the Advanced-TD3 algorithm, after being improved based on the standard TD3 algorithm, outperforms the standard TD3 algorithm and is also superior to the standard DDPG algorithm. Therefore, it can be concluded that the Advanced-TD3 algorithm proposed in this invention can solve the satellite exploration and control problem and outperforms the standard TD3 and standard DDPG algorithms in terms of performance.

[0103] In summary, this invention provides a satellite exploration and control system based on deep reinforcement learning, which utilizes deep reinforcement learning to optimize satellite orbit control, attitude control, and energy management. For example, by training with deep reinforcement learning, the satellite can autonomously select the optimal thruster pulse mode while simultaneously optimizing energy consumption. This significantly improves the satellite's detection efficiency and accuracy, enabling better applications. Furthermore, this invention also relates to a control algorithm based on Advanced-TD3, which offers better adaptability and flexibility, enabling more efficient and precise control strategies to achieve the control objectives.

[0104] A preferred embodiment of the present invention also discloses a satellite exploration control method based on deep reinforcement learning, which comprises the following steps:

[0105] S1. Initialize the basic environment. After initialization, obtain the current state s and output action a, where a is a ternary array representing the satellite's three-axis acceleration action.

[0106] S2. Capture and process actions, update the current environment state, obtain all state and action information, give the reward r for the current learning round, and combine the reward r with the updated state s′;

[0107] S3. Combine all the current information to form a quadruple (s, a, r, s′) and train it. Then, output the next action a' based on the next state s′.

[0108] Other aspects of this embodiment can be found in the foregoing embodiments.

[0109] The embodiments described above are merely preferred embodiments for fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Any person skilled in the art to which this invention pertains can make corresponding additions and modifications to the described specific embodiments, or make equivalent substitutions or changes in other ways. Therefore, the above embodiments are only intended to highlight and describe the objectives and advantages of the present invention, and are not limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A satellite exploration control system based on deep reinforcement learning, characterized by: The system comprises a reinforcement learning module and an environment module. The environment module includes an action module, an initialization module, a state update module, a basic environment module, and a reward calculation module. The reinforcement learning module calls the initialization module, which initializes the basic environment. After initialization, the initialization module returns the current state s. After obtaining the current state s, the reinforcement learning module outputs action a to the environment module, where a is a triple array representing the satellite's three-axis acceleration action. The action module in the environment module captures and processes the actions, which are then passed to the state update module, which calls the basic environment module to update the current environment state. Finally, the reward calculation module obtains all state and action information, provides the reward r for this round of learning, and passes the reward r and the updated state s′ to the reinforcement learning module. The reinforcement learning module combines the current state s, the output action a, the reward r, and the next state s′ to form a quadruple (s, a, r, s′), which is used as the input value of the network and trained. Then, it outputs the next action a' based on the next state s′. The reinforcement learning module includes: TD3 Network Submodule: Includes Actor Network, Critic Network, and Experience Replay Pool; Composite noise module: includes exploration noise and strategy noise; Network add-on module: includes LSTM network submodule and RunningMeanStd normalization function submodule; Before the learning environment, the reinforcement learning module transforms the orbital exploration and control problem into a Markov decision process, which consists of three key elements: state, action, and reward. The rewards are described below: The rewards are divided into reward and penalty parts. The penalty part includes collision penalty and eccentricity penalty, while the reward part includes incentives and a one-time reward upon reaching the goal. In the penalty section, the collision penalty r cp And biased punishment r ep Once set as the trigger condition, training is immediately terminated, all current rewards are calculated, and penalties are assigned. ε is the collision penalty coefficient, and φ is the eccentricity coefficient, as shown in the following formula: R P =εr cp +φr ep (3) If the target is being explored, the reward r for this round is greater than the reward r for the previous round. - If the difference is higher, the difference will be multiplied by a coefficient and added to the current round's reward r; the distance difference reward formula is as follows: R A =αr x +βr y +δr z (4) Where, r x r y r z The reward is the distance between the current position's geocentric inertial coordinates and the target point's coordinates at time i. α, β, and δ are the weights on the three-axis reward, used to balance the smoothness of the reward itself. The distance reward is itself a negative reward. On the other hand, the specific formula for the activation function is: in, It is the balance coefficient of the incentive function, used to ensure the smoothness of the reward. In the incentive function, if the reward of the current state is greater than the reward of the previous state, a positive reward will be given to the current state. Conversely, if the reward of the current state is lower than the reward of the previous state, a negative reward will be given as a reminder. In summary, these can be summarized into a single reward function, the specific formula of which is as follows:

2. The satellite exploration and control system based on deep reinforcement learning as described in claim 1, characterized in that, The control strategy of the reinforcement learning module adopts the Advanced-TD3 algorithm, and the specific steps of the Advanced-TD3 algorithm are as follows: S1. At the start of training, initialize the parameters by loading the hyperparameters for the Actor network, Critic network, experience replay pool, and LSTM network. S2. Perform several rounds of random sampling with the environment module; S3. Interact with the environment under the intervention of compound noise, and obtain a tuple consisting of the current state, action, reward and next state from the environment module; where the current state includes the observation value of the environment and the previous action, and the previous state is the observation value and action of the previous step. S4. Use the RunningMeanStd normalization method to normalize the current state; S5. The input includes the current state and several previous states after being processed by the LSTM network. At the same time, after receiving the input values, the LSTM network models the feature vector sequence and learns the time dependencies between the state sequences. S6. Use reinforcement learning to select the next action from the output of the LSTM network to maximize the expected reward, while updating the parameters of the Actor network and the Critic network.

3. The satellite exploration control system based on deep reinforcement learning as described in claim 1, characterized in that, The status description is as follows: The satellite's motion environment uses a geocentric inertial coordinate system. The x-axis points to the vernal equinox, is a line within the Earth's equatorial plane intersecting the Sun-Earth plane, and points towards 0 degrees ecliptic longitude. The y-axis is perpendicular to the x-axis and points towards 90 degrees ecliptic longitude, i.e., east. The z-axis coincides with the Earth's rotation axis and points towards the North Pole. Based on this, the state space mainly consists of binary arrays, representing the satellite's current position and velocity. At the current time t, the state space is structured as follows: S t =[[x t ,y t ,z t ],[v xt ,v yt ,x zt ]] (1) [x t ,y t ,z t [v] represents the satellite's current position across its three spatial axes. xt ,v yt ,x zt [This represents the magnitude of the satellite's three-axis velocity in its current state.] 4. The satellite exploration control system based on deep reinforcement learning as described in claim 1, characterized in that, The actions described are as follows: The satellite is given a reasonable initial velocity at the initial point. If there are no external forces, only the gravitational force will act on it. Simultaneously, the motion is defined as acceleration along three axes, with the specific formulas as follows: A t =[a xt ,a yt ,a zt ] a∈[-c,c] (2) The magnitude of acceleration is a, and each of the three axes has its corresponding acceleration. The velocity range is within [-c, c], where c represents the maximum acceleration.

5. A satellite exploration control method based on deep reinforcement learning, based on the satellite exploration control system based on deep reinforcement learning as described in claim 1, characterized in that... Follow these steps: S1. Initialize the basic environment. After initialization, obtain the current state s and output action a, where a is a ternary array representing the satellite's three-axis acceleration action. S2. Capture and process actions, update the current environment state, obtain all state and action information, give the reward r for the current learning round, and combine the reward r with the updated state s′; S3. Combine all the current information to form a quadruple (s, a, r, s′) and train it. Then, output the next action a' based on the next state s′.

Images