Reinforcement learning driven random MPC control method for terminal strategy optimization heuristic

Abstract

The invention provides a random MPC control method driven by reinforcement learning and inspired by terminal strategy optimization, and relates to the field of on-orbit avoidance control, and the method comprises the steps: constructing a discrete relative dynamics model; generating expert demonstration data through stochastic model predictive control (SMPC); defining the state and action of reinforcement learning and designing an instant reward function; training a terminal strategy by adopting an offline reinforcement learning algorithm, determining a reinforcement learning state in a prediction time domain and generating corresponding reference control input to obtain an initial reference control sequence; and a random model prediction control SMPC framework is adopted, an initial reference control sequence is introduced into a target function, solving optimization is performed to obtain a current control quantity, and prediction and optimization at the next moment are propelled in a rolling manner. According to the method, through fusion of reinforcement learning and random model prediction control, the spacecraft control has better adaptability and robustness when dealing with orbit perturbation and environment uncertainty, and the overall control performance of spacecraft space debris avoidance is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of on-orbit evasion control technology, specifically to a reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization. Background Technology

[0002] With the continuous increase in the number of spacecraft in low Earth orbit, the density of space debris is constantly rising, significantly increasing the potential collision risk between spacecraft and space debris, seriously threatening the safety of spacecraft operation in orbit. To address this risk, operational management agencies typically forecast rendezvous and approach events between spacecraft and debris in the near future. When the collision probability exceeds a preset threshold, an avoidance maneuver command is issued. To reduce the burden on ground control and improve the avoidance response speed and the autonomous operation capability of spacecraft, autonomous avoidance decision-making and control of spacecraft in orbit has become a research hotspot in recent years. Among these, model predictive control (SMPC) is widely used in spacecraft rendezvous, formation flying, and collision avoidance missions due to its excellent ability to handle constrained optimization problems. Stochastic model predictive control (SMPC), which can adapt to uncertainties in orbit and environment, has become an important research direction in the field of spacecraft avoidance control. Terminal strategy inspiration can be obtained through offline training using data-driven methods such as reinforcement learning, providing a new path for optimizing avoidance control performance.

[0003] In related technologies, spacecraft control and collision avoidance schemes based on stochastic model predictive control (SMPC) are mainly divided into two categories. One category addresses the multi-source disturbance problem in spacecraft attitude control. It first establishes a spacecraft attitude dynamics model and discretizes it into a state-space form. Then, a disturbance observer is designed to estimate external random disturbances, transforming random constraints into deterministic constraints. Finally, a finite-time optimization problem is constructed by combining stochastic model predictive control (SMPC) with anti-interference control. The attitude control variables are solved online to achieve constrained attitude stability control. The other category focuses on satellite collision avoidance scenarios. It uses historical orbit data to estimate the uncertainty distribution of the target orbit, transforming collision probability constraints into chance constraints. By constructing a cost function that includes propellant consumption and collision risk penalties, a stochastic model predictive control (SMPC) optimization problem with probabilistic safety constraints is formed. The avoidance maneuver sequence is obtained through rolling solutions, achieving a balance between collision probability control and fuel consumption / orbit deviation.

[0004] Existing spacecraft evasion control technologies based on stochastic model predictive control (SMPC) still have significant shortcomings, making it difficult to adapt to the autonomous evasion requirements in complex orbits and multi-constraint scenarios. On one hand, solution efficiency is limited. Existing SMPC control methods require solving an optimization problem with probabilistic safety constraints from scratch in each control cycle. This is not only sensitive to initial values ​​and requires a large number of iterations, but also prone to excessively long solution times and difficulty in obtaining feasible solutions in complex scenarios with multiple debris and constraints, failing to meet the real-time requirements of engineering applications. On the other hand, terminal processing is relatively coarse. Existing SMPC-based evasion methods often use fixed-form quadratic terminal costs or simple terminal constraints in the terminal phase, with parameter selection relying heavily on experience. This makes it difficult to accurately characterize the differences in long-term operational safety and fuel consumption among different states near the terminal, thus limiting the overall performance and adaptability of evasion control. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a stochastic MPC control method and system inspired by reinforcement learning-driven terminal policy optimization. This solves the problems of low solution efficiency, poor real-time performance, coarse terminal processing methods, and parameter selection relying on experience in existing SMPC spacecraft avoidance control technologies, resulting in insufficient overall performance and adaptability, making it difficult to meet the autonomous avoidance requirements of complex orbits and multi-constraint scenarios.

[0006] To achieve the above objectives, this application provides the following technical solution: In a first aspect, embodiments of this application provide a stochastic MPC control method inspired by reinforcement learning-driven terminal policy optimization. This stochastic MPC control method includes: acquiring the spacecraft's system state and control inputs; considering orbital perturbations and environmental uncertainties; constructing a discrete relative dynamics model of the spacecraft; generating expert demonstration data representing the state-control trajectory set through stochastic model predictive control (SMPC); defining the reinforcement learning state and actions based on the system state and control inputs; designing an immediate reward function to guide the terminal policy to iterate towards the desired control behavior; constructing a parameterized terminal policy; training the terminal policy using an offline reinforcement learning algorithm based on the expert demonstration data and the immediate reward function; determining the reinforcement learning state in the prediction time domain and generating corresponding reference control inputs after the terminal policy training converges; sorting the reference control inputs to obtain an initial reference control sequence; and using the stochastic model predictive control (SMPC) framework, explicitly introducing the initial reference control sequence into the constructed objective function, solving and optimizing to obtain the current control quantity, and continuously advancing the prediction and optimization for the next moment to achieve closed-loop control for spacecraft space debris avoidance.

[0007] Secondly, embodiments of this application provide a stochastic MPC control system inspired by reinforcement learning-driven terminal policy optimization. The stochastic MPC control system includes: a data generation module, a definition and design module, a training and ranking module, and a prediction and optimization module.

[0008] Specifically, the data generation module acquires the spacecraft's system state and control inputs, considers orbital perturbations and environmental uncertainties, and constructs a discrete relative dynamics model of the spacecraft; it generates expert demonstration data representing the state-control trajectory set through stochastic model predictive control (SMPC); the definition and design module defines the state and actions of reinforcement learning based on the system state and control inputs, and designs an immediate reward function to guide the terminal policy to iterate towards the desired control behavior; the training and ranking module constructs a parameterized terminal policy, trains the terminal policy using an offline reinforcement learning algorithm based on expert demonstration data and the immediate reward function, determines the reinforcement learning state in the prediction time domain and generates the corresponding reference control inputs, and ranks the reference control inputs to obtain the initial reference control sequence; the prediction and optimization module uses the stochastic model predictive control (SMPC) framework, explicitly introduces the initial reference control sequence into the constructed objective function, solves and optimizes to obtain the current control quantity, and continuously advances the prediction and optimization for the next moment to achieve closed-loop control for space debris avoidance of the spacecraft.

[0009] Thirdly, embodiments of this application provide an electronic device, which includes: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the reinforcement learning-driven terminal policy optimization-inspired stochastic MPC control method described in the first aspect above.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the reinforcement learning-driven terminal policy optimization-inspired stochastic MPC control method described in the first aspect above.

[0011] This application provides a reinforcement learning-driven stochastic MPC control method and system inspired by terminal policy optimization. Compared with existing technologies, it has the following advantages: This application constructs a discrete relative dynamics model of the spacecraft by considering orbital perturbations and environmental uncertainties, ensuring that the control strategy design is grounded in realistic dynamics. It utilizes stochastic model predictive control (SMPC) to generate professional and reasonable expert demonstration data, providing data support for offline reinforcement learning training of the terminal strategy. Simultaneously, by defining the states and actions of reinforcement learning and designing an immediate reward function, it accurately guides the terminal strategy to iterate towards the desired control behavior for spacecraft space debris avoidance, effectively avoiding blind strategy training. This application achieves efficient convergence of the terminal strategy by constructing a parameterized terminal strategy and combining it with expert demonstration data and an immediate reward function for offline reinforcement learning training. It can also generate and sort reference control inputs to obtain an initial reference control sequence. Explicitly introducing this initial reference control sequence into the objective function of the SMPC framework for optimization effectively reduces the optimization search range and improves the efficiency and specificity of solving the current control variables. Combined with the rolling propagation characteristics of SMPC, it achieves closed-loop control for spacecraft space debris avoidance. This application integrates reinforcement learning and stochastic model predictive control, giving full play to the advantages of reinforcement learning in policy iterative optimization and adaptive generation of control references, as well as the technical characteristics of SMPC in handling stochastic uncertainties and rolling optimization closed-loop control. This makes spacecraft control more adaptable and robust in dealing with orbital perturbations and environmental uncertainties, and significantly improves the overall control performance of spacecraft space debris avoidance. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization, as provided in an embodiment of this application. Figure 2 This is a flowchart of an offline reinforcement learning training terminal strategy provided in an embodiment of this application; Figure 3 This is a flowchart of a randomized MPC process that takes into account terminal strategies, provided in an embodiment of this application. Figure 4 This is a schematic diagram of a stochastic MPC control system inspired by reinforcement learning-driven terminal policy optimization, provided in an embodiment of this application. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0016] This application provides a reinforcement learning-driven stochastic MPC control method and system inspired by terminal policy optimization. This solves the problems of low solution efficiency, poor real-time performance, coarse terminal processing and parameter selection relying on experience in existing SMPC spacecraft avoidance control technology, resulting in insufficient overall performance and adaptability, and difficulty in meeting the autonomous avoidance requirements of complex orbits and multi-constraint scenarios.

[0017] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0018] The following section first introduces a reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization, provided in the embodiments of this application.

[0019] This application provides a flowchart illustrating a reinforcement learning-driven, terminal policy optimization-inspired stochastic MPC control method, as shown in the embodiments below. Figure 1 As shown, the reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization may include the following steps S110-S140.

[0020] S110. Obtain the spacecraft's system state and control inputs, considering orbital perturbations and environmental uncertainties, and construct a discrete relative dynamics model of the spacecraft. Generate expert demonstration data representing the state-control trajectory set using stochastic model predictive control SMPC. Understandably, defining the discrete relative dynamics model of the spacecraft (the controlled object) and the form of uncertainties / perturbations serves as the common foundation for subsequent expert demonstration data generation, reinforcement learning (RL) environment interaction, and SMPC prediction. Based on the discrete relative dynamics model, traditional stochastic model predictive control SMPC generates expert demonstration data, providing training samples for offline reinforcement learning and solving the problem of online trial-and-error sampling in aerospace scenarios.

[0021] S120. Based on the system state and control input, the state and actions of the reinforcement learning agent are defined; and an immediate reward function is designed to guide the terminal policy to iterate towards the desired control behavior. It is understood that this application delineates the state space and action space of the reinforcement learning RL agent based on the system state and control input, which are the input-output boundaries of the terminal policy. When designing the immediate reward function, rewards can be designed around mission objectives such as spacecraft debris avoidance, control accuracy, and energy consumption to guide the terminal policy to iterate towards the desired control behavior, which directly determines the performance of the trained policy.

[0022] S130. Construct a parameterized terminal policy. Based on expert demonstration data and an immediate reward function, train the terminal policy using an offline reinforcement learning algorithm. After the terminal policy training converges, determine the reinforcement learning state in the prediction time domain and generate the corresponding reference control input. Sort the reference control input to obtain the initial reference control sequence. In essence, after determining the policy network structure, by combining expert demonstration data, reinforcement learning states and actions, and setting the training objective using the immediate reward function, the reference control input (terminal control policy) can be trained and the initial reference control sequence can be output after sorting. This is the foundation for online stochastic model predictive control based on terminal policy heuristics.

[0023] S140. A stochastic model predictive control (SMPC) framework is adopted. An initial reference control sequence is explicitly incorporated into the constructed objective function, and the current control quantity is obtained through optimization. This process is then continuously refined to predict and optimize the next time step, achieving closed-loop control for spacecraft space debris avoidance. In essence, after completing offline reinforcement learning training of the terminal strategy and generating an initial reference control sequence, the online control phase employs the stochastic model predictive control (SMPC) framework. The initial reference control sequence is invoked and incorporated into the objective function to form the final stochastic optimization problem. Solving this problem yields the current control quantity, and the process is then continuously refined to predict and optimize the next time step, thus achieving closed-loop debris avoidance control for the spacecraft.

[0024] The above are specific implementation methods of a reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization, provided in the embodiments of this application. Please refer to them as well. Figures 1-3 This application constructs a discrete relative dynamics model of the spacecraft by considering orbital perturbations and environmental uncertainties, making the control strategy design more in line with the actual dynamics. By using stochastic model predictive control (SMPC) to generate professional and reasonable expert demonstration data, it provides data support for offline reinforcement learning training of terminal strategies. At the same time, by defining the state and action of reinforcement learning and designing an instant reward function, it accurately guides the terminal strategy to iterate towards the expected control behavior of spacecraft space debris avoidance, effectively avoiding the blindness of strategy training.

[0025] Furthermore, this application constructs a parameterized terminal strategy and conducts offline reinforcement learning training by combining expert demonstration data and an immediate reward function, thereby achieving efficient convergence of the terminal strategy. Based on this, reference control inputs can be generated and sorted to obtain an initial reference control sequence. This initial reference control sequence is explicitly introduced into the objective function of the SMPC framework for optimization, which can effectively reduce the optimization search range and improve the efficiency and specificity of solving the current control variables. Combined with the rolling propulsion characteristics of SMPC, closed-loop control for spacecraft space debris avoidance can be achieved.

[0026] Based on this, this application fully leverages the advantages of reinforcement learning in policy iterative optimization and adaptive generation of control references, as well as the technical characteristics of SMPC in handling stochastic uncertainties and rolling optimization closed-loop control, by integrating reinforcement learning and stochastic model predictive control. This makes spacecraft control more adaptable and robust in dealing with orbital perturbations and environmental uncertainties, and significantly improves the overall control performance of spacecraft space debris avoidance.

[0027] In some embodiments, the aforementioned acquisition of the spacecraft's system state and control inputs, considering orbital perturbations and environmental uncertainties, constructs a discrete relative dynamics model of the spacecraft; expert demonstration data representing the state-control trajectory set is generated through stochastic model predictive control SMPC, i.e., the aforementioned S110 may specifically include the following steps: S210, Obtaining the spacecraft's timeline System status and control input System status State quantities related to collision risk, including the relative position and relative velocity of the spacecraft; control inputs. This includes the thrust or velocity increment of orbital maneuvers; S220. Considering orbital perturbations and environmental uncertainties, a discrete relative dynamics model is constructed; the discrete relative dynamics model is used to describe the situation under control input. With disturbance Under its influence, the system state changes from Evolved to The relationship is such that it satisfies the expression: In the formula, Let the spacecraft's discrete-time relative motion dynamics function be the perturbation. This indicates process noise or unmodeled disturbance. It follows a known or estimable probability distribution; S230. Establish a trajectory and trajectory covariance model for space debris based on monitoring data, which is used to calculate the collision probability under different control sequences in a simulation environment. S240. Based on the discrete relative dynamics model, a general stochastic predictive controller is constructed. Given an initial state and uncertainties, the avoidance optimization problem is repeatedly solved to obtain a set of approximately optimal state-control trajectories. In the formula, This indicates that the predictive control of the ordinary stochastic model is in the first... The state generated at a given moment This indicates that the control input is predicted by a common stochastic model. The termination step size for a single simulation or a single evasion task; S250, the deterministic state-control trajectory set is expert demonstration data for offline reinforcement learning.

[0028] In the embodiments of this application, it should be noted that stochastic model predictive control (SMPC) is an extension of model predictive control (MPC) in systems with stochastic uncertainties. Its core is to integrate stochastic optimization ideas, treat uncertainties such as model errors and external disturbances in the system as random variables, replace the hard constraints of traditional robust control with probabilistic constraints, and achieve a balance between system performance and constraint satisfaction probability during the rolling optimization process. It is suitable for various complex dynamic systems with random disturbances.

[0029] Understandably, this application accurately acquires the system state and control input related to spacecraft collision risk, and constructs a discrete relative dynamics model that closely reflects the actual operating state by combining orbital perturbations and environmental uncertainties. This model accurately characterizes the relationship between control inputs, disturbances, and system state evolution. Simultaneously, it establishes a space debris orbit and orbital covariance model based on monitoring data, which can effectively calculate the collision probability under different control sequences, providing a crucial basis for avoidance optimization. Furthermore, based on the discrete relative dynamics model, it constructs a general stochastic model predictive controller to solve the avoidance optimization problem under given initial states and uncertainties, obtaining a near-optimal set of state-control trajectories as expert demonstration data for offline reinforcement learning. This ensures the expert demonstration data is generated based on a realistic dynamics model and collision probability calculation logic, guaranteeing the data's rationality and relevance. It also provides high-quality, real-world space debris avoidance samples for subsequent offline reinforcement learning training of terminal strategies, providing a foundation for accurate training.

[0030] In one example, the state of reinforcement learning is defined as: In the formula, For the first The reinforcement learning state at discrete time points; It represents the orbital covariance matrix or other uncertainty parameters associated with space debris and is used to characterize the orbital prediction error of debris; This represents the number of discrete time steps remaining from the current moment until the evasion task terminates.

[0031] The action definition in reinforcement learning is continuous control input: In the formula, For the first The reinforcement learning actions of each step directly correspond to the control inputs of the spacecraft. .

[0032] The instant reward function satisfies the expression: ; In the formula, For the first Instant rewards for each step; A positive weighting coefficient is used to balance the importance of collision risk, propellant consumption, and mimicking expert behavior. Indicates the system state Apply control input The collision probability is estimated in time. Used to characterize the collision risk of the current decision; A quadratic form used to control the input size, used to measure thrust level or propellant consumption; This quadratic form represents the deviation between the current control input and the control input predicted by a normal stochastic model, and is used to encourage policy learning to approach the expert solution.

[0033] In some embodiments, the aforementioned construction of a parameterized terminal policy involves training the terminal policy using an offline reinforcement learning algorithm based on expert demonstration data and an immediate reward function. After the terminal policy training converges, the reinforcement learning state in the prediction time domain is determined and a corresponding reference control input is generated. The reference control input is then sorted to obtain an initial reference control sequence. Specifically, the aforementioned S130 may include the following steps: S310, Constructing parameterized terminal strategies: In the formula, The policy function takes the reinforcement learning state as input. The output is the first... Step control input ; This is the vector of policy parameters to be trained; S320, Defines the cumulative expected reward within the task time domain: In the formula, For the policy parameter vector The cumulative expected reward; It is a mathematical expectation operator, used to statistically average the uncertainties of random perturbations, state-action trajectories, and collision events; For instant rewards, The step size for task termination; S330. Using offline reinforcement learning algorithms on expert demonstration data and simulation environments, the policy parameter vector is processed. Perform iterative updates to maximize the cumulative expected reward. Training terminal strategies; S340. After the terminal policy training converges, the terminal control policy in the terminal phase of the task avoidance is obtained. and terminal control strategy Fixed, serving as a terminal policy heuristic module; S350. Based on the current state estimate and uncertainty prediction, the reinforcement learning state in the prediction time domain is recursively obtained. And generate the corresponding reference control input: ; In the formula, Indicates the length of the prediction time domain. Indicates the current moment For the future The reference control input generated at each time point The superscript 0 in the text indicates a reference value; Indicates at time The predicted state is obtained by recursively calculating the current state estimate, fragment uncertainty, and remaining task time. S360. Arrange the reference control inputs in chronological order to obtain the initial reference control sequence in the prediction time domain: In the formula, For at any time The initial reference control sequence generated by the terminal policy heuristic module, include An initial reference control input; the initial reference control sequence is used as a reference control term in the objective function during stochastic model predictive control optimization.

[0034] In the embodiments of this application, please refer to Figure 2Understandably, this application constructs a parameterized terminal strategy and defines a cumulative expected reward that includes statistical averages of various uncertainties. Relying on expert demonstration data and a simulation environment, it uses an offline reinforcement learning algorithm to iteratively update the strategy parameters to maximize the cumulative expected reward, achieving accurate training and convergence of the terminal strategy. The converged terminal strategy is then fixed as the terminal strategy heuristic module, providing reliable strategy support for subsequent control. Simultaneously, this application derives the reinforcement learning state in the prediction time domain based on the current state estimation and uncertainty prediction recursion, generating corresponding reference control inputs and sorting them chronologically to obtain an initial reference control sequence. This ensures that the terminal strategy training relies on high-quality expert demonstration data and a clearly guided reward mechanism, guaranteeing the strategy's effectiveness and adaptability to various uncertainties in spacecraft avoidance scenarios. Furthermore, the terminal strategy heuristic module accurately generates an initial reference control sequence matching the prediction time domain, providing reference control terms that fit actual avoidance needs for the subsequent stochastic model predictive control optimization process. This effectively reduces the blindness of subsequent optimization solutions and improves the efficiency and targeting of overall control optimization.

[0035] In some embodiments, the aforementioned Stochastic Model Predictive Control (SMPC) framework explicitly introduces an initial reference control sequence into the constructed objective function, solves and optimizes it to obtain the current control quantity, and continuously advances the prediction and optimization for the next time step to achieve closed-loop control for spacecraft space debris avoidance. Specifically, S140 may include the following steps: S410. Based on the current state estimate and the orbital covariance of space debris, the relative distance between the spacecraft and debris in the prediction time domain is predicted, and collision probability constraints are constructed. S420, at the time The predicted time domain length is Initial reference control sequence ; S430. In the prediction time domain, using the predicted state and control input as decision variables, construct a stochastic model for predictive control optimization. S440, Initial reference control sequence Explicitly incorporate the objective function of the stochastic model predictive control; S450. Construct the state prediction equation and apply target constraints, including state constraints, control constraints, and collision probability constraints. S460. A numerical optimization algorithm is used to solve the problem and obtain the target control sequence at the current time. : In the formula, include One target control input; S470, According to the rolling time domain strategy, It acts on the spacecraft's propulsion system to execute the current evasive maneuver; S480, Order The state estimate and fragment information are updated, the reference control sequence is regenerated, and a new stochastic model is constructed to predict the control problem. The process is repeated until the avoidance task is completed or the collision risk is reduced to below the preset safety threshold.

[0036] In the embodiments of this application, please refer to Figure 3 Understandably, this application first combines the current state estimate with the relative distance prediction based on the orbital covariance of space debris and constructs collision probability constraints to define the core safety boundary for avoidance control optimization. Then, at the corresponding moment, it calls the initial reference control sequence with the predicted time domain length and explicitly integrates it into the SMPC objective function. At the same time, it constructs a stochastic model and builds a complete constraint system containing state, control, and collision probability constraints. It relies on numerical optimization algorithms to accurately solve for the target control sequence and follows a rolling time domain strategy to execute only the control quantity at the current moment. It also iteratively advances by updating the state estimate and debris information in real time, regenerating the reference control sequence, and constructing the optimization problem. This not only provides the optimization solution of the SMPC framework with an initial reference basis that fits the actual avoidance requirements, effectively reducing the optimization search range, improving the solution efficiency and control quantity accuracy, but also strictly ensures the safety of the avoidance process through the application of multi-dimensional constraints. Furthermore, through the closed-loop design of rolling advancement and cyclic iteration, it realizes the dynamic adjustment and continuous optimization of spacecraft space debris avoidance control, ensuring that it can adapt to state and environmental changes in real time during the avoidance mission, until the collision risk is reduced to the safety threshold or the avoidance mission is completed.

[0037] This application adopts a fusion mechanism of terminal strategy heuristic and SMPC rolling solution. In each control cycle, an initial reference control sequence is first generated, and then the sequence is used as the initial control and regularization center for SMPC optimization to obtain the optimal control sequence that satisfies the probabilistic safety constraints. Only the first control variable is executed. By repeating the above process in a rolling manner, the quality of the initial value solution is significantly improved without changing the SMPC constraint structure, thereby improving the solution efficiency, feasibility and fuel economy of collision avoidance.

[0038] In one example, the objective function of the stochastic model satisfies the expression: ; In the formula, To be at the current sampling time Predictive control performance indicators; It is a mathematical expectation operator used to perform statistical averaging on random disturbances and the uncertainties they cause. For at any time For the future The predicted value of the spacecraft's state at a given moment; For the corresponding The desired state at any given moment; To predict the end state in the time domain, The desired terminal state; This indicates the matrix transpose. This is a quadratic penalty term for state deviation. It is a symmetric positive definite state weight matrix; A quadratic penalty term to control the input size, It is a symmetric positive definite control weight matrix used to reflect thrust level or propellant consumption; This is a quadratic penalty term for the deviation between the control input and the reference control input. The weight matrix is ​​a symmetric positive definite matrix, used to encourage the optimization solution to be as close as possible to the reference control sequence generated by the terminal policy, while satisfying the constraints. Indicates the current moment For the future Control inputs generated at each moment; Indicates the current moment For the future Reference control inputs generated at each time point; This is a quadratic penalty term for the terminal state deviation. It is a symmetric positive definite terminal state weight matrix used to ensure that the spacecraft is in a safe and near-desired orbital state at the end of the avoidance maneuver.

[0039] In one example, the state prediction equation satisfies the expression: ; In the formula, This refers to the random perturbation or uncertainty of the prediction step; This is the discrete-time relative motion dynamics function of the spacecraft; For at any time For the future The predicted value of the spacecraft's state at a given moment.

[0040] The objective constraint satisfies the following expression: ; In the formula, This is a set of state constraints used to define the physical boundaries, including orbital position and velocity. The set of control constraints is used to limit the magnitude, direction, and amplitude of thrust in a single maneuver. This indicates that the spacecraft and space debris are in the predicted state. The relative distance; A pre-set safe distance threshold based on task requirements; For probability operators; This is the maximum permissible collision probability threshold, used to ensure that the collision risk at each step in the prediction time domain does not exceed the specified upper limit.

[0041] In some embodiments, this application provides a reinforcement learning-driven, terminal policy optimization-inspired stochastic MPC control system 500, such as... Figure 4 As shown, the random MPC control system 500 may include the following modules: The data generation module 510 is used to acquire the system state and control input of the spacecraft, consider orbital perturbations and environmental uncertainties, construct a discrete relative dynamics model of the spacecraft, and generate expert demonstration data representing the state-control trajectory set through stochastic model predictive control SMPC. Define design module 520 to define the state and actions of reinforcement learning based on system state and control input; and design an immediate reward function to guide the terminal policy to iterate towards the desired control behavior; The training and sorting module 530 is used to construct a parameterized terminal policy. Based on expert demonstration data and an instant reward function, it uses an offline reinforcement learning algorithm to train the terminal policy. After the terminal policy training converges, it determines the reinforcement learning state in the prediction time domain and generates the corresponding reference control input. The reference control input is sorted to obtain the initial reference control sequence. The prediction and optimization module 540 is used to adopt the stochastic model predictive control (SMPC) framework. It explicitly introduces the initial reference control sequence into the constructed objective function, solves and optimizes it to obtain the current control quantity, and continuously advances the prediction and optimization for the next moment to achieve closed-loop control for spacecraft space debris avoidance.

[0042] According to embodiments of this application, any and multiple modules among the data generation module 510, definition and design module 520, training and ranking module 530, and prediction and optimization module 540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module.

[0043] Figure 4 Each module in the system shown has the function of implementing each step in the aforementioned reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization, and can achieve its corresponding technical effect. For the sake of brevity, it will not be elaborated here.

[0044] In some embodiments, this application provides an electronic device, the structural schematic of which is shown below. Figure 5 As shown.

[0045] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0046] Specifically, the processor 610 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0047] Memory 620 may include mass storage for data or instructions. For example, and not limitingly, memory 620 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 620 may include removable or non-removable (or fixed) media. Where appropriate, memory 620 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 620 is non-volatile solid-state memory.

[0048] Memory 620 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described by any of the reinforcement learning-driven terminal policy optimization-inspired stochastic MPC control methods in the above embodiments.

[0049] The processor 610 reads and executes computer program instructions stored in the memory 620 to implement any of the reinforcement learning-driven terminal policy optimization-inspired stochastic MPC control methods in the above embodiments.

[0050] In one example, the electronic device may also include a communication interface 630 and a bus 600. Wherein, such as Figure 5 As shown, the processor 610, memory 620, and communication interface 630 are connected via bus 600 and communicate with each other.

[0051] The communication interface 630 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0052] Bus 600 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 600 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0053] Furthermore, in conjunction with the reinforcement learning-driven terminal policy optimization-inspired stochastic MPC control method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any one of the reinforcement learning-driven terminal policy optimization-inspired stochastic MPC control methods in the above embodiments.

[0054] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0055] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0056] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0057] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization, characterized in that, include: Obtain the system state and control input of the spacecraft, and construct a discrete relative dynamics model of the spacecraft, taking into account orbital perturbations and environmental uncertainties. Expert demonstration data representing the state-control trajectory set is generated by predicting the SMPC using a stochastic model; Based on the system state and the control input, define the state and actions for reinforcement learning; And an instant reward function is designed to guide the terminal strategy to iterate towards the desired controlled behavior; A parameterized terminal strategy is constructed. Based on the expert demonstration data and the instant reward function, the terminal strategy is trained using an offline reinforcement learning algorithm. After the terminal strategy training converges, the reinforcement learning state in the prediction time domain is determined and the corresponding reference control input is generated. The reference control input is sorted to obtain the initial reference control sequence. The Stochastic Model Predictive Control (SMPC) framework is adopted. The initial reference control sequence is explicitly introduced into the constructed objective function, and the current control quantity is obtained by solving and optimizing. The prediction and optimization of the next time step are rolled forward to achieve closed-loop control for spacecraft space debris avoidance.

2. The reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in claim 1, characterized in that, The system state and control input of the spacecraft are obtained, and a discrete relative dynamics model of the spacecraft is constructed, taking into account orbital perturbations and environmental uncertainties. Expert demonstration data representing the state-control trajectory set was generated by predicting the SMPC using a stochastic model, including: Obtain the spacecraft's time System status and control input The system state The control inputs include state quantities related to collision risk, such as the relative position and relative velocity of the spacecraft; This includes the thrust or velocity increment of orbital maneuvers; Considering orbital perturbations and environmental uncertainties, a discrete relative dynamics model is constructed; this discrete relative dynamics model is used to describe the situation under control input. With disturbance Under its influence, the system state changes from Evolved to The relationship is such that it satisfies the expression: In the formula, Let the spacecraft's discrete-time relative motion dynamics function be the perturbation. This indicates process noise or unmodeled disturbance. It follows a known or estimable probability distribution; Based on monitoring data, a model of the orbit and orbital covariance of space debris is established to calculate the collision probability under different control sequences in a simulation environment. Based on the aforementioned discrete relative dynamics model, a general stochastic model predictive controller is constructed. Given an initial state and uncertainties, the avoidance optimization problem is repeatedly solved to obtain a set of approximately optimal state-control trajectories. In the formula, This indicates that the predictive control of the ordinary stochastic model is in the first... The state generated at a given moment This indicates that the control input is predicted by a common stochastic model. The termination step size for a single simulation or a single evasion task; The state-control trajectory set is determined to be expert demonstration data for offline reinforcement learning.

3. The reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in claim 2, characterized in that, The state definition in reinforcement learning is: In the formula, For the first The reinforcement learning state at each discrete time step; It represents the orbital covariance matrix or other uncertainty parameters associated with space debris and is used to characterize the orbital prediction error of debris; This represents the number of discrete time steps remaining from the current moment until the evasion mission terminates; The action definition in reinforcement learning is continuous control input: In the formula, For the first The reinforcement learning actions of each step directly correspond to the control inputs of the spacecraft. ; The instant reward function satisfies the expression: ； In the formula, For the first Instant rewards for each step; A positive weighting coefficient is used to balance the importance of collision risk, propellant consumption, and mimicking expert behavior. Indicates the system state Apply control input The collision probability is estimated in time. Used to characterize the collision risk of the current decision; A quadratic form used to control the input size, used to measure thrust level or propellant consumption; This quadratic form represents the deviation between the current control input and the control input predicted by a normal stochastic model, and is used to encourage policy learning to approach the expert solution.

4. The reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in claim 3, characterized in that, The parameterized terminal strategy is constructed by training the terminal strategy using an offline reinforcement learning algorithm based on the expert demonstration data and the instant reward function. After the terminal strategy training converges, the reinforcement learning state in the prediction time domain is determined and the corresponding reference control input is generated. The reference control input is sorted to obtain an initial reference control sequence, including: Constructing parameterized terminal strategies: In the formula, The policy function takes the reinforcement learning state as input. The output is the first... Step control input ; This is the policy parameter vector to be trained; Defined cumulative expected reward within the task time domain: In the formula, For the policy parameter vector The cumulative expected reward below; It is a mathematical expectation operator, used to statistically average the uncertainties of random perturbations, state-action trajectories, and collision events; For instant rewards, The step size for terminating the task; The policy parameter vector is processed using an offline reinforcement learning algorithm on expert demonstration data and a simulation environment. Perform iterative updates to maximize the cumulative expected reward. Training terminal strategies; After the terminal policy training converges, the terminal control policy for the terminal phase of the task avoidance is obtained. and the terminal control strategy Fixed, serving as a terminal policy heuristic module; The reinforcement learning state in the prediction time domain is obtained recursively based on the current state estimate and uncertainty prediction. And generate the corresponding reference control input: ； In the formula, Indicates the length of the prediction time domain. Indicates the current moment For the future The reference control input generated at each time point The superscript 0 in the text indicates a reference value; Indicates at time The predicted state is obtained by recursively calculating the current state estimate, fragment uncertainty, and remaining task time. Arrange the reference control inputs in chronological order to obtain the initial reference control sequence in the prediction time domain: In the formula, For at any time The initial reference control sequence generated by the terminal policy heuristic module, include An initial reference control input; the initial reference control sequence is used as a reference control term in the objective function during stochastic model predictive control optimization.

5. The reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in claim 3, characterized in that, The aforementioned Stochastic Model Predictive Control (SMPC) framework explicitly incorporates the initial reference control sequence into the constructed objective function, solves for optimization to obtain the current control quantity, and continuously advances the prediction and optimization for the next time step to achieve closed-loop control for spacecraft space debris avoidance, including: Based on the current state estimate and the orbital covariance of space debris, the relative distance between the spacecraft and debris in the prediction time domain is predicted, and collision probability constraints are constructed. At any moment The predicted time domain length is Initial reference control sequence ; In the prediction time domain, a stochastic model is constructed using the predicted state and control input as decision variables to perform predictive control optimization. The initial reference control sequence Explicitly incorporate the objective function of the stochastic model predictive control; Construct a state prediction equation and apply target constraints, including state constraints, control constraints, and collision probability constraints. The target control sequence at the current time is obtained by using a numerical optimization algorithm to solve the problem. : In the formula, include One target control input; According to the rolling time-domain strategy, It acts on the spacecraft's propulsion system to execute the current evasive maneuver; make The state estimate and fragmentation information are updated, the reference control sequence is regenerated, and a new stochastic model is constructed to predict the control problem. The process is repeated until the avoidance task is completed or the collision risk is reduced to below the preset safety threshold.

6. The reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in claim 5, characterized in that, The objective function of the stochastic model satisfies the expression: ； In the formula, To be at the current sampling time Predictive control performance indicators; It is a mathematical expectation operator used to perform statistical averaging on random disturbances and the uncertainties they cause. For at any time For the future The predicted value of the spacecraft's state at a given moment; For the corresponding The desired state at any given moment; To predict the end state in the time domain, The desired terminal state; Indicates matrix transpose; This is a quadratic penalty term for state deviation. It is a symmetric positive definite state weight matrix; A quadratic penalty term to control the input size. It is a symmetric positive definite control weight matrix used to reflect thrust level or propellant consumption; This is a quadratic penalty term for the deviation between the control input and the reference control input. The weight matrix is ​​a symmetric positive definite matrix, used to encourage the optimization solution to be as close as possible to the reference control sequence generated by the terminal policy, while satisfying the constraints. Indicates the current moment For the future Control inputs generated at each moment; Indicates the current moment For the future Reference control inputs generated at each time point; This is a quadratic penalty term for the terminal state deviation. It is a symmetric positive definite terminal state weight matrix used to ensure that the spacecraft is in a safe and near-desired orbital state at the end of the avoidance maneuver.

7. The reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in claim 6, characterized in that, The state prediction equation satisfies the expression: ； In the formula, This represents the random perturbation or uncertainty of the prediction step. This is the discrete-time relative motion dynamics function of the spacecraft; For at any time For the future The predicted value of the spacecraft's state at a given moment; The objective constraint satisfies the expression: ； In the formula, This is a set of state constraints used to define the physical boundaries, including orbital position and velocity. The set of control constraints is used to limit the magnitude, direction, and amplitude of thrust in a single maneuver. Indicates the spacecraft and space debris are in the predicted state. The relative distance; A pre-set safe distance threshold based on task requirements; For probability operators; This is the maximum permissible collision probability threshold, used to ensure that the collision risk at each step in the prediction time domain does not exceed the specified upper limit.

8. A stochastic MPC control system inspired by reinforcement learning-driven terminal policy optimization, characterized in that, include: The data generation module is used to acquire the system state and control input of the spacecraft, and to construct a discrete relative dynamics model of the spacecraft, taking into account orbital perturbations and environmental uncertainties. Expert demonstration data representing the state-control trajectory set is generated by predicting the SMPC using a stochastic model; Define a design module for defining the state and actions of reinforcement learning based on the system state and the control input; And an instant reward function is designed to guide the terminal strategy to iterate towards the desired controlled behavior; The training and ranking module is used to construct a parameterized terminal policy. Based on the expert demonstration data and the instant reward function, the terminal policy is trained using an offline reinforcement learning algorithm. After the terminal policy training converges, the reinforcement learning state in the prediction time domain is determined and the corresponding reference control input is generated. The reference control input is ranked to obtain the initial reference control sequence. The prediction and optimization module is used to employ the stochastic model predictive control (SMPC) framework. It explicitly incorporates the initial reference control sequence into the constructed objective function, solves and optimizes it to obtain the current control quantity, and continuously advances the prediction and optimization for the next time step to achieve closed-loop control for spacecraft space debris avoidance.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements a reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the reinforcement learning-driven stochastic MPC control method inspired by terminal policy optimization as described in any one of claims 1 to 7.

Images