A spacecraft rendezvous and docking control method and system based on safety reinforcement learning

Abstract

The present invention discloses a spacecraft rendezvous and docking control method and system based on safety reinforcement learning, the method comprising: collecting trajectory data in the process of interaction between a safety reinforcement learning agent and a spacecraft rendezvous and docking simulation environment; the safety reinforcement learning agent has a safety reinforcement learning model, including: an action network and a value network; storing the trajectory data in a trajectory storage pool in the form of tuples; using the trajectory data in the trajectory storage pool as input of the safety reinforcement learning model, constructing a Lagrangian function to perform reward and constraint joint optimization on the network model parameters of the safety reinforcement learning model; after iterative training, obtaining an optimized safety reinforcement learning model; inputting the states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scene into the optimized safety reinforcement learning model, and outputting the control action of the tracking spacecraft. The method has computational efficiency, interference robustness and constraint safety.

Description

Technical Field

[0001] The present invention relates to the technical field of spacecraft rendezvous and docking, and more specifically to a spacecraft rendezvous and docking control method and system based on safety reinforcement learning. Background Art

[0002] At present, as the degree of human utilization of space resources continues to increase, the rendezvous and docking of spacecraft has become an important research topic and the research basis of many space missions. Spacecraft rendezvous and docking refers to the process in which two spacecraft flying in orbit, through precise control of the tracking star, realize the rendezvous of the tracking star and the target star in space, and finally connect to each other into a single, larger structure. Spacecraft rendezvous and docking is of vital importance in the field of aerospace. It is not only a key technology for building and maintaining space stations and implementing long-term space exploration missions (such as missions to the moon or Mars), but also the basis for on-orbit replenishment, spacecraft upgrades and repairs. Through rendezvous and docking, multiple small spacecraft can be combined into a larger structure, improving the flexibility and capabilities of space missions. In addition, the development of rendezvous and docking technology has also promoted the advancement of aerospace technology, including automated control, navigation and propulsion systems, and has played an irreplaceable role in enhancing human beings' persistent presence in space and deep space exploration capabilities.

[0003] Close rendezvous requires close cooperation between the two spacecraft, including complex guidance systems, and even the participation of astronauts in the control loop to ensure the smooth progress of the mission. For future space missions, small spacecraft that can autonomously reach non-cooperative targets usually do not have strong maneuvering capabilities, and their onboard computers have limited computing power. In order to meet the needs of approaching targets, it is necessary to study guidance algorithms with high computational efficiency and high robustness.

[0004] In order to complete the process of spacecraft rendezvous and docking, tracking the spacecraft when approaching the target requires considering multiple process constraints. Faced with the multi-constraint problem, model predictive control provides a solution. Model predictive control is a favorable method for handling constraint systems under the framework of optimization theory, reflecting the development of constraint control research from feedback stabilization to system optimization.

[0005] In addition, with the improvement of computing power and the development of related algorithms, reinforcement learning is increasingly being used to solve complex practical problems. For example, patent document CN113724260A A satellite capture method based on deep reinforcement learning is to solve the position of the target satellite through a convolutional neural network; perform semantic segmentation on the image through a feature fusion network to find the position of the docking ring; and input the above information into the reinforcement learning network to obtain a satellite control plan using reinforcement learning.

[0006] However, both the model predictive control method and reinforcement learning control have some shortcomings:

[0007] 1. Model predictive control methods mainly exist:

[0008] 1) High computational complexity: Model predictive control requires solving an optimization problem in each control cycle, which is usually a nonlinear programming problem, especially when considering spacecraft dynamics and constraints. For highly dynamic, multi-constrained space missions, the optimization problem is usually extremely complex and requires a lot of computing resources and time, which is difficult to achieve under the limited spacecraft computing power and urgent real-time control requirements.

[0009] 2) Poor robustness: The capability of the model predictive control method is highly dependent on the accuracy of the mathematical model, so it has poor robustness to model errors and external disturbances. In the spacecraft rendezvous and docking environment, due to unpredictable factors such as the microgravity environment, the nonlinear characteristics of spacecraft dynamics, and unpredictable environmental disturbances (such as space debris, solar radiation pressure, etc.), the robustness of the model predictive control method is poor, resulting in differences between actual behavior and the predicted model, affecting the effectiveness of the control strategy.

[0010] 2. Reinforcement learning control method:

[0011] Reinforcement learning is a machine learning method that allows the algorithm to try different strategies in a simulated environment and learn based on the results (rewards or penalties) to ultimately find the optimal strategy for completing a specific task. However, the reinforcement learning method has the following main shortcomings.

[0012] 1) Poor safety: In high-risk spacecraft operation control, ensuring that the control strategy meets safety constraints is key. However, the optimization goal of traditional reinforcement learning methods is only to maximize rewards. The existence of safety constraints is not considered in the optimization goal, resulting in the trained strategy to maximize rewards as the only goal, which leads to violations of safety constraints. Reinforcement learning strategies may take unsafe or non-optimal actions when exploring effective behaviors, especially in the early stages of learning. In critical tasks such as spacecraft rendezvous and docking, this uncertainty will lead to unacceptable risks.

[0013] There are deficiencies in the model predictive control method and / or reinforcement learning control used in the spacecraft rendezvous and docking control process in the prior art. Therefore, in view of future space missions, considering the high value and fragility of spacecraft, a control method that can improve the safety of spacecraft rendezvous and docking missions is urgently needed, which is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0014] In view of this, the present invention provides a spacecraft rendezvous and docking control method and system based on safety reinforcement learning, which can solve the deficiencies mentioned in the above background technology. During the training process, safety reinforcement learning discovers data-driven control strategies by learning from a large number of simulations or actual operations. These strategies can directly map from system states to control inputs, without having to solve complex optimization problems in each control step, and convert complex optimization problems into forward network inferences, greatly reducing the computational complexity; in safety reinforcement learning, by exploring different strategies and learning based on reward (or penalty) signals, it is possible to adapt to changes in the environment and unknown disturbances. This learning and adaptability can enhance the robustness of the control strategy in the face of model inaccuracies or external disturbances. In addition, the optimization goals of safety reinforcement learning include maximizing rewards and minimizing constraints, so as to ensure that safety constraints are always met from training to deployment, thereby greatly improving the safety of the control algorithm.

[0015] In order to achieve the above object, the present invention adopts the following technical solution:

[0016] In a first aspect, an embodiment of the present invention provides a spacecraft rendezvous and docking control method based on safety reinforcement learning, comprising the following steps:

[0017] S10, collecting trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; the safety reinforcement learning agent has a safety reinforcement learning model, and the safety reinforcement learning model includes: an action network and a value network;

[0018] S20, storing the trajectory data in a trajectory storage pool in the form of tuples;

[0019] S30, using the trajectory data in the trajectory storage pool as the input of the safety reinforcement learning model, constructing a Lagrangian function to perform reward and constraint joint optimization on the network model parameters of the safety reinforcement learning model;

[0020] S40, iterative training is completed, and the optimized security reinforcement learning model is obtained;

[0021] S50, inputting the states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario into the optimized safety reinforcement learning model, and outputting the control action of the tracking spacecraft.

[0022] Furthermore, the step S10 includes:

[0023] S101, constructing a simulation environment and a safety reinforcement learning agent for simulating spacecraft rendezvous and docking; the safety reinforcement learning agent has a safety reinforcement learning model, and the safety reinforcement learning model includes: an action network and a value network;

[0024] S102, setting the states s of the tracking spacecraft and the target spacecraft in the simulation environment, and inputting them into the action network;

[0025] S103, the action network outputs the tracking spacecraft action a according to the state s; the value network outputs the estimated return reward of taking action a under the current state s;

[0026] S104, generating a state s at the next moment according to the tracking spacecraft action a, and generating rewards and constraints at the same time;

[0027] S105. During the simulation process, continue to loop steps S102 to S104 to collect trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; until the simulation ends.

[0028] Furthermore, the reward in step S104 is expressed as follows:

[0029] At time t, the reward r received by the tracking spacecraft is:

[0030]

[0031] in, are the distances between the tracking spacecraft and the target spacecraft at time t-1 and time t respectively.

[0032] Furthermore, the constraints in step S104 are expressed as follows:

[0033]

[0034] Where ω is the angular velocity of the current tracking spacecraft, is the set maximum angular velocity; when the angular velocity is less than the maximum value, the constraint is 0; when the angular velocity is greater than or equal to the maximum value, the tracking spacecraft receives a constraint of 1.

[0035] Furthermore, the step S30 includes:

[0036] S301, determining the optimization goal of safety reinforcement learning, including: maximizing the reward received by the tracking spacecraft and satisfying the constraint conditions received by the tracking spacecraft;

[0037] S302, constructing a Lagrangian function;

[0038] S303, sampling and storing trajectory data {s, a, r, c, s'} from the trajectory storage pool, respectively representing the state at the current moment, the action at the current moment, the reward received at the current moment, the constraint received at the current moment and the state at the next moment;

[0039] S304, using the gradient descent method to update the reward value network and the constraint value network; using the gradient ascent method to update the action network;

[0040] S305, using the gradient descent method to update the Lagrange multiplier;

[0041] S306. Repeat steps S304 to S305 until all sampled and stored trajectory data are traversed; the joint optimization of reward and constraint is completed.

[0042] Furthermore, the mathematical expression formula of step S301 is as follows:

[0043]

[0044] in:

[0045]

[0046] E means to find the mathematical expectation; Indicates the desired action to follow is obtained by the action network π; Indicates that the state s at time t=0 comes from the initial state distribution μ; Indicates that except for the time t=0, the state s at other times is determined by the state transition probability Obtain; b is the upper limit of the constraint; is the expected return reward; γ is the discount factor; is the reward function; is the status of the tracking spacecraft and the target spacecraft at time t; is the execution action of the tracking spacecraft at time t; For the expected constraints, is the constraint function.

[0047] Furthermore, the Lagrangian function of step S302 is expressed as follows:

[0048]

[0049] L(·) is the Lagrangian function; π is the action network; λ is the Lagrangian multiplier.

[0050] Furthermore, in step S304:

[0051] The reward value network loss is expressed as:

[0052]

[0053] The constraint value network loss is expressed as:

[0054]

[0055] V(·) is the mean of the cumulative returns of the value function; is the mean of the cumulative constraints of the value function; r is the reward; c is the constraint;

[0056] The update action network loss is expressed as: .

[0057] Furthermore, in step S305:

[0058] Update the Lagrange multiplier λ, and the loss is expressed as: .

[0059] In a second aspect, an embodiment of the present invention further provides a spacecraft rendezvous and docking control system based on safety reinforcement learning, including:

[0060] An interactive data collection module is used to collect trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; the safety reinforcement learning agent has a safety reinforcement learning model, and the safety reinforcement learning model includes: an action network and a value network;

[0061] A trajectory storage pool module, used for storing the trajectory data in a trajectory storage pool in the form of tuples;

[0062] A safety reinforcement learning optimization module is used to use the trajectory data in the trajectory storage pool as the input of the safety reinforcement learning model, construct a Lagrangian function to perform reward and constraint joint optimization on the network model parameters of the safety reinforcement learning model; after iterative training, the optimized safety reinforcement learning model is obtained;

[0063] The rendezvous and docking control module is used to input the states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario into the optimized safety reinforcement learning model, and output the control action of the tracking spacecraft.

[0064] It can be seen from the above technical solution that compared with the prior art:

[0065] The present invention transforms complex optimization problems into forward network inference by learning from a large amount of data, thereby greatly reducing computational complexity; utilizes exploration-based reinforcement learning methods to improve the robustness of control strategies; and incorporates considerations of safety constraints into the optimization objectives of safety reinforcement learning, so that the intelligent agent can meet safety constraints while maximizing rewards, ultimately obtaining a spacecraft rendezvous and docking control method with computational efficiency, interference robustness, and constraint safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0067] Figure 1 A flow chart of the spacecraft rendezvous and docking control method based on safety reinforcement learning provided by the present invention.

[0068] Figure 2 This is a specific flow chart of step S10 provided by the present invention.

[0069] Figure 3 This is a training flow chart of the security reinforcement learning model provided by the present invention.

[0070] Figure 4 Schematic diagram of the spacecraft rendezvous and docking control system based on safety reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Reference Figure 1 As shown, for future space missions and spacecraft rendezvous and docking scenarios, an embodiment of the present invention discloses a spacecraft rendezvous and docking control method based on safety reinforcement learning, comprising the following steps: S10-S50;

[0073] S10. Collect trajectory data during the interaction between a safety reinforcement learning agent and a spacecraft rendezvous and docking simulation environment; the safety reinforcement learning agent has a safety reinforcement learning model, and the safety reinforcement learning model includes: an action network and a value network.

[0074] The basic model of secure reinforcement learning is the Constrained Markov Decision Process (CMDP). The complete Constrained Markov Decision Process model consists of state s, action a, state transition probability , initial state distribution μ, reward function r(s,a), constraint c(s,a) and constraint upper limit b. The optimization goal of secure reinforcement learning is to maximize the total reward while satisfying the total constraint less than the constraint upper limit b.

[0075] Among them, the spacecraft rendezvous and docking simulation environment is a virtual environment that simulates the rendezvous and docking of spacecraft. By giving the states s of the tracking spacecraft and the target spacecraft and receiving the action a of the tracking spacecraft, the rendezvous and docking process of the tracking spacecraft and the target spacecraft is simulated; the state s refers to the speed and position data of the tracking spacecraft itself at the current moment, as well as the speed and position data of the target spacecraft. In the real rendezvous scenario, the tracking spacecraft's own sensor equipment often obtains its own speed and position, and calculates the speed and position of the target spacecraft. The safe reinforcement learning agent refers to the tracking spacecraft loaded with the safe reinforcement learning model. The safe reinforcement learning model includes an action network π(a|s) and a value network Q(s,a). The action network outputs the tracking spacecraft action a according to the state s, and the value network outputs the expected reward for taking action a under the current state s.

[0076] S20. The trajectory data is stored in a trajectory storage pool in the form of a tuple {s, a, r, c, s'}, which respectively represent the state at the current moment, the action at the current moment, the reward received at the current moment, the constraint received at the current moment, and the state at the next moment; for example, each moment uses 1 second as a time step.

[0077] S30. Use the trajectory data in the trajectory storage pool as the input of the safety reinforcement learning model, and construct a Lagrangian function to jointly optimize the network model parameters of the safety reinforcement learning model by reward and constraint. In this step, the deep neural network parameters in the safety reinforcement learning agent are updated according to the trajectory stored in the trajectory storage pool. Specifically, the constrained optimization problem is solved by constructing a Lagrangian function. The optimization direction is: maximize the reward received by the safety reinforcement learning agent, while ensuring that the constraints received by the safety reinforcement learning agent meet the restrictions to ensure the safety during the docking process.

[0078] S40, iterative training is completed, and the optimized security reinforcement learning model is obtained; during the deployment process, only the trained model is needed.

[0079] S50, inputting the states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario into the optimized safety reinforcement learning model, and outputting the control action of the tracking spacecraft.

[0080] The present invention adopts an optimization method based on secure reinforcement learning, which has computational efficiency, interference robustness, and constraint safety. Secure reinforcement learning is one of the paradigms of machine learning, which is used to describe and solve the problem of how an intelligent agent can achieve maximum rewards while satisfying constraints by learning an optimal strategy in the process of interacting with the environment.

[0081] The following is a detailed description of each of the above steps:

[0082] 1. Step S10 refers to Figure 2 As shown, including: S101~ S105;

[0083] S101, constructing a simulation environment and a safety reinforcement learning agent for simulating spacecraft rendezvous and docking; the safety reinforcement learning agent has a safety reinforcement learning model, and the safety reinforcement learning model includes: an action network and a value network;

[0084] S102, setting the states s of the tracking spacecraft and the target spacecraft in the simulation environment, and inputting them into the action network;

[0085] S103, the action network outputs the tracking spacecraft action a according to the state s; the value network outputs the estimated return reward of taking action a under the current state s;

[0086] S104, generating a state s at the next moment according to the tracking spacecraft action a, and generating rewards and constraints at the same time;

[0087] S105. During the simulation process, continue to loop steps S102 to S104 to collect trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; until the simulation ends.

[0088] The goal of step S10 is to collect trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment.

[0089] The spacecraft rendezvous and docking simulation environment is a simulation environment designed for spacecraft rendezvous and docking scenarios. In this environment, the safety reinforcement learning agent (hereinafter referred to as the agent) will be responsible for tracking the spacecraft. The task is to make the relative position of the tracking spacecraft and the target spacecraft meet the requirements (that is, to complete the rendezvous and docking). The constraint is that the angular velocity of the tracking spacecraft cannot exceed the limit. When the simulation environment is initialized, the agent will receive the state given by the environment. At each subsequent moment, the agent takes action according to the state. After receiving the action, the simulation environment will feedback the state of the agent at the next moment, and give the agent rewards and constraints. The purpose of designing rewards is to guide the agent to complete the goal. At time t, the reward r received by the agent is:

[0090]

[0091] Among them, among them, are the distances between the tracking spacecraft and the target spacecraft at time t-1 and time t, respectively. When the distance between the tracking spacecraft and the target spacecraft is smaller than that at the previous moment, the agent receives a positive reward r. The purpose of designing constraints is to ensure the safety of the action output by the agent, that is, to ensure that the angular velocity of the tracking spacecraft is less than the limit value. The constraint is expressed by the formula:

[0092]

[0093] Where ω is the angular velocity of the current tracking spacecraft, is the maximum angular velocity setting. When the angular velocity is less than the maximum value, the constraint is 0; when the angular velocity is greater than or equal to the maximum value, the agent receives a constraint of 1.

[0094] The secure reinforcement learning agent has a secure reinforcement learning model, including: action network and value network. , characterizes the strategy of the reinforcement learning agent. The input of the strategy function is the current state s, and the output is the probability distribution of the action. The final action a can be obtained by sampling the output action probability distribution. The action network includes components such as a multi-layer perceptron module and an activation function module. The action network processes the state s from the simulation environment through a multi-layer perceptron, extracts relevant information, and finally generates an action after being activated by the activation function;

[0095] Value network, i.e. and The network,consists of a multi-layer perceptron module and an activation function module,,with the input as state s and the output as an estimate of the,corresponding value.

[0096] Value Function , which is used to judge whether the current state of the reinforcement learning agent is good or not. The value function V refers to the agent’s In the case of using the current strategy to interact with the environment, from the state The average of the cumulative returns obtained from Indicates that the agent is in state In the case of using the current strategy to interact with the environment, from the state The mean of the cumulative constraints obtained.

[0097] 2. Track storage pool

[0098] In step S20, the trajectory storage pool stores the data collected from the interaction in the form of a tuple {s, a, r, c, s'} to form a trajectory pool for updating the safe reinforcement learning model of the intelligent agent in the subsequent steps of safe reinforcement learning optimization.

[0099] 3. Security reinforcement learning optimization

[0100] In step S30, the trajectory data in the trajectory storage pool is used as the input of the safety reinforcement learning model. By using the trajectory information, the model parameters in the safety reinforcement learning agent are optimized.

[0101] First, the optimization goal of secure reinforcement learning is divided into two parts: 1. Maximizing the reward received by the agent; 2. Making the constraints received by the agent satisfy the conditions. It can be expressed as:

[0102]

[0103] in:

[0104]

[0105] E means to find the mathematical expectation; Indicates the desired action to follow is obtained by the action network π; Indicates that the state s at time t=0 comes from the initial state distribution μ; Indicates that except for the time t=0, the state s at other times is determined by the state transition probability Obtain; b is the upper limit of the constraint; J(π) is the expected return reward; is the reward function; is the status of the tracking spacecraft and the target spacecraft at time t; is the execution action of the tracking spacecraft at time t; For the expected constraints, is the constraint function. In the equation, γ is the discount factor, which is generally less than 1, and t is the time. In the time series, as t increases, the more distant the reward is, the less impact it has on the current moment.

[0106] The constraint upper limit b specifies the maximum value of the sum of constraints received by the agent during a rendezvous and docking process, and is used to limit the degree of constraint violation during a rendezvous and docking process. For example, when the angular velocity of the spacecraft exceeds the specified maximum value, the force on some structures of the spacecraft will exceed the maximum value allowed by the design, thereby affecting the service life. According to ground tests, when the structure is subjected to a load exceeding the maximum allowable value for b seconds, the spacecraft structure will fail, so the constraint upper limit is set to b.

[0107] The present invention introduces the Lagrangian multiplier method to solve the above constrained optimization problem, balancing the original reward maximization goal and the need to meet safety constraints. The Lagrangian function is constructed as:

[0108]

[0109] After constructing the Lagrangian function, the reward objective and constraint objective are unified into one optimization objective by optimizing the Lagrangian function, so as to maximize the reward while satisfying the constraints. The optimization process is as follows:

[0110] Step 1: Sample the stored trajectory {s,a,r,c,s'} from the trajectory storage pool;

[0111] Step 2: Update the secure reinforcement learning agent;

[0112] Step 2.1: Update the reward value network using gradient descent and Constrained Value Network The loss expressions of the two networks are expressed as:

[0113]

[0114] Step 2.2: Update the action network using gradient ascent , the loss is expressed as:

[0115]

[0116] Step 3: Use gradient descent to update the Lagrange multiplier λ. The loss is expressed as:

[0117]

[0118] Step 4: Repeat steps 2 and 3 until the sampled trajectory is traversed;

[0119] Step 5: Reward-constraint joint optimization is completed.

[0120] 4. Training process:

[0121] In step S40, based on the above steps S10 to S30, the training is completed and the model is obtained. Figure 3 As shown:

[0122] Step 1: Initialize the action network , reward value network and Constrained Value Network .

[0123] Step 2: The agent interacts with the environment to collect trajectories.

[0124] Step 3: Build a trajectory storage pool.

[0125] Step 4: Use reward-constraint joint optimization to construct a Lagrangian function to optimize network parameters;

[0126] Step 5: Training is completed and the model is obtained.

[0127] 5. Deployment docking control:

[0128] During the deployment process, only the trained model is needed. The states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario are input into the optimized safety reinforcement learning model, and the control action of the tracking spacecraft is output, that is, the intelligent agent will make a control action.

[0129] The present invention proposes a spacecraft rendezvous and docking control method based on safety reinforcement learning, which has the following advantages:

[0130] 1. High safety and high reliability. In high-risk spacecraft operation control, ensuring that the control strategy meets safety constraints is the key. The present invention uses a safety reinforcement learning method to add consideration of safety constraints to the optimization objectives, and uses the Lagrangian method to efficiently solve the optimization problem with constraints, so that the intelligent agent can meet the safety constraints while maximizing the reward, thereby improving the safety of the control algorithm during the rendezvous and docking process.

[0131] 2. Low computational complexity and high operational efficiency. During the training process, safety reinforcement learning discovers data-driven control strategies by learning from a large number of simulations or actual operations. These strategies can directly map system states to control methods without having to solve complex mathematical optimization problems in each control step, converting complex optimization problems into simple forward network inferences, greatly reducing computational complexity;

[0132] 3. High robustness and strong anti-interference ability. In the spacecraft rendezvous and docking environment, there are unpredictable factors such as microgravity environment, nonlinear characteristics of spacecraft dynamics, and unpredictable environmental disturbances (such as space debris, solar radiation pressure, etc.). In safe reinforcement learning, by exploring different strategies and learning based on reward and constraint signals, the agent is able to adapt to changes in the environment and unknown disturbances. This learning and adaptation ability can enhance the robustness of the control strategy in the face of model inaccuracies or external disturbances.

[0133] Based on the same inventive concept, an embodiment of the present invention also provides a spacecraft rendezvous and docking control system based on safety reinforcement learning. Since the principle of solving the problem by this system is similar to the aforementioned spacecraft rendezvous and docking control method based on safety reinforcement learning, the implementation of this system can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0134] Reference Figure 4 As shown, a spacecraft rendezvous and docking control system based on safety reinforcement learning includes:

[0135] An interactive data collection module is used to collect trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; the safety reinforcement learning agent has a safety reinforcement learning model, and the safety reinforcement learning model includes: an action network and a value network;

[0136] A trajectory storage pool module, used for storing the trajectory data in a trajectory storage pool in the form of tuples;

[0137] A safety reinforcement learning optimization module is used to use the trajectory data in the trajectory storage pool as the input of the safety reinforcement learning model, construct a Lagrangian function to perform reward and constraint joint optimization on the network model parameters of the safety reinforcement learning model; after iterative training, the optimized safety reinforcement learning model is obtained;

[0138] The rendezvous and docking control module is used to input the states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario into the optimized safety reinforcement learning model, and output the control action of the tracking spacecraft.

[0139] in:

[0140] (1) The interactive data collection module consists of a spacecraft rendezvous and docking simulation environment and a safety reinforcement learning agent. The spacecraft rendezvous and docking simulation environment is a virtual environment that simulates the rendezvous and docking of spacecraft. It simulates the rendezvous and docking process of the tracking spacecraft and the target spacecraft by giving the states s of the tracking spacecraft and the target spacecraft and receiving the tracking spacecraft action a. The safety reinforcement learning agent includes an action network π(a|s) and a value network Q(s,a). The action network outputs the tracking spacecraft action a according to the state s, and the value network outputs the estimated reward for taking action a under the current state s. The workflow of the interactive data collection module is described as follows:

[0141] Step 1: Initialize the simulation environment and generate state s;

[0142] Step 2: The agent receives state s and generates action a using the action network;

[0143] Step 3: The simulation environment receives action a, generates the state s at the next moment, and generates reward r and constraint c;

[0144] Step 4: If this round of simulation has not ended, return to step 2 and continue the loop; if this round of simulation has ended, go to step 5;

[0145] Step 5: The simulation ends and data collection is completed.

[0146] (2) Track storage pool module

[0147] The trajectory storage pool stores the trajectories generated by the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment, which are generated by the interactive data acquisition module, in the form of tuples {s, a, r, c, s'}.

[0148] (3) Security reinforcement learning optimization module

[0149] According to the trajectories stored in the trajectory storage pool, the parameters of the deep neural network in the safety reinforcement learning agent are updated. The reward-constraint joint optimization module solves the constrained optimization problem by constructing a Lagrangian function. The optimization direction is to maximize the rewards received by the safety reinforcement learning agent, while ensuring that the constraints received by the safety reinforcement learning agent meet the restrictions to ensure the safety of the docking process. After the training is completed, the optimized safety reinforcement learning model is obtained.

[0150] (4) Rendezvous and Docking Control Module

[0151] The states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario are input into the optimized safety reinforcement learning model, and the control action of the tracking spacecraft is output.

[0152] The spacecraft rendezvous and docking control system based on safety reinforcement learning provided by the present invention has computational efficiency, interference robustness and constraint safety.

[0153] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0154] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A spacecraft rendezvous and docking control method based on safety reinforcement learning, characterized in that: The following steps are involved: S10, collecting trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; The security reinforcement learning agent has a security reinforcement learning model, and the security reinforcement learning model includes: an action network and a value network; S20, storing the trajectory data in a trajectory storage pool in the form of tuples; S30, using the trajectory data in the trajectory storage pool as the input of the safety reinforcement learning model, constructing a Lagrangian function to perform reward and constraint joint optimization on the network model parameters of the safety reinforcement learning model; S40, iterative training is completed, and the optimized security reinforcement learning model is obtained; S50, inputting the states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario into the optimized safety reinforcement learning model, and outputting the control action of the tracking spacecraft; Wherein, the step S30 includes: S301. Determine the optimization goal of safety reinforcement learning, including: maximizing the reward received by the tracking spacecraft and satisfying the constraints received by the tracking spacecraft. The mathematical expression formula is as follows: in: E means to find the mathematical expectation; a t ~π represents the desired action a t is obtained from the action network π; s0~μ means that the state s at time t=0 comes from the initial state distribution μ; s t ~p means that except for the time t = 0, the state s at other times is determined by the state transition probability p(s t+1 |s t ,a t ) is obtained; b is the upper limit of the constraint; J(π) is the expected return reward; γ is the discount factor; r(s t ,a t ) is the reward function; s t is the status of the tracking spacecraft and the target spacecraft at time t; a t is the execution action of the tracking spacecraft at time t; J c (π) is the expected constraint, c(s t ,a t ) is the constraint function; S302, constructing a Lagrangian function; S303, sampling and storing trajectory data {s, a, r, c, s′} from the trajectory storage pool, respectively representing the state at the current moment, the action at the current moment, the reward received at the current moment, the constraint received at the current moment and the state at the next moment; S304, using the gradient descent method to update the reward value network and the constraint value network; using the gradient ascent method to update the action network; S305, using the gradient descent method to update the Lagrange multiplier; S306. Repeat steps S304 to S305 until all sampled and stored trajectory data are traversed; the joint optimization of rewards and constraints is completed.

2. The spacecraft rendezvous and docking control method based on safety reinforcement learning according to claim 1 is characterized in that: The step S10 comprises: S101, constructing a simulation environment and a safety reinforcement learning agent for simulating spacecraft rendezvous and docking; the safety reinforcement learning agent has a safety reinforcement learning model, and the safety reinforcement learning model includes: an action network and a value network; S102, setting the states s of the tracking spacecraft and the target spacecraft in the simulation environment, and inputting them into the action network; S103, the action network outputs the tracking spacecraft action a according to the state s; the value network outputs the estimated return reward of taking action a under the current state s; S104, generating a state s at the next moment according to the tracking spacecraft action a, and generating rewards and constraints at the same time; S105. During the simulation process, continue to loop steps S102 to S104 to collect trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; until the simulation ends.

3. The spacecraft rendezvous and docking control method based on safety reinforcement learning according to claim 2 is characterized in that: The reward in step S104 is expressed as follows: At time t, the reward r received by the tracking spacecraft is: r=d t-1 -d t Among them, d t-1 d t are the distances between the tracking spacecraft and the target spacecraft at time t-1 and time t respectively.

4. The spacecraft rendezvous and docking control method based on safety reinforcement learning according to claim 2 is characterized in that: The constraints in step S104 are expressed as follows: Where ω is the angular velocity of the current tracking spacecraft, ω c is the set maximum angular velocity; when the angular velocity is less than the maximum value, the constraint is 0; when the angular velocity is greater than or equal to the maximum value, the tracking spacecraft receives a constraint of 1.

5. The spacecraft rendezvous and docking control method based on safety reinforcement learning according to claim 1 is characterized in that: The Lagrangian function of step S302 is expressed as follows: L(π,λ)=J(π)-λ·(J c (p)-b) L(·) is the Lagrangian function; π is the action network; λ is the Lagrangian multiplier.

6. The spacecraft rendezvous and docking control method based on safety reinforcement learning according to claim 5 is characterized in that: In step S304: The reward value network loss is expressed as: L V =(r+V(s′)-V(s)) 2 The constraint value network loss is expressed as: L Vc =(c+V c (s′)-V c (s)) 2 V(·) is the mean of the cumulative returns of the value function; V c (·) is the mean of the cumulative constraints of the value function; r is the reward; c is the constraint; Update action network loss is expressed as: L(π,λ)=V(s)-λ·(V c (s)-b).

7. The spacecraft rendezvous and docking control method based on safety reinforcement learning according to claim 6 is characterized in that: In step S305: Update the Lagrange multiplier λ, and the loss is expressed as: L(π,λ)=V c (s)-b.

8. A spacecraft rendezvous and docking control system based on safety reinforcement learning, characterized in that: include: The interactive data acquisition module is used to collect trajectory data during the interaction between the safety reinforcement learning agent and the spacecraft rendezvous and docking simulation environment; The security reinforcement learning agent has a security reinforcement learning model, and the security reinforcement learning model includes: an action network and a value network; A trajectory storage pool module, used for storing the trajectory data in a trajectory storage pool in the form of tuples; A safety reinforcement learning optimization module is used to use the trajectory data in the trajectory storage pool as the input of the safety reinforcement learning model, construct a Lagrangian function to perform reward and constraint joint optimization on the network model parameters of the safety reinforcement learning model; after iterative training, the optimized safety reinforcement learning model is obtained; A rendezvous and docking control module, used to input the states of the tracking spacecraft and the target spacecraft in the rendezvous and docking scenario into the optimized safety reinforcement learning model, and output the control action of the tracking spacecraft; Among them, in the security reinforcement learning optimization module, the process of the security reinforcement learning model includes: S301. Determine the optimization goal of safety reinforcement learning, including: maximizing the reward received by the tracking spacecraft and satisfying the constraints received by the tracking spacecraft. The mathematical expression formula is as follows: in: E means to find the mathematical expectation; a t ~π represents the desired action a t is obtained from the action network π; s0~μ means that the state s at time t=0 comes from the initial state distribution μ; s t ~p means that except for the time t = 0, the state s at other times is determined by the state transition probability p(s t+1 |s t ,a t ) is obtained; b is the upper limit of the constraint; J(π) is the expected return reward; γ is the discount factor; r(s t ,a t ) is the reward function; s t is the status of the tracking spacecraft and the target spacecraft at time t; a t is the execution action of the tracking spacecraft at time t; J c (π) is the expected constraint, c(s t ,a t ) is the constraint function; S302, constructing a Lagrangian function; S303, sampling and storing trajectory data {s, a, r, c, s′} from the trajectory storage pool, respectively representing the state at the current moment, the action at the current moment, the reward received at the current moment, the constraint received at the current moment and the state at the next moment; S304, using the gradient descent method to update the reward value network and the constraint value network; using the gradient ascent method to update the action network; S305, using the gradient descent method to update the Lagrange multiplier; S306. Repeat steps S304 to S305 until all sampled and stored trajectory data are traversed; the joint optimization of rewards and constraints is completed.

Images