A spacecraft approaching control method based on imitation learning and reinforcement learning fusion

Abstract

This invention relates to the field of spacecraft orbit control technology, and particularly to a spacecraft approach control method based on the fusion of imitation learning and reinforcement learning. The method includes: S1: generating initial and target states, solving for the optimal fuel transfer trajectory offline using a direct method, and constructing an expert dataset; S2: based on the expert dataset, training a neural network controller through imitation learning to obtain a pre-trained model; S3: loading the weights of the pre-trained model, constructing a composite reward function, and training the policy network using a near-end policy optimization algorithm to obtain an optimized policy network model; S4: acquiring relative state data, optimizing the policy network model, and outputting the control quantity for the optimal fuel consumption trajectory transfer in real time. This invention rapidly masters an approximately optimal policy through imitation learning and surpasses the performance of expert data through reinforcement learning, completing control command solutions in milliseconds. It balances terminal control accuracy and fuel consumption optimization, offering advantages such as high solution efficiency, excellent control accuracy, and strong generalization ability.

Description

Technical Field

[0001] This invention relates to the field of spacecraft orbit control technology, and in particular to a spacecraft approach control method based on the fusion of imitation learning and reinforcement learning. Background Technology

[0002] Close-proximity control of spacecraft is a core technology in missions such as on-orbit servicing, rendezvous and docking, and space debris removal. In close-proximity scenarios, due to the short distance and low relative speed, continuous low-thrust propulsion is typically employed to achieve high-precision, low-burnup orbital transfer. While continuous low-thrust propulsion technology offers advantages such as high specific impulse and high control precision, its relatively small thrust amplitude results in a dynamically smooth and time-scaled orbital transfer process. This makes the corresponding optimal burnup control problem mathematically a complex optimization problem with high solution difficulty and computational cost.

[0003] For the problem of optimal orbital transfer of spacecraft with optimal fuel consumption, existing technologies mainly adopt two types of methods under the framework of optimal control theory: indirect methods and direct methods.

[0004] The indirect method, based on Pontryagin's minimum principle, obtains the optimal control law by solving a two-point boundary value problem. Theoretically, it can obtain the global optimum and has good continuity of control variables. However, the initial values ​​of costate variables are difficult to predict during the solution process, and the method is extremely sensitive to boundary conditions, leading to difficulties in solving the two-point boundary value problem and a low success rate.

[0005] The direct method transforms the optimal control problem into a nonlinear programming problem, which is solved by parametric methods (such as the pseudospectral method). It is not sensitive to initial values ​​and is relatively easy to solve, but its accuracy is limited and it is difficult to meet the requirements of high-precision control.

[0006] More importantly, both indirect and direct methods are computationally complex and inefficient, requiring re-solution before each task, making real-time online control under arbitrary initial states impossible. While some existing research attempts to solve two-point boundary value problems using machine learning methods, such as fitting the mapping relationship between state variables and costate variables using deep neural networks, these methods still have significant limitations: firstly, training neural networks requires a large number of training samples, and sample generation still relies on indirect methods, resulting in a significant consumption of computational resources and time in generating training sample sets; secondly, since each training sample is obtained separately through indirect methods, the success rate remains low, and simply using imitation learning is limited by the accuracy limit of expert data, unable to surpass the performance of expert data.

[0007] Therefore, there is an urgent need for a spacecraft fuel-optimal approach control method that can balance high precision and real-time performance. This method should overcome the difficulties of solving indirect methods, break through the limitations of insufficient accuracy in direct methods, and achieve real-time optimal control under any initial state. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a spacecraft approach control method based on the fusion of imitation learning and reinforcement learning, comprising the following steps: S1: Randomly generate initial and target states within the preset spacecraft approach mission space, use the direct method to solve the optimal fuel transfer trajectory corresponding to each set of states offline, extract the state and control variables in the trajectory, and construct an expert dataset; S2: Based on the expert dataset, a neural network controller is trained by imitation learning to minimize the difference between the network output and the expert control sequence, thereby obtaining a pre-trained model with near-optimal policy capability; S3: Load the weights of the pre-trained model as initial policy parameters, construct a composite reward function for optimizing terminal control accuracy and fuel consumption, and refine the policy network in a reinforcement learning environment using a near-end policy optimization algorithm to obtain an optimized policy network model. S4: Real-time acquisition of the spacecraft's relative state data with respect to the target approach point. The optimization strategy network model outputs the control quantity for the spacecraft's optimal orbital transfer based on the relative state data, thus completing the spacecraft's approach closed-loop control.

[0009] Preferably, in step S1, the optimal fuel transfer trajectory corresponding to each set of states is solved offline using a direct method, including: Establish an LVLH relative orbit coordinate system with the target state as the origin, and establish the spacecraft's relative motion dynamics equations based on this coordinate system; A fuel-optimal control model is constructed, and the objective function is set as minimizing the modulus integral of the control acceleration throughout the orbital transfer to characterize fuel consumption. The constraints include the relative motion dynamics equation, the upper limit constraint of the control amplitude, and the terminal relative position and relative velocity error constraint. For each state, the Gaussian pseudospectral method is used to discretize the optimal fuel control model into a nonlinear programming problem, and the optimal fuel transfer trajectory is obtained by solving the problem. The optimal fuel transfer trajectory is interpolated and converted into an equal-time-step sequence that matches subsequent training.

[0010] Preferably, in step S2, training the neural network controller through imitation learning includes: The expert dataset is cleaned and preprocessed to remove abnormal data points that fail the dynamic consistency test or whose control variables exceed physical constraints. The state variables and control variables are normalized separately. The statistics used for normalization are calculated from the training set and pre-stored. A multi-layer fully connected neural network is constructed as the policy network architecture. The behavior cloning method is used for supervised learning. The normalized relative state is used as the network input and the normalized control quantity is used as the network's expected output. The loss function is defined as the mean square error between the neural network's predicted control input and the expert control input. The network weight parameters are iteratively optimized through the backpropagation algorithm until the validation set loss converges or the preset number of training rounds is reached, thus obtaining the pre-trained model.

[0011] Preferably, the multilayer fully connected neural network includes an input layer, several hidden layers, and an output layer, wherein the number of neurons in the input layer is consistent with the dimension of the state vector, the number of neurons in the output layer is consistent with the dimension of the control vector, a ReLU activation function is applied after the hidden layers, and the output layer uses a tanh activation function to limit the output within a preset range.

[0012] Preferably, the formula for calculating the loss function is as follows: ,in For batch size, and For the state-action pairs sampled from the expert dataset, It is a neural network for the state The predicted actions are adjusted by optimizing the network parameters. To minimize the loss function.

[0013] Preferably, in step S3, the composite reward function comprises two parts in stages: a real-time reward throughout the entire process and a one-time reward at the terminal. The real-time reward throughout the entire process includes a position error improvement reward, a staged fuel optimization reward, and a speed error dynamic adjustment reward. The one-time reward at the terminal includes a terminal control accuracy enhancement reward and a terminal milestone and imitation learning improvement reward. The composite reward function is calculated in stages as follows: ,in, , , , and Here, represents the weighting coefficient of each reward item, and t represents the current control time. End-point control precision enhancement bonus item The calculation formula is as follows: ,in, Output the relative position state corresponding to the strategy at time t; Speed ​​error dynamic adjustment bonus The calculation formula is as follows: ,in, For orbital transfer time, Output the relative basic state corresponding to the strategy at time t. This is the upper limit threshold for spacecraft control acceleration.

[0014] Preferably, the position error improvement reward The position error improvement is calculated based on the execution of the current reinforcement learning policy network output action and the pre-trained model output action, respectively. The calculation formula is as follows: Among them, the improvement in position error As shown below: , The position state of the pre-trained model at the next moment corresponding to its action. To reinforce the learning strategy action and the next position state at the next moment; The phased fuel optimization reward The orbital transfer is triggered in the first half based on a positive lift condition due to position error, and in the second half based on a simultaneous decrease in both velocity error and fuel consumption. The calculation formula is as follows: ,in, For orbital transfer time, The middle moment, This is the improvement amount for speed error.

[0015] Terminal milestones and imitation learning improvement rewards Triggered at the terminal, the calculation formula is as follows: ,in, and These are multi-level milestone rewards corresponding to terminal position error and terminal speed error, respectively.

[0016] Preferably, in step S3, a reward function for optimizing terminal control accuracy and fuel consumption is constructed, and policy refinement training is performed through reinforcement learning, including: A reinforcement learning interactive environment is constructed, wherein the state space and action space of the environment are matched with the input dimension and output dimension of the training neural network controller, respectively, and the simulation step size is consistent with the interpolated time step size. The interactive environment is initialized, an initial state is selected from the task space to start the track transfer round, the policy network outputs control actions based on the current state, and after the environment executes the actions, it returns the next moment's state, immediate reward and round termination flag. Collect sample data for the entire round, calculate the advantage function for each state-action pair using the generalized advantage estimation method, and calculate the cumulative reward with target discount for each state. The calculation formula for the generalized advantage estimation method is shown below: ,in, As a discount factor, As a smoothing factor, For value network output, This is a round termination indicator; The total loss function is constructed based on the pruning objective of the PPO algorithm. The policy pruning loss limits the policy update magnitude, the value loss fits the state value, and the entropy loss improves the policy exploration. The policy network and value network are iteratively updated. After each preset training round, the performance of the current model is verified to determine whether the convergence condition is met, and the optimal policy network model that meets the convergence condition is saved.

[0017] Preferably, the formula for calculating the total loss function is as follows: Among them, entropy loss The calculation formula is as follows: Value loss The calculation formula is as follows: , For state value, Accumulate returns for the target discount. For the expectation operator, the pruning loss The calculation formula is as follows: , For the clipping factor, The ratio of the action probabilities of the new strategy to those of the old strategy. The dominant function is calculated using the following formula: ,in, For the probability of the new strategy action, This represents the probability of actions under the old strategy.

[0018] Preferably, in step S4, the control parameters for the spacecraft's optimal burnup orbital transfer are output in real time using the trained policy network model, including: During in-orbit deployment, the pre-trained policy network model and pre-stored normalized parameters are preloaded. The real-time relative state of the spacecraft relative to the target approach point is acquired in real time. The real-time relative state data is processed by normalized parameters to obtain normalized real-time state data. The normalized real-time state data is input into the optimization strategy network model to obtain the normalized control quantity output by the model. The normalized control quantity is then processed by inverse normalization and amplitude limiting, and then sent to the spacecraft propulsion system for execution. Closed-loop feedback iteration is performed with a fixed control cycle until the terminal approaches the required accuracy or the safety boundary is triggered.

[0019] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention generates multiple initial and target states randomly within the preset spacecraft approach mission space, and solves the optimal fuel transfer trajectory corresponding to each state offline using the direct method. It extracts state-control sequence pairs to construct an expert dataset, thereby achieving efficient batch generation of the optimal fuel trajectory. Compared with the traditional indirect method, it avoids the problems of difficulty in guessing the initial value of costate variables and low success rate of solution. At the same time, it provides high-quality training samples covering the mission space for subsequent imitation learning, ensuring the data foundation for policy learning.

[0020] (2) Based on expert datasets, this invention trains neural network controllers by imitation learning, minimizes the difference between network output and expert control sequence, obtains a pre-trained model with near-optimal policy capability, realizes efficient fitting of the solution results of direct method, enables neural network to output near-optimal control commands in millisecond time, solves the technical problem that traditional optimal control methods cannot be applied online in real time, provides a good initial strategy for subsequent reinforcement learning, and significantly improves training efficiency.

[0021] (3) The weights of the pre-trained model are used as the initial policy parameters. A composite reward function for optimizing terminal control accuracy and fuel consumption is constructed. The policy network is refined and trained in a reinforcement learning environment through a near-end policy optimization algorithm to obtain an optimized policy network model. This achieves the performance of expert data, making the trained policy network superior to the expert trajectory solved by the direct method in terms of both terminal control accuracy and fuel consumption. Closed-loop control is completed by acquiring relative state data in real time and outputting control quantities. Finally, the high precision, low fuel consumption and strong real-time performance of spacecraft approach control are achieved. It has the comprehensive advantages of high control accuracy, excellent fuel consumption, fast online calculation and strong generalization ability. Attached Figure Description

[0022] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0023] Figure 1 This is a flowchart of a spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to the present invention; Figure 2 The result of solving the problem using the GPOPS method is shown in the figure. Figure 3 Solving the control trajectory diagram using the GPOPS method; Figure 4 The graph shows the change in the learning loss function. Figure 5 The training results are shown in Figure (a), which is the position error diagram, and Figure (b) is the velocity error diagram. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Obviously, the described embodiments are only some, not all, of the embodiments described in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0025] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a” and “an” used herein, and “the”, may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0026] First embodiment: Please see Figure 1 As shown, this embodiment provides a spacecraft approach control method based on the fusion of imitation learning and reinforcement learning, including the following steps: S1: Randomly generate initial and target states within the preset spacecraft approach mission space, use the direct method to solve the optimal fuel transfer trajectory corresponding to each set of states offline, extract the state and control variables in the trajectory, and construct an expert dataset.

[0027] Preferably, in step S1, the optimal fuel transfer trajectory corresponding to each set of states is solved offline using a direct method, including: An LVLH relative orbit coordinate system is established with the target state as the origin. Based on this coordinate system, the spacecraft's relative motion dynamics equations are established. Specifically, in this embodiment, the orbit transfer mission range is set as a spherical region with a radius of 10 km. Initial relative positions are uniformly selected within a spherical region with a radius of 2 km to 10 km. The initial velocity magnitude is randomly selected between 0 and 10 m / s, and the initial velocity direction is selected such that the angle between it and the vector pointing towards the target is less than 90°, thus obtaining the initial relative state. The orbit transfer time is set according to the initial relative distance or mission requirements. The calculation formula in this example is: ,in This represents the initial relative distance.

[0028] A fuel-optimal control model is constructed, with the objective function set as minimizing the modulus integral of the control acceleration throughout the orbital transfer to characterize fuel consumption. Constraints include the relative motion dynamics equations, upper limit constraints on the control magnitude, and terminal relative position and relative velocity error constraints. Specifically, in this embodiment, the orbital dynamics constraints can be selected from a two-body gravitational model, a CW equation dynamics model, or a TH equation dynamics model, etc. This example selects the CW equation. This indicates the angular velocity of the target point's trajectory, and the maximum acceleration of the three axes is controlled to be less than or equal to 0.1 m / s².

[0029] .

[0030] For each state group, the Gaussian pseudospectral method is used to discretize the fuel optimal control model into a nonlinear programming problem, and the optimal fuel transfer trajectory is obtained by solving it. Specifically, in this embodiment, the MATLAB GPOPS-II toolbox is used to solve the fuel consumption optimal transfer trajectory. GPOPS-II is an adaptive pseudospectral method and belongs to the direct method optimal control solver. In this embodiment, the solver parameters are set as shown in Table 1 below: Table 1: Solver Parameter Configuration Table The input is the initial state. Given the target state [0 0 0 0 0 0], the output is a time series with variable step size and the corresponding 6-dimensional state sequence and 3-dimensional acceleration sequence.

[0031] The optimal fuel transfer trajectory is interpolated and converted into a time-step sequence that matches subsequent training. Specifically, in this embodiment, to facilitate model training, the time step of the output sequence is changed to 1 second through interpolation. Assuming a random initial state [2000m, -9000m, 500m, -4m / s, 2m / s, 0m / s], data for an optimal fuel transfer trajectory are obtained. The curves showing the changes in relative position, relative velocity, and acceleration, as well as the transfer trajectory, are as follows: Figure 2 As shown, the terminal position error is [-3.6588m 33.0126m -2.3297m], and the terminal velocity error is [-0.0191m / s 0.2007m / s -0.0147m / s]. The data format is: columns 1 to 3 are relative position, columns 4 to 6 are relative velocity, and columns 7 to 9 are acceleration. New trajectory data are obtained by repeatedly inputting new initial states. Finally, these are combined to form an expert dataset of 5000 trajectories, with a dataset size of (5277655, 9).

[0032] S2: Based on the expert dataset, a neural network controller is trained through imitation learning to minimize the difference between the network output and the expert control sequence, thereby obtaining a pre-trained model with near-optimal policy capabilities.

[0033] Preferably, in step S2, training the neural network controller through imitation learning includes: Data cleaning and preprocessing are performed on the expert dataset to remove outlier data points that fail dynamic consistency tests or whose control variables exceed physical constraints (such as records containing NaN or infinity).

[0034] To improve the numerical stability and convergence speed of the training process, the state variables and control variables are normalized separately. The statistics used for normalization are calculated from the training set and pre-stored for real-time normalization during subsequent training and online deployment. Specifically, in this embodiment, the state variables and control variables are normalized separately, mapping them to the numerical range of [-1, 1]. The normalization formula is: ,in and These are the mean and standard deviation of the state variables, respectively. and These are the mean and standard deviation of the control quantity, respectively.

[0035] A multi-layer fully connected neural network is constructed as the policy network architecture, and the behavior cloning method is used for supervised learning. The normalized relative state is used as the network input and the normalized control quantity is used as the network's expected output.

[0036] Preferably, the multilayer fully connected neural network includes an input layer, several hidden layers, and an output layer. The number of neurons in the input layer is the same as the dimension of the state vector, and the number of neurons in the output layer is the same as the dimension of the control vector. A ReLU activation function is applied after the hidden layers, and the output layer uses a tanh activation function to limit the output to a preset range. Specifically, in this embodiment, a deep neural network is designed as the policy function. An approximator whose input is the normalized state quantity. The output is the predicted normalized control quantity. . refer to Figure 3 The neural network adopts a fully connected neural network structure, comprising one input layer, two hidden layers, and one output layer. The input layer has six neurons, consistent with the dimension of the state vector, corresponding to the x, y, and z components of relative position and relative velocity, respectively. Each hidden layer has 256 neurons, and each hidden layer is followed by a ReLU activation function. The output layer has three neurons, consistent with the dimension of the control vector, and outputs a three-axis thrust acceleration command. The tanh activation function is used to constrain the output to the range [-1, 1]. The network weights are initialized using the Xavier uniform distribution method, and the biases are initialized to zero.

[0037] The loss function is defined as the mean square error between the neural network's predicted control input and the expert control input. The network weight parameters are iteratively optimized through the backpropagation algorithm until the validation set loss converges or the preset number of training rounds is reached, thus obtaining the pre-trained model.

[0038] Preferably, the formula for calculating the loss function is as follows: ,in For batch size, and For state-action pairs sampled from expert datasets, It is a neural network for the state The predicted actions are adjusted by optimizing the network parameters. The goal is to minimize the loss function so that the output of the neural network is as close as possible to the expert's decision under the same conditions.

[0039] Please refer to Figure 4 The loss function descent process during training includes: Based on the defined loss function, gradient descent is used for network training. The optimization algorithm employs the Adam optimizer, with parameters set as follows: initial learning rate... The value is 0.001. The batch size is set to batch_size=64, and data is randomly sampled from the training set for each parameter update; the total number of training epochs is 82464. During training, the loss value is calculated on the validation set after each training epoch to monitor model performance; an early stopping strategy is adopted, if the validation set loss no longer decreases within a consecutive patience of 20 epochs, training is terminated early to prevent overfitting; finally, the model parameters with the best performance on the validation set are selected as the training result.

[0040] S3: Load the weights of the pre-trained model as the initial policy parameters, construct a composite reward function for optimizing terminal control accuracy and fuel consumption, and refine the policy network in a reinforcement learning environment using a near-end policy optimization algorithm to obtain an optimized policy network model.

[0041] Based on the relative orbital dynamics model, a reinforcement learning interactive environment conforming to the OpenAI Gym interface specification is constructed, specifying the environment simulation step size and control step size. All steps are set to 1 second, which is exactly the same as the time step of trajectory interpolation. The maximum number of steps in one simulation round is... The environment's state space is a 6-dimensional continuous vector, corresponding to the relative position and relative velocity of the current state and the target state along the x, y, and z axes in the LVLH coordinate system, perfectly matching the input dimension of the neural network. The state values ​​are limited to relative position ±50km and relative velocity ±50m / s. The environment's action space is a 3-dimensional continuous vector, corresponding to the control acceleration along the three axes in the LVLH coordinate system, with values ​​strictly limited to [-0.1m / s²]. 2 0.1m / s 2 This should be consistent with the control constraints.

[0042] Simultaneously, an environmental round termination condition is set: the simulation duration reaches the set orbital transfer time. The spacecraft's relative position exceeds the mission space boundary; the spacecraft's terminal position error is less than 1m and the terminal velocity error is less than 0.1m / s, completing the orbit transfer mission ahead of schedule.

[0043] The policy network and value network of the PPO algorithm are constructed separately. Both networks adopt a fully connected neural network structure that is completely consistent with the neural network controller: containing one 6-neuron input layer and two 256-neuron hidden layers (with ReLU activation function after the hidden layer); the output layer of the policy network has 3 neurons and is connected to the tanh activation function, which is completely matched with the neural network controller. The model weights with the optimal loss on the validation set are directly loaded as the initial parameters for pre-training to achieve a warm start of the policy and avoid the problems of low exploration efficiency and slow convergence caused by training from scratch; the output layer of the value network has 1 neuron and no activation function. It is used to estimate the cumulative reward of the current state. The network weights are initialized using the Xavier uniform distribution method, and the bias is initialized to 0.

[0044] Preferably, in step S3, the composite reward function comprises two parts: a phased real-time reward throughout the entire process and a one-time reward at the end. The real-time reward throughout the entire process includes a position error improvement reward, a phased fuel optimization reward, and a speed error dynamic adjustment reward. The one-time reward at the end includes a terminal control accuracy enhancement reward and a terminal milestone and imitation learning improvement reward. The composite reward function is calculated in stages, as follows: ,in, , , , and Here, represents the weighting coefficient of each reward item, and t represents the current control time. To enhance end-point control accuracy, the last 30 steps of the trajectory transfer are selected as the key optimization segment. The current relative state is compared with the near-optimal deceleration state; the smaller the error, the greater the reward. This is a reward item for enhancing end-point control accuracy. The calculation formula is as follows: ,in, Output the relative position state corresponding to the strategy at time t; Speed ​​error dynamic adjustment bonus The calculation formula is as follows: ,in, For orbital transfer time, Output the relative basic state corresponding to the strategy at time t. This is the upper limit threshold for spacecraft control acceleration.

[0045] Preferably, the position error improvement reward The position error improvement is calculated based on the execution of the current reinforcement learning policy network output action and the pre-trained model output action, respectively. The calculation formula is as follows: Among them, the improvement in position error As shown below: , The position state of the pre-trained model at the next moment corresponding to its action. To reinforce the learning strategy action and its next position state, specifically in this embodiment, at each time step t, the current relative state is... The inputs are fed into the current reinforcement learning actor network and the trained imitation learning model, respectively, to obtain the output action of the actor network. The output action of the imitation learning model Perform one iteration on each of the two actions in the environment to obtain the state at the next moment. and Phased fuel optimization rewards The orbital transfer is triggered in the first half based on a positive lift condition due to position error, and in the second half based on a simultaneous decrease in both velocity error and fuel consumption. The calculation formula is as follows: ,in, For orbital transfer time, The middle moment, To improve the speed error, specifically, in this embodiment, the first half of a round is defined as an acceleration phase and the second half as a deceleration phase. During the acceleration phase, if... (Positive improvement in position error): In addition to the position error improvement reward, an extra fuel consumption reward is given, with the weight of the extra reward halved. The expression is as follows: During the deceleration phase, the additional reward is triggered when both speed error and fuel consumption decrease, i.e., both conditions must be met simultaneously: and This system provides immediate rewards based on speed error improvement and dynamic step size weighting. It focuses on improving speed error and, through weights that dynamically change with step size, adapts to a control logic that encourages acceleration in the initial stage and emphasizes deceleration in the later stage. A negative value indicates an increasing speed error, consistent with an accelerating trend; a positive reward encourages further acceleration. If the speed error decreases, the reward is positive and the reward value increases as the speed approaches the terminal, thus strengthening deceleration control.

[0046] Terminal milestones and imitation learning improvement rewards Triggered at the terminal, the calculation formula is as follows: ,in, and The system provides multi-level milestone rewards for terminal position error and terminal speed error. Specifically, in this embodiment, the terminal error of the actor network is compared with the terminal error of the initial imitation learning model simulation. If the error of the actor network is smaller, an improvement reward is given.

[0047] The weighting coefficients for each part of the reward are shown in Table 2 below: Table 2: Weighting Coefficients for Reward Items Preferably, in step S3, a reward function for optimizing terminal control accuracy and fuel consumption is constructed, and policy refinement training is performed through reinforcement learning, including: A reinforcement learning interactive environment is constructed, where the state space and action space of the environment are matched with the input and output dimensions of the training neural network controller, respectively. The simulation step size is consistent with the interpolated time step size. Specifically, in this embodiment, at each time step, the environment outputs the current state. The policy network outputs the mean of the control actions based on the current state, and then samples the final action using a Gaussian distribution with a fixed standard deviation of 0.015. Simultaneously record the logarithmic probability corresponding to this action. ; The interactive environment is initialized, an initial state is selected from the task space to initiate a trajectory transition round, the policy network outputs control actions based on the current state, and after the environment executes the actions, it returns the next time-instance state, immediate reward, and round termination flag. Specifically, in this embodiment, the environment executes actions... Then, return to the state of the next time step. Instant rewards Round End Flag ; We collect sample data from the entire round, and use the generalized advantage estimation method to calculate the advantage function for each state-action pair. This balances the bias and variance of advantage estimation, improves the stability of policy updates, and calculates the cumulative reward with target discount for each state. The formula for the generalized advantage estimation method is shown below: ,in, As a discount factor, As a smoothing factor, For value network output, This serves as a round termination marker. Specifically, in this embodiment, the bias and variance of advantage estimation are balanced to improve the stability of policy updates and to fully collect sample data at each time step. This process continues until the end of the round, completing multiple full rounds of interactive sampling until the total number of samples collected in a single sampling period is no less than 2048 sets. This provides data support for subsequent model updates, and the cumulative reward of the target discount corresponding to each state is calculated. , used for fitting and training value networks; The total loss function is constructed based on the pruning objective of the PPO algorithm to limit the policy update magnitude and avoid training collapse. The policy update magnitude is limited by the policy pruning loss, the state value is fitted by the value loss, and the policy exploration is improved by the entropy loss. The policy network and the value network are iteratively updated. Specifically, in this embodiment, the policy pruning loss is used to optimize the policy network. The pruning mechanism limits the policy update magnitude to a reasonable range to avoid performance degradation caused by excessive differences between the old and new policies. The value loss is used to optimize the value network. The mean squared error loss is used to measure the accuracy of the value network's estimation of the state value. The loss is used to improve the exploration of the policy and prevent the policy from converging to a local optimum too early. After each preset training round, the performance of the current model is verified to determine whether the convergence condition is met, and the optimal policy network model that meets the convergence condition is saved.

[0048] Preferably, the formula for calculating the total loss function is as follows: In this context, the entropy loss is transformed into an optimization objective that maximizes entropy through the use of a negative sign; the entropy loss... The calculation formula is as follows: Value loss The calculation formula is as follows: , For state value, Accumulate returns for the target discount. For the expectation operator, the pruning loss The calculation formula is as follows: , For the clipping factor, The ratio of the action probabilities of the new strategy to those of the old strategy. The dominant function is calculated using the following formula: ,in, For the probability of the new strategy action, This represents the probability of actions under the old strategy.

[0049] The specific configurations of all key hyperparameters and reward function weights for reinforcement learning training are shown in Table 3 below: Table 3: Hyperparameter Configuration Table for Reinforcement Learning S4: Real-time acquisition of the spacecraft's relative state data with respect to the target approach point; optimization strategy network model outputs the control quantity for the spacecraft's optimal orbital transfer based on the relative state data, completing the spacecraft's approach closed-loop control.

[0050] Training process reference Figure 5 After 500 rounds of training, the positional error decreased from the initial 14m to less than 1m, and the velocity error decreased from the initial 1.7m / s to 0.2m / s.

[0051] Preferably, in step S4, the control parameters for the spacecraft's optimal burnup orbital transfer are output in real time using the trained policy network model, including: During in-orbit deployment, the pre-trained policy network model and pre-stored normalized parameters are preloaded. The real-time relative state of the spacecraft relative to the target approach point is acquired in real time. The real-time relative state data is processed by normalized parameters to obtain normalized real-time state data. The normalized real-time state data is input into the optimization strategy network model to obtain the normalized control quantity output by the model. The normalized control quantity is inversely normalized and limited before being sent to the spacecraft propulsion system for execution. The closed-loop feedback iteration is performed with a fixed control cycle until the terminal approach accuracy requirement is met or the safety boundary is triggered. Specifically, in this embodiment, when it is necessary to perform the spacecraft's close-range burn-out optimal low-thrust orbit transfer approach mission, the real-time relative state of the spacecraft with respect to the target approach point is determined. After standardizing the relative state using pre-stored normalized parameters, it is input into the trained deep neural network model to obtain the normalized control quantity output by the model. Then, it is converted into a physical quantity three-axis control acceleration command through pre-stored inverse normalized parameters. Based on this control command, the closed-loop autonomous control of the spacecraft orbit transfer is completed.

[0052] The spacecraft approach control method based on the fusion of imitation learning and reinforcement learning provided in this embodiment can quickly construct a high-quality expert dataset of optimal fuel consumption trajectory. Through imitation learning, it can quickly complete the initial learning of near-optimal strategies, and with the help of reinforcement learning, it can surpass the performance of expert data. It can solve the optimal fuel consumption control command in milliseconds, taking into account both terminal control accuracy and fuel consumption optimization. The solution efficiency, control accuracy and mission success rate are significantly better than traditional optimal control methods, and it can fully meet the application requirements of real-time online control of spacecraft in orbit.

[0053] To verify the beneficial effects of this invention, the imitation learning pre-trained model of this embodiment was used as the control group, and the converged reinforcement learning refined model was used as the experimental group. A comparative test was conducted using 100 sets of randomly initialized initial state samples that had not participated in training. The comparison results of the core performance indicators are shown in Table 4 below: Table 4: Performance Indicator Comparison Table Second Embodiment Based on the same concept, this embodiment also provides a computer device, including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform steps of a spacecraft approach control method based on the fusion of imitation learning and reinforcement learning as described in the embodiment.

[0054] It is understood that, for the aforementioned spacecraft approach control method based on the fusion of imitation learning and reinforcement learning, if all of these methods are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer server or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0055] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0056] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A spacecraft proximity control method based on the fusion of imitation learning and reinforcement learning, characterized in that, Includes the following steps: S1: Randomly generate initial and target states within the preset spacecraft approach mission space, use the direct method to solve the optimal fuel transfer trajectory corresponding to each set of states offline, extract the state and control variables in the trajectory, and construct an expert dataset; S2: Based on the expert dataset, a neural network controller is trained by imitation learning to minimize the difference between the network output and the expert control sequence, thereby obtaining a pre-trained model with near-optimal policy capability; S3: Load the weights of the pre-trained model as initial policy parameters, construct a composite reward function for optimizing terminal control accuracy and fuel consumption, and refine the policy network in a reinforcement learning environment using a near-end policy optimization algorithm to obtain an optimized policy network model. S4: Real-time acquisition of the spacecraft's relative state data with respect to the target approach point. The optimization strategy network model outputs the control quantity for the spacecraft's optimal orbital transfer based on the relative state data, thus completing the spacecraft's approach closed-loop control.

2. The spacecraft proximity control method based on the fusion of imitation learning and reinforcement learning according to claim 1, characterized in that, In step S1, the optimal fuel transfer trajectory for each set of states is solved offline using the direct method, including: Establish an LVLH relative orbit coordinate system with the target state as the origin, and establish the spacecraft's relative motion dynamics equations based on this coordinate system; A fuel-optimal control model is constructed, and the objective function is set as minimizing the modulus integral of the control acceleration throughout the orbital transfer to characterize fuel consumption. The constraints include the relative motion dynamics equation, the upper limit constraint of the control amplitude, and the terminal relative position and relative velocity error constraint. For each state, the Gaussian pseudospectral method is used to discretize the optimal fuel control model into a nonlinear programming problem, and the optimal fuel transfer trajectory is obtained by solving the problem. The optimal fuel transfer trajectory is interpolated and converted into an equal-time-step sequence that matches subsequent training.

3. The spacecraft proximity control method based on the fusion of imitation learning and reinforcement learning according to claim 1, characterized in that, In step S2, the neural network controller is trained by imitation learning, including: The expert dataset is cleaned and preprocessed to remove abnormal data points that fail the dynamic consistency test or whose control variables exceed physical constraints. The state variables and control variables are normalized separately. The statistics used for normalization are calculated from the training set and pre-stored. A multi-layer fully connected neural network is constructed as the policy network architecture. The behavior cloning method is used for supervised learning. The normalized relative state is used as the network input and the normalized control quantity is used as the network's expected output. The loss function is defined as the mean square error between the neural network's predicted control input and the expert control input. The network weight parameters are iteratively optimized through the backpropagation algorithm until the validation set loss converges or the preset number of training rounds is reached, thus obtaining the pre-trained model.

4. The spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to claim 3, characterized in that, The multilayer fully connected neural network includes an input layer, several hidden layers, and an output layer. The number of neurons in the input layer is the same as the dimension of the state vector, and the number of neurons in the output layer is the same as the dimension of the control vector. The hidden layers are followed by a ReLU activation function, and the output layer uses a tanh activation function to limit the output to a preset range.

5. The spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to claim 3, characterized in that, The formula for calculating the loss function is as follows: ,in For batch size, and For the state-action pairs sampled from the expert dataset, It is a neural network for the state The predicted actions are adjusted by optimizing the network parameters. To minimize the loss function.

6. The spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to claim 1, characterized in that, In step S3, the composite reward function comprises two parts in stages: a real-time reward throughout the entire process and a one-time reward at the terminal. The real-time reward throughout the entire process includes a position error improvement reward, a staged fuel optimization reward, and a speed error dynamic adjustment reward. The one-time reward at the terminal includes a terminal control accuracy enhancement reward and a terminal milestone and imitation learning improvement reward. The composite reward function is calculated in stages as follows: ,in, , , , and Here, represents the weighting coefficient of each reward item, and t represents the current control time. End-of-line control precision enhancement reward items The calculation formula is as follows: ,in, Output the relative position state corresponding to the strategy at time t; Speed ​​error dynamic adjustment bonus The calculation formula is as follows: ,in, For orbital transfer time, Output the relative basic state corresponding to the strategy at time t. This is the upper limit threshold for spacecraft control acceleration.

7. The spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to claim 6, characterized in that, The position error improvement reward The position error improvement is calculated based on the execution of the current reinforcement learning policy network output action and the pre-trained model output action, respectively. The calculation formula is as follows: Among them, the improvement in position error As shown below: , The position state of the pre-trained model at the next moment corresponding to its action. To reinforce the learning strategy action and the next position state at the next moment; The phased fuel optimization reward The orbital transfer is triggered in the first half based on a positive lift condition due to position error, and in the second half based on a simultaneous decrease in both velocity error and fuel consumption. The calculation formula is as follows: ,in, For orbital transfer time, The middle moment, This is the improvement amount for speed error; Terminal milestones and imitation learning improvement rewards Triggered at the terminal, the calculation formula is as follows: ,in, and These are multi-level milestone rewards corresponding to terminal position error and terminal speed error, respectively.

8. The spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to claim 1, characterized in that, In step S3, a reward function for optimizing terminal control accuracy and fuel consumption is constructed, and policy refinement training is performed through reinforcement learning, including: A reinforcement learning interactive environment is constructed, wherein the state space and action space of the environment are matched with the input dimension and output dimension of the training neural network controller, respectively, and the simulation step size is consistent with the interpolated time step size. The interactive environment is initialized, an initial state is selected from the task space to start the track transfer round, the policy network outputs control actions based on the current state, and after the environment executes the actions, it returns the next moment's state, immediate reward and round termination flag. Collect sample data for the entire round, calculate the advantage function for each state-action pair using the generalized advantage estimation method, and calculate the cumulative reward with target discount for each state. The calculation formula for the generalized advantage estimation method is shown below: ,in, As a discount factor, As a smoothing factor, For value network output, This is a round termination indicator; The total loss function is constructed based on the pruning objective of the PPO algorithm. The policy pruning loss limits the policy update magnitude, the value loss fits the state value, and the entropy loss improves the policy exploration. The policy network and value network are iteratively updated. After each preset training round, the performance of the current model is verified to determine whether the convergence condition is met, and the optimal policy network model that meets the convergence condition is saved.

9. A spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to claim 8, characterized in that, The formula for calculating the total loss function is as follows: Among them, entropy loss The calculation formula is as follows: , Value loss The calculation formula is as follows: , For state value, Accumulate returns for the target discount. For the expectation operator, the pruning loss The calculation formula is as follows: , For the clipping factor, The ratio of the action probabilities of the new strategy to those of the old strategy. The dominant function is calculated using the following formula: ,in, For the probability of the new strategy action, This represents the probability of actions under the old strategy.

10. A spacecraft approach control method based on the fusion of imitation learning and reinforcement learning according to claim 1, characterized in that, In step S4, the trained policy network model is used to output the control parameters for the spacecraft's optimal burnup orbital transfer in real time, including: During in-orbit deployment, the pre-trained policy network model and pre-stored normalized parameters are preloaded. The real-time relative state of the spacecraft relative to the target approach point is acquired in real time. The real-time relative state data is processed by normalized parameters to obtain normalized real-time state data. The normalized real-time state data is input into the optimization strategy network model to obtain the normalized control quantity output by the model. The normalized control quantity is then processed by inverse normalization and amplitude limiting, and then sent to the spacecraft propulsion system for execution. Closed-loop feedback iteration is performed with a fixed control cycle until the terminal approaches the required accuracy or the safety boundary is triggered.

Images