DQN Variable Dynamics Intelligent Decision-Making Method Assisted by Predictive Correction Expert Knowledge
By using the DQN variable-force intelligent decision-making method with the assistance of predictive correction expert knowledge, the problem of spacecraft struggling to safely avoid obstacles under insufficient aerodynamic forces has been solved, achieving efficient obstacle avoidance and terminal guidance, and improving the maneuverability and guidance accuracy of the spacecraft.
Patent Information
- Application Number
- CN202311366770.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-20
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-10-20
AI Technical Summary
Existing technologies are insufficient to effectively utilize aerodynamic forces during spacecraft reentry to maintain emergency lateral maneuvers and ensure safe obstacle avoidance, especially under conditions of low atmospheric density, making it difficult for spacecraft to complete guidance missions safely and efficiently.
The DQN variable-power intelligent decision-making method, aided by predictive correction expert knowledge, is adopted. By constructing a predictive correction module and a DQN network model, and combining angle of attack, roll angle and power gear decisions, obstacle avoidance and terminal guidance constraints are achieved, thus optimizing the power gear decision of the variable-power aircraft.
It improves the emergency maneuverability and guidance accuracy of the aircraft, ensures flight safety throughout the entire flight and meets terminal constraints, solves the problems of insufficient lateral maneuverability and poor guidance accuracy in traditional methods, and improves learning efficiency and convergence speed.
Smart Images

Figure CN117250990B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of spacecraft reentry guidance technology, specifically to a DQN variable dynamics intelligent decision-making method assisted by predictive correction expert knowledge. Background Technology
[0002] In recent years, spacecraft have been widely used in both military and civilian fields due to their superior characteristics such as high speed, long range, and strong emergency response capabilities. Meanwhile, spacecraft reentry guidance technology has attracted much attention in the aerospace field. Its main task is to design corresponding flight guidance laws based on dynamic equations under conditions of process and control constraints, so that the spacecraft meets the reentry terminal constraints at the landing point. However, pure aerodynamic control methods are difficult to achieve strong maneuverability and fast response speed to ensure safe obstacle avoidance. Researchers have adopted direct force / aerodynamic composite control strategies under high overload conditions to improve the safety and guidance accuracy of spacecraft flyaround maneuvers. Compared with traditional reentry guidance and control technologies, artificial intelligence technology, with its independence from precise modeling information and powerful optimization capabilities, allows variable-power spacecraft to intelligently select the optimal power level under complex flight environments and multi-mission requirements, ensuring optimal flight performance at all times. Therefore, the spacecraft reentry trajectory optimization problem based on reinforcement learning has solved the multi-dimensional and multi-level optimal decision-making problem of variable dynamic adjustment and trajectory planning, and has the advantages of high accuracy, strong stability and good robustness, which has attracted widespread attention from military researchers and experts at home and abroad.
[0003] Current research largely focuses on reentry trajectory optimization for unpowered spacecraft. This inevitably leads to a trade-off between the lateral overload required for obstacle avoidance and the longitudinal overload required to maintain the expected range. This contradiction is particularly pronounced in the initial reentry phase, where the thin atmosphere restricts aerodynamic forces, making it difficult to ensure the spacecraft completes its guidance mission safely and efficiently. Furthermore, most reinforcement learning methods are model-driven, making it difficult to achieve high precision and fast convergence of decision commands. For situations where aerodynamic forces are insufficient to maintain emergency lateral maneuvers for obstacle avoidance, the aforementioned strategies suffer from insufficient lateral maneuverability, slow convergence, and poor guidance accuracy. Therefore, employing predictive correction expert knowledge during the initial reentry phase and guiding a direct force / aerodynamic intelligent composite to enhance maneuverability and guidance accuracy is an effective way to compensate for the shortcomings of reentry trajectory optimization under unpowered guidance switching. This is of great significance for solving missions such as emergency avoidance and precision strikes for spacecraft. Summary of the Invention
[0004] To address the problem that insufficient aerodynamic forces of spacecraft are insufficient to maintain emergency lateral maneuvers and ensure safe obstacle avoidance, this invention provides a DQN variable dynamic intelligent decision-making method assisted by predictive correction expert knowledge.
[0005] This invention is achieved through the following technical solution: a DQN variable dynamic intelligent decision-making method assisted by predictive correction expert knowledge, comprising the following steps:
[0006] a) To address the issue of insufficient aerodynamic forces in spacecraft to maintain emergency lateral control and ensure safe obstacle avoidance, a variable-power spacecraft model and reentry guidance constraints are determined; details are as follows:
[0007] a1) To address the issue that the aerodynamic forces of a spacecraft are insufficient to maintain emergency lateral maneuvers and ensure safe obstacle avoidance, the Earth's rotation angular rate, which has a relatively minor impact, can be ignored. Based on Newton's second law, the reentry kinematic model of a variable-power spacecraft can be determined as follows:
[0008]
[0009] In the formula:
[0010] Where: r is the distance from the Earth's center, θ is the longitude of the spacecraft, φ is the latitude of the spacecraft, V is the speed of the spacecraft, and S... e ρ = ρ0e -h / 7200 Given the current atmospheric density, ρ0 = 1.225 kg / m³ 3 h is the current height; C L C is the lift coefficient. D S is the drag coefficient. ref R is the reference area of the aircraft. c is the Earth's equatorial radius, and m is the mass of the spacecraft;
[0011] a2) Determine reentry guidance constraints, including terminal constraints, obstacle constraints, control quantity constraints, and fuel constraints;
[0012] (1) Terminal constraints:
[0013] Depending on the different phases of the flight mission, variable-powered aircraft need to limit the terminal state of reentry guidance to ensure that it meets the handover conditions in the terminal guidance phase, as follows:
[0014]
[0015] Where: t f θ f φ f h f V f These are the terminal time, terminal longitude, terminal latitude, terminal altitude, and terminal speed, respectively.
[0016] (2) Obstacle constraints:
[0017] Areas formed by factors such as rugged mountains, electromagnetic interference, and enemy detection and interception are collectively referred to as obstacles. Here, obstacles are considered as infinitely tall cylinders, and aircraft can only avoid them by passing through the sides of the cylinder. Specifically, they are described as follows:
[0018]
[0019] Among them, D o,i θ is the distance from the aircraft to the center of obstacle i; o,i φ o,i R o,i These are the longitude, latitude, and radius of the center of the i-th obstacle, respectively.
[0020] (3) Control constraints:
[0021] Angle of attack α and roll angle σ are the main variables for controlling the trajectory of a variable-powered spacecraft during reentry guidance; angle of attack α affects aerodynamic parameters and changes the longitudinal flight trajectory of the spacecraft, using a three-segment equation related to flight speed:
[0022]
[0023] In the formula: α max and α min V1 and V2 are the maximum and minimum allowable angles of attack for the variable-powered spacecraft; V0 is the initial velocity magnitude; the roll angle σ mainly affects lateral maneuverability and changes the lateral flight trajectory of the spacecraft; therefore, in order to ensure the stability of the spacecraft's reentry trajectory, the roll angle σ of the spacecraft must satisfy the following boundary conditions:
[0024] |σ|≤σ max (5)
[0025] Where σ max This is the maximum allowable roll angle amplitude during flight;
[0026] (4) Fuel constraints:
[0027] The formula for fuel consumption of an aircraft is as follows:
[0028]
[0029] Where Ξ represents the fuel mass, k1 is a positive constant, and Ξ0 represents the initial fuel load mass of the aircraft.
[0030] b) Based on the results obtained in step a), design a DQN variable power intelligent decision-making architecture with the assistance of prediction and correction expert knowledge, including a prediction and correction module that can realize obstacle avoidance and terminal guidance constraints, and an intelligent decision-making model for aircraft power gear.
[0031] c) The prediction and correction module determines the tilt angle sign logic based on the relationship between the aircraft's heading angle and the line-of-sight angle threshold between the aircraft and the obstacle, thereby enabling obstacle avoidance and meeting terminal constraint requirements; specifically as follows:
[0032] The tilt angle weighting coefficient ω is corrected based on the range error, and the correction formula is as follows:
[0033]
[0034] In the formula: S go This represents the aircraft's ready-to-fly range; the smaller the tilt angle value, the greater the range. The tilt angle amplitude is updated as follows:
[0035] σ=ωσ min +(1-ω)σ max (8)
[0036] In the formula: σ is the tilt angle command, σ min For the minimum tilt angle, σ max This represents the maximum tilt angle.
[0037] When the aircraft enters the obstacle detection range, it needs to be given emergency avoidance capability; at this time, the tilt angle symbol is based on the aircraft (θ,φ) and the i-th obstacle (θ). o,i ,φ o,i The geometric relationships are determined as follows:
[0038]
[0039] In the formula: Let β be the line-of-sight angle between the aircraft and the center of the i-th obstacle; β = arcsin[D o,i / R o,i Let D be the angle between the line connecting the aircraft to the i-th obstacle and the tangent line. o,i R represents the distance between the aircraft and the i-th obstacle. o,i Let ψ be the radius of the i-th obstacle; d This is the threshold value for the aircraft's heading angle.
[0040] d) Based on the results obtained in steps b) and c), the intelligent decision-making problem of the aircraft's power gear is transformed into a Markov decision process. The state space is defined by the angle of attack, Mach, and the relative distance between the aircraft and the obstacle, and the action space is defined by the discrete power gear. A reward function that considers the collision probability and the terminal constraint deviation is designed, and a DQN network model with three-dimensional input and one-dimensional output is constructed to guide the agent to train and form the optimal power gear decision-making model for the variable-power aerospace vehicle.
[0041] d1) Combining the results obtained in steps b) and c), for the variable power intelligent decision-making problem assisted by expert knowledge in prediction and correction, the aircraft power gear decision problem is transformed into a Markov decision process. The normalized angle of attack command, Mach number, and obstacle center are selected to constitute the environmental state of the current control period t, which can be characterized as:
[0042]
[0043] In the formula: M=V / V s V is the current Mach number. s For the standard speed of sound, M max =V0 / V s V0 is the maximum Mach number during the entire flight, and V0 is the initial velocity of the aircraft. Defined in the following form:
[0044]
[0045] In the formula: R d R is the detection radius of the aircraft. When the distance between the aircraft and the center of the obstacle is less than R... d At that time, the obstacle avoidance procedure is executed. Since intelligent decision-making is based on the process under the control closed loop, the angle of attack and Mach number of the observed values in the state space are inconsistent with the order of magnitude of the distance from the aircraft to the center of the obstacle. Therefore, the observed values are normalized during the actual training process.
[0046] The discrete power gear is used as the action output A for the current control cycle t. t The power gear is the power command F;
[0047] d2) Design the terminal constraint deviation and the reward function considering the collision probability:
[0048] The reward and penalty values for the aircraft under different states are designed using terminal constraint bias and a potential field function with a saturation term. The mission objective is to successfully avoid obstacles by selecting the aircraft's power level. The reward function R for the current control period t is... t Specifically, it can be characterized as:
[0049] R t =r1+r2+r3+r4 (12)
[0050] Where r1 is the reward function for the terminal constraint deviation; the closer the aircraft is to the terminal constraint, the closer it is to the target point, and therefore the larger the reward value is obtained, as designed below:
[0051]
[0052] In the formula, L max The range from the starting point to the target point is defined to normalize the reward value and improve the convergence of the learning algorithm; S des and M des These are the minimum values for range and Mach tolerance deviation, respectively; R1 is the additional bonus value given to the aircraft when its speed and range are within the tolerance deviation range during training, guiding the aircraft to reach the terminal state more accurately.
[0053] Furthermore, to encourage the aircraft to achieve smaller landing point and terminal velocity errors, the following sparse rewards r2 and r3 are designed:
[0054]
[0055]
[0056] In the formula, R2 and R3 are preset positive constants; when the range and speed of the current aircraft are within the corresponding minimum tolerance deviation range, the corresponding positive reward term will be added to the reward function to encourage the aircraft to tend toward the terminal constraint and be able to stay within the minimum deviation tolerance range;
[0057] Furthermore, to address the difficulty of conventional aircraft in emergency obstacle avoidance during the initial reentry phase in a thin atmosphere, a reward function r4 with a saturation term is designed based on the artificial potential field method:
[0058]
[0059] Among them, L o R4 is a normalization factor used to adjust the sparsity of the reward; R4 > 0 is the penalty value for colliding with an obstacle; when the aircraft is outside the obstacle detection radius, there is no risk of the aircraft being intercepted, and the reward is 0; when the aircraft enters the detection radius, r4 will be activated during training to guide the aircraft to gradually move away from the detection range, that is, the closer to the center of the obstacle, the greater the negative reward; when the aircraft cannot avoid the obstacle, the reward function saturates, and the output is a constant negative number -R4.
[0060] d3) Based on the mission requirements of variable-powered aircraft, construct a DQN network structure with three-dimensional input and one-dimensional output:
[0061] The state of the intelligent variable-powered aircraft consists of three variables, therefore the network input contains three neurons; the middle layer consists of two hidden layers, each containing 32 neurons, and each hidden layer is activated using the ReLU function.
[0062] ReLU(x) = max{0,x} (17)
[0063] Where x is a neuron in the hidden layer, and the last hidden layer is directly connected to the one-dimensional output layer, i.e., action A;
[0064] d4) Construct a DQN variable-power intelligent decision-making method assisted by predictive correction expert knowledge, guide the agent to train, and derive the optimal gear decision scheme for the variable-power spacecraft:
[0065] To ensure the accuracy of the estimated Q value, based on the current state S t And Action A t The current network parameters are updated using gradient descent, and a loss function L containing a discount factor η is defined:
[0066]
[0067] in, and S represents the action-value function estimated using the value network and the target network, respectively. t+1 and A t+1 For the next state and the next action, and These are the weights of the value network and the target network, respectively; for each action A executed... t Both can maximize the value function; therefore, an ε-greedy algorithm is used for policy selection.
[0068]
[0069] Finally, the DQN training process involves iterative learning through an "action-environment reward-policy" model, continuously training the agent based on power level and flight status. First, the roll angle command obtained from predictive correction expert knowledge is transmitted to the aircraft to obtain the flight status S under the current command. t Secondly, initialize the parameters of the experience pool and network Q, and apply the magnitude of the Q network parameters to the target Q network. Select random samples and then use an ε-greedy strategy to explore strategies, taking actions A that maximize the Q value and are suitable and feasible. t Interact with the environment based on the selected action to obtain feedback rewards (reward item R). t Next, obtain the action through A. t The next state S t And determine whether the mode requirements of the intelligent aircraft are met, and use the obtained experience data (S) t At ,R t ,S t+1 The parameters are stored in the experience replay pool and updated using gradient descent based on the loss function L. After each N steps, the magnitude of the Q network parameters at that time needs to be transferred to the target network. Finally, the optimal dynamic decision is obtained, which effectively improves the emergency maneuverability and guidance accuracy of the aircraft.
[0070] Compared with existing technologies, this invention has the following advantages: The innovative aspects of the DQN variable-power intelligent decision-making method assisted by predictive correction expert knowledge provided by this invention are as follows: 1) The proposed method is essentially a data / model linked reentry planning algorithm. By introducing predictive correction expert knowledge commonly used in the guidance field, variable-power trajectory planning is described as a power gear decision problem. Compared with using reinforcement learning to train angle of attack, bank angle, and power gear, it greatly reduces the optimization dimension and action space, while avoiding the problems of low learning efficiency and slow convergence speed under pure data-driven approaches; 2) Compared with predictive correction guidance that can avoid obstacles, the proposed strategy can enhance the lateral maneuverability in the initial reentry phase through gear shifting decisions, ensuring flight safety throughout the entire flight and meeting terminal constraints. This invention solves the multi-dimensional and multi-level optimal decision-making problem of variable-power adjustment and trajectory planning: by introducing predictive correction expert knowledge commonly used in the reentry guidance field, the optimization dimension and action space are greatly reduced, improving the learning efficiency and convergence speed of reinforcement learning; by adopting optimal variable-power intelligent decision-making, the shortcomings of traditional guidance laws, such as insufficient lateral maneuverability and poor guidance accuracy, are improved. Attached Figure Description
[0071] Figure 1 This is a structural block diagram of the present invention.
[0072] Figure 2 This is a diagram of the DQN network structure involved in the present invention.
[0073] Figure 3 This is a flowchart of the DQN algorithm involved in the present invention.
[0074] Figure 4 This is a diagram showing the training results of DQN in a specific embodiment of the present invention.
[0075] Figure 5 This is a simulation result diagram of the lateral trajectory of a variable-powered spacecraft according to a specific embodiment of the present invention.
[0076] Figure 6 The diagram shows the simulation results of the aircraft's power gears according to a specific embodiment of the present invention.
[0077] Figure 7 This is a curve showing the change in remaining fuel mass according to a specific embodiment of the present invention. Detailed Implementation
[0078] The present invention will be further described below with reference to specific embodiments.
[0079] A predictive correction expert knowledge-assisted DQN variable dynamic intelligent decision-making method, the structural block diagram of which is as follows: Figure 1 As shown, it includes the following steps:
[0080] a) To address the issue of insufficient aerodynamic forces in spacecraft to maintain emergency lateral control and ensure safe obstacle avoidance, a variable-power spacecraft model and reentry guidance constraints are determined; details are as follows:
[0081] a1) To address the issue that the aerodynamic forces of a spacecraft are insufficient to maintain emergency lateral maneuvers and ensure safe obstacle avoidance, the Earth's rotation angular rate, which has a relatively minor impact, can be ignored. Based on Newton's second law, the reentry kinematic model of a variable-power spacecraft can be determined as follows:
[0082]
[0083] In the formula:
[0084] Where: r is the distance from the Earth's center, θ is the longitude of the spacecraft, φ is the latitude of the spacecraft, V is the speed of the spacecraft, and S... e ρ = ρ0e -h / 7200 Given the current atmospheric density, ρ0 = 1.225 kg / m³ 3 h is the current height; C L C is the lift coefficient. D S is the drag coefficient. ref R is the reference area of the aircraft. c is the Earth's equatorial radius, and m is the mass of the spacecraft;
[0085] a2) Determine reentry guidance constraints, including terminal constraints, obstacle constraints, control quantity constraints, and fuel constraints;
[0086] (1) Terminal constraints:
[0087] Depending on the different phases of the flight mission, variable-powered aircraft need to limit the terminal state of reentry guidance to ensure that it meets the handover conditions in the terminal guidance phase, as follows:
[0088]
[0089] Where: t f θ f φ f h f V fThese are the terminal time, terminal longitude, terminal latitude, terminal altitude, and terminal speed, respectively.
[0090] (2) Obstacle constraints:
[0091] Areas formed by factors such as rugged mountains, electromagnetic interference, and enemy detection and interception are collectively referred to as obstacles. Here, obstacles are considered as infinitely tall cylinders, and aircraft can only avoid them by passing through the sides of the cylinder. Specifically, they are described as follows:
[0092]
[0093] Among them, D o,i θ is the distance from the aircraft to the center of obstacle i; o,i φ o,i R o,i These are the longitude, latitude, and radius of the center of the i-th obstacle, respectively.
[0094] (3) Control constraints:
[0095] Angle of attack α and roll angle σ are the main variables for controlling the trajectory of a variable-powered spacecraft during reentry guidance; angle of attack α affects aerodynamic parameters and changes the longitudinal flight trajectory of the spacecraft, using a three-segment equation related to flight speed:
[0096]
[0097] In the formula: α max and α min V1 and V2 are the maximum and minimum allowable angles of attack for the variable-powered spacecraft; V0 is the initial velocity magnitude; the roll angle σ mainly affects lateral maneuverability and changes the lateral flight trajectory of the spacecraft; therefore, in order to ensure the stability of the spacecraft's reentry trajectory, the roll angle σ of the spacecraft must satisfy the following boundary conditions:
[0098] |σ|≤σ max (5)
[0099] Where σ max This is the maximum allowable roll angle amplitude during flight;
[0100] (4) Fuel constraints:
[0101] The formula for fuel consumption of an aircraft is as follows:
[0102]
[0103] Where Ξ represents the fuel mass, k1 is a positive constant, and Ξ0 represents the initial fuel load mass of the aircraft.
[0104] b) Based on the results obtained in step a), design a DQN variable power intelligent decision-making architecture with the assistance of prediction and correction expert knowledge, including a prediction and correction module that can realize obstacle avoidance and terminal guidance constraints, and an intelligent decision-making model for aircraft power gear.
[0105] c) The prediction and correction module determines the tilt angle sign logic based on the relationship between the aircraft's heading angle and the line-of-sight angle threshold between the aircraft and the obstacle, thereby enabling obstacle avoidance and meeting terminal constraint requirements; specifically as follows:
[0106] The tilt angle weighting coefficient ω is corrected based on the range error, and the correction formula is as follows:
[0107]
[0108] In the formula: S go This represents the aircraft's ready-to-fly range; the smaller the tilt angle value, the greater the range. The tilt angle amplitude is updated as follows:
[0109] σ=ωσ min +(1-ω)σ max (8)
[0110] In the formula: σ is the tilt angle command, σ min For the minimum tilt angle, σ max This represents the maximum tilt angle.
[0111] When the aircraft enters the obstacle detection range, it needs to be given emergency avoidance capability; at this time, the tilt angle symbol is based on the aircraft (θ,φ) and the i-th obstacle (θ). o,i ,φ o,i The geometric relationships are determined as follows:
[0112]
[0113] In the formula: Let β be the line-of-sight angle between the aircraft and the center of the i-th obstacle; β = arcsin[D o,i / R o,i Let D be the angle between the line connecting the aircraft to the i-th obstacle and the tangent line. o,i R represents the distance between the aircraft and the i-th obstacle. o,i Let ψ be the radius of the i-th obstacle; d This is the threshold value for the aircraft's heading angle.
[0114] d) Based on the results obtained in steps b) and c), the intelligent decision-making problem of the aircraft's power gear is transformed into a Markov decision process. The state space is defined by the angle of attack, Mach, and the relative distance between the aircraft and the obstacle, and the action space is defined by the discrete power gear. A reward function that considers the collision probability and the terminal constraint deviation is designed, and a DQN network model with three-dimensional input and one-dimensional output is constructed to guide the agent to train and form the optimal power gear decision-making model for the variable-power aerospace vehicle.
[0115] d1) Combining the results obtained in steps b) and c), for the variable power intelligent decision-making problem assisted by expert knowledge in prediction and correction, the aircraft power gear decision problem is transformed into a Markov decision process. The normalized angle of attack command, Mach number, and obstacle center are selected to constitute the environmental state of the current control period t, which can be characterized as:
[0116]
[0117] In the formula: M=V / V s V is the current Mach number. s For the standard speed of sound, M max =V0 / V s V0 is the maximum Mach number during the entire flight, and V0 is the initial velocity of the aircraft. Defined in the following form:
[0118]
[0119] In the formula: R d R is the detection radius of the aircraft. When the distance between the aircraft and the center of the obstacle is less than R... d At that time, the obstacle avoidance procedure is executed. Since intelligent decision-making is based on the process under the control closed loop, the angle of attack and Mach number of the observed values in the state space are inconsistent with the order of magnitude of the distance from the aircraft to the center of the obstacle. Therefore, the observed values are normalized during the actual training process.
[0120] The discrete power gear is used as the action output A for the current control cycle t. t The power gear is the power command F;
[0121] d2) Design the terminal constraint deviation and the reward function considering the collision probability:
[0122] The reward and penalty values for the aircraft under different states are designed using terminal constraint bias and a potential field function with a saturation term. The mission objective is to successfully avoid obstacles by selecting the aircraft's power level. The reward function R for the current control period t is... t Specifically, it can be characterized as:
[0123] R t =r1+r2+r3+r4 (12)
[0124] Where r1 is the reward function for the terminal constraint deviation; the closer the aircraft is to the terminal constraint, the closer it is to the target point, and therefore the larger the reward value is obtained, as designed below:
[0125]
[0126] In the formula, L max The range from the starting point to the target point is defined to normalize the reward value and improve the convergence of the learning algorithm; S des and M des These are the minimum values for range and Mach tolerance deviation, respectively; R1 is the additional bonus value given to the aircraft when its speed and range are within the tolerance deviation range during training, guiding the aircraft to reach the terminal state more accurately.
[0127] Furthermore, to encourage the aircraft to achieve smaller landing point and terminal velocity errors, the following sparse rewards r2 and r3 are designed:
[0128]
[0129]
[0130] In the formula, R2 and R3 are preset positive constants; when the range and speed of the current aircraft are within the corresponding minimum tolerance deviation range, the corresponding positive reward term will be added to the reward function to encourage the aircraft to tend toward the terminal constraint and be able to stay within the minimum deviation tolerance range;
[0131] Furthermore, to address the difficulty of conventional aircraft in emergency obstacle avoidance during the initial reentry phase in a thin atmosphere, a reward function r4 with a saturation term is designed based on the artificial potential field method:
[0132]
[0133] Among them, L o R4 is a normalization factor used to adjust the sparsity of the reward; R4 > 0 is the penalty value for colliding with an obstacle; when the aircraft is outside the obstacle detection radius, there is no risk of the aircraft being intercepted, and the reward is 0; when the aircraft enters the detection radius, r4 will be activated during training to guide the aircraft to gradually move away from the detection range, that is, the closer to the center of the obstacle, the greater the negative reward; when the aircraft cannot avoid the obstacle, the reward function saturates, and the output is a constant negative number -R4.
[0134] d3) Based on the mission requirements of variable-powered aircraft, construct a three-dimensional input, one-dimensional output DQN network structure, such as... Figure 2 As shown:
[0135] The state of the intelligent variable-powered aircraft consists of three variables, therefore the network input contains three neurons; the middle layer consists of two hidden layers, each containing 32 neurons, and each hidden layer is activated using the ReLU function.
[0136] ReLU(x) = max{0,x} (17)
[0137] Where x is a neuron in the hidden layer, and the last hidden layer is directly connected to the one-dimensional output layer, i.e., action A;
[0138] d4) Construct a DQN variable-power intelligent decision-making method assisted by predictive correction expert knowledge, guide the agent to train, and derive the optimal gear decision scheme for the variable-power spacecraft:
[0139] To ensure the accuracy of the estimated Q value, based on the current state S t And Action A t The current network parameters are updated using gradient descent, and a loss function L containing a discount factor η is defined:
[0140]
[0141] in, and S represents the action-value function estimated using the value network and the target network, respectively. t+1 and A t+1 For the next state and the next action, and These are the weights of the value network and the target network, respectively; for each action A executed... t Both can maximize the value function; therefore, an ε-greedy algorithm is used for strategy selection.
[0142]
[0143] Finally, the DQN training process is as follows: Figure 3 As shown, the agent learns iteratively through an "action-environmental reward-policy" model, continuously training itself based on power level and flight status. First, the tilt angle command obtained from predictive correction expert knowledge is transmitted to the aircraft to obtain the flight status S under the current command. t Secondly, initialize the parameters of the experience pool and network Q, and apply the magnitude of the Q network parameters to the target Q network. Select random samples and then use an ε-greedy strategy to explore strategies, taking actions A that maximize the Q value and are suitable and feasible. t Interact with the environment based on the selected action to obtain feedback rewards (reward item R). t Next, obtain the action through A. t The next state S tAnd determine whether the mode requirements of the intelligent aircraft are met, and use the obtained experience data (S) t A t ,R t ,S t+1 The parameters are stored in the experience replay pool and updated using gradient descent based on the loss function L. After each N steps, the magnitude of the Q network parameters at that time needs to be transferred to the target network. Finally, the optimal dynamic decision is obtained, which effectively improves the emergency maneuverability and guidance accuracy of the aircraft.
[0144] e) The specific parameter settings for the simulation of the proposed DQN variable dynamic intelligent decision-making method assisted by expert knowledge for prediction and correction are as follows:
[0145] e1) The power levels provided by the jet engine of the variable-power spacecraft are set to four, with the power levels and the total mass of the initial fuel of the spacecraft all within reasonable preset parameters. The radius of the Earth's equator and the gravitational acceleration are R and R, respectively. c =6378.14km and g0=9.81m / s 2 .
[0146] e2) The DQN network parameters for variable dynamic intelligent decision-making are shown in Table 1.
[0147] Table 1 DQN Network Parameters
[0148]
[0149] e3) Determine the simulation initialization parameters, including: variable-power spacecraft parameters, initial and terminal states, and obstacle information:
[0150] The mass, reference area, initial parameters, and terminal parameters of the variable-power spacecraft body are all preset within a reasonable range. Other simulation initialization parameters are shown in Table 2.
[0151] Table 2 Simulation Initialization Parameters
[0152]
[0153] e4) Determine the controller's adjustment parameters.
[0154] Minimum and maximum angles of attack α allowed during the motion of a variable-powered spacecraft min and α max The two preset speed nodes V1 and V2 are both within a reasonable range. The remaining controller parameters are shown in Table 3.
[0155] Table 3 Controller Parameters
[0156]
[0157] e5) Simulation results are as follows Figures 4 to 7 As shown, Figure 4 This is a training graph for DQN; Figure 5 For the lateral trajectory of a variable-powered spacecraft, Figure 6 and Figure 7 The figures show the curves representing the changes in the aircraft's power level and remaining fuel mass, respectively.
[0158] The scope of protection claimed by this invention is not limited to the specific embodiments described above. Moreover, for those skilled in the art, this invention can have various modifications and alterations. Any modifications, improvements, and equivalent substitutions made within the concept and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A DQN variable dynamic intelligent decision-making method assisted by expert knowledge for prediction and correction, characterized in that: Includes the following steps: a) Determine the model of the variable-powered spacecraft and the reentry guidance constraints; b) Based on the results obtained in step a), design a DQN variable power intelligent decision-making architecture with the assistance of predictive correction expert knowledge, including a predictive correction module that can realize obstacle avoidance and terminal guidance constraints, and an intelligent decision-making model for aircraft power gear. c) The prediction and correction module determines the sign logic of the roll angle based on the relationship between the aircraft's heading angle and the line-of-sight angle threshold between the aircraft and the obstacle, thereby enabling the aircraft to fly around obstacles and meeting terminal constraint requirements; specifically as follows: Weighting coefficient of tilt angle based on range error The correction is performed using the following formula: (7) In the formula: This represents the aircraft's ready-to-fly range; the smaller the tilt angle value, the greater the range. The tilt angle amplitude is updated as follows: (8) In the formula: This is the yaw angle command. This is the minimum tilt angle. This represents the maximum tilt angle. When the aircraft enters the obstacle detection range, it is given emergency avoidance capability; at this time, the roll angle symbol is based on the aircraft. With the i-th obstacle The geometric relationships are determined as follows: (9) In the formula: Let be the line-of-sight angle between the aircraft and the center of the i-th obstacle; Let be the angle between the line connecting the aircraft to the i-th obstacle and the tangent. This represents the distance between the aircraft and the i-th obstacle. Let be the radius of the i-th obstacle; The threshold for the aircraft's heading angle; d) Based on the results obtained in steps b) and c), the intelligent decision-making problem of the aircraft's power gear is transformed into a Markov decision process. The state space is defined by the angle of attack, Mach number, and the relative distance between the aircraft and the obstacle. The action space is defined by the discrete power gear. A reward function that considers the collision probability and the terminal constraint deviation is designed. A DQN network model with three-dimensional input and one-dimensional output is constructed to guide the agent to train and form the optimal power gear decision-making model for the variable-power aerospace vehicle.
2. The DQN variable dynamic intelligent decision-making method with expert knowledge assistance for prediction and correction as described in claim 1, characterized in that: Step a) is as follows: a1) Ignoring the Earth's rotation angular rate, the reentry kinematic model of the variable-power spacecraft can be determined according to Newton's second law as follows: (1) In the formula: ; Where: r is the distance from the Earth's center. The longitude of the aircraft. Where V is the latitude of the aircraft and V is the speed of the aircraft. This represents the remaining range at the location of the aircraft. The flight path angle of the aircraft. The heading angle of the aircraft. L is the aircraft's tilt angle, D is the lift, and F is the drag. Given the current atmospheric density, h is the current height; The lift coefficient, The drag coefficient, For the aircraft reference area, Where is the Earth's equatorial radius, and m is the mass of the spacecraft; a2) Determine reentry guidance constraints, including terminal constraints, obstacle constraints, control quantity constraints, and fuel constraints; (1) Terminal constraints: Depending on the different phases of the flight mission, the variable-powered aircraft imposes restrictions on the terminal state of reentry guidance to ensure that it meets the handover conditions in the terminal guidance phase, as follows: (2) in: , , , , These are the terminal time, terminal longitude, terminal latitude, terminal altitude, and terminal speed, respectively. (2) Obstacle constraints: The obstacle is considered as an infinitely tall cylinder; the aircraft can only avoid it by moving along either side of the cylinder. Specifically: (3) in, For aircraft to obstacles Distance from the center; , , The first The longitude, latitude, and radius of the center of each obstacle; (3) Control quantity constraints: Angle of Attack and tilt angle Angle of attack is a crucial variable in the control trajectory of a variable-powered aircraft during reentry guidance. Influencing aerodynamic parameters and altering the longitudinal flight trajectory of the spacecraft, a three-stage trajectory related to flight speed is selected: (4) In the formula: and The maximum and minimum angle of attack allowed for a variable-powered aircraft; and Two preset velocity nodes; Initial velocity magnitude; tilt angle It primarily affects lateral maneuverability, altering the spacecraft's lateral flight trajectory; therefore, to ensure the stability of the spacecraft's reentry trajectory, the spacecraft's tilt angle... The following boundary conditions must be met: (5) in This is the maximum allowable roll angle amplitude during flight; (4) Fuel constraints: The formula for fuel consumption of an aircraft is as follows: (6) in, For fuel quality, For positive integers, Initial fuel load mass of the aircraft.
3. The DQN variable dynamic intelligent decision-making method with expert knowledge assistance for prediction and correction as described in claim 2, characterized in that: Step d) is as follows: d1) Combining the results obtained in steps b) and c), the aircraft power gear decision problem is transformed into a Markov decision process. The normalized angle of attack, Mach number, and obstacle center are selected to constitute the environmental state of the current control period t, which can be characterized as: (10) In the formula: Given the current Mach number, For standard speed of sound, This is the maximum Mach number during the entire flight. The initial speed of the aircraft. Defined in the following form: (11) In the formula: The detection radius of the aircraft is defined as the distance between the center of the aircraft and the center of the obstacle being less than [missing information]. At that time, the obstacle avoidance procedure is executed; during actual training, the observation values are normalized. The discrete power gear is used as the action output for the current control cycle t. The power gear is the power command F; d2) Design the terminal constraint bias and the reward function considering the collision probability: The reward value of the aircraft under different states is designed using terminal constraint bias and a potential field function with saturation terms. The mission objective is to successfully avoid obstacles by selecting the aircraft's power level. The reward function for the current control period t is... Specifically, it can be characterized as: (12) in, The reward function is the terminal constraint deviation; the closer the aircraft is to the terminal constraint, the closer it is to the target point, and therefore the larger the reward value. The design is as follows: (13) In the formula, The range from the starting point to the target point is used to normalize the reward value and improve the convergence of the learning algorithm. and These are the minimum values for range and Mach tolerance deviation, respectively; An additional bonus value is given to the aircraft when its speed and range are within the tolerance range during training, guiding the aircraft to reach the terminal state more accurately. In addition, to encourage the aircraft to achieve smaller landing point and terminal velocity errors, the following sparse reward is designed. and : (14) (15) In the formula, and These are preset normal values; when the current range and speed of the aircraft are within the corresponding minimum tolerance deviation range, the corresponding positive reward term will be added to the reward function to encourage the aircraft to tend toward the terminal constraint and be able to stay within the minimum deviation tolerance range; Furthermore, to address the difficulty of conventional aircraft in emergency obstacle avoidance during the initial reentry phase in a thin atmosphere, a reward function with a saturation term is designed based on the artificial potential field method. : (16) in, It is a normalization factor used to adjust the sparsity of rewards; This is the penalty value for colliding with an obstacle; when the aircraft is outside the obstacle's detection radius, there is no risk of interception, and the reward is 0; when the aircraft enters the detection radius, ... During training, the reward function will be activated to guide the aircraft to gradually move away from its detection range; that is, the closer the aircraft is to the center of the obstacle, the greater the negative reward. When the aircraft is unable to avoid the obstacle, the reward function saturates, and the output becomes a constant negative number. ; d3) Based on the mission requirements of variable-powered aircraft, construct a DQN network structure with three-dimensional input and one-dimensional output: The state of the intelligent variable-powered aircraft consists of three variables, therefore the network input contains three neurons; the middle layer consists of two hidden layers, each containing 32 neurons, and each hidden layer uses... Function activation: (17) in, These are neurons in the hidden layer, and the last hidden layer is directly connected to the one-dimensional output layer, i.e., action A. d4) Construct a DQN variable-dynamic intelligent decision-making method assisted by predictive correction expert knowledge, guide the agent to train, and derive the optimal gear decision scheme for the variable-dynamic spacecraft: To ensure the accuracy of the estimated Q value, based on the current state and actions The current network parameters are updated using gradient descent, and a discount factor is defined. loss function : (18) in, and These are the action-value functions estimated using the value network and the target network, respectively. and For the next state and the next action, and These are the weights of the value network and the target network, respectively; for each action executed. Both can maximize the value function, and are used when choosing a strategy. Greedy Algorithm: (19) Finally, the DQN training process involves iterative learning through "action-environment reward-policy," meaning the agent is continuously trained based on power level and flight status. First, the roll angle command obtained from predictive correction expert knowledge is transmitted to the aircraft to obtain the flight status under the current command. Secondly, initialize the parameters of the experience pool and network Q, assign the Q network parameters to the target Q network, select random samples, and then use... - A greedy algorithm uses strategic exploration to take actions that maximize the Q-value and are both appropriate and feasible. The system interacts with the environment based on the selected action to obtain a reward function. Next, obtain the action process. The next state And determine whether the mode requirements of the intelligent aircraft are met, and use the acquired experience data. The data is stored in the experience replay pool, and the Q network parameters are updated using gradient descent based on the loss function L. After every N steps, the current Q network parameters need to be assigned to the target Q network. Finally, the optimal dynamic decision is obtained.
