A fast trajectory optimization method based on pseudo-spectral grid adaptive construction based on reinforcement learning

Abstract

A method for rapid trajectory optimization based on pseudo-spectral grid adaptive construction of reinforcement learning belongs to the field of aircraft control technology. The method is as follows: construct an aircraft dynamics and kinematics model, set the initial value and constraints of the flight trajectory, and construct an aircraft trajectory optimization problem based on the pseudo-spectral method; calculate the residual matrix of the midpoints of two adjacent matching points, and use the second norm of the residual matrix as a network evaluation method for evaluation; obtain a new Online-Actor network through DDPG reinforcement learning to design the pseudo-spectral discrete grid, convert the continuous trajectory optimization problem into a nonlinear programming problem, and use the sequential quadratic programming method to solve it. The present invention uses reinforcement learning to complete the rapid training of neural networks, and quickly and accurately constructs pseudo-spectral discrete grids based on neural networks, which solves the problem of long solution time caused by repeated iterations, and then completes the rapid optimization of aircraft trajectories.

Description

Technical Field

[0001] The invention relates to a trajectory fast optimization method based on reinforcement learning pseudo-spectral grid adaptive construction, belonging to the technical field of aircraft control. Background Art

[0002] Due to the increasing complexity of the aircraft flight environment and the expansion of the flight airspace speed range, the nonlinearity caused by the aerodynamic force in the aircraft dynamics model has become more and more serious. As a result, it is difficult to simply construct a pseudo-spectral discrete grid manually when using the pseudo-spectral method to optimize the aircraft trajectory. Instead, it is necessary to design a pseudo-spectral discrete grid through repeated iterations in order to clearly describe it through the grid. This process is often time-consuming and difficult to meet the needs of online applications. Summary of the invention

[0003] In order to solve the problems existing in the background technology, the present invention provides a trajectory fast optimization method based on reinforcement learning pseudo-spectral grid adaptive construction.

[0004] To achieve the above object, the present invention adopts the following technical solution: a fast trajectory optimization method based on reinforcement learning pseudo-spectral grid adaptive construction, the method comprising the following steps:

[0005] S1: Construct the dynamics and kinematics model of the aircraft, set the initial value and constraints of the flight trajectory, and construct the aircraft trajectory optimization problem based on the pseudo-spectral method;

[0006] S2: Calculate the residual matrix R of the midpoints of two adjacent collocation points, and use the bi-norm ||R||2 of the residual matrix R as the network evaluation method for evaluation;

[0007] S3: Get a new Online-Actor network through DDPG reinforcement learning;

[0008] S4: The pseudo-spectral discrete grid is designed through the network trained by S3, the continuous trajectory optimization problem is transformed into a nonlinear programming problem, and the sequential quadratic programming method is used to solve it.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] The present invention utilizes reinforcement learning to complete the rapid training of neural networks, adopts intelligent neural networks to characterize the differences and laws of aircraft nonlinear problems, and then quickly and accurately constructs pseudo-spectral discrete grids based on neural networks. Reinforcement learning is used to replace the adaptive solution iterative process of the pseudo-spectral discrete grid, thereby solving the problem of long solution time caused by repeated iterations, thereby fully utilizing the rapidity of the pseudo-spectral trajectory optimization solution, and further completing the rapid optimization of the aircraft trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0012] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0013] The application of artificial intelligence in aircraft is becoming more and more mature. Therefore, the accuracy and cost advantages brought by reinforcement learning can be used to predict and analyze the grid based on the flight state, which provides the possibility for the rapid construction of pseudo-spectral discrete grids and the rapid optimization of aircraft trajectories. A rapid trajectory optimization method based on reinforcement learning pseudo-spectral grid adaptive construction is provided, which includes the following steps:

[0014] S1: Construct the dynamics and kinematics model of the aircraft, set the initial value and constraints of the flight trajectory, and construct the aircraft trajectory optimization problem based on the pseudo-spectral method;

[0015] The aircraft dynamics and kinematics model is as follows:

[0016]

[0017] In formula (1):

[0018] is the first derivative of r, r is the distance from the center of the earth;

[0019] is the first derivative of θ, θ is the longitude;

[0020] for The first derivative of is latitude;

[0021] is the first-order derivative of V, V is the flight speed of the aircraft;

[0022] is the first derivative of γ, γ is the flight path angle;

[0023] is the first-order derivative of ψ, ψ is the flight heading angle;

[0024] σ is the tilt angle;

[0025] L is the aerodynamic lift;

[0026] D is the aerodynamic drag;

[0027] and:

[0028]

[0029] In formula (2):

[0030] C L (α,Ma) is the lift coefficient determined by the angle of attack α and the Mach number Ma;

[0031] C D (α,Ma) is the drag coefficient determined by the angle of attack α and the Mach number Ma;

[0032] q is the dynamic pressure;

[0033] S ref is the reference area of ​​the aircraft;

[0034] And set the above-mentioned state quantities, namely, the distance from the center of the earth of the aircraft, longitude, latitude, flight speed, flight path angle, and flight heading angle, as well as the process and terminal upper and lower limits.

[0035] S2: Calculate the residual matrix R of the midpoints of two adjacent collocation points, and use the bi-norm ||R||2 of the residual matrix R as the network evaluation method for evaluation;

[0036] S201: Divide the fixed aircraft trajectory into N interval segments, and the time of each interval is [t k ,t k+1 ], k = 1, ..., N, the number of points in each time interval is [M1, M2 ..., M N ], select two adjacent matching points in a single segment (t i ,t i+1 ), that is,

[0037]

[0038] S202: Define state quantity s t for:

[0039]

[0040] S203: Define the action amount a t for:

[0041] a t =[M1,…,M N ] (4)

[0042] S204: Define the reward function as:

[0043] r t =-||R|| (5)

[0044] S205: Get the state variable of the midpoint of two adjacent collocation points by Lagrange interpolation method That is, the control variables of the aircraft's distance from the center of the earth, longitude, latitude, speed, flight path angle, heading angle, and the midpoint of two adjacent points That is, angle of attack and angle of roll;

[0045] S206: Calculate the residual matrix R of the midpoints of two adjacent collocation points by using the approximate values ​​of the state variables and control variables at the midpoints of the discrete collocation points in each interval of time:

[0046]

[0047] In formula (6):

[0048] is the state variable;

[0049] F is the aircraft dynamics and kinematics model in equation (1);

[0050] is the approximate value of the state variable at the midpoint of the discrete collocation;

[0051] is the approximate value of the control variable at the midpoint of the discrete collocation;

[0052] S207: Using the binary norm ||R||2 of the residual matrix R as a network evaluation method, that is, the smaller the binary norm ||R||2 of the residual matrix R is, the better the evaluation is.

[0053] S3: Get a new Online-Actor network through DDPG reinforcement learning;

[0054] S301: Initialize the agent in DDPG reinforcement learning, including four neural networks, namely Online-Actor network μ(s t ), Target-Actor network μ′(s t ), Online-Critic Network And Target-Critic Network Setting up the Online-Actor network μ(s t ) and Target-Actor network μ′(s t ) have the same parameters and structures; set the Online-Critic network and the Target-Critic network The parameters and structures are the same; and the design parameters [M1,…,M N ];

[0055] S302: Set the number of rounds K of DDPG reinforcement learning. In each round, randomly initialize the design parameters M1,…M N ; Randomly initialize the aircraft state quantity instruction The DDPG reinforcement learning algorithm is used to train the parameters of the agent and obtain the reward function sequence r obtained by the agent after training. t = {r t1 ,r t2 ,...,r tN} and the corresponding design parameter sequence;

[0056] S303: Calculate the derivative value of each parameter in the design parameter sequence by forward difference

[0057]

[0058] In formula (7):

[0059] Represents the εth design parameter M ε ,ε=1,…,N, the derivative value of the reward function of the i,jth design parameter in the design sequence;

[0060] S304: Update the design parameters in the reward function based on formula (7):

[0061]

[0062] In formula (8):

[0063] is the learning rate;

[0064] S305: Design parameters [M1,…,M] updated based on equation (8) N ], and return to S302; waiting for the design parameters [M1,…,M N ] After the adjustment process converges, enter S306;

[0065] S306: Based on the design parameters obtained in S304 [M1,…,M N ] gives the designed reward function; based on the reward function, returns to S302, sets the number of DDPG reinforcement learning rounds K = 1, and obtains a new Online-Actor network μ(s t ).

[0066] S4: The pseudo-spectral discrete grid is designed through the network trained by S3, the continuous trajectory optimization problem is transformed into a nonlinear programming problem, and the sequential quadratic programming method is used to solve it.

[0067] S401: Based on the new Online-Actor network μ(st ), using the aircraft time series state quantity to obtain the optimal discrete grid [M1,…,M N ], and based on this grid, the trajectory optimization problem is described: the state variable X at the discrete point i (i=0,...,K) and control variable U k (k=1,...,K) and the initial time t0 and the final time t f , so that the performance index is minimized and the constraints of the dynamic equation, terminal state constraints, control quantity constraints, boundary conditions of the original optimal control problem and process constraints are satisfied.

[0068] The boundary conditions of the original optimal control problem are:

[0069] φ(X0,t0,X f ,t f )=0 (9)

[0070] In formula (9):

[0071] φ is the constraint function of the upper and lower bounds of the terminal state quantity;

[0072] X0 is the initial state quantity;

[0073] X f is the terminal state quantity;

[0074] The process constraints are:

[0075] C(X i ,U k ,τ k ; t0,t f )≤0(k=1,…,K) (10)

[0076] In formula (10):

[0077] C is the constraint function of the upper and lower bounds of the set process state quantity;

[0078] τ k The time for the distribution point;

[0079] Thus, the original optimal control problem is transformed into a general nonlinear programming problem, namely:

[0080]

[0081] In formula (11):

[0082] y is the design variable including state variables, control variables and endpoint time, which is solved by sequential quadratic programming method;

[0083] E is the optimization index;

[0084] gj is the inequality constraint, which contains the process constraint of the original problem;

[0085] h m It is the equality constraint condition, including the end constraint of the original problem and the dynamic equation constraint.

[0086] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0087] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A fast trajectory optimization method based on reinforcement learning pseudo-spectral grid adaptive construction, characterized by: The method comprises the following steps: S1: Construct the dynamics and kinematics model of the aircraft, set the initial value and constraints of the flight trajectory, and construct the aircraft trajectory optimization problem based on the pseudo-spectral method; The dynamic and kinematic models of the aircraft described in S1 are as follows: In formula (1): is the first derivative of r, r is the distance from the center of the earth; is the first derivative of θ, θ is the longitude; for The first derivative of is latitude; is the first-order derivative of V, V is the flight speed of the aircraft; is the first derivative of γ, γ is the flight path angle; is the first-order derivative of ψ, ψ is the flight heading angle; σ is the tilt angle; L is the aerodynamic lift; D is the aerodynamic drag; and: In formula (2): C L (α,Ma) is the lift coefficient determined by the angle of attack α and the Mach number Ma; C D (α,Ma) is the drag coefficient determined by the angle of attack α and the Mach number Ma; q is the dynamic pressure; S ref is the reference area of ​​the aircraft; S2: Calculate the residual matrix R of the midpoints of two adjacent collocation points, and use the bi-norm ||R||2 of the residual matrix R as the network evaluation method for evaluation; The S2 comprises the following steps: S201: Divide the fixed aircraft trajectory into N interval segments, and the time of each interval is [t k ,t k+1 ], k = 1, ..., N, the number of points in each time interval is [M1, M2 ..., M N ], select two adjacent matching points in a single segment (t i ,t i+1 ), that is, S202: Define state quantity s t for: S203: Define the action amount a t for: a t =[M1,…,M N ] (4) S204: Define the reward function as: r t ＝-||R|| (5) S205: Get the state variable of the midpoint of two adjacent collocation points by Lagrange interpolation method That is, the control variables of the aircraft's distance from the center of the earth, longitude, latitude, speed, flight path angle, heading angle, and the midpoint of two adjacent points That is, angle of attack and angle of roll; S206: Calculate the residual matrix R of the midpoints of two adjacent matching points: In formula (6): is the state variable; F is the aircraft dynamics and kinematics model in equation (1); is the approximate value of the state variable at the midpoint of the discrete collocation; is the approximate value of the control variable at the midpoint of the discrete collocation; S207: using the binary norm ||R||2 of the residual matrix R as a network evaluation method, that is, the smaller the binary norm ||R||2 of the residual matrix R is, the better the evaluation is; S3: Get a new Online-Actor network through DDPG reinforcement learning; The S3 comprises the following steps: S301: Initialize the agent in DDPG reinforcement learning, including four neural networks, namely Online-Actor network μ(s t ), Target-Actor network μ′(s t ), Online-Critic Network And Target-Critic Network Setting up the Online-Actor network μ(s t ) and Target-Actor network μ′(s t ) have the same parameters and structures; set the Online-Critic network and the Target-Critic network The parameters and structures are the same; and the design parameters [M1,…,M N ]; S302: Set the number of rounds K of DDPG reinforcement learning. In each round, randomly initialize the design parameters M1,…M N ; Randomly initialize the aircraft state quantity instruction The DDPG reinforcement learning algorithm is used to train the parameters of the agent and obtain the reward function sequence r obtained by the agent after training. t = {r t1 ,r t2 ,...,r tN } and the corresponding design parameter sequence; S303: Calculate the derivative value of each parameter in the design parameter sequence by forward difference In formula (7): Represents the εth design parameter M ε ,ε=1,…,N, the derivative value of the reward function of the i,jth design parameter in the design sequence; S304: Update the design parameters in the reward function based on formula (7): In formula (8): is the learning rate; S305: Design parameters [M1,…,M] updated based on equation (8) N ], and return to S302; waiting for the design parameters [M1,…,M N ] After the adjustment process converges, enter S306; S306: Based on the design parameters obtained in S304 [M1,…,M N ] gives the designed reward function; based on the reward function, returns to S302, sets the number of DDPG reinforcement learning rounds K = 1, and obtains a new Online-Actor network μ(s t ); S4: The pseudo-spectral discrete grid is designed through the network trained by S3, the continuous trajectory optimization problem is transformed into a nonlinear programming problem, and the sequential quadratic programming method is used to solve it.

2. The method for rapid trajectory optimization based on reinforcement learning pseudo-spectral grid adaptive construction according to claim 1, characterized in that: The S4 comprises the following steps: S401: Based on the new Online-Actor network μ(s t ), using the aircraft time series state quantity to obtain the optimal discrete grid [M1,…,M N ], and based on this grid, the trajectory optimization problem is described: the state variable X at the discrete point i (i=0,...,K) and control variable U k (k=1,...,K) and the initial time t0 and the final time t f , so that the performance index is minimized and the constraints of the dynamic equation, terminal state constraints, control quantity constraints, boundary conditions of the original optimal control problem and process constraints are satisfied. The boundary conditions of the original optimal control problem are: φ(X0,t0,X f ,t f )=0 (9) In formula (9): φ is the constraint function of the upper and lower bounds of the terminal state quantity; X0 is the initial state quantity; X f is the terminal state quantity; The process constraints are: C(X i ,U k ,τ k ;t0,t f )≤0(k=1,…,K) (10) In formula (10): C is the constraint function of the upper and lower bounds of the set process state quantity; τ k The time for the distribution point; Thus, the original optimal control problem is transformed into a general nonlinear programming problem, namely: In formula (11): y is the design variable including state variables, control variables and endpoint time, which is solved by sequential quadratic programming method; E is the optimization index; g j is the inequality constraint, which contains the process constraint of the original problem; h m It is an equality constraint, including the end constraint of the original problem and the dynamic equation constraint.

Images