High-reliability online trajectory planning method based on convex optimization and complex constraint

Abstract

A high-reliability on-line trajectory planning method based on convex optimization and complex constraints belongs to the technical field of on-line trajectory planning of aircrafts, and mainly comprises the following steps: establishing a three-dimensional particle motion model of an aircraft under a half-speed system; determining constraints needing to be met in the flight process and constructing an optimal control model; discretizing the optimal control model, and converting the discretized optimal control model into a finite dimensional convex programming problem; generating an initial trajectory conjecture by using deep reinforcement learning; and performing sequence convex optimization solution based on the initial trajectory guessing. According to the method, efficient and reliable operation of a trajectory planning algorithm in a complex environment is realized through a multi-stage framework design of convex optimization sample generation, reinforcement learning optimization strategy and sequence convex optimization solution, and the method can be widely applied to the fields of aircraft penetration, path planning, precise guidance and the like, and has important engineering application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of online trajectory planning of aircraft, in particular to a high-reliability online trajectory planning method based on convex optimization and complex constraints. BACKGROUND

[0002] During flight, the aircraft needs to use limited control capability to achieve penetration and meet the harsh state (such as position, speed, and track angle) of mid-end guidance handover. The traditional analytical guidance law cannot meet the demand, and an online trajectory planning method that can meet many complex constraints is urgently needed. The convex optimization method has high solving efficiency and can handle various constraints (such as range, heat flow, drop speed, drop angle, etc.) in the process and terminal, and has been increasingly applied in the solution of optimal control problems. At the same time, with the continuous development of intelligent algorithms such as reinforcement learning, new ideas are also provided for the online trajectory planning of aircraft.

[0003] Existing convex optimization and reinforcement learning methods have their own advantages, but also face the following main problems:

[0004] 1. Strong dependence of initial trajectory for convex optimization method: Existing convex optimization algorithms usually need to process complex nonlinear models through linearization, and the linearization process depends on the quality of the initial trajectory. Low-quality initial trajectory often leads to slow convergence or failure to converge.

[0005] 2. Insufficient trajectory feasibility of reinforcement learning: Although the trajectory generated by reinforcement learning meets the real-time requirement, it often lacks optimality and reliability.

[0006] 3. Contradiction between real-time and complexity: In complex environments, existing technologies are difficult to simultaneously consider the real-time and reliability of trajectory planning.

[0007] Therefore, there is an urgent need for a high-reliability trajectory planning method that combines the advantages of convex optimization and reinforcement learning, which can meet complex constraint conditions and improve real-time performance and engineering application feasibility. SUMMARY

[0008] The present disclosure provides a high-reliability online trajectory planning method based on convex optimization and complex constraints, which uses convex optimization and reinforcement learning methods to plan flight trajectories that can meet many complex constraints in real time, effectively solves the problem that real-time and reliability are difficult to be considered in traditional trajectory planning methods, and improves the success rate and safety of aircraft task execution under multiple constraint conditions.

[0009] The high-reliability online trajectory planning method based on convex optimization and complex constraints provided by the present disclosure mainly includes the following steps:

[0010] S1, a three-dimensional particle motion model of the aircraft in the semi-velocity system is established;

[0011] S2, determining the constraint condition in the flight process and constructing an optimal control model;

[0012] S3, linearizing and discretizing the optimal control model, converting the nonlinear infinite-dimensional non-convex optimal control problem into a linear finite-dimensional convex programming problem;

[0013] S4, for the discrete convex programming problem, constructing a step-by-step progressive online trajectory planning framework of "convex optimization generates samples → uses supervised learning to pre-train a deterministic trajectory planning model → uses reinforcement learning to further optimize the model → solves online sequence quickly", including the following steps:

[0014] Sample data is generated by a sequential convex optimization solution algorithm;

[0015] Supervised learning generates a deterministic trajectory planning model;

[0016] Then, the trajectory initial value is optimized using reinforcement learning to establish an initial trajectory guess;

[0017] Based on the initial guess, sequential convex optimization is solved.

[0018] Further, the three-dimensional particle motion model in step S1 is:

[0019] (1)

[0020] In the formula: is the longitude; is the latitude; y is the altitude; V is the aircraft speed; is the speed inclination angle; is the speed deflection angle; m is the engine mass; P is the engine thrust; X is the aerodynamic drag, Y is the normal aerodynamic lift, and Z is the lateral aerodynamic force; is the attack angle; is the sideslip angle; is the engine flow rate; R is the earth radius; g is the standard gravity constant of the earth.

[0021] Further, in step S2, the constraint condition in the flight process includes:

[0022] (2)

[0023] In the formula: X is the aircraft state vector; is the aircraft heat flow; q is the dynamic pressure; n is the overload; is the initial time, is the terminal time, , are the initial state and terminal state of the aircraft, , respectively are the angle of attack minimum and maximum constraints, , respectively are the sideslip angle minimum and maximum constraints, is the heat flux coefficient, is the maximum heat flux constraint, , respectively are the dynamic pressure minimum and maximum constraints, is the maximum overload constraint;

[0024] Finally, the optimal control model is constructed as follows:

[0025]

[0026] where J is the objective function, U is the aircraft control vector, Q, R, K are the weight coefficients of each term in the objective function, F is the dynamic equation, is the constraint function.

[0027] Further, the step S3 specifically comprises:

[0028] The optimal control problem shown in formula (3) in step S2 is linearized and discretized to convert it into a convex programming parameter optimization problem:

[0029] Assuming the number of discrete points is N, the inequality constraints in formula (3) are changed to satisfy respectively at the discrete points, and each inequality constraint is converted into an inequality constraint at N discrete points;

[0030] Based on the initial reference trajectory, the equality and inequality constraints of formula (3) are linearized by using first-order or second-order Taylor expansion;

[0031] At the same time, the dynamic equation constraint of formula (3) is discretized by using any one of trapezoidal method, Euler method, Runge-Kutta method, thereby completing the conversion of the nonlinear infinite-dimensional optimal control problem into a finite-dimensional convex programming problem.

[0032] Further, the step-by-step progression in the online trajectory planning framework in step S4 specifically comprises:

[0033] Sample data is generated by a sequential convex optimization solution algorithm: according to the working area of the aircraft, a number of initial states of the aircraft and target point positions are generated by using Latin hypercube sampling, and a number of demonstration trajectories are batch-generated by using a sequential convex optimization method;

[0034] The supervised learning generates a deterministic trajectory planning model: a deep neural network model is constructed, and an implicit mapping relationship between an environment situation and a flight trajectory is learned from a demonstration trajectory based on supervised learning, the environment situation including an initial state of the aircraft and a target point position, so as to provide a pre-trained strategy function for subsequent reinforcement learning;

[0035] The initial trajectory guess is established by using reinforcement learning to optimize the trajectory initial value: it is assumed that a strategy function generated by an optimal initial trajectory of the aircraft obeys a normal distribution, a probability density function is constructed by predicting a mean value and a variance of the strategy function based on the strategy function constructed by supervised learning, so as to realize autonomous exploration of the aircraft in a continuous action space and realize further optimization of the strategy function.

[0036] After obtaining the initial trajectory guess provided by the reinforcement learning, it only needs to be used as an initial guess of sequence convex optimization to perform sequence optimization solving.

[0037] Further, the action space and the state space of the reinforcement learning are set as follows:

[0038] The action space of the aircraft is defined as The state space S includes: a real-time state of the aircraft , a relative position relationship between the aircraft and the target point , wherein:

[0039]

[0040] , wherein, and are a longitude difference, a height difference, a latitude difference and a remaining distance of the aircraft from the target point.

[0041] Further, in the step S4, the reward function of the reinforcement learning only considers whether the generated initial trajectory guess can help the subsequent sequence convex optimization to converge, and the initial trajectory guess that can converge is rewarded, and a penalty is given according to the number of iterations required for the sequence convex optimization to converge.

[0042] Further, in the step S4, the reward function of the reinforcement learning includes: SCP convergence reward and convergence efficiency reward ; wherein:

[0043] is used to reward whether the reinforcement learning provides an initial value available for SCP, is used to measure the contribution to the SCP convergence efficiency, and the specific expression is as follows:

[0044] (5)

[0045] wherein, the number of iterations required for SCP convergence, The smaller the better the initial trajectory.

[0046] Further, in the step S4, an optimization solution algorithm with real-time operation capability is used to solve the discrete optimal control problem, including: projection method, or primal-dual interior point method.

[0047] Compared with the prior art, the beneficial effects of the present disclosure are: ①Using reinforcement learning to generate high-quality initial trajectory guesses effectively reduces the number of iterations of convex optimization algorithms and greatly improves the real-time performance of trajectory planning;

[0048] ②By optimizing the initial trajectory, the present method can quickly converge to the optimal solution, meeting the high dynamic requirements of the aircraft in complex combat environments;

[0049] ③By combining reinforcement learning with convex optimization, the present method has both the flexibility of reinforcement learning and the rigor of convex optimization, ensuring that the generated trajectory not only meets various process constraints (such as heat flow, overload, and dynamic pressure) and terminal constraints, but also significantly improves the feasibility and safety of the trajectory, making it suitable for actual engineering needs;

[0050] ④Through the multi-level framework design of "convex optimization to generate samples → reinforcement learning to optimize strategies → sequential convex optimization solution", the present method realizes efficient operation of the trajectory planning algorithm in complex environments and can be widely applied in the fields of aircraft penetration, path planning, and precision guidance, etc., having important engineering application value;

[0051] ⑤Effectively solves the problem that real-time performance and reliability are difficult to balance in traditional trajectory planning methods, improving the success rate and safety of aircraft task execution under multiple constraint conditions. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters designate the same components throughout the figures thereof.

[0053] Figure 1 is an example of a complex multi-constrained online trajectory planning framework according to the present disclosure. DETAILED DESCRIPTION

[0054] Preferred embodiments of the present disclosure will be described in greater detail below, with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0055] The present disclosure proposes a high-reliability online trajectory planning algorithm based on convex optimization and complex constraints. The main steps are as follows:

[0056] Step one, establish a three-dimensional particle motion model of the aircraft in the semi-velocity system.

[0057] (6)

[0058] In the formula: is the longitude; is the latitude; y is the altitude; V is the aircraft speed; is the speed inclination angle; is the speed deflection angle; m is the engine mass; P is the engine thrust; X is the aerodynamic drag, Y is the normal aerodynamic lift, and Z is the lateral aerodynamic force; is the attack angle; is the sideslip angle; is the engine flow rate; R is the Earth's radius; g is the standard gravity constant of the Earth.

[0059] Step two, determine the constraints that need to be met during flight and perform mathematical modeling.

[0060] Under the action of program thrust, the aircraft is controlled by adjusting the attack angle and sideslip angle of the aircraft, so that it meets various constraints, such as initial position and speed, terminal position, landing speed and landing angle, heat flow, overload and dynamic pressure during the process, thrust, attack angle and speed inclination angle amplitude, which can be expressed as follows:

[0061] (7)

[0062] In the formula: X is the aircraft state vector; is the aircraft heat flow; q is the dynamic pressure; n is the overload; is the initial time, is the terminal time, , are the initial state and terminal state of the aircraft, , are the minimum and maximum constraints of the attack angle, , are the minimum and maximum constraints of the sideslip angle, is the heat flow coefficient, For maximum heat flux constraint, , These are the minimum and maximum dynamic pressure constraints, respectively. This is the maximum overload constraint.

[0063] The final optimal control model is as follows:

[0064] (7)

[0065] In the formula, U is the aircraft control vector, J is the objective function, Q, R, and K are the weighting coefficients of each term in the objective function, and F is the dynamic equation. For constraint functions.

[0066] Step 3: Linearization and discretization of the optimal control problem.

[0067] The optimal control problem shown in Equation (3) is a continuous infinite-dimensional optimization problem, which needs to be linearized and discretized before it can be converted into a convex programming parameter optimization problem. Based on the initial reference trajectory, first-order and second-order Taylor expansions are used to linearize the equality and inequality constraints in Equation (3). Assuming the number of discrete points is N, the inequality constraints in Equation (3) are then satisfied at each discrete point, and each inequality constraint is converted into an inequality constraint at N discrete points. At the same time, the dynamic equation constraints in Equation (3) can usually be discretized using the trapezoidal method, Euler method, or Runge-Kutta method. Thus, the conversion from the nonlinear infinite-dimensional optimal control problem to the finite-dimensional convex programming problem is completed.

[0068] The optimization problem of equation (3) can be solved using efficient optimization algorithms with real-time computing capabilities (such as the projection method, the original dual interior point method, etc.).

[0069] Step 4: Generate initial values ​​for convex optimization based on deep reinforcement learning.

[0070] Due to the complex aerodynamic forces in equation (1), convex optimization in practice mostly adopts the sequential convex optimization method, which requires linear Taylor expansion of the state equation on the initial conjectured reference trajectory. The convergence efficiency and optimality of convex optimization are both limited by the quality of the initial conjectured trajectory. Therefore, in order to ensure the high reliability of convex optimization in online applications, it is crucial to provide a good initial trajectory conjecture for convex optimization.

[0071] Therefore, this embodiment proposes a complex multi-constraint online trajectory planning framework that is progressively advanced and mutually reinforcing with convex optimization and reinforcement learning: "convex optimization to generate samples → pre-training a deterministic model based on supervised learning → further upgrading the model based on reinforcement learning → fast online sequence solution." (See attached figure.) Figure 1 As shown, the online trajectory planning process according to this framework mainly includes:

[0072] Sample data is generated by the sequential convex optimization algorithm;

[0073] Supervised learning generates a deterministic trajectory planning model;

[0074] Reinforcement learning is used to optimize the initial trajectory value to establish an initial trajectory guess;

[0075] Based on the initial guess, sequential convex optimization is solved.

[0076] Sample data is generated by the sequential convex optimization algorithm: According to the working area of the aircraft, a large number of initial states of the aircraft and target point positions are generated by Latin hypercube sampling, and a large number of demonstration trajectories are batch-generated by using the sequential convex optimization method.

[0077] Supervised learning generates a deterministic trajectory planning model: A deep neural network model is constructed, and based on supervised learning, the implicit mapping relationship between the environment situation (i.e., the initial state of the aircraft and the target point position) and the flight trajectory is learned from the demonstration trajectory. A pre-trained strategy function is provided for subsequent reinforcement learning.

[0078] Reinforcement learning is used to optimize the initial trajectory value to establish an initial trajectory guess: In this framework, to achieve knowledge transfer, without loss of generality, it is assumed that the strategy function generated by the optimal initial trajectory of the aircraft obeys a normal distribution. Based on the strategy function constructed by supervised learning, the mean and variance of the prediction strategy function are constructed to realize the autonomous exploration of the aircraft in the continuous action space and realize the further optimization of the strategy function.

[0079] At the same time, to ensure the convergence of the reinforcement learning algorithm, careful design of the state space and action space of reinforcement learning is involved. The action space of the aircraft is defined as The state space S is composed of the real-time state of the aircraft , the relative position relationship between the aircraft and the target point

[0080] (8)

[0081] where and are the longitude difference, height difference, latitude difference, and remaining distance of the aircraft from the target point, respectively.

[0082] Finally, when designing the reward function of reinforcement learning, only the initial trajectory guess generated needs to be considered to help the subsequent sequential convex optimization quickly converge. The initial trajectory guess that can successfully converge is rewarded, and according to the number of iterations required for the convergence of the sequential convex optimization, a penalty is given to ensure that the initial trajectory guess provided by the reinforcement learning can effectively guarantee the rapid convergence of the convex optimization. The final reward function contains the SCP convergence reward​ and convergence efficiency reward Two people aspects. For providing a reward for whether reinforcement learning provides an initial value available for SCP, For measuring The contribution to the convergence efficiency of SCP, and the specific expression is as follows:

[0083] (9)

[0084] Wherein, The number of iterations required for SCP convergence, The smaller the better the initial trajectory.

[0085] Step five, online trajectory planning of complex multi-constraints.

[0086] After obtaining the initial trajectory guess provided by reinforcement learning, it only needs to be used as the initial guess of sequential convex optimization to solve the sequential optimization. Under the action of a good initial solution, the fast convergence of convex optimization is guaranteed.

[0087] The above technical solutions are only exemplary embodiments of the present application. For those skilled in the art, on the basis of the application disclosed herein, various types of improvements or modifications can be easily made without being limited to the methods described in the above embodiments. Therefore, the above-described method is only preferred and has no limiting meaning.

Claims

1. A high-reliability online trajectory planning method based on convex optimization and complex constraints, comprising the following steps: S1, establishing a three-dimensional particle motion model of the aircraft in the semi-velocity system; S2, determining the constraint conditions in the flight process and constructing an optimal control model; S3, linearizing and discretizing the optimal control model, converting the nonlinear infinite-dimensional non-convex optimal control problem into a linear finite-dimensional convex programming problem; S4, for the discrete convex programming problem, a step-by-step progressive online trajectory planning framework is constructed, including the following steps: Sample data is generated by a sequential convex optimization solving algorithm; A deterministic trajectory planning model is generated by supervised learning; An initial trajectory guess is established by optimizing the trajectory initial value using reinforcement learning; Based on the initial guess, sequential convex optimization solving is performed.

2. The method of claim 1, wherein, The three-dimensional particle motion model in step S1 is: （1） where: is longitude; is latitude; y is altitude; V is aircraft velocity; is velocity bank angle; is velocity drift angle; m is engine mass; P is engine thrust; X is aerodynamic drag, Y is normal aerodynamic lift, Z is lateral aerodynamic force; is angle of attack; is sideslip angle; is engine flow rate; R is earth radius; g is earth standard gravity constant.

3. The method of claim 2, wherein, In step S2, the constraint conditions in the flight process include: （2） where X is the vehicle state vector; q is the heat flux; q is the dynamic pressure; n is the overload; t0 is the initial time, tf is the final time, , X0 and Xf are the initial and final vehicle states, respectively, , a0 and a1 are the minimum and maximum angle of attack constraints, respectively, , b0 and b1 are the minimum and maximum sideslip angle constraints, respectively, is the heat flux coefficient, is the maximum heat flux constraint, , q0 and q1 are the minimum and maximum dynamic pressure constraints, respectively, is the maximum overload constraint; The final optimal control model is constructed as follows: where J is the objective function, U is the aircraft control vector, Q, R, K are weight coefficients of each term in the objective function, F is the dynamics equation, is the constraint function.

4. The method of claim 3, wherein, Step S3 specifically includes: The optimal control problem shown in formula (3) in step S2 is linearized and discretized to convert it into a convex programming parameter optimization problem: Assuming the number of discrete points is N, the inequality constraint in formula (3) becomes respectively satisfy at discrete points, and each inequality constraint is converted into an inequality constraint at N discrete points; Based on the initial reference trajectory, the equality and inequality constraints of formula (3) are linearized using first or second order Taylor expansion; At the same time, the dynamic equation constraint of formula (3) is discretized using any one of trapezoidal method, Euler method, Runge-Kutta method, thereby completing the conversion of the nonlinear infinite-dimensional optimal control problem into a finite-dimensional convex programming problem.

5. The method of claim 4, wherein, The step-by-step progressive online trajectory planning framework in step S4 specifically includes: Sample data is generated by a sequential convex optimization solving algorithm: according to the working area of the aircraft, Latin hypercube sampling is used to generate several initial states of the aircraft and target point positions, and a sequential convex optimization method is used to batch generate several demonstration trajectories; A deterministic trajectory planning model is generated by supervised learning: a deep neural network model is constructed, and based on supervised learning, the implicit mapping relationship between the environment situation and the flight trajectory is learned from the demonstration trajectory, providing a pre-trained strategy function for subsequent reinforcement learning, the environment situation includes: the initial state of the aircraft and the target point position; An initial trajectory guess is established by optimizing the trajectory initial value using reinforcement learning: assuming that the strategy function generated by the optimal initial trajectory of the aircraft obeys a normal distribution, based on the strategy function constructed by supervised learning, the mean and variance of the prediction strategy function are used to construct a probability density function, realizing the autonomous exploration of the aircraft in the continuous action space and realizing the further optimization of the strategy function; After obtaining the initial trajectory guess provided by reinforcement learning, it only needs to be used as the initial guess of sequential convex optimization to perform sequential optimization solving.

6. The method of claim 5, wherein, The action space and state space of the reinforcement learning are set as follows: The action space of the aircraft is defined as The state space S comprises: The relative position of the aircraft to the target point wherein: wherein, and respectively are the longitudinal, altitude, latitude and remaining distance differences of the aircraft from the target point.

7. The method of claim 5, wherein, In the step S4, the reward function of the reinforcement learning only considers whether the generated initial trajectory guess can help the subsequent sequence convex optimization converge, and gives a reward to the initial trajectory guess that can converge, and gives a penalty according to the number of iterations required for the sequence convex optimization to converge.

8. The method of claim 7, wherein, The reward function of the reinforcement learning in the step S4 comprises: SCP convergence reward and convergence efficiency reward ; wherein: providing a reward for whether reinforcement learning provides initial values usable by SCP, for measuring contribution to SCP convergence efficiency, which is expressed as follows: （5） wherein, the number of iterations needed for SCP convergence, The smaller the better the initial trajectory.

9. The method of any one of claims 1-8, wherein, In the step S4, the discrete optimal control problem is solved by using an optimization solution algorithm with real-time computing capability, including a projection method or a primal-dual interior point method.

Images