A hybrid strategy driven model predictive control method for unmanned aerial vehicles

Abstract

The present invention proposes a hybrid strategy driven unmanned aerial vehicle model predictive control method. Firstly, according to the unknown dynamic environment characteristics of the unmanned aerial vehicle performing complex tasks, the complex task is decomposed into two subtasks with relatively simple objectives, and a model predictive controller is designed. According to the perception information of the real-time environment and the estimated information of its own state, a proportional coefficient of the mixture of the two subtask cost functions is given, so that the unmanned aerial vehicle can switch between the subtasks in real time according to the state changes of itself and the environment; the strategy parameters are updated by an iterative search method, so that the mixed proportional coefficient given by the strategy reaches the optimal value under the task requirements; according to the perception and estimation of the real-time environment and its own state by the unmanned aerial vehicle, a relatively optimal mixed proportional coefficient of the subtask cost function is quickly given, which is of great significance for further research on the unmanned aerial vehicle performing complex tasks in unknown environments.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a hybrid strategy driven unmanned aerial vehicle model predictive control method. Background Art

[0002] UAV technology is in a stage of rapid development. Not only has there been a number of major advances in the field of academic research, but more and more UAVs are also being put into practical applications, such as military reconnaissance, targeted strikes, cruise maintenance, pesticide spraying, etc. As an important supporting technology for UAVs to carry out various flight missions, UAV motion control is gaining more and more attention.

[0003] However, in the field of UAV motion control, there are many major problems that need to be solved. The execution of complex tasks in complex dynamic environments is a major problem faced by UAVs. It generally requires UAVs to traverse dynamic environments containing moving obstacles (such as pedestrians, flying birds, and passing vehicles) and perform complex tasks that are usually composed of a variety of simple flight tasks (such as crossing a fixed area, tracking specific targets, etc.). The motion control technology of UAVs is mainly divided into model-based control methods and model-free control methods. Model-based control methods use the dynamic model of the controlled object to predict and adapt the output characteristics or future dynamics of the controlled object. The implementation difficulty is relatively low and the control effect is relatively stable. Model predictive control method is a more commonly used model-based control method. Rawlings et al. clearly explained this method in the literature (Model Predictive Control: Theory and Design, 2009). This method predicts the state transfer of the controlled object within a fixed future time length through the dynamic model of the controlled object, and solves the optimal control input of the controlled object in real time by minimizing the preset cost function related to a series of future states of the controlled object. It has the advantages of flexible solution, strong versatility, fast calculation speed and low resource consumption, and is widely used in the field of UAV motion control. Foehn et al. proposed a model predictive controller that takes into account both motion control objectives and visual tracking objectives in the literature (PAMPC: Perception-aware model predictive control for quadrotors, 2018 IEEE / RSJInternational Conference on Intelligent Robots and Systems). This method enables the UAV to adjust its own posture while performing specific flight tasks, so that the onboard visual sensor is always aimed at a set of specific targets.

[0004] However, the model predictive control method solves the control input of the drone by minimizing the preset cost function. Therefore, for each specific task, the corresponding cost function needs to be carefully designed and debugged, which greatly limits the application of the model predictive control method in more complex tasks. In addition, this feature also makes it difficult for drones using the model predictive control method to cope with dynamic changes in the environment that cannot be predicted in advance, because these changes cannot be considered in advance in the design of the cost function. Reinforcement learning algorithms have a strong ability to cope with complex tasks in dynamic environments. Song et al. proposed a method in the literature (Policy search for model predictive control with application to agile drone flight, 2018 IEEE Transactions on Robotics) to use a reinforcement learning policy search algorithm to provide an upper-level control policy for the model predictive control method, so that the drone can successfully perform the task of crossing the swing door in a known dynamic environment. However, there is no method to apply the policy-driven model predictive control method to the execution of tasks in dynamic unknown environments. Summary of the invention

[0005] In view of this, the present invention provides a hybrid strategy driven UAV model predictive control method, which can successfully cope with the unknown dynamics in the mission environment and enable the UAV to successfully perform complex tasks in an unknown dynamic environment, which is of great significance for further research on UAVs' response to unknown dynamic environments.

[0006] A hybrid strategy driven UAV model predictive control method includes the following steps:

[0007] S1, preset a hyperparameter λ as the mixing ratio coefficient, decompose the task M into subtasks M1 and M2 according to the preset mixing ratio coefficient λ, and design the cost function C1 corresponding to the subtask M1 and the cost function C2 corresponding to the subtask M2, and mix the cost functions C1 and C2 into function C, design a model predictive controller with C as the cost function, and generate the future flight trajectory of the drone according to the designed model predictive controller, and then design a preset reward function A according to the generated flight trajectory;

[0008] S2, perform multiple independent repeated sampling of the preset Gaussian strategy G to obtain the sampling results λ of multiple Gaussian strategies G [1] ,λ [2] , ...λ [i] , ...λ [N] , N is the number of sampling times, i = 1, 2, ... N, the preset reward function A obtained in step S1 is used to calculate the reward value R (λ [i]), with R(λ [i] ) is the parameter λ [i] The corresponding weights are updated through the Monte Carlo expectation maximization method to update the mean μ and variance σ of the Gaussian strategy G, so that the reward R(λ [i] ) has the largest expected value, and the mean μ of the Gaussian strategy G after the update is successful * The mixing ratio coefficient λ as the policy output * ;

[0009] S3, at any time during the UAV's mission, the UAV's perception information e of the environment, its own state estimation information x and the output λ of the Gaussian strategy G at any time * Store in offline database B;

[0010] S4, preset neural network F, preset neural network F provides corresponding mixing ratio coefficient λ′ according to the drone's perception information e of the environment and its own state estimation information x, uses the drone's environment perception information e and its own state estimation information x in the offline database B in step S3 as data samples, and the corresponding Gaussian strategy G output λ * As labels, the preset neural network F is trained so that the mixing ratio coefficient λ′ approaches λ * , complete the hybrid strategy driven UAV model predictive control;

[0011] In the step S1, the cost function C is the objective function of the model predictive controller for solving the optimization problem, and the solution goal of the model predictive controller is to minimize the value of the cost function C;

[0012] The method of mixing the cost functions C1 and C2 into the function C is as follows: the value of the cost function C1 of the subtask M1 is multiplied by the mixing ratio coefficient λ, plus the value of the cost function C2 of the subtask M2 is multiplied by (1-λ), where 0≤λ≤1;

[0013] In step S2, at any time t during the UAV's mission execution, the process of updating the Gaussian strategy G parameters by the Monte Carlo expectation maximization method is required to finally obtain the mixing ratio coefficient λ that is only locally optimal at that arbitrary time t. * ;

[0014] In step S3, the data dimension and type of the drone's environmental perception information e and its own state estimation information x are the same;

[0015] In the step S4, when the preset neural network F is trained, the environment perception information e is subtracted from the self-state estimation information x, and the obtained difference d is used as the input of the preset neural network F;

[0016] The input dimension of the preset neural network F is the same as the drone environment perception information e and the drone's own state estimation information x, and the output λ′ dimension of the preset neural network F is 1;

[0017] In the process of training the preset neural network F, the drone's perception information e of the environment and its own state estimation information x at a certain moment in the mission are mixed with the output mixing ratio coefficient λ of the Gaussian strategy G after the parameter update process at that moment. * Match the configuration and set the mixing ratio coefficient λ * As the label corresponding to the environmental perception information e and the self-state estimation information x.

[0018] Beneficial Effects

[0019] (1) The present invention uses a reinforcement learning algorithm and a neural network offline training method to extract high-level strategies for the first time to perform adaptive proportional mixing of sub-task objectives, designs a hybrid strategy-driven model predictive controller, and enables the UAV to smoothly switch between sub-tasks according to the real-time dynamics of the environment. This is of great significance for further research on how UAVs can cope with unknown dynamic environments.

[0020] (2) The present invention proposes a hybrid strategy driven model predictive control method for unmanned aerial vehicles. First, in view of the unknown dynamic environment characteristics of the unmanned aerial vehicle performing complex tasks, the complex task M is decomposed into two subtasks M1 and M2 with relatively simple objectives, and a cost function C1 is designed for subtask M1 and a cost function C2 is designed for subtask M2 to express the subtask objectives, which are used in the model predictive control process so that the unmanned aerial vehicle can perform each subtask independently. A model predictive controller is designed, which uses the mixed result of the cost function C1 of subtask M1 and the cost function C2 of subtask M2 at a set ratio λ as the cost function, so that the control objective of the unmanned aerial vehicle can switch smoothly between subtasks M1 and M2. Then, an upper-level strategy G in the form of a Gaussian distribution is designed, which gives the proportional coefficient λ of the mixing of the two subtask cost functions according to the perception information e of the real-time environment and the estimated information x of its own state. * , so that the drone can switch between subtasks M1 and M2 in real time according to the changes in its own and the environment's state; next, an iterative optimization framework based on maximizing the expected return is designed to update the parameters of the Gaussian strategy G through iterative search, so that the mixing ratio coefficient λ given by the Gaussian strategy G * Reach the local optimum under the requirements of task M; finally, design a strategy in the form of a neural network F, and use the iterative optimization process to collect the Gaussian strategy G output λ *The data of state perception e and estimated information x are used to train a strategy in the form of a neural network F to achieve a working effect similar to that of the iteratively optimized Gaussian strategy G. According to the UAV's perception and estimation of the real-time environment and its own state, the mixed proportion coefficient λ′ of the cost functions C1 and C2 of the better subtasks M1 and M2 is quickly given, which is of great significance for further research on UAVs performing complex tasks in unknown environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flow chart of the method of the present invention;

[0022] Figure 2 This is an operating effect diagram of an embodiment of the hybrid strategy driven unmanned aerial vehicle model predictive control method provided by the present invention. DETAILED DESCRIPTION

[0023] The present invention is described in detail below with reference to the accompanying drawings and embodiments. The following embodiments or drawings are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0024] The present invention provides a hybrid strategy driven UAV model predictive control method, which realizes the execution of complex tasks by the UAV in an unknown dynamic environment through subtask cost functions C1 and C2, hybrid ratio coefficient λ, Gaussian strategy G and neural network F.

[0025] The model predictive control method driven by the hybrid strategy in the present invention obtains the hybrid ratio coefficient λ of the two subtask cost functions C1 and C2 in the model predictive controller by sampling a Gaussian distribution strategy G. * , find the optimal mean μ and variance σ of the Gaussian strategy G 2 The optimization function is as follows:

[0026]

[0027] Among them, λ [i] It refers to the i-th mixing ratio coefficient among several different mixing ratio coefficients obtained by repeatedly sampling the Gaussian strategy, R(λ [i] ) refers to the i-th mixing ratio coefficient y [i] The corresponding reward value, p μ，σ (λ [i] ) means that under the action of the parameter, the sampling result is λ [i] The optimization function aims to find the policy parameters μ, σ that can maximize the expected reward value R(λ) obtained by the policy sampling result y.

[0028] The optimization function searches for the optimal strategy parameters through the Monte Carlo expectation maximization algorithm. In the present invention, the strategy parameters μ, σ are updated as follows: the Gaussian strategy G is sampled N times independently to obtain a total of N mixing ratio coefficients λ, and their corresponding reward values ​​R(λ are calculated respectively. [i] ), and then update the policy parameters μ, σ as follows:

[0029]

[0030]

[0031] like Figure 1 As shown, an embodiment of the present invention provides a hybrid strategy driven UAV model predictive control method, comprising the following steps:

[0032] S1, the model predictive controller solves the following constrained optimization problem during operation:

[0033]

[0034] in, is the UAV dynamic model, g(x,u)=0, h(x,u)≤0 is the constraint of the UAV. C is the total cost function, p is the control target state, t p is the expected time when the drone reaches the control target state, and λ is the mixing ratio coefficient. The total task is decomposed into two subtasks, and the cost functions C1 and C2 are defined respectively. The total cost function is the sum of the cost functions C1 and C2 of the two subtasks in proportion.

[0035]

[0036] in is the target state and the corresponding expected time of subtask 1, is the target state and corresponding expected time of subtask 2.

[0037] S2, at each moment during the UAV mission, initialize the Gaussian distribution strategy G to π(μ, σ), and perform N samplings to obtain N different mixing ratio coefficients [λ] = [λ [1] ,λ [2] ,...,λ [N] ]. For each mixing ratio coefficient λ, calculate its corresponding reward value r, and get [r] = [r [1] , r [2] , ..., r [N] ].

[0038] S3, update the strategy parameters μ, σ according to the Monte Carlo expected value maximization method described in formula (2). And take the strategy mean μ after successful update* The mixing ratio coefficient λ as the policy output * The current moment UAV’s perception information of the environment e, the UAV’s own state estimation information x and the current moment strategy output λ * Stored in offline database B.

[0039] S4, repeat the above steps S2 to S3. During the repetition process, if the drone successfully performs the mission, the next mission scenario is initialized until the storage space of the offline database B is full. Then, the neural network strategy F is initialized to f w , whose input is ex and output is λ′. Taking e,x stored in the offline database B as data samples, the corresponding λ * is the label pair neural network f w Train to make λ′ approach λ * , we get a strategy in the form of a neural network F that can provide a mixed ratio coefficient λ′ based on the real-time perception e and estimated information x of the UAV.

[0040] See also Figure 2 , Figure 2 The effect diagram of a hybrid strategy driven unmanned aerial vehicle model predictive control method provided by the present invention running on an example. The flight mission is to cross a horizontally swinging door and reach the set midpoint. The dynamic equation of the door is unknown, and its future motion state is predicted by a neural network. The door-passing task is decomposed into two subtasks, one is to fly following the motion state of the door, and the other is to cross the door at the future moment predicted by the neural network. The target state of the following motion subtask is set to the real-time motion state of the door, the expected time to reach the target state is 0.2 seconds, the expected time of the crossing motion is the future moment when the position error between the drone and the center point of the door at the crossing moment is minimized, and the target state is the predicted motion state of the door at the crossing moment. The initial position of the drone is [-5, 0, 1], in meters, the initial speed is 0 meters per second, the hanging position of the door is [2, 0, 3], in meters, the distance from the center point of the door to the hanging point is 2 meters, the door is 1 meter wide and 0.8 meters high, and it performs a free pendulum motion around the hanging point. The simulation results show the effectiveness of the invented hybrid strategy driven unmanned aerial vehicle model predictive control method.

[0041] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hybrid strategy driven UAV model predictive control method, characterized in that The following steps are involved: S1, preset a hyperparameter λ as the mixing ratio coefficient, decompose task M into subtasks M1 and M2 according to the preset mixing ratio coefficient λ, and design the cost function C1 corresponding to subtask M1 and the cost function C2 corresponding to subtask M2, mix the cost functions C1 and C2 into function C, design a model predictive controller with C as the cost function, and generate the future flight trajectory of the drone according to the designed model predictive controller, and then design a preset reward function A according to the generated flight trajectory; S2, perform multiple independent repeated sampling of the preset Gaussian strategy G to obtain the sampling results λ of multiple Gaussian strategies G [1] ,λ [2] , …λ [i] , …λ [N] , N is the number of sampling times, i = 1, 2, ... N, the preset reward function A obtained in step S1 is used to calculate the reward value R (λ [i] ), with R(λ [i] ) is the parameter λ [i] The corresponding weights are updated through the Monte Carlo expectation maximization method to update the mean μ and variance σ of the Gaussian strategy G, so that the reward R(λ [i] ) has the largest expected value, and the mean μ of the Gaussian strategy G after the update is successful * The mixing ratio coefficient λ as the policy output * ; Find the optimal mean μ and variance σ of the Gaussian strategy G 2 The optimization function is as follows: Among them, p μ,σ (λ [i] ) means that under the action of the parameter, the sampling result is λ [i] The probability of, ζ is the weight adjustment coefficient; S3, at any time during the UAV's mission, the UAV's perception information e of the environment, its own state estimation information x and the output λ of the Gaussian strategy G at any time * Store in offline database B; S4, first preset the neural network F, and then preset the neural network F to provide the corresponding mixing ratio coefficient λ according to the drone's perception information e of the environment and its own state estimation information x ′ , using the drone's environmental perception information e and its own state estimation information x in the offline database B in step S3 as data samples, the output λ of the corresponding Gaussian strategy G * The preset neural network F is trained so that the mixing ratio coefficient λ′ approaches λ * , complete the hybrid strategy driven UAV model predictive control; In the step S1, the cost function C is the objective function of the model predictive controller for solving the optimization problem, and the solution goal of the model predictive controller is to minimize the value of the cost function C; The method of mixing the cost functions C1 and C2 into the function C is as follows: the value of the cost function C1 of the subtask M1 is multiplied by the mixing ratio coefficient λ, plus the value of the cost function C2 of the subtask M2 is multiplied by (1-λ), where 0≦λ≦1; In the step S2, at any time t during the UAV's mission execution, the process of updating the Gaussian strategy G parameters by the Monte Carlo expectation maximization method is required to finally obtain the mixing ratio coefficient λ that is only locally optimal at that arbitrary time t. * .

2. The hybrid strategy driven UAV model predictive control method according to claim 1, characterized in that: In step S3, the data dimension and type of the drone's environmental perception information e and its own state estimation information x are the same.

3. The hybrid strategy driven UAV model predictive control method according to claim 2 is characterized by: In the step S4, when the preset neural network F is trained, the environment perception information e is subtracted from the self-state estimation information x, and the obtained difference d is used as the input of the preset neural network F.

4. The hybrid strategy driven UAV model predictive control method according to claim 3 is characterized by: The input dimension of the preset neural network F is the same as the drone environment perception information e and the drone's own state estimation information x. The output λ of the preset neural network F is ′ The dimension is 1.

5. The hybrid strategy driven UAV model predictive control method according to claim 1, characterized in that: In the process of training the preset neural network F, the drone's perception information e of the environment and its own state estimation information x at a certain moment in the mission are mixed with the output mixing ratio coefficient λ of the Gaussian strategy G after the parameter update process at that moment. * Match the configuration and set the mixing ratio coefficient λ * As the label corresponding to the environmental perception information e and the own state estimation information x.

Images