Parameter setting method and device for unmanned aerial vehicle control system based on brain-like mechanism
By optimizing the PID parameters of the UAV control system using a brain-like mechanism-based reinforcement learning method, the problems of low parameter tuning accuracy and high time cost in the existing technology are solved, and the UAV can achieve efficient and stable flight in complex environments.
Patent Information
- Application Number
- CN202511552988.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-29
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2045-10-29
AI Technical Summary
Existing methods for parameter tuning in UAV control systems rely on human experience and trial-and-error approaches, resulting in low accuracy, high time costs, and difficulty in achieving efficient and stable flight performance in complex environments.
A reinforcement learning method based on brain-like mechanisms is adopted, which uses neural networks to replace the human brain for parameter tuning. By establishing a UAV simulation model and a reinforcement learning agent based on the SAC algorithm, the PID controller parameters are optimized.
This technology has enabled the UAV control system to achieve adaptability and robustness in complex environments, improved flight performance and stability, reduced noise interference, shortened parameter optimization time, and improved the efficiency of the control system.
Smart Images
Figure CN121028511B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of unmanned aerial vehicle (UAV) flight control, and in particular to a method and apparatus for tuning the parameters of an UAV control system based on a brain-like mechanism. Background Technology
[0002] With the widespread application of drones in military, civilian, and commercial fields, the performance requirements for drone control systems are constantly increasing. Precise tuning of control system parameters is crucial for drone stability, flight performance, and navigation accuracy. Through precise parameter tuning methods, drones can achieve better flight performance under different environmental conditions, including higher speed, better maneuverability, and faster response. Drone flight in complex environments demands higher robustness and adaptability from the control system; therefore, more refined parameter tuning methods are needed to adapt to complex and changing environmental conditions.
[0003] Proper parameter tuning methods can improve the stability and flight safety of UAVs, reduce the likelihood of loss of control or unexpected events, and ensure the safe completion of flight missions. Optimizing parameter tuning methods can improve the navigation accuracy of UAVs, including position control and flight path tracking, meeting the needs of applications requiring high navigation accuracy. Optimizing control system parameters can reduce the energy consumption cost of UAVs, extend flight time, and improve endurance, thereby increasing flight efficiency. Different types of UAVs have varying requirements for control systems in different application scenarios; therefore, it is necessary to develop different parameter tuning methods for specific application scenarios to meet diverse needs.
[0004] Currently, the controllers widely used in the UAV field are simple and easy-to-control PID controllers. The mainstream methods for controller parameter tuning are the empirical method and the trial-and-error method. The empirical method is a PID parameter tuning method based on experience and practical application experience. It typically adjusts PID parameters based on the experience accumulated by engineers or operators in actual applications. The empirical method is highly subjective; the adjustment process mainly relies on personal experience, which can lead to unstable system performance under different environments, requiring multiple adjustments to achieve the desired effect. The trial-and-error method is a method to determine PID parameters through multiple trials and adjustments. The trial-and-error method is time-consuming, requiring multiple attempts and adjustments. Furthermore, for some complex control systems, the trial-and-error method may not find the optimal combination of PID parameters, requiring more complex mathematical models and calculation methods. Therefore, current manual parameter tuning has limitations in both accuracy and time cost, failing to achieve ideal results. Summary of the Invention
[0005] The purpose of this application is to provide a method and apparatus for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism. It aims to extract and use a brain-like mechanism for the manual tuning method of PID controller parameters in UAV control systems. It proposes a parameter generation method based on reinforcement learning, using neural networks to replace the human brain in the parameter tuning trial and error process, which can enable complex UAV control systems to obtain more accurate controller parameters.
[0006] To achieve the above objectives, this application provides the following solution:
[0007] Firstly, this application provides a method for tuning the parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism, including:
[0008] S1: Establish a drone simulation model;
[0009] S2: Establish a UAV control system based on a PID controller;
[0010] S3: Given the control expectation of the PID controller, and based on the current flight state of the UAV simulation model or the actual UAV system, generate the initial state s of the reinforcement learning environment;
[0011] S4: Input the initial state s into the pre-trained policy network, and the policy network outputs a set of action space A, which is the control parameter of the PID controller;
[0012] S5: Load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller;
[0013] S6: The updated PID controller generates control commands based on the control expectation and sends them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, which is denoted as the new state s'.
[0014] S7: Calculate the reward value r based on the difference between the new state s' and the control expectation;
[0015] S8: Store the tuples s, A, r, s' into the experience replay pool, and use the SAC algorithm to update the parameters of the policy network and the value network.
[0016] S9: Take the new state s' as the current state s of the next time step, and repeat steps S4 to S8 until the termination condition set by the system is reached; the termination condition includes: reaching the predetermined response time, the control system response is completed and reaches a stable state, or the system is judged to be unstable.
[0017] Optionally, the policy network is obtained through the following training process:
[0018] Collect UAV mission data, configure and load the UAV simulation model;
[0019] Set the hyperparameters of the reinforcement learning environment, including discount factor γ, target smoothing coefficient τ, learning rate lr, temperature parameter α, and the size of the hidden layer of the neural network;
[0020] Initialize the policy network and value network to be trained;
[0021] In the simulation environment, the agent interacts with the environment, generates parameters, controls the system response, calculates rewards, and collects data (s,A,r,s').
[0022] The collected data is used to update the policy network parameters based on the SAC algorithm.
[0023] Determine whether the training has reached the convergence condition or the maximum number of rounds. If not, repeat the above steps. If it has, save the trained policy network.
[0024] Optionally, the action space A includes 14 dimensions of PID control parameters, specifically including:
[0025] k1: Pitch attitude control proportional coefficient pitch_kp;
[0026] k2: Pitch attitude control differential coefficients pitch_kd;
[0027] k3: Altitude control ratio coefficient (altitude_kp);
[0028] k4: Altitude control integral coefficient (altitude_ki);
[0029] k5: Airspeed control proportional coefficient based on pitch angle (airspeed_pitch_kp);
[0030] k6: Airspeed control integral coefficient based on pitch angle (airspeed_pitch_ki);
[0031] k7: Airspeed_throttle_kp, a proportional coefficient for airspeed control based on throttle.
[0032] k8: Integral coefficient for airspeed control based on throttle;
[0033] k9: Roll attitude control proportional coefficient roll_kp;
[0034] k10: Roll attitude control differential coefficient roll_kd;
[0035] k11: Course_kp: Track maintenance control proportional coefficient;
[0036] k12: Integral coefficient for track maintenance control, course_ki;
[0037] k13: Sideslip holding control proportional coefficient beta_kp;
[0038] k14: Integral coefficient for sideslip hold control, beta_ki.
[0039] Optionally, the initial state includes:
[0040] North and east position and altitude coordinates in inertial coordinate system pn, pd, h Velocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t ].
[0041] Optionally, the calculation of the reward value r combines the following four parts:
[0042] Steady-state error reward, overshoot reward, system stability reward, and system instability reward.
[0043] Optionally, the expression for the steady-state error reward is as follows:
[0044] ;
[0045] in, As a reward for the steady-state error, For steady-state error, The time for each step.
[0046] Optionally, the expression for the overshoot reward is as follows:
[0047] ;
[0048] in, The reward is for the overshoot portion. For systematic error, This is the expected value.
[0049] Optionally, the expression for the system's stable reward is as follows:
[0050] ;
[0051] in, Partial rewards for system stability For systematic error, This is the expected value.
[0052] Optionally, the expression for the system instability reward is as follows:
[0053] ;
[0054] in, The reward is for the part of the system that is unstable; "instable" means unstable and "stable" means not unstable.
[0055] Secondly, this application provides a parameter tuning device for a drone control system based on a brain-like mechanism, comprising:
[0056] The simulation model building module is used to create UAV simulation models;
[0057] The UAV control system construction module is used to build a UAV control system based on a PID controller.
[0058] The initial state determination module is used to generate the initial state s of the reinforcement learning environment based on the control expectation of the PID controller and the current flight state of the UAV simulation model or the actual UAV system.
[0059] The action space determination module is used to input the initial state s into a pre-trained policy network, and the policy network outputs a set of action spaces A, which are the control parameters of the PID controller.
[0060] The PID controller update module is used to load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller.
[0061] The execution module is used to generate control commands based on the control expectation using the updated PID controller, and send them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, denoted as the new state s'.
[0062] The reward value calculation module is used to calculate the reward value r based on the difference between the new state s' and the control expectation;
[0063] The parameter update module is used to store tuples s, A, r, s' into the experience replay pool and update the parameters of the policy network and value network using the SAC algorithm.
[0064] The termination module is used to take the new state s' as the current state s for the next time step, repeating the process from the UAV control system construction module to the parameter update module until the system's set termination conditions are met. The termination conditions include: reaching a predetermined response time, the control system completing its response and reaching a stable state, or the system determining that it is unstable.
[0065] According to the specific embodiments provided in this application, this application has the following technical effects:
[0066] This application provides a method and apparatus for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism. This method can learn and update parameters in real time according to environmental changes and system state, exhibiting strong adaptability and better adaptability to complex and ever-changing flight environments. It also possesses a certain learning ability, continuously improving and optimizing control parameters through interaction with the environment and experience accumulation, thereby enhancing the flight performance and stability of the UAV. Furthermore, it can better handle noise, interference, and uncertainties in the environment, exhibiting good robustness and enabling more stable flight mission execution. Finally, it can optimize and adjust parameters in a shorter time, improving the efficiency and performance of the UAV control system.
[0067] It is applicable to a variety of flight missions, including search and rescue, aerial photography, and cruise monitoring, and has strong versatility and applicability. Attached Figure Description
[0068] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0069] Figure 1 A block diagram of a method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a neuromorphic mechanism, provided in an embodiment of this application;
[0070] Figure 2 A flowchart illustrating a method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a neuromorphic mechanism, provided in an embodiment of this application;
[0071] Figure 3 The diagram below illustrates the convergence effect. Detailed Implementation
[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0073] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0074] Specifically, the parameter tuning method for UAV control systems based on brain-like mechanisms in this application consists of three parts: a UAV simulation model based on the Matlab-Simulink platform, a UAV control system based on a PID controller, and a reinforcement learning agent based on brain-like mechanisms with the SAC (Soft Actor Critic) algorithm as its core.
[0075] (1) UAV simulation model based on Matlab-Simulink platform
[0076] The UAV simulation model serves as the controlled object of the control system, used for inputting and extracting experimental data.
[0077] (2) Unmanned Aerial Vehicle Control System Based on PID Controller
[0078] This control system is the control system for the controlled object (UAV simulation model), used to receive parameters generated by the brain-like mechanism tuning method and generate instructions to be transmitted to the simulation model.
[0079] (3) Brain-like mechanism-based reinforcement learning agent with SAC (Soft Actor Critic) algorithm as the core.
[0080] Reinforcement learning agents are the core of methods that use reinforcement learning to replace human brain parameter tuning. These agents are trained using the SAC (Soft Actor Critic) algorithm. SAC is a deep reinforcement learning algorithm used to train agents to learn and perform tasks in unknown environments. It is a maximum entropy reinforcement learning algorithm suitable for handling problems with continuous action spaces, enabling agents to learn and perform tasks in continuous action spaces.
[0081] The basic idea of this application is to train a model using the initial flight state data of the UAV, and save the trained model as a selectable policy network, which is then loaded and used when the specific method is applied.
[0082] In one exemplary embodiment, such as Figure 1 and Figure 2 As shown, a method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, it includes the following steps S1-S9. Wherein:
[0083] S1: Establish a drone simulation model.
[0084] This simulation model is a UAV simulation model based on the Matlab-Simulink platform. It is the controlled object of the control system and is used to input and extract experimental data.
[0085] S2: Establish a UAV control system based on a PID controller.
[0086] This control system is the control system for the controlled object, used to receive parameters generated by the brain-like mechanism tuning method and generate instructions to be input into the simulation model.
[0087] S3: Given the control expectation of the PID controller, i.e. Figure 1 In Based on the current flight state of the UAV simulation model or the actual UAV system, the initial state s of the reinforcement learning environment is generated.
[0088] S4: Input the initial state s into the pre-trained policy network, and the policy network outputs a set of action space A, which is the control parameters of the PID controller.
[0089] Specifically, based on factors such as task requirements, the control expectation of the controller is given. Simultaneously, based on the flight state of the UAV simulation model, the initial state of the controlled object is generated by the simulation environment trained using a reinforcement learning algorithm. The policy network to be trained is loaded, and the initial state is fed into the policy network. Finally, a set of Kp, Ki, and Kd control parameters for all PID controllers of the UAV control system is generated through the policy network; where...
[0090] The training environment for reinforcement learning algorithms consists of the following elements:
[0091] (1) Action space
[0092] Based on the scenario setting of the UAV control system, the action space of the environment is set as the PID control parameters of all control loops, A=[k1, k2, k3 … k14], a total of 14 dimensions;
[0093] The seven sets of control parameters are derived from: Pitch Attitude Hold, Altitude Hold Using Commanded Pitch, Airspeed Hold Using Commanded Pitch, Airspeed Hold Using Throttle, Roll Attitude Loop, Course Hold, and Sidelip Hold. Pitch and Roll Attitude Hold use PD control, while Altitude Hold, Airspeed Hold, Throttle-based Airspeed, Course Hold, and Sidelip Hold use PI control.
[0094] As shown in Table 1, where A = [k1, k2, k3 … k14] specifically means:
[0095] k1: pitch_kp is the proportional coefficient of the pitch attitude control loop, used to adjust the response of the pitch attitude.
[0096] k2: pitch_kd is the differential coefficient of the pitch attitude control loop, used to suppress pitch attitude overshoot and oscillation.
[0097] k3: altitude_kp is the proportional coefficient of the altitude control loop, which adjusts the response effect to altitude changes.
[0098] k4: altitude_ki is the integral coefficient of the altitude control loop, which accumulates the error to eliminate steady-state error.
[0099] k5: airspeed_pitch_kp is an airspeed control proportional coefficient based on pitch angle, which adjusts the pitch control related to airspeed.
[0100] k6: airspeed_pitch_ki is the integral coefficient of airspeed control based on pitch angle, accumulating airspeed-related errors.
[0101] k7: airspeed_throttle_kp is an airspeed control proportional coefficient based on throttle, which adjusts the effect of throttle on airspeed.
[0102] k8: airspeed_throttle_ki is the integral coefficient of airspeed control based on throttle, accumulating throttle-related errors.
[0103] k9: roll_kp is the proportional coefficient of the roll attitude control loop, used to adjust the dynamic response of the roll attitude.
[0104] k10: roll_kd is the differential coefficient of the roll attitude control loop, which provides damping for the roll motion to reduce oscillations.
[0105] k11: course_kp is the proportional coefficient of the heading control loop, which adjusts the response speed of heading adjustments.
[0106] k12: course_ki is the integral coefficient of the heading control loop, used to eliminate the steady-state error of heading control.
[0107] k13: beta_kp is the proportional coefficient of the sideslip control loop, which adjusts the aircraft's sideslip control.
[0108] k14: beta_ki is the integral coefficient of the sideslip control loop, used to eliminate steady-state error in sideslip control.
[0109] Table 1 PID control parameters for all control loops
[0110]
[0111] (2) State space
[0112] Select the following flight state variables: north and east position and altitude coordinates in the inertial coordinate system. pn, pd, h Velocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t There are a total of 19 state variables.
[0113] S5: Load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller.
[0114] S6: The updated PID controller generates control commands based on the control expectation and sends them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, which is denoted as the new state s'.
[0115] The generated action is a set of controller parameters (k1) ~ k 14The data is input to the PID controller, resulting in an updated control system with all controller parameters. This updated control system generates control commands for each part of the controller loop, representing the desired control for that single step, and inputs these commands to the controlled object, i.e., the UAV simulation model. Upon receiving the control commands, the UAV system completes a one-step response, generating the necessary state information in the state space (north, east, and altitude coordinates in the inertial coordinate system). pn, pd, h Velocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t (There are 19 dimensions in total.) It also transmits state information and other data to the reinforcement learning environment;
[0116] S7: Calculate the reward value r based on the difference between the new state s' and the control expectation.
[0117] Based on the above performance indicators, this application designs a four-part reward:
[0118] 1) Steady-state error
[0119] This application selects three sets of state variables—attitude, velocity, and position—that are most relevant to the completion of the flight mission, obtains the absolute value of the difference between the three sets of state variables and the expected value, and takes the negative reward as the reward value for evaluating the steady-state error of each step.
[0120] ;
[0121] ;
[0122] in, As a reward for the steady-state error, For steady-state error, The time for each step, abs It is an absolute value.
[0123] 2) Overshoot
[0124] This application transforms the control system performance represented by overshoot into an environmentally adapted model. Since the system may not reach a steady state within a given time during the initial training phase, the percentage by which the maximum value of the system response exceeds the steady-state value, represented by overshoot, is transformed into a negative reward with a large absolute value for a certain percentage deviation from the expectation. The outermost loop's expected state variable is selected as the overshoot index for judgment.
[0125] ;
[0126] in, The reward is for the overshoot portion. For systematic error, This is the expected value.
[0127] 3) System stability
[0128] Since the PID controller parameters are updated at each step, it is impossible to determine whether the system has reached stability. Therefore, as a reward design for system stability performance, all data points of the entire simulation time for each step are taken, and the difference is calculated with the expected value to obtain the error. When all error values of a certain state variable are within the range of Δ=0.02, a small positive reward is given; when the error range of all state variables is within Δ=0.02, a larger positive reward is given.
[0129] ;
[0130] ;
[0131] in, Partial rewards for system stability For systematic error, This is the expected value.
[0132] 4) System instability
[0133] When the error deviation of the system state variables exceeds a certain value, the system is judged to be unstable, a very small negative reward is given, and the episode ends. If the system is not unstable, the episode continues, and a very small positive reward is given.
[0134] ;
[0135] ;
[0136] in, Rewards for system instability For systematic error, This is the expected value.
[0137] S8: Store the tuples s, A, r, s' into the experience replay pool, and use the SAC algorithm to update the parameters of the policy network and the value network.
[0138] S9: Take the new state s' as the current state s of the next time step, and repeat steps S4 to S8 until the termination condition set by the system is reached; the termination condition includes: reaching the predetermined response time, the control system response is completed and reaches a stable state, or the system is judged to be unstable.
[0139] Specifically, it includes:
[0140] Analyze the altitude state obtained from the UAV simulation model h airspeedVa If the absolute value of the difference between the attitude state (theta, phi, psi) and the controller's control expectation exceeds 40% of the expectation, the system is judged to be unstable. If the system is unstable, the controller is turned off, the simulation is stopped, and the next reset and initialization training is started.
[0141] Analyze the data obtained from the UAV simulation model to determine whether the given response time has been reached or whether the control system has completed the task response. If the response time reaches the initial set time of the controller or the system has reached a stable state (the error level is kept within the expected 2%), the controller is turned off and the simulation is stopped. If not, the training continues from step one until the effect is good.
[0142] The technical solution in this application will be further described below using a small fixed-wing UAV as an example:
[0143] (1) Build a UAV simulation model based on the Matlab-Simulink platform
[0144] The controlled object of the control system is used to input and extract experimental data.
[0145] (2) Constructing a UAV control system based on a PID controller
[0146] The control system of the controlled object is used to receive parameters generated by the brain-like mechanism tuning method and generate instructions to be input into the simulation model.
[0147] (3) Brain-like mechanism-based reinforcement learning agent with SAC (Soft Actor Critic) algorithm as the core
[0148] Reinforcement learning methods are used to replace human brain parameter tuning methods; the reinforcement learning agent is trained based on the SAC algorithm. The SAC (Soft Actor Critic) algorithm is a deep reinforcement learning algorithm used to train an agent to learn and perform tasks in unknown environments. It is a maximum entropy reinforcement learning algorithm, suitable for handling problems with continuous action spaces, and can train an agent to learn and perform tasks in continuous action spaces.
[0149] The training environment for reinforcement learning algorithms consists of the following elements:
[0150] a) Action space
[0151] Based on the scenario setting of the UAV control system, the action space of the environment is set as the PID control parameters of all control loops, A=[k1, k2, k3 … k14], a total of 14 dimensions;
[0152] b) State space
[0153] Select the following flight state variables: north and east position and altitude coordinates in the inertial coordinate system. [pn, pd, h] Velocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t There are a total of 19 state variables.
[0154] In this embodiment of the application, the initial flight state data of the UAV is used to train the model, and the trained model is saved as a selectable policy network, which is then read and loaded for use when the specific method is applied.
[0155] Table 2 Initial Flight Status Data of UAV
[0156]
[0157] Taking a small fixed-wing UAV with a flight altitude within 100 meters as a specific example, the detailed implementation of the entire process from strategy model training to deployment is as follows:
[0158] S1~S6 are the parameter generation strategy network training process.
[0159] S1, Collect task data and load simulation model.
[0160] Data on the flight mission objectives and scenarios of a small fixed-wing UAV are analyzed and integrated into simulation model requirements parameters. These parameters are then loaded into the UAV simulation model, and the flight state range and command range are set. In this embodiment, the mission objective is to control the stable flight of the UAV. The mission scenario is imported using a basic atmospheric model at an altitude of 100 meters, and this data is then imported into the simulation model. Furthermore, the flight state range is set as the simulation termination marker for flight failure, with a maximum pitch angle not exceeding 45° and a maximum roll angle not exceeding 180°.
[0161] S2, Set reinforcement learning environment parameters
[0162] Based on the task scenario and simulation model limitations of S1, the initialization parameters, namely the controller commands and initial states, are set in the reinforcement learning environment. The initial parameters are as follows:
[0163] Table 3 Initial Parameters
[0164]
[0165] S3, Reinforcement learning generates control parameters
[0166] The untrained policy network to be trained is loaded, and the initial state is fed into the network. Finally, a set of Kp, Ki, Kd control parameters of all PID controllers of the UAV control system are generated through the policy network. The policy network is modeled as a three-layer fully connected neural network with 256 neurons in each layer.
[0167] S4, Parameter Update and System Response
[0168] The control parameters are input into the controller to obtain the updated control system. The control system generates control commands through calculation and inputs them into the controlled object, i.e., the UAV simulation model. After receiving the control commands, the UAV system completes a one-step response and transmits status information and other data to the reinforcement learning environment.
[0169] S5, Training and updating the policy network
[0170] After the reinforcement learning environment extracts and calculates the data required for training, such as state and reward, the state and reward data are fed into the evaluation network for training, and finally a new policy network is generated after training.
[0171] S6, Check system status
[0172] Analyze the data obtained from the UAV simulation model to determine if the system is unstable. If the system is unstable, shut down the controller, stop the current simulation, and start the next resetting and initialization training. Analyze the data obtained from the UAV simulation model to determine if the given response time has been reached, or if the control system has completed the task response. If it has been completed, shut down the controller and stop the simulation. If it has not been completed, continue training from step one until the results are satisfactory. The training result is considered satisfactory when the environment converges and the system response reaches the ideal state. Figure 3 The reward curve shows the convergence after 200 training rounds, with each round having a step size of 100 steps (i.e., a system response time of 100 seconds).
[0173] The following process loads the model into the control system of a small fixed-wing drone.
[0174] S7, Load the trained policy network.
[0175] Based on mission requirements and other factors, the control expectation of the controller is given, and the initial state of the controlled object is obtained according to the flight state of the UAV. The pre-trained policy network for a specific UAV or flight mission is loaded, and the initial state is fed into the network. Finally, a set of Kp, Ki, Kd control parameters for all PID controllers of the UAV control system is generated through the policy network.
[0176] S8, Controller Response
[0177] The control parameters are input to the controller to obtain the updated control system. The control system generates control commands through calculation and transmits them to the controlled object, i.e., the UAV system. After receiving the control commands, the fixed-wing UAV system completes a response of one step, records the flight data of this step, and transmits status information and other data to the reinforcement learning environment.
[0178] S9, Data Transmission and Parameter Regeneration
[0179] The reinforcement learning environment extracts and calculates data such as state s and reward r, and feeds the current state and reward information into the policy network. The policy network generates a set of Kp, Ki, Kd control parameters for all PID controllers of the UAV control system.
[0180] S10, Determine system status
[0181] Determine whether the UAV control system is interrupted or terminated. If the control system continues to operate normally, repeat steps S8 to S9 until the controller is interrupted or the task ends, then shut down the tuning system.
[0182] Based on the same inventive concept, this application also provides a device for tuning parameters of a UAV control system based on a brain-like mechanism, used to implement the aforementioned method for tuning parameters of a UAV control system based on a brain-like mechanism. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the device for tuning parameters of a UAV control system based on a brain-like mechanism provided below can be found in the limitations of the method for tuning parameters of a UAV control system based on a brain-like mechanism described above, and will not be repeated here.
[0183] In one exemplary embodiment, a parameter tuning device for a drone control system based on a brain-like mechanism is provided, comprising:
[0184] The simulation model building module is used to create UAV simulation models;
[0185] The UAV control system construction module is used to build a UAV control system based on a PID controller.
[0186] The initial state determination module is used to generate the initial state s of the reinforcement learning environment based on the control expectation of the PID controller and the current flight state of the UAV simulation model or the actual UAV system.
[0187] The action space determination module is used to input the initial state s into a pre-trained policy network, and the policy network outputs a set of action spaces A, which are the control parameters of the PID controller.
[0188] The PID controller update module is used to load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller.
[0189] The execution module is used to generate control commands based on the control expectation using the updated PID controller, and send them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, denoted as the new state s'.
[0190] The reward value calculation module is used to calculate the reward value r based on the difference between the new state s' and the control expectation;
[0191] The parameter update module is used to store tuples s, A, r, s' into the experience replay pool and update the parameters of the policy network and value network using the SAC algorithm.
[0192] The termination module is used to take the new state s' as the current state s of the next time step, and repeat the process from the UAV control system construction module to the parameter update module until the termination conditions set by the system are met. The termination conditions include: reaching a predetermined response time, the control system completing the response and reaching a stable state, or the system judging that it is unstable.
[0193] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0194] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism, characterized in that, The method for tuning the parameters of the unmanned aerial vehicle (UAV) control system based on the neuromorphic mechanism includes: S1: Establish a drone simulation model; S2: Establish a UAV control system based on a PID controller; S3: Given the control expectation of the PID controller, and based on the current flight state of the UAV simulation model or the actual UAV system, generate the initial state s of the reinforcement learning environment; S4: Input the initial state s into the pre-trained policy network, and the policy network outputs a set of action space A, which is the control parameter of the PID controller; S5: Load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller; S6: The updated PID controller generates control commands based on the control expectation and issues them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes a one-step action, generating new flight state data, which is recorded as the new state. ; S7: Based on the new state Calculate the reward value r based on the difference between the expected and controlled outcome. S8: tuple Store the data in the experience replay pool and use the SAC algorithm to update the parameters of the policy network and value network; S9: New Status As the current state of the next time step, repeat steps S4 to S8 until the termination conditions set by the system are met; the termination conditions include: reaching a predetermined response time, the control system completing the response and reaching a stable state, or the system judging that it is unstable; The action space A contains 14 dimensions of PID control parameters, specifically including: k1: Pitch attitude control proportional coefficient ; k2: Differential coefficient for pitch attitude control ; k3: Height control ratio coefficient ; k4: Integral coefficient for height control ; k5: Airspeed control proportional coefficient based on pitch angle ; k6: Airspeed control integral coefficient based on pitch angle ; k7: Throttle-based airspeed control proportional coefficient ; k8: Integral coefficient for throttle-based airspeed control ; k9: Roll attitude control proportional coefficient ; k10: Differential coefficient for roll attitude control ; k11: Track Holding Control Proportion Coefficient ; k12: Integral coefficient for track maintenance control ; k13: Sideslip Holding Control Proportion Coefficient ; k14: Integral coefficient for sideslip holding control .
2. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 1, characterized in that, The policy network is obtained through the following training process: Collect UAV mission data, configure and load the UAV simulation model; Set the hyperparameters of the reinforcement learning environment, including discount factor γ, target smoothing coefficient τ, learning rate lr, temperature parameter α, and the size of the hidden layer of the neural network; Initialize the policy network and value network to be trained; In the simulation environment, the agent interacts with the environment, generates parameters, controls the system response, calculates rewards, and collects data. ; The collected data is used to update the policy network parameters based on the SAC algorithm. Determine whether the training has reached the convergence condition or the maximum number of rounds. If not, repeat the above steps. If it has, save the trained policy network.
3. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 1, characterized in that, The initial state includes: North and East position and altitude coordinates in inertial coordinate system velocity in body coordinate system attitude angle angular velocity Aerodynamic parameters rudder .
4. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 1, characterized in that, The calculation of the reward value r comprises the following four parts: Steady-state error reward, overshoot reward, system stability reward, and system instability reward.
5. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 4, characterized in that, The expression for the steady-state error reward is as follows: ； in, As a reward for the steady-state error, For steady-state error, The time for each step.
6. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 4, characterized in that, The expression for the overshoot reward is as follows: ； in, The reward is for the overshoot portion. For systematic error, This is the expected value.
7. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 4, characterized in that, The expression for the system's stable reward is as follows: ； in, Partial rewards for system stability For systematic error, This is the expected value.
8. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 4, characterized in that, The expression for the system instability reward is as follows: ； in, Rewards for system instability Instability, It has not become unstable.
9. A parameter tuning device for an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism, characterized in that, The neuromorphic mechanism-based unmanned aerial vehicle (UAV) control system parameter tuning device includes: The simulation model building module is used to create UAV simulation models; The UAV control system construction module is used to build a UAV control system based on a PID controller. The initial state determination module is used to generate the initial state s of the reinforcement learning environment based on the control expectation of the PID controller and the current flight state of the UAV simulation model or the actual UAV system. The action space determination module is used to input the initial state s into a pre-trained policy network, and the policy network outputs a set of action spaces A, which are the control parameters of the PID controller. The PID controller update module is used to load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller. The execution module is used to generate control commands based on the control expectation using the updated PID controller, and issue them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size, generating new flight state data, which is recorded as the new state. ; The reward value calculation module is used to calculate the reward value based on the new state. Calculate the reward value r based on the difference between the expected and controlled outcome. The parameter update module is used to update tuples. Store the data in the experience replay pool and use the SAC algorithm to update the parameters of the policy network and value network; Termination module, used to change the new state As the current state s of the next time step, the UAV control system construction module is repeated to the parameter update module until the system-set termination conditions are met; the termination conditions include: reaching a predetermined response time, the control system completing the response and reaching a stable state, or the system judging that it is unstable; The action space A contains 14 dimensions of PID control parameters, specifically including: k1: Pitch attitude control proportional coefficient ; k2: Differential coefficient for pitch attitude control ; k3: Height control ratio coefficient ; k4: Integral coefficient for height control ; k5: Airspeed control proportional coefficient based on pitch angle ; k6: Airspeed control integral coefficient based on pitch angle ; k7: Throttle-based airspeed control proportional coefficient ; k8: Integral coefficient for throttle-based airspeed control ; k9: Roll attitude control proportional coefficient ; k10: Differential coefficient for roll attitude control ; k11: Track Holding Control Proportion Coefficient ; k12: Integral coefficient for track maintenance control ; k13: Sideslip Holding Control Proportion Coefficient ; k14: Integral coefficient for sideslip holding control .