Unmanned aerial vehicle control system parameter setting method and device based on brain-like mechanism

Abstract

The invention discloses an unmanned aerial vehicle control system parameter setting method and device based on a brain-like mechanism, and relates to the field of unmanned aerial vehicle flight control, and the method comprises the steps: building an unmanned aerial vehicle simulation model and a PID controller; giving a control expectation, and obtaining an initial state s; inputting the initial state s into a pre-trained strategy network and a group of action spaces A, loading control parameters into a PID (Proportion Integration Differentiation) controller, and updating the parameters of the PID controller; generating a control instruction according to the control expectation, issuing the control instruction to the unmanned aerial vehicle simulation model, executing an action of one step length, and generating new flight state data s'; calculating a reward value r; storing the tuples s, A, r and s'into an experience playback pool, and updating parameters of a policy network and a value network; and taking the new state s'as the current state s of the next time step, and repeating the steps until a termination condition set by the system is reached. According to the invention, a complex unmanned aerial vehicle control system can obtain more accurate controller parameters.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle flight control, in particular to a method and device for parameter setting of an unmanned aerial vehicle control system based on a brain-like mechanism. BACKGROUND

[0002] With the wide application of unmanned aerial vehicles in military, civilian and commercial fields, the performance requirements of unmanned aerial vehicle control systems are continuously improving. The accurate setting of control system parameters is crucial for the stability, flight performance, navigation accuracy and other aspects of unmanned aerial vehicles. Through accurate parameter setting methods, unmanned aerial vehicles can have better flight performance in different environmental conditions, including higher speed, better maneuverability and faster response speed. Unmanned aerial vehicles require control systems with higher robustness and adaptability in complex environments, so more refined parameter setting methods are needed to adapt to complex and variable environmental conditions.

[0003] Reasonable parameter setting methods can improve the stability and flight safety of unmanned aerial vehicles, reduce the possibility of loss of control or accidents, and ensure the safe completion of flight missions. By optimizing the parameter setting method, the navigation accuracy of the unmanned aerial vehicle can be improved, including position control, route tracking and other aspects, to meet the requirements of high-precision navigation in application scenarios. By optimizing the control system parameters, the energy consumption cost of the unmanned aerial vehicle can be reduced, the flight time can be extended, and the endurance can be improved, thereby improving the flight efficiency. Different types of unmanned aerial vehicles have different requirements for control systems in different application scenarios, so different parameter setting methods need to be developed for specific application scenarios to meet diverse needs.

[0004] The controller widely used in the field of unmanned aerial vehicles is the simple and easy-to-control PID controller, and the mainstream method of controller parameter tuning is the empirical method and trial-and-error method. The empirical method is a PID parameter setting method based on experience and practical application experience. It is usually based on the experience accumulated by engineers or operators in practical applications to adjust the PID parameters. The empirical method is highly subjective, and the adjustment process mainly depends on personal experience, which is more subjective and may lead to unstable performance of the system in different environments, requiring multiple adjustments to achieve the desired effect. The trial-and-error method is a method of determining PID parameters through multiple trials and adjustments. The trial-and-error method has high time cost and requires multiple attempts and adjustments, and for some complex control systems, the trial-and-error method may not be able to find the optimal PID parameter combination, and more complex mathematical models and calculation methods are needed. Therefore, the current human parameter setting has limitations in both accuracy and time cost. SUMMARY

[0005] The purpose of this application is to provide a method and apparatus for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism. It aims to extract and use a brain-like mechanism for the manual tuning method of PID controller parameters in UAV control systems. It proposes a parameter generation method based on reinforcement learning, using neural networks to replace the human brain in the parameter tuning trial and error process, which can enable complex UAV control systems to obtain more accurate controller parameters.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for tuning the parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism, including: S1: Establish a drone simulation model; S2: Establish a UAV control system based on a PID controller; S3: Given the control expectation of the PID controller, and based on the current flight state of the UAV simulation model or the actual UAV system, generate the initial state s of the reinforcement learning environment; S4: Input the initial state s into the pre-trained policy network, and the policy network outputs a set of action space A, which is the control parameter of the PID controller; S5: Load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller; S6: The updated PID controller generates control commands based on the control expectation and sends them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, which is denoted as the new state s'. S7: Calculate the reward value r based on the difference between the new state s' and the control expectation; S8: Store the tuples s, A, r, s' into the experience replay pool, and use the SAC algorithm to update the parameters of the policy network and the value network. S9: Take the new state s' as the current state s of the next time step, and repeat steps S4 to S8 until the termination condition set by the system is reached; the termination condition includes: reaching the predetermined response time, the control system response is completed and reaches a stable state, or the system is judged to be unstable.

[0007] Optionally, the policy network is obtained through the following training process: Collect UAV mission data, configure and load the UAV simulation model; Set the hyperparameters of the reinforcement learning environment, including discount factor γ, target smoothing coefficient τ, learning rate lr, temperature parameter α, and the size of the hidden layer of the neural network; Initialize the policy network and value network to be trained; In the simulation environment, the agent interacts with the environment, generates parameters, control system responses, calculates rewards, and collects data (s, A, r, s'); Using the collected data, the policy network parameters are updated based on the SAC algorithm; Determine whether the training has reached the convergence condition or the maximum number of rounds. If not, repeat the above steps. If so, save the trained policy network.

[0008] Optionally, the action space A contains 14-dimensional PID control parameters, specifically including: k1: pitch_kp, a pitch attitude control proportional coefficient; k2: pitch_kd, a pitch attitude control differential coefficient; k3: altitude_kp, an altitude control proportional coefficient; k4: altitude_ki, an altitude control integral coefficient; k5: airspeed_pitch_kp, an airspeed control proportional coefficient based on the pitch angle; k6: airspeed_pitch_ki, an airspeed control integral coefficient based on the pitch angle; k7: airspeed_throttle_kp, an airspeed control proportional coefficient based on the throttle; k8: airspeed_throttle_ki, an airspeed control integral coefficient based on the throttle; k9: roll_kp, a roll attitude control proportional coefficient; k10: roll_kd, a roll attitude control differential coefficient; k11: course_kp, a course keeping control proportional coefficient; k12: course_ki, a course keeping control integral coefficient; k13: beta_kp, a beta keeping control proportional coefficient; k14: beta_ki, a beta keeping control integral coefficient.

[0009] Optionally, the initial state includes: North, East position and height coordinates in the inertial coordinate system pn, pd, h Velocity in the body coordinate system u, v, w Attitude angle phi, theta, psi Angular rate p, q, r Aerodynamic parameters Va, alpha, beta Rudder amount delta_e, delata_a, delta_r, delta_t

[0010] ​Optionally, the calculation of the reward value r integrates the following four parts: a steady-state error reward, an overshoot reward, a system stability reward, and a system instability reward.

[0011] Optionally, the expression of the steady-state error reward is as follows: ; wherein, is the steady-state error part reward, is the steady-state error, is the time of each step.

[0012] Optionally, the expression of the overshoot reward is as follows: ; wherein, is the overshoot part reward, is the system error, is the expected value.

[0013] Optionally, the expression of the system stability reward is as follows: ; wherein, is the system stability part reward, is the system error, is the expected value.

[0014] Optionally, the expression of the system instability reward is as follows: ; wherein, is the system instability part reward, instable is instability, and stable is non-instability.

[0015] In a second aspect, the application provides a UAV control system parameter setting device based on a brain-like mechanism, comprising: a simulation model construction module, configured to establish a UAV simulation model; a UAV control system construction module, configured to establish a UAV control system based on a PID controller; an initial state determination module, configured to give a control expectation of the PID controller, and generate an initial state s of a reinforcement learning environment based on a current flight state of the UAV simulation model or an actual UAV system; an action space determination module, configured to input the initial state s to a pre-trained policy network, and output a set of action spaces A from the policy network, wherein the action spaces A are control parameters of the PID controller; a PID controller updating module, configured to load the control parameters into the PID controller and update parameters of the PID controller to obtain an updated PID controller; an execution module, configured to generate a control instruction according to the control expectation by using the updated PID controller, and deliver the control instruction to the UAV simulation model or the actual UAV system, so that the UAV simulation model or the actual UAV system performs an action of one step to generate new flight state data, denoted as new state s'; a reward value calculation module, configured to calculate a reward value r according to a difference between the new state s' and the control expectation; a parameter updating module, configured to store the tuple s, A, r, s' into an experience replay pool, and update parameters of the policy network and the value network by using the SAC algorithm; a termination module, configured to take the new state s' as a current state s of a next time step, and repeat the UAV control system construction module to the parameter updating module until a termination condition set by the system is reached; the termination condition includes: a predetermined response duration is reached, the control system response is completed to reach a stable state, or the system is judged to be unstable According to the specific embodiments provided in the present application, the present application has the following technical effects: The UAV control system parameter setting method and device based on the brain-like mechanism provided in the present application can learn and update parameters in real time according to environmental changes and system states, has strong self-adaptive ability, can better adapt to complex and changeable flight environments, has certain learning ability, can continuously improve and optimize control parameters through interaction with the environment and experience accumulation, improve flight performance and stability of the UAV, can better handle noise, interference and uncertainty in the environment, has good robustness, can more stably perform flight tasks, and can realize optimization and adjustment of parameters in a shorter time, improve efficiency and performance of the UAV control system. It can be applied to various types of flight tasks, including search and rescue, aerial photography, cruise monitoring, etc., and has strong universality and applicability. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 A block diagram of a UAV control system parameter setting method based on a brain-like mechanism provided in an embodiment of the present application; Figure 2 A flowchart of a parameter setting method of a UAV control system based on a brain-like mechanism is provided for an embodiment of the present application. Figure 3 A reference diagram for convergence effect is provided. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0019] The above purposes, features and advantages of the present application will be more apparent and easy to understand. The present application will be further described in detail below with reference to the drawings and specific embodiments.

[0020] Specifically, the parameter setting method of the UAV control system based on the brain-like mechanism in the present application includes three parts: a UAV simulation model based on a Matlab-Simulink platform, a UAV control system based on a PID controller, and a reinforcement learning agent based on a brain-like mechanism with SAC (Soft Actor Critic) algorithm as the core.

[0021] (1) UAV simulation model based on Matlab-Simulink platform The UAV simulation model is used as the control object of the control system to import and extract experimental data.

[0022] (2) UAV control system based on PID controller The control system is the control system of the control object (UAV simulation model), which is used to receive the parameters generated by the brain-like mechanism setting method and generate instructions to import into the simulation model.

[0023] (3) Reinforcement learning agent based on brain-like mechanism with SAC (Soft Actor Critic) algorithm as the core.

[0024] The reinforcement learning agent is the core of using reinforcement learning method instead of brain parameter setting method. The reinforcement learning agent is trained based on SAC algorithm. SAC (Soft Actor Critic) algorithm is a deep reinforcement learning-based algorithm used to train agents to learn and perform tasks in unknown environments. It is a maximum entropy reinforcement learning-based algorithm suitable for handling continuous action space problems, enabling agents to learn and perform tasks in continuous action space.

[0025] The basic idea of ​​this application is to train a model using the initial flight state data of the UAV, and save the trained model as a selectable policy network, which is then loaded and used when the specific method is applied.

[0026] In one exemplary embodiment, such as Figure 1 and Figure 2 As shown, a method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, it includes the following steps S1-S9. Wherein: S1: Establish a drone simulation model.

[0027] This simulation model is a UAV simulation model based on the Matlab-Simulink platform. It is the controlled object of the control system and is used to input and extract experimental data.

[0028] S2: Establish a UAV control system based on a PID controller.

[0029] This control system is the control system for the controlled object, used to receive parameters generated by the brain-like mechanism tuning method and generate instructions to be input into the simulation model.

[0030] S3: Given the control expectation of the PID controller, i.e. Figure 1 In Based on the current flight state of the UAV simulation model or the actual UAV system, the initial state s of the reinforcement learning environment is generated.

[0031] S4: Input the initial state s into the pre-trained policy network, and the policy network outputs a set of action space A, which is the control parameters of the PID controller.

[0032] Specifically, based on factors such as task requirements, the control expectation of the controller is given. Simultaneously, based on the flight state of the UAV simulation model, the initial state of the controlled object is generated by the simulation environment trained using a reinforcement learning algorithm. The policy network to be trained is loaded, and the initial state is fed into the policy network. Finally, a set of Kp, Ki, and Kd control parameters for all PID controllers of the UAV control system is generated through the policy network; where... The training environment for reinforcement learning algorithms consists of the following elements: (1) Action space Based on the scenario setting of the UAV control system, the action space of the environment is set as the PID control parameters of all control loops, A=[k1, k2, k3 … k14], a total of 14 dimensions; The seven sets of control parameters are derived from: Pitch Attitude Hold, Altitude Hold Using Commanded Pitch, Airspeed Hold Using Commanded Pitch, Airspeed Hold Using Throttle, Roll Attitude Loop, Course Hold, and Sidelip Hold. Pitch and Roll Attitude Hold use PD control, while Altitude Hold, Airspeed Hold, Throttle-based Airspeed, Course Hold, and Sidelip Hold use PI control.

[0033] As shown in Table 1, where A = [k1, k2, k3 … k14] specifically means: k1: pitch_kp is the proportional coefficient of the pitch attitude control loop, used to adjust the response of the pitch attitude.

[0034] k2: pitch_kd is the differential coefficient of the pitch attitude control loop, used to suppress pitch attitude overshoot and oscillation.

[0035] k3: altitude_kp is the proportional coefficient of the altitude control loop, which adjusts the response effect to altitude changes.

[0036] k4: altitude_ki is the integral coefficient of the altitude control loop, which accumulates the error to eliminate steady-state error.

[0037] k5: airspeed_pitch_kp is an airspeed control proportional coefficient based on pitch angle, which adjusts the pitch control related to airspeed.

[0038] k6: airspeed_pitch_ki is the integral coefficient of airspeed control based on pitch angle, accumulating airspeed-related errors.

[0039] k7: airspeed_throttle_kp is an airspeed control proportional coefficient based on throttle, which adjusts the effect of throttle on airspeed.

[0040] k8: airspeed_throttle_ki is the integral coefficient of airspeed control based on throttle, accumulating throttle-related errors.

[0041] k9: roll_kp is the proportional coefficient of the roll attitude control loop, used to adjust the dynamic response of the roll attitude.

[0042] k10: roll_kd is the differential coefficient of the roll attitude control loop, which provides damping for the roll motion to reduce oscillations.

[0043] k11: course_kp is the proportional coefficient of the heading control loop, which adjusts the response speed of heading adjustments.

[0044] k12: course_ki is the integral coefficient of the heading control loop, used to eliminate the steady-state error of heading control.

[0045] k13: beta_kp is the proportional coefficient of the sideslip control loop, which adjusts the aircraft's sideslip control.

[0046] k14: beta_ki is the integral coefficient of the sideslip control loop, used to eliminate steady-state error in sideslip control.

[0047] Table 1 PID control parameters for all control loops

[0048] (2) State space Select the following flight state variables: north and east position and altitude coordinates in the inertial coordinate system. pn, pd, h Velocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t There are a total of 19 state variables.

[0049] S5: Load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller.

[0050] S6: The updated PID controller generates control commands based on the control expectation and sends them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, which is denoted as the new state s'.

[0051] The generated action is a set of controller parameters (k1) ~ k 14 The data is input to the PID controller, resulting in an updated control system with all controller parameters. This updated control system generates control commands for each part of the controller loop, representing the desired control for that single step, and inputs these commands to the controlled object, i.e., the UAV simulation model. Upon receiving the control commands, the UAV system completes a one-step response, generating the necessary state information in the state space (north, east, and altitude coordinates in the inertial coordinate system). pn, pd, hVelocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t (There are 19 dimensions in total.) It also transmits state information and other data to the reinforcement learning environment; S7: Calculate the reward value r based on the difference between the new state s' and the control expectation.

[0052] Based on the above performance indicators, this application designs a four-part reward: 1) Steady-state error This application selects three sets of state variables—attitude, velocity, and position—that are most relevant to the completion of the flight mission, obtains the absolute value of the difference between the three sets of state variables and the expected value, and takes the negative reward as the reward value for evaluating the steady-state error of each step.

[0053] ; ; in, As a reward for the steady-state error, For steady-state error, The time for each step, abs It is an absolute value.

[0054] 2) Overshoot This application transforms the control system performance represented by overshoot into an environmentally adapted model. Since the system may not reach a steady state within a given time during the initial training phase, the percentage by which the maximum value of the system response exceeds the steady-state value, represented by overshoot, is transformed into a negative reward with a large absolute value for a certain percentage deviation from the expectation. The outermost loop's expected state variable is selected as the overshoot index for judgment.

[0055] ; in, The reward is for the overshoot portion. For systematic error, This is the expected value.

[0056] 3) System stability Since the PID controller parameters are updated at each step, it is impossible to determine whether the system has reached stability. Therefore, as a reward design for system stability performance, all data points of the entire simulation time for each step are taken, and the difference is calculated with the expected value to obtain the error. When all error values ​​of a certain state variable are within the range of Δ=0.02, a small positive reward is given; when the error range of all state variables is within Δ=0.02, a larger positive reward is given.

[0057] ; ; in, Partial rewards for system stability For systematic error, This is the expected value.

[0058] 4) System instability When the error deviation of the system state variables exceeds a certain value, the system is judged to be unstable, a very small negative reward is given, and the episode ends. If the system is not unstable, the episode continues, and a very small positive reward is given.

[0059] ; ; in, Rewards for system instability For systematic error, This is the expected value.

[0060] S8: Store the tuples s, A, r, s' into the experience replay pool, and use the SAC algorithm to update the parameters of the policy network and the value network.

[0061] S9: Take the new state s' as the current state s of the next time step, and repeat steps S4 to S8 until the termination condition set by the system is reached; the termination condition includes: reaching the predetermined response time, the control system response is completed and reaches a stable state, or the system is judged to be unstable.

[0062] Specifically, it includes: Analyze the altitude state obtained from the UAV simulation model h airspeed Va If the absolute value of the difference between the attitude state (theta, phi, psi) and the controller's control expectation exceeds 40% of the expectation, the system is judged to be unstable. If the system is unstable, the controller is turned off, the simulation is stopped, and the next reset and initialization training is started. Analyze the data obtained from the UAV simulation model to determine whether the given response time has been reached or whether the control system has completed the task response. If the response time reaches the initial set time of the controller or the system has reached a stable state (the error level is kept within the expected 2%), the controller is turned off and the simulation is stopped. If not, the training continues from step one until the effect is good.

[0063] The technical solution in this application will be further described below using a small fixed-wing UAV as an example: (1) Build a UAV simulation model based on the Matlab-Simulink platform The controlled object of the control system is used to input and extract experimental data.

[0064] (2) Constructing a UAV control system based on a PID controller The control system of the controlled object is used to receive parameters generated by the brain-like mechanism tuning method and generate instructions to be input into the simulation model.

[0065] (3) Brain-like mechanism-based reinforcement learning agent with SAC (Soft Actor Critic) algorithm as the core Reinforcement learning methods are used to replace human brain parameter tuning methods; the reinforcement learning agent is trained based on the SAC algorithm. The SAC (Soft Actor Critic) algorithm is a deep reinforcement learning algorithm used to train an agent to learn and perform tasks in unknown environments. It is a maximum entropy reinforcement learning algorithm, suitable for handling problems with continuous action spaces, and can train an agent to learn and perform tasks in continuous action spaces.

[0066] The training environment for reinforcement learning algorithms consists of the following elements: a) Action space Based on the scenario setting of the UAV control system, the action space of the environment is set as the PID control parameters of all control loops, A=[k1, k2, k3 … k14], a total of 14 dimensions; b) State space Select the following flight state variables: north and east position and altitude coordinates in the inertial coordinate system. [pn, pd, h] Velocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t There are a total of 19 state variables.

[0067] In this embodiment of the application, the initial flight state data of the UAV is used to train the model, and the trained model is saved as a selectable policy network, which is then read and loaded for use when the specific method is applied.

[0068] Table 2 Initial Flight Status Data of UAV

[0069] Taking a small fixed-wing UAV with a flight altitude within 100 meters as a specific example, the specific implementation of the entire process from strategy model training to deployment is as follows: S1~S6 are the parameter generation strategy network training process.

[0070] S1, Collect task data and load simulation model. Data on the flight mission objectives and scenarios of a small fixed-wing UAV are analyzed and integrated into simulation model requirements parameters. These parameters are then loaded into the UAV simulation model, and the flight state range and command range are set. In this embodiment, the mission objective is to control the stable flight of the UAV. The mission scenario is imported using a basic atmospheric model at an altitude of 100 meters, and this data is then imported into the simulation model. Furthermore, the flight state range is set as the simulation termination marker for flight failure, with a maximum pitch angle not exceeding 45° and a maximum roll angle not exceeding 180°.

[0071] S2, Set reinforcement learning environment parameters Based on the task scenario and simulation model limitations of S1, the initialization parameters, namely the controller commands and initial states, are set in the reinforcement learning environment. The initial parameters are as follows: Table 3 Initial Parameters

[0072] S3, Reinforcement learning generates control parameters The untrained policy network to be trained is loaded, and the initial state is fed into the network. Finally, a set of Kp, Ki, Kd control parameters of all PID controllers of the UAV control system are generated through the policy network. The policy network is modeled as a three-layer fully connected neural network with 256 neurons in each layer. S4, Parameter Update and System Response The control parameters are input into the controller to obtain the updated control system. The control system generates control commands through calculation and inputs them into the controlled object, i.e., the UAV simulation model. After receiving the control commands, the UAV system completes a one-step response and transmits status information and other data to the reinforcement learning environment. S5, Training and updating the policy network After the reinforcement learning environment extracts and calculates the data required for training, such as state and reward, the state and reward data are fed into the evaluation network for training, and finally a new policy network is generated after training. S6, Check system status Analyze the data obtained from the UAV simulation model to determine if the system is unstable. If the system is unstable, shut down the controller, stop the current simulation, and start the next resetting and initialization training. Analyze the data obtained from the UAV simulation model to determine if the given response time has been reached, or if the control system has completed the task response. If it has been completed, shut down the controller and stop the simulation. If it has not been completed, continue training from step one until the results are satisfactory. The training result is considered satisfactory when the environment converges and the system response reaches the ideal state.Figure 3 The reward curve shows that the system response time is 100 seconds, with each round having a step size of 100. After training for 200 rounds, the system reaches convergence.

[0073] The following process loads the model into the control system of a small fixed-wing drone.

[0074] S7, Load the trained policy network. Based on mission requirements and other factors, the control expectation of the controller is given, and the initial state of the controlled object is obtained according to the flight state of the UAV. The pre-trained policy network for a specific UAV or flight mission is loaded, and the initial state is fed into the network. Finally, a set of Kp, Ki, Kd control parameters for all PID controllers of the UAV control system is generated through the policy network. S8, Controller Response The control parameters are input to the controller to obtain the updated control system. The control system generates control commands through calculation and transmits them to the controlled object, i.e., the UAV system. After receiving the control commands, the fixed-wing UAV system completes a response of one step, records the flight data of this step, and transmits status information and other data to the reinforcement learning environment. S9, Data Transmission and Parameter Regeneration The reinforcement learning environment extracts and calculates data such as state s and reward r, and feeds the current state and reward information into the policy network. The policy network generates a set of Kp, Ki, Kd control parameters for all PID controllers of the UAV control system. S10, Determine system status Determine whether the UAV control system is interrupted or terminated. If the control system continues to operate normally, repeat steps S8 to S9 until the controller is interrupted or the task ends, then shut down the tuning system.

[0075] Based on the same inventive concept, this application also provides a device for tuning parameters of a UAV control system based on a brain-like mechanism, used to implement the aforementioned method for tuning parameters of a UAV control system based on a brain-like mechanism. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the device for tuning parameters of a UAV control system based on a brain-like mechanism provided below can be found in the limitations of the method for tuning parameters of a UAV control system based on a brain-like mechanism described above, and will not be repeated here.

[0076] In one exemplary embodiment, a parameter tuning device for a drone control system based on a brain-like mechanism is provided, comprising: The simulation model building module is used to create UAV simulation models; The UAV control system construction module is used to build a UAV control system based on a PID controller. The initial state determination module is used to generate the initial state s of the reinforcement learning environment based on the control expectation of the PID controller and the current flight state of the UAV simulation model or the actual UAV system. The action space determination module is used to input the initial state s into a pre-trained policy network, and the policy network outputs a set of action spaces A, which are the control parameters of the PID controller. The PID controller update module is used to load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller. The execution module is used to generate control commands based on the control expectation using the updated PID controller, and send them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, denoted as the new state s'. The reward value calculation module is used to calculate the reward value r based on the difference between the new state s' and the control expectation; The parameter update module is used to store tuples s, A, r, s' into the experience replay pool and update the parameters of the policy network and value network using the SAC algorithm. The termination module is used to take the new state s' as the current state s of the next time step, and repeat the process from the UAV control system construction module to the parameter update module until the termination conditions set by the system are met. The termination conditions include: reaching a predetermined response time, the control system completing the response and reaching a stable state, or the system judging that it is unstable.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism, characterized in that, The method for tuning the parameters of the unmanned aerial vehicle (UAV) control system based on the neuromorphic mechanism includes: S1: Establish a drone simulation model; S2: Establish a UAV control system based on a PID controller; S3: Given the control expectation of the PID controller, and based on the current flight state of the UAV simulation model or the actual UAV system, generate the initial state s of the reinforcement learning environment; S4: Input the initial state s into the pre-trained policy network, and the policy network outputs a set of action space A, which is the control parameter of the PID controller; S5: Load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller; S6: The updated PID controller generates control commands based on the control expectation and sends them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, which is denoted as the new state s'. S7: Calculate the reward value r based on the difference between the new state s' and the control expectation; S8: Store the tuples s, A, r, s' into the experience replay pool, and use the SAC algorithm to update the parameters of the policy network and the value network. S9: Take the new state s' as the current state of the next time step, and repeat steps S4 to S8 until the termination condition set by the system is met; the termination condition includes: reaching the predetermined response time, the control system response is completed and reaches a stable state, or the system is judged to be unstable.

2. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 1, characterized in that, The policy network is obtained through the following training process: Collect UAV mission data, configure and load the UAV simulation model; Set the hyperparameters of the reinforcement learning environment, including discount factor γ, target smoothing coefficient τ, learning rate lr, temperature parameter α, and the size of the hidden layer of the neural network; Initialize the policy network and value network to be trained; In the simulation environment, the agent interacts with the environment, generates parameters, controls the system response, calculates rewards, and collects data (s,A,r,s'). The collected data is used to update the policy network parameters based on the SAC algorithm. Determine whether the training has reached the convergence condition or the maximum number of rounds. If not, repeat the above steps. If it has, save the trained policy network.

3. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 1, characterized in that, The action space A contains 14 dimensions of PID control parameters, specifically including: k1: Pitch attitude control proportional coefficient pitch_kp; k2: Pitch attitude control differential coefficients pitch_kd; k3: Altitude control ratio coefficient (altitude_kp); k4: Altitude control integral coefficient (altitude_ki); k5: Airspeed control proportional coefficient based on pitch angle (airspeed_pitch_kp); k6: Airspeed control integral coefficient based on pitch angle (airspeed_pitch_ki); k7: Airspeed_throttle_kp, a proportional coefficient for airspeed control based on throttle. k8: Integral coefficient for airspeed control based on throttle; k9: Roll attitude control proportional coefficient roll_kp; k10: Roll attitude control differential coefficient roll_kd; k11: Course_kp: Track maintenance control proportional coefficient; k12: Integral coefficient for track maintenance control, course_ki; k13: Sideslip holding control proportional coefficient beta_kp; k14: Integral coefficient for sideslip hold control, beta_ki.

4. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 1, characterized in that, The initial state includes: North and east position and altitude coordinates in inertial coordinate system pn, pd, h Velocity in body coordinate system u, v, w ], attitude angle [ phi, theta, psi ], angular velocity [ p, q, r Aerodynamic parameters Va, alpha, beta ], rudder [ delta_e, delata_a, delta_r, delta_t ].

5. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 1, characterized in that, The calculation of the reward value r comprises the following four parts: Steady-state error reward, overshoot reward, system stability reward, and system instability reward.

6. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 5, characterized in that, The expression for the steady-state error reward is as follows: ； in, As a reward for the steady-state error, For steady-state error, The time for each step.

7. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 5, characterized in that, The expression for the overshoot reward is as follows: ； in, The reward is for the overshoot portion. For systematic error, This is the expected value.

8. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 5, characterized in that, The expression for the system's stable reward is as follows: ； in, Partial rewards for system stability For systematic error, This is the expected value.

9. The method for tuning parameters of an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism according to claim 5, characterized in that, The expression for the system instability reward is as follows: ； in, The reward is for the part of the system that is unstable; "instable" means unstable, and "stable" means not unstable.

10. A parameter tuning device for an unmanned aerial vehicle (UAV) control system based on a brain-like mechanism, characterized in that, The neuromorphic mechanism-based unmanned aerial vehicle (UAV) control system parameter tuning device includes: The simulation model building module is used to create UAV simulation models; The UAV control system construction module is used to build a UAV control system based on a PID controller. The initial state determination module is used to generate the initial state s of the reinforcement learning environment based on the control expectation of the PID controller and the current flight state of the UAV simulation model or the actual UAV system. The action space determination module is used to input the initial state s into a pre-trained policy network, and the policy network outputs a set of action spaces A, which are the control parameters of the PID controller. The PID controller update module is used to load the control parameters into the PID controller, update the parameters of the PID controller, and obtain the updated PID controller. The execution module is used to generate control commands based on the control expectation using the updated PID controller, and send them to the UAV simulation model or the actual UAV system. The UAV simulation model or the actual UAV system then executes an action of one step size to generate new flight state data, denoted as the new state s'. The reward value calculation module is used to calculate the reward value r based on the difference between the new state s' and the control expectation; The parameter update module is used to store tuples s, A, r, s' into the experience replay pool and update the parameters of the policy network and value network using the SAC algorithm. The termination module is used to take the new state s' as the current state s of the next time step, and repeat the process from the UAV control system construction module to the parameter update module until the termination conditions set by the system are met. The termination conditions include: reaching a predetermined response time, the control system completing the response and reaching a stable state, or the system judging that it is unstable.

Images