Method and device for controlling unpowered circling of fixed-wing unmanned aerial vehicle
Through parallel evaluation reinforcement learning and PID hierarchical control methods, the problem of hover control of fixed-wing drones when they lose power is solved, and the anti-interference ability and safety in harsh environments are improved.
Patent Information
- Application Number
- CN202410929178.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-26
Smart Images

Figure CN119024877A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of UAV control technology, and in particular relates to a method and device for controlling unpowered hovering of a fixed-wing UAV. Background Art
[0002] With the development of aviation and manufacturing industries, drones have developed rapidly and efficiently in the past decade. In the civil and military fields, drones are favored because of their high cost-effectiveness, high flexibility and high safety. With the increasing maturity of fixed-wing drone technology, more and more fixed-wing drones have been put into work in different fields, and the corresponding safety and reliability issues are becoming increasingly severe. The biggest difference from manned aircraft is that due to the lack of on-site operation by pilots, drones are often unable to take corresponding emergency measures when a failure occurs. In the absence of insufficient human intervention, communication failures, bad weather, and sensor failures may all cause crashes.
[0003] Intelligent drone behavior decision-making refers to the use of artificial intelligence and digital system technology to enable drones to independently analyze, understand and respond to various situations in the environment, so as to make decisions and perform tasks autonomously. By combining sensor technology, machine learning algorithms and autonomous control systems, drones can perceive and understand the surrounding environment in real time, plan appropriate action plans, and make corresponding adjustments when necessary to deal with emergencies. This intelligent drone behavior decision-making not only improves the efficiency and accuracy of task execution, but also enhances the safety and stability of the system, providing reliable technical support for the widespread use of drones in various application fields.
[0004] However, compared with rotary-wing UAVs, fixed-wing UAVs have more types of onboard sensors, more complex dynamic models, and are more susceptible to weather effects. Especially when the aircraft loses power and encounters bad weather, safe forced landing is particularly important. Traditional control algorithms often require a lot of optimization to improve control efficiency, and the control effect is not ideal, resulting in poor anti-interference ability during unpowered autonomous hovering and inability to land safely. Summary of the invention
[0005] In order to solve the above problems, the present application provides a method and device for controlling unpowered hovering of a fixed-wing UAV, based on a hierarchical control method of parallel judgment reinforcement learning and PID, so that the fixed-wing UAV can perform hovering flight missions in complex environments when it loses power.
[0006] The first aspect of the present application provides a method for controlling unpowered hovering of a fixed-wing UAV, mainly comprising:
[0007] Step S1, obtaining the initially given hovering point, hovering radius, initial position and drone status information;
[0008] Step S2: Periodically extract the environmental information of the drone, and update the state information of the drone according to the turbulence speed at the location of the drone;
[0009] Step S3, inputting the state information and environmental information of the UAV into a pre-trained parallel evaluation reinforcement learning model, and outputting the heading angle instruction of the UAV, wherein the parallel evaluation reinforcement learning model takes the state information and environmental information of the UAV as input in advance, takes the heading angle instruction as output, takes conforming to the ideal hovering trajectory as the optimization goal, and optimizes the strategy parameters of the parallel evaluation reinforcement learning model through multiple iterative updates;
[0010] Step S4, outputting the heading angle command as the roll angle command through PID control;
[0011] Step S5: Control the UAV to roll based on the roll angle command to achieve unpowered hovering of the UAV.
[0012] Preferably, step S2 further comprises:
[0013] Update the drone speed using the following formula:
[0014]
[0015] Among them, V old and V new Represents the speed of the drone before and after the update, V t is the turbulent velocity, CD0 and CL0 represent the zero lift coefficient and zero lift coefficient respectively, m is the aircraft mass, △t is the control period
[0016] Update the drone heading angle using the following formula:
[0017]
[0018] Among them, ψ old and ψ new Represent the heading angles before and after the update, C is the coupling coefficient, is the current rolling angle, S is the wing area;
[0019] Update the drone's roll angle using the following formula:
[0020]
[0021] in, and are the rolling angles before and after the update, and ψ is the current heading angle.
[0022] Preferably, before step S3, the method further includes training the parallel evaluation reinforcement learning model. During the training process, the turbulent velocity at the location of the UAV is obtained by:
[0023] A grid matrix of a specified size is formed within the preset size range, and a wind speed value is randomly generated from the interval [-15m / s, 15m / s] for each node of the grid matrix;
[0024] The grid matrix is smoothed by using spline interpolation to obtain the turbulence field within the preset size range.
[0025] Preferably, in step S3, training the parallel judgment reinforcement learning model further includes:
[0026] The training samples containing the drone status information and environment information are simultaneously input into the judgment network and the execution network;
[0027] The evaluation network outputs the heading angle command of the UAV through multiple network layers, and the execution network integrates the heading angle command of the UAV, the UAV status information and the environmental information, and outputs the evaluation value through multiple network layers;
[0028] The judgment value is compared with an objective value calculated by a pre-constructed judgment function to optimize the policy parameters of the parallel judgment reinforcement learning model, wherein the judgment function is set to have a set high score when the flight trajectory conforms to an ideal hovering trajectory.
[0029] Preferably, the activation function used by the last network layer of the evaluation network is the tanh function, and the activation functions used by other network layers of the evaluation network are Relu functions.
[0030] Preferably, in step S3, before inputting the state information and environmental information of the drone into the pre-trained parallel judgment reinforcement learning model, the method further includes:
[0031] Step S31, calculating the dynamic standard deviation of the input data of the current node according to the average value of the input data of multiple previous nodes;
[0032] Step S32: Perform zero-mean normalization processing on the input data of the current node according to the dynamic standard deviation to obtain the processed input data of the current node.
[0033] The second aspect of the present application provides a fixed-wing UAV unpowered hovering control device which mainly includes:
[0034] The initial parameter acquisition module is used to obtain the initial given hovering point, hovering radius, initial position and drone status information;
[0035] The UAV status information acquisition module is used to periodically extract the UAV's environmental information and update the UAV's status information according to the turbulence speed at the UAV's location;
[0036] The heading angle command output module is used to input the state information and environmental information of the UAV into the pre-trained parallel evaluation reinforcement learning model, and output the heading angle command of the UAV. The parallel evaluation reinforcement learning model takes the state information and environmental information of the UAV as input in advance, takes the heading angle command as output, takes the ideal hovering trajectory as the optimization goal, and optimizes the strategy parameters of the parallel evaluation reinforcement learning model through multiple iterative updates;
[0037] The roll angle command output module is used to output the rolling angle command through PID control;
[0038] The UAV control module is used to control the UAV's roll based on the roll angle command to achieve unpowered hovering of the UAV.
[0039] Preferably, the drone status information acquisition module includes:
[0040] The drone speed update unit is used to update the drone speed using the following formula:
[0041]
[0042] Among them, V old and V new Represents the speed of the drone before and after the update, V t is the turbulent velocity, CD0 and CL0 represent the zero lift coefficient and zero lift coefficient respectively, m is the aircraft mass, △t is the control period
[0043] The drone heading angle update unit is used to update the drone heading angle using the following formula:
[0044]
[0045] Among them, ψ old and ψ new Represent the heading angles before and after the update, C is the coupling coefficient, is the current rolling angle, S is the wing area;
[0046] The drone roll angle update unit is used to update the drone roll angle using the following formula:
[0047]
[0048] in, and are the rolling angles before and after the update, and ψ is the current heading angle.
[0049] Preferably, the heading angle command output module further comprises:
[0050] A dynamic standard deviation calculation unit, used to calculate the dynamic standard deviation of the input data of the current node according to the average value of the input data of multiple previous nodes;
[0051] The input data normalization unit is used to perform zero-mean normalization processing on the input data of the current node according to the dynamic standard deviation to obtain the processed input data of the current node.
[0052] A third aspect of the present application provides a drone, comprising a processor, wherein the processor is configured to execute the fixed-wing drone unpowered hovering control method as described above.
[0053] This application not only overcomes the problem of slow control response, but also can effectively deal with external and model uncertainty disturbances, significantly improving the robustness of control. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a flow chart of a preferred embodiment of the unpowered hovering control method of the fixed-wing UAV of the present application.
[0055] Figure 2 This application Figure 1 Schematic diagram of turbulent field gridding in the illustrated embodiment.
[0056] Figure 3 This is a schematic diagram of the turbulent field after spline interpolation.
[0057] Figure 4 It is a schematic diagram of the evaluation network and execution network architecture.
[0058] Figure 5 This is a schematic diagram comparing the hovering effects of multiple algorithm drones. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the implementation of this application will be described in more detail in combination with the drawings in the implementation of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described implementation is a part of the implementation of this application, not all of the implementations. The implementation described below with reference to the drawings is exemplary and is intended to be used to explain this application, and cannot be understood as a limitation on this application. Based on the implementation in this application, all other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The implementation of this application is described in detail below in combination with the drawings.
[0060] The first aspect of the present application provides a method for controlling a fixed-wing UAV to hover without power, such as Figure 1 As shown, it mainly includes:
[0061] Step S1, obtaining the initially given hovering point, hovering radius, initial position and drone status information;
[0062] Step S2: Periodically extract the environmental information of the drone, and update the state information of the drone according to the turbulence speed at the location of the drone;
[0063] Step S3, inputting the state information and environmental information of the UAV into a pre-trained parallel evaluation reinforcement learning model, and outputting the heading angle instruction of the UAV, wherein the parallel evaluation reinforcement learning model takes the state information and environmental information of the UAV as input in advance, takes the heading angle instruction as output, takes conforming to the ideal hovering trajectory as the optimization goal, and optimizes the strategy parameters of the parallel evaluation reinforcement learning model through multiple iterative updates;
[0064] Step S4, outputting the heading angle command as the roll angle command through PID control;
[0065] Step S5: Control the UAV to roll based on the roll angle command to achieve unpowered hovering of the UAV.
[0066] In the present application, steps S1 to S3 are to output a heading angle instruction using a parallel evaluation reinforcement learning model, step S4 is to convert the heading angle instruction into a roll angle instruction through PID control, and step S5 is used to control the hovering motion of the drone according to the roll angle instruction.
[0067] In step S3, a pre-trained parallel judgment reinforcement learning model is used. From the perspective of use, this application provides the input data of the model through steps S1-S2. It can be understood that from the perspective of model training, these input data are also needed. The difference is that since the purpose of this application is to adapt to turbulent weather when the drone needs to fly without power, when using the model, the drone can obtain the turbulent speed of its location in real time, while when training the model, it is necessary to artificially construct a turbulent field.
[0068] Therefore, in some optional implementations, step S3 further includes training the parallel evaluation reinforcement learning model. During the training process, the turbulent velocity at the location of the drone is obtained by:
[0069] A grid matrix of a specified size is formed within the preset size range, and a wind speed value is randomly generated from the interval [-15m / s, 15m / s] for each node of the grid matrix;
[0070] The grid matrix is smoothed by using spline interpolation to obtain the turbulence field within the preset size range.
[0071] In the generation of turbulence, this embodiment first generates a 5000 feet * 5000 feet area to form a turbulent field later, and then grids the turbulent field in units of 1000 feet to obtain a 6*6 (0, 1000 feet, 2000 feet, 3000 feet, 4000 feet, 5000 feet) grid matrix. Each element in this matrix is set to a random number uniformly distributed between 0 and 1, and this row of random numbers is linearly mapped to the interval [-15m / s, 15m / s], such as Figure 2 The positive and negative represent the direction of the wind relative to the aircraft. A wind speed of 15m / s is roughly equivalent to a force 7 wind in reality.
[0072] In practice, the relationship between wind speed and position is not linear, so nonlinear processing is required. Here, spline interpolation is used to smooth the grid matrix:
[0073]
[0074] Among them, S(x,y) represents the value estimated at the point (x,y), a ij is the interpolation coefficient, B i (x) and B j (y) is the spline basis function. The turbulent field after spline interpolation is as follows Figure 3 As shown in the figure, the turbulence velocity used by the training model is provided in this environment to complete the hovering and hovering transfer tasks during the training process.
[0075] In step S2, the drone environment information is the coordinates of the drone relative to the hovering point (center of the circle) and the local wind speed, which is the turbulent velocity measured or extracted from the turbulent field. Then, the drone status information is updated based on the turbulent velocity, mainly updating the speed, heading angle and roll angle. The drone status information is shown in Table 1 below.
[0076] Table 1 UAV status data
[0077]
[0078]
[0079] In some optional implementations, step S2 further includes:
[0080] Update the drone speed using the following formula:
[0081]
[0082] Among them, Vold and V new Represents the speed of the drone before and after the update, V t is the turbulent velocity, CD0 and CL0 represent the zero lift coefficient and zero lift coefficient respectively, m is the aircraft mass, △t is the control period
[0083] Update the drone heading angle using the following formula:
[0084]
[0085] Among them, ψ old and ψ new Represent the heading angles before and after the update, C is the coupling coefficient, is the current rolling angle, S is the wing area;
[0086] Update the drone's roll angle using the following formula:
[0087]
[0088] in, and are the rolling angles before and after the update, and ψ is the current heading angle.
[0089] It should also be noted that for the heading angle and roll angle, the turbulence velocity V t There are positive and negative directions, representing the left and right sides of the drone respectively. This direction will affect the direction of change of the heading angle and roll angle.
[0090] In the process of using and training the model, in addition to the difference in turbulent fields in the input data, the architecture of using and training the model is also slightly different. This application constructs a parallel evaluation reinforcement learning model, which uses a parallel structure of multiple deep neural networks, mainly including an evaluation network (Appraisal Network) and an execution network (ImplementNetwork). The evaluation network is applied in the process of training the model to evaluate the output results of the execution network, so that the model can train better strategy parameters. When using the model, the execution network can be used directly.
[0091] In some optional implementations, in step S3, training the parallel judgment reinforcement learning model further includes:
[0092] The training samples containing the drone status information and environment information are simultaneously input into the judgment network and the execution network;
[0093] The evaluation network outputs the heading angle command of the UAV through multiple network layers, and the execution network integrates the heading angle command of the UAV, the UAV status information and the environmental information, and outputs the evaluation value through multiple network layers;
[0094] The judgment value is compared with an objective value calculated by a pre-constructed judgment function to optimize the policy parameters of the parallel judgment reinforcement learning model, wherein the judgment function is set to have a set high score when the flight trajectory conforms to an ideal hovering trajectory.
[0095] like Figure 4 As shown in the figure, the evaluation network and the execution network use a two-dimensional convolutional network when processing state information. This is to capture the spatial relationship and local features between states, and to explore the state space more effectively during the learning process, thereby improving learning efficiency and performance. The action output by the execution network is the heading angle command. At the same time, the heading angle command will also be input into the evaluation network, and after dimension unification by appraisal_fc3, it will be weighted with the output of appraisal_fc2. This is done to allow the evaluation network to output the evaluation value based on the current state and action, that is, the evaluation network will make a value judgment based on the current action taken. In order to solve the problem of overestimation of the evaluation value, the evaluation value is fitted and a smaller evaluation value is finally selected, thereby greatly alleviating the problem of overestimation of the evaluation value.
[0096] In the process of interaction between the drone and the environment, a judgment function needs to be set to limit the flight behavior of the drone. In the construction of the judgment function, a continuous method is used to design a segment judgment function. The continuous judgment function provides a reward signal at each step of the drone, which means that the drone can get corresponding feedback at each step. In essence, the drone hovering task is a more flexible task type, so using continuous rewards can accelerate the learning process and improve the stability of the algorithm.
[0097] For hovering missions when the drone loses engine power, it is necessary to keep the actual flight trajectory of the drone as close to the planned ideal trajectory as possible. In addition, the flight state should be kept as stable as possible to avoid making dangerous and exciting movements in a short period of time. Based on these two points, the evaluation function is divided into four main parts.
[0098] (1) Direction deviation penalty and reward: By calculating the error between the current position of the UAV and the ideal position, it is determined whether the flight direction of the UAV is within a reasonable range.
[0099] (2) Trajectory deviation penalties and rewards: The ideal trajectory is transformed into a range area that is approximately circular. If the drone continues to fly in this area, the agent will continue to be rewarded; if it exceeds this range for a long time, the agent will continue to be punished.
[0100] (3) Stability rewards and penalties: In hovering tasks, the ideal roll angle of the drone should remain constant; the ideal heading angle should change regularly. By penalizing sudden changes in posture, the network can be trained to output a stable heading angle.
[0101] (4) Time smoothing: By adding time smoothing to the judgment function, the output of the neural network can maintain continuity and stability, and the impact of noise can be alleviated to a certain extent.
[0102] In the construction of the judgment function, this application uses an intuitive judgment function that is easy for researchers to understand. In the traditional UAV behavior control problem, the UAV model formula is more in line with the physical laws of the real world. When these formulas are converted into judgment functions, it is often necessary to significantly adjust some weight coefficients, and these coefficients usually differ by several orders of magnitude, which is not conducive to the adjustment of hyperparameters and network training. The judgment function should formally describe the mission objectives. By designing a suitable judgment function, it can be clearly told what kind of behavior the drone should pursue in the process of completing the task, and the quality of the behavior can be quantified according to the specific nature of the task.
[0103] In summary, the judgment function mainly judges the quality of drone flight. It is an objective value, and the score is added or subtracted according to the quality of flight. During the training process, when the value of the judgment function converges at a high level, it indicates that the model has been trained. The judgment network mainly outputs the action output by the execution network to judge and obtain the judgment value, which is an estimated value, which is different from the objective value obtained by the previous judgment function. The role of the judgment network is to continuously make the predicted value close to the objective value during the training process, so that the network can know whether its actions are good or bad, thereby improving performance.
[0104] In some optional implementations, the activation function used by the last network layer of the evaluation network is the tanh function, and the activation function used by other network layers of the evaluation network is the Relu function. The Relu activation functions are connected to improve the fitting ability of the model, and the tanh function limits the output of the execution network to between (-1, 1), and then scales and outputs a specific heading angle.
[0105] In some optional implementations, in step S3, before inputting the state information and environment information of the drone into the pre-trained parallel judgment reinforcement learning model, the method further includes:
[0106] Step S31, calculating the dynamic standard deviation of the input data of the current node according to the average value of the input data of multiple previous nodes;
[0107] Step S32: Perform zero-mean normalization processing on the input data of the current node according to the dynamic standard deviation to obtain the processed input data of the current node.
[0108] It is understandable that the use of a parallel judgment reinforcement learning model actually treats the attitude control wandering of a fixed-wing UAV as a Markov process. The state input of the parallel judgment reinforcement learning model consists of various features. Before these features are input into the network, they need to be normalized to avoid the degradation of the model generalization performance caused by data scale differences, that is, to normalize the input data of different scales to between (-1,1). In addition to improving the generalization ability of the model, this embodiment also uses a dynamic paradigm to save computing resources and improve efficiency through a recursive model. Compared with static normalization, dynamic normalization allows the network to have stronger adaptability because changes in the mean and variance do not completely depend on historical data.
[0109] As shown in step S31, the input parameters are first dynamically processed by the upper and lower formulas:
[0110]
[0111] Among them, x is the input parameter, Indicates the average value of the input parameter at the previous n time nodes, so as to calculate the average value of the input parameter at the current node Then the standard deviation σ of the input parameter is obtained after the variance standard deviation is calculated. new .
[0112] After the variance is obtained, the input parameters can be processed by zero mean standardization in step S32 to standardize them into a normal distribution with a mean of 0 and a standard deviation of 1. The zero mean standardization expression is as follows:
[0113]
[0114] x1 is the input parameter after standardization.
[0115] This application uses a parallel-criteria reinforcement learning method to train the UAV agent. The network architecture of the parallel-criteria reinforcement learning method allows the UAV to obtain more information in the interaction with the environment, so as to execute more correct flight strategies and adjust the flight attitude in real time.
[0116] The flight dynamics of fixed-wing UAVs are very complex, and the entire system is highly nonlinear, so it is impossible to fit all fixed-wing aircraft with a unified model. However, the parallel evaluation reinforcement learning method proposed in this application can simulate the flight environment and flight data of different types of aircraft to obtain a more accurate nonlinear model.
[0117] This application uses dynamic normalization, which, on the one hand, keeps the network input data at the same order of magnitude, and on the other hand, enables the UAV agent to be adaptive in different environments. In addition, the dynamic normalization process enriches the amount of information in the data, thereby improving the training efficiency of the model.
[0118] The technical effect is further described in detail below through experiments.
[0119] First, we set up the evaluation function, assign different weights to direction deviation, trajectory deviation, stability, and time smoothing, and build a continuous evaluation function for subsequent adjustments. At the same time, we build a parallel evaluation reinforcement learning network scale and set the initial learning rate.
[0120] Then it is divided into two groups of experiments. In one group of experiments, the initial hovering point, hovering radius and initial position are set. Using dynamic normalization, 1500 rounds of training are carried out in the environment to ensure the convergence of the judgment function, so that the network can understand the control strategy of the drone. In another group of experiments, a simulation environment with a turbulent field is established, and it is set to a smooth turbulent field, and the wind speed is set between [-15,15]. The initial hovering point, hovering radius and initial position are set, and the hovering transfer task under turbulence is performed. Also train for 1500 rounds. After completing the training, the method proposed in this application and other methods are tested at the same time and the effects are compared.
[0121] Effect comparison Figure 5 As shown, Figure 5 In the figure, the small standard dotted circle is the circling circle given in the first group of experiments, the large standard dotted circle on the left is the circling circle given in the second group of experiments, and the solid line is the corresponding UAV simulation trajectory.
[0122] from Figure 5It can be seen that in a non-turbulent environment, all three methods show good results. Among them, the average heading change of the forward point tracking algorithm is the smallest, making its hovering trajectory the smoothest. The average deviation of the two reinforcement learning algorithms, parallel judgment reinforcement learning and DDPG, is smaller, so they are closer to the ideal hovering trajectory. However, after the turbulent field is added to the environment, the turbulent field greatly affects the lateral control of the drone, and because the influence of turbulence is constantly changing, the attitude of the drone needs to be adjusted frequently. In a turbulent environment, the forward point tracking algorithm shows obvious lack of robustness. This is because too much error accumulated by the previous step length leads to poor anti-interference ability of the algorithm in turbulence. Parallel judgment reinforcement learning uses turbulence as information input, and during the training process, the judgment function allows the agent to achieve a certain balance between heading angle output and roll angle stability. In the process of policy optimization of DDPG and parallel judgment reinforcement learning, parallel judgment reinforcement learning not only considers the cumulative reward, but also considers the long-term feasibility of the strategy, which enables parallel judgment reinforcement learning to better explore the environment during training. Therefore, parallel-criteria reinforcement learning is more robust in turbulent environments.
[0123] The second aspect of the present application provides a fixed-wing UAV unpowered hovering control device corresponding to the above method, mainly comprising:
[0124] The initial parameter acquisition module is used to obtain the initial given hovering point, hovering radius, initial position and drone status information;
[0125] The UAV status information acquisition module is used to periodically extract the UAV's environmental information and update the UAV's status information according to the turbulence speed at the UAV's location;
[0126] The heading angle command output module is used to input the state information and environmental information of the UAV into the pre-trained parallel evaluation reinforcement learning model, and output the heading angle command of the UAV. The parallel evaluation reinforcement learning model takes the state information and environmental information of the UAV as input in advance, takes the heading angle command as output, takes the ideal hovering trajectory as the optimization goal, and optimizes the strategy parameters of the parallel evaluation reinforcement learning model through multiple iterative updates;
[0127] The roll angle command output module is used to output the rolling angle command through PID control;
[0128] The UAV control module is used to control the UAV's roll based on the roll angle command to achieve unpowered hovering of the UAV.
[0129] In some optional implementations, the drone status information acquisition module includes:
[0130] The drone speed update unit is used to update the drone speed using the following formula:
[0131]
[0132] Among them, V old and V new Represents the speed of the drone before and after the update, V t is the turbulent velocity, CD0 and CL0 represent the zero lift coefficient and zero lift coefficient respectively, m is the aircraft mass, △t is the control period
[0133] The drone heading angle update unit is used to update the drone heading angle using the following formula:
[0134]
[0135] Among them, ψ old and ψ new Represent the heading angles before and after the update, C is the coupling coefficient, is the current rolling angle, S is the wing area;
[0136] The drone roll angle update unit is used to update the drone roll angle using the following formula:
[0137]
[0138] in, and are the rolling angles before and after the update, and ψ is the current heading angle.
[0139] In some optional implementations, the heading angle command output module further includes:
[0140] A dynamic standard deviation calculation unit, used to calculate the dynamic standard deviation of the input data of the current node according to the average value of the input data of multiple previous nodes;
[0141] The input data normalization unit is used to perform zero-mean normalization processing on the input data of the current node according to the dynamic standard deviation to obtain the processed input data of the current node.
[0142] A third aspect of the present application provides a drone, comprising a processor, wherein the processor is configured to execute the fixed-wing drone unpowered hovering control method as described above.
[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for controlling unpowered hovering of a fixed-wing UAV, characterized in that: include: Step S1, obtaining the initially given hovering point, hovering radius, initial position and drone status information; Step S2: Periodically extract the environmental information of the drone, and update the state information of the drone according to the turbulence speed at the location of the drone; Step S3, inputting the state information and environmental information of the UAV into a pre-trained parallel evaluation reinforcement learning model, and outputting the heading angle instruction of the UAV, wherein the parallel evaluation reinforcement learning model takes the state information and environmental information of the UAV as input in advance, takes the heading angle instruction as output, takes conforming to the ideal hovering trajectory as the optimization goal, and optimizes the strategy parameters of the parallel evaluation reinforcement learning model through multiple iterative updates; Step S4, outputting the heading angle command as the roll angle command through PID control; Step S5: Control the UAV to roll based on the roll angle command to achieve unpowered hovering of the UAV.
2. The method for controlling the unpowered hovering of a fixed-wing UAV according to claim 1, characterized in that: Step S2 further comprises: Update the drone speed using the following formula: Among them, V old and V new Represents the speed of the drone before and after the update, V t is the turbulent velocity, CD0 and CL0 represent the zero lift coefficient and zero lift coefficient respectively, m is the aircraft mass, Δt is the control period, and the heading angle of the UAV is updated through the following formula: Among them, ψ old and ψ new Represent the heading angles before and after the update, C is the coupling coefficient, is the current roll angle, S is the wing area; Update the drone's roll angle using the following formula: in, and are the rolling angles before and after the update, and ψ is the current heading angle.
3. The method for controlling the unpowered hovering of a fixed-wing UAV according to claim 1, characterized in that: Before step S3, the parallel judgment reinforcement learning model is further trained. During the training process, the turbulence velocity at the location of the UAV is obtained by: A grid matrix of a specified size is formed within the preset size range, and a wind speed value is randomly generated from the interval [-15m / s, 15m / s] for each node of the grid matrix; The grid matrix is smoothed by using spline interpolation to obtain the turbulence field within the preset size range.
4. The method for controlling the unpowered hovering of a fixed-wing UAV as claimed in claim 3, characterized in that: In step S3, training the parallel judgment reinforcement learning model further includes: The training samples containing the drone status information and environment information are simultaneously input into the judgment network and the execution network; The evaluation network outputs the heading angle command of the UAV through multiple network layers, and the execution network integrates the heading angle command of the UAV, the UAV status information and the environmental information, and outputs the evaluation value through multiple network layers; The judgment value is compared with an objective value calculated by a pre-constructed judgment function to optimize the policy parameters of the parallel judgment reinforcement learning model, wherein the judgment function is set to have a set high score when the flight trajectory conforms to an ideal hovering trajectory.
5. The method for controlling the unpowered hovering of a fixed-wing UAV as claimed in claim 4, characterized in that: The activation function used by the last network layer of the evaluation network is the tanh function, and the activation function used by other network layers of the evaluation network is the Relu function.
6. The method for controlling the unpowered hovering of a fixed-wing UAV according to claim 1, characterized in that: In step S3, before the state information and environment information of the drone are input into the pre-trained parallel judgment reinforcement learning model, the method further includes: Step S31, calculating the dynamic standard deviation of the input data of the current node according to the average value of the input data of multiple previous nodes; Step S32: Perform zero-mean normalization processing on the input data of the current node according to the dynamic standard deviation to obtain the processed input data of the current node.
7. A fixed-wing UAV unpowered hovering control device, characterized in that: include: The initial parameter acquisition module is used to obtain the initial given hovering point, hovering radius, initial position and drone status information; The UAV status information acquisition module is used to periodically extract the UAV's environmental information and update the UAV's status information according to the turbulence speed at the UAV's location; The heading angle command output module is used to input the state information and environmental information of the UAV into the pre-trained parallel evaluation reinforcement learning model, and output the heading angle command of the UAV. The parallel evaluation reinforcement learning model takes the state information and environmental information of the UAV as input in advance, takes the heading angle command as output, takes the ideal hovering trajectory as the optimization goal, and optimizes the strategy parameters of the parallel evaluation reinforcement learning model through multiple iterative updates; The roll angle command output module is used to output the rolling angle command through PID control; The UAV control module is used to control the UAV's roll based on the roll angle command to achieve unpowered hovering of the UAV.
8. The unpowered hovering control device for a fixed-wing UAV as claimed in claim 7, characterized in that: The drone status information acquisition module includes: The drone speed update unit is used to update the drone speed using the following formula: Among them, V old and V new Represents the speed of the drone before and after the update, V t is the turbulent velocity, CD0 and CL0 represent the zero lift coefficient and zero lift coefficient respectively, m is the aircraft mass, Δt is the control period, and the UAV heading angle update unit is used to update the UAV heading angle through the following formula: Among them, ψ old and ψ new Represent the heading angles before and after the update, C is the coupling coefficient, is the current rolling angle, S is the wing area; The drone roll angle update unit is used to update the drone roll angle using the following formula: in, and are the rolling angles before and after the update, and ψ is the current heading angle.
9. The unpowered hovering control device for a fixed-wing UAV as claimed in claim 7, characterized in that: The heading angle command output module further includes: A dynamic standard deviation calculation unit, used to calculate the dynamic standard deviation of the input data of the current node according to the average value of the input data of multiple previous nodes; The input data normalization unit is used to perform zero-mean normalization processing on the input data of the current node according to the dynamic standard deviation to obtain the processed input data of the current node.
10. A drone, characterized in that: It includes a processor, which is configured to execute the fixed-wing UAV unpowered hovering control method as described in any one of claims 1-6.
