A method for constructing a drone control model and a drone control method

Abstract

The unmanned aerial vehicle control model construction method and the unmanned aerial vehicle control method, the unmanned aerial vehicle control model construction method is constructed through setting up the compound reward function containing the attitude guide reward item and the nonlinear position penalty item, the quaternion attitude guide reward realizes the accurate guidance without singularity to the large angle rotation of any axis, the nonlinear position penalty provides the necessary fault tolerance space at the moment of turning over, and forms the strong constraint to the position drift, realizes the dynamic balance between the large maneuver action and the fixed point hovering, effectively solves the common position drift problem in the existing deep reinforcement learning control, and significantly improves the steady state accuracy and flight stability of the unmanned aerial vehicle after executing the large maneuver action.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, specifically to a method for constructing a UAV control model and a UAV control method. Background Technology

[0002] With the rapid development of drone technology, it has been widely applied in many fields such as aerial surveying and mapping, power line inspection, logistics distribution, and military reconnaissance. In actual operation scenarios, drones (such as quadcopter drones) often need to complete rapid, high-maneuvering maneuvers (such as a 180-degree roll around the X-axis of the drone). These maneuvers require the drone to complete a large change in attitude in a very short time; at the same time, after the maneuver is executed, attitude oscillations must be quickly suppressed and position drift offset to achieve high-precision steady-state hovering. This dual requirement of "rapid maneuvering" and "precise steady-state" poses stringent challenges to the dynamic response speed, nonlinear characteristic adaptability, and steady-state control accuracy of the control algorithm.

[0003] Currently, the most commonly used attitude control methods for UAVs, including PID control (proportional-integral-derivative) and deep reinforcement learning-based control methods, cannot meet the high-precision hovering requirements after large maneuvers. Existing deep reinforcement learning-based control methods achieve end-to-end training using standard reinforcement learning algorithms (such as DQN and primitive PPO), inputting UAV state information into the neural network, which directly outputs control actions. The network parameters are updated based on reward signals from environmental feedback, gradually optimizing the control strategy. However, due to the lack of specificity in the reward function design, the "reward hacking" phenomenon easily occurs. To obtain sparse attitude rewards, the UAV may learn to rapidly rotate and sweep across the target angle, leading to attitude oscillations and position drift after large maneuvers, making steady-state hovering impossible; or, to avoid penalties such as crashing into the ground, it may choose to escape the target altitude, resulting in problems such as "flying higher and higher" or severe position drift. Summary of the Invention

[0004] In view of the above problems, this application provides a method for constructing a UAV control model, which solves the problem that in existing UAV control methods based on deep reinforcement learning, UAVs are prone to attitude oscillation and position drift after completing large maneuvers, making it difficult to achieve steady-state hovering.

[0005] To achieve the above objectives, the inventors provide a method for constructing a drone control model, which includes the following steps:

[0006] A physics engine-based UAV simulation environment is constructed, and a state space is defined; the state information in the state space includes the UAV's position, quaternion attitude, linear velocity, and angular velocity.

[0007] Construct a policy network and a value network; wherein the policy network is used to receive state information and output the next action command of the UAV; the value network is used to receive state information and perform state value estimation.

[0008] A composite reward function is set up, which includes an attitude guidance reward term and a nonlinear position penalty term. The attitude guidance reward term calculates the reward value based on the geodesic error between the current attitude and the target attitude using quaternion attitude calculation. The nonlinear position penalty term calculates the penalty value based on the square of the deviation between the current position of the UAV and the target position.

[0009] Based on the current state information of the UAV, the system interacts with the UAV simulation environment through a policy network, collects interaction data and stores it in an experience buffer, and repeats the interaction process until the amount of data collected reaches a preset number of steps. The data in the experience buffer is used as the training set for the centralized update network. The interaction data includes the current state information, the next action command, the reward and the next state information. The reward is calculated by the composite reward function.

[0010] Based on the interaction data, the value network is used to estimate the state value corresponding to each state information, calculate the advantage function and reward objective, and iteratively update the parameters of the policy network and the value network based on the deep reinforcement learning algorithm until the preset convergence condition is reached, and output the trained policy network as the UAV control model.

[0011] Furthermore, the composite reward function also includes a stability regularization penalty, which includes penalizing one or more of the Euler angles, linear velocities, angular velocities, and control variable rates of change of the non-target axes of the UAV.

[0012] Furthermore, the composite reward function also includes an angular velocity gating reward term, which is configured to provide a first reward value when the attitude error, position error, and angular velocity amplitude of the UAV are all less than their respective preset thresholds; otherwise, the first reward value is not provided.

[0013] Furthermore, the angular velocity gating reward item is also configured to provide a second reward value when the UAV meets the conditions that the attitude error, position error and angular velocity amplitude are less than their respective preset thresholds for multiple consecutive time steps, and the second reward value is greater than the first reward value.

[0014] Furthermore, when the UAV meets the condition that the attitude error, position error, and angular velocity amplitude are less than their respective preset thresholds for multiple consecutive time steps, the interactive data acquisition for the current round is terminated.

[0015] Further, collect interaction data, including the following steps:

[0016] Obtain the current status information of the drone and perform normalization processing;

[0017] The normalized state information is input into the policy network, the next action command of the UAV is output, the action command is mapped into motor control command and then sent to the UAV simulation environment;

[0018] The UAV simulation environment performs dynamic calculations based on the motor control commands and returns the next state information;

[0019] Based on the current state information, the next action instruction, and the next state information, the composite reward function is called to calculate the reward value;

[0020] The current state information, the action command, the reward value, and the next state information are stored as a set of interactive data in a buffer, and the above steps are repeated until the data collection of the preset time step is completed.

[0021] Furthermore, during the process of collecting interactive data, the action commands output by the policy network are superimposed with exploration noise.

[0022] Furthermore, when iteratively updating the parameters of the policy network and the value network based on the deep reinforcement learning algorithm, a maximum number of training steps is set; if the preset convergence condition is not reached within the maximum number of training steps, the parameters of the UAV simulation environment and the deep reinforcement learning algorithm are initialized, and a new round of training begins.

[0023] Furthermore, the deep reinforcement learning algorithm is a proximal policy optimization algorithm.

[0024] A method for controlling an unmanned aerial vehicle (UAV) includes the following steps:

[0025] Obtain the current status information of the drone;

[0026] Normalize the current state information;

[0027] The normalized state information is input into the UAV control model to obtain the next action command of the UAV; the UAV control model is constructed using the UAV control model construction method described above.

[0028] The next action command of the drone is transmitted to the motor to control the drone's movements.

[0029] Unlike existing technologies, the above-mentioned technical solution sets up a composite reward function that includes an attitude guidance reward term and a nonlinear position penalty term. The quaternion attitude guidance reward enables precise guidance without singularities for large-angle rotations on any axis, while the nonlinear position penalty provides necessary fault tolerance space at the moment of flipping while forming a strong constraint on position drift, thus achieving a dynamic balance between large maneuvers and stationary hovering. This effectively solves the position drift problem commonly found in existing deep reinforcement learning control and significantly improves the steady-state accuracy and flight stability of UAVs after performing large maneuvers.

[0030] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0031] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0032] In the accompanying drawings of the instruction manual:

[0033] Figure 1 This is a flowchart illustrating the UAV control model construction method described in a specific implementation.

[0034] Figure 2 This is a schematic diagram of the training process for the UAV control model as described in the specific implementation method;

[0035] Figure 3 This is a schematic diagram illustrating the calculation process of the composite reward function as described in the specific implementation method.

[0036] Figure 4 This is a schematic diagram illustrating the process of collecting interactive data as described in a specific implementation method. Detailed Implementation

[0037] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0038] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0039] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0040] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0041] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0042] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0043] As understood in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0044] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0045] The processor described in the embodiments of this application can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures that can implement the same or equivalent functions as the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0046] The computer program involved in the embodiments can be stored in a computer device readable storage medium, which includes, but is not limited to, disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc., and also includes other biological, physical, or chemical structures that can achieve the same or equivalent functions as the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium involved can be one of the above-mentioned media types, or a combination of the above-mentioned media types. In different embodiments, the computer program involved in the embodiments can be centrally stored in a single medium, or distributed and stored in multiple media. The memory containing the computer device readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device, or can be connected to the device involved in the embodiments as an external device or part of an external device. In some embodiments, the memory having a computer device readable storage medium is deployed locally; in other embodiments, the memory may be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as computer device access to the memory is enabled. Furthermore, the computer program involved in the embodiments may be stored in plaintext / ciphertext form, or it may be designed as training data, integrated and recombined through model training and implicitly stored in the parameter states of a deep neural network or other machine learning model.

[0047] A method for constructing a UAV control model addresses the rapid attitude changes and nonlinear dynamics inherent in UAV high-maneuvering maneuvers. By setting a composite reward function including an attitude guidance reward and a nonlinear position penalty, the quaternion attitude guidance reward enables precise, singularity-free guidance for large-angle rotations along any axis. The nonlinear position penalty provides necessary tolerance space during rollovers while strongly constraining position drift, achieving a dynamic balance between high-maneuvering maneuvers and stationary hovering. This effectively solves the common position drift problem in existing deep reinforcement learning control methods, significantly improving the steady-state accuracy and flight stability of UAVs after performing high-maneuvering maneuvers.

[0048] In this application, large-scale UAV maneuvers refer to UAVs completing significant attitude changes or trajectory changes in a short period of time. These mainly include roll-type maneuvers around the X-axis (such as 180-degree half-roll to achieve forward flight to inverted flight and hovering, 360-degree roll, 90-degree side flight attitude change, etc.), pitch-type maneuvers around the Y-axis (such as 180-degree loop, 90-degree vertical climb or dive, forward somersault and back somersault, etc.), yaw-type maneuvers around the Z-axis (such as 180-degree quick turn, 360-degree spin reconnaissance, etc.), as well as compound maneuvers (such as Immelmann roll, S-shaped maneuver, barrel roll) and multi-axis linkage maneuvers (such as sudden stop hovering, sudden 90-degree turn, inverted flight hovering, etc.).

[0049] The following combination Figure 1-3 This paper provides a specific implementation method for constructing a UAV control model, which includes the following steps:

[0050] S10. Construct a physics engine-based UAV simulation environment and define a state space; the state information in the state space includes the UAV's position, quaternion attitude, linear velocity, and angular velocity;

[0051] S20. Construct a policy network and a value network; wherein, the policy network is used to receive state information and output the next action command of the UAV; the value network is used to receive state information and perform state value estimation.

[0052] S30. Set a composite reward function, which includes an attitude guidance reward term and a nonlinear position penalty term; the attitude guidance reward term calculates the reward value based on the geodesic error between the current attitude and the target attitude using quaternion attitude calculation; the nonlinear position penalty term calculates the penalty value based on the square of the deviation between the current position of the UAV and the target position.

[0053] S40. Based on the current state information of the UAV, the system interacts with the UAV simulation environment through a policy network, collects interaction data and stores it in an experience buffer, and repeats the interaction process until the amount of data collected reaches a preset number of steps. The data in the experience buffer is used as the training set for the centralized update network. The interaction data includes the current state information, the next action command, the reward and the next state information. The reward is calculated by the composite reward function.

[0054] S50. Based on the interaction data, use the value network to estimate the state value corresponding to each state information, calculate the advantage function and reward objective; and iteratively update the parameters of the policy network and the value network based on the deep reinforcement learning algorithm until the preset convergence condition is reached, and output the trained policy network as the UAV control model.

[0055] In step S10, a physics engine-based UAV simulation environment is constructed to simulate the flight state and dynamic characteristics of the UAV in the real physical world. The physics engine includes, but is not limited to, any one of Gazebo, PyBullet, AirSim, or FlightGoggles, which can provide high-precision rigid body dynamics calculations, collision detection, and visualization interaction functions. Specifically, based on the structure of the target UAV, a rigid body model of the UAV (such as an X-shaped quadcopter, a cross-shaped quadcopter, a hexacopter, or a fixed wing, etc.) is established in the physics engine, and dynamic parameters are configured to ensure that the dynamic response in the simulation environment is consistent with that of a real UAV.

[0056] The state information in the observation space is a 13-dimensional vector used to describe the UAV's flight state, serving as input to the policy network and value network. It includes the UAV's position, quaternion attitude, linear velocity, and angular velocity. The UAV's position is its three-dimensional spatial coordinates in the inertial coordinate system. The quaternion attitude is represented using quaternion notation, avoiding the singularity problem of Euler angle notation in high-maneuver flight scenarios, ensuring that the state space is a continuous and smooth manifold structure under any attitude. The linear velocity is the UAV's three-dimensional linear velocity in the inertial coordinate system, and the angular velocity is the UAV's three-dimensional angular velocity in the body coordinate system. To improve the training efficiency of the policy network and value network, the state information is normalized before being input into them.

[0057] In step S20, the policy network can adopt a flat, single-network architecture multilayer perceptron (MLP) to achieve direct mapping from the UAV state to the thrust of the underlying actuators. Specifically, the input layer of the policy network contains 13 neurons, matching the 13-dimensional state information, used to receive normalized position, quaternion attitude, linear velocity, and angular velocity; its hidden layer includes a multilayer fully connected network and a nonlinear activation function, used to perform nonlinear mapping on the high-dimensional and strongly coupled flight state features; the output layer dimension of the policy network matches the control commands of the corresponding UAV actuators. Taking a quadcopter UAV as an example, the output layer contains 4 neurons, corresponding to the thrust percentage or motor speed commands of the four rotors, respectively. Through this flat, single-network architecture, the policy network can directly map the normalized state information into continuous underlying action commands, eliminating the multi-level calculation process of traditional cascade control (such as from the position loop to the attitude loop and then to motor allocation), effectively eliminating cascade control delay, and thus meeting the stringent requirements for extremely high dynamic response speed when the UAV performs rapid, high-maneuver maneuvers.

[0058] In step S20, the value network also employs a multilayer perceptron (MLP) architecture. Specifically, the input layer of the value network contains 13 neurons, sharing the same input layer structure as the policy network, and is used to receive normalized 13-dimensional state information (including position, quaternion pose, linear velocity, and angular velocity). Its hidden layer includes a multilayer fully connected network and a nonlinear activation function, used to extract high-order features of the state space. The output layer of the value network contains one neuron, used to output the scalar value evaluation value of the current state (i.e., the state value estimate). This evaluation value is subsequently invoked during centralized network updates to calculate the advantage function and reward objective, thereby accurately evaluating the merits of the actions performed by the policy network and guiding the gradient update direction of the policy network.

[0059] See Figure 3 As shown, in step S30, a composite reward function is constructed to guide the policy network to learn the correct actions, including orientation guidance reward. ) and nonlinear position penalty (Strict Position Constraint, ), i.e., composite reward function :

[0060] ;

[0061] The posture-guided reward The geodesic error between the current attitude and the target attitude (e.g., a 180-degree roll) is calculated using quaternions. This guides the drone to perform a flip. The formula is:

[0062]

[0063] in, To align with the positive reward coefficient (e.g., 20.0). This is the negative penalty coefficient for deviation (e.g., 1.0).

[0064] The nonlinear position penalty Instead of the traditional linear penalty, the quadratic penalty is used to strictly limit drone escape or drift. The formula is:

[0065]

[0066] in, Current position For the target location, This is the position penalty coefficient (e.g., 20.0).

[0067] Taking the 180-degree large-maneuver roll control of an unmanned aerial vehicle (UAV) as an example, the nonlinear position penalty function is as follows:

[0068] .

[0069] Specific calculation example:

[0070] Scenario A (Minor Drift): When the drone's positional deviation is extremely small, for example, only 0.05 meters away from the target point ( When ), the penalty value is:

[0071] Scenario B (Severe Drift): When the drone deviates from the target point by 1.0 meter, the penalty value is:

[0072] Advantages compared to linear penalties:

[0073] If a linear penalty is used (e.g.) ):

[0074] The penalty for scenario A is

[0075] The penalty for scenario B is

[0076] As can be seen from this embodiment, under the nonlinear (squared) penalty of this invention, the penalty for the tiny positional fluctuation (0.05m) generated by the UAV during the flip is extremely small (-0.05), which gives the policy network the necessary "fault tolerance space" and degree of freedom of action during extreme flips; however, once the UAV attempts to escape the target area (e.g., the error reaches 1m), the penalty value will increase dramatically to -20.0. This nonlinear design forces the agent to implicitly learn thrust vector compensation while flipping violently, ultimately achieving high-precision fixed-point inverted hovering (steady-state drift less than 0.03 meters).

[0077] In some embodiments, the composite reward function further includes a stability regularization penalty. Then the composite reward function :

[0078] ;

[0079] The stability regularization penalty term To ensure hardware deployability and eliminate high-frequency control jitter, penalties are imposed on one or more of the following: Euler angles, linear velocities, angular velocities, and rates of change of control variables, including those of the UAV's non-target axes. The formula is:

[0080] ;

[0081] in, , , , , represents the weight constants for each penalty term.

[0082] In some embodiments, the composite reward function further includes an angular velocity gating reward term (Anti-Spinning Mechanism). Then the composite reward function :

[0083] ;

[0084] The angular velocity gating reward item The configuration is as follows: a first reward value is provided when the attitude error, position error, and angular velocity amplitude of the UAV are all less than their respective preset thresholds; otherwise, the first reward value is not provided. That is, a reward value is provided if and only if the state set is satisfied. Time (i.e.) rad and and :

[0085] Each step grants an initial reward value for active braking: ;

[0086] If the set is not satisfied The condition is then And the counter was successfully cleared consecutively.

[0087] Of course, the angular velocity gating reward can also be configured as follows: when the UAV meets the condition that the attitude error, position error, and angular velocity amplitude are all less than their respective preset thresholds for multiple consecutive time steps, a second reward value is provided, and the second reward value is greater than the first reward value. That is:

[0088] If the above state is maintained continuously Step (such as) If a step is taken, a second reward value will be given (e.g., ...). ).

[0089] By employing "angular velocity gating" and "continuous success determination" mechanisms, the model is forced to learn to "brake" and hover, rather than rotating to pass the target angle, effectively avoiding motion oscillations. If a second reward value is obtained—that is, when the drone meets the conditions that attitude error, position error, and angular velocity amplitude are less than their respective preset thresholds for multiple consecutive time steps—the current round of interactive data acquisition can be terminated. This allows the model to complete the action and lock the attitude in the shortest time (shortest round length), demonstrating extremely high maneuverability, superior to strategies that simply pursue cumulative reward duration.

[0090] See Figure 4 As shown, step S40, collecting interactive data, may include the following steps:

[0091] Obtain the current status information of the drone and perform normalization processing;

[0092] The normalized state information is input into the policy network, the next action command of the UAV is output, the action command is mapped into motor control command and then sent to the UAV simulation environment;

[0093] The UAV simulation environment performs dynamic calculations based on the motor control commands and returns the next state information;

[0094] Based on the current state information, the next action instruction, and the next state information, the composite reward function is called to calculate the reward value;

[0095] The current state information, the action command, the reward value, and the next state information are stored as a set of interactive data in a buffer, and the above steps are repeated until the data collection of the preset time step is completed.

[0096] During the data collection process, at each time step, a deterministic action can be output based on the current state of the policy network. Noise interference is introduced during training, i.e., exploration noise is superimposed during the training phase to enhance the ability to explore the action space. The exploration noise is preferably zero-mean Gaussian noise, whose variance gradually decreases as the training progresses, achieving a smooth transition from exploration-based to utilization-based learning.

[0097] See Figure 2As shown, in step S50, in some embodiments, when iteratively updating the parameters of the policy network and the value network based on the deep reinforcement learning algorithm, a maximum number of training steps is set as one of the termination conditions of the training process. The maximum number of training steps is preset according to the task complexity, the computing resources of the simulation environment, and the expected training time. During the training process, the performance of the policy network is monitored in real time, including evaluation indicators such as cumulative reward value, success rate of large maneuvering actions, and steady-state hovering accuracy. If the preset convergence condition is not reached within the maximum number of training steps, it is determined that the current training round has failed to learn the optimal policy that meets the task requirements. At this time, the training reset mechanism is automatically triggered. Specifically, the configuration parameters of the UAV simulation environment are initialized, including resetting... The initial position, initial attitude, environmental physical parameters, and random seed of the UAV are initialized. At the same time, the network parameters of the deep reinforcement learning algorithm are initialized, including the weight parameters of the policy network, value network, policy target network, and value target network. The historical interaction data in the experience buffer is cleared, and a new round of training is started. Through the above reset and retraining mechanism, the problem of training failure caused by improper selection of initial parameters, insufficient exploration, or getting stuck in local optima during training can be effectively avoided. The algorithm is given the ability to try to find the global optimal policy multiple times. Successful experience is gradually accumulated in multiple training rounds. Finally, the UAV control model that can stably converge to the preset convergence condition within the maximum number of training steps is output, which significantly improves the robustness and success rate of the training process.

[0098] The deep reinforcement learning algorithm in this application can be the soft actor-critic (SAC) algorithm, the dual-delay deep deterministic policy gradient (TD3) algorithm, the proximal policy optimization (PPO) algorithm, etc., with the proximal policy optimization algorithm being preferred. The proximal policy optimization algorithm limits the magnitude of each policy update by introducing a pruning mechanism or adaptive KL divergence penalty, ensuring that the difference between the old and new policies remains within a controllable range. In the highly dynamic and strongly nonlinear task scenario of UAV high maneuver control, the policy network needs to learn two objectives simultaneously: high maneuvering actions and precise hovering. The action space changes drastically. The conservative update strategy of the proximal policy optimization algorithm can prevent a single update from deviating excessively from the original policy, ensuring the smooth progress of the training process.

[0099] This application also provides a drone control method, which is used to deploy a trained drone control model to an actual flight platform to achieve end-to-end real-time control from state perception to action execution; it includes the following steps:

[0100] Obtain the current status information of the drone;

[0101] Normalize the current state information;

[0102] The normalized state information is input into the UAV control model to obtain the next action command of the UAV; the UAV control model is constructed using the UAV control model construction method described above.

[0103] The next action command of the drone is transmitted to the motor to control the drone's movements.

[0104] The aforementioned status information includes the UAV's position, quaternion attitude, linear velocity, and angular velocity. The UAV's position is obtained through a Global Positioning System (GPS) or visual odometry, the quaternion attitude is calculated by an inertial measurement unit (IMU), the linear velocity information is obtained through accelerometer integration or differential GPS, and the angular velocity information is directly read from a gyroscope.

[0105] The above-mentioned UAV control method achieves a smooth transition between the training and deployment phases, enabling the UAV to execute large maneuvers quickly and accurately and enter a high-precision steady-state hover during actual flight, effectively improving the control performance and flight stability of the UAV in high-maneuver scenarios.

[0106] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A method for constructing a UAV control model, characterized in that, Includes the following steps: A physics engine-based UAV simulation environment is constructed, and a state space is defined; the state information in the state space includes the UAV's position, quaternion attitude, linear velocity, and angular velocity. Construct a policy network and a value network; wherein the policy network is used to receive state information and output the next action command of the UAV; the value network is used to receive state information and perform state value estimation. A composite reward function is set up, which includes an attitude guidance reward term and a nonlinear position penalty term. The attitude guidance reward term calculates the reward value based on the geodesic error between the current attitude and the target attitude using quaternion attitude calculation. The nonlinear position penalty term calculates the penalty value based on the square of the deviation between the current position of the UAV and the target position. Based on the current state information of the UAV, the system interacts with the UAV simulation environment through a policy network, collects interaction data and stores it in an experience buffer, and repeats the interaction process until the amount of data collected reaches a preset number of steps. The data in the experience buffer is used as the training set for the centralized update network. The interaction data includes the current state information, the next action command, the reward and the next state information. The reward is calculated by the composite reward function. Based on the interaction data, the value network is used to estimate the state value corresponding to each state information, calculate the advantage function and reward objective, and iteratively update the parameters of the policy network and the value network based on the deep reinforcement learning algorithm until the preset convergence condition is reached, and output the trained policy network as the UAV control model.

2. The method for constructing a UAV control model according to claim 1, characterized in that, The composite reward function also includes a stability regularization penalty, which includes penalties for one or more of the following: Euler angles, linear velocity, angular velocity, and rate of change of control variables of the UAV's non-target axes.

3. The method for constructing a UAV control model according to claim 2, characterized in that, The composite reward function further includes an angular velocity gating reward term, which is configured to provide a first reward value when the attitude error, position error, and angular velocity amplitude of the UAV are all less than their respective preset thresholds; otherwise, the first reward value is not provided.

4. The method for constructing a UAV control model according to claim 3, characterized in that, The angular velocity gating reward item is further configured to provide a second reward value when the UAV meets the conditions that the attitude error, position error and angular velocity amplitude are less than their respective preset thresholds for multiple consecutive time steps. The second reward value is greater than the first reward value.

5. The method for constructing a UAV control model according to claim 4, characterized in that, When the UAV meets the condition that the attitude error, position error and angular velocity amplitude are less than their respective preset thresholds for multiple consecutive time steps, the interactive data acquisition of the current round is also terminated.

6. The method for constructing a UAV control model according to claim 1, characterized in that, Collecting interactive data includes the following steps: Obtain the current status information of the drone and perform normalization processing; The normalized state information is input into the policy network, the next action command of the UAV is output, the action command is mapped into motor control command and then sent to the UAV simulation environment; The UAV simulation environment performs dynamic calculations based on the motor control commands and returns the next state information; Based on the current state information, the next action instruction, and the next state information, the composite reward function is called to calculate the reward value; The current state information, the action command, the reward value, and the next state information are stored as a set of interactive data in a buffer, and the above steps are repeated until the data collection of the preset time step is completed.

7. The method for constructing a UAV control model according to claim 1, characterized in that, During the process of collecting interactive data, the action commands output by the policy network are superimposed with exploration noise.

8. The method for constructing a UAV control model according to claim 1, characterized in that, When iteratively updating the parameters of the policy network and the value network based on the deep reinforcement learning algorithm, a maximum number of training steps is set; if the preset convergence condition is not reached within the maximum number of training steps, the parameters of the UAV simulation environment and the deep reinforcement learning algorithm are initialized, and a new round of training begins.

9. The method for constructing a UAV control model according to claim 1, characterized in that, The deep reinforcement learning algorithm is a near-end policy optimization algorithm.

10. A method for controlling an unmanned aerial vehicle (UAV), characterized in that, Includes the following steps: Obtain the current status information of the drone; Normalize the current state information; The normalized state information is input into the UAV control model to obtain the next action command of the UAV; the UAV control model is constructed using the UAV control model construction method according to any one of claims 1 to 6; The next action command of the drone is transmitted to the motor to control the drone's movements.

Images