Flapping-wing aircraft hovering method and device based on deep reinforcement learning, terminal and storage medium
By constructing action and state space models using deep reinforcement learning algorithms and training policy networks, the problem of low hovering control accuracy of tailless micro flapping-wing aircraft was solved, achieving high-precision flight control and multi-aircraft formation flight.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-03-10
AI Technical Summary
Existing tailless micro flapping-wing aircraft driven by dual motors have low control precision when hovering, and the power system output is asymmetrical and the air disturbance is strong, which makes it difficult to model the aerodynamic characteristics, weakens the natural stability of the aircraft, and requires high computing speed and robustness.
A deep reinforcement learning algorithm is used to construct action space and state space models. A state space model is constructed by sensor constraints, and the initial state is randomized. A reward function and boundary conditions are set. A policy network is trained using a deep deterministic policy gradient algorithm to control the simulated aircraft to hover.
It improves flight control precision, enabling hovering, path following, and formation flying of micro flapping-wing aircraft, and has good robustness and generalization ability.
Smart Images
Figure CN121635378A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of aircraft control and machine learning technology, and in particular to a hovering method, device, terminal and storage medium for flapping-wing aircraft based on deep reinforcement learning. Background Technology
[0002] Micro flapping-wing aircraft are a type of biomimetic aircraft that achieve flight by mimicking the flapping of wings of birds or insects, exhibiting high maneuverability and flexibility. Compared to traditional rotorcraft, micro flapping-wing aircraft demonstrate better resistance to interference in complex environments and reduce noise; and compared to fixed-wing aircraft, they can hover and take off and land vertically without relying on high-speed airflow, making them suitable for indoor and confined space missions. However, due to the complex structure and highly nonlinear aerodynamic characteristics of flapping-wing aircraft, their flight control design faces greater challenges.
[0003] Existing tailless micro flapping-wing aircraft driven by dual motors eliminate the traditional tail, achieving flight attitude adjustment through independent control of the left and right wings. However, this design faces significant challenges during hovering. First, the asymmetric and time-varying output of the flapping-wing aircraft's power system reduces its control precision. Second, the high-speed flapping of the wings causes strong air disturbances, making it difficult to accurately model aerodynamic characteristics. Finally, the tailless design weakens the aircraft's natural stability, requiring real-time compensation for attitude deviations through control algorithms, which places higher demands on the controller's computational speed and robustness.
[0004] Therefore, the existing technology still suffers from low control precision, and the technology needs further improvement. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a hovering method, device, terminal and storage medium for flapping-wing aircraft based on deep reinforcement learning, in order to solve the problem of low control accuracy of existing flapping-wing aircraft.
[0006] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention provides a hovering method for flapping-wing aircraft based on deep reinforcement learning, comprising: A motion space model is constructed based on the instantaneous wing kinematic modulation theory; A state-space model is constructed based on sensor constraints, and the initial state is randomized; Set the reward function and boundary conditions according to the characteristics of the aircraft and the hovering target; The policy network is trained using the deep deterministic policy gradient algorithm based on the action space model, the state space model, the reward function, and the boundary conditions. The trained policy network is deployed on a flight simulator, which is then used to control the flight simulator to hover.
[0007] In one implementation, constructing the motion space model based on instantaneous wing kinematic modulation theory includes: Based on the instantaneous wing kinematics modulation theory, a motion space model is constructed for real-time adjustment of the lift, roll moment, pitch moment, and yaw moment of the simulated aircraft.
[0008] In one implementation, the step of constructing a state-space model based on sensor constraints and randomizing the initial state includes: Historical observation data is imported through a sliding window to obtain the linear velocity, angular velocity, real-time position error, and attitude error of the simulated aircraft. Based on the linear velocity, the angular velocity, the real-time position error, and the attitude error, the state space model is constructed, and the initial state of the state space model is randomized.
[0009] In one implementation, setting the reward function and boundary conditions based on the aircraft characteristics and the hovering target includes: The position error penalty, attitude error penalty, velocity penalty, and angular velocity penalty of the simulated aircraft are determined based on the characteristics of the aircraft and the hovering target. The reward function and the boundary conditions are set based on the position error penalty, the attitude error, the velocity penalty, and the angular velocity penalty.
[0010] In one implementation, training the policy network using a deep deterministic policy gradient algorithm based on the action space model, the state space model, the reward function, and the boundary conditions includes: The system controls multiple simulated aircraft to fly in parallel based on the action space model and the state space model, and collects flight data from multiple simulated aircraft. Training sequences are generated based on the flight data, and the policy network is optimized using a deep deterministic policy gradient algorithm according to the reward function and the boundary conditions, so that the policy network outputs deterministic actions to achieve hovering of the aircraft.
[0011] In one implementation, generating training sequences based on the flight data and optimizing the policy network using a deep deterministic policy gradient algorithm according to the reward function and the boundary conditions includes: In each simulation environment, an initial state is constructed based on initial observation data. ; Initialize the policy network according to the initial state. A Gaussian distribution of the generated actions is obtained, and the actions are sampled from the Gaussian distribution. After the actions are applied to the simulated aircraft in the simulation environment, the reward is calculated according to the reward function. State is constructed based on forward dynamics calculation results. The policy network predicts and generates corresponding actions, and generates state-action-reward sequences until the data collected by all parallel environments reaches the size of the rolling buffer. When the amount of data in the rolling buffer reaches a preset value, the value network and the policy network are updated using the state-action-reward sequence.
[0012] In one implementation, deploying the trained policy network on a simulated aircraft and controlling the simulated aircraft to hover includes: The effective torque output by the motor is applied to the flapping shaft to drive the wing to move, and the aerodynamic force calculated by the blade method is applied to the wing to simulate the motion of the simulated aircraft in the air. During the simulation, the torso of the simulated aircraft is fixed, the force acting on the fixture is measured, and the force-torque acting on the simulated aircraft is inferred. Obtain the relationship between the measured voltage input and the force-torque output with the control input set to zero; The voltage input is adjusted according to the relationship to apply average force and torque to the simulated aircraft in order to control the simulated aircraft to hover.
[0013] Secondly, the present invention provides a hovering device for flapping-wing aircraft based on deep reinforcement learning, comprising: The motion space module is used to construct a motion space model based on the instantaneous wing kinematic modulation theory. The state-space module is used to construct a state-space model based on sensor constraints and to randomize the initial state. The reward setting module is used to set the reward function and boundary conditions based on the characteristics of the aircraft and the hovering target. The policy network training module is used to train the policy network using the deep deterministic policy gradient algorithm based on the action space model, the state space model, the reward function, and the boundary conditions. The hovering control module is used to deploy the trained policy network on the simulated aircraft and control the simulated aircraft to hover.
[0014] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a deep reinforcement learning-based flapping-wing aircraft hovering program, and the deep reinforcement learning-based flapping-wing aircraft hovering program, when executed by the processor, is used to implement the operation of the deep reinforcement learning-based flapping-wing aircraft hovering method as described in the first aspect.
[0015] Fourthly, the present invention also provides a computer-readable storage medium storing a deep reinforcement learning-based flapping-wing aircraft hovering program, which, when executed by a processor, is used to implement the operation of the deep reinforcement learning-based flapping-wing aircraft hovering method as described in the first aspect.
[0016] The present invention, by employing the above technical solution, has the following effects: 1) This invention addresses the hovering flight problem of a tailless micro flapping-wing aircraft driven by dual motors, employing a deep reinforcement learning algorithm to train a control strategy. This strategy can serve as a low-level attitude controller, directly mapping the flapping-wing aircraft's state to motor voltage outputs, thus improving flight control accuracy.
[0017] 2) This invention constructs a reasonable action space and state space, designs a reward function and boundary conditions, and trains the system through multiple parallel simulation environments. The resulting strategy has good effect on the hovering flight attitude control of micro flapping-wing aircraft.
[0018] 3) The underlying pose controller trained by this invention can be used for aircraft path following and multi-aircraft formation flight, and has good robustness and generalization ability. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0020] Figure 1 This is a flowchart of the hovering method for flapping-wing aircraft based on deep reinforcement learning in this invention.
[0021] Figure 2 This is a basic framework diagram of the deep reinforcement learning training aircraft principle in this invention.
[0022] Figure 3 This is a schematic diagram of gradient training for a deep deterministic strategy in this invention.
[0023] Figure 4This is a functional schematic diagram of the terminal in one implementation of the present invention.
[0024] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0026] Exemplary methods Micro flapping-wing aircraft are a type of biomimetic aircraft that achieve flight by mimicking the flapping of wings of birds or insects, exhibiting high maneuverability and flexibility. Compared to traditional rotorcraft, micro flapping-wing aircraft demonstrate better resistance to interference in complex environments and reduce noise; and compared to fixed-wing aircraft, they can hover and take off and land vertically without relying on high-speed airflow, making them suitable for indoor and confined space missions. However, due to the complex structure and highly nonlinear aerodynamic characteristics of flapping-wing aircraft, their flight control design faces greater challenges.
[0027] Existing tailless micro flapping-wing aircraft driven by dual motors eliminate the traditional tail, achieving flight attitude adjustment through independent control of the left and right wings. However, this design faces significant challenges during hovering. First, the asymmetric and time-varying output of the flapping-wing aircraft's power system reduces its control precision. Second, the high-speed flapping of the wings causes strong air disturbances, making it difficult to accurately model aerodynamic characteristics. Finally, the tailless design weakens the aircraft's natural stability, requiring real-time compensation for attitude deviations through control algorithms, which places higher demands on the controller's computational speed and robustness.
[0028] To address the above technical problems, this invention provides a hovering method for flapping-wing aircraft based on deep reinforcement learning. The method includes: constructing an action space model based on instantaneous wing kinematic modulation theory; constructing a state space model based on sensor constraints and randomizing the initial state; setting a reward function and boundary conditions based on aircraft characteristics and the hovering target; training a policy network using a deep deterministic policy gradient algorithm based on the action space model, the state space model, the reward function, and the boundary conditions; and deploying the trained policy network on a simulated aircraft to control the simulated aircraft to hover. This invention effectively enables hovering, path following, and formation flying of a dual-motor driven tailless micro flapping-wing aircraft, improving flight control accuracy and exhibiting good robustness and generalization ability.
[0029] This embodiment employs deep reinforcement learning to train the aircraft's policy network. Compared to traditional flight control methods based on dynamic models, deep reinforcement learning, through trial and error and feedback mechanisms, can directly learn the optimal policy from the interaction between the aircraft and its environment, without requiring precise mathematical modeling. This method is particularly suitable for complex, highly nonlinear, and multidimensional dynamic systems, such as hovering control of flapping-wing aircraft. Reinforcement learning can adapt to environmental changes and adjust the control policy in real time, thereby achieving more robust flight performance. Furthermore, reinforcement learning has significant advantages in solving high-dimensional state-space problems; for example, it can simultaneously optimize attitude stability and energy efficiency, providing an innovative solution for hovering control of a dual-motor driven tailless micro flapping-wing aircraft.
[0030] like Figure 1 As shown, this embodiment of the invention provides a hovering method for flapping-wing aircraft based on deep reinforcement learning, including the following steps: Step S100: Construct an action space model based on the instantaneous wing kinematic modulation theory.
[0031] In this embodiment, a tailless micro flapping-wing aircraft driven by two motors is provided (in this embodiment, it is used as a simulated aircraft). This aircraft drives the wings with two brushless DC motors on the left and right sides respectively, so as to achieve precise adjustment of lift and three-axis control torque, enabling it to have multiple flight capabilities, including stable hovering, path following and multi-aircraft formation flight capabilities.
[0032] To achieve precise adjustment of lift and three-axis control torque of the tailless micro flapping-wing aircraft, this embodiment designs an attitude controller based on the aircraft's dynamic model to control the aircraft's three-axis stability; and constructs a physical model in a simulation environment to verify the controllability of force and torque generation by simulating the relationship between motor input and output torque.
[0033] Specifically, in one implementation of this embodiment, step S100 includes the following steps: Step S101: Based on the instantaneous wing kinematics modulation theory, construct an action space model for real-time adjustment of the lift, roll moment, pitch moment, and yaw moment of the simulated aircraft.
[0034] In this embodiment, the motion space model can achieve wing kinematic modulation based on the real-time adjustment of the left and right motor voltages, thereby controlling lift, roll torque, pitch torque, and yaw torque. Specifically, the motion space model can be selected based on voltage-related variables, where the instantaneous voltages of the left and right motors are directly defined as u_0 and u_1. However, experiments show that directly using this motion space for training makes it difficult to achieve periodic wing flapping. Therefore, it is necessary to introduce instantaneous wing kinematic modulation theory.
[0035] Furthermore, the method constructs a motion space model based on the instantaneous wing kinematic modulation theory, which includes the following parameters: ΔU_a is the amplitude change of the motor sinusoidal voltage signal, which can adjust the overall lift; U_r is the amplitude difference between the two motor voltage signals, which is used to adjust the roll torque; Up is the motor voltage bias, which is used to provide pitch torque; Δσ is the split wing flapping parameter, which controls the speed difference between the wing's up and down flapping by changing its value, thereby adjusting the yaw torque.
[0036] like Figure 1 As shown, this embodiment of the invention provides a hovering method for flapping-wing aircraft based on deep reinforcement learning, including the following steps: Step S200: Construct a state-space model based on sensor constraints and randomize the initial state.
[0037] In this embodiment, after constructing the action space model based on the instantaneous wing kinematic modulation theory, it is also necessary to construct the state space model based on sensor constraints. The state space model includes the aircraft's real-time position error, attitude error, and linear velocity and angular velocity. During the aircraft simulation, this embodiment uses a sliding window to introduce historical observation data (including sensor constraint parameters) to construct the state space model, thereby reducing the impact of system delay on control accuracy.
[0038] Specifically, in one implementation of this embodiment, step S200 includes the following steps: Step S201: Import historical observation data through a sliding window to obtain the linear velocity, angular velocity, real-time position error, and attitude error of the simulated aircraft; Step S202: Based on the linear velocity, the angular velocity, the real-time position error, and the attitude error, construct the state space model and randomize the initial state of the state space model.
[0039] In this embodiment, the state-space model Can be adopted The system was constructed using a 40-step historical observation and action sliding window to address system latency and asynchronous control issues. Indicates the current position With the target location Errors in the Cartesian coordinate system. Furthermore, This represents the current rotation matrix. This represents the linear velocity of the drone. This indicates the angular velocity of the drone.
[0040] Furthermore, to enhance the robustness of the control strategy and facilitate its transfer to a real micro flapping-wing aircraft, this embodiment employs domain randomization technology to randomly initialize the state-space model. Specifically, the aircraft is initially placed along the Z-axis, with its height randomly varying between 400 mm and 600 mm, while the hovering target position remains constant at (0 mm, 0 mm, 500 mm). Euler angles Initially, all angles are set to 0 degrees, and each angle is randomly varied within a range of ±5 degrees.
[0041] Furthermore, this embodiment analyzes the mass characteristics of different airfoils and generates a multivariate regression function. During training, each wingspan is randomly varied by ±2 mm from a baseline of 90 mm, and the URDF file (Uniform Robot Description Format) is updated accordingly. The airfoil's mass and moment of inertia are automatically generated by the regression function.
[0042] like Figure 1 As shown, this embodiment of the invention provides a hovering method for flapping-wing aircraft based on deep reinforcement learning, including the following steps: Step S300: Set the reward function and boundary conditions based on the aircraft characteristics and hovering target.
[0043] In this embodiment, in order to achieve high-efficiency training, a reward function and boundary conditions are set according to the characteristics of the aircraft and the hovering target. The reward function comprehensively considers the following factors: penalty for aircraft position error, as well as penalty for attitude error, velocity, and angular velocity. In this embodiment, the accuracy of aircraft hovering control is improved by gradually reducing the boundary conditions.
[0044] Specifically, in one implementation of this embodiment, step S300 includes the following steps: Step S301: Determine the position error penalty, attitude error penalty, velocity penalty, and angular velocity penalty of the simulated aircraft based on the aircraft characteristics and the hovering target; Step S302: Set the reward function and the boundary conditions based on the position error penalty, the attitude error, the velocity penalty, and the angular velocity penalty.
[0045] In this embodiment, the final control strategy is obtained through two training sessions, each with different boundary conditions and reward function settings. In the first training session, the boundary conditions are set as a cuboid with the aircraft's center of mass as its geometric center, the side length determined by the wingspan and the accuracy of the hovering target. The second training session narrows the boundary. Within each training segment, whenever the micro-flapping-wing aircraft flies out of this boundary, the environment is reset, and a new training segment begins. Furthermore, the time boundary is set to... The reward function structure is defined as follows: .
[0046] In one feasible implementation of this embodiment, p The initial value is set to 25, increasing by 5 when the UAV approaches within 25 mm of the hovering target point. The remaining values represent penalties for position error, attitude, linear velocity, and angular velocity, respectively, with coefficients set as follows: p All physical quantities can be calculated from the state space data during the training process.
[0047] like Figure 1 As shown, this embodiment of the invention provides a hovering method for flapping-wing aircraft based on deep reinforcement learning, including the following steps: Step S400: Train the policy network using the deep deterministic policy gradient algorithm based on the action space model, the state space model, the reward function, and the boundary conditions.
[0048] In this embodiment, parallel simulation and deep reinforcement learning are used to accelerate policy training. By constructing a reasonable state and action space, designing a reward function, and evaluating the effects of various algorithms, a policy network that can be used for underlying pose control is obtained.
[0049] Specifically, this embodiment employs parallel simulation technology, which uses multiple threads to synchronously collect flight data in multiple simulation environments; it utilizes a deep deterministic policy gradient algorithm to optimize the policy network, enabling the network to output deterministic actions to achieve aircraft hovering; and it performs domain randomization processing on the aircraft state during training to improve the generalization ability of the policy.
[0050] like Figure 2 As shown, Figure 2 This is a basic framework diagram of the deep reinforcement learning training aircraft principle in this embodiment. The following is a brief introduction to the training principle framework of this embodiment.
[0051] In this reinforcement learning framework, the agent learns by continuously interacting with the environment: First, observe the current state. According to its strategy Select Action Receive rewards from the environment ; Then, the agent transitions to the next state. From these interactions, the agent learns about the environment and updates its policy. This interaction can be represented as a Markov decision process, which includes a state transition model. State space Action space ,award and discount factor .
[0052] In this embodiment, , It's difficult to know for sure that the agent learns using Monte Carlo sampling, which is model-free learning. Define the reward. The sum of rewards weighted over time, where the weights are... It is a discount factor between 0 and 1, designed to balance the relative weight of current rewards and future rewards. The closer the value is to 1, the more emphasis is placed on long-term rewards; however, setting it too close to 1 may negatively impact the hovering performance of aircraft. The actor-judge algorithm, most commonly used in deep reinforcement learning, approximates the action-value function using a value neural network and trains a policy network using policy gradients to approximate the policy. .
[0053] In the deep deterministic policy gradient algorithm used in this embodiment, the policy network outputs deterministic actions without randomness, unlike the actor-judge policy network which outputs random probability density. Furthermore, both the policy network and the value network consist of two sub-networks, which reduces bootstrapping, i.e., avoids updating the network's own parameters with estimates made by the network itself.
[0054] Furthermore, to address the asynchronous control issues arising from applying reinforcement learning to robotics, this embodiment incorporates actions taken prior to the policy as part of the state definition. One of the key assumptions of the Markov formula is synchronous execution, meaning that the observed state remains unchanged before an action is executed. Go, for example, satisfies this assumption. However, in robotic systems such as aircraft, execution is asynchronous. As the state is measured, transmitted, and actions are calculated and executed, the robot's state constantly changes, and the system's next state does not directly depend on the measured state but rather on the state delayed after measurement. This violates the aforementioned assumptions to some extent. Therefore, this embodiment incorporates actions taken prior to the policy as part of the state definition to mitigate the impact of asynchronous control.
[0055] Specifically, in one implementation of this embodiment, step S400 includes the following steps: Step S401: Control multiple simulated aircraft to fly in parallel according to the action space model and the state space model, and collect flight data of multiple simulated aircraft; Step S402: Generate a training sequence based on the flight data, and optimize the policy network using a deep deterministic policy gradient algorithm according to the reward function and the boundary conditions, so that the policy network outputs deterministic actions to achieve hovering of the aircraft.
[0056] In one implementation of this embodiment, step S402 includes the following steps: Step S4021: In each simulation environment, construct the initial state based on the initial observation data. ; Step S4022: Initialize the policy network according to the initial state. A Gaussian distribution of the generated actions is obtained, and the actions are sampled from the Gaussian distribution. After the actions are applied to the simulated aircraft in the simulation environment, the reward is calculated according to the reward function. Step S4023: Construct the state based on the forward dynamics calculation results The policy network predicts and generates corresponding actions, and generates state-action-reward sequences until the data collected by all parallel environments reaches the size of the rolling buffer. Step S4024: When the amount of data in the rolling buffer reaches a preset value, the value network and the policy network are updated using the state-action-reward sequence.
[0057] In this embodiment, a multi-stage training strategy is used to train the policy network, including: an initial stage using small batches of data and a low discount factor for rapid policy optimization; and subsequent stages gradually increasing the batch size and discount factor to enhance the long-term planning capability of the policy.
[0058] As an example, this embodiment establishes an independent simulation environment on each CPU core used for parallel training, and improves data collection efficiency by using 20 parallel simulation environments during training. By executing simulations simultaneously in multiple environments, the speed and diversity of data acquisition are significantly improved, accelerating the training process. The specific parallel training method is as follows: S1, the sequence required for training The generation process includes the following steps: In each simulation environment, state s1 is constructed from initial observations, and historical observations and actions are set to zero to ensure that all simulations start from uniform initial conditions.
[0059] S2, the initial policy network is based on A Gaussian distribution of actions is generated, and actions are sampled from it. This Gaussian distribution provides the probability distribution of possible actions for the policy network, and more state space is explored by introducing randomness.
[0060] S3 applies actions to a micro flapping-wing aircraft in a simulated environment and calculates rewards based on a reward function. The reward function evaluates the aircraft's performance based on its current state and actions, providing feedback for policy network improvement.
[0061] S4, constructing the state based on the forward dynamics calculation results. The policy network predicts and generates actions, and the above process is repeated to generate state-action-reward sequences until the data collected by all parallel environments reaches the size of the rolling buffer.
[0062] When the amount of data in the rolling buffer reaches a preset value, this data can be used to update the value network and the policy network.
[0063] Through the training process described above, the policy network successfully enables a micro flapping-wing aircraft to hover within a 50mm diameter range around its initial center of mass. Despite the relatively close initial and target positions, the trained hovering policy demonstrates the ability to generalize to more distant targets and achieves hovering at any set point in Cartesian space. Furthermore, the policy supports vertical, forward, and lateral flight, allowing the aircraft to fly from its initial position to a distant target point, thus forming an efficient attitude controller.
[0064] Furthermore, to demonstrate the versatility of the strategy, this embodiment provides trajectory tracking of a micro flapping-wing aircraft along a circular path and controlled flight test results for multiple aircraft. These results show that the trained strategy can not only complete simple hovering tasks but also handle more complex flight tasks, demonstrating its robustness and practicality in various scenarios.
[0065] like Figure 3 As shown, in this embodiment, training is performed based on the depth-based certainty policy gradient algorithm, as detailed below: In the deep deterministic policy gradient algorithm used in this embodiment, its policy network outputs deterministic actions without randomness, unlike the actor-judge policy network which outputs random probability density. Furthermore, both the policy network and the value network are composed of two sub-networks, which reduces bootstrapping, i.e., avoids updating the network's own parameters with estimates made by the network itself.
[0066] In one feasible implementation, the batch size for updating the neural network is set to 256 during the first training iteration. The discount factor γ is set to 0.99, meaning that the algorithm focuses more on recent rewards while also considering future rewards when calculating the payoff. In the second training iteration, the batch size is increased to 512 to utilize more data for each update, thereby improving training stability and efficiency. Simultaneously, the discount factor γ is increased to 0.995 to more fully consider the expected future rewards, allowing the policy network to better plan long-term actions. In both training iterations, five independent segments are evaluated every 20,000 steps. These evaluation segments are used to monitor and record the performance of the policy network, providing immediate feedback during training. This evaluation method allows for detailed observation of the policy's improvement at different training stages and its adaptability to hovering tasks.
[0067] like Figure 1 As shown, this embodiment of the invention provides a hovering method for flapping-wing aircraft based on deep reinforcement learning, including the following steps: Step S500: Deploy the trained policy network on the simulated aircraft and control the simulated aircraft to hover.
[0068] In this embodiment, the triaxial forces and torques during the flapping of the aircraft's wings are verified using a simulation environment. The left and right wings are driven by brushless DC motors, and torque control is achieved by adjusting the wing kinematics. In this embodiment, aerodynamic forces are calculated and the relationship between forces and torques is measured in the simulation based on the Bullet physics engine to optimize the control input.
[0069] Specifically, in one implementation of this embodiment, step S500 includes the following steps: Step S501: Apply the effective torque output by the motor to the flapping shaft to drive the wing to move, and apply the aerodynamic force calculated by the blade method to the wing to simulate the motion of the simulated aircraft in the air. Step S502: During the simulation process, fix the body part of the simulated aircraft, measure the force acting on the fixture, and infer the force-torque acting on the simulated aircraft. Step S503: Obtain the relationship between the measured voltage input and the force-torque output when the control input is set to zero; Step S504: Adjust the voltage input according to the relationship to apply average force and torque to the simulated aircraft to control the simulated aircraft to hover.
[0070] In this embodiment, in the dual-motor driven micro flapping-wing aircraft, two independent brushless DC motors are used to drive the left and right wings respectively. The wing kinematics are controlled by changing the motor voltage signal, thereby changing the wing kinematics to generate a three-axis control torque.
[0071] Furthermore, in a simulation environment based on the Bullet physics engine: the effective torque output by the motor... Acting on the flapping axis, it drives the wing to move; aerodynamic forces calculated by the blade method are applied to the flapping wing to simulate the motion of a micro flapping-wing aircraft in the air.
[0072] Furthermore, this embodiment can fix the aircraft's torso in the simulation, measure the force acting on the clamp, and infer the force and torque acting on the aircraft; the voltages of the left and right motors are set as follows: , The control input is set to zero at this time to verify the relationship between the measured voltage input and the force and torque output when the control input is zero. In addition, starting from the aircraft's stationary state, the control input is changed, and the average force and torque applied to the aircraft over a period of time are measured.
[0073] Through the above technical solutions, the embodiments of the present invention can efficiently verify the three-axis control torque and related parameters of flapping-wing aircraft, and meet actual control requirements.
[0074] As an example, in this embodiment, the policy network is deployed on a simulated aircraft to achieve hovering of a target within a 50 mm diameter range around the initial center of mass. Although the initial and target positions are relatively close during training, the final hovering policy demonstrates the ability to generalize to more distant targets. This policy enables the micro flapping-wing aircraft to hover at any set point in Cartesian space and to perform vertical, forward, and lateral flight. This allows the aircraft to fly from its initial position to a distant target point under the control of the policy. This is an effective pose controller. Experimental results show that the obtained policy can effectively achieve hovering, path following, and formation flying, exhibiting good robustness and generalization ability.
[0075] This embodiment achieves the following technical effects through the above technical solution: 1) This embodiment addresses the hovering flight problem of a tailless micro flapping-wing aircraft driven by dual motors, employing a deep reinforcement learning algorithm to train a control strategy. This strategy can serve as a low-level attitude controller, directly mapping the flapping-wing aircraft's state to motor voltage outputs, thus improving flight control accuracy.
[0076] 2) This embodiment constructs a reasonable action space and state space, designs reward functions and boundary conditions, and trains through multiple parallel simulation environments. The resulting strategy has good effect on hovering flight attitude control of micro flapping-wing aircraft.
[0077] 3) The underlying pose controller trained in this embodiment can be used for aircraft path following and multi-aircraft formation flight, and has good robustness and generalization ability.
[0078] Exemplary device Based on the above embodiments, the present invention also provides a hovering device for flapping-wing aircraft based on deep reinforcement learning, comprising: The motion space module is used to construct a motion space model based on the instantaneous wing kinematic modulation theory. The state-space module is used to construct a state-space model based on sensor constraints and to randomize the initial state. The reward setting module is used to set the reward function and boundary conditions based on the characteristics of the aircraft and the hovering target. The policy network training module is used to train the policy network using the deep deterministic policy gradient algorithm based on the action space model, the state space model, the reward function, and the boundary conditions. The hovering control module is used to deploy the trained policy network on the simulated aircraft and control the simulated aircraft to hover.
[0079] This embodiment achieves the following technical effects through the above technical solution: 1) This embodiment addresses the hovering flight problem of a tailless micro flapping-wing aircraft driven by dual motors, employing a deep reinforcement learning algorithm to train a control strategy. This strategy can serve as a low-level attitude controller, directly mapping the flapping-wing aircraft's state to motor voltage outputs, thus improving flight control accuracy.
[0080] 2) This embodiment constructs a reasonable action space and state space, designs reward functions and boundary conditions, and trains through multiple parallel simulation environments. The resulting strategy has good effect on hovering flight attitude control of micro flapping-wing aircraft.
[0081] 3) The underlying pose controller trained in this embodiment can be used for aircraft path following and multi-aircraft formation flight, and has good robustness and generalization ability.
[0082] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 4 As shown.
[0083] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.
[0084] When executed by the processor, this computer program is used to implement a hovering method for flapping-wing aircraft based on deep reinforcement learning.
[0085] It will be understood by those skilled in the art that Figure 4 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0086] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a deep reinforcement learning-based flapping-wing hovering program, which, when executed by the processor, is used to implement the above-described deep reinforcement learning-based flapping-wing hovering method.
[0087] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a deep reinforcement learning-based flapping-wing aircraft hovering program, which, when executed by a processor, is used to implement the above-described deep reinforcement learning-based flapping-wing aircraft hovering method.
[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.
[0089] In summary, this invention provides a hovering method, device, terminal, and storage medium for flapping-wing aircraft based on deep reinforcement learning. The method includes: constructing an action space model based on instantaneous wing kinematic modulation theory; constructing a state space model based on sensor constraints and randomizing the initial state; setting a reward function and boundary conditions based on aircraft characteristics and the hovering target; training a policy network using a deep deterministic policy gradient algorithm based on the action space model, the state space model, the reward function, and the boundary conditions; and deploying the trained policy network on a simulated aircraft to control the simulated aircraft to hover. This invention effectively enables hovering, path following, and formation flying of a dual-motor driven tailless micro flapping-wing aircraft, improving flight control accuracy and exhibiting good robustness and generalization ability.
[0090] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for hovering of flapping wing aerial vehicle based on deep reinforcement learning, the method comprising: The application relates to a method for training a hovering flight control strategy of a simulation aircraft. The method comprises the following steps: constructing an action space model according to a transient wing kinematics modulation theory; constructing a state space model according to sensor limitations and randomizing an initial state; setting a reward function and boundary conditions according to aircraft characteristics and a hovering target; training a policy network by using a deep deterministic policy gradient algorithm according to the action space model, the state space model, the reward function and the boundary conditions; 2. The deep reinforcement learning based flapping aerial vehicle hovering method according to claim 1, wherein, arranging the trained policy network on a simulation aircraft to control the simulation aircraft to hover. The constructing an action space model according to a transient wing kinematics modulation theory comprises:
3. The deep reinforcement learning based flapping aerial vehicle hovering method of claim 1, wherein, constructing an action space model for adjusting lift, roll moment, pitch moment and yaw moment of the simulation aircraft in real time according to the transient wing kinematics modulation theory. The constructing a state space model according to sensor limitations and randomizing an initial state comprises: introducing historical observation data through a sliding window to obtain linear velocity, angular velocity, real-time position error and attitude error of the simulation aircraft; 4. The deep reinforcement learning based flapping aerial vehicle hovering method of claim 1, wherein, constructing the state space model according to the linear velocity, the angular velocity, the real-time position error and the attitude error and randomizing an initial state of the state space model. The setting a reward function and boundary conditions according to aircraft characteristics and a hovering target comprises: determining a position error penalty, attitude error, velocity penalty and angular velocity penalty of the simulation aircraft according to the aircraft characteristics and the hovering target; 5. The deep reinforcement learning based flapping aerial vehicle hovering method of claim 1, wherein, setting the reward function and the boundary conditions according to the position error penalty, the attitude error, the velocity penalty and the angular velocity penalty. The training a policy network by using a deep deterministic policy gradient algorithm according to the action space model, the state space model, the reward function and the boundary conditions comprises: controlling multiple simulation aircrafts to fly in parallel according to the action space model and the state space model and collecting flight data of the multiple simulation aircrafts; 6. The deep reinforcement learning based flapping aerial vehicle hovering method according to claim 5, wherein, generating a training sequence based on the flight data and optimizing the policy network by using a deep deterministic policy gradient algorithm according to the reward function and the boundary conditions, so that the policy network outputs a deterministic action to realize aircraft hovering. In each simulation environment, an initial state is constructed from initial observation data ; initializing the policy network, according to the initial state generating a Gaussian distribution of actions, sampling an action from the Gaussian distribution, applying the action to a simulated aircraft in a simulated environment, and calculating a reward according to the reward function; constructing a state based on forward dynamics calculations generating a corresponding action from the policy network prediction and generating a state-action-reward sequence until the data collected by all parallel environments reaches the size of the rolling buffer; The generating a training sequence based on the flight data and optimizing the policy network by using a deep deterministic policy gradient algorithm according to the reward function and the boundary conditions comprises:
7. The deep reinforcement learning based flapping aerial vehicle hovering method of claim 1, wherein, when the data amount of the rolling buffer reaches a preset value, updating a value network and the policy network by using the state-action-reward sequence. The arranging the trained policy network on a simulation aircraft to control the simulation aircraft to hover comprises: applying an effective torque output by a motor to a flapping shaft to drive wing movement and exerting an aerodynamic force calculated by a blade method on the wing to simulate movement of the simulation aircraft in air; fixing a torso part of the simulation aircraft during simulation, measuring force acting on a clamp and inferring force-moments acting on the simulation aircraft. Obtaining a relationship between a voltage input and a force-torque output measured with the control input set to zero; Adjusting the voltage input according to the relationship to apply an average force and torque to the simulated aircraft to control the simulated aircraft to hover.
8. A flapping-wing aerial vehicle hovering device based on deep reinforcement learning, characterized in that, The method comprises: An action space module configured to construct an action space model according to instantaneous wing kinematics modulation theory; A state space module configured to construct a state space model according to sensor limitations and randomize initial states; A reward setting module configured to set a reward function and boundary conditions according to aircraft characteristics and hovering targets; A policy network training module configured to train a policy network according to the action space model, the state space model, the reward function, and the boundary conditions using a deep deterministic policy gradient algorithm; A hovering control module configured to deploy the trained policy network on a simulated aircraft to control the simulated aircraft to hover.
9. A terminal, characterized by comprising: The method comprises: A processor and a memory, wherein the memory stores a deep reinforcement learning flapping wing aircraft hovering program, and the deep reinforcement learning flapping wing aircraft hovering program is used to implement operations of the deep reinforcement learning flapping wing aircraft hovering method according to any one of claims 1-7 when executed by the processor.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a deep reinforcement learning flapping wing aircraft hovering program, and the deep reinforcement learning flapping wing aircraft hovering program is used to implement operations of the deep reinforcement learning flapping wing aircraft hovering method according to any one of claims 1-7 when executed by the processor.