Data-driven insect flapping flight control method based on deep reinforcement learning

Abstract

The application provides a data-driven insect flapping flight control method based on deep reinforcement learning, which comprises the following steps: constructing an insect model according to obtained geometric structure images of real insects, parameterizing modeling of a wing beating cycle based on a real insect flight data set, generating a wing motion trajectory, driving wings of the insect model based on discretization processing of a blade element theory by using the wing motion trajectory, calculating by using a quasi-steady aerodynamic model to obtain total air force and total aerodynamic moment, and applying the total air force and the total aerodynamic moment to the insect model, and calculating angular velocity and angular acceleration according to Euler motion equations to update a motion state of the insect model in a simulation environment. The application establishes a general control strategy which can autonomously learn, the strategy can dynamically map high-dimensional flight state to optimal wing kinematics parameters through continuous interaction with the environment, so that intelligent and adaptive control of insect flight is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biological simulation, in particular to a data-driven insect flapping flight control method based on deep reinforcement learning. BACKGROUND

[0002] Realistic simulation of living organisms, especially flying animals, has a wide range of potential applications, such as virtual reality, entertainment, digital ecological simulation, and biodiversity modeling. However, this direction remains a challenging and attractive research topic. In recent years, researchers have made great efforts in simulating different types of flying species, covering a wide range of flying organisms from large vertebrates (such as birds, bats) to small insects (such as butterflies, fruit flies).

[0003] In order to achieve realistic simulation of animal flight, various methods have been proposed to model wing kinematics and aerodynamics. Early methods are based on quasi-steady aerodynamic models, which globally optimize the wing kinematics offline to generate bird flight along predefined trajectories. Subsequently, researchers proposed a data-driven paradigm based on quasi-steady aerodynamic algorithms, using motion capture data to learn continuous controllers, enabling simulated birds to autonomously match their wing rhythms according to training data. However, this method is highly dependent on the quality and diversity of the captured data, making it difficult to generalize to maneuvers not adequately covered in the training set. On this basis, subsequent research has expanded the application of quasi-steady aerodynamic models to butterfly flight, updating wing parameters through manually designed functions. However, this artificial setting may produce flight behaviors that deviate from physical reality, such as sustained upward flight without wing flapping. In addition, these methods have a common limitation: they rely on pre-defined control paradigms, whether global optimization along a specific path, matching in a motion database, or hand-designed control laws. Therefore, the resulting insect simulation individuals lack a general learning mechanism, making it difficult to autonomously discover and adapt to high-level flight strategies in complex or novel environments.

[0004] Compared with bird flight, insect flight presents additional challenges. First, due to the small size of insects, it is extremely difficult to obtain comprehensive motion data, especially accurate wing trajectory data. Second, insect aerodynamics is mainly dominated by unsteady flow effects, so relying solely on quasi-steady aerodynamic models is insufficient to accurately describe the force generation mechanism required for insect flight. SUMMARY

[0005] In view of the above situation, the main purpose of the present application is to provide a data-driven insect flapping flight control method based on deep reinforcement learning to solve the above technical problems.

[0006] The application provides a data-driven insect flapping flight control method based on deep reinforcement learning, and the method comprises the following steps:

[0007] Step 1: constructing an insect model according to the obtained geometric structure image of a real insect;

[0008] Step 2: parameterizing modeling of a wing beating cycle based on a real insect flight data set, and generating a wing motion trajectory;

[0009] Step 3: driving the wings of the insect model based on the leaf element theory discretization processing by using the wing motion trajectory, and calculating by using a quasi-steady aerodynamic model to obtain total air force and total aerodynamic moment;

[0010] Step 4: applying the total air force and the total aerodynamic moment to the insect model, and calculating the angular velocity and the angular acceleration according to the Euler motion equation to update the motion state of the insect model in the simulation environment;

[0011] Step 5: configuring a virtual camera sensor for the insect model in the simulation environment to render an environment image, and encoding the environment image by using a convolutional neural network to obtain visual features; splicing the visual features and the motion state of the insect model to obtain a complete environment observation state;

[0012] Step 6: training a policy network based on reinforcement learning; inputting the complete environment observation state into the policy network for processing, and obtaining a control action by guiding the total reward, wherein the control action comprises the beating amplitude and the frequency of the left and right wings;

[0013] Step 7: feeding back the control action to the insect model to drive the wings to produce new motion and change the flight state of the insect model, so as to realize the control of the insect flapping flight.

[0014] Compared with the prior art, the application has the beneficial effects that:

[0015] 1: Based on the existing insect flapping data set, the natural flapping cycle is parameterized and modeled, and the natural flapping process driven by the real data is represented as a smooth and connected periodic motion, so that physically reasonable flapping kinematics can be generated, the physical characteristics and dynamic laws of the natural wing beating are effectively retained, and the authenticity of the flapping motion is significantly improved;

[0016] 2: Based on the traditional quasi-constant aerodynamic model, two key unsteady aerodynamic components are introduced, which can more accurately reflect the unsteady aerodynamic effect in the insect flight process, thereby significantly improving the accuracy of the flapping flight air force;

[0017] 3. This invention establishes a general control strategy that can learn autonomously. This strategy can dynamically map high-dimensional flight states to optimal wing kinematic parameters through continuous interaction with the environment, thereby achieving intelligent and adaptive control of insect flight.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the steps of a data-driven insect flapping-wing flight control method based on deep reinforcement learning proposed in this invention.

[0020] Figure 2 This is a structural diagram of the insect model proposed in this invention. Figure 2 Image (a) shows a 3D model of the blue morpho butterfly and the hoverfly. Figure 2 (b) shows the anatomical structures of the blue morpho butterfly and the hoverfly;

[0021] Figure 3 This is a general framework diagram of a data-driven insect flapping-wing flight control method based on deep reinforcement learning proposed in this invention;

[0022] Figure 4 This is a diagram illustrating the flapping of an insect's wings and its body posture. Figure 4 In the diagram, (a) represents the three flapping angles of an insect's wing. Figure 4 (b) represents the three posture angles of the insect's body;

[0023] Figure 5 This is a schematic diagram illustrating the discretization of wings based on leaf element theory.

[0024] Figure 6 A diagram showing the results of an insect's learning of basic flight skills;

[0025] Figure 7 This is a diagram showing the result of imitating the flight trajectory of an insect;

[0026] Figure 8 A diagram illustrating the results of insect learning to avoid visual obstacles.

[0027] Figure 9 A comparison image of the real Morpho blue butterfly and the Morpho blue butterfly simulated by this method;

[0028] Figure 10 This is a comparison image of real hoverflies and hoverflies simulated by this method. Figure 10 (a) in the image represents a real hoverfly. Figure 10 (b) in the figure represents the hoverfly simulated by this method;

[0029] Figure 11 This is a comparison image of the Blue Morpho butterfly simulated by existing methods and simulated by our method. Figure 11 (a) in the image shows the gliding flight of the blue morpho butterfly simulated by existing methods. Figure 11 (b) in the figure represents the gliding flight of the blue morpho butterfly simulated by this method. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0031] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0032] Please see Figure 1 This embodiment provides a data-driven insect flapping-wing flight control method based on deep reinforcement learning, the method comprising the following steps:

[0033] Step 1: Construct an insect model based on the obtained geometric structure image of a real insect.

[0034] Please see Figure 2 In step 1, the insect model consists of a head, thorax, abdomen, and wings. The wings are connected to the thorax via a hierarchical hinge structure. The head, thorax, and abdomen together form the body.

[0035] Step 2: Based on real insect flight datasets, perform parametric modeling of wing flapping period to generate wing motion trajectory.

[0036] Please see Figure 3 and Figure 4 In step 2, based on a real insect flight dataset, the wingbeat period is parametrically modeled to generate the wing motion trajectory, which specifically includes the following sub-steps:

[0037] Based on a real insect flight dataset, the average trajectory is extracted from multiple wing flapping cycles and fitted using Fourier series. The fitted result is defined as the baseline flapping cycle, and the following relationship exists in the process:

[0038] ;

[0039] in, Indicates the flapping angle, This represents the average angle over one period. and All indicate the first Fourier coefficients of the first harmonic, Represents the dimensionless time normalized to a single flapping period;

[0040] By introducing a frequency-amplitude modulation mechanism, the reference flapping period is generalized by adjusting the frequency and amplitude to obtain the extended flapping motion. The following relationship exists in the corresponding process:

[0041] ;

[0042] in, A function representing the three flapping angles. This represents the magnitude of the three angular components that vary over time. This represents the frequency at which the three angular components are variable over time. This indicates the amplitude of the baseline flapping period. The frequency representing the reference flapping wing period, Indicates the corner of the slap. Indicates the sweeping angle. Indicates the rotation angle.

[0043] Furthermore, when the flapping amplitude changes continuously, discontinuities may occur between adjacent flapping cycles. By introducing a smooth transition function for the flapping amplitude based on the Sigmoid function, smooth amplitude interpolation is ensured between adjacent flapping cycles to generate the wing motion trajectory. The corresponding relationship is as follows:

[0044] ;

[0045] in, A smooth transition function representing the flapping amplitude. Indicates the first The amplitude of each flapping cycle, Indicates the first The amplitude of each flapping cycle, This represents the smoothing factor, used to control the steepness of the transition; Indicates the start time of the transition interval. Indicates the end time of the transition interval.

[0046] It should be noted that, unlike traditional methods that require repeating the motion twice within the transition interval to achieve smooth flapping, our method does not require additional forward recalculation, thus enabling more efficient and seamless fusion between consecutive flapping cycles.

[0047] Step 3: Drive the wings of the insect model based on leaf element theory discretization using the wing motion trajectory, and use the quasi-steady-state aerodynamic model to calculate the total aerodynamic force and total aerodynamic torque.

[0048] Please see Figure 5 In step 3, the wings of the insect model discretized based on leaf element theory are driven by the wing motion trajectory, and the total aerodynamic force and total aerodynamic torque are calculated using a quasi-steady-state aerodynamic model. This includes the following sub-steps:

[0049] Based on leaf element theory, the wings of the insect model are discretized along the span direction into a series of equidistant leaf elements, and each leaf element is modeled as a two-dimensional airfoil.

[0050] The translational lift and drag acting on the blade element are calculated using a quasi-steady-state aerodynamic model to obtain the translational lift and drag of the blade element. The following relationship exists in the corresponding process:

[0051] ;

[0052] in, Indicates the first The translational lift of a leaf element Indicates the first Translational resistance of a leaf element Represents the air density constant. Indicates the first The chord length of a leaf element, Indicates the first The radial width of each leaf element, Indicates the action on the first Airflow velocity on the surface of each leaf element This represents the vector modulo operation. Indicates the first The lift coefficient of each leaf element, Indicates the first The drag coefficient of each leaf element, Indicates the first The angle of attack of a leaf element;

[0053] The lift coefficient has the following relationship: ;

[0054] The drag coefficient has the following relationship: ;

[0055] in, This indicates the sensitivity of the quasi-steady-state pressure coefficient to the angle of attack. This indicates taking the sine value. This indicates taking the cosine value. This indicates the offset of the drag coefficient;

[0056] The angle of attack of leaf element is related by the following formula:

[0057] ;

[0058] in, This indicates taking the tangent value. This represents the component of air velocity in the direction normal to the leaf element surface. This represents the component of air velocity in the tangential direction at the leaf element surface;

[0059] Furthermore, while the quasi-steady-state aerodynamic model can effectively capture the overall trend of aerodynamic force generation during insect flapping flight, it essentially ignores the key unsteady mechanisms that dominate during rapid wing acceleration and commutation. Specifically, the dynamic interaction between the insect model's wings and the surrounding fluid triggers transient aerodynamic forces that cannot be explained by the steady-state assumption. To compensate for this deficiency, two main types of unsteady aerodynamic force components are introduced into the quasi-steady-state aerodynamic model: additional mass force and rotational circulation force.

[0060] The additional mass force has the following relationship: ;

[0061] The rotational circulation force has the following relationship: ;

[0062] in, Indicates the first The added mass force of each leaf element, Indicates the first The acceleration in the normal direction of the leaf element surface; where... Indicates the first The rotational force of a leaf element The time derivative of the wing rotation angle. Indicates the contribution of rotational circulation;

[0063] definition To act on the first The first wing The total aerodynamic force of each leaf element is such that the instantaneous resultant force exerted by the wings on the body and the instantaneous torque exerted by the wings on the body have the following relationship:

[0064] ;

[0065] in, Indicates the action on the first The first wing The total aerodynamics of each leaf element Indicates the first The instantaneous combined force exerted by the wings on the body, Indicates the first The instantaneous torque exerted by each wing on the body, Indicates the body's center of mass to the first The first wing The lever arm of the leaf element center;

[0066] The total aerodynamic force is calculated based on the instantaneous resultant force exerted by the wings on the body, and the following relationship exists in the corresponding process:

[0067] ;

[0068] in, Indicates total aerodynamics;

[0069] The total aerodynamic torque is calculated based on the instantaneous torque exerted by the wings on the body, and the following relationship exists in the corresponding process:

[0070] ;

[0071] in, This represents the total aerodynamic torque.

[0072] It should be noted that the total aerodynamics It is the primary driving force for insect flight propulsion, used to generate forward velocity and position updates; total aerodynamic torque It acts on the rotation of the body around its main axes to adjust the insect's posture, including roll, pitch and yaw control.

[0073] Furthermore, the difference between experimental measurements and predictions using a quasi-steady-state aerodynamic model is measured using mean square error, and the following relationship exists in the corresponding process:

[0074] ;

[0075] in, This represents the mean square error between experimental measurements and model predictions. Indicates the number of wing beat cycles. This indicates the number of time sampling points per cycle. Indicates the first Within the first wing flapping cycle Experimental aerodynamics at a given moment Indicates the first Within the first wing flapping cycle Predicted aerodynamic forces at a given moment.

[0076] Step 4: Apply the total aerodynamic force and total aerodynamic torque to the insect model, and calculate the angular velocity and angular acceleration according to the Euler equation of motion to update the motion state of the insect model in the simulation environment.

[0077] In step 4, the total aerodynamic force and total aerodynamic torque are applied to the insect model, and the angular velocity and angular acceleration are calculated according to the Euler equations of motion to update the motion state of the insect model in the simulation environment. This includes the following sub-steps:

[0078] The total aerodynamic force and total aerodynamic torque are applied to the body of the insect model, and the angular velocity and angular acceleration are obtained by solving the Euler equations of motion. The following relationship exists in the corresponding process:

[0079] ;

[0080] in, Represents angular acceleration in the body coordinate system. Represents the moment of inertia. Represents the angular velocity in the body coordinate system;

[0081] The moment of inertia has the following relationship:

[0082] ;

[0083] in, Indicates along The inertial components of the axis, Indicates along The inertial components of the axis, Indicates along The inertial components of the axis, Indicates the total mass of the insects. Indicates the length of an insect's body. Indicates the width of the insect's body. Indicates the thickness of an insect's body;

[0084] The motion state of the insect model in the simulation environment is updated using angular velocity and angular acceleration.

[0085] Step 5: Configure a virtual camera sensor for the insect model in the simulation environment to render and generate environmental images, and encode them through a convolutional neural network to obtain visual features; stitch the visual features with the movement state of the insect model to obtain a complete environmental observation state;

[0086] Step 6: Obtain a policy network based on reinforcement learning training; input the complete environmental observation state into the policy network for processing, and obtain control actions through the guidance of a hierarchical reward function. The control actions include the flapping amplitude and frequency of the left and right wings.

[0087] It should be noted that, when using the insect model as an agent, the training process is organized into three stages of progressive complexity: primary, intermediate, and advanced.

[0088] In the initial stage, the agent learns to maintain flight stability and basic maneuvering skills, such as forward flight, climb, gliding, and turning, by interacting with the environment.

[0089] In the intermediate stage, the strategy aims to mimic insect movement data (provided in the form of target position and posture), enabling the agent to reproduce biologically realistic trajectories and kinematic patterns.

[0090] In the advanced stage, the policy network introduces a virtual camera sensor, enabling the agent to perceive the environment and learn obstacle avoidance behavior while maintaining insect flight characteristics.

[0091] At all stages, the policy maps the observed state to control actions to drive the agent toward the desired motion and task objectives.

[0092] In step 6, the total reward includes the target reward, kinematic reward, dynamic reward, balance reward, and collision avoidance reward;

[0093] The function expression for the total reward is:

[0094] ;

[0095] in, Indicates time step Total reward Indicates a time step. All represent learnable weights. Indicates time step The target reward, Indicates time step Kinematic rewards, Indicates time step The dynamic reward, Indicates time step Balanced rewards, Indicates time step Collision avoidance rewards;

[0096] The function expression for the target reward is:

[0097] ;

[0098] in, Represents an exponential function. This represents the distance vector from the agent's current position to the target position. This indicates the scope of influence of controlling rewards.

[0099] The functional expression for kinematic reward is:

[0100] ;

[0101] in, This represents the flapping amplitude of the intelligent agent. This represents the flapping frequency of the intelligent agent. This represents the mean of the actual flapping amplitude data. The standard deviation of the actual flapping amplitude data. This represents the mean of the actual flapping frequency data. The standard deviation of the actual flapping frequency data;

[0102] The functional expression for the dynamic reward is:

[0103] ;

[0104] in, Indicates velocity and acceleration. The set representing velocity and acceleration. This indicates taking the maximum value. This represents the maximum value of velocity and acceleration. Represents the minimum values ​​of velocity and acceleration;

[0105] The functional expression for balancing rewards is:

[0106] ;

[0107] in, This represents the angular velocity of the agent's rolling motion. This represents the angular acceleration of the agent's rolling motion;

[0108] The function expression for the collision avoidance reward is:

[0109] ;

[0110] in, This indicates the number of obstacles detected. This indicates the range of visual perception of the intelligent agent. Indicates the intelligent agent and the first The distance to the obstacle.

[0111] Step 7: Feedback the control action to the insect model to drive the wings to generate new movements, change the flight state of the insect model, and achieve control of the insect's flapping flight.

[0112] Please see Figures 6 to 11 Simulation results and comparisons of the present invention:

[0113] Basic maneuvering skills learning. In the initial stage, the insect agent is first trained to avoid crashes and maintain flight stability. Once basic stability is achieved, specific mission objectives are introduced to guide the agent in learning basic maneuvering skills, including forward flight, climb, gliding, and turning. Figure 6 As shown, setting the target at a higher altitude prompts the agent to climb, while setting it at a lower altitude induces gliding. Similarly, by placing the target in the appropriate spatial orientation, the agent can be guided to perform forward flight and turning maneuvers.

[0114] Biological flight trajectory imitation. In the intermediate stage, the training focus shifts from general maneuvers to the imitation of natural insect flight. The policy is trained using reference datasets to reproduce biologically authentic trajectories and kinematic patterns. In the experiments, we used approximately 142 flight trajectories of the blue morpho butterfly and 879 flight trajectories of the hoverfly. These datasets provide detailed kinematic and dynamic information and can be used as reference targets. By minimizing the deviation from the reference trajectory, the agent learns to replicate characteristic flight behaviors (such as...). Figure 7 (As shown).

[0115] Visual obstacle avoidance learning. In the advanced training phase, visual obstacle avoidance learning is incorporated, a process divided into two progressive stages. In the first stage, the insect agent avoids simple geometric obstacles, such as cylinders and cubes; in the second stage, the obstacles are replaced by more complex virtual plants, including flowers and trees. Through this training process, the insect learns to initially perceive the relative positions of obstacles and, based on this perception, autonomously adjusts its wing kinematic parameters to generate aerodynamics that guide flight away from obstacles, thereby achieving effective obstacle avoidance (e.g., ...). Figure 8 (As shown).

[0116] Comparison with real Blue Morpho butterfly videos. In this comparative experiment, we simulated a virtual Blue Morpho butterfly and directly compared it with real Blue Morpho butterfly videos. The results show (e.g.) Figure 9 As shown, the simulated Blue Morpho butterfly exhibits a significant periodic overall pitch motion, which originates from the intermittent lift generated by low-frequency wing flapping. This pitch motion is a natural result of wing-body coupled vibrations, consistent with the flight movements of real butterflies.

[0117] A comparison with a real video of a hoverfly. Unlike butterflies, hoverflies maintain a highly stable body movement during flight, thanks to the continuous lift generated by the high-frequency flapping of their single pair of slender wings. Figure 10 The results show a comparison between real hoverflies and hoverflies simulated by this method. Figure 10 (a) in the image represents a real hoverfly. Figure 10 (b) in the figure represents the hoverfly simulated by this method. Experimental results show that the hoverfly simulated by the method presented in this paper can reproduce flight movements that are highly similar to those of real individuals.

[0118] Comparison with existing methods. We compare our proposed method with recent state-of-the-art methods for butterfly flight simulation. Unlike our framework, this method does not couple kinematics and dynamics based on physical principles, but rather updates wing kinematics through artificially designed rules; the resulting flight behavior is inconsistent with physical reality, for example, the butterfly continuously rises without flapping its wings (e.g., Figure 11 (as shown in (a)); in contrast, as Figure 11 As shown in (b), the method presented in this paper enables butterflies to ascend by accelerating wing flapping and descend during gliding, thus following the physical laws of natural flight.

[0119] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0120] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0121] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0122] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A data-driven insect flapping-wing flight control method based on deep reinforcement learning, characterized in that, The method includes the following steps: Step 1: Construct an insect model based on the obtained geometric structure image of a real insect; Step 2: Based on real insect flight datasets, perform parametric modeling of wingbeat period to generate wing motion trajectory; Step 3: Drive the wings of the insect model discretized based on leaf element theory using the wing motion trajectory, and use the quasi-steady-state aerodynamic model to calculate the total aerodynamic force and total aerodynamic torque. Step 4: Apply the total aerodynamic force and total aerodynamic torque to the insect model, and calculate the angular velocity and angular acceleration according to the Euler equation of motion to update the motion state of the insect model in the simulation environment; Step 5: Configure a virtual camera sensor for the insect model in the simulation environment to render and generate environmental images, and encode them through a convolutional neural network to obtain visual features; Visual features are stitched together with the movement state of the insect model to obtain a complete environmental observation state; Step 6: Obtain the policy network based on reinforcement learning training; input the complete environmental observation state into the policy network for processing, and obtain the control action through the guidance of the total reward; Step 7: Feedback the control action to the insect model to drive the wings to generate new movements, change the flight state of the insect model, and achieve control of the insect's flapping flight; In step 2, based on a real insect flight dataset, the wingbeat period is parametrically modeled to generate the wing motion trajectory, which specifically includes the following sub-steps: Based on a real insect flight dataset, the average trajectory is extracted from multiple wing flapping cycles and fitted using Fourier series. The fitted result is defined as the baseline flapping cycle, and the following relationship exists in the process: ； in, Indicates the flapping angle, This represents the average angle over a period of time. and All indicate the first Fourier coefficients of the first harmonic, Represents the dimensionless time normalized to a single flapping period; By introducing a frequency-amplitude modulation mechanism, the reference flapping period is generalized by adjusting the frequency and amplitude to obtain the extended flapping motion. The following relationship exists in the corresponding process: ； in, A function representing the three flapping angles. This represents the magnitude of the three angular components that vary over time. This represents the frequency at which the three angular components are variable over time. This indicates the amplitude of the baseline flapping period. The frequency representing the reference flapping wing period, Indicates the corner of the slap. Indicates the sweeping angle. Indicates the rotation angle; When the flapping amplitude changes continuously, discontinuities occur between adjacent flapping cycles. By introducing a smooth transition function for the flapping amplitude based on the Sigmoid function, smooth amplitude interpolation is achieved between adjacent flapping cycles to generate the wing motion trajectory. The following relationship exists in the corresponding process: ； in, A smooth transition function representing the flapping amplitude. Indicates the first The amplitude of each flapping cycle, Indicates the first The amplitude of each flapping cycle, Represents the smoothing factor. Indicates the start time of the transition interval. Indicates the end time of the transition interval.

2. The data-driven insect flapping-wing flight control method based on deep reinforcement learning according to claim 1, characterized in that, In step 1, the insect model consists of a head, thorax, abdomen, and wings. The wings are connected to the thorax via a hierarchical hinge structure, and the head, thorax, and abdomen together form the body.

3. The data-driven insect flapping-wing flight control method based on deep reinforcement learning according to claim 2, characterized in that, In step 3, the wings of the insect model discretized based on leaf element theory are driven by the wing motion trajectory, and the total aerodynamic force and total aerodynamic torque are calculated using a quasi-steady-state aerodynamic model. This includes the following sub-steps: Based on leaf element theory, the wings of the insect model are discretized along the span direction into a series of equidistant leaf elements, and each leaf element is modeled as a two-dimensional airfoil. The translational lift and translational drag acting on the blade element are calculated using a quasi-steady-state aerodynamic model to obtain the translational lift and translational drag of the blade element. definition To act on the first The first wing The total aerodynamic force of each leaf element was calculated, and the instantaneous resultant force and instantaneous torque exerted by the wings on the body were obtained. The total aerodynamic force is calculated based on the instantaneous resultant force exerted by the wings on the body; The total aerodynamic torque is calculated based on the instantaneous torque exerted by the wings on the body.

4. The data-driven insect flapping-wing flight control method based on deep reinforcement learning according to claim 3, characterized in that, In the step of calculating the translational lift and translational drag acting on the blade element using a quasi-steady-state aerodynamic model to obtain the translational lift and translational drag of the blade element, the following relationship exists: ； in, Indicates the first The translational lift of a leaf element Indicates the first Translational resistance of a leaf element Represents the air density constant. Indicates the first The chord length of a leaf element, Indicates the first The radial width of each leaf element, Indicates the action on the first Airflow velocity on the surface of each leaf element This represents the vector modulo operation. Indicates the first The lift coefficient of each leaf element, Indicates the first The drag coefficient of each leaf element, Indicates the first Angle of attack of a leaf element; The lift coefficient has the following relationship: ; The drag coefficient has the following relationship: ; in, This indicates the sensitivity of the quasi-steady-state pressure coefficient to the angle of attack. This indicates taking the sine value. This indicates taking the cosine value. This indicates the offset of the drag coefficient; The angle of attack of leaf element is related by the following formula: ； in, This indicates taking the tangent value. This represents the component of air velocity in the direction normal to the leaf element surface. This represents the component of air velocity in the tangential direction at the leaf element surface; In definition To act on the first The first wing In the steps of calculating the total aerodynamic force of each leaf element and obtaining the instantaneous resultant force and instantaneous torque exerted by the wings on the body, the following relationship exists: ； in, Indicates the action on the first The first wing The total aerodynamics of each leaf element Indicates the first The instantaneous combined force exerted by the wings on the body, Indicates the first The instantaneous torque exerted by each wing on the body, Indicates the body's center of mass to the first The first wing The lever arm of the leaf element center; In the step of calculating the total aerodynamic force based on the instantaneous resultant force exerted by the wings on the body, the following relationship exists: ； in, Indicates total aerodynamics; In the step of calculating the total aerodynamic torque based on the instantaneous torque exerted by the wings on the body, the following relationship exists: ； in, This represents the total aerodynamic torque.

5. The data-driven insect flapping-wing flight control method based on deep reinforcement learning according to claim 4, characterized in that, In step 4, the total aerodynamic force and total aerodynamic torque are applied to the insect model, and the angular velocity and angular acceleration are calculated according to the Euler equations of motion to update the motion state of the insect model in the simulation environment. This includes the following sub-steps: The total aerodynamic force and total aerodynamic torque are applied to the body of the insect model, and the angular velocity and angular acceleration are obtained by solving the Euler equations of motion. The following relationship exists in the corresponding process: ； in, Represents angular acceleration in the body coordinate system. Represents the moment of inertia. Represents the angular velocity in the body coordinate system; The moment of inertia has the following relationship: ； in, Indicates along The inertial components of the axis, Indicates along The inertial components of the axis, Indicates along The inertial components of the axis, Indicates the total mass of the insects. Indicates the length of an insect's body. Indicates the width of the insect's body. Indicates the thickness of an insect's body; The motion state of the insect model in the simulation environment is updated using angular velocity and angular acceleration.

6. The data-driven insect flapping-wing flight control method based on deep reinforcement learning according to claim 5, characterized in that, In step 6, the total reward includes target reward, kinematic reward, dynamic reward, balance reward and collision avoidance reward, and the control action includes the flapping amplitude and frequency of the left and right wings; The function expression for the total reward is: ； in, Indicates time step Total reward Indicates a time step. , , , , All represent learnable weights. Indicates time step The target reward, Indicates time step Kinematic rewards, Indicates time step Dynamic reward, Indicates time step Balanced rewards, Indicates time step Collision avoidance rewards; The function expression for the target reward is: ； in, Represents an exponential function. This represents the distance vector from the agent's current position to the target position. This indicates the scope of influence of the controlled rewards. In this context, insect models are used as intelligent agents. The functional expression for kinematic reward is: ； in, This represents the flapping amplitude of the intelligent agent. This represents the flapping frequency of the intelligent agent. This represents the mean of the actual flapping amplitude data. The standard deviation of the actual flapping amplitude data. This represents the mean of the actual flapping frequency data. The standard deviation of the actual flapping frequency data; The functional expression for balancing rewards is: ； in, This represents the angular velocity of the rolling motion of a flying insect. The angular acceleration representing the rolling motion of a flying insect; The function expression for the collision avoidance reward is: ； in, Indicates the number of obstacles detected. This indicates the range of visual perception of the intelligent agent. Indicates the intelligent agent and the first The distance to the obstacle.

Images