Unmanned aerial vehicle stunt flight control method, device, equipment, medium and product

Abstract

The invention discloses an unmanned aerial vehicle stunt flight control method, device and equipment, a medium and a product. The method is applied to an unmanned aerial vehicle and comprises the following steps: acquiring unmanned aerial vehicle attitude information corresponding to the unmanned aerial vehicle at a current moment; inputting the unmanned aerial vehicle attitude information into an unmanned aerial vehicle stunt strategy network corresponding to the target stunt flight action to obtain target unmanned aerial vehicle control information corresponding to the unmanned aerial vehicle at the current moment; and controlling the unmanned aerial vehicle to execute the target unmanned aerial vehicle control information, obtaining unmanned aerial vehicle attitude information at the next moment, and repeatedly executing the steps of determining the target unmanned aerial vehicle control information, executing the target unmanned aerial vehicle control information and obtaining the unmanned aerial vehicle attitude information under the condition that the unmanned aerial vehicle attitude information does not reach preset ending attitude information. And controlling the unmanned aerial vehicle to complete the target stunt flight action. According to the technical scheme provided by the embodiment of the invention, the effect of controlling the unmanned aerial vehicle to autonomously complete the stunt flight action based on the strategy network is realized, and the intelligence of the stunt flight of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) intelligent control technology, and in particular to a UAV aerobatic flight control method, device, equipment, medium and product. Background Art

[0002] Drone aerobatics, as a flying method with high technical difficulty and outstanding visual effects, is widely used in entertainment performances, racing events and other fields.

[0003] In related technologies, drone aerobatic flights usually rely on manual remote control or pre-programmed flight trajectories. However, manual remote control aerobatic flights require high operating skills from users and require high personnel training costs; pre-programmed flight trajectories are difficult to cope with changes in the external environment and lack adaptability and real-time decision-making capabilities. Summary of the invention

[0004] The present invention provides a method, device, equipment, medium and product for controlling unmanned aerial vehicle aerobatic flight, so as to achieve the effect of controlling the unmanned aerial vehicle to autonomously complete aerobatic flight actions based on a policy network, thereby improving the intelligence of the unmanned aerial vehicle aerobatic flight.

[0005] According to one aspect of the present invention, a method for controlling aerobatic flight of a drone is provided, the method being applied to a drone and comprising:

[0006] Acquire the drone attitude information corresponding to the drone at the current moment; wherein the drone attitude information is used to characterize the motion attitude of the drone; and the drone attitude information corresponds to the target aerobatic action to be performed;

[0007] Inputting the drone attitude information into a drone aerobatic strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; wherein the drone aerobatic strategy network is obtained by training a second deep network model based on a reinforcement learning algorithm and a drone simulation model corresponding to the drone;

[0008] The drone is controlled to execute the target drone control information, and the drone attitude information of the drone at the next moment after the current moment is obtained. When the drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic action.

[0009] According to another aspect of the present invention, there is provided a drone aerobatic flight control device, which is configured in a drone and includes:

[0010] The posture information acquisition module is used to acquire the posture information of the drone corresponding to the current moment; wherein the posture information of the drone is used to characterize the motion posture of the drone; and the posture information of the drone corresponds to the target aerobatic action to be performed;

[0011] A control information determination module is used to input the drone attitude information into a drone aerobatic strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; wherein the drone aerobatic strategy network is obtained by training a second deep network model based on a reinforcement learning algorithm and a drone simulation model corresponding to the drone;

[0012] The special effects flight action completion module is used to control the drone to execute the target drone control information and obtain the drone attitude information of the drone at the next moment after the current moment. When the drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic flight action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic flight action.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the unmanned aerial vehicle aerobatic flight control method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the unmanned aerial vehicle aerobatic flight control method described in any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program, and the computer program, when executed by a processor, implements the method for controlling the aerobatic flight of a drone as described in any embodiment of the present invention.

[0019] The technical solution of the embodiment of the present invention obtains the drone attitude information corresponding to the drone at the current moment; further, the drone attitude information is input into the drone stunt strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; further, the drone is controlled to execute the target drone control information, and the drone attitude information of the drone at the next moment of the current moment is obtained. When the drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic action, thereby solving the technical problems in the related technologies that have high professional requirements and are time-consuming and laborious, and realizing the effect of controlling the drone to autonomously complete aerobatic actions based on the strategy network, thereby improving the intelligence of the drone aerobatic flight, and the strategy network obtained by reinforcement learning can realize more accurate drone attitude control, and the application of reinforcement learning algorithm to drone aerobatic control can realize efficient aerobatic strategy with high accuracy and generalization.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a method for controlling aerobatic flight of a drone provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flow chart of a method for controlling aerobatic flight of a drone provided according to Embodiment 2 of the present invention;

[0024] Figure 3 is a schematic structural diagram of a UAV aerobatic flight control device provided according to Embodiment 3 of the present invention;

[0025] Figure 4 It is a schematic diagram of the structure of an electronic device for implementing the unmanned aerial vehicle aerobatic flight control method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] Figure 1 This is a flow chart of a method for controlling aerobatic flight of a drone provided in the first embodiment of the present invention. This embodiment can be applied to the case of controlling a drone to perform aerobatic flight actions. The method is applied to a drone and can be executed by a drone aerobatic flight control device. The drone aerobatic flight control device can be implemented in the form of hardware and / or software. The drone aerobatic flight control device can be configured in a terminal and / or a server. Figure 1 As shown, the method includes:

[0030] S110: Obtain the drone attitude information corresponding to the drone at the current moment.

[0031] Among them, the drone, which can also be called an unmanned aerial vehicle, is an unmanned robot that flies by remote control or autonomously. In general, in addition to being able to complete normal flight actions, drones can also complete maneuvers that are not required for normal flight, namely aerobatic actions. The current moment is any moment when the drone is completing aerobatic actions. The drone attitude information can be used to characterize the motion attitude of the drone. The drone attitude information can be used to describe the position and direction of the drone in space. Optionally, the drone attitude information may include at least one of the drone yaw angle, the drone pitch angle, the drone yaw angle, the drone sideslip angle, the attitude quaternion, the drone speed, the flight angular velocity, the drone angle of attack, and the drone overload. It should be noted that the drone attitude information corresponds to the target aerobatic action to be performed. In other words, the drone attitude parameters to be obtained can be pre-determined according to the target aerobatic action to be performed. Furthermore, in the process of the drone completing the target aerobatic action, the corresponding parameter values ​​at each moment can be obtained according to the predetermined drone attitude parameters. Furthermore, the parameter values ​​corresponding to the acquired drone attitude parameters at the current moment can be used as the drone attitude information corresponding to the drone at the current moment. Targeted aerobatic maneuvers generally refer to maneuvers that are not required for normal flight and that can be performed by the drone. Targeted aerobatic maneuvers generally include sharp changes in slope, abnormal pitch or nose-down attitudes, and abnormal accelerations such as sudden changes in slope, pitch, and speed. Such maneuvers require the drone to have a high degree of maneuverability and stability, as well as precise flight control capabilities. Optionally, targeted aerobatic maneuvers include, but are not limited to, somersaults, half somersaults, and S-shaped maneuvers.

[0032] It should be noted that there are multiple ways to obtain the drone attitude information: the drone attitude information can be collected by a sensor set on the drone; or the drone can be detected by a control terminal associated with the drone to collect the drone attitude information corresponding to the drone at each moment;

[0033] As an optional implementation of the first embodiment of this embodiment, when determining the target aerobatic action to be performed by the drone, the drone attitude parameter corresponding to the target aerobatic action can be determined. Further, in the process of the drone performing the target aerobatic action, the drone attitude information corresponding to the drone at each moment can be obtained by the sensor set on the drone according to the predetermined drone attitude parameter. Furthermore, when the drone attitude information corresponding to the drone at the current moment is obtained, the flight control strategy for the drone to perform the target aerobatic action can be determined based on the drone attitude information.

[0034] It should be noted that the drone attitude information corresponding to the drone at the initial moment is the preset initial attitude information or the default attitude information. Exemplarily, it is assumed that the drone attitude information includes the drone roll angle, the drone pitch angle, the drone yaw angle, the drone sideslip angle, the attitude quaternion, the flight speed, the flight angular velocity, the drone angle of attack, and the drone overload; the drone attitude information corresponding to the drone at the initial moment may be the drone roll angle, the drone pitch angle, the drone yaw angle, the drone sideslip angle, the attitude quaternion, the flight speed, the flight angular velocity, the drone angle of attack, and the drone overload are all zero.

[0035] S120: Inputting the drone attitude information into the drone aerobatic strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment.

[0036] Among them, the drone stunt strategy network is obtained by training the second deep network model based on the reinforcement learning algorithm and the drone simulation model corresponding to the drone.

[0037] Among them, the drone stunt strategy network can be a neural network model that takes the drone attitude information corresponding to the drone at the current moment as an input object to determine the decision action corresponding to the drone at the current moment based on the input object. The drone stunt strategy network is a strategy network obtained by training the second deep network model based on the reinforcement learning algorithm and the drone simulation model corresponding to the drone. The drone simulation model can be a model that uses mathematics and computer technology to simulate the drone system. The drone simulation model is based on the relevant theories such as the flight kinematics, aerodynamics and flight control principles of the drone, and then uses this model as a basis for simulation experiments and analysis. In this embodiment, after determining the drone to perform the target aerobatic action, a simulation model corresponding to the drone can be constructed according to the relevant power parameters, aerodynamic parameters and flight control parameters of the drone to obtain the drone simulation model. Further, the drone simulation model and the second deep network model can be trained based on the reinforcement learning algorithm to obtain the drone stunt strategy network. It can be understood that the strategy network can be a deep neural network model that deploys a strategy function, and its corresponding input object can be the state information of the environment in which the agent is located, and its corresponding output object can be the decision action determined according to the input state. In this embodiment, the input object corresponding to the drone stunt strategy network can be the drone attitude information corresponding to the drone when the drone performs the target aerobatic action, and the corresponding output object is the target drone control information determined according to the input drone attitude information. The drone stunt strategy network can be a deep neural network model of any model structure. The model structure of the drone stunt strategy network is not limited here, and can be customized according to actual needs. Exemplarily, the drone stunt strategy network includes at least a multilayer perceptron (Mult ilayer Perceptron, MLP) and a recurrent neural network (Recurrent Neural Network, RNN). The target drone control information can be used to control the drone to perform the target aerobatic action. The target drone control information can be understood as the drone control instruction that controls the drone to perform the corresponding action. That is to say, when the target drone control information is obtained, the drone can be directly controlled to execute the obtained target drone control information to change the position and direction of the drone in space. Optionally, the target drone control information includes but is not limited to throttle information, roll control surface control information, pitch control surface control information, and yaw control surface control information. It should be noted that, in the case where the drone attitude information corresponds to the target aerobatic flight action, the target drone control information may also correspond to the target aerobatic flight action.

[0038] It should be noted that the UAV aerobatic strategy network may correspond to the target aerobatic maneuvers, that is, different target aerobatic maneuvers may use different UAV aerobatic strategy networks to determine the aerobatic strategies.

[0039] It should also be noted that the drone acrobatic strategy network can be deployed in the drone or in a control terminal associated with the drone. Preferably, the drone acrobatic strategy network can be deployed in the drone. Furthermore, when the drone attitude information of the drone at the current moment is obtained by the sensor set on the drone, the drone attitude information can be directly input into the drone acrobatic strategy network deployed in the drone, and the target drone control information corresponding to the drone at the current moment can be obtained. Furthermore, the drone can be directly controlled to execute the target drone control information to change the position and direction of the drone in space. The advantage of such a setting is that the control efficiency of the drone acrobatic flight is improved, and the strategy response is directly based on the drone without the help of the drone control terminal, which improves the response efficiency of the drone acrobatic flight, and achieves the effect of reducing labor costs while improving the prediction accuracy and efficiency of the acrobatic flight strategy.

[0040] In practical applications, the training of the drone acrobatic strategy network usually involves testing the physical drone so that the physical drone repeatedly completes the target acrobatic flight action many times, and in the process of completing the target acrobatic flight action, the corresponding drone attitude information and drone control information at each moment are collected as training sample data. Then, the pre-constructed neural network model is trained based on the constructed training sample data to obtain the drone acrobatic strategy network. However, the above-mentioned model training may result in a long training cycle and a complicated training process, which may lead to low training efficiency. In addition, there may be a small amount of training sample data, which may lead to poor performance of the trained drone acrobatic strategy network and low strategy prediction accuracy.

[0041] In view of the above situation, in this embodiment, a drone simulation model corresponding to the drone can be constructed. Then, the drone simulation model can be run to construct training sample data. Furthermore, the pre-constructed neural network model can be trained based on the constructed training sample data to obtain a drone stunt strategy network. The advantage of this setting is that different training scenarios and parameter settings can be simulated through the simulation model to find the optimal training strategy and resource allocation scheme. This helps to reduce unnecessary waste of computing resources and improve training efficiency; and it can speed up the training process and shorten the training time; as well as improve the generalization ability and robustness of the model and reduce the cost of model training.

[0042] As an optional implementation of the first embodiment of this invention, after the drone acrobatic strategy network is trained, the drone acrobatic strategy network can be deployed in the drone. Furthermore, when the drone attitude information corresponding to the drone at the current moment is obtained, the drone attitude information can be input into the drone acrobatic strategy network deployed in the drone. Furthermore, the drone attitude information can be processed based on the drone acrobatic strategy network, and the target drone control information corresponding to the drone at the current moment can be output.

[0043] S130, controlling the UAV to execute the target UAV control information, and obtaining the UAV attitude information of the UAV at the next moment after the current moment. When the UAV attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, repeating the steps of determining the target UAV control information, executing the target UAV control information and obtaining the UAV attitude information until the UAV is controlled to complete the target aerobatic action.

[0044] Among them, the preset end posture information may be predetermined drone posture information corresponding to the completion of the aerobatic action. Exemplarily, assuming that the target aerobatic action is a somersault action, the corresponding preset end posture information may be the drone posture information corresponding to the drone when the drone completes the somersault action; assuming that the target aerobatic action is a half-somersault roll action, the corresponding preset end posture information may be the drone posture information corresponding to the drone when the drone completes the half-somersault roll action; assuming that the target aerobatic action is an S-breaking action, the corresponding preset end posture information may be the drone posture information corresponding to the drone when the drone completes the S-breaking action. The determination standard for the drone to complete the target aerobatic action may be to determine that the drone has completed the target aerobatic action when the acquired drone posture information reaches the preset end posture information.

[0045] In this embodiment, when the target drone control information is obtained, the drone can be controlled to execute the target drone control information. Furthermore, the drone attitude information corresponding to the drone at the next moment of the current moment can be determined according to the drone attitude information corresponding to the current moment and the target drone control information corresponding to the current moment, that is, when the drone executes the target drone control information corresponding to the current moment, the drone attitude information corresponding to the current moment can be adjusted, and then the adjusted drone attitude information can be used as the drone attitude information corresponding to the drone at the next moment of the current moment.

[0046] In the case of obtaining the drone attitude information of the drone at the next moment, the drone attitude information at the next moment can be compared with the preset end attitude information corresponding to the target aerobatic action. Further, in the case of determining that the acquired drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, the drone attitude information at the next moment can be input into the drone aerobatic strategy network, and the target drone control information corresponding to the drone at the next moment can be obtained. Further, the drone can be controlled to execute the target drone control information to adjust the corresponding drone attitude information at the next moment, and obtain the corresponding drone attitude information at the next moment of the next moment. Further, the steps of determining the target drone control information, executing the target drone control information, and obtaining the drone attitude information are repeated until the acquired drone attitude information reaches the preset end attitude information and the drone completes the target aerobatic action.

[0047] It should be noted that in order to improve the model performance of the drone stunt strategy network in the application process of the drone stunt strategy network, and improve the accuracy of the drone stunt strategy network in strategy prediction. In the process of applying the drone stunt strategy network based on the drone, the drone stunt strategy network can also be optimized. Optionally, the method also includes: deploying the drone stunt strategy network in the drone, and updating the model parameters of the drone stunt strategy network based on the drone attitude information, the target drone control information and the response results.

[0048] The response result may be a user's feedback on the drone control information. Generally, the accuracy of the target drone control information output by the drone acrobatic strategy network may be determined based on the response result.

[0049] As an optional implementation of the first embodiment of this invention, when a drone stunt strategy network is obtained, the drone stunt strategy network can be deployed in a drone. Furthermore, when the drone attitude information is obtained based on the drone, the drone attitude information can be processed based on the drone stunt strategy network, the target drone control information corresponding to the drone attitude information can be determined, and the user's response result to the target drone control information feedback can be received. Furthermore, the model parameters of the drone stunt strategy network can be corrected according to the drone attitude information, the target drone control information and the response result. Thus, the model performance can be continuously optimized during the network application process of the drone stunt strategy network.

[0050] The technical solution of the embodiment of the present invention obtains the drone attitude information corresponding to the drone at the current moment; further, the drone attitude information is input into the drone stunt strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; further, the drone is controlled to execute the target drone control information, and the drone attitude information of the drone at the next moment of the current moment is obtained. When the drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic action, thereby solving the technical problems in the related technologies that have high professional requirements and are time-consuming and laborious, and realizing the effect of controlling the drone to autonomously complete aerobatic actions based on the strategy network, thereby improving the intelligence of the drone aerobatic flight, and the strategy network obtained by reinforcement learning can realize more accurate drone attitude control, and the application of reinforcement learning algorithm to drone aerobatic control can realize efficient aerobatic strategy with high accuracy and generalization.

[0051] Embodiment 2

[0052] Figure 2 This is a flow chart of a method for controlling aerobatic flight of a drone provided in the second embodiment of the present invention. Based on the above embodiment, before applying the drone aerobatic strategy network, this embodiment can first obtain training samples, and train the second deep network model based on the training samples to obtain a trained drone aerobatic strategy network. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or similar to the above embodiments are not repeated here.

[0053] like Figure 2 As shown, the method includes:

[0054] S210. Extract multiple training samples from a pre-built experience replay pool, wherein the training samples include the first simulation posture information at a historical moment, the first UAV control information and its corresponding probability information, advantage parameters, and expected value parameters.

[0055] Among them, the experience replay pool can be a data structure for storing the interaction experience between the agent and the environment, and is a replay buffer for storing samples of the interaction between the agent and the environment. The samples stored in the experience replay pool can be used to train the policy network. In this embodiment, a plurality of training samples are stored in the experience replay pool, and the training samples are constructed based on the simulation interaction data between the drone simulation model corresponding to the drone and the first deep network model. Furthermore, the drone simulation model can be used as the agent, and the first deep network model can be used as the environment where the agent is located. The first deep network model can be a neural network model whose model parameters are initial values ​​or default values. The training samples include the first simulation posture information at a historical moment, the first drone control information at a historical moment, the probability information corresponding to the first drone control information, the advantage parameter at a historical moment, and the expected value parameter at a historical moment. The first simulation posture information at a historical moment can be the drone posture information corresponding to the drone simulation model at a historical moment. The first simulation posture information is determined based on the drone simulation model corresponding to the drone. The first drone control information at a historical moment may be the drone control information output after the drone attitude information corresponding to the historical moment is input into a pre-built neural network model, and while outputting the drone control information, the probability information corresponding to the drone control information is output, and the output probability information may be used as the probability information corresponding to the first drone control information. The first drone control information is used to control the drone to perform a target aerobatic action. The advantage parameter may be used to characterize how much advantage, on average, it is to perform a certain action in a given state compared to performing any action according to the current strategy. The advantage parameter corresponding to the historical moment may be used to characterize how much advantage it is to perform the first drone control information under the drone attitude information at the historical moment compared to performing any action according to the current strategy. The advantage parameter may be determined based on a pre-set advantage function. The expected value parameter may be a parameter obtained by pre-determined value evaluation of the drone attitude information at the historical moment. The expected value parameter may be the value score that the expected drone attitude information can achieve.

[0056] In this embodiment, a drone simulation model corresponding to the drone can be constructed, and a first deep network model can be constructed. Furthermore, a simulated interaction can be performed through the drone simulation model and the first deep network model, and simulated interaction data corresponding to at least one simulated interaction round can be obtained. Furthermore, a training sample can be constructed based on the simulated interaction data, and the constructed training sample can be placed in the experience replay pool. Furthermore, in the model training stage, multiple training samples can be extracted from the experience replay pool to perform model training based on the training samples.

[0057] Optionally, before extracting multiple training samples from a pre-constructed experience replay pool, it also includes: obtaining simulation interaction data corresponding to the simulation interaction round; determining the advantage parameter and expected value parameter corresponding to at least one historical moment based on the simulation interaction data; for at least one historical moment, constructing a training sample based on the first simulation posture information, first UAV control information, probability information of the first UAV control information, advantage parameter and expected value parameter corresponding to the historical moment; and storing the at least one constructed training sample in the experience replay pool.

[0058] Among them, the simulation interaction round can be understood as an interaction process between the drone simulation model and the first deep network model starting from an initial state to interact until a terminal state corresponding to the initial state is obtained. The simulation interaction data can be understood as the data obtained during the interaction process. The simulation interaction data includes a simulation posture information queue, a drone control information queue, a reward parameter queue and a probability information queue. The simulation posture information queue includes the first simulation posture information corresponding to at least one historical moment. The drone control information queue includes the first drone control information corresponding to at least one historical moment. The reward parameter queue includes the reward parameter corresponding to at least one historical moment. The probability information queue includes the probability information of the first drone control information corresponding to at least one historical moment. The reward parameter can be understood as the evaluation or reward made for the first simulation posture information at a historical moment and its corresponding first drone control information. In general, the first simulation posture information at a historical moment and its corresponding first drone control information are input into a predetermined reward function, and the output obtained is the reward parameter at the historical moment.

[0059] In this embodiment, a drone simulation model corresponding to the drone can be constructed, and a first deep network model can be constructed. Further, a simulation interaction can be performed through the drone simulation model and the first deep network model, and simulation interaction data corresponding to at least one simulation interaction round can be obtained.

[0060] Optionally, obtaining simulation interaction data corresponding to the simulation interaction round includes: for at least one simulation interaction round: calling a pre-built UAV simulation model and a pre-built first deep network model; wherein the first deep network model includes at least a policy network; determining the UAV attitude parameters according to the target aerobatic action to be performed, and obtaining the first simulation attitude information at the initial moment based on the UAV attitude parameters, and inputting the first simulation attitude information into the policy network in the first deep network model to obtain the first actual control strategy at the initial moment; wherein the first actual control strategy includes at least one first candidate UAV control information and its corresponding probability information; determining the first UAV control information corresponding to the initial moment according to the at least one first candidate UAV control information and its corresponding probability information, and inputting the first UAV control information into the UAV simulation model to obtain the first UAV control information at the initial moment. The first simulation posture information corresponding to the next moment at the moment; the first simulation posture information corresponding to the next moment and the expected posture information are processed according to a preset reward and punishment function to obtain a reward parameter corresponding to the initial moment; the steps of determining the first actual control strategy, determining the first UAV control information, obtaining the first simulation posture information and determining the reward function are repeated until the preset round end condition is met; a simulation posture information queue is constructed based on at least one first simulation posture information obtained, and a UAV control information queue is constructed based on at least one first UAV control information obtained, and a reward parameter queue is constructed based on at least one reward parameter obtained, and a probability information queue is constructed based on at least one probability information obtained; the simulation posture information queue, the UAV control information queue, the reward parameter queue and the probability information queue are used as simulation interaction data corresponding to the simulation interaction round.

[0061] Among them, the drone attitude parameter may be an attitude parameter required to be detected to complete the target aerobatic action. Optionally, the drone attitude parameter may include at least one of the drone yaw angle, the drone pitch angle, the drone yaw angle, the drone sideslip angle, the attitude quaternion, the flight speed, the flight angular velocity, the drone angle of attack, and the drone overload. The first simulation attitude information at the initial moment may be the preset initial attitude information or the default attitude information. Exemplarily, the first simulation attitude information at the initial moment may be zero. The preset round end condition may be a pre-set condition for determining whether the current simulation interaction round can be ended. Optionally, the preset round end condition includes that the first drone control information obtained does not reach the expected attitude target, or that the current number of iterations has reached a preset number threshold, etc. Exemplarily, the expected attitude target may be a 45° flip every 3 seconds. It should be noted that at least one historical moment included in the simulation interaction data includes the initial moment.

[0062] It should be noted that aerobatic maneuvers are usually mainly related to the attitude angle of the drone. When performing aerobatic maneuvers, the drone's attitude changes greatly, which is likely to cause a sudden change in the attitude angle. This brings great difficulties to the design of reward and punishment functions. In order to better adapt to aerobatic tasks, in this embodiment, quaternions can be used as reward and punishment functions to help the drone simulation model learn to achieve the corresponding attitude. In addition, considering the tolerable range of real drones, some constraints are generally added, for example, limiting the size of the drone's sideslip angle (if it exceeds the preset sideslip angle threshold, a penalty is given) to prevent the drone from exceeding the tolerable range of the drone in order to achieve the expected attitude target in actual applications.

[0063] Optionally, the first simulated attitude information includes an attitude quaternion and a sideslip angle of the UAV; the first simulated attitude information corresponding to the next moment and the expected attitude information are processed according to a preset reward and punishment function to obtain a reward parameter corresponding to the initial moment, including: determining the quaternion corresponding to the expected attitude information to obtain the expected attitude quaternion; performing differential processing on the attitude quaternion and the expected attitude quaternion to obtain a differential quaternion; determining the rotation angle corresponding to the attitude quaternion according to the differential quaternion; processing the rotation angle, a preset rotation angle threshold, the UAV sideslip angle and a preset sideslip angle range according to a preset reward and punishment function to obtain a reward parameter corresponding to the initial moment.

[0064] Among them, the attitude quaternion is a four-dimensional complex number, usually composed of four real numbers (q0, q1, q2, q3), which can be regarded as a row vector q = [q0 q1 q2 q3]. In attitude solution, attitude quaternions are usually used to represent and update the attitude of an object. The sideslip angle of a drone refers to the angle between the flight velocity vector (or incoming flow direction) of the drone and the longitudinal symmetry plane of the fuselage during flight. The sideslip angle of a drone reflects the horizontal lateral tilt angle of the drone relative to the flight direction. The rotation angle can be used to characterize the degree of difference between the attitude information of the drone and the expected attitude information. The preset reward and punishment function may correspond to the target aerobatic action to be performed. The preset rotation angle threshold may be the minimum rotation angle used to determine whether the degree of difference between the attitude information of the drone and the expected attitude information satisfies the preset attitude difference target. The preset sideslip angle range may be a predetermined variable range of the sideslip angle. The preset sideslip angle range includes at least the maximum sideslip angle. Generally speaking, the closer the rotation angle is to the preset rotation angle threshold, the larger the reward value will be; when the sideslip angle of the drone exceeds the preset sideslip angle range, a penalty value will be added.

[0065] As an optional implementation of the first embodiment of the present invention, the expected attitude information can be converted into a quaternion form to obtain the expected attitude quaternion. Further, the expected attitude quaternion and the attitude quaternion can be differentially processed to obtain a differential quaternion. Further, the absolute value of the real part of the differential quaternion can be determined, and the arccosine value of the absolute value can be determined. Further, the obtained arccosine value can be multiplied by a preset value, and the obtained product can be used as the rotation angle corresponding to the attitude quaternion. Further, the rotation angle, the preset rotation angle threshold, the UAV sideslip angle, and the preset sideslip angle range can be processed according to a preset reward and punishment function to obtain a reward parameter corresponding to the initial moment.

[0066] As an optional implementation of this embodiment, a drone simulation model corresponding to multiple types of drones can be pre-constructed, and the drone identification and the drone simulation model can be associated and stored. Further, after determining the drone to be executed for the target aerobatic action, the drone simulation model corresponding to it can be retrieved according to the drone identification of the drone, and the pre-constructed first deep network model can be retrieved, and the first deep network model at least includes a policy network. Further, the drone attitude parameters corresponding to the target aerobatic action to be executed can be determined, and the drone attitude parameters can be assigned values ​​to serve as the first simulation attitude information at the initial moment. Further, the first simulation attitude information at the initial moment can be input into the policy network in the first deep network model, and the first actual control strategy at the initial moment is output, and the first actual control strategy includes at least one first candidate drone control information and probability information corresponding to each first candidate drone control information. Further, at least one first candidate drone control information and its corresponding probability information can be processed according to a preset strategy selection method to select a first candidate drone control information from at least one first candidate drone control information, and the selected first candidate drone control information is used as the first drone control information at the initial moment. Further, the first drone control information at the initial moment can be input into the drone simulation model to obtain the first simulation posture information at the next moment of the initial moment. Further, the first simulation posture information and the expected posture information at the next moment can be processed according to the preset reward and punishment function to obtain the reward parameter at the initial moment. Further, the steps of determining the first actual control strategy, determining the first drone control information, obtaining the first simulation posture information and determining the reward parameter are repeatedly executed until the preset round end condition is met, and the current simulation interaction round is ended. Further, a simulation posture information queue can be constructed based on at least one first simulation posture information obtained, and a drone control information queue can be constructed based on at least one first drone control information obtained, and a reward parameter queue can be constructed based on at least one reward parameter obtained, and a probability information queue can be constructed based on at least one probability information obtained. Further, the simulation posture information queue, the drone control information queue, the reward parameter queue and the probability information queue can be used as the simulation interaction data corresponding to the current simulation interaction round.

[0067] In the case of obtaining simulation interaction data, the simulation attitude information queue, the drone control information queue and the reward parameter queue in the simulation interaction data can be processed according to a preset advantage function to determine the advantage parameter corresponding to at least one historical moment, and the simulation attitude information queue, the drone control information queue and the reward parameter queue in the simulation interaction data can be processed according to a preset value function to obtain the expected value parameter corresponding to at least one historical moment. Further, for at least one historical moment, a training sample is constructed based on the first simulation attitude information corresponding to the historical moment, the first drone control information, the probability information corresponding to the first drone control information, the advantage parameter and the expected value parameter, and the at least one constructed training sample is stored in the experience replay pool.

[0068] Furthermore, in the reinforcement learning process, multiple training samples can be retrieved from the experience replay pool to train the policy network based on the multiple training samples.

[0069] S220. For multiple training samples, the first simulation posture information of the historical moments in the training samples are respectively input into the value network and the strategy network in the second deep network model to obtain the actual value parameters corresponding to the value network and the second actual control strategy corresponding to the strategy network.

[0070] Among them, the second deep network model can be a neural network model in the model training process or the reinforcement learning process. The second deep network model has the same model structure as the first deep network model, and the basis for correcting the model parameters is inconsistent. In other words, the first deep network model is a neural network model used to simulate the simulation environment model and to interact with the drone simulation model, and is a neural network model in the interaction process; the basis for correcting the model parameters of the first deep network model is the second deep network model, that is, the model parameters in the first deep network model are synchronized according to the model parameters in the second deep network model at every preset period, thereby realizing distributed training. The second deep network model is applied to the reinforcement learning stage, that is, the second deep network model is trained based on the reinforcement learning algorithm to obtain a trained drone stunt strategy network; the basis for correcting the model parameters of the second deep network model is the loss value. The second deep network model includes a value network and a strategy network. The value network can be a deep neural network that takes the drone attitude information at a historical moment as an input object to evaluate the drone attitude information at a historical moment. The value network can include a value function. The input of the value network can be the drone attitude information at a historical moment, and the output can be the value obtained after evaluating the drone attitude information at a historical moment. The policy network can be understood as a neural network model that takes the drone posture information at a historical moment as an input object to determine the control information of the second drone at a historical moment based on the input object.

[0071] In this embodiment, when multiple training samples are extracted from the experience replay pool, for the multiple training samples, the first simulation posture information of the historical moments in the training samples can be respectively input into the value network and the policy network in the second deep network model. Then, the input first simulation posture information can be processed based on the value network, and the actual value parameters corresponding to the value network can be output; and the input first simulation posture information can be processed based on the policy network, and the second actual control policy corresponding to the policy network can be output.

[0072] Among them, the actual value parameter can be a value parameter obtained after the value network evaluates the first simulation posture information at the historical moment. The second control strategy can be a drone control strategy determined by the strategy network in the second deep network model based on the first simulation posture information at the historical moment. The second actual control strategy includes at least one second candidate drone control information and its corresponding probability information. The second candidate drone control information can be understood as optional drone control information.

[0073] S230. Modify the model parameters in the value network according to the actual value parameters and the expected value parameters.

[0074] In this embodiment, when the actual value parameters output by the value network are obtained, the model parameters of the value network can be corrected according to the actual value parameters and the expected value parameters.

[0075] Optionally, the model parameters in the value network are modified according to the actual value parameters and the expected value parameters, including: performing loss processing on the actual value parameters and the expected value parameters according to a preset loss function to obtain a first loss value; and modifying the model parameters in the value network based on the first loss value.

[0076] The loss function may be a pre-set function used to characterize the difference between the actual output and the expected output of the value network. In this embodiment, the loss function may be a mean square error loss function.

[0077] As an optional implementation of this embodiment, the actual value parameter and the expected value parameter can be processed according to a preset loss function, and the obtained loss value can be used as the first loss value. Further, the model parameters in the value network can be modified according to the first loss value.

[0078] S240, modifying the model parameters in the strategy network according to the second actual control strategy, the first UAV control information and its corresponding probability information, and the advantage parameter, until the training ends when the preset convergence condition corresponding to the strategy network is reached, and obtaining the UAV stunt strategy network corresponding to the target aerobatic action.

[0079] It should be noted that there is no time-series execution order for S230 and S240 , and these two steps can be executed in parallel.

[0080] The preset convergence condition may be a preset end condition of the strategy network training process. Optionally, the preset convergence condition may include that the training error is less than a preset error, the error change trend tends to be stable, or the current training iteration number reaches a preset number.

[0081] In this embodiment, after obtaining the second actual control strategy, the loss value can be determined according to the second actual control strategy, the first drone control information, the probability information corresponding to the first drone control information, and the advantage parameter. Furthermore, the model parameters in the policy network in the second deep network model can be modified according to the loss value.

[0082] Optionally, the second actual control strategy includes at least one second candidate UAV control information and its corresponding probability information; the model parameters in the strategy network are modified according to the second actual control strategy, the first UAV control information and its corresponding probability information, and the advantage parameter, including: determining the second UAV control information consistent with the first UAV control information from at least one second candidate UAV control information; determining the ratio between the probability information corresponding to the second UAV control information and the probability information corresponding to the first UAV control information to obtain a first numerical value; determining the product between the first numerical value and the advantage parameter to obtain a second loss value, and modifying the model parameters in the strategy network according to the second loss value.

[0083] As an optional implementation of this embodiment, when the second actual control strategy is obtained, the obtained second actual control strategy includes at least one second candidate drone control information and probability information corresponding to each second candidate drone control information. Further, the second drone control information consistent with the first drone control information can be determined from at least one second candidate drone control information according to the first drone control information. Further, the ratio between the probability information corresponding to the second drone control information and the probability information corresponding to the first drone control information can be determined, and the obtained ratio is used as the first numerical value. Further, the product between the first numerical value and the advantage parameter can be determined, and the obtained product is used as the second loss value. Further, the model parameters in the strategy network can be corrected according to the second loss value.

[0084] For example, assuming that the first drone control information is decision action 2, the four second candidate drone control information included in the second actual control strategy are decision action 1, decision action 2, decision action 3, and decision action 4. The second drone control information selected from the four second candidate drone control information according to the first drone control information is decision action 2.

[0085] After obtaining the second loss value, the model parameters in the policy network can be corrected according to the second loss value. Specifically, the training error of the loss function in the policy network, that is, the loss parameter, can be used as a condition for detecting whether the current loss function has reached convergence, for example, whether the training error is less than the preset error or whether the error change trend tends to be stable, or whether the current number of model iterations is equal to the preset number, etc. If the detection reaches the convergence condition, for example, the training error of the loss function is less than the preset error or the error change tends to be stable, it indicates that the current policy network training is completed, and the iterative training can be stopped at this time. If it is detected that the convergence condition is not reached at present, the current first training sample can be further obtained to train the policy network until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the policy network obtained by the current training can be used as the drone stunt policy network.

[0086] S250: Obtain the drone attitude information corresponding to the drone at the current moment.

[0087] S260: Input the drone attitude information into the drone aerobatic strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment.

[0088] S270, control the UAV to execute the target UAV control information, and obtain the UAV attitude information of the UAV at the next moment after the current moment. When the UAV attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, repeat the steps of determining the target UAV control information, executing the target UAV control information and obtaining the UAV attitude information until the UAV is controlled to complete the target aerobatic action.

[0089] The technical solution of the embodiment of the present invention extracts multiple training samples from a pre-constructed experience replay pool, wherein the training samples include the first simulation posture information at the historical moment, the first UAV control information and its corresponding probability information, advantage parameters and expected value parameters; the first UAV control information is used to control the UAV to perform the target aerobatic action; the training samples are constructed based on the simulation interaction data between the UAV simulation model corresponding to the UAV and the first deep network model; further, for multiple training samples, the first simulation posture information at the historical moment in the training samples is respectively input into the value network and the strategy network in the second deep network model to obtain the actual value parameters corresponding to the value network and the second actual control strategy corresponding to the strategy network; further, the model parameters in the value network are corrected according to the actual value parameters and the expected value parameters; and the model parameters in the strategy network are corrected according to the second actual control strategy, the first UAV control information and its corresponding probability information and advantage parameters until the training is terminated when the preset convergence condition corresponding to the strategy network is reached, and the UAV stunt strategy network corresponding to the target aerobatic action is obtained, thereby achieving the effect of improving the training effect and training speed of the strategy network, and the UAV stunt strategy network can be trained more accurately and conveniently, thereby improving the decision-making performance of the UAV stunt strategy network.

[0090] Embodiment 3

[0091] Figure 3 Schematic diagram of the structure of a UAV aerobatic flight control device provided by the third embodiment of the present invention. Figure 3 As shown, the device includes: a posture information acquisition module 310, a control information determination module 320 and a special effect flight action completion module 330.

[0092] Among them, the attitude information acquisition module 310 is used to obtain the attitude information of the drone corresponding to the current moment; wherein the attitude information of the drone is used to characterize the motion attitude of the drone; the attitude information of the drone corresponds to the target aerobatic action to be performed; the control information determination module 320 is used to input the attitude information of the drone into the drone aerobatic strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; wherein the drone aerobatic strategy network is obtained by training the second deep network model based on the reinforcement learning algorithm and the drone simulation model corresponding to the drone; the special effect flight action completion module 330 is used to control the drone to execute the target drone control information, and obtain the drone attitude information of the drone at the next moment of the current moment. When the attitude information of the drone does not reach the preset end attitude information corresponding to the target aerobatic action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic action.

[0093] The technical solution of the embodiment of the present invention obtains the drone attitude information corresponding to the drone at the current moment; further, the drone attitude information is input into the drone stunt strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; further, the drone is controlled to execute the target drone control information, and the drone attitude information of the drone at the next moment of the current moment is obtained. When the drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic action, thereby solving the technical problems in the related technologies that have high professional requirements and are time-consuming and laborious, and realizing the effect of controlling the drone to autonomously complete aerobatic actions based on the strategy network, thereby improving the intelligence of the drone aerobatic flight, and the strategy network obtained by reinforcement learning can realize more accurate drone attitude control, and the application of reinforcement learning algorithm to drone aerobatic control can realize efficient aerobatic strategy with high accuracy and generalization.

[0094] Optionally, the device further includes: a training sample extraction module, a second control strategy determination module, a model parameter correction module and a stunt strategy network determination module. Among them, the training sample extraction module is used to extract multiple training samples from a pre-constructed experience replay pool, wherein the training samples include the first simulation posture information at the historical moment, the first drone control information and its corresponding probability information, advantage parameters and expected value parameters; the first simulation posture information is determined based on the drone simulation model corresponding to the drone; the training samples are constructed based on the simulation interaction data between the drone simulation model corresponding to the drone and the first deep network model; the second control strategy determination module is used to input the first simulation posture information at the historical moment in the training sample into the value network and the strategy network in the second deep network model for multiple training samples, respectively, to obtain the actual value parameters corresponding to the value network and the second actual control strategy corresponding to the strategy network; the model parameter correction module is used to correct the model parameters in the value network according to the actual value parameters and the expected value parameters; and the stunt strategy network determination module is used to correct the model parameters in the strategy network according to the second actual control strategy, the first drone control information and its corresponding probability information and advantage parameters, until the training is terminated when the preset convergence condition corresponding to the strategy network is reached, and the drone stunt strategy network corresponding to the target stunt flight action is obtained.

[0095] Optionally, the device further includes: an interactive data acquisition module, a parameter determination module, a training sample construction module and a sample storage module. The interactive data acquisition module is used to acquire the simulation interaction data corresponding to the simulation interaction round; wherein the simulation interaction data includes a simulation posture information queue, a drone control information queue, a reward parameter queue and a probability information queue; the simulation posture information queue includes the first simulation posture information corresponding to at least one historical moment; the drone control information queue includes the first drone control information corresponding to at least one historical moment; the reward parameter queue includes the reward parameter corresponding to at least one historical moment; the probability information queue includes the probability information of the first drone control information corresponding to at least one historical moment; the parameter determination module is used to determine the advantage parameter and expected value parameter corresponding to at least one historical moment according to the simulation interaction data; the training sample construction module is used to construct a training sample for at least one historical moment based on the first simulation posture information, the first drone control information, the probability information, the advantage parameter and the expected value parameter corresponding to the historical moment; the sample storage module is used to store at least one of the constructed training samples in the experience replay pool.

[0096] Optionally, the interactive data acquisition module includes: a model retrieval unit, a first control strategy determination unit, a first control information determination unit, a reward parameter determination unit, a step repetition execution unit, a queue construction unit and an interactive data determination unit. Among them, for at least one simulation interaction round: a model retrieval unit, used to retrieve a pre-built UAV simulation model and a pre-built first deep network model; wherein the first deep network model at least includes a policy network; a first control strategy determination unit, used to determine the UAV attitude parameters according to the target aerobatic action to be performed, and obtain the first simulation attitude information at the initial moment based on the UAV attitude parameters, and input the first simulation attitude information into the policy network in the first deep network model to obtain the first actual control strategy at the initial moment; wherein the first actual control strategy includes at least one first candidate UAV control information and its corresponding probability information; a first control information determination unit, used to determine the first UAV control information corresponding to the initial moment according to at least one first candidate UAV control information and its corresponding probability information, and input the first UAV control information into the UAV simulation model to obtain the first simulation attitude information corresponding to the next moment of the initial moment; a reward A reward parameter determination unit, used to process the first simulation posture information and expected posture information corresponding to the next moment according to a preset reward and punishment function to obtain the reward parameter corresponding to the initial moment; a step repetition execution unit, used to repeatedly execute the steps of determining the first actual control strategy, determining the first drone control information, obtaining the first simulation posture information and determining the reward function until the preset round end condition is met; a queue construction unit, used to construct a simulation posture information queue based on at least one of the first simulation posture information obtained, and to construct a drone control information queue based on at least one of the first drone control information obtained, and to construct a reward parameter queue based on at least one of the reward parameters obtained, and to construct a probability information queue based on at least one of the probability information obtained; an interaction data determination unit, used to use the simulation posture information queue, the drone control information queue, the reward parameter queue and the probability information queue as the simulation interaction data corresponding to the simulation interaction round.

[0097] Optionally, the first simulation posture information includes posture quaternion and UAV sideslip angle; the reward parameter determination unit includes: a quaternion determination subunit, a differential processing subunit, a rotation angle determination subunit and a reward parameter determination subunit. Among them, the quaternion determination subunit is used to determine the quaternion corresponding to the expected posture information to obtain the expected posture quaternion; the differential processing subunit is used to perform differential processing on the posture quaternion and the expected posture quaternion to obtain the differential quaternion; the rotation angle determination subunit is used to determine the rotation angle corresponding to the posture quaternion according to the differential quaternion; the reward parameter determination subunit is used to process the rotation angle, the preset rotation angle threshold, the UAV sideslip angle and the preset sideslip angle range according to a preset reward and punishment function to obtain the reward parameter corresponding to the initial moment.

[0098] Optionally, the model parameter correction module includes: a first loss value determination unit and a model parameter correction unit. The first loss value determination unit is used to perform loss processing on the actual value parameter and the expected value parameter according to a preset loss function to obtain a first loss value; and the model parameter correction unit is used to correct the model parameters in the value network based on the first loss value.

[0099] Optionally, the second actual control strategy includes at least one second candidate UAV control information and its corresponding probability information; the stunt strategy network determination module includes: a second UAV control information determination unit, a probability ratio determination unit and a second loss value determination unit. Among them, the second UAV control information determination unit is used to determine the second UAV control information consistent with the first UAV control information from at least one of the second candidate UAV control information; the probability ratio determination unit is used to determine the ratio between the probability information corresponding to the second UAV control information and the probability information corresponding to the first UAV control information to obtain a first value; the second loss value determination unit is used to determine the product between the first value and the advantage parameter to obtain a second loss value, and modify the model parameters in the strategy network according to the second loss value.

[0100] Optionally, the device further comprises: a model parameter updating module. The model parameter updating module is used to deploy the drone acrobatic strategy network in the drone and update the model parameters of the drone acrobatic strategy network based on the drone attitude information, the target drone control information and the response result.

[0101] The unmanned aerial vehicle aerobatic flight control device provided in the embodiment of the present invention can execute the unmanned aerial vehicle aerobatic flight control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0102] Embodiment 4

[0103] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0104] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0105] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a drone aerobatic flight control method.

[0107] In some embodiments, the drone aerobatic control method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the drone aerobatic control method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the drone aerobatic control method in any other appropriate manner (e.g., by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0110] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a target blockchain network, and the Internet.

[0113] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for controlling aerobatic flight of an unmanned aerial vehicle, characterized in that: Applied in drones, including: Acquire the drone attitude information corresponding to the drone at the current moment; wherein the drone attitude information is used to characterize the motion attitude of the drone; and the drone attitude information corresponds to the target aerobatic action to be performed; Inputting the drone attitude information into a drone aerobatic strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; wherein the drone aerobatic strategy network is obtained by training a second deep network model based on a reinforcement learning algorithm and a drone simulation model corresponding to the drone; The drone is controlled to execute the target drone control information, and the drone attitude information of the drone at the next moment after the current moment is obtained. When the drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic action.

2. The method for controlling aerobatic flight of a drone according to claim 1, characterized in that: Also includes: Extract multiple training samples from a pre-constructed experience replay pool, wherein the training samples include first simulation posture information at a historical moment, first drone control information and corresponding probability information, advantage parameters and expected value parameters; the first simulation posture information is determined based on a drone simulation model corresponding to the drone; the training samples are constructed based on simulation interaction data between the drone simulation model corresponding to the drone and the first deep network model; For the plurality of training samples, the first simulation posture information of the historical moments in the training samples are respectively input into the value network and the strategy network in the second deep network model to obtain the actual value parameters corresponding to the value network and the second actual control strategy corresponding to the strategy network; Modifying the model parameters in the value network according to the actual value parameter and the expected value parameter; and, The model parameters in the strategy network are modified according to the second actual control strategy, the first UAV control information and its corresponding probability information, and the advantage parameter, until the training ends when the preset convergence condition corresponding to the strategy network is reached, and a UAV stunt strategy network corresponding to the target aerobatic action is obtained.

3. The method for controlling the aerobatic flight of a drone according to claim 2, characterized in that: Also includes: Acquire simulation interaction data corresponding to the simulation interaction round; wherein the simulation interaction data includes a simulation posture information queue, a drone control information queue, a reward parameter queue and a probability information queue; the simulation posture information queue includes at least one first simulation posture information corresponding to a historical moment; the drone control information queue includes at least one first drone control information corresponding to a historical moment; the reward parameter queue includes at least one reward parameter corresponding to a historical moment; the probability information queue includes at least one probability information of the first drone control information corresponding to a historical moment; Determine an advantage parameter and an expected value parameter corresponding to at least one historical moment according to the simulation interaction data; For at least one of the historical moments, constructing a training sample based on the first simulation posture information, the first UAV control information, the probability information of the first UAV control information, the advantage parameter, and the expected value parameter corresponding to the historical moment; The at least one constructed training sample is stored in an experience replay pool.

4. The method for controlling the aerobatic flight of a drone according to claim 3, characterized in that: The obtaining of simulation interaction data corresponding to the simulation interaction round includes: For at least one simulation interaction round: Retrieving a pre-built UAV simulation model and a pre-built first deep network model; wherein the first deep network model at least includes a policy network; Determine the attitude parameters of the drone according to the target aerobatic action to be performed, obtain the first simulated attitude information at the initial moment based on the attitude parameters of the drone, and input the first simulated attitude information into the policy network in the first deep network model to obtain the first actual control strategy at the initial moment; wherein the first actual control strategy includes at least one first candidate drone control information and its corresponding probability information; Determine the first UAV control information corresponding to the initial moment according to at least one of the first candidate UAV control information and its corresponding probability information, and input the first UAV control information into the UAV simulation model to obtain the first simulation attitude information corresponding to the next moment of the initial moment; Processing the first simulation posture information and the expected posture information corresponding to the next moment according to a preset reward and punishment function to obtain a reward parameter corresponding to the initial moment; Repeat the steps of determining the first actual control strategy, determining the first drone control information, acquiring the first simulation posture information, and determining the reward parameter until a preset round end condition is met; Constructing a simulation posture information queue based on at least one of the first simulation posture information that has been obtained, and constructing a drone control information queue based on at least one of the first drone control information that has been obtained, and constructing a reward parameter queue based on at least one of the reward parameters that has been obtained, and constructing a probability information queue based on at least one of the probability information that has been obtained; The simulation attitude information queue, the drone control information queue, the reward parameter queue and the probability information queue are used as simulation interaction data corresponding to the simulation interaction round.

5. The method for controlling the aerobatic flight of a drone according to claim 4, characterized in that: The first simulated attitude information includes an attitude quaternion and a sideslip angle of the drone; the first simulated attitude information and the expected attitude information corresponding to the next moment are processed according to a preset reward and punishment function to obtain a reward parameter corresponding to the initial moment, including: Determine the quaternion corresponding to the expected posture information to obtain the expected posture quaternion; Performing differential processing on the attitude quaternion and the expected attitude quaternion to obtain a differential quaternion; Determining a rotation angle corresponding to the attitude quaternion according to the differential quaternion; The rotation angle, the preset rotation angle threshold, the UAV sideslip angle and the preset sideslip angle range are processed according to a preset reward and punishment function to obtain a reward parameter corresponding to the initial moment.

6. The method for controlling the aerobatic flight of a drone according to claim 2, characterized in that: The modifying of the model parameters in the value network according to the actual value parameter and the expected value parameter includes: Performing loss processing on the actual value parameter and the expected value parameter according to a preset loss function to obtain a first loss value; Model parameters in the value network are modified based on the first loss value.

7. The method for controlling the aerobatic flight of a drone according to claim 2, characterized in that: The second actual control strategy includes at least one second candidate UAV control information and its corresponding probability information; the model parameters in the strategy network are modified according to the second actual control strategy, the first UAV control information and its corresponding probability information and the advantage parameter, including: Determine second drone control information consistent with the first drone control information from at least one of the second candidate drone control information; Determine a ratio between probability information corresponding to the second UAV control information and probability information corresponding to the first UAV control information to obtain a first value; The product between the first value and the advantage parameter is determined to obtain a second loss value, and the model parameters in the policy network are modified according to the second loss value.

8. The method for controlling aerobatic flight of a drone according to claim 1, characterized in that: Also includes: The UAV stunt strategy network is deployed in the UAV, and the model parameters of the UAV stunt strategy network are updated based on the UAV attitude information, the target UAV control information and the response result.

9. A UAV aerobatic flight control device, characterized in that: Configured in the drone, including: The posture information acquisition module is used to acquire the posture information of the drone corresponding to the current moment; wherein the posture information of the drone is used to characterize the motion posture of the drone; and the posture information of the drone corresponds to the target aerobatic action to be performed; A control information determination module is used to input the drone attitude information into a drone aerobatic strategy network corresponding to the target aerobatic action to obtain the target drone control information corresponding to the drone at the current moment; wherein the drone aerobatic strategy network is obtained by training a second deep network model based on a reinforcement learning algorithm and a drone simulation model corresponding to the drone; The special effects flight action completion module is used to control the drone to execute the target drone control information and obtain the drone attitude information of the drone at the next moment after the current moment. When the drone attitude information does not reach the preset end attitude information corresponding to the target aerobatic flight action, the steps of determining the target drone control information, executing the target drone control information and obtaining the drone attitude information are repeated until the drone is controlled to complete the target aerobatic flight action.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the unmanned aerial vehicle aerobatic flight control method according to any one of claims 1 to 8.

Images