Unmanned aerial vehicle evasion strategy determination method and device based on reinforcement learning, equipment and medium

Abstract

The invention discloses an unmanned aerial vehicle evasion strategy determination method and device based on reinforcement learning, equipment and a medium, and relates to the technical field of unmanned aerial vehicle autonomous control, and the method comprises the steps: constructing an unmanned aerial vehicle six-degree-of-freedom model, an action library, and three-degree-of-freedom particle models of other aircrafts in a flight state at present, maneuvering characteristics of other aircrafts are restrained, and target tracking of other aircrafts is achieved according to the relative motion situation. Building a reinforcement learning environment and a model on a programming language platform, inputting the position attitude information of the unmanned aerial vehicle and other aircrafts into the model, and outputting a flight action for the unmanned aerial vehicle to avoid other aircrafts. And if the avoidance is successful, giving a reward value to the model based on maneuvering area situations of other aircrafts. According to the reward function based on maneuvering situation quantification of other aircrafts, the unmanned aerial vehicle can better perceive danger and action effects in training, the unmanned aerial vehicle is promoted to adopt more effective avoidance actions, and therefore the avoidance success rate when the unmanned aerial vehicle actually encounters other aircrafts is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous control of unmanned aerial vehicles, and in particular to a method, device, equipment and medium for determining an unmanned aerial vehicle avoidance strategy based on reinforcement learning. Background Art

[0002] In low-altitude logistics transportation, drones flying in densely intersecting routes face great threats from other aircraft, which may have the characteristics of high speed, strong maneuverability, and changeable routes, which bring many obstacles to drone flights. At present, as logistics transportation is transforming from manned to unmanned, and from manual operation to fully autonomous development, autonomous decision-making has become an inevitable trend in the development of the low-altitude economy.

[0003] Traditional methods for drones to avoid obstacles include expert system method, differential game method, optimal control method, etc. The expert system method relies on the prior knowledge of experts. Once the subsystems of other aircraft or aircraft change, the experts need to analyze the new subsystem and give a new avoidance maneuver strategy again; both differential game method and optimal control method rely on clear and complete mathematical models, and solving complex differential equations is essential. However, the problem of aircraft avoiding other aircraft involves multiple complex nonlinear systems, and errors are inevitable in modeling each subsystem, which greatly increases the difficulty of solving the problem by the above methods. In addition, most of the methods for drones to avoid other aircraft are completed under more restrictive conditions, and can only vaguely explain which maneuvers are more advantageous in a certain type of encounter situation, and the scalability is not strong. Therefore, the above problems urgently need to be solved by technical personnel in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for determining a drone avoidance strategy based on reinforcement learning, which can get rid of the dependence on expert prior knowledge, avoid the interference of modeling errors on avoidance decisions, promote the reinforcement learning model to optimize the avoidance strategy, form a virtuous circle, and steadily improve the avoidance success rate, greatly enhancing the scalability of the avoidance strategy. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a method for determining a drone avoidance strategy based on reinforcement learning, comprising:

[0006] Build a six-degree-of-freedom model of a UAV in the engineering computing simulation platform and establish a UAV motion library;

[0007] A three-degree-of-freedom particle model of other aircraft currently in flight is constructed in the engineering computing simulation platform, the maneuverability characteristics of other aircraft are constrained based on constraint rules, and the tracking of the UAV by other aircraft is achieved according to the relative motion situation; the relative motion situation is the relative motion situation between the other aircraft and the UAV;

[0008] Build a reinforcement learning environment in the programming language platform and establish a corresponding reinforcement learning model;

[0009] Inputting the position and attitude information of the UAV and the position and attitude information of the other aircraft into the reinforcement learning model, so as to output the flight actions in the UAV action library through the reinforcement learning model;

[0010] When the UAV avoids the other aircraft based on the flight action, it is judged whether the UAV avoids successfully based on the maneuvering characteristics of the other aircraft. If successful, a corresponding reward value is given to the reinforcement learning model based on the situation of the maneuvering area of ​​the other aircraft to encourage the reinforcement learning model to optimize the avoidance strategy.

[0011] Optionally, the other aircraft maneuverability characteristics include the flight time of other aircraft, the safety range of other aircraft, and the tolerable overload of other aircraft, and the constraining the maneuverability characteristics of other aircraft based on the constraint rules includes:

[0012] Setting the simulation time of other aircraft in the engineering calculation simulation platform so as to constrain the flight time of the other aircraft through the simulation time of the other aircraft;

[0013] The size of the simulation space is set in the engineering calculation simulation platform so as to constrain the safety range of the other aircraft through the size of the simulation space;

[0014] The overload that the other aircraft can withstand is controlled to be no greater than a preset overload threshold, so as to constrain the overload that the other aircraft can withstand; wherein the overload that the other aircraft can withstand includes tangential overload and normal overload.

[0015] Optionally, judging whether the UAV successfully avoids based on the maneuverability characteristics of the other aircraft includes:

[0016] Within the flight time of the other aircraft and within the tolerable overload of the other aircraft, determine whether the distance between the other aircraft and the drone is greater than a preset value. If so, determine that the drone has successfully avoided the obstacle.

[0017] Optionally, the situation of the other aircraft maneuvering area includes the distance between the other aircraft and the UAV, the relative speed between the other aircraft and the UAV, the collision probability of the other aircraft, and the situation level of the area where the other aircraft is located; wherein the situation level represents the importance of the area where the other aircraft is located to the UAV when avoiding it.

[0018] Optionally, giving the reinforcement learning model a corresponding reward value based on the situation of other aircraft in the maneuvering area to encourage the reinforcement learning model to optimize the avoidance strategy includes:

[0019] Based on the distance between the other aircraft and the UAV, a first reward value is given to the reinforcement learning model through a first reward function to encourage the reinforcement learning model to optimize the avoidance strategy; wherein, in the first reward function, the distance between the other aircraft and the UAV is negatively correlated with the reward value.

[0020] Optionally, giving the reinforcement learning model a corresponding reward value based on the situation of other aircraft in the maneuvering area to encourage the reinforcement learning model to optimize the avoidance strategy includes:

[0021] Based on the relative speed between the other aircraft and the UAV, a second reward value is given to the reinforcement learning model through a second reward function to encourage the reinforcement learning model to optimize the avoidance strategy; wherein, in the second reward function, the relative speed between the other aircraft and the UAV is negatively correlated with the reward value.

[0022] Optionally, giving the reinforcement learning model a corresponding reward value based on the situation of other aircraft in the maneuvering area to encourage the reinforcement learning model to optimize the avoidance strategy includes:

[0023] If the situation level of the area where the other aircraft is located is the first priority and the collision probability of the other aircraft is greater than the preset probability threshold, a third reward value is given to the reinforcement learning model to encourage the reinforcement learning model to optimize the avoidance strategy;

[0024] If the situation level of the area where the other aircraft is located is the second priority or the collision probability of the other aircraft is not greater than the preset probability threshold, giving the reinforcement learning model a fourth reward value to encourage the reinforcement learning model to optimize the avoidance strategy;

[0025] The first priority is higher than the second priority, and the third reward value is greater than the fourth reward value.

[0026] In a second aspect, the present application discloses a device for determining a drone avoidance strategy based on reinforcement learning, comprising:

[0027] The UAV model building module is used to build a six-degree-of-freedom model of the UAV in the engineering computing simulation platform and establish a UAV motion library;

[0028] Other aircraft model building module, used to build three-degree-of-freedom particle models of other aircraft currently in flight in the engineering calculation simulation platform;

[0029] Other aircraft characteristic constraint module, used for constraining the maneuvering characteristics of other aircraft based on constraint rules, and realizing tracking of the UAV by other aircraft according to relative motion situation; the relative motion situation is the relative motion situation between the other aircraft and the UAV;

[0030] The reinforcement learning model building module is used to build a reinforcement learning environment in the programming language platform and establish a corresponding reinforcement learning model;

[0031] A flight action output module, used for inputting the position and attitude information of the UAV and the position and attitude information of the other aircraft into the reinforcement learning model, so as to output the flight actions in the UAV action library through the reinforcement learning model;

[0032] The avoidance strategy optimization module is used to determine whether the UAV has successfully avoided the other aircraft based on the maneuvering characteristics of the other aircraft when the UAV avoids the other aircraft based on the flight action. If successful, a corresponding reward value is given to the reinforcement learning model based on the situation of the maneuvering area of ​​the other aircraft to encourage the reinforcement learning model to optimize the avoidance strategy.

[0033] In a third aspect, the present application discloses an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the aforementioned disclosed method for determining the drone avoidance strategy based on reinforcement learning.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for determining a drone avoidance strategy based on reinforcement learning.

[0037] It can be seen that the present application proposes a method for determining a UAV avoidance strategy based on reinforcement learning, including: building a six-degree-of-freedom model of a UAV in an engineering computing simulation platform, and establishing a UAV action library; building a three-degree-of-freedom particle model of other aircraft currently in flight in the engineering computing simulation platform, constraining the maneuverability characteristics of other aircraft based on constraint rules, and realizing tracking of the UAV by other aircraft according to the relative motion situation; building a reinforcement learning environment in a programming language platform and establishing a corresponding reinforcement learning model; inputting the position and posture information of the UAV and the position and posture information of other aircraft into the reinforcement learning model, so as to output the flight action in the UAV action library through the reinforcement learning model; when the UAV avoids other aircraft based on the flight action, judging whether the UAV avoids successfully based on the maneuverability characteristics of other aircraft, and if successful, giving the reinforcement learning model a corresponding reward value based on the maneuverability area situation of the other aircraft, so as to encourage the reinforcement learning model to optimize the avoidance strategy. It can be seen that this application builds a six-degree-of-freedom model of the drone and a three-degree-of-freedom particle model of other aircraft currently in flight on the engineering computing simulation platform, and combines the reinforcement learning environment and the reinforcement learning model to enable the drone to autonomously learn the response strategy. Even if other aircraft use a new guidance subsystem, the drone does not need to wait for expert analysis, but relies on its own accumulated learning results in the simulation environment to quickly select the appropriate flight action from the action library for avoidance, getting rid of the dependence on the expert's prior knowledge. Secondly, the reinforcement learning model in this application is not based on the calculation results of the theoretical model that may have errors, but is based on the response mode learned from similar scenarios in the past, and flexibly adjusts the flight action of the drone, effectively avoiding the interference of modeling errors on avoidance decisions. Furthermore, after the drone successfully avoids other aircraft, this application gives the reinforcement learning model a corresponding reward value based on the situation of the maneuvering area of ​​other aircraft. In this way, the reinforcement learning model is prompted to optimize the avoidance strategy and form a virtuous circle. Finally, this application uses the engineering computing simulation platform to flexibly adjust parameters to simulate the characteristics of a variety of different environments, so that drones can rely on the avoidance capabilities acquired through training in various complex and changing scenarios, steadily improve the avoidance success rate, and greatly enhance the scalability of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 This is a flow chart of a method for determining a drone avoidance strategy based on reinforcement learning disclosed in this application;

[0040] Figure 2 A schematic diagram of a basic framework of reinforcement learning disclosed in this application;

[0041] Figure 3 A schematic diagram of an interactive simulation process between another aircraft and a UAV disclosed in the present application;

[0042] Figure 4 This is a schematic diagram of the structure of a device for determining a drone avoidance strategy based on reinforcement learning disclosed in this application;

[0043] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Traditional methods for UAVs to avoid other aircraft include expert system method, differential game method, optimal control method, etc. The expert system method relies on the prior knowledge of experts. Once the subsystem of other aircraft or aircraft changes, the expert needs to analyze the new subsystem and give a new avoidance maneuver strategy again; differential game method and optimal control method both rely on clear and complete mathematical models, and solving complex differential equations is essential. However, the problem of aircraft avoiding other aircraft involves multiple complex nonlinear systems, and errors are inevitable in modeling each subsystem, which greatly increases the difficulty of solving the problem with the above methods. In addition, the method for UAVs to avoid other aircraft can usually only vaguely explain which maneuvers are more advantageous in a certain type of encounter situation, and the scalability is not strong.

[0046] To this end, the embodiment of the present application proposes a UAV avoidance strategy determination scheme based on reinforcement learning, which can get rid of the dependence on expert prior knowledge, avoid the interference of modeling errors on avoidance decisions, promote the reinforcement learning model to optimize the avoidance strategy, form a virtuous circle, and steadily improve the avoidance success rate, greatly enhancing the scalability of the avoidance strategy.

[0047] The present application embodiment discloses a method for determining a drone avoidance strategy based on reinforcement learning, see Figure 1 As shown, the method includes:

[0048] Step S11: Build a six-degree-of-freedom model of the UAV in the engineering computing simulation platform and establish a UAV motion library.

[0049] In this embodiment, the drone is first idealized and regarded as an ideal rigid body with bilateral symmetry, and it is clear that the control of the drone mainly depends on the engine thrust and aerodynamic control surfaces. Then, before establishing the motion equation, the following two conditions need to be met: (1) The drone is regarded as a rigid body, and the mass of the drone remains constant; the ground coordinate axis system is selected as the inertial coordinate system, on this basis, the curvature of the earth is ignored, and it is assumed that the gravitational acceleration does not change with the change of flight altitude; for drones with symmetrical layout, the coordinate axis system of the body is clearly defined. The plane is the symmetry plane of the drone, so that the geometry and mass distribution of the drone are centered around Plane symmetry; (2) Six degrees of freedom are used to describe the spatial motion of the drone. The spatial motion of the drone specifically includes linear motion (displacement of the center of mass) and angular motion (rotation around the center of mass). Linear motion includes the increase and decrease of flight speed, lifting and lowering, and lateral movement. Angular motion includes pitch angle motion, yaw angle motion, and roll angle motion. Finally, a six-degree-of-freedom model of the drone is built on the engineering computing simulation platform (Matlab) so that the drone's motion state in space can be accurately simulated and analyzed through the six-degree-of-freedom model of the drone.

[0050] Drone six-degree-of-freedom model:

[0051] ;

[0052] in, represents the speed of the drone, represents the track deviation angle, represents the track inclination, represents the climb rate, represents the velocity component of the UAV in the x-axis direction, represents the velocity component of the UAV in the y-axis direction, Represents the velocity component of the drone in the z-axis direction.

[0053] ;

[0054] ;

[0055] ;

[0056] in, represents the derivative of the drone's velocity, Indicates the quality of the drone, Indicates resistance, Indicates thrust, represents the angle of attack, represents the derivative of the angle of attack, represents the sideslip angle, represents the derivative of the sideslip angle, represents the acceleration due to gravity, represents the derivative of the track angle, represents lift, represents the velocity roll angle, represents the derivative of the velocity roll angle, represents the lateral force, represents the derivative of the track inclination, represents the pitch rate, represents the derivative of the pitch rate, represents the rolling rate, represents the derivative of the roll rate, is the yaw rate, represents the derivative of the yaw rate, , , Represents the components of the torque on the x, y, and z axes. Represents the angular momentum.

[0057] For the established UAV six-degree-of-freedom model, based on the damper and stabilization design, the control law of the longitudinal and lateral motion of the UAV is designed to obtain the outer loop control command. , Indicates speed command, Indicates altitude command. Indicates the aircraft roll command. By setting the command sequence, the drone can complete the following seven basic maneuvers: uniform straight flight, maximum force acceleration, maximum force deceleration, climb, dive, horizontal left turn, and horizontal right turn. It should be pointed out that the design of the avoidance maneuver library is usually divided into two categories: one is a typical maneuver library based on classic maneuvers, and the other is a basic control library based on commonly used maneuvering methods. This embodiment obtains the drone action library by setting the command sequence, so that the drone can complete the above seven maneuvers. In this way, it can be executed and switched at any time, which is flexible and simple, and is not limited by the completion time of the maneuver.

[0058] In this embodiment, the updating and switching of maneuvering instructions are realized based on the following logic: the updating of maneuvering instructions takes the step length of maneuvering decision as the cycle, so as to switch the maneuvering action instructions, wherein the maneuvering action determined by the maneuvering decision will last for a maneuvering decision step length, and the duration of the maneuvering action instruction corresponding to the maneuvering action is a command update cycle. In addition, the updating of maneuvering instructions has the characteristics of continuity. Each time the instruction value is updated, it is based on the speed, height, and roll angle instruction values ​​when the previous maneuvering action is completed, and the corresponding increase and decrease operations are performed on this basis. Secondly, considering the characteristics of the constructed drone model itself, its height and roll angle can change significantly within 5s, and this characteristic can ensure that a maneuvering action can be accurately executed. Therefore, in this embodiment, the time interval of the maneuvering decision based on the reinforcement learning algorithm is set to 5s. Finally, when it is necessary to switch the maneuvering action, the speed, height, and roll angle instruction values ​​should be adaptively changed synchronously. For example, when the aircraft completes the maximum boost climbing maneuver and starts to perform a uniform straight-line level maneuver, the speed command value should be set to the actual speed at the moment of switching from the climbing maneuver to the uniform straight-line motion, and the altitude command value should also be the altitude corresponding to that moment. In this way, the UAV flight state can be guaranteed to be stable, continuous and in line with the flight logic, thereby achieving a reasonable and smooth switching of the maneuvering command.

[0059] Step S12: Build a three-degree-of-freedom particle model of other aircraft currently in flight in the engineering calculation simulation platform, constrain the maneuverability characteristics of other aircraft based on constraint rules, and enable other aircraft to track the UAV according to the relative motion posture; the relative motion posture is the relative motion posture between the other aircraft and the UAV.

[0060] Build a reinforcement learning environment in the programming language platform and establish a corresponding reinforcement learning model.

[0061] In this embodiment, a simplified three-degree-of-freedom particle model of other aircraft is established in the engineering calculation simulation platform:

[0062] ;

[0063] in, Indicates tangential overload, Indicates normal overload, represents the track deviation angle, Represents the derivative of the track angle.

[0064] In this embodiment, the maneuverability characteristics of other aircraft include the flight time of other aircraft, the safety range of other aircraft and the overload that other aircraft can withstand. Furthermore, the maneuverability characteristics of other aircraft are constrained based on constraint rules, including: setting the simulation time of other aircraft in the engineering computing simulation platform so that the flight time of other aircraft can be constrained by the simulation time of other aircraft; setting the simulation space size in the engineering computing simulation platform so that the safety range of other aircraft can be constrained by the simulation space size; controlling the overload that other aircraft can withstand to be no more than a preset overload threshold so as to constrain the overload that other aircraft can withstand; wherein, the overload that other aircraft can withstand includes tangential overload and normal overload. For example, for the flight time of other aircraft, the simulation time of other aircraft is set to 10 seconds in the engineering calculation simulation platform, which means that the actual flight time of other aircraft also fluctuates around this magnitude, so as to constrain them. For the safety range of other aircraft, the simulation space size is set to a cube space with a length, width and height of 10 kilometers in the engineering calculation simulation platform, so that the safety range of other aircraft is set within this space. For the overload that other aircraft can withstand, the simulation space size is set to 10 seconds. ,in, Indicates tangential overload, Indicates normal overload, Indicates the preset tangential overload threshold, Indicates the preset normal overload threshold.

[0065] In this embodiment, the tracking of the UAV by other aircraft is achieved according to the relative motion situation, and the relative motion situation is the relative motion situation between the other aircraft and the UAV. The relative motion model used to describe the relative motion situation is as follows:

[0066] ;

[0067] in, represents the time derivative of the distance between other aircraft and the UAV, Indicates the relative heading angle between other aircraft and the UAV. represents the time derivative of the relative heading angle, Indicates the speed of other aircraft, is the guidance angle of other aircraft, is the guidance angle of the UAV, represents the time derivative of the guidance angle, Indicates the heading angle of other aircraft, Indicates the heading angle of the drone, is the guidance coefficient, which reflects the response of the guidance system to the change of heading angle. The pitch control and yaw control of other aircraft are independent of each other, so it can be obtained:

[0068] ;

[0069] in, Indicates the pitch angle command of the pitch channel. Indicates the yaw angle command of the yaw channel; and Respectively The derivatives of the components in both pitch and yaw channels, and They represent the guidance coefficients on the pitch and yaw channels respectively. By adjusting the guidance coefficient values ​​on the two channels, other aircraft can achieve better tracking effects.

[0070] Step S13: Building a reinforcement learning environment in the programming language platform and establishing a reinforcement learning model.

[0071] Build a reinforcement learning environment and establish a reinforcement learning model on the programming language platform (python), and then establish UDP (User Datagram Protocol) communication between matlab and python to prepare for subsequent information transmission and model training. When building a reinforcement learning environment, in addition to using OpenAI Gym (a toolkit for developing and comparing reinforcement learning algorithms), TensorFlow (an open source machine learning framework) or PyTorch (an open source machine learning library), some auxiliary tools will also be used. For example, NumPy provides basic support for data processing, and Pandas is used for data cleaning and analysis, especially when processing large amounts of data generated by the interaction between the intelligent agent and the environment, it can quickly organize and count data.

[0072] Step S14: inputting the position and attitude information of the UAV and the position and attitude information of other aircraft into the reinforcement learning model, so as to output the flight actions in the UAV action library through the reinforcement learning model.

[0073] In order to effectively build an intelligent decision-making mechanism, this embodiment introduces the Markov decision process. The Markov decision process consists of five tuples: Among them, represents the state space, represents the action space, represents the state transition probability, represents the reward function, which is used to measure the rewards obtained by the agent when taking actions in different states. Represents the discount factor used to calculate the cumulative return. Figure 2As shown in Figure 1, the agent continuously interacts with the environment, tries different actions, adjusts its behavior strategy according to the rewards obtained, and finally learns the optimal strategy that can obtain the maximum cumulative reward in the environment. The construction of the drone selects the position and attitude information Position and attitude information with other aircraft , in order to fully characterize the three-dimensional state space, where , , Indicates the location coordinates of the drone, represents the roll angle of the drone, Indicates the pitch angle of the drone, represents the yaw angle of the drone, , , Indicates the position coordinates of other aircraft, represents the roll angle of other aircraft, Indicates the pitch angle of other aircraft, Indicates the yaw angle of other aircraft. With this information, the position and attitude of the drone and other aircraft in space can be accurately described. Action Space The establishment of is by numbering the basic maneuvers from 1 to 7, represented by the corresponding maneuvering action numbers. These basic maneuvers cover uniform straight-line level flight, maximum force acceleration, maximum force deceleration, climbing, diving, horizontal left turn, horizontal right turn, etc. In this way, a discrete action space is constructed, so that the decision of the intelligent agent can correspond to a specific and executable action. The specific operation process of this step is: the position and attitude information of the UAV and the position and attitude information of other aircraft are input into the reinforcement learning model constructed based on the Markov decision process. The reinforcement learning model has a strong analytical ability after learning a large amount of historical data and continuously optimizing the algorithm. It deeply analyzes the input state information, and after complex calculations and logical judgments, it finally outputs the appropriate flight action from the pre-established UAV action library. This process realizes the efficient conversion from current state information to specific action decisions.

[0074] Step S15: When the UAV avoids the other aircraft based on the flight action, whether the UAV avoids successfully is determined based on the maneuvering characteristics of the other aircraft. If successful, a corresponding reward value is given to the reinforcement learning model based on the situation of the maneuvering area of ​​the other aircraft to encourage the reinforcement learning model to optimize the avoidance strategy.

[0075] In this embodiment, within the flight time of other aircraft and within the overload that other aircraft can bear, it is determined whether the distance between other aircraft and the drone is greater than a preset value. If so, it is determined that the drone avoids successfully, and the corresponding reward value is given to the reinforcement learning model based on the situation of the maneuvering area of ​​other aircraft to encourage the reinforcement learning model to optimize the avoidance strategy. The situation of the maneuvering area of ​​other aircraft includes the distance between other aircraft and the drone, the relative speed between other aircraft and the drone, the relative angle between other aircraft and the drone, the collision probability of other aircraft, and the situation level of the area where other aircraft are located. Among them, the situation level represents the importance of the area where other aircraft are located in avoiding drones. Exemplarily, in a low-altitude route logistics transportation scenario, there is a logistics command and dispatch center, which is surrounded by dense drone transportation base stations and dense routes. This area is set as the first priority area, and an open area on the edge of the city has only a few scattered routes. This area is set as the second priority area. In the first embodiment, based on the distance between other aircraft and the drone, the first reward value is given to the reinforcement learning model through the first reward function to encourage the model to optimize the avoidance strategy. For example, the first reward function can be: , represents the first reward value, is a constant, indicating the effect of distance on reward. In this way, the distance between other aircraft and the UAV is negatively correlated with the reward value, so that other aircraft can get higher rewards when approaching the UAV, thus encouraging other aircraft to approach the UAV.

[0076] In a second embodiment, based on the relative speed of other aircraft and the UAV, a second reward value is given to the reinforcement learning model through a second reward function to encourage the model to optimize the avoidance strategy. For example, the first reward function can be: , represents the second reward value, is a constant, which indicates the influence of relative speed on reward. In this way, the relative speed between other aircraft and the UAV is negatively correlated with the reward value. When the relative speed between other aircraft and the UAV is large, the probability of collision increases. At this time, a negative reward value is set to inhibit the UAV from choosing an evasive path with too large a relative speed. In addition, the maneuvering area of ​​other aircraft can also be the relative angle between other aircraft and the UAV. The greater the change in the relative angle between other aircraft and the UAV, the greater the difficulty of collision. At this time, a positive reward value is set to encourage the UAV to choose an evasive path with a large change in relative angle.

[0077] In the third embodiment, if the situation level of the area where other aircraft are located is the first priority and the collision probability of other aircraft is greater than the preset probability threshold, the reinforcement learning model is given a third reward value to encourage the model to optimize the avoidance strategy; if the situation level of the area where other aircraft are located is the second priority or the collision probability of other aircraft is not greater than the preset probability threshold, the reinforcement learning model is given a fourth reward value to encourage the model to optimize the avoidance strategy. The first priority is higher than the second priority, and the third reward value is greater than the fourth reward value. That is, when other aircraft are in a high priority area and the probability of collision is high, a positive reward is given. If they are in a low priority area or the probability of success is low, a negative reward or a lower positive reward is given. It should be pointed out that for the above-mentioned probability of collision, this embodiment can establish a collision probability model based on factors such as the flight status of other aircraft and the movement of drones. This probability can be used as part of the situation quantification to reflect the collision probability of other aircraft in a certain state.

[0078] See also Figure 3 As shown, Figure 3 A schematic diagram of the interactive simulation process of other aircraft and unmanned aerial vehicles is disclosed. First, the initial situation is set, including but not limited to the initial position, speed, attitude and other related parameters of other aircraft and unmanned aerial vehicles, to lay the foundation for the subsequent simulation process. After the initial situation setting is completed, the other aircraft position attitude information iteration link is entered. In this way, during the simulation process, the position and attitude information of other aircraft will be continuously updated to simulate the dynamic changes of other aircraft during flight. Further, the corresponding decision action is input to avoid other aircraft. After the decision action is input, the parameters of the unmanned aerial vehicle will be iteratively updated accordingly to reflect the actual response and dynamic changes made by the unmanned aerial vehicle according to the decision action. Further, it is judged whether the flight of other aircraft has ended. If the flight of other aircraft has not ended, the process will return to the other aircraft position attitude information iteration link, continue to update the information of other aircraft, and repeat the subsequent decision action input and unmanned aerial vehicle parameter iteration steps to form a loop until the flight of other aircraft ends. When the flight of other aircraft ends, the simulation end judgment link is entered. Here, it is judged whether the entire simulation process is over. If the simulation has not ended, the process will return to the initial situation setting link, reset the initial situation, and start a new round of simulation. If the simulation ends, the simulation results are output. The simulation results include whether the drone successfully avoids other aircraft, the flight trajectory of other aircraft, the flight trajectory of the drone, various performance indicators and other related information. These results can be used for subsequent analysis, evaluation and decision-making.

[0079] In summary, (1) by using the other aircraft tracking algorithm based on the proportional guidance law, other aircraft can realize autonomous tracking maneuvers of the UAV; (2) by establishing a basic maneuvering action library for the UAV, the flight state of the UAV is made stable, continuous and reasonable, and the logical switching of maneuvering instructions is realized; (3) by using a reinforcement learning-based UAV avoidance method for other aircraft proposed in this application, the UAV can eventually autonomously learn an effective maneuvering avoidance strategy without relying on prior knowledge of avoiding other aircraft.

[0080] In low-altitude logistics transportation, drone A is responsible for package delivery and flies along the planned route at a speed of 8 meters per second and an altitude of 120 meters. During the flight, it encounters drone B (regarded as other aircraft) flying in the opposite direction at a speed of 7 meters per second, and there is a risk of collision between the two. Before executing the mission, technicians build a six-degree-of-freedom model and action library of drone A and a three-degree-of-freedom particle model of drone B on the engineering calculation simulation platform. At the same time, constraints are imposed on the flight time, safety range and tolerable overload of drone B. For example, the simulation flight time of drone B is set to 15 seconds, and the size of the simulation space (safety range) is set to a cube with a length, width and height of 10 kilometers each, so that its tolerable overload is controlled within a certain threshold. A reinforcement learning environment and model are built on the programming language platform, and a large amount of data is used for training to form a rich action library, including acceleration, deceleration, climbing, diving, turning and other actions. Drone A obtains its own and drone B's position and posture information in real time through onboard sensors, and transmits this information to the reinforcement learning model. The model is built based on the Markov decision process. According to the pre-designed reward function, the reward values ​​of different actions are calculated by integrating factors such as distance and relative speed. According to the calculation results, the model selects the action output of "turning quickly to the right and slowing down appropriately" from the action library. The flight control system of UAV A executes the corresponding action after receiving the command. During the execution process, the system monitors in real time. If the distance between the two aircraft is greater than 50 meters within the flight time and the tolerable overload range of UAV B, it is determined that UAV A has successfully avoided. Then, according to the situation of the maneuvering area of ​​UAV B, such as the degree of danger and collision probability of the area, the corresponding reward value is given to the reinforcement learning model. If UAV B is in a dangerous area and the collision probability is high, a high reward value is given; if it is in a safe area or the collision probability is low, a low reward value is given to optimize the model decision strategy. In the end, UAV A successfully avoids UAV B and continues to complete the package delivery task. Compared with traditional methods, this decision-making method based on reinforcement learning can quickly respond to complex airspace conditions and is suitable for high-density flight scenarios of urban low-altitude logistics.

[0081] It can be seen that the present application proposes a method for determining a UAV avoidance strategy based on reinforcement learning, including: building a six-degree-of-freedom model of a UAV in an engineering computing simulation platform, and establishing a UAV action library; building a three-degree-of-freedom particle model of other aircraft currently in flight in the engineering computing simulation platform, constraining the maneuverability characteristics of other aircraft based on constraint rules, and realizing tracking of the UAV by other aircraft according to the relative motion situation; building a reinforcement learning environment in a programming language platform and establishing a corresponding reinforcement learning model; inputting the position and posture information of the UAV and the position and posture information of other aircraft into the reinforcement learning model, so as to output the flight action in the UAV action library through the reinforcement learning model; when the UAV avoids other aircraft based on the flight action, judging whether the UAV avoids successfully based on the maneuverability characteristics of other aircraft, and if successful, giving the reinforcement learning model a corresponding reward value based on the maneuverability area situation of the other aircraft, so as to encourage the reinforcement learning model to optimize the avoidance strategy. It can be seen that this application builds a six-degree-of-freedom model of the drone and a three-degree-of-freedom particle model of other aircraft currently in flight on the engineering computing simulation platform, and combines the reinforcement learning environment and the reinforcement learning model to enable the drone to autonomously learn the response strategy. Even if other aircraft use a new guidance subsystem, the drone does not need to wait for expert analysis, but relies on its own accumulated learning results in the simulation environment to quickly select the appropriate flight action from the action library for avoidance, getting rid of the dependence on the expert's prior knowledge. Secondly, the reinforcement learning model in this application is not based on the calculation results of the theoretical model that may have errors, but is based on the response mode learned from similar scenarios in the past, and flexibly adjusts the flight action of the drone, effectively avoiding the interference of modeling errors on avoidance decisions. Furthermore, after the drone successfully avoids other aircraft, this application gives the reinforcement learning model a corresponding reward value based on the situation of the maneuvering area of ​​other aircraft. In this way, the reinforcement learning model is prompted to optimize the avoidance strategy and form a virtuous circle. Finally, this application uses an engineering computing simulation platform to flexibly adjust parameters to simulate the characteristics of a variety of different environments, so that drones can rely on the avoidance capabilities acquired through training in various complex and changing scenarios, steadily improve the avoidance success rate, and greatly enhance the scalability of the strategy.

[0082] Correspondingly, the embodiment of the present application also discloses a device for determining a drone avoidance strategy based on reinforcement learning, see Figure 4 As shown, the device comprises:

[0083] The UAV model building module 11 is used to build a UAV six-degree-of-freedom model in the engineering computing simulation platform and establish a UAV motion library;

[0084] Other aircraft model building module 12, used to build three-degree-of-freedom particle models of other aircraft currently in flight in the engineering calculation simulation platform;

[0085] The other aircraft characteristic constraint module 13 is used to constrain the maneuverability characteristics of other aircraft based on the constraint rules, and to enable other aircraft to track the UAV according to the relative motion situation; the relative motion situation is the relative motion situation between the other aircraft and the UAV;

[0086] A reinforcement learning model building module 14 is used to build a reinforcement learning environment in a programming language platform and establish a corresponding reinforcement learning model;

[0087] A flight action output module 15, used for inputting the position and attitude information of the UAV and the position and attitude information of the other aircraft into the reinforcement learning model, so as to output the flight actions in the UAV action library through the reinforcement learning model;

[0088] The avoidance strategy optimization module 16 is used to determine whether the UAV has successfully avoided the other aircraft based on the maneuvering characteristics of the other aircraft when the UAV avoids the other aircraft based on the flight action. If successful, a corresponding reward value is given to the reinforcement learning model based on the situation of the maneuvering area of ​​the other aircraft to encourage the reinforcement learning model to optimize the avoidance strategy.

[0089] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0090] It can be seen that the present application proposes a method for determining a UAV avoidance strategy based on reinforcement learning, including: building a six-degree-of-freedom model of a UAV in an engineering computing simulation platform, and establishing a UAV action library; building a three-degree-of-freedom particle model of other aircraft currently in flight in the engineering computing simulation platform, constraining the maneuverability characteristics of other aircraft based on constraint rules, and realizing tracking of the UAV by other aircraft according to the relative motion situation; building a reinforcement learning environment in a programming language platform and establishing a corresponding reinforcement learning model; inputting the position and posture information of the UAV and the position and posture information of other aircraft into the reinforcement learning model, so as to output the flight action in the UAV action library through the reinforcement learning model; when the UAV avoids other aircraft based on the flight action, judging whether the UAV avoids successfully based on the maneuverability characteristics of other aircraft, and if successful, giving the reinforcement learning model a corresponding reward value based on the maneuverability area situation of the other aircraft, so as to encourage the reinforcement learning model to optimize the avoidance strategy. It can be seen that this application builds a six-degree-of-freedom model of the drone and a three-degree-of-freedom particle model of other aircraft currently in flight on the engineering computing simulation platform, and combines the reinforcement learning environment and the reinforcement learning model to enable the drone to autonomously learn the response strategy. Even if other aircraft use a new guidance subsystem, the drone does not need to wait for expert analysis, but relies on its own accumulated learning results in the simulation environment to quickly select the appropriate flight action from the action library for avoidance, getting rid of the dependence on the expert's prior knowledge. Secondly, the reinforcement learning model in this application is not based on the calculation results of the theoretical model that may have errors, but is based on the response mode learned from similar scenarios in the past, and flexibly adjusts the flight action of the drone, effectively avoiding the interference of modeling errors on avoidance decisions. Furthermore, after the drone successfully avoids other aircraft, this application gives the reinforcement learning model a corresponding reward value based on the situation of the maneuvering area of ​​other aircraft. In this way, the reinforcement learning model is prompted to optimize the avoidance strategy and form a virtuous circle. Finally, this application uses an engineering computing simulation platform to flexibly adjust parameters to simulate the characteristics of a variety of different environments, so that drones can rely on the avoidance capabilities acquired through training in various complex and changing scenarios, steadily improve the avoidance success rate, and greatly enhance the scalability of the strategy.

[0091] Furthermore, an embodiment of the present application also provides an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0092] Figure 5A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display screen 23, an input and output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the method for determining a drone avoidance strategy based on reinforcement learning disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0093] In this embodiment, the power supply 26 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 24 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0094] In addition, the memory 22, as a carrier for storing resources, may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include a computer program 221, and the storage method may be temporary storage or permanent storage. In addition to the computer program that can be used to complete the method for determining a drone avoidance strategy based on reinforcement learning performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 221 may further include a computer program that can be used to complete other specific tasks.

[0095] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned reinforcement learning-based drone avoidance strategy determination method is implemented.

[0096] For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0097] The various embodiments in this application are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0098] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0100] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0101] The above is a detailed introduction to the method, device, equipment, and storage medium for determining a drone avoidance strategy based on reinforcement learning provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for determining a drone avoidance strategy based on reinforcement learning, characterized in that: include: Build a six-degree-of-freedom model of a UAV in the engineering computing simulation platform and establish a UAV motion library; A three-degree-of-freedom particle model of other aircraft currently in flight is constructed in the engineering computing simulation platform, the maneuverability characteristics of other aircraft are constrained based on constraint rules, and the tracking of the UAV by other aircraft is achieved according to the relative motion situation; the relative motion situation is the relative motion situation between the other aircraft and the UAV; Build a reinforcement learning environment in the programming language platform and establish a corresponding reinforcement learning model; Inputting the position and attitude information of the UAV and the position and attitude information of the other aircraft into the reinforcement learning model, so as to output the flight actions in the UAV action library through the reinforcement learning model; When the UAV avoids the other aircraft based on the flight action, it is judged whether the UAV avoids successfully based on the maneuvering characteristics of the other aircraft. If successful, a corresponding reward value is given to the reinforcement learning model based on the situation of the maneuvering area of ​​the other aircraft to encourage the reinforcement learning model to optimize the avoidance strategy.

2. The method for determining a drone avoidance strategy based on reinforcement learning according to claim 1, characterized in that: The other aircraft maneuverability characteristics include the flight time of other aircraft, the safety range of other aircraft, and the tolerable overload of other aircraft. The constraint rules are used to constrain the maneuverability characteristics of other aircraft, including: Setting the simulation time of other aircraft in the engineering calculation simulation platform so as to constrain the flight time of the other aircraft through the simulation time of the other aircraft; The size of the simulation space is set in the engineering calculation simulation platform so as to constrain the safety range of the other aircraft through the size of the simulation space; The overload that the other aircraft can withstand is controlled to be no greater than a preset overload threshold, so as to constrain the overload that the other aircraft can withstand; wherein the overload that the other aircraft can withstand includes tangential overload and normal overload.

3. The method for determining a drone avoidance strategy based on reinforcement learning according to claim 2 is characterized in that: The determining whether the UAV successfully avoids the maneuver based on the maneuver characteristics of the other aircraft includes: Within the flight time of the other aircraft and within the tolerable overload of the other aircraft, determine whether the distance between the other aircraft and the drone is greater than a preset value. If so, determine that the drone has successfully avoided the obstacle.

4. The method for determining a drone avoidance strategy based on reinforcement learning according to any one of claims 1 to 3, characterized in that: The situation of the maneuvering area of ​​other aircraft includes the distance between the other aircraft and the UAV, the relative speed between the other aircraft and the UAV, the collision probability of the other aircraft and the situation level of the area where the other aircraft is located; wherein the situation level represents the importance of the area where the other aircraft is located to the UAV when avoiding it.

5. The method for determining a drone avoidance strategy based on reinforcement learning according to claim 4 is characterized in that: Giving the reinforcement learning model a corresponding reward value based on the situation of other aircraft in the maneuvering area to encourage the reinforcement learning model to optimize the avoidance strategy includes: Based on the distance between the other aircraft and the UAV, a first reward value is given to the reinforcement learning model through a first reward function to encourage the reinforcement learning model to optimize the avoidance strategy; wherein, in the first reward function, the distance between the other aircraft and the UAV is negatively correlated with the reward value.

6. The method for determining a drone avoidance strategy based on reinforcement learning according to claim 4 is characterized in that: Giving the reinforcement learning model a corresponding reward value based on the situation of other aircraft in the maneuvering area to encourage the reinforcement learning model to optimize the avoidance strategy includes: Based on the relative speed between the other aircraft and the UAV, a second reward value is given to the reinforcement learning model through a second reward function to encourage the reinforcement learning model to optimize the avoidance strategy; wherein, in the second reward function, the relative speed between the other aircraft and the UAV is negatively correlated with the reward value.

7. The method for determining a drone avoidance strategy based on reinforcement learning according to claim 4, characterized in that: Giving the reinforcement learning model a corresponding reward value based on the situation of other aircraft in the maneuvering area to encourage the reinforcement learning model to optimize the avoidance strategy includes: If the situation level of the area where the other aircraft is located is the first priority and the collision probability of the other aircraft is greater than the preset probability threshold, a third reward value is given to the reinforcement learning model to encourage the reinforcement learning model to optimize the avoidance strategy; If the situation level of the area where the other aircraft is located is the second priority or the collision probability of the other aircraft is not greater than the preset probability threshold, giving the reinforcement learning model a fourth reward value to encourage the reinforcement learning model to optimize the avoidance strategy; The first priority is higher than the second priority, and the third reward value is greater than the fourth reward value.

8. A device for determining a drone avoidance strategy based on reinforcement learning, characterized in that: include: The UAV model building module is used to build a six-degree-of-freedom model of the UAV in the engineering computing simulation platform and establish a UAV motion library; Other aircraft model building module, used to build three-degree-of-freedom particle models of other aircraft currently in flight in the engineering calculation simulation platform; Other aircraft characteristic constraint module, used for constraining the maneuvering characteristics of other aircraft based on constraint rules, and realizing tracking of the UAV by other aircraft according to relative motion situation; the relative motion situation is the relative motion situation between the other aircraft and the UAV; The reinforcement learning model building module is used to build a reinforcement learning environment in the programming language platform and establish a corresponding reinforcement learning model; A flight action output module, used for inputting the position and attitude information of the UAV and the position and attitude information of the other aircraft into the reinforcement learning model, so as to output the flight actions in the UAV action library through the reinforcement learning model; The avoidance strategy optimization module is used to determine whether the UAV has successfully avoided the other aircraft based on the maneuvering characteristics of the other aircraft when the UAV avoids the other aircraft based on the flight action. If successful, a corresponding reward value is given to the reinforcement learning model based on the situation of the maneuvering area of ​​the other aircraft to encourage the reinforcement learning model to optimize the avoidance strategy.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method for determining a drone avoidance strategy based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, it implements the method for determining a drone avoidance strategy based on reinforcement learning as described in any one of claims 1 to 7.

Images