Unmanned aerial vehicle battle control method, device and equipment and storage medium

Abstract

The invention provides an unmanned aerial vehicle battle control method and device, equipment and a storage medium. The method comprises the following steps: acquiring state information of a friend or foe unmanned aerial vehicle; obtaining target reference angle information through an air combat decision network based on the state information of the friend or foe unmanned aerial vehicle; inputting the target reference angle information into an attitude angle controller to obtain target reference angular rate information; inputting the target reference angular rate information into an angular rate controller to obtain target control torque information; and inputting the target control moment information into the constructed unmanned aerial vehicle kinetic model to obtain the control state information of the our unmanned aerial vehicle. Thus, under the condition that the dynamic detailed information of the unmanned aerial vehicle model is not informed, optimal control over the attitude angle and the angular rate of the unmanned aerial vehicle can be achieved, the unmanned aerial vehicle is guided to occupy a favorable position, and therefore the adverse effect caused by a mass point model is avoided.

Description

Technical Field

[0001] The present application relates to the field of aircraft control technology, and in particular to a method, device, equipment and storage medium for controlling unmanned aerial vehicle combat. Background Art

[0002] Fixed-wing UAV is a complex controlled object that integrates multivariable, uncertain, nonlinear, fast time-varying, strong coupling, static instability and underactuation. Its flight control technology has always been the focus and difficulty of research in the aviation field. At present, traditional control methods based on classical theory, such as PID control algorithm, are widely used in the field of flight control because of their simple structure, easy implementation, and independence from system mathematical models. In the design of UAV air combat decision-making algorithms, existing methods mostly summarize basic air combat actions, use machine learning methods to learn air combat strategies, and obtain action value functions under various air combat situations. In actual flight, the maneuvering action with the highest benefit in the current state is obtained by looking up the table and executed.

[0003] However, the control method based on traditional theory has at least the following problems when designing fixed-wing UAV controllers and air combat decision controllers:

[0004] In traditional one-on-one air combat problems against drones, the models used are mostly particle models, which do not consider the impact of aerodynamic-related state quantities such as angle of attack and sideslip angle on the dynamic position of the drone. Summary of the invention

[0005] The present application provides a drone combat control method, device, equipment and storage medium to solve the defects of one-to-one drone combat control in the prior art.

[0006] According to the first aspect of the present application, a method for controlling UAV combat is provided, the method comprising: obtaining status information of enemy and friendly UAVs; obtaining target reference angle information through an air combat decision network based on the status information of enemy and friendly UAVs; inputting the target reference angle information into an attitude angle controller to obtain target reference angular velocity information; inputting the target reference angular velocity information into an angular velocity controller to obtain target control torque information; inputting the target control torque information into a constructed UAV dynamics model to obtain control status information of our UAV.

[0007] According to a second aspect of the present invention, there is provided a UAV combat control device, comprising: a first information acquisition module, used to acquire status information of enemy and friendly UAVs; a decision module, used to obtain target reference angle information through an air combat decision network based on the status information of enemy and friendly UAVs; an attitude angle control module, used to input the target reference angle information into an attitude angle controller to obtain target reference angular rate information; an angular rate control module, used to input the target reference angular rate information into an angular rate controller to obtain target control torque information; a second information acquisition module, used to input the target control torque information into a constructed UAV dynamics model to obtain control status information of our UAV.

[0008] According to a third aspect of the present application, an electronic device is provided, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, any one of the above-mentioned drone combat control methods is implemented.

[0009] According to a fourth aspect of the present application, a computer-readable storage medium is provided, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the above-mentioned drone combat control methods is implemented.

[0010] In summary, the drone combat method, device, equipment and storage medium provided by the present application have at least the following beneficial effects: the target reference angle information tracked by the attitude angle controller is output based on the state information of the enemy and friendly drones through the air combat decision network, the target reference angular velocity information tracked by the angular velocity controller is output by the attitude angle controller, and the reference torque information is output by the angular velocity controller. By directly inputting the reference torque information into the attitude dynamic model of the drone, the control state information of the drone can follow the attitude reference signal. In addition, by forming two inner and outer loops by the angular velocity controller and the attitude angle controller, the optimal control of the attitude angle and angular velocity of the drone can be achieved without knowing the detailed dynamic information of the drone model, and the drone of our side can be guided to occupy a favorable position, thereby solving the adverse effects brought by the particle model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A principle block diagram of a drone combat control method according to an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram showing a flow chart of a drone combat control method provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram showing relative characteristic quantities of friendly and enemy drones provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of a guided reward function provided in an embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram showing the iterative change of a strategy network parameter of an angular rate controller provided in an embodiment of the present application is shown;

[0017] Figure 6 A schematic diagram showing the iterative change of a strategy network parameter of an attitude angle controller provided in an embodiment of the present application is shown;

[0018] Figure 7 A schematic diagram showing an angle response of a drone provided by an embodiment of the present application is shown;

[0019] Figure 8 A schematic diagram showing the evolution of an average total step length of an air combat game provided by an embodiment of the present application is shown;

[0020] Fig. 9 A schematic diagram showing the evolution of the average cumulative reward for air combat games provided by an embodiment of the present application is shown;

[0021] Fig.10 A schematic diagram of an air combat flight trajectory provided by an embodiment of the present application is shown;

[0022] Fig.11 Another schematic diagram of an air combat flight trajectory provided by an embodiment of the present application is shown;

[0023] Fig.12 A schematic diagram of another air combat flight trajectory provided by an embodiment of the present application is shown;

[0024] Fig.13 A schematic diagram showing the structure of a drone combat control device provided in an embodiment of the present application is shown;

[0025] Fig.14 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the above and other features and advantages of the present application more clear, the present application is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0027] In the following description, many specific details are set forth to provide a thorough understanding of the present application. However, it is apparent to those skilled in the art that specific details need not be adopted to practice the present application. In other cases, well-known steps or operations are not described in detail to avoid blurring the present application.

[0028] Before introducing the present application in detail, it is necessary to make the following assumptions about the UAV dynamics model involved in the present application: (1) The aircraft body is regarded as a rigid body whose mass does not change with time, and the influence of the elasticity of the aircraft body is ignored. (2) The influence of the wind field on the flight dynamics is ignored, that is, the airspeed is equal to the ground speed. (3) The influence of the earth's rotation and ground curvature on the flight dynamics is ignored. (4) It is assumed that the layout and mass distribution of the UAV are plane-symmetrical. (5) The UAV adopts the bank-to-turn (BTT) maneuver.

[0029] This application constructs four coordinate systems to conveniently represent the position, velocity, acceleration, and aerodynamic force of the drone. The four coordinate systems are:

[0030] (1) Ground inertial coordinate system origin Take a point on the ground in a low-latitude area. The coordinate system is fixed to the ground and regarded as an inertial coordinate system. The unit vector of the z-axis is The plumb bob points downward, and the x-axis is the unit vector The unit vector pointing north, the y axis Pointing to the east, they together form a right-hand coordinate system.

[0031] (2) Body coordinate system origin Take the coordinate system at the center of mass of the drone, which is fixed to the drone, and the unit vector of the x-axis The unit vector of the y-axis coincides with the fuselage axis of the drone and points to the nose. The unit vector of the z-axis is perpendicular to the symmetry plane of the drone and points to the right side of the fuselage. It is in the symmetry plane of the drone and points perpendicular to the x-axis downwards.

[0032] (3) Airflow coordinate system origin Take the coordinate system at the center of mass of the drone, which is fixed to the drone, and the unit vector of the x-axis The unit vector of the z-axis coincides with the velocity vector of the drone and has the same direction. In the symmetry plane of the drone, and perpendicular to the x-axis pointing downwards, the unit vector of the y-axis is Pointing to the right side of the fuselage and Together they form a right-handed rectangular coordinate system.

[0033] (4) Track coordinate system origin Take the coordinate system at the center of mass of the drone, which is fixed to the drone, and the unit vector of the x-axis The unit vector of the z-axis is consistent with the direction of the drone's speed. In the plumb plane containing the flight velocity vector, The unit vector of the y-axis points vertically downward. and Together they form a right-handed rectangular coordinate system.

[0034] The drone battle control method provided in the embodiment of the present application is applied to a one-on-one drone battle scenario and can be executed by the drone battle control device provided in the embodiment of the present application, which can be configured in an electronic device.

[0035] The UAV combat control method proposed in the embodiment of the present application mainly includes two parts: the design of the UAV attitude controller based on reinforcement learning and the design of the air combat decision controller based on deep reinforcement learning. Among them, the attitude controller includes an attitude angle controller and an angular rate controller. The core idea of ​​the UAV attitude controller design is: directly acting on the six-degree-of-freedom dynamic equation of the UAV, so that the UAV attitude follows the inferred attitude reference signal. Figure 1 The figure shows the main decision-making architecture schematic diagram of the method, which includes a decision layer 11 and a control layer 12. The control layer 12 is divided into two inner and outer loops, namely an angular rate control loop and an attitude angle control loop. The optimal controller parameters of the two loops are learned based on the strategy reinforcement learning algorithm. The core idea of ​​the air combat decision controller design is: the decision layer is based on the state information of the own UAV and the enemy UAV, and the characteristic value is calculated to more accurately describe the air combat situation. At the same time, the deep neural network will make maneuver decisions according to the current situation to achieve the purpose of guiding our UAV to occupy an advantageous position.

[0036] Figure 1 A block diagram showing a method for controlling a drone battle according to an embodiment of the present application is shown. Figure 1 As shown, the combat control decision architecture of the present application mainly includes: a decision layer 11 and a control layer 12, wherein the decision layer 11 may include an air combat decision network 111. The control layer 12 includes an attitude angle controller 121 and an angular rate controller 122.

[0037] In addition, the decision layer may further include a control allocation 112. The control allocation 112 is used to solve the reference attitude angle of the corresponding UAV based on the angle dynamic equation and the target reference roll angle output by the air combat decision network.

[0038] Figure 2 A flowchart of a method for controlling a drone battle provided by an embodiment of the present application is shown as follows: Figure 2As shown, the UAV combat control method may include the following steps.

[0039] S21, obtain the status information of enemy and friendly UAVs.

[0040] The state information of the enemy and friendly drones involved in the embodiments of the present application reflects the overall situation of the current air battle and the relative situation of the drones of each party, and may include but is not limited to the basic state information of our drones and the basic state information of the enemy drones. Among them, the basic state information may include position information, heading angle and roll angle.

[0041] S22, based on the status information of enemy and friendly UAVs, obtains the target reference angle information through the air combat decision network.

[0042] The air combat decision network involved in the embodiment of the present application can make maneuver decisions according to the current air combat situation, so as to achieve the purpose of guiding our drone to occupy an advantageous position. The air combat decision network can be constructed according to a deep neural network.

[0043] In one embodiment of the present application, the state information of enemy and friendly UAVs is input into the air combat decision network to determine a maneuvering strategy, and the target reference angle information is obtained based on the maneuvering actions in the maneuvering strategy.

[0044] S23, inputting the target reference angle information into the attitude angle controller to obtain the target reference angular velocity information.

[0045] The attitude angle controller involved in the embodiment of the present application can be called an angle loop controller. It can not only be used to output the target reference angular rate information corresponding to the maneuvering strategy, but also compare the actual attitude angle with the target angle information, and generate the corresponding optimal angle control strategy according to the comparison result to achieve the angle adjustment of the drone.

[0046] S24, inputting the target reference angular velocity information into the angular velocity controller to obtain the target control torque information.

[0047] The angular rate controller involved in the embodiment of the present application can be called an angular rate loop controller. It can not only be used to output the target control torque information corresponding to the maneuvering strategy, but also generate the optimal angular rate control strategy of the drone according to the difference between the actual angular rate and the target reference angular rate, so as to achieve the angular rate adjustment of the drone.

[0048] S25, input the target control torque information into the constructed UAV dynamics model to obtain the control state information of our UAV.

[0049] The embodiment of the present application needs to construct a UAV dynamics model before performing the above operations, and the UAV dynamics model involved may include a UAV attitude dynamics model and a UAV position dynamics model. The UAV attitude dynamics model may include an angular rate dynamics model and an angle dynamics model.

[0050] The moment of inertia matrix of the drone can be expressed as:

[0051]

[0052] The UAV angle dynamic model can be expressed as:

[0053]

[0054] Among them, α is the angle of attack of the UAV, is the derivative of the angle of attack with respect to time, β is the sideslip angle of the UAV, is the time derivative of the sideslip angle, and μ is the roll angle, is the derivative of the roll angle with respect to time. m is the mass of the drone, g is the acceleration of gravity, V is the speed of the drone, ρ is the air density, S w is the reference area of ​​the UAV wing, C L is the lift coefficient, T is the engine thrust, C Yβ is the lateral force coefficient.

[0055] The attitude dynamics model of the UAV can be expressed as:

[0056]

[0057] Among them, p, q, and r are the components of the angular velocity of the drone on the body coordinate axis. are the derivatives of each angular velocity component with respect to time, L, M, N are the components of the control torque on the body coordinate axis, c1 to c9 are composed of inertia products, which can be specifically expressed as

[0058]

[0059] The UAV position dynamics model can be expressed as:

[0060]

[0061] in, is the coordinate of the center of mass of the UAV in the ground inertial coordinate system, γ, ψ V are the track inclination angle and heading angle of the airflow coordinate system relative to the ground inertial coordinate system, n z represents the normal overload caused by the angle of attack of the UAV, n x Indicates the axial overload caused by the UAV thrust.

[0062] n z Produced by the angle of attack α, it can be expressed as:

[0063]

[0064] It should be noted that since the UAV adopts a banked turning maneuver, the sideslip angle is small, so the lateral overload caused by the sideslip angle is ignored. μ is the roll angle, which determines the direction of the normal overload in the vertical and velocity vector planes.

[0065] The control state information of our drone involved in the embodiment of the present application is the attitude angle, angular velocity and position information obtained by controlling the drone torque according to the target control torque information.

[0066] It should be noted that the air combat decision network constitutes the decision layer of the UAV control system, and the attitude angle controller and angular rate controller constitute the control layer. The control goal of the control layer is to make the attitude angle of the UAV track the reference attitude angle of the decision layer. Formula (2) to Formula (4) constitute the six-degree-of-freedom model of the UAV.

[0067] In the above embodiment, the air combat decision network outputs the target reference angle information tracked by the attitude angle controller based on the state information of the enemy and friendly UAVs, the attitude angle controller outputs the target reference angular velocity information tracked by the angular velocity controller, and the angular velocity controller outputs the reference torque information, and the reference torque information is directly input into the attitude dynamic model of the UAV, so that the control state information of the UAV can follow the attitude reference signal. In addition, by forming two inner and outer loops by the angular velocity controller and the attitude angle controller, the optimal control of the attitude angle and angular velocity of the UAV can be achieved without knowing the detailed dynamic information of the UAV model, and the UAV of our side can be guided to occupy a favorable position, thereby solving the adverse effects brought by the particle model.

[0068] In the face of a one-on-one air combat scenario, especially in a one-on-one air combat confrontation scenario between drones at the same altitude, the combat process can be regarded as the optimal decision-making process of a random dynamic system. Therefore, in some embodiments, the drone combat control method also includes: constructing a Markov model to describe the air combat problem, that is, using the Markov model to train the air combat decision network, and the Markov model includes a state part, an action part, a reward function and a state transfer function. Among them, the state part includes the basic state quantity of both the enemy and us, which can be expressed as the following formula.

[0069]

[0070] Among them, the superscript r represents our information, and the superscript b represents the enemy's information. Indicates position information, heading angle The value range is [-π,π), the roll angle μ r ,μ b The value range of depends on the maneuverability of the UAV.

[0071] Define the state space set as S1 and the action space set as A1. The Markov model can also be described as the interaction process between the drone and the environment. At time step t, the current state is Drone selects action in A1 And execute, the environment will transfer function according to the state Feedback the status of the next moment And the reward value The process of the drone choosing an action based on the current state depends on the strategy π1: S1→A1. The strategy π1 is a mapping from the state space to the action space.

[0072] Based on the aforementioned six-degree-of-freedom model, and in order to maintain the altitude of the drone, the roll angle μ and the angle of attack α of the drone in the action information must satisfy the following relationship.

[0073]

[0074] In this way, the target reference roll angle μ r , the target reference angle of attack α can be determined r , at the same time, the target reference sideslip angle β r Keep 0, therefore, in the Markov model, the control reference information of the UAV can be obtained according to the required roll angle, and then the controller is designed to act on the UAV dynamics model to track the control reference signal

[0075] That's it.

[0076] In the action part, since the tracking performance is taken into consideration, the roll angle rate is used as the action and the following constraints are met.

[0077]

[0078] It should be noted that according to formula (8), it can be seen that the action space composed of actions is a continuous space. Therefore, compared with the common discrete workspace, it can be more in line with the actual control capability and allow the drone to make more precise maneuvers.

[0079] In order to facilitate the subsequent description of the relative situation between drones, from our perspective, based on the basic state information, we can define the relative feature quantity, R LOS is the relative distance to the enemy UAV, θ AA ∈[-π,π) is the azimuth angle relative to the enemy drone, θ ATA∈[-π,π) is the antenna angle of the enemy UAV relative to the local UAV. Figure 3 A schematic diagram showing relative characteristic quantities of friendly and enemy drones provided in an embodiment of the present application is shown. Figure 3 The red drone represents our drone, and the blue drone represents the enemy drone.

[0080] Since the goal of drone air combat is to overcome the enemy's maneuvering influence and control our drone to maneuver to a relatively advantageous position, so as to meet the conditions for firing and attacking. The relatively advantageous position is that our drone is located in the rear area of ​​the enemy drone, and the nose is pointed at the enemy drone. The distance between the enemy and our drone is affected by the attack range and collision avoidance, and has upper and lower bounds. In order to quantitatively describe the relative advantageous position, based on the above-mentioned feature vector, the target reward function r is given in the form of pseudo code. goal (s1) is as follows:

[0081] For example, in R LOS When it is less than 0.2 m, r goal = -10. LOS Greater than 1 meter and less than 10 meters, |θ AA |<π / 3 and |θ ATA |<π / 6, r goal = 10. LOS Greater than 1 meter and less than 10 meters, |θ AA |>5π / 6 and |θ ATA |>2π / 3, r goal = -10. In other cases, r goal =0.

[0082] The state transition function can be described as in, Indicates the initial state of the time interval. a1 is a constant value, indicating the drone action. Indicates the status at the end of a time interval.

[0083] It should be noted that the time interval is the state transition time interval, which can be 0.25 seconds. In addition, according to action a1, the target reference angle information x Ωr =[α r β r μ r ] T .

[0084] In the above embodiment, in order to solve the problem of one-on-one air combat between UAVs, a one-on-one air combat maneuvering strategy is obtained through a one-on-one air combat Markov decision model, so that actions can be given according to the relative situation of the enemy and friendly UAVs, and then the target reference angle information that the attitude angle controller needs to track can be obtained.

[0085] In one embodiment, the maneuver strategy of the enemy UAV is obtained by the α-β pruning Minimax search algorithm. In one embodiment of the present application, the search depth of the enemy UAV is set to 3, and the value function is as follows:

[0086]

[0087] Where R d is the desired distance between the two aircraft, and k is the parameter for adjusting the distance and angle weights.

[0088] In the above embodiment, the optimal strategy of the enemy UAV that meets the current combat situation is determined by the pruned Minimax search algorithm, thereby determining the state information of the enemy UAV.

[0089] The goal of the one-on-one air combat problem of drones is to use a deep reinforcement learning algorithm to learn a one-on-one air combat maneuver strategy based on the one-on-one air combat Markov model. This strategy can give an action a1 based on the relative state of the drone and design an attitude angle controller to track the attitude reference signal.

[0090] In some embodiments, S22, based on the status information of enemy and friendly UAVs, obtaining the target reference angle information through the air combat decision network includes: constructing a feature vector based on the status information of enemy and friendly UAVs, and inputting the feature vector into the air combat decision network trained based on the Markov model, so as to use the air combat decision network to output the target reference angle information.

[0091] The feature vector involved in the embodiment of the present application is a feature set that can comprehensively and intuitively reflect the air combat situation, and can be expressed as the following formula.

[0092]

[0093] Among them, s f represents the feature vector, R LOS is the relative distance to the enemy UAV, θ AA is the azimuth relative to the enemy UAV, θ ATA is the antenna angle of the enemy UAV relative to the host, They represent our heading angle and the enemy's heading angle respectively.

[0094] In view of the vast state space and continuous action space faced by drone air combat, it is difficult to learn the optimal strategy without sufficient exploration. However, excessive exploration will significantly consume computing resources and slow down the efficiency of strategy learning. Therefore, in the sampling learning process, it is necessary to cleverly find a balance between exploration and data utilization. In addition, drone air combat is essentially a game of confrontation between the two sides. The use of random strategies can effectively prevent the enemy from carrying out targeted attacks, thereby improving the combat winning rate. In view of this, the embodiment of the present application proposes to use a reinforcement learning framework to train the one-to-one air combat strategy of drones. Through the designed random strategy framework, drones can generate random air combat maneuvers in the continuous action domain. This randomness greatly enhances the unpredictability of air combat strategies, making it difficult for the enemy to grasp our action intentions.

[0095] In some embodiments, the drone combat control method also includes: constructing an air combat decision network based on a reinforcement learning algorithm model; generating an experience replay pool based on sample data of historical games; and iteratively updating the air combat decision network based on the experience replay pool until convergence conditions are met.

[0096] In one embodiment of the present application, the reinforcement learning algorithm model can be a soft actor-critic (SAC) algorithm model. That is, the embodiment of the present application solves the optimal solution of the air combat decision network based on the SAC algorithm model to output the target reference angle information. The SAC algorithm model combines maximum entropy reinforcement learning and a policy gradient-based method, and by maximizing expected returns and policy entropy, the strategy can both explore the environment and utilize known information, thereby improving the stability and performance of the strategy.

[0097] Specifically, the SAC algorithm model architecture includes two parts: the strategy network and the value network. That is, the initial air combat decision network includes the strategy network and the value network. Among them, the input of the strategy network is the feature vector, and the output of the strategy network is the action probability density function, which can be expressed as Among them, the actions that the drone needs to perform in the execution strategy It is randomly drawn according to the action probability density function and belongs to a random strategy.

[0098] In order to balance the exploration of state and the use of data, the first objective function of the SAC algorithm model can be constructed based on the maximum entropy theory. That is, on the basis of the traditional objective function, the policy entropy is introduced, and the first objective function can be expressed as the following formula.

[0099]

[0100] Among them, H is the maximum time limit, Indicates that from the current state Starting from, based on the strategy π1, interacting with the environment, all possible state-action distributions are achieved. is the cumulative value of the traditional reward function, γ 1v1 is the discount factor, is the entropy of the action probability density function.

[0101] It should be noted that the purpose of policy network training in the SAC algorithm model is to maximize the objective function. This also means that for states with high cumulative rewards, the policy is allowed to have lower entropy and be more stable, while for states with low cumulative rewards, the policy entropy tends to be larger, encouraging exploration of different actions.

[0102] The value network in the SAC algorithm model is a parameterized soft action value network, which can be expressed as According to the Bellman equation, the soft action value network satisfies the following relationship.

[0103]

[0104] Among them, θ1 is the parameter of the value network. Indicates that the action is executed at time t Custom rewards obtained when is the soft state value function.

[0105] The soft state value function is the sum of the traditional value function and the policy entropy, which can be expressed as the following formula.

[0106]

[0107] in, represents the expectation of action a1 according to the probability distribution of π1, Soft Action Value Network Indicates in status Execute action when The probability distribution of .

[0108] The training process of the air combat decision network includes two steps: strategy evaluation and strategy improvement. In the strategy evaluation step, the parameterized soft action value function Gradient descent is performed to minimize the second objective function. The second objective function can be expressed as the following formula.

[0109]

[0110] in, Indicates that the experience replay pool A batch of tuples randomly selected from for training, It is the target of the soft action value network update.

[0111] In the policy improvement step, the goal of updating the policy network is to minimize the information divergence between the policy distribution and the indexed action value network distribution, that is, to minimize the difference between the action distribution and the action value distribution. Its objective function is the fourth objective function, which can be expressed as follows.

[0112]

[0113] in, represents actions with probability distribution according to π1 expectations.

[0114] Since the reward function of the existing air combat decision model is sparse, the drone will only receive reward feedback in some state areas, which will reduce the learning efficiency. Therefore, the embodiment of the present application proposes a guidance reward function that is continuous in the entire state space. The guidance reward function can be expressed as the following formula.

[0115]

[0116] in, It is a small constant set to avoid singularity in the denominator, and k is a parameter for adjusting the angle and distance weights.

[0117] Figure 4 A schematic diagram of a guided reward function provided by an embodiment of the present application is shown as follows: Figure 4 As shown in the figure, when the yaw angles of both the enemy and our drones are zero, the guidance reward function values ​​obtained by our drone relative to the enemy drone at different positions. The blue arrow represents the direction of the enemy drone's nose.

[0118] In the actual training process, the reward function used is the weighted sum of the environment target reward function and the guidance reward function. In an embodiment of the present application, the reward function r1 can be expressed as the following formula.

[0119] r1=0.8×r goal +0.2×r g . (16)

[0120] Although the soft state value function can be estimated according to Formula 12 during the strategy evaluation process, that is, it is estimated by the soft action value function and the strategy function. However, in order to further improve the stability of training, an independent parameterized soft state value network can be constructed. Estimate the soft-state value function separately.

[0121] In the process of updating the soft state value network, the third objective function needs to be minimized. The third objective function can be expressed as the following formula.

[0122]

[0123] in, Represents the state according to D probability distribution expectations, Represents the state-value network.

[0124] At the same time, construct the target state value network The structure of the network and the soft state value network Exactly the same, the parameters of the target state value network After a fixed time interval, it will be synchronized with the parameter θ2 of the soft state value network. After a fixed time interval, it will be synchronized with the parameter θ2 of the state value network. Introducing into formula 13, we can get the following formula.

[0125]

[0126] In this way, in the objective function of the soft action value network update, the target of the soft action value network update, that is, the D-target item, will not change frequently, which helps to improve the stability of learning.

[0127] The experience replay pool involved in the embodiment of the present application is used to store the experience generated during the drone battle so that it can be reused during the training process to optimize the target strategy. The experience replay pool is stored in the form of a quadruple, which includes actions randomly extracted from the strategy distribution. The state at time t And the state at time t+1 And synthesis rewards

[0128] In one embodiment of the present application, the training process of the air combat decision network, that is, the iterative update of the air combat decision network, specifically includes the following steps: initializing the soft action value network Parameters, state value network Parameters, target state value network Parameters and experience replay pool For each game, first, randomly initialize the state of each drone, and then in each state transition step, according to the current state of the drone Calculate the eigenvector Distribution by strategy Randomly extract actions And execute, the environment feedback is the state transfer function to get the next state With reward goal , and calculate the composite reward And the tuple Deposit into experience replay pool In each gradient descent step, the experience replay pool Extract a fixed-size batch of tuples from the , and update the soft state value network according to Formula 17 Update the soft action value network according to formula 18, update the strategy network π1 according to formula 14, and at every fixed time interval, The network parameter θ2 is assigned to the target state value network parameter The above updating process ends when the convergence condition is reached, that is, when the objective function of each network converges.

[0129] In the above embodiment, the feature vector is used to more accurately describe the air combat situation, and the reinforcement learning architecture is used to build an air combat decision network, so that maneuver decisions can be made according to the current situation, thereby achieving the purpose of guiding our drone to occupy an advantageous position.

[0130] The angular rate dynamics model, that is, Formula 3, can be written in the following dynamic equation form.

[0131]

[0132] y ω =x ω (19)

[0133] Among them, the angular rate state quantity is Its components represent the angular rate of rotation of the drone in the body coordinate system The projections on each axis, is the control torque, y ω is the system output, F(x ω ) is the system matrix, B τc is the input matrix.

[0134]

[0135] It can be seen from Formula 20 that there are nonlinearities and couplings in the UAV angular velocity dynamic equation. Therefore, the embodiment of the present application proposes a technical solution for constructing an angular velocity controller based on a policy reinforcement learning model, that is, constructing an optimal angular velocity controller, so that the optimal control of the UAV angular velocity loop can be achieved through input and output data.

[0136] In some embodiments, the drone combat control method may further include: a step of obtaining an angular rate controller based on a policy reinforcement learning algorithm.

[0137] The attitude angle controller involved in the embodiment of the present application can learn the optimal angular rate control strategy without the need for detailed drone model dynamic information.

[0138] In one embodiment of the present application, the step of obtaining the angular velocity controller based on a policy reinforcement learning algorithm may specifically include: constructing an angular velocity controller based on an augmented system of angular velocity dynamics; iteratively updating the angular velocity controller based on multiple iterations of control strategies, control information of our drone in historical periods, and status information until convergence conditions are met.

[0139] The angular rate dynamic augmentation system involved in the embodiment of the present application can be constructed according to the dynamic model of the angular rate of the drone. Specifically, it is first assumed that the reference angular rate of the drone satisfies the following dynamic equation.

[0140]

[0141] y ωr =x ωr , (twenty one)

[0142] in, is the smooth function to be determined, y ωr is the output of the reference angular rate.

[0143] Combining formulas 19 and 21, the following augmented system of angular rate dynamics can be obtained.

[0144]

[0145]

[0146] in, is the state quantity, is the augmented system matrix, is the angular rate tracking error.

[0147] Since the angular rate controller in the embodiment of the present application is constructed based on policy reinforcement learning, the angular rate controller includes a value function and a policy function.

[0148] The task of the angular rate controller is to make the angular rate of the drone track the reference angular rate. In order to achieve the optimal control of the angular rate of the drone, the value function is defined as the following formula.

[0149]

[0150] in They are all positive definite real constant matrices, and γ0>0 is the discount factor.

[0151] To seek optimal control The value function is minimized. Differentiate formula 23 and combine it with formula 22 to obtain the Bellman equation as follows.

[0152]

[0153] in, represents the partial differential of the value function to the augmented system, i.e., the state quantity in formula 22. Each control strategy corresponds to a corresponding value function V ω ,set up is the optimal value function corresponding to the optimal control, which will satisfy the following Hamilton-Jacobi-Bellman equation.

[0154]

[0155] Due to optimal control Should meet Therefore, the optimal control strategy can be expressed as the following formula, which can make the closed-loop system asymptotically stable.

[0156]

[0157] The augmented system of angular rate dynamics, namely Formula 22, can be rewritten as follows.

[0158]

[0159]

[0160] Among them, u τ is the actual control input, is the control strategy that needs to be learned iteratively, and the superscript n indicates that the strategy is the nth generation strategy.

[0161] The new system (i.e., Formula 27) is equivalent to the original augmented system (i.e., Formula 22). By differentiating the value function and combining the Bellman equation (i.e., Formula 24) with the optimal control strategy (i.e., Formula 26), the following value function differential equation can be obtained.

[0162]

[0163] Multiply both sides of formula 28 by And integrating it we can get the following equation.

[0164]

[0165] Where T is the time interval.

[0166] By a fixed actual control strategy u τ , and based on the measured state quantity e during the integration process ω , we can solve formula 29, and evaluate and improve the n-th generation strategy at the same time, and get and

[0167] In one embodiment of the present application, the training process of the angular rate controller may include the following steps: First, a stable initial control strategy u is applied to the angular rate controller. τ0 With exploration noise u τnoise , and obtain the control information and status information of our drone in the historical period. Then, use the given nth generation iteration strategy Solve formula 29 with the collected data to get the strategy evaluation With the improved strategy Finally, Updated to Iterate the calculation until the convergence condition is met. The convergence condition is and The absolute value of the difference is within the preset threshold.

[0168] In the process of solving formula 29, since the value function to be solved in formula 29 With the policy function It is usually nonlinear and difficult to solve directly, so two neural networks are used for function fitting, which can be expressed as the following formula.

[0169]

[0170]

[0171] in and is the neural network’s estimate, and is a polynomial basis function, let l ω1 With l ω2 Respectively represent the number of neurons in the two neural networks, and is the weight matrix.

[0172] definition And assume that the constant matrix of the value function is is a diagonal matrix, then the neural network in Formula 30 is substituted into Formula 29 to obtain the following formula.

[0173]

[0174] in is the Bellman estimate error, Represents the weight matrix The j-th column of can use the least squares method to minimize the Bellman estimation error and solve the weight parameters in Formula 30 for the next round of iteration.

[0175] when When it approaches zero, Formula 31 can be rewritten as follows.

[0176]

[0177]

[0178]

[0179]

[0180] The historical period involved in the embodiment of the present application may be from time point t1 to time point t N , that is, from the acquisition time point t1 to the time point t N The control information and status information of our drone, that is, the N segments a ω (t) and b ω (t). In order for the equation to have a unique solution, it is implicitly required that N ≥ l ω1 +3l ω2 .

[0181] The least squares solution formula of formula 32 can be expressed as the following formula.

[0182]

[0183] A ω =[a ω (t1)...a ω (t N )]

[0184] B ω =[b ω (t1)...b ω (t N )] T .

[0185] In the above embodiment, an angular rate controller of the UAV is constructed based on an augmented system of angular rate dynamics, and each function is solved according to a policy reinforcement learning algorithm, so that the optimal control strategy can be learned using only state quantities and input data under the condition of uncertainty of model parameters.

[0186] For the attitude dynamics model of the UAV, that is, Formula 3, in order to eliminate the influence of gravity acceleration g and engine thrust T on the design of the attitude angle controller, the actual attitude angle control quantity is constructed as the following formula.

[0187]

[0188] in, is the virtual control variable of angular rate to be designed.

[0189] The UAV attitude angle dynamics model can be rewritten as the following formula.

[0190]

[0191]

[0192]

[0193] in, Due to the high flying speed of drones, they are considered to be a small amount.

[0194] The UAV attitude angle dynamics model can be rewritten as the following dynamic equation.

[0195]

[0196] y Ω =x Ω (36)

[0197] Among them, x Ω =[α β μ] T , is the attitude angle state quantity of the UAV, and its components are the angle of attack, sideslip angle and roll angle. ω is the angular rate control quantity, y Ω is the system output, F(x Ω ) is the system matrix, B ωc (x Ω ) is the input matrix.

[0198]

[0199] It can be seen from Formula 36 that there are nonlinearities and couplings in the dynamic equation of the attitude angle of the UAV. Therefore, the embodiment of the present application proposes a technical solution for constructing an attitude angle controller based on a policy reinforcement learning model, that is, constructing an optimal controller, so that the optimal control of the attitude angle loop of the UAV can be achieved by inputting and outputting data.

[0200] In some embodiments, the drone combat control method may further include: obtaining an attitude angle controller based on a strategy reinforcement learning algorithm.

[0201] The attitude angle controller constructed by the policy reinforcement learning algorithm involved in the embodiment of the present application can learn the optimal attitude angle control strategy without the need for detailed drone model dynamic information.

[0202] Obtaining an attitude angle controller based on a policy reinforcement learning algorithm can specifically include: constructing an attitude angle controller based on an augmented system of angle dynamics; iteratively updating the attitude angle controller based on an initial control strategy, control information of our UAV during a target period, and state information until convergence conditions are met.

[0203] The angle dynamic augmentation system involved in the embodiment of the present application can be constructed according to the attitude dynamics model of the drone. Specifically, it is first assumed that the drone reference attitude angle satisfies the following dynamic equation.

[0204]

[0205] y Ω r = x Ω r, (38)

[0206] Among them, x Ωr =[α r β r μ r ] T , S Ωr (x Ωr ) is the smooth function to be determined, y Ωr is the output of the reference signal.

[0207] It should be pointed out that the common maneuvering method of drones is banked turning. Therefore, in the actual learning process, there are two types of angle state reference signals for drones: the first type is that the angle of attack and sideslip angle are zero, and the roll angle changes greatly; the second type is that the roll angle and sideslip angle are zero, and the angle of attack changes within an appropriate range.

[0208] Combining Formula 36 with Formula 38, the following angular dynamic augmented system can be obtained.

[0209]

[0210]

[0211] Among them, the state quantity is the augmented system matrix, e Ω Represents the angular tracking error.

[0212] Since the attitude angle controller in the embodiment of the present application is constructed based on policy reinforcement learning, the attitude angle controller includes a value function and a policy function.

[0213] In order to reduce the UAV angle tracking error and achieve optimal control, the value function of the attitude angle controller is defined as follows:

[0214]

[0215] in, And the discount factor γ1>0.

[0216] Similar to the angular rate controller, the following equation can be constructed, which can be solved to obtain the evaluation of the current strategy And the improved strategy

[0217]

[0218] in, represents the angular velocity value function gain matrix, e Ω represents the angle tracking error, ω represents the angular velocity of the UAV, and γ1 represents the discount factor.

[0219] In one embodiment of the present application, the training process of the attitude angle controller includes the following steps: First, a stable initial control strategy u is applied to the attitude angle controller. τ0 With exploration noise u τnoise , and obtain the historical period of our drone

[0220] Then, use the given nth generation iteration strategy Solve formula 41 with historical data to get the strategy evaluation With the improved strategy Finally, Updated to Iterate the calculation until the convergence condition is met. The convergence condition is and The absolute value of the difference is within the preset threshold.

[0221] In the process of solving formula 41, a neural network is constructed and Fitting value function With the policy function The least square method is used to solve formula 41 to obtain the neural network parameters. It should be noted that the solution process of formula 41 is similar to the solution process of formula 31, which will not be repeated here.

[0222] In the above embodiment, an attitude angle controller of the UAV is constructed based on an augmented system of attitude angle dynamics, and each function is solved according to a policy reinforcement learning algorithm, so that the optimal control strategy can be learned using only state quantities and input data under the condition of uncertainty of model parameters.

[0223] In order to verify the effectiveness of the UAV combat method proposed in this embodiment, the fixed-wing UAV control system is firstly integrated and designed in Matlab / Simulink and Python environment, and a simulation experiment is carried out. The main simulation process is as follows.

[0224] In the simulation test, the parameters of the drone are set as follows: g = 9.8m / s 2 The mass of the drone is m = 1kg, and the moment of inertia matrix is The air density is ρ = 1.205 kg / m at 20 degrees Celsius and a standard atmospheric pressure. 3 , the speed of the drone is constant V = 15m / s, and the wing reference area S w =0.32m 2 , lift coefficient C L =1.25, lateral force coefficient C Yβ =-0.3.

[0225] For the angular rate controller, the reinforcement learning algorithm hyperparameters are set to Q τ =5I3,R τ =0.8I3, where I3 is the third-order unit matrix, the discount factor γ0 = 0.001, the integration time interval T = 0.045s, and a total of N = 400 segments of integration data were collected. In the iterative learning process, the policy network parameters The second norm of changes with the number of iterations is as follows Figure 5 shown.

[0226] For the attitude angle controller, the reinforcement learning algorithm hyperparameters are set to Q ω =5I3,R ω =0.8I3, discount factor γ1 = 0.001, integration time interval T = 0.045s, a total of N = 400 segments of integration data were collected. In the iterative learning process, the policy network parameters The second norm of changes with the number of iterations is as follows Figure 6 As shown in Figure 1, in order to test the performance of the learned optimal attitude angle controller, a reference signal is given, where the maximum roll angular rate is about 40° / s.

[0227] α r =0.5236,

[0228] β r =0,

[0229] μ r =1.19sin(0.7t),

[0230] The angle response curve of the drone is as follows: Figure 7 As shown in the figure, it can be seen that the UAV attitude angle tracks the reference attitude angle well and has a fast response speed.

[0231] For the air combat decision network, the hyperparameters are set as follows: the maximum time limit for each round is 40 seconds, and a total of 1,000 rounds are played. The initial state of the drone is randomly generated within a suitable range, the learning rate is 0.0003, the batch capacity is 256, and the experience replay pool D size is 1,000,000. Policy network π1, soft action value network State Value Network The structure of is a fully connected network with three hidden layers and 256 neurons in each layer. The Adam optimizer is selected for gradient descent, with the momentum parameter set to 0.9 and the gamma parameter set to 0.99.

[0232] During the training process, the average total step length per game changes as follows Figure 8 The average cumulative reward per game changes as shown below: Fig. 9 As shown in the figure, we can see that the average total step length of the game shows a trend of increasing first and then decreasing. The reason is that at the beginning, our strategy tends to be a random strategy, and the enemy can easily win, thus shortening the total step length of the game. As the training progresses, it becomes increasingly difficult for the enemy to win, and the average total step length shows an upward trend until our side learns a better air combat strategy and can quickly win, and the average total step length decreases again. As for the average cumulative reward of our side per game, it gradually increases as the training progresses.

[0233] In order to more intuitively demonstrate the drone combat control method provided in the embodiment of the present application, from the perspective of the red team, three representative air combat initial positions are selected for verification. The three initial positions are shown in Table 1 below.

[0234] Table 1

[0235]

[0236] When the initial state is an offensive posture, the air combat trajectory is as follows Fig.10 As shown; when the initial state is defensive, the air combat trajectory is as follows Fig.11 As shown; when the initial state is moving towards each other, the air combat trajectory is as follows Fig.12 Shown

[0237] Another aspect of the embodiment of the present application provides a drone combat control device. Fig.13 A schematic diagram of the structure of a drone combat control device provided in an embodiment of the present application is shown. Fig.13 As shown, the UAV combat control device may include the following modules.

[0238] The first information acquisition module 1301 is used to obtain the status information of friendly and enemy drones.

[0239] The decision module 1302 is used to obtain target reference angle information through the air combat decision network based on the status information of enemy and friendly UAVs.

[0240] The attitude angle control module 1303 is used to input the target reference angle information into the attitude angle controller to obtain the target reference angular rate information.

[0241] An angular rate control module 1304 is used to input the target reference angular rate information into an angular rate controller to obtain target control torque information;

[0242] The second information obtaining module 1305 is used to input the target control torque information into the constructed UAV dynamics model to obtain the control state information of our UAV.

[0243] In the above embodiment, the air combat decision network outputs the target reference angle information tracked by the attitude angle controller based on the state information of the enemy and friendly UAVs, the attitude angle controller outputs the target reference angular velocity information tracked by the angular velocity controller, and the angular velocity controller outputs the reference torque information, and the reference torque information is directly input into the attitude dynamic model of the UAV, so that the control state information of the UAV can follow the attitude reference signal. In addition, by forming two inner and outer loops by the angular velocity controller and the attitude angle controller, the optimal control of the attitude angle and angular velocity of the UAV can be achieved without knowing the detailed dynamic information of the UAV model, and the UAV of our side can be guided to occupy a favorable position, thereby solving the adverse effects brought by the particle model.

[0244] In some embodiments, the decision module 1302 is used to construct a feature vector based on the state information of the enemy and friendly drones, and input the feature vector into an air combat decision network trained based on a Markov model to use the air combat decision network to output target reference angle information.

[0245] In some embodiments, the decision module 1302 is specifically used to solve the optimal solution of the air combat decision network based on the reinforcement learning algorithm SAC to output the target reference angle information.

[0246] In some embodiments, the drone combat control device may further include an angular rate controller acquisition module, which is used to obtain the angular rate controller based on a policy reinforcement learning algorithm.

[0247] The angular rate controller acquisition module is specifically used to construct an angular rate controller based on an augmented system of angular rate dynamics, wherein the value function and the strategy function in the angular rate controller are both represented by neural networks; based on multiple iterations of control strategies, control information and status information of our UAV in historical periods, the angular rate controller is iteratively updated until the convergence conditions are met.

[0248] In some embodiments, the UAV combat control device may further include an attitude angle controller acquisition module, which is used to obtain the attitude angle controller based on a strategy reinforcement learning algorithm.

[0249] The attitude angle controller acquisition module is specifically used to construct an attitude angle controller based on an angle dynamic augmented system, wherein the value function and the strategy function in the attitude angle controller are both represented by neural networks; based on multiple iterations of control strategies, the control information and state information of our UAV in the target period, the attitude angle controller is iteratively updated until the convergence conditions are met.

[0250] In some embodiments, the device also includes a model building module for building a UAV dynamics model, wherein the UAV dynamics model includes: a position dynamics model and an attitude dynamics model, and the attitude dynamics model includes an angular rate dynamics model and an angle dynamics model.

[0251] In some embodiments, the device also includes an enemy strategy setting module, which is used to set the enemy drone strategy by an α-β pruning Minimax search algorithm.

[0252] It should be understood that the specific features, operations and details described hereinabove with respect to the method of the present application may also be similarly applied to the device and system of the present application, or vice versa. In addition, each step of the method of the present application described above may be performed by a corresponding component or unit of the device or system of the present application.

[0253] It should be understood that each module / unit of the device of the present application can be implemented in whole or in part by software, hardware, firmware or a combination thereof. Each module / unit can be embedded in the processor of the electronic device or independent of the processor in the form of hardware or firmware, or can be stored in the memory of the electronic device in the form of software for the processor to call to perform the operation of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0254] like Fig.14 As shown, the present application provides an electronic device 1400, which includes a processor 1401 and a memory 1402 storing computer program instructions. When the processor 1401 executes the computer program instructions, each step of the above-mentioned unmanned combat control method is implemented. The electronic device 1400 can be generally a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities.

[0255] In one embodiment, the electronic device 1400 may include a processor, a memory, a network interface, a communication interface, etc. connected through a system bus. The processor of the electronic device 1400 may be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device 1400 may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system, a computer program, etc. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device 1400 may be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the method of the present application are executed.

[0256] The present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned drone combat control method is implemented.

[0257] Those skilled in the art will appreciate that the method steps of the present application can be completed by instructing related hardware such as the electronic device 1400 or a processor through a computer program, and the computer program can be stored in a non-temporary computer-readable storage medium, and the computer program causes the steps of the present application to be executed when it is executed. Depending on the circumstances, any reference to memory, storage, or other media herein may include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0258] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0259] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A drone combat control method, characterized in that: The method comprises: Obtain status information of enemy and friendly drones; Based on the state information of the friendly and enemy UAVs, target reference angle information is obtained through an air combat decision network; Inputting the target reference angle information into the attitude angle controller to obtain target reference angular rate information; Inputting the target reference angular velocity information into an angular velocity controller to obtain target control torque information; The target control torque information is input into the constructed UAV dynamics model to obtain the control state information of our UAV.

2. The method according to claim 1, characterized in that Based on the state information of the enemy and friendly UAVs, the target reference angle information obtained through the air combat decision network includes: A feature vector is constructed based on the state information of the friendly and enemy UAVs, and the feature vector is input into an air combat decision network trained based on a Markov model, so as to utilize the air combat decision network to output target reference angle information.

3. The method according to claim 2, characterized in that Outputting target reference angle information by using the air combat decision network includes: The optimal solution of the air combat decision network is solved based on the reinforcement learning algorithm SAC to output the target reference angle information.

4. The method according to claim 1, characterized in that The method further includes: a step of obtaining the angular rate controller based on a policy reinforcement learning algorithm, specifically including: An angular rate controller is constructed based on an augmented system of angular rate dynamics, wherein a value function and a strategy function in the angular rate controller are both represented by a neural network; Based on multiple iterations of control strategies, control information of our UAV in historical periods, and state information, the angular rate controller is iteratively updated until convergence conditions are met.

5. The method according to claim 1, characterized in that The method further includes: a step of obtaining the attitude angle controller based on a policy reinforcement learning algorithm, specifically including: A posture angle controller is constructed based on an augmented system of angle dynamics, wherein both the value function and the strategy function in the posture angle controller are represented by a neural network; Based on the multiple iterations of the control strategy, the control information and the state information of our UAV in the target period, the attitude angle controller is iteratively updated until the convergence condition is met.

6. The method according to claim 1, characterized in that Before obtaining the status information of the enemy and friendly drones, the method further includes: A UAV dynamics model is constructed, wherein the UAV dynamics model includes a position dynamics model and an attitude dynamics model, wherein the attitude dynamics model includes an angular rate dynamics model and an angle dynamics model.

7. The method according to claim 1 or 2, characterized in that: The method further comprises: The enemy drone strategy is set by the α-β pruning Minimax search algorithm.

8. A drone combat control device, characterized in that: The device comprises: The first information acquisition module is used to obtain the status information of enemy and friendly UAVs; A decision module, used to obtain target reference angle information through an air combat decision network based on the state information of the enemy and friendly UAVs; An attitude angle control module is used to input the target reference angle information into an attitude angle controller to obtain target reference angular rate information; An angular rate control module, used for inputting the target reference angular rate information into an angular rate controller to obtain target control torque information; The second information obtaining module is used to input the target control torque information into the constructed UAV dynamics model to obtain the control state information of our UAV.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the drone combat control method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the drone combat control method according to any one of claims 1 to 7 is implemented.

Images