Unmanned group game confrontation strategy generation method and device

Abstract

The invention provides an unmanned group game confrontation strategy generation method and apparatus. The method comprises the steps of establishing a kinetic model of an unmanned aerial vehicle based on position information and attitude information of the unmanned aerial vehicle; wherein the unmanned aerial vehicles comprise an own unmanned aerial vehicle and an enemy unmanned aerial vehicle, and the attitude information comprises Euler angles of the unmanned aerial vehicles; based on the dynamical model, after interference terms generated by position and attitude control coupling of the unmanned aerial vehicle are removed, an augmentation system equation of the own unmanned aerial vehicle and the enemy unmanned aerial vehicle is established; based on the augmented system equation, establishing a corresponding game performance function; based on the game performance function, establishing a Bellman equation corresponding to the augmented system equation; iteratively solving the Bellman equation by using a least square method to obtain virtual control input of the own unmanned aerial vehicle and the enemy unmanned aerial vehicle; and controlling the own unmanned aerial vehicle to follow the enemy unmanned aerial vehicle based on the virtual control input of the own unmanned aerial vehicle and the enemy unmanned aerial vehicle. A model-free differential game algorithm is designed, and the algorithm is applied to iterative interaction with the environment and other unmanned aerial vehicles to obtain an optimal pursuit strategy.

Description

Technical Field

[0001] The present application relates to the field of aircraft control technology, and in particular to a method and device for generating an unmanned group game confrontation strategy. Background Art

[0002] In the past few decades, the game confrontation of unmanned swarm systems has been one of the research hotspots. Especially for drone swarm systems, due to their stronger collaborative perception, decision-making and execution capabilities, drone swarms have gradually become one of the carriers of game confrontation. However, in the game, it is difficult for us to obtain accurate model information of the enemy group, which puts our unmanned group at a disadvantage. Therefore, it is very important to generate a confrontation game strategy when the model information is unknown.

[0003] Among them, the pursuit-escape game problem is the core problem of the entire adversarial game. In previous studies, the pursuit-escape problem is often described as a differential game problem, which is solved based on optimal control theory. For example, some scholars have obtained the solution of the game by solving the augmented algebraic Riccati equation; some scholars have expanded this method and realized the offline solution of the Riccati equation. However, solving this equation requires obtaining relevant information of the entire game system. Due to the high nonlinearity of the model and the inaccuracy of modeling, the information is not completely known. Therefore, it is necessary to solve the problem of generating game strategies under unknown system information. Summary of the invention

[0004] The present application provides a method and device for generating an unmanned group game confrontation strategy, which is used to solve the problem of generating a game strategy under unknown system information in the prior art.

[0005] According to the first aspect of the present application, a method for generating an unmanned group game confrontation strategy is provided, the method comprising: establishing a dynamic model of the drone based on the position information and attitude information of the drone; wherein the drone includes a own drone and an enemy drone, and the attitude information includes the Euler angle of the drone; establishing an augmented system equation of the own drone and the enemy drone based on the dynamic model; establishing its corresponding game performance function based on the augmented system equation; establishing a Bellman equation corresponding to the augmented system equation based on the game performance function; iteratively solving the Bellman equation using the least squares method to obtain virtual control inputs of the own drone and the enemy drone; and controlling the own drone to pursue the enemy drone based on the virtual control inputs of the own drone and the enemy drone.

[0006] Optionally, the method also includes: removing interference terms generated by the coupling of the position and attitude control of the UAV from the dynamic model; establishing the augmented system equations of the own UAV and the enemy UAV based on the dynamic model includes: establishing the augmented system equations of the own UAV and the enemy UAV based on the dynamic model after removing the interference terms.

[0007] Optionally, a Bellman equation corresponding to the augmented system equation is established based on the game performance function, including: establishing an equivalent equation of the Riccati equation of the game performance function based on the game performance function and the saddle point condition of the Nash equilibrium; establishing the Bellman equation corresponding to the augmented system equation based on the equivalent equation of the Riccati equation and the augmented system equation using the linear system fitting principle.

[0008] Optionally, the unmanned group game confrontation strategy generation method of the present application also includes: obtaining relative state information of one's own UAV and the enemy UAV; wherein the relative state information includes: the angle difference between the velocity vector of one's own UAV and the line of sight angle of the two UAVs, the angle difference between the velocity vector of the enemy UAV and the line of sight angle of the two UAVs, the modulus of the distance difference between the two UAVs, the velocity vector angle between the two UAVs and the modulus of the speed difference between the two UAVs; based on the relative state information, determining the threat value to which one's own UAV is subjected; wherein the threat value includes the angle threat, distance threat and speed threat to which one's own UAV is subjected; determining the Markov state transition probability of one's own UAV based on the threat value to which one's own UAV is subjected; based on the Markov state transition probability of one's own UAV, and using the reinforcement learning method to solve the cumulative discounted reward function of one's own UAV, obtain the target enemy UAV assigned to one's own UAV.

[0009] Optionally, the unmanned group game confrontation strategy generation method of the present application also includes: obtaining the position, speed and posture information of the enemy drone; based on the position, speed and posture information of the enemy drone, and using a pre-trained fuzzy neural network model to predict the behavior pattern of the enemy drone.

[0010] According to a second aspect of the present invention, there is provided an unmanned group game confrontation strategy generation device, comprising: a dynamic model establishment module, used to establish a dynamic model of the unmanned aerial vehicle based on the position information and posture information of the unmanned aerial vehicle; wherein the unmanned aerial vehicle includes a own unmanned aerial vehicle and an enemy unmanned aerial vehicle, and the posture information includes the Euler angle of the unmanned aerial vehicle; an augmented system establishment module, used to establish an augmented system equation of the own unmanned aerial vehicle and the enemy unmanned aerial vehicle based on the dynamic model; a game performance function establishment module, used to establish its corresponding game performance function based on the augmented system equation; a Bellman equation establishment module, used to establish the Bellman equation corresponding to the augmented system equation based on the game performance function; a virtual control input determination module, used to iteratively solve the Bellman equation using the least squares method to obtain the virtual control input of the own unmanned aerial vehicle and the enemy unmanned aerial vehicle; a control module, used to control the own unmanned aerial vehicle to pursue the enemy unmanned aerial vehicle based on the virtual control input of the own unmanned aerial vehicle and the enemy unmanned aerial vehicle.

[0011] Optionally, the device also includes: an interference term removal module, used to remove the interference terms generated by the position and attitude control coupling of the UAV from the dynamic model; the augmented system establishment module is further used to establish the augmented system equations of the own UAV and the enemy UAV based on the dynamic model after removing the interference terms.

[0012] Optionally, the game performance function establishment module further includes a first submodule for establishing an equivalent equation of the Riccati equation of the game performance function based on the game performance function and the saddle point condition of the Nash equilibrium; and a second submodule for establishing the Bellman equation corresponding to the augmented system equation based on the equivalent equation of the Riccati equation and the augmented system equation using the linear system fitting principle.

[0013] Optionally, the unmanned group game confrontation strategy generation device of the present application also includes: a first acquisition module, used to obtain relative state information between the own drone and the enemy drone; wherein the relative state information includes the angle difference between the velocity vector of the own drone and the line of sight angle of the two drones, the angle difference between the velocity vector of the enemy drone and the line of sight angle of the two drones, the modulus of the distance difference between the two drones, the velocity vector angle between the two drones, and the modulus of the speed difference between the two drones; a threat value determination module, used to determine the threat value of the own drone based on the relative state information; wherein the threat value includes the angle threat, distance threat and speed threat to the own drone; a Markov state transition probability determination module, used to determine the Markov state transition probability of the own drone based on the threat value to the own drone; a target allocation module, used to solve the cumulative discounted reward function of the own drone based on the Markov state transition probability of the own drone and using the reinforcement learning method to obtain the target enemy drone assigned to the own drone.

[0014] Optionally, the unmanned group game confrontation strategy generation device of the present application also includes: a second acquisition module, used to obtain the position, speed and posture information of the enemy drone; a behavior pattern prediction module, used to predict the behavior pattern of the enemy drone based on the position, speed and posture information of the enemy drone and using a pre-trained fuzzy neural network model.

[0015] According to a third aspect of the present application, an electronic device is provided, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, any of the above-mentioned methods for generating unmanned group game confrontation strategies is implemented.

[0016] According to a fourth aspect of the present application, a computer-readable storage medium is provided, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the above-mentioned unmanned group game confrontation strategy generation methods is implemented.

[0017] In summary, the unmanned group game confrontation strategy generation method and device provided by the present application have at least the following beneficial effects:

[0018] The unmanned group game confrontation strategy generation method of the present application includes: establishing a dynamic model of the drone based on the position information and attitude information of the drone; wherein the drone includes a self-drone and an enemy drone, and the attitude information includes the Euler angle of the drone; establishing the augmented system equations of the self-drone and the enemy drone based on the dynamic model; establishing the corresponding game performance function based on the augmented system equation; establishing the Bellman equation corresponding to the augmented system equation based on the game performance function; using the least squares method to iteratively solve the Bellman equation to obtain the virtual control input of the self-drone and the enemy drone; controlling the self-drone to pursue the enemy drone based on the virtual control input of the self-drone and the enemy drone. The present application realizes obtaining the control input of both the self-drone and the enemy based on the differential game algorithm when the drone model is unknown, thereby obtaining the pursuit strategy of the self-drone against the enemy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a method for generating an unmanned group game confrontation strategy provided in an embodiment of the present application;

[0021] Figure 2 A flow chart for generating the optimal differential game strategy in the game confrontation provided by the embodiment of the present application;

[0022] Figure 3 A flow chart of a target allocation method provided for an embodiment of the present application;

[0023] Figure 4 A flowchart of target behavior prediction provided for an embodiment of the present application;

[0024] Figure 5 A fuzzy neural network structure diagram provided for an embodiment of the present application;

[0025] Figure 6 A flowchart of another method for generating an unmanned group game confrontation strategy provided by an embodiment of the present application;

[0026] Figure 7 A diagram of the win rate evolution during the training phase provided for an embodiment of the present application;

[0027] Figure 8 A 3V3 game scenario trajectory diagram provided for an embodiment of the present application;

[0028] Fig. 9 A performance function evolution diagram of the first group of pursuers in a 3V3 game scenario provided in an embodiment of the present application;

[0029] Fig.10 A structural diagram of an unmanned group game confrontation strategy generation device provided in an embodiment of the present application;

[0030] Fig.11 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above and other features and advantages of the present application more clear, the present application is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0032] In the following description, many specific details are set forth to provide a thorough understanding of the present application. However, it is apparent to those skilled in the art that specific details need not be adopted to practice the present application. In other cases, well-known steps or operations are not described in detail to avoid blurring the present application.

[0033] The unmanned group game confrontation strategy generation method provided in the embodiment of the present application can be executed by the unmanned group game confrontation strategy generation device provided in the embodiment of the present application, and the device can be configured in an electronic device.

[0034] refer to Figure 1 The present application provides a method for generating an unmanned group game confrontation strategy, the method comprising:

[0035] S110, establishing a dynamic model of the drone based on the position information and attitude information of the drone; wherein the drone includes a friendly drone and an enemy drone, and the attitude information includes the Euler angle of the drone.

[0036] Specifically, define p j =[p x,j p y,j p z,j ] T ∈R 3×1 is the position of UAV j in the inertial coordinate system, Ω j =[φ j θ j ψ j ] T ∈R 3×1is its Euler angle, which is used to describe its attitude information. Then the dynamic model of UAV j is:

[0037]

[0038] Among them, M j is the mass of drone j, is its inertia matrix, represents the acceleration of UAV j in the inertial coordinate system, represents the angular acceleration of UAV j, represents the angular velocity of UAV j, is the nonlinear Coriolis term, is the rotation matrix from the body coordinate system to the inertial coordinate system, τ j is the control torque generated by the propeller, is the control force generated by the propeller. k,j is the rotational angular velocity of the kth propeller of UAV j, k = 1, 2, 3, 4, g is the acceleration due to gravity. 3,3 ∈R 3×1 Represents a three-dimensional vector, where the third parameter is 1 and the rest are 0. t,j ,n ω,j and n τ,j All are normal numbers.

[0039] To simplify the expression, define For the control input instruction, define {p x,j p y,j p z,j ψ j} is the system status, then:

[0040]

[0041] in, u τ,j =[u φ,j u θ,j u ψ,j ] T ∈R 3×1 .u virtual,j is the virtual control input of UAV j, defined as:

[0042]

[0043] Among them, φ d,j ,θ d,j and ψ d,j are the expected attitude angles of UAV j, s(·) and c(·) represent sin(·) and cos(·), respectively. is a coupling term related to the state.

[0044] S120, establish the augmented system equations of the friendly UAV and the enemy UAV based on the dynamic model.

[0045] Preferably, according to formula (2), the interference term generated by the coupling of the position and attitude control of the UAV can be removed first: Then, based on the dynamic model after removing the interference term, the augmented system equations of the friendly UAV and the enemy UAV are established, where the position dynamic model of the friendly UAV j can be rewritten as:

[0046]

[0047] in, A=[0 6×3 c 6,1 c 6,2 c 6,3 ]∈R 6×6 , B=[c 6,4 c 6,5 c 6,6 ]∈R 6×3 ,

[0048] U j =b p,j u virtual,j -gc 3,3 ∈R 3×1 Similarly, the position dynamics model of the enemy UAV k can be obtained as:

[0049]

[0050] Where C represents the input matrix. If the enemy drone and the friendly drone have the same model, then C = B. In the pursuit-escape game modeling, the goal of drone j is to catch up with drone k on the basis of minimum energy consumption, while drone k will try its best to avoid the pursuit of drone j. According to the above dynamics formula, the augmented system equation can be obtained as follows:

[0051]

[0052] in, In the augmented state.

[0053] S130, establishing a corresponding game performance function based on the augmented system equation.

[0054] Specifically, the game performance function established based on the augmented system equation is:

[0055]

[0056] Where Q = [c 6,1 c 6,2 c6,3 0 6×3 ]∈R 6×6 is the weight matrix, R∈R 3×3 >0,S∈R 3×3 < 0. Friendly UAV j hopes to maximize performance function J, while enemy UAV k tries to minimize J.

[0057] S140, establishing the Bellman equation corresponding to the augmented system equation based on the game performance function.

[0058] One implementation method includes: establishing an equivalent equation of the Riccati equation of the game performance function based on the game performance function and the saddle point condition of the Nash equilibrium; establishing the Bellman equation corresponding to the augmented system equation based on the equivalent equation of the Riccati equation and the augmented system equation using the linear system fitting principle.

[0059] Specifically, in order to ensure that the friendly drone can chase the enemy drone, for the above-mentioned game performance function, the saddle point condition of the Nash equilibrium is:

[0060]

[0061] The existing solution method is to obtain the optimal control input for both parties by solving the Riccati algebraic equation, namely:

[0062]

[0063] At this time, the corresponding optimal input is:

[0064]

[0065] However, since P in equation (9) is often a large-scale sparse matrix, it is difficult to obtain its analytical solution directly by solving the equation. A common method is to obtain its approximate solution by iterative solution. Define K j,0 ,K k,0 is the feedback control input matrix of any stabilizable system, that is P n >0 is a positive solution to the following equation:

[0066]

[0067] Among them, K j,n ,K k,n ,n=1,2,...satisfies the following equation:

[0068]

[0069] Then, the following properties can be obtained: * ≤P n+1 ≤Pn ,

[0070] lim n→∞ P n =P * .

[0071] However, due to the uncertainty of model parameters and modeling errors, it is difficult to obtain an accurate system model in the actual solution process, and it is difficult to obtain the optimal game control input based on the above method.

[0072] In this paper, a data-driven differential adversarial game algorithm is designed, which can obtain the optimal control input of both parties under the condition that the system model is unknown. Then formula (6) can be rewritten as:

[0073]

[0074] When the time interval δt>0, according to the linear system The fitting principle and taking into account equations (11), (12) and (13) yields the Bellman equation:

[0075]

[0076] Among them, Q n is the augmented state gain matrix, Q n =Q+K j,n T R j,n +K k,n T SK k,n , Q is the constant gain matrix, R is the first state gain matrix, and S is the second state gain matrix. For the augmented state, p j =[p x,j p y,j p z,j ] T ∈R 3×1 is the position of UAV j in the inertial coordinate system, p k =[p x,k p y,k p z,k ] T ∈R 3×1 is the position of UAV k in the inertial coordinate system, A=[0 6×3 c 6,1 c 6,2 c 6,3 ]∈R 6×6 , B=[c 6,4 c6,5 c 6,6 ]∈R 6×3 , K j,n is the feedback control input matrix of UAV j, K k,n is the feedback control input matrix of UAV k, U j is the control quantity of UAV j, U k is the control quantity of UAV k.

[0077] Based on equations (11), (12) and (13), the proof process of equation (14) is as follows: Considering is continuously differentiable, we can get:

[0078]

[0079] Substituting equation (13) into equation (15), the Bellman equation can be further written as:

[0080]

[0081] Considering After mathematical operations, we can get:

[0082]

[0083] Finally, substituting equations (11) and (12) into the above equation, we can obtain:

[0084]

[0085] According to the definition of differential, we can finally get equation (14), and the proof is complete.

[0086] S150, using the least square method to iteratively solve the Bellman equation to obtain the virtual control inputs of the friendly UAV and the enemy UAV.

[0087] In order to simplify the calculation process and make full use of historical data, this application obtains the required parameters through the least squares method.

[0088] definition

[0089]

[0090] Among them, p ij represents the value of the i-th row and j-th column of the matrix P, Representation Matrix The value of the i-th row, then the right side of the equal sign of formula (14) can be further rewritten as:

[0091]

[0092] Define z as the number of integrations and define the following operator:

[0093]

[0094] Then formula (14) can be finally rewritten as:

[0095]

[0096] in,

[0097]

[0098] If Ω n If it is a full-rank matrix, K can be optimized by the least squares method. j,n ,K k,n :

[0099]

[0100] Observing the above derivation process, we can get that the system matrix A in the system model is Replace, B and The controlled quantity U j and U k Replacement, and the system state and control quantity can be measured online, that is, the optimal strategy can be generated by collecting system information in the game environment. The algorithm process is as follows Figure 2 First, the UAV control input U that can stabilize the system is initially introduced j,0 ,U k,0 And add exploration noise e to the control input j,0 ,e k,0 , and then collect the drone status data and drone control input data in the game system to obtain the system input U j,0 =u j,0 +e j,0 and U k,0 =u k,0 +e k,0 ; Then, the updated control input is iteratively solved through the Bellman equation to determine whether the control input converges. If it converges, the control input is the optimal strategy. Otherwise, the iterative solution process is repeated until convergence.

[0101] Once the optimal control input U is determined j,n+1 ,U k,n+1 , then the virtual control input can be obtained according to the system equation:

[0102]

[0103] S160, controlling the own UAV to chase the enemy UAV based on the virtual control input of the own UAV and the enemy UAV.

[0104] Specifically, after obtaining the virtual control input, taking the own UAV j as an example, the desired attitude angle {φ d,j ,θ d,j} and thrust input can be calculated by the following formula:

[0105]

[0106] Among them, s(·), c(·), as(·) represent sin(·), cos(·), arcsin(·) respectively. Therefore, the attitude control of the own UAV can be realized based on control methods such as PID or MPC to pursue the enemy UAV.

[0107] In one embodiment, in order to improve the efficiency of target allocation in group games, refer to Figure 3 The embodiment of the present application also provides a target allocation method, that is, a strategy for allocating target enemy drones to one's own drones. That is, in a group game confrontation, it is necessary to first allocate the target enemy drones to be confronted to one's own drones, so as to further generate a game confrontation strategy for the target enemy drones for one's own drones. The target allocation method described in the embodiment of the present application includes:

[0108] S310, obtaining relative state information of the own drone and the enemy drone; wherein the relative state information includes the angle difference between the velocity vector of the own drone and the sight angle of the two drones, the angle difference between the velocity vector of the enemy drone and the sight angle of the two drones, the modulus of the distance difference between the two drones, the velocity vector angle between the two drones, and the modulus of the velocity difference between the two drones. The two drones are the own drone and the enemy drone.

[0109] Specifically, assuming that the number of our drone group is N red Equal to or greater than the number of enemy drone groups N blue , and in the game process, the own drone j,j∈N red We can obtain the enemy drone k, k∈N blue Relative status information in represents the angle difference between the velocity vector of UAV j and the sight angles of the two UAVs; represents the angle difference between the velocity vector of UAV k and the sight angles of the two UAVs; The modulus representing the distance difference between the two drones; Represents the velocity vector angle between the two drones; The modulus that represents the velocity difference between the two drones.

[0110] S320, determining the threat value of the own UAV based on the relative state information; wherein the threat value includes the angle threat, distance threat and speed threat of the own UAV.

[0111] Specific, definition is the set of N enemy drones closest to drone j. In order to further describe the game environment in which drone j is located, define Describe the drone j is subject to from the drone Threats:

[0112]

[0113] The right side represents the angle threat, distance threat, and speed threat to drone j, and the specific calculation method is:

[0114]

[0115] Where, d, V max Represent the minimum avoidance distance and maximum speed respectively.

[0116] S330, determining the Markov state transition probability of the own UAV based on the threat value to the own UAV.

[0117] The Markov state transition probability of one's own UAV can be determined based on historical data, that is, the Markov state transition probability of one's own UAV can be determined based on historical UAV confrontation scenarios and predefined rules.

[0118] According to formula (26), the total threat to the own UAV j can be calculated as:

[0119]

[0120] The optimal target allocation strategy can be implemented based on reinforcement learning theory: Definition is the action space of the own drone j, where 0 means that drone j does not make any allocation. Indicates that the assignment target of UAV j is k∈N blue enemy drones, that is, our drone j will attack the kth enemy drone. Define Π j ={π j (1),π j (2),...,π j (t)} is the sequential decision action of own drone j, where π j (t)∈A represents the target allocation result of drone j at time t. Optimal allocation target result It is not only related to itself, but also to the environment. In the reinforcement learning algorithm, the Markov state transition probability of the own drone j is defined as:

[0121] P j {μ j ′|π j ,μ j}=P{μ j (t+1)|π j (t),μ j (t)}; (29)

[0122] Among them, π j (t),μ j (t) represents the action and threat level of the own UAV j at time t, μ j (t+1) is the threat level at time t+1, and the state transition probability can be determined using historical data and expert knowledge.

[0123] S340, based on the Markov state transition probability of the own UAV, and using the reinforcement learning method to solve the cumulative discounted reward function of the own UAV, obtain the target enemy UAV assigned to the own UAV.

[0124] Specifically, for the time length 0<t<∞, after a given discount factor β∈(0,1), the own drone j needs to maximize the cumulative discounted reward function R j :

[0125]

[0126] Among them, r j (π j (t),μ j (t)) is the reward value of environmental feedback, which is inversely proportional to the threat level and is defined as:

[0127]

[0128] The above process transforms the allocation problem of the own drone j into the problem of finding the maximum cumulative discounted reward function through reinforcement learning. According to formula (30), it can be obtained that the target allocation process of drone j is affected by the state of the enemy drone. Define the optimal reward amount and the optimal sequential decision action to satisfy the following formula:

[0129]

[0130] The above optimal values ​​can then be solved based on the Q-learning algorithm to obtain the target enemy drones assigned by the own drones.

[0131] In addition, in some game scenarios, for example, there is a ground target in N red At the same time, N blueIn order to confuse and consume the friendly drones, the enemy drone swarm has three behavior modes: penetration, feint attack and escape. Feint attack means that the enemy drone swarm does not attack any target when flying over, including friendly drones and ground targets; escape means that the enemy drone flies back to the base when detecting the attack of friendly drones; penetration means that the enemy drone attacks at least one ground area or friendly drones. Therefore, a behavior pattern prediction algorithm is needed to obtain the behavior pattern of the enemy drone, so as to obtain a matching game confrontation strategy based on the predicted behavior pattern.

[0132] In recent years, researchers have proposed many behavior prediction methods, such as behavior analysis and prediction methods based on clustering and topic modeling, prediction methods based on long short-term memory neural networks and gated recurrent unit networks, etc. These methods are data-driven and require all system information, which results in low computational accuracy and efficiency.

[0133] Therefore, in order to improve the computational accuracy and efficiency of predicting the behavior pattern of enemy drones in group games, reference Figure 4 In some embodiments of the present application, the unmanned group game confrontation strategy generation method of the present application also provides a behavior pattern prediction strategy for enemy unmanned aerial vehicles, which strategy includes:

[0134] S410, obtaining the position, speed and attitude information of the enemy UAV.

[0135] S420, based on the position, speed and attitude information of the enemy UAV, and using a pre-trained fuzzy neural network model, predicts the behavior pattern of the enemy UAV.

[0136] Specifically, this application predicts the behavior pattern of enemy drones based on the prediction algorithm of the fuzzy neural network model. Figure 5 , the fuzzy neural network model is constructed as follows:

[0137] First, define the position, speed and attitude information S of the enemy drone blue is the input of the fuzzy network. Considering that there are N blue enemy drones, the entire input vector can be described as There are N blue ×S blue Then, define a blue ×S blue ×m nodes fuzzy layer, where m is the fuzzy node of each input node; in the present invention, since there are three possible behavior modes of the enemy UAV, m = 3. Each input node corresponds to m fuzzy nodes, and each fuzzy node has a corresponding membership function:

[0138]

[0139] Where q=1,...,N blue ×S blue ; p = 1, ..., m. Then, a fuzzy rule calculation layer is constructed, which contains three nodes. Each node represents a rule:

[0140]

[0141] Finally, an output layer is introduced to add and output the numerical results through the softmax function:

[0142]

[0143] Among them, ω p represents the weights of different rules, and softmax(·) is the activation function. The final behavior pattern can be predicted based on the output value. p represents the rule parameter of the pth rule calculation node, m represents the number of rule calculation nodes, and y represents the output of the output node.

[0144] In the initial stage, training data can be constructed based on the fused observation data of the enemy drone group; in the training stage, the number of nodes in each layer needs to be determined in advance, and the hyperparameters that need to be trained are mainly c qp ,ω p , the optimal parameters can be obtained by stochastic gradient descent and reverse optimization. This method can further reduce the search space scale of the reverse optimization method and improve the training speed of the fuzzy neural network.

[0145] See also Figure 6 The overall game confrontation process shown in the figure first collects the information required for predicting the behavior of the enemy drone, including position, speed and attitude information, and obtains the optimal parameters based on the trained fuzzy neural network. c qp ,ω p , thereby predicting the enemy UAV behavior pattern based on the optimal parameters; collecting the information required for assigning target enemy UAVs, and using the optimal target assignment strategy obtained based on reinforcement learning to determine the target enemy UAVs assigned to one's own UAVs; then, collecting the information required for the game confrontation strategy (for specific information and process, refer to Figure 1 Corresponding introduction), obtain the optimal virtual input parameters of the own drone and the enemy drone, use the currently obtained drone state information and the differential game strategy to obtain the game confrontation result, that is, obtain the strategy for the own drone to pursue the assigned target enemy drone.

[0146] To facilitate understanding of the present application, the present application scheme is introduced below by taking a specific embodiment as an example. Assume that each party has 3 drones participating in the game confrontation, and the parameters of the drones are shown in Table 1.

[0147] Table 1 UAV parameters

[0148]

[0149]

[0150] At the initial stage, the initial positions of the 6 drones are: red,1 (0) = [-200-100-100] T m, p red,2 (0) = [200100 100] T m, p red,3 (0) = [500 -200 -400] T m, p blue,1 (0) = [-350-50100] T m, p blue,2 (0) = [50300200] T m, p blue,3 (0) = [200-500-500] T m, the initial velocity is 0, and the position of the ground target is

[000] T m. β in equation (30) is set to 0.8, and the weight matrix in equation (7) is set to R = I 3×3 ,S=-I 3×3 ,||P n+1 -P n The constant in ||<ε is chosen to be 0.1. In order to meet the continuous excitation condition required by the system, exploration noise is added to the control input: e j,0 =e k,0 =0.4e -0.3t sin 2 (1.4t)cos(5t)+0.4e -0.3t cos(3t)sin 3 (5t).

[0151] In the training phase, the enemy drone’s behavior pattern is a random behavior pattern among the three behavior patterns of attack, feint and escape. In this phase, 500 training iterations are conducted, and 100 confrontation games are conducted in parallel in each training. The evolution of the winning rate in the training phase is shown in the figure below: Figure 7As shown in the figure, the winning rate of our own drones gradually increases from 40% to about 90%. In the game verification stage, three enemy drones fly towards ground targets, and their predetermined intention is to directly attack ground targets. When being detected by our own drones, the behavior pattern prediction algorithm is executed. At this time, the behavior pattern of the enemy drone is predicted to be an attack with a confidence level of 0.8. Subsequently, the target allocation algorithm is executed, and the target allocation result given by the algorithm is that our own drone 1 attacks enemy drone 1, our own drone 2 attacks enemy drone 2, and our own drone 3 attacks enemy drone 3, which can maximize the reward. Finally, the three groups of drones enter the differential game stage. The confrontation trajectories of the three groups of drones are shown in the figure. Figure 8 As shown. If the attack radius of the own drone is set to 10m, then the own drone successfully hits [172113] T m,[1214 25] T m,[21 259] T m destroys the corresponding enemy drone. The performance function evolution value of the first group of confrontation is as follows: Fig. 9 As shown, it can be seen that the function value reaches 71278 around 4 seconds and no longer changes, which is the Nash equilibrium point.

[0152] In summary, this application designs a model-free differential game algorithm, and applies the algorithm to iteratively interact with the environment and other drones to obtain the optimal pursuit strategy. At the same time, it is theoretically proved that the stability of the proposed model-free pursuit differential game algorithm can be guaranteed. Furthermore, a target allocation algorithm is proposed, which uses reinforcement learning theory to achieve rapid allocation of attack targets. In order to overcome the problems of limited allocation time and incomplete information, a threat value containing key information is first constructed to represent the state of the enemy drone, and then expert knowledge and historical data are introduced to improve the allocation efficiency. In addition, this application also designs a predictive behavior algorithm to predict the behavior of the enemy in the adversarial game. The algorithm overcomes the problems of low precision and low computational efficiency of traditional methods based on fuzzy networks.

[0153] refer to Fig.10 The present application provides an unmanned group game confrontation strategy generation device, comprising:

[0154] The dynamic model building module 1001 is used to build a dynamic model of the drone based on the position information and attitude information of the drone; wherein the drone includes a friendly drone and an enemy drone, and the attitude information includes the Euler angle of the drone.

[0155] The augmented system establishment module 1002 is used to establish augmented system equations of the own UAV and the enemy UAV based on the dynamic model.

[0156] The game performance function establishment module 1003 is used to establish the corresponding game performance function based on the augmented system equation.

[0157] The Bellman equation establishing module 1004 is used to establish the Bellman equation corresponding to the augmented system equation based on the game performance function.

[0158] The virtual control input determination module 1005 is used to iteratively solve the Bellman equation using the least square method to obtain the virtual control inputs of the own UAV and the enemy UAV.

[0159] The control module 1006 is used to control the own UAV to chase the enemy UAV based on the virtual control input of the own UAV and the enemy UAV.

[0160] In some embodiments of the present application, the device further comprises:

[0161] An interference term removal module, used to remove interference terms generated by the coupling of the position and attitude control of the UAV from the dynamic model;

[0162] The augmented system establishment module is further used to establish augmented system equations of the own UAV and the enemy UAV based on the dynamic model after removing the interference term.

[0163] In some embodiments of the present application, the gaming performance function establishment module further includes:

[0164] The first submodule is used to establish an equivalent equation of the Riccati equation of the game performance function based on the game performance function and the saddle point condition of the Nash equilibrium;

[0165] The second submodule is used to establish the Bellman equation corresponding to the augmented system equation based on the equivalent equation of the Riccati equation and the augmented system equation using the linear system fitting principle.

[0166] In some embodiments of the present application, the Bellman equation corresponding to the augmented system equation is:

[0167]

[0168] Among them, Q n =Q+K j,n T R j,n +K k,n T SK k,n , For the augmented state, p j =[p x,j p y,j p z,j ] T ∈R 3×1 is the position of UAV j in the inertial coordinate system, pk =[p x,k p y,k p z,k ] T ∈R 3×1 is the position of UAV k in the inertial coordinate system, A=[0 6×3 c 6,1 c 6,2 c 6,3 ]∈R 6×6 , B=[c 6,4 c 6,5 c 6,6 ]∈R 6×3 , K j,n is the feedback control input matrix of UAV j, K k,n is the feedback control input matrix of UAV k, U j is the control quantity of UAV j, U k is the control quantity of UAV k.

[0169] In some embodiments of the present application, the unmanned group game confrontation strategy generation device of the present application further includes:

[0170] The first acquisition module is used to acquire relative state information of the own UAV and the enemy UAV; wherein the relative state information includes: the angle difference between the velocity vector of the own UAV and the sight angle of the two UAVs, the angle difference between the velocity vector of the enemy UAV and the sight angle of the two UAVs, the modulus of the distance difference between the two UAVs, the velocity vector angle between the two UAVs, and the modulus of the velocity difference between the two UAVs;

[0171] A threat value determination module is used to define the threat value of the own UAV based on the relative state information; wherein the threat value includes the angle threat, distance threat and speed threat of the own UAV;

[0172] A Markov state transition probability determination module is used to determine the Markov state transition probability of the friendly UAV based on the threat value and using historical data and expert knowledge;

[0173] The target assignment module is used to obtain the target enemy drone assigned by the own drone based on the Markov state transition probability of the own drone and the use of reinforcement learning method to solve the cumulative discounted reward function of the own drone.

[0174] In some embodiments of the present application, the unmanned group game confrontation strategy generation device of the present application further includes:

[0175] The second acquisition module is used to obtain the position, speed and attitude information of the enemy UAV;

[0176] A behavior pattern prediction module is used to predict the behavior pattern of an enemy UAV based on the position, speed and attitude information of the enemy UAV and using a pre-trained fuzzy neural network model;

[0177] The fuzzy neural network model is trained in the following manner to obtain a training data set, wherein the training data set includes multiple groups of data samples, each of which includes the position, speed and attitude information of multiple sample enemy drones and corresponding behavior patterns: a fuzzy neural network model is constructed, wherein the fuzzy neural network model includes an input layer, a fuzzy layer, a fuzzy rule calculation layer and an output layer, wherein the fuzzy layer includes a preset number of fuzzy nodes determined according to the number of behavior patterns, and each fuzzy node has a corresponding membership function, and the formula of the membership function is:

[0178]

[0179] Among them, x q is the position, speed and attitude information of the enemy drone corresponding to the input node q in the input layer, is the membership function corresponding to the fuzzy node connected to the input node q, c qp is the first target parameter, v qp is the second target parameter. The fuzzy rule calculation layer includes multiple rule calculation nodes. The rule calculation formula of each rule calculation node is:

[0180]

[0181] The output layer includes an output node, and the formula of the output node is:

[0182]

[0183] Among them, ω p is the third target parameter, softmax(·) is the activation function, and the training data set is input into the constructed fuzzy neural network model. Based on the hybrid algorithm combining machine gradient descent and reverse optimization, the first target parameter, the second target parameter and the third target parameter in the fuzzy neural network model are trained to obtain the pre-trained intent interpretation model.

[0184] In summary, this application designs a model-free differential game algorithm applied to the above-mentioned device, and applies the algorithm to iteratively interact with the environment and other drones to obtain the optimal pursuit strategy. At the same time, it is theoretically proved that the stability of the proposed model-free differential game algorithm can be guaranteed.

[0185] Furthermore, this application proposes a target allocation algorithm that uses reinforcement learning theory to achieve rapid allocation of attack targets. In order to overcome the problems of limited allocation time and incomplete information, a threat value containing key information is first constructed to represent the state of the enemy drone, and then expert knowledge and historical data are introduced to improve the allocation efficiency.

[0186] In addition, the present application also designs a behavior prediction algorithm to predict the behavior of the enemy in the adversarial game, which overcomes the problems of low accuracy and low computational efficiency of traditional methods based on fuzzy networks.

[0187] It should be understood that the specific features, operations and details described hereinabove with respect to the method of the present application may also be similarly applied to the device and system of the present application, or vice versa. In addition, each step of the method of the present application described above may be performed by a corresponding component or unit of the device or system of the present application.

[0188] It should be understood that each module / unit of the device of the present application can be implemented in whole or in part by software, hardware, firmware or a combination thereof. Each module / unit can be embedded in the processor of the electronic device or independent of the processor in the form of hardware or firmware, or can be stored in the memory of the electronic device in the form of software for the processor to call to perform the operation of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0189] like Fig.11 As shown, the present application provides an electronic device 1100, which includes a processor 1101 and a memory 1102 storing computer program instructions. When the processor 1101 executes the computer program instructions, each step of the above-mentioned method for generating unmanned group game confrontation strategies is implemented. The electronic device 1100 can be a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities in a broad sense.

[0190] In one embodiment, the electronic device 1100 may include a processor, a memory, a network interface, a communication interface, etc. connected through a system bus. The processor of the electronic device 1100 may be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device 1100 may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system, a computer program, etc. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device 1100 may be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the method of the present application are executed.

[0191] The present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned unmanned group game confrontation strategy generation method is implemented.

[0192] Those skilled in the art will appreciate that the method steps of the present application can be completed by instructing related hardware such as the electronic device 1100 or a processor through a computer program, and the computer program can be stored in a non-temporary computer-readable storage medium, and the computer program causes the steps of the present application to be executed when it is executed. Depending on the circumstances, any reference to memory, storage, or other media herein may include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0193] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating unmanned group game confrontation strategies, characterized in that: The method comprises: A dynamic model of the drone is established based on the position information and attitude information of the drone; wherein the drone includes a friendly drone and an enemy drone, and the attitude information includes the Euler angle of the drone; Based on the dynamic model, the augmented system equations of the friendly UAV and the enemy UAV are established; Establishing the corresponding game performance function based on the augmented system equation; Establishing the Bellman equation corresponding to the augmented system equation based on the game performance function; The Bellman equation is iteratively solved using the least square method to obtain virtual control inputs of the friendly UAV and the enemy UAV; Based on the virtual control input of the own drone and the enemy drone, the own drone is controlled to pursue the enemy drone.

2. The method for generating unmanned group game confrontation strategy according to claim 1, characterized in that: The method further comprises: Remove interference items generated by the coupling of the position and attitude control of the UAV from the dynamic model; The augmented system equations of the own UAV and the enemy UAV established based on the dynamic model include: The augmented system equations of the friendly UAV and the enemy UAV are established based on the dynamic model after removing the interference term.

3. The method for generating unmanned group game confrontation strategy according to claim 1, characterized in that: The step of establishing the Bellman equation corresponding to the augmented system equation based on the game performance function includes: Based on the game performance function and the saddle point condition of Nash equilibrium, an equivalent equation of the Riccati equation of the game performance function is established; Based on the equivalent equation of the Riccati equation and the augmented system equation, the Bellman equation corresponding to the augmented system equation is established by utilizing the linear system fitting principle.

4. The method for generating unmanned group game confrontation strategy according to claim 1, characterized in that: The method further comprises: Acquire relative status information of the own UAV and the enemy UAV; wherein the relative status information includes: the angle difference between the velocity vector of the own UAV and the sight angle of the two UAVs, the angle difference between the velocity vector of the enemy UAV and the sight angle of the two UAVs, the modulus of the distance difference between the two UAVs, the velocity vector angle between the two UAVs, and the modulus of the velocity difference between the two UAVs; Determine the threat value of the own UAV based on the relative state information; wherein the threat value includes the angle threat, distance threat and speed threat of the own UAV; Determining the Markov state transition probability of the own UAV based on the threat value to the own UAV; Based on the Markov state transition probability of the own UAV, the cumulative discounted reward function of the own UAV is solved by using a reinforcement learning method to obtain a target enemy UAV assigned to the own UAV.

5. A method for generating unmanned group game confrontation strategies according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtain the position, speed and attitude information of the enemy drone; Based on the position, speed and attitude information of the enemy UAV, a pre-trained fuzzy neural network model is used to predict the behavior pattern of the enemy UAV.

6. An unmanned group game confrontation strategy generation device, characterized in that: The device comprises: A dynamic model building module, used to build a dynamic model of the drone based on the position information and attitude information of the drone; wherein the drone includes a friendly drone and an enemy drone, and the attitude information includes the Euler angle of the drone; An augmented system establishment module, used to establish augmented system equations of the own UAV and the enemy UAV based on the dynamic model; A game performance function establishment module, used to establish the corresponding game performance function based on the augmented system equation; A Bellman equation establishment module, used to establish the Bellman equation corresponding to the augmented system equation based on the game performance function; A virtual control input determination module is used to iteratively solve the Bellman equation using the least square method to obtain virtual control inputs of the own UAV and the enemy UAV; The control module is used to control the own drone to pursue the enemy drone based on the virtual control input of the own drone and the enemy drone.

7. The device for generating unmanned group game confrontation strategy according to claim 6, characterized in that: The device also includes: An interference term removal module, used to remove interference terms generated by the coupling of the position and attitude control of the UAV from the dynamic model; The augmented system establishment module is further used to establish augmented system equations of the own UAV and the enemy UAV based on the dynamic model after removing the interference term.

8. The device for generating unmanned group game confrontation strategy according to claim 6, characterized in that: The gaming performance function establishment module further includes: A first submodule is used to establish an equivalent equation of the Riccati equation of the game performance function based on the game performance function and the saddle point condition of the Nash equilibrium; The second submodule is used to establish the Bellman equation corresponding to the augmented system equation based on the equivalent equation of the Riccati equation and the augmented system equation using the linear system fitting principle.

9. The device for generating unmanned group game confrontation strategy according to claim 6, characterized in that: The device also includes: The first acquisition module is used to acquire relative state information of the own UAV and the enemy UAV; wherein the relative state information includes: the angle difference between the velocity vector of the own UAV and the sight angle of the two UAVs, the angle difference between the velocity vector of the enemy UAV and the sight angle of the two UAVs, the modulus of the distance difference between the two UAVs, the velocity vector angle between the two UAVs, and the modulus of the velocity difference between the two UAVs; A threat value determination module, used to determine the threat value of the own UAV based on the relative state information; wherein the threat value includes the angle threat, distance threat and speed threat of the own UAV; A Markov state transition probability determination module, used to determine the Markov state transition probability of the own UAV based on the threat value to the own UAV; The target allocation module is used to solve the cumulative discounted reward function of the own UAV based on the Markov state transition probability of the own UAV and use the reinforcement learning method to obtain the target enemy UAV assigned to the own UAV.

10. The device for generating unmanned group game confrontation strategy according to any one of claims 6 to 9, characterized in that: The device also includes: The second acquisition module is used to obtain the position, speed and attitude information of the enemy UAV; The behavior pattern prediction module is used to predict the behavior pattern of the enemy UAV based on the position, speed and posture information of the enemy UAV and using a pre-trained fuzzy neural network model.

Images