Aircraft intelligent many-to-many game target distribution method, electronic equipment and storage medium

Abstract

The invention discloses an aircraft intelligent many-to-many game target distribution method, electronic equipment and a storage medium, and belongs to the technical field of aircraft control. In order to realize many-to-many fast game target distribution of aircrafts, an aircraft game confrontation relative motion model is established; designing a training environment of aircraft intelligent many-to-many game confrontation, a state space of a multi-agent and an action space of the agent, collecting an observation value of the aircraft in the training environment as the state space of the multi-agent, taking an input number of the aircraft as the action space of the agent, and calculating fuel consumption of the aircraft and an obtained reward value; and constructing an OW-QMIX agent structure, then inputting a state space of multiple agents, an action space of the agents and the obtained reward value into the OW-QMIX agent structure, outputting an aircraft intelligent many-to-many game target allocation strategy training result, and then performing simulation verification. According to the invention, many-to-many fast game target distribution of aircrafts is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of aircraft control, and particularly relates to an aircraft intelligent many-to-many game target allocation method, an electronic device and a storage medium. BACKGROUND

[0002] In the process of many-to-many game confrontation of multiple aircrafts, the spatial position distribution of the confrontation targets usually covers a large spatial range, so the aircraft needs to allocate tasks to the game targets. By optimizing the game target allocation process, a better game strategy is obtained. Since the process of many-to-many game confrontation of multiple aircrafts has the characteristics of fast time-varying, the conventional algorithm is difficult to adapt to the rapid changes of the game confrontation form in the game confrontation process. Therefore, research needs to be carried out on the problem of many-to-many fast game target allocation of aircrafts. SUMMARY

[0003] The problem to be solved by the application is to realize many-to-many fast game target allocation of aircrafts, and an aircraft intelligent many-to-many game target allocation method, an electronic device and a storage medium are proposed.

[0004] To achieve the above purpose, the application realizes the following technical solutions:

[0005] An aircraft intelligent many-to-many game target allocation method, comprising the following steps:

[0006] S1. Establishing an aircraft game confrontation relative motion model;

[0007] S2. Designing a training environment of aircraft intelligent many-to-many game confrontation, a state space of multiple agents and an action space of agents based on the aircraft game confrontation relative motion model obtained in step S1, collecting observation values of the aircraft in the training environment as the state space of multiple agents, input numbers of the aircraft as the action space of agents, and calculating fuel consumption and reward values obtained by the aircraft;

[0008] S3. Constructing an OW-QMIX agent structure, then inputting the state space of multiple agents, the action space of agents and the reward values obtained in step S2 into the OW-QMIX agent structure, and outputting aircraft intelligent many-to-many game target allocation strategy training results;

[0009] S4. Simulating and verifying the aircraft intelligent many-to-many game target allocation strategy training results obtained in step S3.

[0010] Further, the specific implementation method of step S1 comprises the following steps:

[0011] S1.1. Constructing a three-degree-of-freedom kinematics equation of the aircraft flying outside the atmosphere, and the expression is:

[0012]

[0013] wherein, is the rate of change of the velocity of the aircraft relative to the x-axis direction, V is the displacement vector and velocity vector of the aircraft in the ballistic system relative to the launch inertial system, is the rate of change of the velocity of the aircraft relative to the y-axis direction, is the rate of change of the velocity of the aircraft relative to the z-axis direction, θ is the flight path angle, ψ P is the heading angle;

[0014]

[0015] wherein, is the rate of change of the velocity of the aircraft relative to the pitch angle , is the rate of change of the velocity of the aircraft relative to the yaw angle ψ m , is the rate of change of the velocity of the aircraft relative to the roll angle γ m , ω xB is the angular velocity of the aircraft relative to the x-axis direction, ω yB is the angular velocity of the aircraft relative to the y-axis direction, ω zB is the angular velocity of the aircraft relative to the z-axis direction;

[0016] S1.2. Constructing the dynamic equation of the aircraft flying outside the atmosphere, the expression is:

[0017]

[0018] wherein, m is the mass of the aircraft, r is the geocentric vector of the aircraft, V is the displacement vector and velocity vector of the aircraft in the ballistic system relative to the launch inertial system, Ω is the angular velocity of the ballistic system relative to the launch inertial system, P is the engine thrust of the game aircraft, F is the aerodynamic force and interference force other than the thrust, H is the center of mass momentum moment of the game aircraft, M P is the moment of the engine of the aircraft due to the eccentricity of the thrust, M is the aerodynamic moment and interference moment, ω is the angular velocity of the aircraft body system relative to the launch inertial system;

[0019] S1.3. Setting the position of the aircraft at time T as r M = [x M , y M , z M ] T , the position of the target aircraft at time T is r T = [x T , y T , z T ] T , constructing the motion position parameter model of the aircraft-target aircraft, the expression is:

[0020] The expression of relative position Δr is:

[0021]

[0022] According to the relative distance, the expression of relative velocity v is obtained by derivation:

[0023]

[0024] After obtaining the relative velocity, the expression of the corresponding distance approach rate is obtained:

[0025]

[0026] The expressions of line-of-sight inclination angle q g and line-of-sight deflection angle q f are:

[0027]

[0028] wherein, q f = (-π, π];

[0029] Since the target adopts proportional guidance, the expression of the aircraft-target line-of-sight angle rate is obtained:

[0030]

[0031] wherein, is the aircraft-target line-of-sight inclination rate, is the aircraft-target line-of-sight deflection rate.

[0032] Further, the specific implementation method of step S2 includes the following steps:

[0033] S2.1. Designing a training environment for intelligent multi-to-multi game confrontation of the aircraft based on the aircraft game confrontation relative motion model obtained in step S1;

[0034] Initialization setting is performed, including randomly generating a plurality of aircraft and game targets, and randomly determining the speed size and direction of the aircraft. At the start time of simulation, each aircraft respectively performs detection, and obtains the relative position and speed information of the game targets in its field of view. The position initialization range of the aircraft is set to 0-1 km in the x-axis direction, wherein the x-axis direction position of the target is 10-11 km, the y-axis direction position is -1-12 km, the z-axis direction position is -10-10 km, the speed initialization size is taken in the value range of 5000-5500 m / s, and the speed inclination angle and deflection angle direction are taken in the value range of -10°-10°.

[0035] S2.2. Define the expression of the state space S of the multi-agent as:

[0036]

[0037] wherein Δx1, Δy1, Δz1 represent the three-axis relative position of the first game aircraft, is the relative distance change rate of the first game aircraft;

[0038] Define the expression of the action space A of the agent as:

[0039] A = [num] (10)

[0040] wherein the action space of the agent is the corresponding game aircraft target allocation result, and num is the corresponding number of the game aircraft;

[0041] S2.3. Input the observation value of each aircraft into the agent network respectively, and the agent network outputs the corresponding number of the game aircraft. The aircraft group simulates the relative motion model of the game aircraft, and calculates the fuel consumption of the aircraft in the process of game confrontation. In combination with the game confrontation result, the reward obtained by the allocation algorithm is calculated.

[0042] The expression of the fuel consumption of the aircraft is:

[0043]

[0044] wherein Fuel is the fuel consumption index, n is the total number of aircrafts, i is any one in n, N represents the total overload of aircraft maneuver, and t represents the total time length of aircraft maneuver.

[0045] The expression of the reward reward obtained by the allocation algorithm is:

[0046]

[0047] wherein Fuel i is the fuel consumption index of the i-th aircraft, and the reward is the sum of the game confrontation reward value and the fuel consumption reward value.

[0048] Further, the specific implementation method of step S3 includes the following steps:

[0049] S3.1. The state space of the multi-agent, the action space of the agent and the reward value obtained are input into the OW-QMIX agent structure obtained in step S2, the first step of the input is randomly selected as S0, the flight trajectory simulation is performed by using the dynamic equation in step S1, the state space S' of the next time is randomly obtained after completing a time step of 0.1 s, the data is composed into a four-tuple (S, A, S', r) and stored in the experience pool D, and the maximum number of rounds of 100000 is set as the experience pool cycle stop condition;

[0050] then the parameters of the current network are set as θ now , the updated network parameters are θ new , the forward propagation of the single-agent network is performed to obtain the expression of the updated i-th single-agent value function Q i ′ as follows:

[0051] Q i ′=q i (S i ′,A i ;θ now ) (13)

[0052] S3.2. For the value distribution problem in multi-agent learning, the value decomposition network VDN is used for multi-agent training, the network weight of the OW-QMIX agent structure is set as Q(S, A; θ), the experience pool is set as D, the learning rate is set as α, and the joint action value function is set as Q tot (S, A) with the expression as follows:

[0053]

[0054] wherein Q i is the i-th single-agent value function;

[0055] S3.3. When the multi-agent needs to extract a decentralized policy, the VDN ensures that the operation of the global maximum independent variable and the operation of the individual maximum independent variable on the agent network have the same result, and the expression is as follows:

[0056]

[0057] Each single agent collects information through its agent network, inputs the information into the mixing network after summarizing, the mixing network is composed of a super network with an absolute value function as a heuristic function, and finally Q tot (S, A) is calculated.

[0058] S3.4. The expressions of the TD target y TD,i and the TD error δ TD,i are as follows:

[0059] yTD,i = reward + gamma * Q i (16)

[0060] delta TD,i = Q tot - y TD (17)

[0061] Set the expression of the gradient g obtained by back propagation i The expression of the gradient g obtained by back propagation is as follows:

[0062]

[0063] The expression of the gradient g obtained by back propagation is as follows:

[0064] theta new ← theta now + alpha * delta TD,i * g i (19)

[0065] S3.5. Repeat steps S3.1-S3.4 until the output aircraft intelligent many-to-many game target allocation strategy training result is obtained.

[0066] Further, the specific implementation method of step S4 includes the following steps:

[0067] S4.1. Determine that the number of simulated multiple aircrafts is 16, the number of targets is 12, and the parameters of the aircrafts and targets are randomly selected for initialization when the environment is initialized.

[0068] S4.2. Set the hyperparameters of the OW-QMIX network, and the hardware environment used for simulation is: CPU i5-11400F, GPU GTX1060, and in the normal working condition, the aircraft does not exist partial or total failure problem, and the simulation verification is carried out.

[0069] An electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the aircraft intelligent many-to-many game target allocation method when executing the computer program.

[0070] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the aircraft intelligent many-to-many game target allocation method.

[0071] The beneficial effects of the present application are:

[0072] The aircraft intelligent many-to-many game target allocation method provided by the application discloses a distributed aircraft target allocation method based on an optimized weighted hybrid Q learning, considers limited observation between aircrafts and different communication topological structures, uses an OW-QMIX algorithm to train distributed intelligent agents, and performs relevant multi-agent algorithm simulation.

[0073] The aircraft intelligent many-to-many game target allocation method provided by the application discloses a distributed aircraft target allocation method based on an optimized weighted hybrid Q learning, considers limited observation between aircrafts and different communication topological structures, uses an OW-QMIX algorithm to train distributed intelligent agents, and performs relevant multi-agent algorithm simulation. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 The flowchart of the aircraft intelligent many-to-many game target allocation method provided by the application;

[0075] Figure 2 The schematic diagram of the overall application process of the OW-QMIX algorithm provided by the application;

[0076] Figure 3 The intelligent agent structure diagram of the QMIX algorithm provided by the application;

[0077] Figure 4 The training environment design diagram of the application;

[0078] Figure 5 The algorithm training process curve of the normal working environment of the application;

[0079] Figure 6 The algorithm test result of the normal working environment of the application;

[0080] Figure 7 The algorithm training process curve of the partial failure environment of the application;

[0081] Figure 8 The algorithm test result of the partial failure environment of the application. DETAILED DESCRIPTION

[0082] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, i.e., the specific embodiments described are only a part of the embodiments of the present application, but not all the specific embodiments. The components of the specific embodiments of the present application generally described and shown in the drawings herein can be arranged and designed in various different configurations, and the present application can also have other embodiments.

[0083] Therefore, the detailed description of the specific embodiments of the present application provided below in the drawings is not intended to limit the scope of the claimed present application, but only represents selected specific embodiments of the present application. Based on the specific embodiments of the present application, all other specific embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0084] In order to further understand the inventive content, characteristics and effects of the present application, the following specific embodiments are exemplified, and the drawings are Figure 1 - the drawings Figure 8 are as follows: Specific embodiment one:

[0086] A method for intelligent multi-to-multi game target allocation of an aircraft, comprising the following steps:

[0087] S1. Establishing a game confrontation relative motion model of the aircraft;

[0088] Further, the specific implementation method of step S1 comprises the following steps:

[0089] S1.1. Constructing a three-degree-of-freedom kinematics equation of the aircraft flying outside the atmosphere, expressed as:

[0090]

[0091] wherein, is the rate of change of the velocity of the aircraft relative to the x-axis direction, V is the displacement vector and velocity vector of the aircraft in the ballistic system relative to the inertial system, is the rate of change of the velocity of the aircraft relative to the y-axis direction, is the rate of change of the velocity of the aircraft relative to the z-axis direction, θ is the track angle, ψ P is the heading angle;

[0092]

[0093] wherein, is the rate of change of the velocity of the aircraft relative to the x-axis direction, V is the displacement vector and velocity vector of the aircraft in the ballistic system relative to the inertial system, is the rate of change of the velocity of the aircraft relative to the y-axis direction, is the rate of change of the velocity of the aircraft relative to the z-axis direction, θ is the track angle, ψm the rate of change of the velocity of the aircraft, the roll angle γ of the aircraft m the rate of change of the velocity of the aircraft, ω xB the angular velocity of the aircraft relative to the x-axis direction, ω yB the angular velocity of the aircraft relative to the y-axis direction, ω zB the angular velocity of the aircraft relative to the z-axis direction;

[0094] S1.2. Construct the dynamic equation of the aircraft flying outside the atmosphere, expressed as:

[0095]

[0096] where m is the mass of the aircraft, r is the geocentric radius of the aircraft, V is the displacement vector and velocity vector of the aircraft in the ballistic system relative to the launch inertial system, Ω is the angular velocity of the ballistic system relative to the launch inertial system, P is the thrust of the game aircraft engine, F is the aerodynamic force and disturbance force, H is the mass moment of inertia of the game aircraft, M P is the moment of the aircraft engine due to the eccentricity of the thrust, M is the aerodynamic moment and disturbance moment, and ω is the angular velocity of the aircraft body system relative to the launch inertial system;

[0097] S1.3. Set the position of the aircraft at time T as r M = [x M , y M , z M ] T , the position of the target aircraft at time T is r T = [x T , y T , z T ] T , construct the motion position parameter model of the aircraft-target aircraft, expressed as:

[0098] The expression of the relative position Δr is:

[0099]

[0100] According to the relative distance, the expression of the relative velocity v is obtained by derivation:

[0101]

[0102] After obtaining the relative velocity, the expression of the corresponding distance approach rate is obtained:

[0103]

[0104] The expressions of the line-of-sight inclination angle q g and the line-of-sight deflection angle q f are:

[0105]

[0106] wherein, q f = (-π, π];

[0107] Since the target adopts proportional guidance, the expression of the aircraft-target line-of-sight angle rate is obtained as follows:

[0108]

[0109] wherein, is the aircraft-target line-of-sight inclination rate, is the aircraft-target line-of-sight deflection rate;

[0110] S2. Design a training environment for aircraft intelligent multi-to-multi game confrontation, a state space of multi-agents and an action space of agents based on the aircraft game confrontation relative motion model obtained in step S1, collect observation values of the aircraft in the training environment as the state space of the multi-agents, input numbers of the aircraft as the action space of the agents, and calculate fuel consumption of the aircraft and reward values obtained by the aircraft;

[0111] Further, the specific implementation method of step S2 includes the following steps:

[0112] S2.1. Design a training environment for aircraft intelligent multi-to-multi game confrontation based on the aircraft game confrontation relative motion model obtained in step S1.

[0113] Initialization setting is performed, including randomly generating a plurality of aircraft and game targets, and randomly determining a speed size and a direction of the aircraft. At a simulation start time, each aircraft respectively performs detection, and obtains relative position and speed information of the game targets in a field of view thereof. The position initialization range of the aircraft is set as 0-1 km in the x-axis direction, wherein the position of the target in the x-axis direction is 10-11 km, the position in the y-axis direction is -1-12 km, the position in the z-axis direction is -10-10 km, the speed initialization size is set as 5000-5500 m / s, and the speed inclination and deflection directions are set as -10°-10°.

[0114] S2.2. The expression of the state space S of the multi-agents is defined as follows:

[0115]

[0116] In the formula, Δx1, Δy1 and Δz1 represent three-axis relative positions of the first game aircraft, is a relative distance change rate of the first game aircraft;

[0117] The expression for defining the action space A of the agent is:

[0118] A = [num] (10)

[0119] wherein the action space of the agent is the corresponding game aircraft target allocation result, and num is the corresponding number of the game aircraft;

[0120] S2.3. Input the observation value of each aircraft into the agent network respectively, and the agent network outputs the corresponding number of the game aircraft. The aircraft group simulates the relative motion model of the game aircraft, and calculates the fuel consumption of the aircraft during the game confrontation process. The reward obtained by the allocation algorithm is calculated in combination with the game confrontation result;

[0121] The expression of the fuel consumption of the aircraft is:

[0122]

[0123] wherein Fuel is the fuel consumption index, n is the total number of aircrafts, i is any one in n, N represents the total overload of aircraft maneuver, and t represents the total time length of aircraft maneuver;

[0124] The expression of the reward obtained by the allocation algorithm is:

[0125]

[0126] wherein Fuel i is the fuel consumption index of the i-th aircraft, and the reward is the sum of the game confrontation reward value and the fuel consumption reward value.

[0127] Further, the reward is a normalized reward, which includes the game confrontation reward and the fuel consumption reward. The final reward is normalized according to the number of aircrafts as the final reward value;

[0128] S3. Constructing the OW-QMIX agent structure, then inputting the state space of the multi-agent, the action space of the agent and the reward value obtained in step S2 into the OW-QMIX agent structure, and outputting the training result of the aircraft intelligent multi-to-multi game target allocation strategy;

[0129] Further, the specific implementation method of step S3 includes the following steps:

[0130] S3.1. The state space of the multi-agent, the action space of the agent and the reward value obtained are input into the OW-QMIX agent structure obtained in step S2, the first step of input is randomly selected as S0, the flight trajectory simulation is performed by using the dynamic equation in step S1, the state space S' of the next time is randomly obtained after completing a time step 0.1 s, the data is composed into a four-tuple (S, A, S', r) and stored in the experience pool D, and the maximum number of rounds is set as 100000 times as the experience pool cycle stop condition;

[0131] The parameter of the current network is set as θ now , the updated network parameter is θ new , the expression of the i-th single-agent value function Q i ' obtained by forward propagation of the single-agent network is:

[0132] Q i ' = q i (S i ', A i ; θ now ) (13)

[0133] S3.2. For the value distribution problem in multi-agent learning, the value decomposition network VDN is used for multi-agent training, the network weight of the OW-QMIX agent structure is set as Q(S, A; θ), the experience pool is set as D, the learning rate is set as α, and the joint action value function is set as Q tot (S, A), and the expression is:

[0134]

[0135] Wherein, Q i is the i-th single-agent value function;

[0136] S3.3. When the multi-agent needs to extract a decentralized strategy, the VDN ensures that the operation of the global maximum independent variable and the operation of the single maximum independent variable on the agent network have the same result, and the expression is:

[0137]

[0138] Each single agent collects information through its agent network, inputs the information into the mixing network after summarizing, and finally calculates Q tot (S, A) by the mixing network composed of a super network with an absolute value function as a heuristic function.

[0139] S3.4. The expressions of the TD target y TD,i and the TD error δ TD,i are:

[0140] yTD,i = reward + γQ i (16)

[0141] δ TD,i = Q tot - y TD (17)

[0142] Set the expression for obtaining the gradient g of back propagation i

[0143]

[0144] Set the expression for updating the network parameters

[0145] θ new ← θ now + α · δ TD,i · g i (19)

[0146] S3.5. Repeat steps S3.1-S3.4 until the output aircraft intelligent many-to-many game target assignment strategy training result is obtained.

[0147] S4. Simulate and verify the aircraft intelligent many-to-many game target assignment strategy training result obtained in step S3.

[0148] Further, the specific implementation method of step S4 includes the following steps:

[0149] S4.1. Determine that the number of simulated multi-aircraft is 16, the number of targets is 12, and the environment is initialized by randomly selecting the parameter disturbance of the aircraft and the target for initialization;

[0150] S4.2. Set the hyperparameters of the OW-QMIX network, and the hardware environment used for simulation is: CPU i5-11400F, GPU GTX1060, and there is no partial or total failure problem of the aircraft in the normal working condition, and the simulation verification is performed.

[0151] Further, the setting of the hyperparameters of the OW-QMIX network is shown in Table 1

[0152] Table 1 Hyperparameter setting

[0153]

[0154]

[0155] ​In the task allocation of multiple UAVs to multiple targets, the number of simulated UAVs is 16, and the number of targets is 12. When initializing the environment, the parameters of the UAVs and targets are randomly selected for initialization. The initial parameters of the 16 UAVs in one training case are shown in Table 2 as follows:

[0156] Table 2 Initial parameters of UAVs

[0157]

[0158] At the same time, the initial parameters of the targets are set as shown in Table 3

[0159] Table 3 Initial parameters of targets

[0160]

[0161] 1. Simulation results and analysis under normal working conditions are as follows:

[0162] After setting the hyperparameters, the hardware environment used for simulation is: CPU i5-11400F, GPU GTX1060. In the normal working condition, there is no partial or total failure problem of the UAVs. The training process of the QMIX algorithm and the OW-QMIX algorithm is as shown in Figure 5 . In the figure, the horizontal axis is the number of training scenes, and the vertical axis is the total reward value obtained by the agent. Taking the initial environment as an example, after changing the initial conditions of the environment many times, including the initial position and initial velocity, and changing the number of game targets and UAVs between 12-16. Finally, the normalized indicators of each algorithm are listed in Table 4 as follows: the normalized reward mentioned above is the completion degree value obtained by normalizing the interception result and fuel consumption:

[0163] Table 4 Algorithm index completion degree statistics

[0164]

[0165] The index distribution of the three algorithms is shown in Figure 6 .

[0166] 2. Simulation results and analysis under partial failure conditions are as follows:

[0167] In this simulation, there is a partial failure problem of individual UAVs. The failure mode is to use a random number of UAVs to fail, and the failed UAVs cannot participate in the task. The training process of the QMIX algorithm and the OW-QMIX algorithm is as shown in Figure 7 . The horizontal axis is the number of training scenes, and the vertical axis is the total reward value obtained by the agent. After many simulations, the normalized indicators of each algorithm are listed in Table 5 as follows:

[0168] Table 5 Algorithm index completion degree statistics

[0169]

[0170] The index distribution of the three algorithms is shown in Figure 8

[0171] In summary, through Monte Carlo targeting test, in the presence of multiple aircraft individual failure, the distributed multi-aircraft multi-to-multi game target allocation algorithm based on OW-QMIX algorithm is stronger than QMIX algorithm in convergence speed; the same as simulation 1, it can be seen that the convergence speed and index completion degree of OW-QMIX algorithm are stronger than QMIX algorithm and greedy algorithm, and in general, OW-QMIX algorithm has great advantages. Specific implementation two:

[0173] An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the aircraft intelligent multi-to-multi game target allocation method described in the specific implementation one when executing the computer program.

[0174] The computer device of the present application can be a device including a processor and a memory, such as a single-chip microcomputer including a central processing unit. Moreover, the processor is used to execute the computer program stored in the memory to realize the steps of the above-mentioned aircraft intelligent multi-to-multi game target allocation method.

[0175] The so-called processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0176] ​The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices. THREE SPECIFIC EMBODIMENTS

[0178] A computer readable storage medium, having stored thereon a computer program, wherein the computer program is executed by a processor to implement the aircraft intelligent many-to-many game target allocation method in the first specific embodiment.

[0179] The computer readable storage medium of the present application can be any form of storage medium readable by the processor of the computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc., and the computer readable storage medium has stored thereon a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned aircraft intelligent many-to-many game target allocation method can be implemented.

[0180] The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0181] It has to be noted that the terms "first", "second", and the like in connection with an entity or action refer to this entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without further constraints, exclude the presence of additional elements of the process, method, article, or apparatus.

[0182] While the application has been described with reference to specific implementations thereof, it should be understood that various modifications and substitutions can be made by those skilled in the art without departing from the scope of the present application. Especially, features of the specific implementations disclosed herein can be combined in any manner, unless structural conflicts arise, and the combinations are not exhaustively described in the present specification only for the sake of brevity and conciseness. Therefore, the application is not limited to the specific implementations disclosed herein but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for allocating targets for an intelligent many-to-many game involving aircraft, characterized in that: The steps include: S1. Establish a relative motion model for aircraft game confrontation; S2. Based on the relative motion model of the aircraft game confrontation obtained in step S1, design a training environment, multi-agent state space, and agent action space for the aircraft intelligent multi-to-multi game confrontation. Collect aircraft observations in the training environment as the multi-agent state space, and aircraft input numbers as the agent action space. Calculate the fuel consumption and reward values ​​of the aircraft. S3. Build the OW-QMIX agent structure, then input the multi-agent state space, agent action space, and reward value obtained in step S2 into the OW-QMIX agent structure, and output the training results of the aircraft intelligent many-to-many game target allocation strategy; S4. Simulate and verify the training results of the aircraft intelligent many-to-many game target allocation strategy obtained in step S3.

2. The method for allocating targets of an intelligent many-to-many aircraft game according to claim 1, characterized in that: The specific implementation method of step S1 includes the following steps: S1.

1. Construct the three-degree-of-freedom kinematic equations for an aircraft flying outside the atmosphere, expressed as: in, is the velocity change rate of the aircraft relative to the x-axis, V is the displacement radius and velocity vector of the aircraft in the ballistic system relative to the inertial system, is the velocity change rate of the aircraft relative to the y-axis, is the velocity change rate of the aircraft relative to the z-axis, θ is the track angle, ψ P is the heading angle; in, is the pitch angle The rate of change of velocity, is the yaw angle ψ m The rate of change of velocity, is the roll angle γ m The rate of change of velocity, ω xB is the angular velocity of the aircraft relative to the x-axis, ω yB is the angular velocity of the aircraft relative to the y-axis, ω zB is the angular velocity of the aircraft relative to the z-axis; S1.

2. Construct the dynamic equations for the aircraft flying outside the atmosphere, which are expressed as follows: Among them, m is the mass of the aircraft, r is the center of gravity of the aircraft, V is the displacement radius and velocity vector of the aircraft in the ballistic system relative to the launch inertial system, Ω is the angular velocity of the ballistic system relative to the launch inertial system, P is the thrust of the game aircraft engine, F is the aerodynamic force and interference force other than the thrust, H is the center of mass momentum of the game aircraft, M P is the torque generated by the aircraft engine due to thrust eccentricity, M is the aerodynamic torque and interference torque, and ω is the angular velocity of the aircraft system relative to the inertial system; S1.

3. Set the position of the aircraft at time T to r M =[x M ,y M ,z M ] T , the position of the target aircraft at time T is r T =[x T ,y T ,z T ] T , construct the motion position parameter model of the aircraft-target aircraft, the expression is: The expression of relative position Δr is: According to the relative distance, the expression of relative velocity v is derived as follows: After obtaining the relative speed, the corresponding distance closing rate is obtained The expression is: Sight angle q g and the line of sight deflection angle q f The expression is: Among them, q f =(-π,π]; Since the target adopts proportional guidance, the expression of the aircraft-target line of sight angular rotation rate is: in, is the aircraft-target line of sight inclination rate, is the aircraft-target line of sight deflection rate.

3. The method for allocating targets of an intelligent many-to-many aircraft game according to claim 2, characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. Design a training environment for intelligent multi-to-multi aircraft game confrontation based on the aircraft relative motion model obtained in step S1. Perform initialization settings, including randomly generating multiple aircraft and game targets, and randomly determining the speed and direction of the aircraft. At the start of the simulation, each aircraft performs detection and obtains the relative position and speed information of the game target in its field of view; the aircraft position initialization range is set to 0-1km in the x-axis direction, of which the target's x-axis position is 10-11km, the y-axis position is -1-12km, and the z-axis position is -10-10km. The speed initialization value range is 5000-5500m / s, and the speed inclination and deflection direction value range is -10° to 10°. S2.

2. The expression for defining the state space S of a multi-agent is: Where Δx1, Δy1, and Δz1 represent the three-axis relative positions of the first game aircraft. is the relative distance change rate of the first game aircraft; The expression defining the action space A of the agent is: A=[num] (10) Among them, the action space of the agent is the corresponding game aircraft target allocation result, and num is the corresponding number of the game aircraft; S2.

3. Each aircraft's observations are fed into the agent network. The agent network outputs the corresponding number of the aircraft in the game. The aircraft group is simulated based on the relative motion model of the aircraft game competition. During the aircraft game competition, the fuel consumption of the aircraft is calculated, and the rewards obtained by the distribution algorithm are calculated based on the results of the game competition. The expression for the fuel consumption of an aircraft is: Where Fuel is the fuel consumption index, n is the total number of aircraft, i is any one of n, N is the total overload of aircraft maneuvers, and t is the total duration of aircraft maneuvers; The reward obtained by the allocation algorithm is expressed as: Among them, Fuel i is the fuel consumption index of the i-th aircraft, and the reward is the sum of the game confrontation reward value and the fuel consumption reward value.

4. The method for allocating targets of an intelligent many-to-many aircraft game according to claim 3, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. Input the multi-agent state space, agent action space, and reward values ​​obtained in step S2 into the OW-QMIX agent structure. Randomly select the first step of the input as S0. Use the dynamic equations in step S1 to simulate the flight trajectory. After completing a time step of 0.1s, randomly obtain the next state space S′. This data is organized into a 4-tuple (S, A, S′, r) and stored in the experience pool D. Set the experience pool loop termination condition to a maximum number of rounds of 100,000. Then set the parameters of the current network to θ now , the network parameters after the update are θ new , forward propagation is performed on the single-agent network to obtain the updated i-th single-agent value function Q i The expression of ′ is: Q i ′=q i (S i ′,A i ;θ now ) (13) S3.

2. Aiming at the value distribution problem in multi-agent learning, we use the value decomposition network (VDN) for multi-agent training. We set the network weight of the OW-QMIX agent structure to Q(S, A; θ), the experience pool to D, the learning rate to α, and the joint action value function to Q. tot (S,A), the expression is: Among them, Q i is the value function of the i-th single agent; S3.

3. When multiple agents need to extract decentralized strategies, VDN ensures that the global maximum independent variable operation and the individual maximum independent variable operation on the agent network have the same results, and the expression is: Each single agent collects information through its agent network, summarizes it and inputs it into the hybrid network for calculation. The hybrid network is composed of a super network with the absolute value function as the heuristic function, and finally calculates Q tot (S,A); S3.

4. Set TD target y TD,i and TD error δ TD,i The expression is: y TD,i =reward+γQ i ′ (16) δ TD,i =Q tot -y TD (17) Set up back propagation to get the gradient g i The expression is: g i =▽ θ Q i (S i ,A i ;θ now ) (18) The expression for setting the updated network parameters is: i new ←θ now +a·d TD,i ·g i (19) S3.

5. Repeat steps S3.1-S3.4 until the training results of the aircraft intelligent many-to-many game target allocation strategy are output.

5. The method for allocating targets of an intelligent many-to-many aircraft game according to claim 4, characterized in that: The specific implementation method of step S4 includes the following steps: S4.

1. Determine the number of simulated multi-robots to be 16 and the number of targets to be 12. Initialize the environment by randomly selecting perturbations of the parameters of the vehicles and targets. S4.

2. Set the hyperparameters of the OW-QMIX network. The simulation hardware environment used is: CPU i5-11400F, GPU GTX1060. Under normal working conditions, there is no partial or complete failure of the aircraft. Simulation verification is performed.

6. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for allocating targets of an intelligent many-to-many game of aircraft according to any one of claims 1 to 5 are realized.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for allocating targets of an intelligent many-to-many game for aircraft as described in any one of claims 1 to 5 is implemented.

Images