Unmanned aerial vehicle asynchronous frozen game confrontation training method and device, equipment and medium

Abstract

The application discloses an unmanned aerial vehicle asynchronous frozen game confrontation training method and device, equipment and medium, belong to unmanned aerial vehicle near distance air combat maneuver decision field. Among them, the method comprises the following steps: constructing an unmanned aerial vehicle decision model, and initializing the model parameters; a coupling reward function considering angle and distance is designed, and the unmanned aerial vehicle decision model is trained according to the coupling reward function; the unmanned aerial vehicle decision model of our side and the enemy side is trained by using the asynchronous frozen game confrontation training method, and the trained model is used to control the unmanned aerial vehicle. The application can improve the near distance air combat model training speed and the model maneuver decision performance through the coupling reward function. In addition, based on the asynchronous frozen game confrontation method, the performance of different reinforcement learning models can be objectively and reasonably evaluated in a direct confrontation manner, and the performance of the reinforcement learning model can be effectively and continuously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle (UAV) close air combat maneuver decision-making, and in particular to a UAV asynchronous frozen game confrontation training method, device, equipment and medium. BACKGROUND

[0002] As an important part of modern military confrontation, air combat is complex and high-risk, which brings great challenges to the decision-making process. In particular, in close air combat, quick and accurate maneuver decision-making is crucial. The widespread use of unmanned aerial vehicles (UAVs) plays an increasingly important role in modern air combat due to their flexibility, safety and low cost. Unmanned aerial vehicles not only perform reconnaissance and attack tasks, but also play a key role in high-risk air combat environments. However, the difficulty of decision-making in air combat comes from the rapid changes in the environment, the complex maneuvers of enemy aircraft, and the real-time requirements of the decision-making process.

[0003] In order to improve the autonomous air combat decision-making capability of unmanned aerial vehicles, many studies have attempted to use game theory and optimization theory to solve this problem. Although these methods have certain effects in theory, they often face great computational pressure and uncertainty when solving real-time complex problems. Therefore, in recent years, deep reinforcement learning (DRL) has gradually become an important research direction in the field of unmanned aerial vehicle air combat, with its ability to quickly adapt to complex problems in high-dimensional environments, deep reinforcement learning provides a more flexible and efficient autonomous decision-making solution for unmanned aerial vehicles.

[0004] However, when using deep reinforcement learning, one of the challenges is the design of the reward function. Most related researches consider different situation information rewards separately, such as angle rewards and distance rewards, which are added together in a certain proportion to form a comprehensive reward. One of the challenges is the weight selection problem. Secondly, the decision-making model obtained by using a single training is ordinary, which is not conducive to improving the performance of the decision-making model. SUMMARY

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a UAV asynchronous frozen game confrontation training method, device, equipment and medium based on deep reinforcement learning.

[0006] The first technical solution adopted by the present application is:

[0007] A UAV asynchronous frozen game confrontation training method, comprising the following steps:

[0008] Constructing a UAV decision-making model and initializing the model parameters;

[0009] Designing a coupling reward function that considers angle and distance comprehensively, and training the UAV decision-making model according to the coupling reward function;

[0010] An asynchronous frozen game confrontation training method is used to train the unmanned aerial vehicle decision-making model of the friend and the enemy, and the trained model is used to control the unmanned aerial vehicle.

[0011] Further, the expression of the coupling reward function is as follows:

[0012]

[0013] s.t.α2D best -γ2=β2D best ,α2>0,β2<0

[0014] In the formula, D represents the distance between the unmanned aerial vehicles of the two parties, D best represents the optimal air combat distance, ATA represents the angle between the speed vector and the distance vector of the unmanned aerial vehicle of the friend relative to the target unmanned aerial vehicle; AA represents the angle between the speed vector of the unmanned aerial vehicle of the enemy and the distance vector between the unmanned aerial vehicle of the enemy and the unmanned aerial vehicle of the friend; α2, β2, γ2 are parameters for adjusting the spatial form of the reward function, and satisfy the constraint α2D best -γ2=β2D best , which ensures that the reward function of D≤D best is connected with the reward function of D≥D best in space, which is conducive to training convergence.

[0015] Further, the unmanned aerial vehicle decision-making model is a deep reinforcement learning network model, and the unmanned aerial vehicle decision-making model comprises an Actor network and a Critic network.

[0016] The unmanned aerial vehicle decision-making model is trained in the following manner:

[0017] The collected position and speed information of the unmanned aerial vehicle of the friend and the unmanned aerial vehicle of the enemy are integrated to obtain environment situation information s;

[0018] The environment situation information s is input into the Actor network to obtain the action a of the current unmanned aerial vehicle;

[0019] The new environment situation information s′ is obtained by executing the action a;

[0020] The reward r of the action a is obtained according to the environment situation information s′ and the coupling reward function;

[0021] It is judged whether the game is over according to the new environment situation information s′, and the result B is recorded as True if the game is won, or False if the game is drawn or lost, and the result B and other information are formed as {s, a, r, s′, B} and saved into an experience replay pool M;

[0022] The environment situation information s is updated, so that s=s′;

[0023] Data from the sampled experience replay pool M is used to update the network parameters.

[0024] Furthermore, the environmental situation information s is defined as:

[0025]

[0026] In the formula, x, y, and z represent the three-dimensional position information of the UAV, the subscripts r and b represent the friendly UAV and the enemy UAV, respectively; v represents the velocity, v x v y v z These represent the components of velocity along the three axes.

[0027] Furthermore, the data in the sampled experience replay pool M is used to update the network parameters, including:

[0028] m samples are sampled from the experience replay set M, {s} i ,a i ,r i ,s′ i B i}, i = 1, ..., m, calculate the current target Q value y using the Critic network. j ;

[0029] The parameters ω of the current Critic network are updated through backpropagation of the gradient of the neural network.

[0030] The Actor updates all parameters θ of the current network through backpropagation of the gradient of the neural network.

[0031] Regularly update the parameters of the Critic and Actor target networks:

[0032] ω'←τω+(1-τ)ω′

[0033] θ'←τθ+(1-τ)θ'

[0034] In the formula, τ is the update coefficient.

[0035] Furthermore, the target Q value y j The calculation formula is as follows:

[0036]

[0037] In the formula, γ is the discount factor, and R j Q′(φ(S′) is the reward value obtained at the current step size. j )) represents the next state S′ j The expected reward obtained in the Q-value function, π θ′ φ(S′j ) represents the current state S' j by the policy function π θ The optimal action of the decision is obtained, and ω' is the parameter of the Q value function Q';

[0038] The loss of the Critic network is calculated by using the mean square error loss function, and the expression is as follows:

[0039]

[0040] In the formula, S j is the current state space, A j is the action decision made under the current state;

[0041] The loss of the Actor network uses the negative value of the target Q returned by the state estimation network, and the expression is as follows:

[0042]

[0043] Further, the asynchronous frozen game confrontation training method comprises:

[0044] Initialize the unmanned aerial vehicle decision model of the enemy and oneself, and the initial model of the self is A0, and the initial model of the enemy is B0;

[0045] Fix the model B0, and train the model A0, and obtain the model A1 after training convergence;

[0046] Fix the model A1, and train the model B0, and obtain the model B1 after training convergence;

[0047] After there are at least two models of A and B models, in each round of training, the fixed party extracts the latest model from the model library with a probability of λ, and extracts the historical model with a probability of 1-λ, until one of the enemy and the self appears reward convergence negative, and the asynchronous frozen game confrontation training process is ended;

[0048] After the victory or defeat is determined, the unmanned aerial vehicle decision model of different reinforcement learning is evaluated, and the model of the winning party is evaluated to be stronger, that is, the designed deep reinforcement learning method is more excellent.

[0049] The second technical scheme adopted by the application is:

[0050] An unmanned aerial vehicle asynchronous frozen game confrontation training device comprises:

[0051] A model construction module is used for constructing an unmanned aerial vehicle decision model and initializing model parameters;

[0052] A model training module is configured to design a coupling reward function considering the angle and distance comprehensively, and train the UAV decision model according to the coupling reward function.

[0053] An adversarial training module is configured to train the UAV decision model of the own side and the enemy side by using an asynchronous frozen game adversarial training method, and use the trained model to control the UAV.

[0054] The third technical solution adopted by the present application is:

[0055] An electronic device includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the above-mentioned UAV asynchronous frozen game adversarial training method.

[0056] The fourth technical solution adopted by the present application is:

[0057] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement the above-mentioned UAV asynchronous frozen game adversarial training method.

[0058] The fifth technical solution adopted by the present application is:

[0059] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to make the computer device execute the above-mentioned UAV asynchronous frozen game adversarial training method.

[0060] The present application has the following advantages: the present application can improve the UAV close-range air combat model training speed and model maneuvering decision performance through the coupling reward function. In addition, based on the asynchronous frozen game adversarial method, the performance of different reinforcement learning models can be objectively and reasonably evaluated by direct confrontation, and the performance of the reinforcement learning model can be effectively and continuously improved. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for facilitating the clear description of part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without paying creative labor on the premise.

[0062] Figure 1 is a schematic diagram of coupling the reward three-dimensional space in the embodiments of the present application;

[0063] Figure 2 is a schematic diagram of training of the unmanned aerial vehicle decision model in the embodiments of the present application;

[0064] Figure 3 is a schematic diagram of asynchronous frozen game confrontation training in the embodiments of the present application;

[0065] Figure 4 is a step flow chart of an unmanned aerial vehicle asynchronous frozen game confrontation training method in the embodiments of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for facilitating the description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0067] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0068] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0069] In the description of the present application, the words such as arrangement, installation, connection and the like should be understood in a broad sense unless otherwise explicitly limited, and the person skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0070] In order to make the depth reinforcement learning maneuver decision model of the unmanned aerial vehicle of the present application free from considering the weight problems of angle and distance, the present application designs a coupling reward model considering angle and distance, which greatly improves the training speed and the maneuver decision performance of the model. Further, the present application proposes an asynchronous frozen game training method to improve the maneuver decision model performance of the unmanned aerial vehicle, proposes a new method for distinguishing the performance of different depth reinforcement learning maneuver decision models, and overcomes the problem that different models are difficult to compare with each other in the form of confrontation due to overfitting of reinforcement learning.

[0071] Embodiment 1

[0072] As shown in Figure 4 , the present embodiment provides an asynchronous frozen game confrontation training method for unmanned aerial vehicle, comprising the following steps:

[0073] S1, constructing an unmanned aerial vehicle decision model and initializing the model parameters.

[0074] In the present embodiment, the unmanned aerial vehicle decision model is a depth reinforcement learning network model, including an Actor network and a Critic network. After constructing the model, the parameters of the depth reinforcement learning network model are initialized, and the state information of the unmanned aerial vehicle is initialized.

[0075] S2, designing a coupling reward function considering angle and distance, and training the unmanned aerial vehicle decision model according to the coupling reward function.

[0076] In the present embodiment, the coupling reward function is used to evaluate the good or bad degree in the current environment, and the parameters α, β and γ in the function are used to adjust the steepness of the reward. Specifically, the expression of the coupling reward function is as follows:

[0077]

[0078] s.t.α2D best -γ2=β2D best ,α2>0,β2<0

[0079] In some embodiments, the battlefield environment situation information is obtained by the fire control radar of the unmanned aerial vehicle, and data integration and normalization processing are performed to obtain environment state data satisfying the depth reinforcement learning neural network, including real-time position, flight speed, overload and orientation angle of the unmanned aerial vehicle. The situation information after secondary processing is defined as:

[0080] [x r ,yr ,z r ,v r ,v rx ,v ry ,v rz ,x b ,y b ,z b ,v b ,v bx ,v by ,v bz ]

[0081] where x, y, z represent the three-dimensional position information of the UAV, and the subscripts r and b represent the friendly UAV and the enemy UAV, respectively; v represents the speed, and v x , v y , v z represent the components of the speed in the three axes, respectively.

[0082] After integrating the collected position and speed information of the friendly and enemy UAVs, the environmental situation information s is obtained; the environmental situation information s is input into the Actor network to obtain the execution action a of the current UAV.

[0083] The UAV executes the action a of the Actor network to obtain new environmental situation information s', and updates the environmental situation information. According to the coupling reward function, the reward r of the current action a is obtained, which is specifically as follows:

[0084] r = f(s')

[0085] According to the new environmental situation information s', it is judged whether to end, and the result is saved in B, and the data {s, a, r, s', B} is saved to the experience replay pool M;

[0086] The environmental situation information s is updated to s = s';

[0087] The data in the experience replay pool M is sampled, and the network parameters are updated.

[0088] As an optional implementation, the specific process of updating the network parameters is as follows:

[0089] A1, sample m samples from the experience replay set M, {s i ,a i ,r i ,s′ i ,B i}, i = 1, …, m, and calculate the current target Q value y j :

[0090]

[0091] where γ is the discount factor, and Q' is computed by the Target_Critic network.

[0092] A2, Critic network loss uses mean squared error loss function:

[0093]

[0094] Update all parameters ω of the Critic current network by gradient backpropagation through the neural network.

[0095] A3, Actor network loss uses the negative value of Q returned by the state estimation network:

[0096]

[0097] Update all parameters θ of the Actor current network by gradient backpropagation through the neural network.

[0098] A4, periodically update the Critic target network and Actor target network parameters. Network parameter updates use soft updates, only a little bit is updated each iteration.

[0099] ω'←τω+(1—τ)ω′

[0100] θ'←τθ+(1-τ)θ′

[0101] where τ is the update coefficient, generally take a small value.

[0102] A5, loop iteration steps A1-A4 until the termination condition.

[0103] S3, use the asynchronous frozen game confrontation training method to train the unmanned aerial vehicle decision-making model of our side and the enemy side, and use the trained model to control the unmanned aerial vehicle.

[0104] In some embodiments, after the deep reinforcement learning model converges, different deep reinforcement learning model confrontation training is performed, denoted as the initial model of our side as A0 and the initial model of the enemy side as B0; the model B0 is fixed, the model A0 is trained, and the model A1 is obtained after the model A0 is trained and converged; the model A1 is fixed, the model B0 is trained, and the model B1 is obtained after the model B0 is trained and converged.

[0105] After there are at least two models of A and B models, each time the training is alternated, in each round of training, the fixed party extracts the latest model from its model library with a certain probability λ, and extracts the historical model with a probability of 1-λ. Until one of the enemy and me appears reward convergence is negative, then the asynchronous frozen game confrontation training process is ended.

[0106] After the winner is determined, the different reinforcement learning near combat maneuver decision models can be evaluated, and the model of the winner is stronger, that is, the designed deep reinforcement learning method is better.

[0107] The above method is further described below in combination with the drawings and specific embodiments.

[0108] (1) in combination Figure 1 The coupling reward model described in the application is designed as follows:

[0109] Step 1: determine the required optimal air combat distance D best , that is Figure 1 The position of the dividing line between the two curved surfaces.

[0110] Step 2: determine the three parameters α, β and γ, which mainly adjust the concave and convex degree of the curved surface.

[0111] (2) in combination Figure 2 The embodiment of the application is based on the standard reinforcement learning model, and the coupling reward reinforcement learning model training is carried out, which specifically includes the following steps:

[0112] Step 1: initialize the unmanned aerial vehicle environment setting, including position, speed, etc.

[0113] Step 2: process the position and speed, and splice into state s

[0114] s=[x r ,y r ,z r ,v r ,v rx ,v ry ,v rz ,x b ,y b ,z b ,v b ,v bx ,v by ,v bz ]

[0115] In the formula, x, y and z respectively represent three-dimensional position information of the unmanned aerial vehicle, and the subscripts r and b respectively represent the unmanned aerial vehicle and the enemy unmanned aerial vehicle; v represents the speed, and v x , v y , v z respectively represent the components of the speed in the three axes.

[0116] Step 3: collect the position and attitude information of the enemy and our unmanned aerial vehicles by using the fire control radar, set the integrated information as s. Input s into the Actor network to obtain the execution action a of the current unmanned aerial vehicle.

[0117] Step 4: Get new environment information by performing action a on each drone, set as s', update the environment situation information. According to the set reward and punishment rules, combine Figure 2 , get the reward r of action a, as follows:

[0118] r=f(s′)

[0119] In the formula, the function f is a self-defined evaluation function, which aims to evaluate the degree of good or bad of the action a strategy in the current environment.

[0120] Step 5: Determine whether to end according to s', save the result in D. Save the data {s, a, r, s', D} to the experience replay pool M.

[0121] Step 6: Update the state s, make s=s'.

[0122] Step 7: Update the experience pool information in real time. Randomly sample m samples from the experience replay set M, {s i ,a i ,r i ,s′ i ,B i}, i=1,…,m, calculate the current target Q value y j :

[0123]

[0124] In the formula, γ is the discount factor, and Q' is calculated by the Target_Critic network.

[0125] Step 8: The Critic network loss uses the mean square error loss function:

[0126]

[0127] Update all parameters ω of the Critic current network through gradient backpropagation of the neural network.

[0128] Step 9: The Actor network loss uses the negative value of Q returned by the state estimation network:

[0129]

[0130] Update all parameters θ of the Actor current network through gradient backpropagation of the neural network.

[0131] Step 10: Update the Critic target network and Actor target network parameters regularly. Network parameter updates use soft updates, and only a little bit is updated each iteration.

[0132] ω'←τω+(1-τ)ω′

[0133] θ'←τθ+(1-τ)θ′

[0134] Wherein, p is an update coefficient, generally takes a smaller value.

[0135] Step 11: loop the above steps until the termination condition is met.

[0136] (3) combined Figure 3 , the process of asynchronous frozen game confrontation training is:

[0137] Step 1: initialize the enemy and me reinforcement learning initial model, that is, A0, B0; first fix the model B0, train the model A0, and get the model A1.

[0138] Step 2: fix the model A1, train the model B0, until the model B0 model converges to the model B1.

[0139] Step 2: after A and B models exist at least two models, each time the training is alternated, and in each round of training, the fixed party extracts the latest model from its model library with a certain probability λ, and extracts the past historical model with a probability of 1-λ. Until one of the enemy and me appears reward convergence is negative, then the asynchronous frozen game confrontation training process is ended.

[0140] Step 3: after the victory and defeat are divided through step 2, the different reinforcement learning close combat maneuver decision models can be evaluated, and the winning side is evaluated as stronger, that is, the designed deep reinforcement learning method is more excellent.

[0141] Embodiment 2

[0142] The embodiment provides an unmanned aerial vehicle asynchronous frozen game confrontation training device, comprising:

[0143] A model construction module is configured to construct an unmanned aerial vehicle decision model and initialize model parameters.

[0144] A model training module is configured to design a coupling reward function considering angle and distance comprehensively, and train the unmanned aerial vehicle decision model according to the coupling reward function.

[0145] An antagonistic training module is configured to use an asynchronous frozen game confrontation training method to train the unmanned aerial vehicle decision model of the enemy and me, and use the trained model to control the unmanned aerial vehicle.

[0146] Since the device is an unmanned aerial vehicle asynchronous frozen game confrontation training device of the embodiment of the application, and the principle of solving the problem of the device is similar to that of the method, the implementation of the device can be referred to the implementation process of the above method embodiment, and the repeated parts will not be described here.

[0147] Embodiment 3

[0148] The electronic device includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method. Figure 4 The method is a UAV asynchronous frozen game confrontation training method.

[0149] It can be understood that the memory can include a random access memory (RAM) and a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, and the like; and the data storage area can store data created according to the use of the server, and the like.

[0150] The processor can include one or more processing cores. The processor connects various parts within the server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0151] Since the electronic device is a corresponding electronic device of the UAV asynchronous frozen game confrontation training method of the embodiments of the present application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0152] Embodiment 4

[0153] The embodiment of the present application also provides a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the method as Figure 4 The method shown is a UAV asynchronous frozen game confrontation training method.

[0154] Those skilled in the art can understand that all or part of the steps of the various methods in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including a Read-Only Memory (ROM), a Random Access Memory (RAM), a Programmable Read-only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an Electrically-Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disk memories, magnetic disk memories, magnetic tape memories, or any other computer readable medium that can be used to carry or store data.

[0155] Since the storage medium is a storage medium corresponding to the UAV asynchronous frozen game confrontation training method of the embodiment of the present application, and the principle of solving problems of the storage medium is similar to that of the method, therefore, the implementation of the storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described here.

[0156] Embodiment 5

[0157] In some possible implementation manners, each aspect of the method of the embodiment of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the method of the unmanned aerial vehicle asynchronous frozen game confrontation training method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. The executable computer program codes or "codes" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0158] It should be understood that various parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods can be realized in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized in hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application-specific integrated circuits (ASICs) having appropriate combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and the like.

[0159] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the present specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0160] The above-described embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for training an unmanned aerial vehicle (UAV) asynchronous frozen game confrontation, characterized in that, The method comprises the following steps: constructing a UAV decision model and initializing model parameters; designing a coupling reward function considering both angle and distance, and training the UAV decision model according to the coupling reward function; training the UAV decision models of both parties by using an asynchronous frozen game confrontation training method, and using the trained models to control UAVs; The expression of the coupling reward function is as follows: In the formula, represents the distance between the two UAVs, represents the optimal air combat distance, ATA represents the angle between the velocity vector of the UAV of the own side and the distance vector relative to the target UAV; AA represents the angle between the velocity vector of the UAV of the enemy side and the distance vector between it and the UAV of the own side; , , is used for adjusting the space form parameters of the reward function; The asynchronous frozen game confrontation training method comprises: Initialize the decision model of the unmanned aerial vehicles of both sides, and let the initial model of our side be , and the initial model of the enemy side be ; Fixed model , the model is trained , and the model is obtained after training convergence ; Fixed model , the model is trained , and the model is obtained after training convergence ; After both A and B models exist at least two or more models, each time alternating training, in each round of training, the fixed party extracts from its model library with a probability the latest model, and with a probability extracts the past historical model, until one of the two parties appears reward convergence to negative, ending the asynchronous frozen game confrontation training process; After determining the winner, the UAV decision models of different reinforcement learning are evaluated, and the model of the winning party is evaluated as stronger, that is, the deep reinforcement learning method designed by the winning party is more excellent. 2.The method of claim 1, wherein, The UAV decision model is a deep reinforcement learning network model, and the UAV decision model comprises an Actor network and a Critic network. The UAV decision model is trained by the following method: Integrate the position and speed information of the unmanned aerial vehicle of the enemy and the unmanned aerial vehicle of the party to obtain environmental situation information ; obtaining environmental situation information inputting the actor network to obtain an action of the current unmanned aerial vehicle ; by performing an action , obtaining new environmental situation information ; According to environmental situation information and coupling reward function, get action reward ; According to the new environmental situation information determines whether the game is over, and the result B is recorded as True for a win, False for a draw or a loss, and the result B and other information are formed together as saved into the experience replay pool M; Updating environmental situation information , make ; Sample data in the experience replay pool M and update network parameters. 3.The method of claim 2, wherein, The environmental situation information is defined as: In the formula, respectively represent three-dimensional position information of the unmanned aerial vehicle, and subscript and respectively represent the unmanned aerial vehicle and the enemy unmanned aerial vehicle; represent the speed, respectively represent the components of the speed in the three axes. 4.The method of claim 2, wherein, The sampling of data in the experience replay pool M and the updating of network parameters comprise: sampling a sample from the experience replay set M, computing a current target value Q by the Critic network;​​ updating all parameters of the Critic current network by gradient backpropagation through the neural network ; updating all parameters of the current network of the actor by gradient backpropagation of the neural network ; Periodically update the Critic target network and the Actor target network parameters: In the formula, is an update coefficient.

5. The unmanned aerial vehicle asynchronous frozen game confrontation training method according to claim 4, characterized in that, Object Q Values The calculation formula is as follows: wherein, is a discount factor, is a reward value obtained at the current step, denotes the next state expected reward obtained in the Q-value function, denotes the optimal action based on the current state by the policy function the decision optimal action obtained, is a parameter of the Q-value function . Calculate the loss of the Critic network by using a mean square error loss function, and the expression is as follows: In the formula, is the current state, is the decision action under the current state; The loss of the actor network uses the negative of the target returned by the state estimation network, expressed as follows: Q ​ 6. An unmanned aerial vehicle asynchronous frozen game confrontation training device, characterized in that, It comprises: A model construction module is configured to construct a UAV decision model and initialize model parameters; A model training module is configured to design a coupling reward function considering both angle and distance, and train the UAV decision model according to the coupling reward function; An adversarial training module is configured to train the UAV decision models of both parties by using an asynchronous frozen game confrontation training method, and use the trained models to control UAVs; The expression of the coupling reward function is as follows: In the formula, represents the distance between the two UAVs, represents the optimal air combat distance, ATA represents the angle between the velocity vector of the UAV of the own side and the distance vector relative to the target UAV; AA represents the angle between the velocity vector of the UAV of the enemy side and the distance vector between it and the UAV of the own side; , , is a parameter for adjusting the space form of the reward function; The asynchronous frozen game confrontation training method comprises: Initialize the decision model of the unmanned aerial vehicles of both sides, and let the initial model of our side be , and the initial model of the enemy side be ; Fixed model , the model is trained , and the model is obtained after training converges ; Fixed model , the model is trained, and the model is obtained after training convergence After both A and B models exist at least two or more models, each time alternating training, in each round of training, the fixed party extracts from its model library with a probability the latest model, and with a probability extracts the past historical model, until one of the two parties appears reward convergence to negative, end the asynchronous frozen game confrontation training process; After determining the winner, the UAV decision models of different reinforcement learning are evaluated, and the model of the winning party is evaluated as stronger, that is, the deep reinforcement learning method designed by the winning party is more excellent.

7. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one program, which is loaded and executed by the processor to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is loaded and executed by the processor to implement the method of any one of claims 1 to 5.

Images