Unmanned aerial vehicle asynchronous freezing game confrontation training method, device, equipment and medium

Abstract

The invention discloses an unmanned aerial vehicle asynchronous freezing game confrontation training method, device and equipment and a medium, and belongs to the field of unmanned aerial vehicle short-distance air combat maneuvering decision making. The method comprises the following steps: constructing an unmanned aerial vehicle decision model, and initializing model parameters; designing a coupling reward function comprehensively considering the angle and the distance, and training an unmanned aerial vehicle decision model according to the coupling reward function; and training the unmanned aerial vehicle decision models of our party and enemies by adopting an asynchronous freezing game confrontation training method, and using the trained model to control the unmanned aerial vehicle. By coupling the reward function, the training speed of the near-distance air combat model of the unmanned aerial vehicle and the maneuvering decision-making performance of the model can be improved. Besides, based on the asynchronous freezing game confrontation method, the performance of different reinforcement learning models can be objectively and reasonably evaluated in a direct confrontation mode, and the performance of the reinforcement learning models is effectively and continuously improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) close-range air combat maneuvering decision-making, and in particular to a method, device, equipment and medium for asynchronous freezing game confrontation training of unmanned aerial vehicles. Background Art

[0002] As an important part of modern military confrontation, air combat has brought great challenges to the decision-making process due to its complexity and high risk, especially in close-range air combat, where fast and accurate maneuvering decisions are crucial. The widespread use of unmanned aerial vehicles (UAVs) plays an increasingly important role in modern air combat due to their flexibility, safety and low cost. UAVs can not only perform reconnaissance and attack missions, but also play a key role in high-risk air combat environments. However, the difficulty of decision-making in air combat comes from the rapid changes in the environment, the complex maneuvers of enemy aircraft, and the real-time requirements of the decision-making process.

[0003] In order to improve the autonomous air combat decision-making ability of UAVs, many studies have tried to use methods such as game theory and optimization theory to solve this problem. Although these methods have certain effects in theory, they often face great computational pressure and uncertainty when solving real-time complex problems. Therefore, in recent years, deep reinforcement learning (DRL) has gradually become an important research direction in the field of UAV air combat. With its ability to quickly adapt to complex problems in high-dimensional environments, deep reinforcement learning provides UAVs with a more flexible and efficient autonomous decision-making solution.

[0004] However, when using deep reinforcement learning, when considering the design of the reward function, most related studies consider different situation information rewards separately, such as adding the angle reward to the distance reward in a certain proportion to form a comprehensive reward. One of the challenges is the weight selection problem. Secondly, the decision model obtained by using a single training is mediocre, which is not conducive to improving the performance of the decision model. Summary of the invention

[0005] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a method, device, equipment and medium for asynchronous freezing game adversarial training of drones based on deep reinforcement learning.

[0006] The first technical solution adopted by the present invention is:

[0007] A method for asynchronous freezing game confrontation training of a drone comprises the following steps:

[0008] Build a drone decision model and initialize the model parameters;

[0009] Design a coupled reward function that comprehensively considers angle and distance, and train the drone decision model based on the coupled reward function;

[0010] An asynchronous frozen game adversarial training method is used to train the decision-making models of our and enemy drones, and the trained models are used to control the drones.

[0011] Furthermore, the expression of the coupled reward function is as follows:

[0012]

[0013] stα2D best -γ2=β2D best ,α2>0,β2<0

[0014] In the formula, D represents the distance between the two drones, D best represents the optimal air combat distance, ATA represents the angle between the velocity vector and the distance vector of our drone relative to the target drone; AA represents the angle between the velocity vector of the enemy drone and the distance vector between it and our drone; α2, β2, γ2 are the parameters for adjusting the spatial morphology of the reward function to satisfy the constraint α2D best -γ2=β2D best When D≤D best The reward function is similar to D ≥ D best The reward function of is connected in space, which is conducive to training convergence.

[0015] Furthermore, the drone decision model is a deep reinforcement learning network model, and the drone decision model includes an Actor network and a Critic network;

[0016] The drone decision model is trained in the following way:

[0017] Integrate the collected position and speed information of our UAV and enemy UAV to obtain environmental situation information s;

[0018] Input the environmental situation information s into the Actor network to obtain the current drone action a;

[0019] By executing action a, new environmental situation information s′ is obtained;

[0020] According to the environmental situation information s′ and the coupled reward function, the reward r of action a is obtained;

[0021] Determine whether the game is over based on the new environmental situation information s′. If it is a win, record the result B as True, if it is a draw or a loss, record it as False, and save the result B together with other information into {s, a, r, s′, B} in the experience replay pool M;

[0022] Update the environmental situation information s so that s = s′;

[0023] Sample the data in the experience replay pool M and update the network parameters.

[0024] Furthermore, the environmental situation information s is defined as:

[0025]

[0026] In the formula, x, y, and z represent the three-dimensional position information of the UAV, respectively; subscripts r and b represent our UAV and enemy UAV, respectively; v represents the speed, v x 、v y 、v z Represents the components of velocity in three axes respectively.

[0027] Furthermore, the sampling experience replays the data in the pool M and updates the network parameters, including:

[0028] Sample m samples from the experience replay set M, {s i ,a i ,r i ,s′ i ,B i}, i = 1, ..., m, calculate the current target Q value y through the Critic network j ;

[0029] Update all parameters ω of the Critic's current network through the gradient back propagation of the neural network;

[0030] Update all parameters θ of the Actor's current network through gradient backpropagation of the neural network;

[0031] Regularly update the Critic target network and Actor target network parameters:

[0032] ω'←τω+(1-τ)ω′

[0033] θ'←τθ+(1-τ)θ'

[0034] Where τ is the update coefficient.

[0035] Furthermore, the target Q value y j The calculation formula is as follows:

[0036]

[0037] In the formula, γ is the discount factor, R j is the reward value obtained at the current step, Q′(φ(S′ j )) represents the next state S′ j The expected reward obtained in the Q-value function, π θ′ φ(S′j ) represents the current state S′ j Through the policy function π θ The optimal decision action obtained, ω′ is the parameter of the Q-value function Q′;

[0038] The mean square error loss function is used to calculate the loss of the Critic network, and the expression is as follows:

[0039]

[0040] In the formula, S j is the current state space, A j Decision making for the current state;

[0041] The loss of the actor network uses the negative value of the target Q returned by the state estimation network and is expressed as follows:

[0042]

[0043] Furthermore, the asynchronous freezing game adversarial training method comprises:

[0044] Initialize the decision-making models of the enemy and our drones, and note that our initial model is A0 and the enemy's initial model is B0;

[0045] Fix model B0, train model A0, and obtain model A1 after training convergence;

[0046] Fix model A1, train model B0, and obtain model B1 after training convergence;

[0047] After there are at least two models of both A and B, each time the training is alternating, in each round of training, the fixed party extracts the latest model from its model library with probability λ, and extracts the previous historical model with probability 1-λ, until the reward converges to negative for one of the two parties, and the asynchronous frozen game adversarial training process ends;

[0048] After the winner is determined, the drone decision-making models of different reinforcement learning are evaluated, and the winner is judged to have a stronger model, that is, the deep reinforcement learning method it designed is better.

[0049] The second technical solution adopted by the present invention is:

[0050] A drone asynchronous freezing game confrontation training device, comprising:

[0051] Model building module, used to build the UAV decision model and initialize the model parameters;

[0052] The model training module is used to design a coupled reward function that comprehensively considers angle and distance, and train the drone decision model based on the coupled reward function;

[0053] The adversarial training module is used to train the decision-making models of our and enemy drones using an asynchronous freezing game adversarial training method, and use the trained models to control the drones.

[0054] The third technical solution adopted by the present invention is:

[0055] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the asynchronous freezing game adversarial training method for a drone as described above.

[0056] The fourth technical solution adopted by the present invention is:

[0057] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned drone asynchronous freezing game adversarial training method.

[0058] The fifth technical solution adopted by the present invention is:

[0059] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned drone asynchronous freezing game confrontation training method.

[0060] The beneficial effect of the present invention is that the present invention can improve the training speed of the UAV close-range air combat model and the model maneuver decision-making performance by coupling the reward function. In addition, based on the asynchronous frozen game confrontation method, the performance of different reinforcement learning models can be objectively and reasonably evaluated by direct confrontation, and the performance of the reinforcement learning model can be effectively and continuously improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 is a schematic diagram of a coupled reward three-dimensional space in an embodiment of the present invention;

[0063] Figure 2 is a schematic diagram of the training of the UAV decision model in an embodiment of the present invention;

[0064] Figure 3 is a schematic diagram of asynchronous freezing game adversarial training in an embodiment of the present invention;

[0065] Figure 4 It is a flowchart of the steps of a method for asynchronous freezing game confrontation training of a drone in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0067] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0068] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0069] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution.

[0070] In order to exempt the training of our drone's deep reinforcement learning maneuver decision model from the weight problem of angle and distance, the present invention designs a coupled reward model that comprehensively considers angle and distance, which greatly improves the training speed and model maneuver decision performance. Furthermore, the present invention proposes an asynchronous frozen game training method to improve the performance of the drone maneuver decision model, and proposes a new method to distinguish the performance of different deep reinforcement learning maneuver decision models, overcoming the problem of overfitting of reinforcement learning that makes it difficult to compare different models in a confrontational form.

[0071] Example 1

[0072] like Figure 4 As shown, this embodiment provides a drone asynchronous freezing game confrontation training method, comprising the following steps:

[0073] S1. Build a drone decision model and initialize the model parameters.

[0074] In this embodiment, the drone decision model is a deep reinforcement learning network model, including an Actor network and a Critic network. After the model is constructed, the parameters of the deep reinforcement learning network model are initialized, and the drone state information is initialized.

[0075] S2. Design a coupled reward function that comprehensively considers angle and distance, and train the drone decision model based on the coupled reward function.

[0076] In this embodiment, the coupled reward function is used to evaluate the degree of goodness in the current environment. The α, β, and γ parameters in the function are used to adjust the steepness of the reward. Specifically, the expression of the coupled reward function is as follows:

[0077]

[0078] stα2D best -γ2=β2D best ,α2>0,β2<0

[0079] In some embodiments, battlefield environment situation information is obtained through the UAV fire control radar, and data integration and normalization are performed to obtain environmental status data that meets the requirements of the deep reinforcement learning neural network, including the real-time position, flight speed, overload, heading angle, etc. of the UAV. The situation information after secondary processing is defined as:

[0080] [x r ,yr ,z r ,v r ,v rx ,v ry ,v rz ,x b ,y b ,z b ,v b ,v bx ,v by ,v bz ]

[0081] In the formula, x, y, and z represent the three-dimensional position information of the UAV, respectively; subscripts r and b represent our UAV and enemy UAV, respectively; v represents the speed, v x 、v y 、v z Represents the components of velocity in three axes respectively.

[0082] After integrating the collected position and speed information of our and enemy drones, we obtain the environmental situation information s; we input the environmental situation information s into the Actor network to obtain the execution action a of the current drone.

[0083] Execute the Actor network drone to perform action a, obtain new environmental situation information s′, and update the environmental situation information. And according to the coupled reward function, obtain the reward r of the current action a, as follows:

[0084] r = f(s′)

[0085] Determine whether to end based on the new environmental situation information s′, save the result in B, and save the data {s, a, r, s′, B} to the experience replay pool M;

[0086] Update the environmental situation information s so that s = s′;

[0087] Sample the data in the experience replay pool M and update the network parameters.

[0088] As an optional implementation, the specific process of updating the network parameters is as follows:

[0089] A1. Sample m samples from the experience playback set M, {s i ,a i ,r i ,s′ i ,B i}, i = 1, ..., m, calculate the current target Q value y through the Critic network j :

[0090]

[0091] Where γ is the discount factor and Q′ is calculated by the Target_Critic network.

[0092] A2. Critic network loss uses mean square error loss function:

[0093]

[0094] Update all parameters ω of the Critic current network through the gradient back propagation of the neural network.

[0095] A3. Actor network loss uses the negative value of Q returned by the state estimation network:

[0096]

[0097] Update all parameters θ of the Actor's current network through the gradient backpropagation of the neural network.

[0098] A4. Regularly update the parameters of the Critic target network and the Actor target network. The network parameters are updated using soft updates, with only a small amount updated each iteration.

[0099] ω'←τω+(1-τ)ω′

[0100] θ'←τθ+(1-τ)θ′

[0101] Where τ is the update coefficient, which generally takes a relatively small value.

[0102] A5. Loop through steps A1-A4 until the termination condition occurs.

[0103] S3. Use the asynchronous frozen game adversarial training method to train the decision-making models of our and enemy drones, and use the trained models to control the drones.

[0104] In some embodiments, after the deep reinforcement learning model converges, adversarial training of different deep reinforcement learning models is performed. Our initial model is denoted as A0 and the enemy's initial model is denoted as B0; fix model B0, train model A0, and wait for model A0 training to converge to obtain model A1; fix model A1, train model B0, and wait until model B0 converges to obtain model B1.

[0105] After there are at least two models of both A and B, each time the training is alternating, in each round of training, the fixed party extracts the latest model from its model library with a certain probability λ, and extracts the previous historical model with a probability 1-λ. Until one of the enemy and us has a negative reward convergence, the asynchronous frozen game adversarial training process ends.

[0106] After the winner is determined, the different reinforcement learning close-range air combat maneuver decision-making models can be evaluated, and the winner will be evaluated as having a stronger model, that is, the deep reinforcement learning method it designed is better.

[0107] The above method is further described below in conjunction with the accompanying drawings and specific embodiments.

[0108] (1) Combination Figure 1 The coupling reward model of the present invention is designed to be carried out in the following steps:

[0109] Step 1: Determine the optimal air combat distance D required best , that is Figure 1 The location of the dividing line between the mid-surfaces.

[0110] Step 2: Determine the three parameters α, β, and γ. These three parameters mainly adjust the concavity of the surface.

[0111] (2) Combination Figure 2 The embodiment of the present invention uses the standard reinforcement learning model as a benchmark to perform coupled reward reinforcement learning model training, which specifically includes the following steps:

[0112] Step 1: Initialize the drone environment settings, including position, speed, etc.

[0113] Step 2: Process the position and velocity and concatenate them into state s

[0114] s=[x r ,y r ,z r ,v r ,v rx ,v ry ,v rz ,x b ,y b ,z b ,v b ,v bx ,v by ,v bz ]

[0115] In the formula, x, y, and z represent the three-dimensional position information of the UAV, respectively; subscripts r and b represent our UAV and enemy UAV, respectively; v represents the speed, v x 、v y 、v z Represents the components of velocity in three axes respectively.

[0116] Step 3: Use the fire control radar to collect the position and attitude information of the enemy and friendly drones, and set the integrated information as s. Input s into the Actor network to obtain the execution action a of the current drone.

[0117] Step 4: By executing action a on each drone, new environmental information is obtained, set as s′, and the environmental situation information is updated. And according to the set reward and punishment rules, combined with Figure 2 , we get the reward r for action a as follows:

[0118] r = f(s′)

[0119] In the formula, function f is a custom evaluation function, whose purpose is to evaluate the quality of the action a strategy in the current environment.

[0120] Step 5: Determine whether to end based on s′ and save the result in D. Save the data {s, a, r, s′, D} to the experience replay pool M.

[0121] Step 6: Update state s so that s=s′.

[0122] Step 7: Update the experience pool information in real time. Randomly sample m samples from the experience playback set M, {s i ,a i ,r i ,s′ i ,B i}, i = 1, ..., m, calculate the current target Q value y through the Critic network j :

[0123]

[0124] Where γ is the discount factor and Q′ is calculated by the Target_Critic network.

[0125] Step 8: The Critic network loss uses the mean square error loss function:

[0126]

[0127] Update all parameters ω of the Critic current network through the gradient back propagation of the neural network.

[0128] Step 9: The Actor Network loss uses the negative value of Q returned by the state estimation network:

[0129]

[0130] Update all parameters θ of the Actor's current network through the gradient backpropagation of the neural network.

[0131] Step 10: Regularly update the parameters of the Critic target network and the Actor target network. The network parameter update adopts soft update, and only a little bit is updated in each iteration.

[0132] ω'←τω+(1-τ)ω′

[0133] θ'←τθ+(1-τ)θ′

[0134] Where: ρ is the update coefficient, which generally takes a relatively small value.

[0135] Step 11: Iterate the above steps until the termination condition is met.

[0136] (3) Combination Figure 3 , the process of asynchronous freezing game adversarial training is:

[0137] Step 1: Initialize the initial reinforcement learning models of both sides, namely A0 and B0; first fix model B0, train model A0, and obtain model A1.

[0138] Step 2: Fix model A1 and train model B0 until model B0 converges to model B1.

[0139] Step 2: After there are at least two models of both A and B, each time the training is alternating, in each round of training, the fixed party extracts the latest model from its model library with a certain probability λ, and extracts the previous historical model with a probability 1-λ. Until one of the enemy and us has a negative reward convergence, the asynchronous frozen game adversarial training process ends.

[0140] Step 3: After the winner is determined in step 2, the different reinforcement learning close-range air combat maneuver decision models can be evaluated. The winner is considered to have a stronger model, which means that the deep reinforcement learning method it designed is better.

[0141] Example 2

[0142] This embodiment provides a drone asynchronous freezing game confrontation training device, including:

[0143] Model building module, used to build the UAV decision model and initialize the model parameters;

[0144] The model training module is used to design a coupled reward function that comprehensively considers angle and distance, and train the drone decision model based on the coupled reward function;

[0145] The adversarial training module is used to train the decision-making models of our and enemy drones using an asynchronous freezing game adversarial training method, and use the trained models to control the drones.

[0146] Since the device is a drone asynchronous freezing game adversarial training device of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0147] Example 3

[0148] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 4 A drone asynchronous freezing game adversarial training method is shown.

[0149] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0150] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0151] Since the electronic device is an electronic device corresponding to the asynchronous freezing game adversarial training method for a drone in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0152] Example 4

[0153] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 4 A drone asynchronous freezing game adversarial training method is shown.

[0154] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0155] Since the storage medium is a storage medium corresponding to the drone asynchronous freezing game adversarial training method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0156] Example 5

[0157] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the asynchronous freeze game adversarial training method for drones according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0158] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0159] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0160] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A drone asynchronous freezing game adversarial training method, characterized in that: The following steps are involved: Build a drone decision model and initialize the model parameters; Design a coupled reward function that comprehensively considers angle and distance, and train the drone decision model based on the coupled reward function; An asynchronous frozen game adversarial training method is used to train the decision-making models of our and enemy drones, and the trained models are used to control the drones.

2. The method for asynchronous freezing game confrontation training of unmanned aerial vehicles according to claim 1 is characterized in that: The expression of the coupled reward function is as follows: s.t.α2D best -γ2=β2D best ,α2>0,β2<0 In the formula, D represents the distance between the two drones, D best represents the optimal air combat distance, ATA represents the angle between the velocity vector and the distance vector of our UAV relative to the target UAV; AA represents the angle between the velocity vector of the enemy UAV and the distance vector between it and our UAV; α2, β2, γ2 are the parameters for adjusting the spatial morphology of the reward function.

3. The method for asynchronous freezing game confrontation training of unmanned aerial vehicles according to claim 1 is characterized in that: The drone decision model is a deep reinforcement learning network model, and the drone decision model includes an Actor network and a Critic network; The drone decision model is trained in the following way: Integrate the collected position and speed information of our UAV and enemy UAV to obtain environmental situation information s; Input the environmental situation information s into the Actor network to obtain the current drone action a; By executing action a, new environmental situation information s′ is obtained; According to the environmental situation information s′ and the coupled reward function, the reward r of action a is obtained; Determine whether the game is over based on the new environmental situation information s′. If it is a win, record the result B as True, if it is a draw or a loss, record it as False, and save the result B together with other information into {s, a, r, s′, B} in the experience replay pool M; Update the environmental situation information s so that s = s′; Sample the data in the experience replay pool M and update the network parameters.

4. The method for asynchronous freezing game confrontation training of unmanned aerial vehicles according to claim 3 is characterized in that: The environmental situation information s is defined as: In the formula, x, y, and z represent the three-dimensional position information of the UAV, respectively; subscripts r and b represent our UAV and enemy UAV, respectively; v represents the speed, v x 、v y 、v z Represents the components of velocity in three axes respectively.

5. The method for asynchronous freezing game confrontation training of unmanned aerial vehicles according to claim 3 is characterized in that: The sampled experience replays the data in the pool M and updates the network parameters, including: Sample m samples from the experience replay set M, {s i , a i , r i , s′ i , B i }, i = 1, ..., m, calculate the current target Q value y through the Critic network j ; Update all parameters ω of the Critic's current network through the gradient back propagation of the neural network; Update all parameters θ of the Actor's current network through gradient backpropagation of the neural network; Regularly update the Critic target network and Actor target network parameters: ω′←τω+(1-τ)ω′ θ′←τθ+(1-τ)θ′ Where τ is the update coefficient.

6. The method for asynchronous freezing game confrontation training of unmanned aerial vehicles according to claim 5, characterized in that: Target Q value y j The calculation formula is as follows: In the formula, γ is the discount factor, R j is the reward value obtained at the current step, Q′(φ(S′ j )) represents the next state S′ j The expected reward obtained in the Q-value function, π θ′ φ(S′ j ) represents the current state S′ j Through the policy function π θ The optimal decision action obtained, ω′ is the parameter of the Q-value function Q′; The mean square error loss function is used to calculate the loss of the Critic network, and the expression is as follows: In the formula, S j is the current state, A j is the decision action in the current state; The loss of the actor network uses the negative value of the target Q returned by the state estimation network and is expressed as follows:

7. The method for asynchronous freezing game confrontation training of unmanned aerial vehicles according to claim 1, characterized in that: The asynchronous freezing game adversarial training method comprises: Initialize the decision-making models of the enemy and our drones, and note that our initial model is A0 and the enemy's initial model is B0; Fix model B0, train model A0, and obtain model A1 after training convergence; Fix model A1, train model B0, and obtain model B1 after training convergence; After there are at least two models of both A and B, each time the training is alternating, in each round of training, the fixed party extracts the latest model from its model library with probability λ, and extracts the previous historical model with probability 1-λ, until the reward converges to negative for one of the two parties, and the asynchronous frozen game adversarial training process ends; After the winner is determined, the drone decision-making models of different reinforcement learning are evaluated, and the winner is judged to have a stronger model, that is, the deep reinforcement learning method it designed is better.

8. A drone asynchronous freezing game confrontation training device, characterized in that: include: Model building module, used to build the UAV decision model and initialize the model parameters; The model training module is used to design a coupled reward function that comprehensively considers angle and distance, and train the drone decision model based on the coupled reward function; The adversarial training module is used to train the decision-making models of our and enemy drones using an asynchronous freezing game adversarial training method, and use the trained models to control the drones.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images