Multi-scene generalization training method for unmanned aerial vehicle to break through air defense and attack target and application

Abstract

The invention discloses a multi-scene generalization training method for an unmanned aerial vehicle to break through air defense and attack a target. The multi-scene generalization training method comprises the following steps: designing a plurality of training scenes of various types; key information of an A-party unmanned fighter aircraft, a B-party air defense system and a B-party target in the training scene is obtained; extracting state information from the key information; converting the state information with the time sequence information into feature information through a long short-term memory (LSTM) network; the unmanned fighter controlled through a reinforcement learning algorithm interacts with the training scene to generate training data; and multi-scene generalization training is carried out, and alternate training and model parameter updating are carried out in multiple types of multiple training scenes. According to the method, the generalization ability of the unmanned fighter controlled by the deep reinforcement learning algorithm in a real scene is improved, so that the unmanned fighter controlled by the reinforcement learning algorithm can break through a B-party air defense system and attack a B-party target when facing B-party complex and changeable air defense system deployment. The invention further discloses application of the method.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicle flight control, and relates to a multi-scenario generalized training method and application of an unmanned aerial vehicle breaking through air defense and attacking a target. Background Art

[0002] With the development of artificial intelligence technology, the control performance of drones has been significantly improved, and drones have gradually been used in the military field. Relying on drones to complete patrols and strategic military strikes can better grasp the initiative of the war. Therefore, it is of great significance to improve the ability of unmanned fighter jets to break through the air defense system of party B and strike the target point.

[0003] Traditional air combat decision-making algorithms mainly include algorithms based on optimization models, genetic algorithms, and intelligent optimization algorithms. Algorithms based on optimization models can analyze and process complex problems in the form of mathematical models, solve optimal solutions with high reliability, but they need to establish a correct model for the problem, which consumes time and effort, and the solutions for complex environments may not be globally optimal. Genetic algorithms have the ability to search globally, can explore the solution space in multiple dimensions, and have good robustness for complex problems, but they need to select appropriate evaluation functions and crossover mutation operations, otherwise they may fall into local optimality. Intelligent optimization algorithms are suitable for searching for optimal solutions in multiple dimensions, and can search without prior knowledge, but they are highly random and the search results are unstable.

[0004] With the development of artificial intelligence technology, deep reinforcement learning algorithms are becoming more and more mature in various fields. Deep reinforcement learning algorithms learn by interacting with the environment. Reinforcement learning is widely used in the field of drones. Drones learn how to detect and track targets through reinforcement learning algorithms, and use reinforcement learning to help drones defend and attack in the military field. However, there is a huge difference between the training environment of drones and the real environment, which causes drones controlled by reinforcement learning algorithms to make unreasonable decisions in the real environment. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a generalized training method for training unmanned aerial vehicles to break through the air defense system of Party B and attack the targets of Party B based on a reinforcement learning algorithm in multiple scenarios. The core of the method is to improve the generalization ability of unmanned combat aircraft controlled by the reinforcement learning algorithm in real scenarios by training the reinforcement learning algorithm in multiple scenarios. After searching, no prior art or report related to the present invention has been found. The present invention designs multiple categories of training scenarios, and the unmanned combat aircraft controlled by the deep reinforcement learning algorithm can successfully complete the task of breaking through the air defense system of Party B and attacking the targets of Party B in different scenario settings.

[0006] The present invention proposes a multi-scenario generalized training method for a UAV to break through air defense and attack a target, comprising the following steps:

[0007] Step 1: Design multiple types of unmanned combat aircraft training scenarios (including training phase scenarios and test phase scenarios);

[0008] Step 2: Obtain key information about Party A’s fighter jets, Party B’s air defense system, and Party B’s targets in the training scenario;

[0009] Step 3: Extract state information s from the key information in step 2; convert the state information s with time series information into feature information e through a long short-term memory network (LSTM); that is, process the key information of the fighter jet of party A, the air defense system of party B, and the target of party B as the state information of the reinforcement learning algorithm;

[0010] Step 4: The unmanned fighter controlled by the reinforcement learning algorithm interacts with the training scene, plans the direction of the unmanned fighter, saves the trajectory data including the state transition process and reward information, and learns the strategy of the unmanned fighter to break through the air defense system of party B and attack the target of party B;

[0011] Step 5: Conduct multi-scenario generalization training, perform alternating training in multiple training scenarios of different types, store the trajectory data trained in multiple scenarios in the experience buffer pool, update the model parameters through the trajectory data in multiple scenarios, and improve the generalization ability of the reinforcement learning algorithm, so that the unmanned fighter controlled by the reinforcement learning algorithm can break through the complex and changing air defense system deployment of Party B and attack Party B's targets.

[0012] In the step 1, designing multiple types of unmanned combat aircraft training scenarios includes the following steps:

[0013] Step a1. Set the two parties as Party A and Party B;

[0014] Step a2. Design the initial scenario t0: Party A initializes an unmanned fighter carrying an attack missile, and Party B is set to an air defense system consisting of N (N≥1) air defense facilities and a Party B base (the target point attacked by Party A);

[0015] Step a3. Design various types of training or testing scenarios:

[0016] (1) The first type T a :Based on t0, the positions of the air defense system facilities and the target points of party B are fixed, and the positions of the unmanned fighters of party A are randomly modified to generate n (n≥3) training scenarios, T a ={t a1 , t a2 ,...,t an}; Generate n2 (n2 ≥ 1 test scenarios,

[0017] (2) The second type T b :Based on t0, the positions of the A-side UCAV and the B-side base are fixed, and the initial positions of multiple air defense facilities of the B-side are changed to generate m (m ≥ 3) training scenarios, T b ={t b1 , t b2 ,...,t bm}; Generate m2 (m2 ≥ 1) test scenarios,

[0018] (3) The third type T c :Based on t0, the positions of A's unmanned fighter and B's air defense facilities are fixed, and the position of B's ​​base is randomly changed to generate k (k ≥ 3) training scenarios, T c ={t c1 , t c2 ,...,t ck}; Generate k2 (k2 ≥ 1) test scenarios,

[0019] Step a4. Save the generated n+m+k scenes as scenes in the training phase; and use these n+m+k scenes as scenes in the testing phase;

[0020] Step a5. Save the generated n2+m2+k2 scenarios as test phase scenarios, that is, the test phase shares n+m+k+n2+m2+k2 scenarios.

[0021] In the step 2, obtaining key information from the training scene includes the following steps:

[0022] Step b1. Filter out the longitude information lon of Party A's unmanned combat aircraft from the information provided by the training scenario red , latitude information lat red and direction information head red ;

[0023] Step b2. During the training process, if the B-party facility (including the B-party air defense facility or the B-party base) enters the reconnaissance range of the A-party unmanned fighter, the information of the B-party facility is filtered out from the B-party information provided by the training scene. If the B-party facility is the B-party base (the A-party target point), the longitude information of the B-party base is filtered out. and latitude information If the B party facility is an air defense facility, filter out the longitude information of the air defense system facility Latitude information And direction information i≥1.

[0024] Key Information Where C represents the number of air defense facilities that constitute the air defense system of Party B.

[0025] In the step 3, extracting the status information from the key information in step 2 includes the following steps:

[0026] Step c1. Calculate the distance between the A-side UCAV and the B-side facility using the formula:

[0027]

[0028] Where r represents the radius of the earth, r≈6371km, arcsin is the inverse sine function, sin is the sine function, and cos is the cosine function;

[0029] Use d i (0≤i≤C) represents the distance between the A-side unmanned fighter and the i-th B-side facility; i=0 represents the B-side base; 1≤i≤C represents the i-th B-side air defense facility, and C is the number of B-side air defense facilities.

[0030] Step c2. Calculate the difference in the orientation between the A-side UCAV and the B-side air defense facility using the formula:

[0031]

[0032] Use H j (1≤j≤C) represents the direction difference between the unmanned combat aircraft of party A and the j-th air defense facility of party B, and C represents the number of air defense facilities of party B.

[0033] Step c3. If d i >r red , represents the distance between the ith B-party facility and the A-party unmanned combat aircraft (d i ) is greater than the detection range of the A-side UCAV (r red ), that is, the i-th B-party facility is not within the detection range of the A-party unmanned combat aircraft, let d i = 0, and if the B facility is the B air defense facility, let H i =0.

[0034] Step c4. Based on the results of steps c1 to c3, the information o at a certain time t t =(head red ,d0,d1,…,d C , H1, H2,…, H C), including the direction of Party A's unmanned fighter, the distance between Party A's unmanned fighter and Party B's different facilities (including bases and air defense facilities), and the direction difference between Party A's unmanned fighter and Party B's different air defense facilities.

[0035] Step c5. According to the result of step c4, the status information s at time t t =[o t-2 ,o t-1 ,o t ], if t-2 or t-1 is less than 0, that is, t-2 or t-1 has no corresponding actual time, then o t-2 and t-1 Fill with 0.

[0036] In step 3, the state information s with time series information is converted into feature information e through a long short-term memory network (LSTM). Taking time t as an example, s t is the state information at time t, and its conversion process includes the following steps:

[0037] Step d1. Calculate the forget gate:

[0038] f t =σ(W f ·[h t-1 ,s t ]+b f ),

[0039] Step d2. Calculate the input gate:

[0040] i t =σ(W i ·[h t-1 ,s t ]+b i ),

[0041]

[0042]

[0043] Step d3. Calculate the output gate:

[0044] o t =σ(W o ·[h t-1 ,s t ]+b o ),

[0045] h t =o t *tanh(C t )

[0046] h t That is, the state information s at time tt The transformed feature information t ; where σ is the sigmoid function, tanh is the tanh activation function, · represents the Hadamard product, [] represents the concatenation of vectors, * represents matrix multiplication, and W f , W i , W C , b f , b i , b C , b o are the learnable parameters of the model.

[0047] In the step 4, DDPG (Deep Deterministic Policy Gradient) is used as a reinforcement learning algorithm. DDPG combines deep neural networks and deterministic policy gradient methods. It is an algorithm of the Actor-Critic framework. The Actor network (policy network) accepts the current state as input and outputs the action value under the current state; the Critic network (value network) accepts the current state and the action output by the Actor network as input, and outputs the Q value, which is used to evaluate the quality of the action value output by the Actor network; the Actor network uses gradient ascent to update parameters and maximize the value function; the Critic network uses gradient descent to update parameters and fit the value function; the interaction process between DDPG and the training scene includes the following steps:

[0048] Step e1. Through the Actor strategy network, the next action a is predicted based on the feature information e obtained in step 3. Action a is that the A-side unmanned fighter jet moves towards the head red , calculate the direction value of the A-side unmanned fighter at the next moment:

[0049] head red,t+1 =(head red,t +a+180)%180

[0050] Where a represents the action value taken, and % represents the remainder operation;

[0051] Step e2. The simulation platform simulates the process of A's unmanned fighter flying according to the new heading value, and obtains the state s at the next time t+1 t+1 ' and reward r t+1 ;

[0052] In step 4, the trajectory data includes the state transition process and reward information, and the trajectory data can be expressed as:

[0053] P={[s1,a1,r1,done1,s′2],[s′2,a2,r2,done2,s′3],…,[s′N ,a N ,r N ,done N ,s′ N+1 ]}

[0054] Where N represents the length of the trajectory data; taking time t as an example, the state transition process at time t is [s t ,a t ,r t ,done t ,s' t+1 ], where s t represents the state at time t, a t represents the action value taken at time t, r t Indicates the reward obtained at time t, done t Indicates whether the simulation process ends at time t, s' t+1 Indicates the state at the next moment t+1;

[0055] The reward information includes three parts: the distance between the A-side UCAV and the B-side base, and the reward r dis , the penalty reward for A's unmanned fighter being shot down by B's air defense system is r lose , the winning reward for A's unmanned fighter attacking B's base is r win , the complete reward information r of the current time step is as follows:

[0056] r=α1r dis +α2r lose +α3r win

[0057] Among them, α1, α2, α3 are weight coefficients;

[0058] When the A-side UAV fighter is destroyed by the B-side air defense system or the A-side UAV fighter destroys the B-side base, done t = True, the simulation ends; otherwise done t =Fasle, simulation continues.

[0059] In step 5, multi-scenario generalization training is performed, which means starting from the T in the training phase. a 、T b and T cIn the n+m+k scenes, a scene is randomly selected for training each time, and the training trajectory data is saved in the experience replay pool. Each trajectory data has the same sampling weight; the experience replay pool stores trajectory data P under multiple categories of scenes, which comes from the training trajectory data under multiple types of training scenes designed; when updating network parameters, trajectory data is randomly sampled from the experience replay pool, and the sampled trajectory data comes from different training scenes. The specific training process includes:

[0060] Step f1. Randomly select a scene from the scenes of the training phase generated in step 1 as the current simulation scene;

[0061] Step f2. For each simulation scenario, execute the process of step 4 above (e1, e2);

[0062] Step f3. Randomly select a batch of samples from the experience replay pool containing multiple scene trajectory data for training;

[0063] Step f4. Calculate TD-error: Use the current policy network (actor) and value network (critic) to calculate TD-error, which is the error between the target value and the current estimated value. TD-error is calculated as:

[0064] TD_error=r+γQ′(s′,μ′(s′))-Q(s,a)

[0065] Among them, r is the reward of the current time step, γ is the discount factor, Q' is the target value network, μ' is the target policy, Q is the current estimated value, s' is the state information at the next moment, s is the state information at the current moment, and a is the value of the action currently taken.

[0066] Step f5. Update the parameters of the Critic network: Use TD-error to update the parameters of the value network. The loss function of the value network is: Where y = r + γQ'(s' + μ'(s')) is the target value. Use gradient descent to update the parameters: Fitting value function, where θ Q are the parameters of the value network and α is the learning rate.

[0067] Step f6. Update the actor network parameters: Use the gradient of the current policy network to update the parameters so that the policy can better approach the optimal policy. The loss function of the policy is: L = -Q(s, μ(s)), where s is the current state information and μ(s) is the action generated by the Actor network; use the gradient ascent method to update the parameters: Maximize the value function, where θμ is the parameter of the policy network and β is the learning rate.

[0068] Step f7. Update the target network: Regularly use the current network parameters to update the target network parameters to maintain stability. The parameter update method of the target network is: θ'←τθ+(1-τ)θ', where θ' is the parameter of the target network, θ is the parameter of the current network, and τ is a constant less than 1, which is used to adjust the weights of the two parameters.

[0069] Step f8. Repeat f1 to f7 until the algorithm converges or the maximum number of training times is reached.

[0070] The present invention also provides application of the above method in multi-scenario generalization training of UAVs breaking through air defense and attacking targets.

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] (1) Compared with traditional algorithms, deep reinforcement learning algorithms can adaptively adjust strategies to adapt to different combat scenarios in air combat decision-making problems, and make them more interpretable through the reward value of each action recorded in the trajectory data during execution;

[0073] (2) Using a deep deterministic policy gradient algorithm and experience replay pool can improve the speed of model convergence and reduce the risk of overfitting;

[0074] (3) Training the reinforcement learning algorithm in a single scenario can make the unmanned fighter controlled by the reinforcement algorithm perform well in the scenario, but the generalization ability is poor. When facing unknown environments and tasks, the unmanned fighter may perform worse. Compared with training the reinforcement learning algorithm in a single scenario, training the unmanned fighter controlled by the reinforcement learning algorithm in multiple training scenarios of various types to learn to break through the air defense system of Party B and attack Party B's targets can make the unmanned fighter have better generalization ability in real combat scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 It is a principle diagram of the training method of the present invention.

[0077] Figure 2 It is a scene diagram that fixes the location of Party B's facilities and modifies the location of Party A's unmanned combat aircraft.

[0078] Figure 3It is a scenario diagram that fixes the positions of Party A's unmanned fighter jets and Party B's target points and modifies the deployment of Party B's air defense facilities.

[0079] Figure 4 It is a scene diagram that fixes the positions of party A's unmanned fighter jets and party B's air defense facilities and modifies the position of party B's target points.

[0080] Figure 5 It is a training flow chart of the present invention.

[0081] Figure 6 This is a comparison chart of the training time and winning rate of algorithms trained in multiple scenarios and single scenario;

[0082] In the picture, 1. Party B’s air defense system facilities, 2. Party B’s target point location, 3. Party A’s unmanned fighter.

[0083] Figure 2 , Figure 3 , Figure 4 Only some training or test scene diagrams are shown. The actual training and test scenes include but are not limited to the scenes shown. Figure 6 The horizontal axis (Step) represents the number of simulations for training, and the vertical axis (Winning percentage@5) represents the average winning percentage of the last five simulations. DETAILED DESCRIPTION

[0084] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0085] The present invention discloses a multi-scenario generalization training method for a drone to break through air defense and attack a target. By training an unmanned fighter controlled by a deep reinforcement learning algorithm to interact with the training scene in multiple types of training scenes, the unmanned fighter can still break through the air defense system of party B and attack the target point of party B when facing the complex and changing air defense system deployment of party B in a real scene. The present invention comprises the following steps: (1) designing multiple types of training scenes; (2) obtaining key information of party A's unmanned fighter, party B's air defense system and party B's target in the training scene; (3) extracting state information from the key information of step (2); converting the state information with time series information into feature information through a long short-term memory network (LSTM); (4) generating training data by interacting with the training scene through the unmanned fighter controlled by the reinforcement learning algorithm; (5) performing multi-scenario generalization training, and performing alternating training and model parameter updating in multiple types of multiple training scenes. The method of the present invention improves the generalization ability of the deep reinforcement learning algorithm to control the unmanned combat aircraft in real scenarios, so that the unmanned combat aircraft controlled by the reinforcement learning algorithm can break through the air defense system of party B and attack the targets of party B when facing the complex and changeable air defense system deployment of party B.

[0086] Specifically, the present invention provides a multi-scenario generalized training method for a drone to break through air defense and attack a target, the method comprising the following steps:

[0087] Step 1: Design multiple UCAV training scenarios of various types;

[0088] Step 2: Obtain key information about Party A’s fighter jets, Party B’s air defense system, and Party B’s targets in the training scenario;

[0089] Step 3: Extract state information s from the key information in step 2; convert the state information s with time series information into feature information e through a long short-term memory network (LSTM);

[0090] Step 4: The unmanned fighter controlled by the reinforcement learning algorithm interacts with the training scene, plans the direction of the unmanned fighter, saves the trajectory data including the state transition process and reward information, and learns the strategy of the unmanned fighter to break through the air defense system of party B and attack the target of party B;

[0091] Step 5: Conduct multi-scenario generalization training, perform alternating training in multiple training scenarios of different types, store the trajectory data trained in multiple scenarios in the experience buffer pool, update the model parameters through the trajectory data in multiple scenarios, and improve the generalization ability of the reinforcement learning algorithm, so that the unmanned fighter controlled by the reinforcement learning algorithm can break through the complex and changing air defense system deployment of Party B and attack Party B's targets.

[0092] More specifically, taking the simulation training on the Mozi platform as an example, first, design multiple unmanned combat aircraft training and test scenarios of different types, such as Figure 2 , Figure 3 and Figure 4 As shown:

[0093] a1. Assume that the two parties are Party A and Party B;

[0094] a2. Design the initial scenario t0: Party A initializes an unmanned fighter with attack missiles, and Party B is set to N (N≥1) air defense facilities (in the Mozi platform, it can be set to an air defense missile platoon) and a Party B base (Party A's target point);

[0095] a3. Design various types of training or testing scenarios:

[0096] (1) The first type T a ,like Figure 2 As shown in the figure: Based on t0, the positions of the air defense system facilities and the target points of party B are fixed, and the positions of the unmanned fighters of party A are randomly modified to generate n (n≥3) training scenarios, T a ={t a1 , t a2 ,...,t an}; Generate n2 (n2 ≥ 1) test scenarios,

[0097] (2) The second type T b ,like Figure 3 As shown in the figure: Based on t0, the positions of the A-side UCAV and the B-side base are fixed, and the initial positions of multiple air defense facilities of the B-side are changed to generate m (m ≥ 3) training scenarios, T b ={t b1 , t b2 ,...,t bm}; Generate m2 (m2 ≥ 1) test scenarios,

[0098] (3) The third type T c ,like Figure 4 As shown in the figure: Based on t0, the positions of party A's unmanned fighter and party B's air defense facilities are fixed, and the position of party B's base changes randomly, generating k (k ≥ 3) training scenarios, T c ={t c1 , t c2 ,...,t ck}; Generate k2 (k2 ≥ 1) test scenarios,

[0099] a4. Save the generated n+m+k scenes as training scenes; and use these n+m+k scenes as test scenes;

[0100] a5. Save the generated n2+m2+k2 scenes as test scenes, that is, n+m+k+n2+m2+k2 scenes are shared in the test phase.

[0101] Then, the open source Mozi AI development kit communicates with the simulated scene of the Mozi platform, and filters out key information from the obtained scene information:

[0102] b1. Filter out the longitude information lon of Party A’s unmanned fighter from the scene information provided by the Mozi communication component red , latitude information lat red , head towards information red ;

[0103] b2. During the training process, if the B party facility (including the B party air defense facility or the B party base) enters the reconnaissance range of the A party unmanned fighter, the key information of the B party is filtered out from the B party information provided by the Mozi communication component. If the B party facility is an air defense missile platoon, the longitude information of the air defense missile platoon is filtered out. Latitude information And direction information If the B facility is the B base, filter out the longitude information of the B base and latitude information

[0104] The key information obtained from the communication component is then converted into status information as follows:

[0105] c1. Calculate the distance between the A-side UCAV and the B-side facility using the formula:

[0106]

[0107] Where r represents the radius of the earth, r≈6371km, arcsin is the inverse sine function, sin is the sine function, and cos is the cosine function;

[0108] Use d i (0≤i≤C) represents the distance between the A-side unmanned fighter and the i-th B-side facility; i=0 represents the B-side base; 1≤i≤C represents the i-th B-side air defense facility, and C is the number of B-side air defense facilities.

[0109] c2. Calculate the difference in direction between the A-side UCAV and the B-side air defense facility using the formula:

[0110] H j =head red -headblue

[0111] Use H j (1≤j≤C) represents the direction difference between the unmanned combat aircraft of party A and the j-th air defense missile platoon of party B, and C represents the number of air defense missile platoons of party B.

[0112] c3. If d i >r red , that is, the i-th B-party facility is not within the detection range of A, let d i = 0, and if the i-th facility is the air defense missile platoon of party B, let H i =0.

[0113] c4. According to the results of (c1, c2, c3), the information o at a certain time t t =(head red ,d0,d1,…,d C ,H1,H2,…,H C ).

[0114] c5. According to the result of (c4), the state information s at time t t =[o t-2 ,o t-1 ,o t ].

[0115] The specific training steps are as follows: Figure 1 As shown in the figure, unlike the traditional training method, during the training process, it is necessary to randomly load training scenes from multiple scenes of multiple categories, and extract samples from the experience replay pool that stores the trajectory data of multiple categories of training scenes for parameter update. The specific steps are as follows:

[0116] d1. Randomly select a scene from the multiple scenes generated in step 1 as the current simulation scene;

[0117] d2. For each simulation scenario, perform the following steps:

[0118] 1) Obtain data from the training scene and process the obtained data into state information s according to step 3;

[0119] 2) Convert the state information s with time series information into feature information e through the long short-term memory network (LSTM);

[0120] 3) Through the DDPG strategy network, based on the feature information e, the action a to be taken in the next step is predicted. Action a is that the A-side unmanned fighter jet moves towards the head red The change value of the next moment is the head red,t+1 =(head red,t +a+180)%180;

[0121] 4). The simulation platform simulates the process of A's unmanned fighter flying according to the new direction value and obtains the state s at the next moment. t+1 ' and reward r t+1 , the reward r is determined according to the situation during the simulation:

[0122] (1) During the simulation, whether the A-side UCAV is shot down or not, if it is shot down, the penalty reward is r lose =0, otherwise r lose = -1;

[0123] (2) During the simulation, whether the A-side UCAV attacks the B-side target or not, if it does, the winning reward is r win =1, otherwise the winning reward is r win =0;

[0124] (3) During the simulation, the distance between the A-side UCAV and the B-side target is dis. The closer the distance, the greater the distance. dis The bigger;

[0125] (4) r = α1r dis +α2r lose +α3r win , where α1, α2, α3 are weight coefficients;

[0126] 5). Store the state transition (s,a,r,done,s') in the experience replay pool for subsequent training.

[0127] d3. Randomly select a batch of samples from the experience replay pool containing multiple scene trajectory data for training;

[0128] d4. Calculate TD-error: Use the current policy network (actor) and value network (critic) to calculate TD-error, which is the error between the target value and the current estimated value. TD-error is calculated as follows:

[0129] TD e rror=r+γQ′(s+μ′(s′))-Q(s,a)

[0130] Where r is the reward at the current time step, γ is the discount factor, Q' is the target value network, μ' is the target policy, and Q is the current estimated value.

[0131] d5. Update the parameters of the Critic network: Use TD-error to update the parameters of the value network. The loss function of the value network is: Where y = r + γQ'(s' + μ'(s')) is the target value. Use gradient descent to update the parameters: where θ Q are the parameters of the value network and α is the learning rate.

[0132] d6. Update actor network parameters: Use the gradient of the current policy network to update the parameters so that the policy can better approach the optimal policy. The loss function of the policy is: L = -Q(s, μ(s)), where s is the current state information and μ(s) refers to the action generated by the Actor network; use the gradient ascent method to update the parameters: L, where θμ is the parameter of the policy network and β is the learning rate.

[0133] d7. Update the target network: Regularly use the current network parameters to update the target network parameters to maintain stability. The target network parameters are updated as follows: θ'←τθ+(1-τ)θ', where θ' is the parameter of the target network, θ is the parameter of the current network, and τ is a constant less than 1.

[0134] d8. Repeat d1 to d7 until the algorithm converges or the maximum number of training times is reached.

[0135] After the training phase, the designed training scenario T a 、T b 、T c And test scenarios The scenes are selected in turn as test scenes. The test process is similar to the above training process, but the parameter update process is not performed.

[0136] Comparative Example

[0137] In conjunction with the instruction manual Figure 6 , in a single scenario, the reinforcement learning algorithm is trained to learn the strategy of controlling the unmanned fighter to break through the air defense of party B and attack the target of party B. Compared with the present invention, there are the following differences:

[0138] First, the total time consumed by the training of the present invention is relatively low. If there are n scenes for training in a single scenario, assuming that the training difficulty is the same, and a single scenario requires x time, then these n scenarios require n*x time to complete. However, the present invention can have a certain adaptability to the i-th scenario by training the strategies learned in the first i scenes, thus reducing the time for learning in the i-th scenario, and the time required to complete the training is less than n*x. Figure 6 Among the four scenarios shown, Single i means training in a single scenario, and Multii means training in multiple scenarios. It can be seen that in multi-scenario training, after 20 to 30 steps, it can win in five consecutive simulations, while in single-scenario training, it takes 80 to 100 steps to win in five consecutive simulations.

[0139] Secondly, since the space explored by the reinforcement learning algorithm in the present invention is large enough, the generalization performance in real scenarios is good; when training in a single scenario, there is a problem that the space explored by the reinforcement learning method is not large enough, and in real situations, the deployment of Party B's air defense system and the location of Party B's base are prone to change, which will lead to large differences between real scenarios and unmanned fighter training scenarios, and thus lead to the reinforcement learning algorithm trained in a single scenario being unable to make reasonable decisions in real situations; and the present invention takes into account that in real situations, Party A's unmanned fighter may find that the initial position of Party B's air defense system or base may be different, the position of Party B's base relative to Party B's air defense system may be different, or the layout of Party B's multiple air defense facilities may be different, and three different scenarios are designed. Multiple scenarios are randomly generated in each scenario, and mixed training is performed in multiple scenarios of multiple categories. Moreover, the test scenario also includes simulation scenarios that are different from the training scenarios, which can better verify the generalization ability of the unmanned fighter controlled by the reinforcement learning algorithm in real scenarios. Figure 6 In the training situation shown, the Single 1 curve indicates that the agent needs more than 100 iterations to win in 5 consecutive simulations in scenario 1, while the Multi 1 curve indicates that the agent only needs 20 iterations to win in 5 simulations in scenario 1. This shows that the experience learned by the agent in scenario 0, scenario 2, and scenario 3 can be generalized to scenario 1.

[0140] Finally, in the testing phase, the scene in the training phase is used as the testing scene, that is, [T a 、T b 、T c ] as the test scenario, the present invention and the algorithm trained in a single scenario can complete the task of breaking through the air defense of party B and attacking the target of party B; but the new scenario that has not appeared in the training phase is used as the test scenario, that is, the algorithm selected The scenarios in the test scenarios were used, and the winning rate of the present invention in the test scenarios exceeded that of the algorithm trained in a single scenario. 30 scenarios were randomly selected for testing, and the present invention was able to complete the task in 26 scenarios. The algorithm trained in a single scenario could only complete the task in 5 new scenarios.

[0141] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A multi-scenario generalized training method for a drone to break through air defense and attack a target, characterized in that: The method comprises the following steps: Step 1: Design multiple UCAV training scenarios of various types; Step 2: Obtain key information about Party A’s fighter jets, Party B’s air defense system, and Party B’s targets in the training scenario; Step 3: Extract state information s from the key information in step 2; convert the state information s with time series information into feature information e through the long short-term memory network LSTM; Step 4: The unmanned fighter controlled by the reinforcement learning algorithm interacts with the training scene, plans the direction of the unmanned fighter, saves the trajectory data including the state transition process and reward information, and learns the strategy of the unmanned fighter to break through the air defense system of party B and attack the target of party B; Step 5: Conduct multi-scenario generalization training, perform alternating training in multiple training scenarios of different types, store the trajectory data trained in multiple scenarios in the experience buffer pool, update the model parameters through the trajectory data in multiple scenarios, and improve the generalization ability of the reinforcement learning algorithm, so that the unmanned fighter controlled by the reinforcement learning algorithm can break through the complex and changing air defense system deployment of Party B and attack Party B's targets.

2. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 1 is characterized in that: In step 1, by adjusting the initial position of the fighter jet of party A or the positions of multiple air defense facilities constituting the air defense system of party B or the position of the target of party B, various types of training scenarios and test scenarios can be designed; different types of training scenarios or test scenarios can be further divided into: a)T a :The positions of multiple air defense facilities and targets of party B are fixed, and the positions of fighters of party A are randomly changed. n training scenarios are generated. a ={t a1 , t a2 , ..., t an Generate n2 test scenarios, Among them, n≥3, n2≥1; b)T b :The positions of the fighter jets of party A and the targets of party B are fixed, the initial positions of multiple air defense facilities of party B are changed, and m training scenarios are generated. b ={t b1 , t b2 , ..., t bm Generate m2 test scenarios, Among them, m≥3, m2≥1; c)T c :The positions of A's fighters and B's multiple air defense facilities are fixed, and the position of B's ​​base changes randomly. Generate k training scenarios, T c ={t c1 , t c2 , ..., t ck Generate k2 test scenarios, Among them, k≥3, k2≥1.

3. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 1 is characterized in that: In step 2, the key information K includes the longitude value lon of the fighter jet of party A. red , latitude value lat red , towards the value head red ; And the longitude values ​​of multiple air defense facilities of side B Latitude value And the orientation value And the longitude value of Party B's target and latitude values Right now Where C represents the number of air defense facilities that constitute the air defense system of Party B.

4. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 1 is characterized in that: In step 3, extracting state information from the key information in step 2 and then converting the state information with time series information into feature information through the long short-term memory network LSTM means processing the key information of the fighter jet of party A, the air defense system of party B, and the target of party B as the state information of the reinforcement learning algorithm. The specific operation is as follows: a) Calculate the distance between the fighter jet of party A and the facilities of party B by the formula: Where r represents the radius of the earth, arcsin is the inverse sine function, sin is the sine function, and cos is the cosine function; Use d i represents the distance between the A fighter and the i-th B facility, 0≤i≤C; b) Calculate the difference in direction between the fighter jets of party A and the air defense facilities of party B, using the formula: Use H j represents the direction difference between the A fighter and the j-th B air defense facility, 1≤j≤C; c) If d i >r red , indicating that the distance between the i-th B-party facility and the A-party unmanned fighter is greater than the detection range of the A-party unmanned fighter, that is, the i-th B-party facility is not within the detection range of the A-party unmanned fighter. Let d i = 0, and if the B facility is an air defense facility, let H i =0; d) Based on the results of a), b), and c), get the information o at a certain time t t =(head red , d0, d1, ..., d C , H1, H2, ..., H C ), including the direction of the A-party unmanned fighter, the distance between the A-party unmanned fighter and the different facilities of the B-party, and the direction difference between the A-party unmanned fighter and the different air defense facilities of the B-party; e) According to the result of d), the state information s at time t t =[o t-2 , o t-1 , o t ], if t-2 or t-1 is less than 0, that is, t-2 or t-1 has no corresponding actual time, then o t-2 and t-1 Fill with 0; f) The state information s with time series information is converted into feature information e through the long short-term memory network LSTM.

5. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 4 is characterized in that: The state information s at time t t Convert to feature information t The process includes the following steps: Step d1. Calculate the forget gate: f t =σ(W f ·[h t-1 ,s t ]+b f ), Step d2. Calculate the input gate: I t =σ(W i ·[h t-1 ,s t ]+b i ), Step d3. Calculate the output gate: the t =σ(W o ·[h t-1 ,s t ]+b o ), h t =o t *tanh(C t ) h t That is, the state information s at time t t The transformed feature information t ; where σ is the sigmoid function, tanh is the tanh activation function, · represents the Hadamard product, [] represents the concatenation of vectors, * represents matrix multiplication, and W f , W i , W C , b f , b i , b C , b o are the learnable parameters of the model.

6. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 1 is characterized in that: Use the deep deterministic policy gradient algorithm DDPG to train unmanned fighter jets to break through the air defense system of party B and attack party B's targets; the deep deterministic policy gradient algorithm combines deep neural networks and deterministic policy gradient methods. It is an algorithm of the Actor-Critic framework. The Actor network uses gradient ascent to update parameters and maximize the value function, and the Critic network uses gradient descent to update parameters and fit the value function. The loss function for Actor network updates is: L = -Q(s, μ(s)), where s is the state and μ(s) refers to the action generated by the Actor network; The Actor network uses gradient ascent to update parameters: L, where θμ is the parameter of the policy network and β is the learning rate; The loss function of the Critic network update is: Where y = r + γQ′(s′ + μ′(s′)) is the target value; The Critic network uses the gradient descent method to update parameters: L, where θ Q are the parameters of the value network and α is the learning rate.

7. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 6 is characterized in that: The interaction process between DDPG and the training scenario includes the following steps: Step e1. Through the Actor strategy network, the next action a is predicted based on the feature information e obtained in step 3. Action a is that the A-side unmanned fighter jet moves towards the head red , calculate the direction value of the A-side unmanned fighter at the next moment: head red，t+1 =(head red，t +a+180)%180, Where a represents the action value taken, and % represents the remainder operation; Step e2. The simulation platform simulates the process of A's unmanned fighter flying according to the new heading value, and obtains the state s at the next time t+1 t+1 ′ and reward r t+1 .

8. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 1 is characterized in that: In step 4, the generated trajectory data comes from T a , T b , T c Multiple scenes,trajectory data under three types of scenarios: P={[s1,a1,r1,done1,s'2],[s'2,a2,r2,done2,s'3],...,[s′ N ,a N ,r N ,done N ,s′ N+1 ]}, Where N represents the length of the trajectory data; the state transition process at time t is [s t , a t , r t , done t , s′ t+1 ], where s t represents the state at time t, a t represents the action value taken at time t, r t Indicates the reward obtained at time t, done t Indicates whether the simulation process ends at time t, s′ t+1 Indicates the state at the next moment t+1; In step 4, the reward value consists of three parts: the distance reward r between the fighter and the target of side B dis , the penalty reward for a fighter jet being shot down by the air defense system of side B is r lose , the winning reward for the fighter attacking the target of side B is r win ; The complete current time step reward function is as follows: r＝α1r dis +α2r lose +α3r win ， Among them, α1, α2, α3 are weight coefficients; When the A-side UAV fighter is destroyed by the B-side air defense system or the A-side UAV fighter destroys the B-side base, done t = True, the simulation ends; otherwise done t =Fasle, simulation continues.

9. The multi-scenario generalized training method for a drone to break through air defense and attack a target according to claim 1, characterized in that: In step 5, multi-scenario generalization training is performed, which means starting from T a , T b and T c Among the n+m+k scenes, one scene is randomly selected for training each time, and the training trajectory data is saved in the experience replay pool, and each trajectory data has the same weight; the experience replay pool stores trajectory data P under multiple types of scenes, and the trajectory data P comes from the training trajectory data under multiple types of training scenes designed; When updating network parameters, data is randomly sampled from the experience replay pool, and the sampled data comes from different training scenarios.

10. Application of the method according to any one of claims 1 to 9 in multi-scenario generalized training of UAVs breaking through air defense and attacking targets.

Images