Unmanned fighter avoidance decision-making method based on data and knowledge driving and intelligent agent

Abstract

The invention provides an unmanned fighter avoidance decision-making method based on data and knowledge driving and an intelligent agent, which simulate a way from human understanding rules to rule utilization, and is based on a strategy updating mode of knowledge and data hybrid driving. Sample mixing of an offline database and an online interaction chain and strategy mixing of a knowledge decision base and an evasion agent jointly drive strategy updating, rule knowledge is extracted from existing historical data and decision experience, and an unmanned aerial vehicle is guided to continuously interact and feed back with a multi-stage air combat task environment. And an accurate and efficient avoidance scheme is generated.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent decision-making, and in particular relates to an unmanned aircraft avoidance decision-making method and an intelligent agent based on data and knowledge driving. Background Art

[0002] Unmanned Combat Aerial Vehicle (UCAV) has played a great role in local combat in recent years due to its strong survivability, low cost-effectiveness, and flexible configuration. It has been widely used in the military field. Autonomous air combat decision-making is one of the key technologies for winning air combat, and reasonable avoidance of incoming targets is the primary condition for air combat decision-making. With the development of information technology, modern air combat missions have shown multi-level and multi-stage characteristics. Facing dynamic and changeable opponent strategies and complex strong confrontation environments, how to improve the survivability is a key problem that high-value unmanned combat aircraft need to solve urgently to complete air combat missions. In modern warfare, missiles have the characteristics of high speed, large radius collision area, and unfixed routes. They have great destructive power on aircraft and become the biggest threat to unmanned combat aircraft. Therefore, the literature 1 "Wang Panpan, Chen Mou, Wu Qingxian, et al. Unmanned combat aircraft threat avoidance maneuver decision-making method based on experience intuition [J]. Journal of Nanjing University of Aeronautics and Astronautics, 2022, 54(01): 77-85. DOI: 10. 16356 / j.1005-2615.2022.01.009.” It is proposed that the generation of accurate missile avoidance decision-making scheme is crucial to improving the air combat survivability of unmanned combat aircraft.

[0003] Unmanned aircraft avoidance decision-making has always been a hot topic in air combat research. Currently, it is mainly concentrated in three aspects: (1) Classical control theory methods. For example, Reference 2 "Carr, Ryan, W, et al. Solution of a Pursuit-Evasion Game Using a Near-Optimal Strategy[J]. Journal of Guidance, Control, and Dynamics: A Publication of the American Institute of Aeronautics andAstronautics Devoted to the Technology of Dynamics and Control, 2018." proposed using unilateral optimal control method and nonlinear programming method to solve missile avoidance problem and generate maneuvering scheme. Reference 3 "Li Fei, Yu Lei, Zhou Zhongliang, Fu Zhaowang, & Zhang Tao. (2014). Nonlinear model predictive control avoidance strategy for fighter terminal maneuvers. Journal of National University of Defense Technology, 36(3), 8." proposed a nonlinear model predictive control idea. After establishing the optimal control model, the Gaussian pseudo-spectral method was used to solve the maneuvering strategy of the aircraft to avoid missiles at the terminal. Reference 4 "Li Peiyu, Yang Zhen, & Luo Yazhong. (2023). Collision avoidance maneuver planning method based on nonlinear relative motion equations. Manned Spaceflight, 29(5), 581-589. “An analytical collision avoidance maneuver optimization model based on nonlinear relative motion equations is proposed to realize the planning of maneuver schemes. Reference 5 “Guo Qiang, He Shengjie, Cheng Jialin, Wang Xinghu, Sun Gen, & Guo Fei. (2022). A threat assessment method for unmanned combat aircraft to autonomously avoid missiles. Aviation Science and Technology, 33(7), 7” proposes an artificial potential field method to evaluate the threat of missiles by analyzing the potential field angle and distance, and control the unmanned combat aircraft in real time to complete autonomous avoidance of missiles through the optimal trajectory. These avoidance methods rely on clear and complete mathematical models to solve the optimal maneuver strategy; (2) Intelligent optimization algorithms, for example, Reference 6 “Riihim·a·ki V. Using Genetic Algorithms toGenerate an Initial solution to the Aircraft-Missile–Problem[J].Student Research Projects, Systems Analysis Laboratory, Helsinki University of Technology, 2001, 25 (3): 95-98” proposed using genetic algorithms to generate initial solutions to the aircraft missile problem. Reference 7 “Yang Liu, Luo Jixun, Hu Zhaohui. Fighter maneuver avoidance optimization method based on θ-QPSO [J]. Firepower and Command Control, 2014. DOI: CNKI: SUN: HLYZ.0.2014-10-014” proposed using phase angle encoded quantum particle swarm optimization algorithm to optimize the parameters of the nonlinear programming problem of maneuver avoidance. Although the maneuver strategy can be effectively generated, the optimization algorithm is limited by large computational complexity and few optimization parameters, making it difficult to be widely used; (3) Reinforcement learning algorithm. Reinforcement learning has made significant progress in decision-making problems in recent years, such as reference 8 “Wang Z, Yan H, Wei C, et al. Research on autonomous driving decision-making strategies based deep reinforcement learning [J]. arxiv preprint arxiv:2408.03084, 2024", Document 9"Meng Q, Hussain S, Luo F, et al. An online reinforcement learning-based energy management strategy for microgrids with centralized control[J]. IEEE Transactions onIndustry Applications, 2024", Document 10"SHUFORD, Jeff. Deep learning unleashing the power of AI in decision-making. Journal of ArtificialIntelligence General science (JAIGS) ISSN: 3006-4023, 2024, 1.1.", Document 11 "YUAN, Zhecheng, et al. Rl-vigen: A reinforcement learning benchmark for visual generalization.Advances in Neural Information Processing Systems, 2024, 36". Inspired by this, more and more scholars have applied it to aircraft maneuvering decisions, enabling aircraft to autonomously learn and optimize their maneuvering strategies to cope with complex air combat environments. For example, reference 12 "Song Hongchuan, Zhan Hao, Xia Lu, et al. Research on fighter aircraft avoidance of medium-range air-to-air missiles based on deep deterministic policy gradient algorithm [J]. Advances in Aeronautical Engineering, 2021, 12(3): 85-94." proposed an interactive training based on the DDPG algorithm through the intelligent agent and the missile aircraft pursuit model, which can effectively reduce the dependence of the missile avoidance problem on prior knowledge while generating an end-to-end missile avoidance maneuver strategy from state parameters to control quantities. Reference 13 "Fan Xinlei, Li Dong, Zhang Wei, et al. Research on missile avoidance decision training based on deep reinforcement learning [J]. Electro-Optics and Control, 2021, 28(1): 81-85” proposed an improved DDPG algorithm to autonomously generate maneuvering plans for the carrier aircraft, effectively realizing the evasion decision against the incoming missile. Reference 14 “Lee GT, Kim C O. Autonomous control of combat unmanned aerial vehicles to evade surface-to-air missiles using deep reinforcement learning[J]. IEEE Access, 2020, 8: 226724-226736” proposed a reinforcement learning method based on amplified imitation effect (AIE) to effectively generate maneuvering strategies that can simultaneously complete guided missile avoidance, shortest path flight and formation flight. Reference 15 “Chen K, Lei J, Li B. The research on evasion strategy of unpowered aircraft based on deep reinforcement learning[C] / / Journal of Physics: Conference Series. IOP Publishing, 2022, 2252(1): 012072” proposed a Monte Carlo reinforcement learning mechanism for DQN The algorithm was improved to improve sampling efficiency and achieve end-to-end avoidance strategy learning; Reference 16 "MM Özbek and E.Koyuncu, Missile Evasion Maneuver Generation with Model-free Deep Reinforcement Learning, 2023 10th International Conference on Recent Advances in Air and Space Technologies (RAST), Istanbul, Turkiye, 2023, 1-6, doi: 10.1109 / RAST57548.2023.10197852” proposed a real-time generation of the best missile evasion maneuvering decision for fighter jets in complex combat environments based on double-delayed deep deterministic policy gradient. These methods mostly study the optimal maneuvering strategy in the case of a single air-to-air missile strike, and the fighter mission is fixed. When the number and type of missiles change, or the mission presents multi-stage divergence, the model's maneuvering scheme is difficult to achieve effective missile evasion while completing the mission.

[0004] Traditional air combat avoidance problems only consider avoiding incoming missiles, and most of them are single-target strikes. It is difficult to cope with the modern complex battlefield environment where the battlefield situation changes rapidly and the opponent's strategy is dynamic and changeable. Moreover, in real confrontation scenarios, unmanned fighters usually have different combat missions in each confrontation stage, and the incoming targets they face are usually multiple missiles of different types. These greatly increase the difficulty of solving maneuver strategies. How to improve the survival rate of fighters while ensuring the completion of combat missions is crucial. Summary of the invention

[0005] The main problem solved by the present invention is how to improve the survival rate of fighter jets while ensuring the completion of combat missions, and enhance the efficiency of unmanned fighter jets in maneuvering and evading missiles. A data- and knowledge-driven unmanned fighter jet avoidance decision-making method and intelligent agent are provided.

[0006] In order to solve the above technical problems, the technical solution adopted is: A data- and knowledge-driven UAV avoidance decision-making method includes the following steps: Step 1: Obtain the flight status and target unit information of the unmanned aircraft; Step 2: Formalize the sequential decision-making process of the UAV air combat mission based on the flight status of the UAV; Step 3: Construct an avoidance decision model based on the interactive self-attention mechanism; Step 4: Use a data- and knowledge-based hybrid-driven strategy update method to train the avoidance decision model to obtain an avoidance decision agent; Step 5: Input the flight status of the UAV and the target unit information into the avoidance decision agent, and generate a maneuver avoidance plan in real time during the sequential decision-making process of the air combat mission.

[0007] Furthermore, the sequential decision-making process of the unmanned aircraft air combat mission is formally modeled based on the flight state of the unmanned aircraft: Modeling of sequential decision making based on the Markov decision process, including five elements ; in, is the state space, which is a set of enemy and friendly situation information at each time point in a confrontation round, where yes The moment state code, is the total number of time steps in this adversarial round; is the action space, which is the set of all possible discrete actions, where m is the total number of actions; is the state transfer function: it represents the evolution of the combat situation. Given the current state , the agent takes action After that, the combat scenario evolves to the next state ,Right now ; is the reward function, is the discount factor.

[0008] Furthermore, building an avoidance decision model based on the interactive self-attention mechanism means: In the AC algorithm framework, a self-attention mechanism for block interaction between enemy and friendly situations is introduced in both the actor network and the critic network to capture the relative correlation information at different positions in the situation sequence and mine the global information of both sides.

[0009] Furthermore, the obtained battlefield situation is time-series encoded, and the situation information of our side is respectively and enemy target unit information The relative time step information about the sequence markers is injected into the self-attention mechanism module to make each dimension +1, and then the temporally encoded situation information is input into the self-attention mechanism module.

[0010] Furthermore, the self-attention mechanism of the block interaction of enemy and friendly situations refers to: Step 3.1: Arrange information according to the original situation Split the representation information into our complete situation representation and incomplete intelligence representation of the enemy , N represents the number of observed target units, and D represents the feature dimension; Step 3.2: Complete characterization of our side , connect a fully connected forward network for dimensional mapping, whose operation includes two linear transformations and a ReLU activation output:

[0011] , Represent the weights of two linear transformations, , Respectively represent the thresholds of two linear transformations; Step 3.3: By introducing an incomplete enemy intelligence representation and The self-attention representation mechanism is used to represent the interaction between the enemy and us:

[0012] in, and ; refers to and Dimensions; Step 3.4: The interactively represented information is dimensionally mapped through a fully connected network.

[0013] Furthermore, the strategy updating method based on knowledge and data hybrid drive refers to driving the strategy update through the sample mixing of offline database-online interaction chain and the strategy mixing of knowledge decision library-avoidance agent.

[0014] Furthermore, the offline database-online interaction chain sample mixing means that after the avoidance decision model starts training, the avoidance decision sequence of the avoidance agent is collected online. As a sample of online interactive chain, one side is in the increasing factor In the linear increasing mode, the samples of the online interaction chain are gradually replaced by the batch samples extracted from the offline database and added to the experience replay pool in a stacked manner; The offline database stores the avoidance decision sequences of multiple domain experts at historical moments. The online interactive chain stores the avoidance decision sequence generated by the avoidance decision model in the process of continuous online interactive feedback with the multi-stage air combat mission environment. , both avoidance decision sequences include battlefield states , Decision Action , action reward , and the updated situation of the battlefield after executing the action .

[0015] Furthermore, the strategy mixing of the knowledge decision base and the avoidance agent refers to mixing the corresponding strategy under the current situation of the knowledge decision base and the corresponding strategy generated by the avoidance agent through weight distribution; The knowledge decision database stores avoidance rules consisting of three parts: expert experience description, decision experience data and basic behavior logic. : <if then >, expert experience description and decision experience data are all from domain experts. Expert experience description is the representational data of key situations and behavior triggers, and decision experience data is the deduction data in typical confrontation scenarios. The basic behavior logic is mainly unstructured information, which comes from manually extracted combat rules, regulations, and pilot experience. The strategy of avoiding intelligent agents is to use the situation in the real-time interaction process with the multi-stage air combat mission environment. Evasion actions generated in .

[0016] Furthermore, the reward function is a multivariate reward function: The reward function is a multivariate reward function: ; in, The reward and penalty value are determined by whether the player survives in the end. It is a procedural reward used to guide the agent to maintain a safe distance from the threatening target and is calculated by the target distance Dis: ; For event rewards, the discrete value is determined by the completion of multi-stage tasks. Each time a stage task is completed, a reward or penalty value is given. The multi-stage refers to the three stages of the task: air confrontation, air escort, and attracting firepower; , , They are the distribution weights of the three rewards, which are assigned by experience.

[0017] The present invention also provides an unmanned aircraft avoidance decision-making intelligent agent based on data and knowledge driving, which is implemented using a unmanned aircraft avoidance decision-making method based on data and knowledge driving.

[0018] By adopting the above technical solution, the present invention has the following beneficial effects: The present invention provides an unmanned fighter aircraft avoidance decision-making method and intelligent agent based on data and knowledge driving, which imitates the way humans recognize rules and then use them, and is based on a strategy updating mode driven by a mixture of knowledge and data. The strategy updating is jointly driven by a sample mixture of an offline database and an online interactive chain and a strategy mixture of a knowledge decision library and an avoidance intelligent agent. Rule knowledge is extracted from existing historical data and decision-making experience, and the unmanned fighter aircraft is guided to generate accurate and efficient avoidance plans in the process of continuous interactive feedback with a multi-stage air combat mission environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of the confrontation between the red and blue sides; Figure 2 It is a data-knowledge dual-driven intelligent avoidance decision-making technology framework; Figure 3 An avoidance agent based on interactive self-attention mechanism; Figure 4 It is a strategy update model based on a hybrid drive of knowledge and data; FIG5( a ) is a reward value comparison diagram of the performance comparison between the avoidance decision model proposed in the present invention and DQN, and FIG5( b ) is a comparison diagram of the avoidance success rate; Figure 6 (a) is a survival rate comparison diagram of the avoidance effect of the DQN model in the three-stage task, and Figure 6 (b) is a score comparison diagram for the red and blue sides; Figure 7 (a) is a survival rate comparison diagram of the avoidance effect of the rule model in the three-stage task, and Figure 7 (b) is a score comparison diagram for the red and blue sides; FIG8 (a) is a survival rate comparison diagram of the avoidance effect schematic diagram of the avoidance decision model proposed in the present invention in three-stage tasks, and FIG8 (b) is a score comparison diagram of the red and blue sides; Fig. 9 This is a schematic diagram for comparing survival times. DETAILED DESCRIPTION

[0020] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The present invention is further explained below in conjunction with specific implementation modes.

[0022] Considering that future combat styles will mostly be joint operations of sea, land and air, this application is based on Figure 1The confrontation scenario shown in the figure is studied. Among them, the combat mission of the red side is that the three unmanned fighters #r3, #r4, and #r5 form a formation to attack the blue side's sea targets and ground targets in the second and third stages. The unmanned fighters #r1 and #r2 perform different defense tasks against enemy targets in three stages; the two unmanned fighters #b1 and #b2 of the blue side conduct air interception in the first stage under the warning of the early warning aircraft #b0, and the ship #b3 and the ground fire strike unit #b4 perform the second and third stage air defense tasks respectively. The weapons configuration of the red and blue sides are mainly missiles, as shown in Table 1.

[0023] Table 1. Countermeasures and weapons configuration

[0024] In this application, #r1 is embedded in an avoidance decision model. Its three-stage combat missions are air confrontation, air escort, and attracting firepower. The overall goal is to attract the enemy's three-stage firepower strikes of a total of 10 missiles for the Red Army's first formation. In order to complete this multi-stage defense mission, the ability to avoid missiles is of paramount importance.

[0025] Figures 2 to 9 A specific embodiment of the unmanned aircraft avoidance decision method based on data and knowledge driving of the present invention is shown, comprising the following steps: A data- and knowledge-driven approach to UAV avoidance decision making, such as Figure 2 As shown, the following steps are included: Step 1: Obtain the flight status of the UAV and target unit information. In this embodiment, the situation information of both parties is obtained through a variety of detection devices, where the UAV is the UAV and the target unit information is the enemy.

[0026] Step 2: Formalize the sequential decision-making process of the UAV’s air combat mission based on the UAV’s flight status.

[0027] In this embodiment, the avoidance decision of the unmanned aircraft is an online dynamic planning process. The intelligent agent responds to the current enemy and friendly situation information and obtains new situation information. This process is repeated until the mission is completed. In order to formally describe the avoidance behavior of the unmanned aircraft, this application performs sequential decision modeling based on the Markov decision process (MDP). The sequential decision process of the unmanned aircraft air combat mission based on the formal modeling of the flight state of the unmanned aircraft refers to: Modeling of sequential decision making based on the Markov decision process, including five elements ; in, is the state space, which is a set of enemy and friendly situation information at each time point in a confrontation round, where yes The moment state code, is the total number of time steps in the confrontation round. In this embodiment, for the combat unit, a single observation value is a one-dimensional vector composed of 40 elements, the first 16 elements include our unmanned combat unit information and target unit information, and the last 24 elements are missile group information. The information length of the unmanned combat unit information and the target unit information is 8, including longitude, latitude, clearance height, heading angle, speed, pitch angle, type, and enemy identification, a total of 8 elements. Since in the actual background, missile information will only appear when the unmanned combat aircraft is engaged, the environment will always monitor the missile information. If the environment monitors the missile information, the missile information will be added to the missile information slot of the one-dimensional vector of the observation value. In the embodiment, the threat level of the missile is sorted according to the monitoring information, and the first three missiles are added to the enemy situation information for priority avoidance.

[0028] is the action space, which is the set of all possible discrete actions, where m is the total number of actions; the UAV actions include three categories: a. Adjusting the altitude of the UAV, which is mainly divided into three actions: ascending, descending, and maintaining altitude; b. Adjusting the speed of the UAV, there are four expected speed settings: low speed, cruise, military, and afterburner; c. Adjusting the heading angle of the UAV, the total heading angle interval is discretely cut into 12 angles. The above three can be executed in parallel, and the actions of the UAV at each moment are obtained according to the following logic: altitude adjustment state, speed adjustment state, and target heading angle index.

[0029] is the state transfer function. It represents the evolution of combat situation. Given the current state , the agent takes action After that, the combat scenario evolves to the next state ,Right now ; The traditional solution platform gives discrete numerical reward values ​​based on obvious turning points of the situation and staged confrontation results, but this method makes it difficult to train the agent due to sparse rewards. Therefore, this embodiment combines domain knowledge and expert guidance to design a multivariate reward function consisting of result rewards, incentive rewards, and event rewards: In this embodiment, the reward function R is a multivariate reward function:

[0030] in, The reward and penalty value are determined by whether the player survives in the end. It is a procedural reward used to guide the agent to maintain a safe distance from the threatening target and is calculated by the target distance Dis: ; For event rewards, the discrete value is determined by the completion of multi-stage tasks. Each time a stage task is completed, a reward or penalty value is given. The multi-stage refers to the three stages of the task: air confrontation, air escort, and attracting firepower; , , They are the distribution weights of the three rewards, which are assigned by experience.

[0031] is the discount factor, which needs to be as large as possible, because this article focuses more on long-term returns.

[0032] Step 3: Construct an avoidance decision model based on the interactive self-attention mechanism.

[0033] For multi-stage air combat missions, the key factors to improve the survival rate of unmanned combat aircraft are the situational awareness and decision-making capabilities of the model. Reinforcement learning has strong decision-making capabilities, but the performance of reinforcement learning algorithms is restricted by perception capabilities, and traditional reinforcement learning methods have slow strategy convergence and low data utilization. Therefore, this embodiment improves the reinforcement learning model.

[0034] In this embodiment, building an avoidance decision model based on the interactive self-attention mechanism means: like Figure 3 As shown in the figure, in the AC algorithm framework, the self-attention mechanism of enemy and friendly situation block interaction is introduced in both the actor network and the critic network to capture the relative correlation information at different positions of the situation sequence and mine the global information of both sides. Based on the AC algorithm, the evasive agent that can maneuver to evade missiles and complete multi-stage air combat tasks is trained. The introduction of the self-attention mechanism can enhance the situational awareness ability and improve the human-like decision-making level of the evasive decision-making model.

[0035] In this embodiment, considering that the self-attention mechanism does not contain time-series processing methods such as convolution and recursion, in order to make full use of the time-series characteristics of situation information during the confrontation process, this application first performs time-series encoding on the battlefield situation, such as Figure 3 As shown in the situation intelligence obtained, the situation information of our side is and enemy target unit information Inject the relative time step information about the sequence tag into , making each dimension +1. Specifically, the time length intercepted from the complete task chain is Continuous historical data for each time point t Our situation information at all times and enemy target unit information The relative time information with a length of 1 in the time segment is concatenated. In addition, the missing information is padded with 0 and a status mark is introduced. , record the total number of enemies and ourselves N+1 complement information of unit status information, N is the number of target units.

[0036] In this embodiment, in view of the correlation between multi-source information inputs, the self-attention mechanism is a single-head self-attention mechanism, which is used to effectively mine the semantic features between the enemy and our situation information obtained at the current moment:

[0037] in, Refers to Query( ) and Key( ), All are situation inputs after temporal encoding.

[0038] The introduction of the single-head self-attention mechanism Self-Attention makes it easier to capture the interdependent features of situation information with long time series distances. Because if it is RNN or LSTM, it needs to be calculated sequentially. For long-distance interdependent features, it takes several time steps of information accumulation to connect the two. The longer the distance, the smaller the possibility of effective capture. However, the self-attention mechanism will directly connect any two pieces of information in the situation through a calculation step during the calculation process, so the distance between long-distance dependent features is greatly shortened, which is conducive to the effective use of these features.

[0039] After residual connection and layer normalization, the block interaction module of the enemy and our situation is introduced to represent the information relationship between the enemy and our situation: Step 3.1: Arrange information based on the original enemy and friendly situation Split the representation information into our complete situation representation and incomplete intelligence representation of the enemy .

[0040] Step 3.2: Complete characterization of our side , connect a fully connected forward network for dimensional mapping, whose operation includes two linear transformations and a ReLU activation output:

[0041] , Represent the weights of two linear transformations, , Respectively represent the thresholds of two linear transformations; Step 3.3: By introducing an incomplete enemy intelligence representation and The self-attention representation mechanism is used to represent the interaction between the enemy and us:

[0042] in, and , Refers to Query and Key Dimension.

[0043] Step 3.4: Dimensionally map the interaction information through a fully connected network.

[0044] This embodiment enhances the situation awareness capability of the decision-making model by introducing a self-attention mechanism for enemy and friendly block interaction.

[0045] Step 4: Use a data- and knowledge-driven hybrid strategy update method to train the avoidance decision model to obtain an avoidance decision agent. In order to imitate the path from human beings’ understanding of rules to their use of rules, a knowledge- and data-driven hybrid strategy update model is used.

[0046] In this embodiment, the strategy update method based on knowledge and data hybrid drive refers to driving the strategy update through the offline database-online interactive chain sample hybrid and the knowledge decision library-avoidance agent strategy hybrid. In order to imitate the human's way of understanding rules and using rules, rule knowledge is extracted from existing historical data and decision-making experience, guiding the unmanned fighter to generate accurate and efficient avoidance solutions in the process of continuous interactive feedback with the multi-stage air combat mission environment by making full use of data information and large-scale dynamic exploration. The specific method is as follows: Figure 4 shown.

[0047] In this embodiment, the offline database-online interaction chain sample mixing refers to collecting the avoidance decision sequence of the avoidance agent online after the avoidance decision model starts training. As a sample of online interactive chain, one side is in the increasing factor In the linear incremental mode, the samples of the online interaction chain are gradually replaced by the batch samples extracted from the offline database in a stacked manner and added to the experience replay pool.

[0048] In this embodiment, the offline database stores the avoidance decision sequences of experts in multiple fields at historical moments. The online interaction chain stores the avoidance decision sequence generated by the avoidance agent in the process of continuous online interaction feedback with the multi-stage air combat mission environment. , both avoidance sequences include battlefield status , Decision Action , action reward , and the updated situation of the battlefield after executing the action .

[0049] A sample hybrid approach to implement an offline database and an online interactive chain is: , Per A fixed time step with a rate speed increases.

[0050] A linear incremental method is used to guide the two types of samples to achieve sample mixing in different proportions at different training stages, which leads to the conversion of the sampling method. Different sample sampling methods are adopted in different strategy learning stages to improve the algorithm performance and data utilization from the perspective of sample sampling method, and further improve the performance of the offline-online hybrid reinforcement learning algorithm.

[0051] The offline database is constructed by: given a battlefield t Environmental observation value at all times , for each rule in the knowledge decision base :<if x then y> The prior situation x is used to learn the action weights based on the fully connected network ,exist n Generate a strategy based on the maximum probability of actions :

[0052] Get environmental feedback , which constitutes a historical data fragment ,Through interaction with the environment, data fragments are continuously generated to form an offline database.

[0053] In this embodiment, the strategy mixing of the knowledge decision base and the avoidance agent refers to mixing the strategy corresponding to the current situation of the knowledge decision base and the corresponding strategy generated by the avoidance agent through weight distribution; The knowledge decision database stores avoidance rules consisting of three parts: expert experience description, decision experience data and basic behavior logic. : <if then >, expert experience description and decision experience data are all from domain experts. Expert experience description is the representational data of key situations and behavior triggers, and decision experience data is the deduction data in typical confrontation scenarios. The basic behavior logic is mainly unstructured information, which comes from manually extracted combat rules, regulations, and pilot experience. The strategy of avoiding intelligent agents is to use the situation in the real-time interaction process with the multi-stage air combat mission environment. Generate evasive maneuvers .

[0054] Avoid the two parts of the agent output, one is the weight parameter , one is the Q value , the overall action value is obtained by strategy mixing calculation:

[0055] based on Strategy selection action , ; Get environmental feedback , forming an online interactive chain: .

[0056] Step 5: Input the flight status of the UAV and the target unit information into the avoidance decision agent, and generate a maneuver avoidance plan in real time during the sequential decision-making process of the air combat mission.

[0057] Experimental verification In order to evaluate the performance and conduct application analysis on the avoidance decision model proposed in this application, a multi-stage air combat mission is set based on a simulation platform. The intelligent avoidance decision model is embedded in the fighter #r1, and the missile avoidance maneuvers are learned during the online interaction with the simulation environment. By comparing with the classic reinforcement learning algorithm DQN and the rule-based maneuver model, (1) the avoidance decision performance of the avoidance decision model in the entire air combat confrontation mission, (2) the avoidance effect of the avoidance decision model facing the mission objectives of each stage, and (3) the survival time of the intelligent avoidance technology model in the entire air combat process are analyzed, thereby realizing model performance analysis and air combat effectiveness evaluation.

[0058] The comparison algorithms selected in the experiment are the classic reinforcement learning model - DQN, whose network adopts LSTM+FCN, and the rule script model that comes with the simulation platform. For the avoidance decision algorithm and DQN proposed in the application, the model reward parameters are given in Table 2. Figure 1 In the multi-stage air combat mission set in the figure, fighter #r1 faces three different targets in three stages to attract firepower and conduct missile defense. When the threatening target appears, it starts the autonomous decision-making mode of maneuvering evasive action. If it is shot down, the round of simulation ends. After 10,000 simulations, the model training times reaches about 5,000 times. The reward value of the model and the success rate of evading missiles are recorded during the training process to analyze the avoidance decision-making performance of the model. The specific results are given in Figure 5.

[0059] Table 2 Model reward parameters

[0060] As can be seen from Figure 5 (a), both DQN and the avoidance decision model proposed in this embodiment can converge stably after training for about 5000 times. Although the reward obtained by the avoidance decision model proposed in this application is higher than that of DQN, the fluctuation is relatively obvious. This is because the situation is quite different under the task drive at different stages. After introducing the self-attention mechanism of enemy and friendly block interaction, the avoidance agent is more sensitive to the situation and actively explores different strategies based on the potential global information mined, resulting in obvious ups and downs. However, it is also because the sufficient exploration model is easier to make the optimal decision. As shown in Figure 5 (b), compared with DQN, the avoidance decision model proposed in this application can speed up the convergence speed while increasing the overall avoidance success rate by 10%. The overall avoidance success rate reaches more than 75%, which is about 18% higher than the rule script model. It can be seen that for this multi-stage mission, the evasion effects of the three models are not very ideal. In order to analyze the effectiveness of each stage, we evaluate the evasion effects of the three models through the survival rate of fighter #r1 in each round facing four types of targets #b1, #b2, #b3 and #b4 in air-to-air confrontation, air-to-sea penetration and air-to-ground strike, as well as the final scores of the red and blue teams, as shown in Figures 6-8.

[0061] It can be seen from Figures 6 (a) to 8 (a) that the three models have the highest survival rate in the first stage of air-to-air confrontation, reaching about 85%, 78%, and 99% respectively. However, the defense effects in the air-sea penetration and air-to-ground strike stages are very unsatisfactory, especially for the two missiles against the anti-missile vehicle, with survival rates of only about 5%, 0%, and 20%. Although the avoidance decision model of this application has been improved to a certain extent compared with the DQN and rule models, the avoidance effect is still poor. This is determined by the characteristics of the weapon itself. The air defense missile itself is very fast and has a strong terminal guidance capability. It is difficult to avoid it through tactical actions. Moreover, the anti-missile vehicle can only start precise strikes when the target enters the hit area. This is a pain point and difficulty for the long-term pre-planning of the avoidance agent. In addition, by observing Figures 6(b) to 8(b), it can be seen that the autonomous decision-making scheme of the rule model is difficult to adapt to the complex multi-stage air combat mission scenario, resulting in the red team scoring lower than the blue team and becoming the losing team. However, both DQN and the avoidance decision model of the present application can achieve attack and penetration of the enemy when fighter #r1 completes fire attraction and multi-stage avoidance. In addition, in order to have a more intuitive evaluation of the missile evasion capability of the evasive agent, we counted the survival time of fighter #r1 in each round, such as Fig. 9 As shown, it can be seen that compared with DQN and rule models, the avoidance decision model of the present application can greatly increase the survival time of fighter jets during air combat, improve the survival rate of unmanned fighter jets, and provide a living force guarantee for the completion of multi-stage tasks.

[0062] The present invention also provides an unmanned aircraft avoidance decision-making intelligent agent based on data and knowledge driving, which is implemented using a unmanned aircraft avoidance decision-making method based on data and knowledge driving.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data- and knowledge-driven unmanned aircraft avoidance decision-making method, characterized in that: The method comprises the following steps: Step 1: obtaining the flight status of the unmanned aircraft and target unit information; Step 2: Formalize the sequential decision-making process of the UAV air combat mission based on the flight status of the UAV; Step 3: Construct an avoidance decision model based on the interactive self-attention mechanism; Step 4: Use a data- and knowledge-based hybrid-driven strategy update method to train the avoidance decision model to obtain an avoidance decision agent; Step 5: Input the flight status of the UAV and the target unit information into the avoidance decision-making agent, and generate a maneuver avoidance plan in real time during the sequential decision-making process of the air combat mission.

2. The drone avoidance decision method according to claim 1, characterized in that: The sequential decision-making process of unmanned aircraft air combat missions based on the formal modeling of unmanned aircraft flight status refers to: Modeling of sequential decision making based on the Markov decision process, including five elements ; in, is the state space, which is a set of enemy and friendly situation information at each time point in a confrontation round, where yes The moment state code, is the total number of time steps in this adversarial round; is the action space, which is the set of all possible discrete actions, where m is the total number of actions; is the state transfer function: it represents the evolution of the combat situation. Given the current state , the agent takes action After that, the combat scenario evolves to the next state ,Right now ; is the reward function, is the discount factor.

3. The drone avoidance decision method according to claim 2, characterized in that: Building an avoidance decision model based on the interactive self-attention mechanism means: In the AC algorithm framework, a self-attention mechanism for block interaction between enemy and friendly situations is introduced in both the actor network and the critic network to capture the relative correlation information at different positions in the situation sequence and mine the global information of both sides.

4. The drone avoidance decision method according to claim 3, characterized in that: The obtained battlefield situation is time-series encoded, and the situation information of our side is respectively and enemy target unit information The relative time step information about the sequence markers is injected into the self-attention mechanism module to make each dimension +1, and then the temporally encoded situation information is input into the self-attention mechanism module.

5. The drone avoidance decision method according to claim 4, characterized in that: The self-attention mechanism of the block interaction of enemy and friendly situations refers to: Step 3.1: Arrange information according to the original situation Split the representation information into our complete situation representation and incomplete intelligence representation of the enemy , N represents the number of observed target units, and D represents the feature dimension; Step 3.2: Complete characterization of our side , connect a fully connected forward network for dimensional mapping, whose operation includes two linear transformations and a ReLU activation output: , Represent the weights of two linear transformations, , Respectively represent the thresholds of two linear transformations; Step 3.3: By introducing an incomplete enemy intelligence representation and The self-attention representation mechanism is used to represent the interaction between the enemy and us: in, and ; refers to and Dimensions; Step 3.4: The interactively represented information is dimensionally mapped through a fully connected network.

6. The drone avoidance decision method according to claim 5, characterized in that: The strategy updating method based on knowledge and data hybrid drive refers to driving the strategy update through the sample mixing of offline database-online interaction chain and the strategy mixing of knowledge decision library-avoidance agent.

7. The drone avoidance decision method according to claim 6, characterized in that: The offline database-online interaction chain sample mixture refers to the online collection of the avoidance decision sequence of the avoidance agent after the avoidance decision model starts training. As a sample of online interactive chain, one side is in the increasing factor In the linear increasing mode, the samples of the online interaction chain are gradually replaced by the batch samples extracted from the offline database and added to the experience replay pool in a stacked manner; The offline database stores the avoidance decision sequences of multiple domain experts at historical moments. The online interactive chain stores the avoidance decision sequence generated by the avoidance decision model in the process of continuous online interactive feedback with the multi-stage air combat mission environment. , both avoidance decision sequences include battlefield states , Decision Action , action reward , and the updated situation of the battlefield after executing the action .

8. The drone avoidance decision method according to claim 7, characterized in that: The strategy mixing of the knowledge decision base and the avoidance agent refers to mixing the corresponding strategy under the current situation of the knowledge decision base and the corresponding strategy generated by the avoidance agent through weight distribution; The knowledge decision database stores avoidance rules consisting of three parts: expert experience description, decision experience data and basic behavior logic. : <if then >, expert experience description and decision experience data are all from domain experts. Expert experience description is the representational data of key situations and behavior triggers, and decision experience data is the deduction data in typical confrontation scenarios. The basic behavior logic is mainly unstructured information, which comes from manually extracted combat rules, regulations, and pilot experience. The strategy of avoiding intelligent agents is to use the situation in the real-time interaction process with the multi-stage air combat mission environment. Evasion actions generated in .

9. The drone avoidance decision method according to claim 2, characterized in that: The reward function is a multivariate reward function: in, The reward and penalty value are determined by whether the player survives in the end. It is a process reward used to guide the agent to maintain a safe distance from the threat target. Dis calculate: ; For event rewards, the discrete value is determined by the completion of multi-stage tasks. Each time a stage task is completed, a reward or penalty value is given. The multi-stage refers to the three stages of the task: air confrontation, air escort, and attracting firepower; , , They are the distribution weights of the three rewards, which are assigned by experience.

10. A data- and knowledge-driven unmanned aircraft avoidance decision-making intelligent agent, characterized in that: The steps of the unmanned aircraft avoidance decision method based on data and knowledge driven are implemented using any one of claims 1 to 9.

Images