DQN-based decision-making method for aviation multi-platform collaborative intelligent confrontation
Through multi-platform intelligent collaborative behavior modeling based on DQN algorithm, the defects of relying on rule base and human-in-loop in the existing technology are solved, and the self-learning and self-evolution of the joint combat simulation platform of aviation force is realized, and the situation of inexperienced is optimized. Explore combat strategy and simulate system efficiency under the simulator.
Patent Information
- Application Number
- CN202111244551.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-10-26
AI Technical Summary
The existing aviation force joint combat simulation platform relies on the rule base and human-in-the-loop, lacks self-learning and self-evolution capabilities, and it is difficult to explore the optimal combat strategy without experience. The participation of humans leads to the slow propulsion of the simulation system and low efficiency.
Using multi-platform intelligent collaborative behavior modeling method based on DQN algorithm, we build a collaborative combat behavior model of the scheduling layer, formation layer, and stand-alone layer, design reward functions and state space, form a reinforcement learning framework, and obtain multi-platform collaboration through iterative learning training Combat strategy.
It realizes self-learning and self-evolution of combat strategies in the absence of experience, optimizes combat command decisions, and improves the operation efficiency and strategy exploration capabilities of simulation systems.
Smart Images

Figure CN114330093B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the fields of combat situation analysis technology, intelligent game technology, aviation modeling technology and simulation technology, and in particular to a DQN-based aviation multi-platform collaborative intelligent confrontation decision-making method. Background Art
[0002] After years of experience accumulation, the existing aviation force joint combat simulation platform has relatively mature physical and behavioral modeling for single weapon platforms, but the command decision-making for coordinating joint operations between multiple platforms is still based on rule bases, which are formed by the command experience of experts. In addition, there are also some multi-platform joint confrontation simulation systems where the command decision-making is human-in-the-loop, and the command decision-making of experienced combat commanders is connected to the simulation system. These methods are highly dependent on existing experience, and the command decision-making based on the rule base does not have the ability of self-learning and self-evolution; and the command decision-making based on the rule base does not have the ability of self-learning and self-evolution; the command decision-making system based on human-in-the-loop has high requirements for combat commanders and the advancement of the simulation system is slowed down due to human participation, which reduces the operating efficiency of the simulation system.
[0003] Therefore, for completely new combat scenarios, without any accumulated combat experience, how to explore the optimal combat strategy and achieve self-learning and self-evolution of combat strategies has become an urgent problem that needs to be solved. Summary of the invention
[0004] In view of this, the present disclosure provides an aviation multi-platform intelligent collaborative behavior modeling method based on the DQN algorithm to solve the problems of the aviation multi-platform intelligent collaborative behavior modeling method based on the DQN algorithm in the prior art.
[0005] The present disclosure provides a method for modeling aviation multi-platform intelligent collaborative behavior based on a DQN algorithm, including:
[0006] S1 builds a coordinated combat behavior model of the scheduling layer, formation layer, and single-machine layer based on the DQN algorithm and rule set as the command and decision-making platform of the scheduling layer, formation layer, and single-machine layer;
[0007] S2 constructs the state space of the scheduling layer based on the characteristics of multi-platform air combat of aviation forces and the importance level of relevant factors affecting the results of multi-platform joint operations;
[0008] S3 constructs the behavior space of the scheduling layer based on the operational characteristics of different force formations in the aviation system;
[0009] S4 designs the reward function in the DQN algorithm according to the specific combat objectives;
[0010] S5 forms a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation based on the DQN algorithm, and conducts iterative learning training of multi-platform joint command decision-making based on the reinforcement learning framework to obtain a multi-platform collaborative combat strategy.
[0011] Said S1 comprises:
[0012] Based on the DQN algorithm, a scheduling layer behavior decision model is constructed;
[0013] Based on the rule set, construct the combat behavior models of the formation layer and the single-machine layer respectively;
[0014] A collaborative combat behavior model is constructed based on the scheduling layer behavior decision model, the formation layer combat behavior model and the single-machine layer combat behavior model.
[0015] The S2 specifically includes:
[0016] According to the characteristics of multi-platform air combat of aviation forces and the importance level of relevant factors affecting the results of multi-platform joint operations, key elements are extracted to construct the state space vector of the scheduling layer DQN algorithm. Among them, the key elements include the coordinate positions of the enemy and our aircraft, the types of enemy and our forces, the headings of the enemy and ours, the amount of our bombs carried, and the coverage ratio of the enemy and our detection information domain;
[0017] Based on the state space vector, the battlefield situation of the aviation multi-platform joint confrontation is mathematically represented to construct the scheduling layer state space.
[0018] The S3 specifically includes:
[0019] Based on the combat characteristics of different force formations in the aviation system, a set of executable tasks for different force platform formations is formed, and based on the executable task set, the scheduling layer behavior space of the aviation multi-platform equipment is constructed.
[0020] The S3 further includes:
[0021] Design an action parsing algorithm for the DQN output, and based on the action parsing algorithm for the DQN output, establish a mapping relationship between a single action sequence number output by the DQN algorithm and an action space of multi-platform aviation equipment.
[0022] The S4 specifically includes:
[0023] Design the components of the reward function according to the specific combat objectives, where the components include the number of losses of different force entities and the ratio of our information detection range to the enemy's information detection range;
[0024] Based on the components of the reward function, and according to the importance of different force platforms to the final combat goal, the weight coefficients of the components of the reward function are determined;
[0025] A reward function is constructed based on the reward element and the weight coefficient.
[0026] The reward function consists of two parts: a reward value and a penalty value.
[0027] The S5 comprises:
[0028] Based on the DQN algorithm, the scheduling layer state space, the scheduling layer behavior space and the reward function are integrated to construct a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation;
[0029] Based on the reinforcement learning framework, iterative learning and training of multi-platform joint command decision-making is carried out to obtain the aviation multi-platform collaborative intelligent confrontation strategy.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] 1. This paper introduces deep reinforcement learning into the command decision-making modeling of the aviation multi-platform confrontation formation layer. Compared with the traditional command decision-making model based on rule sets or finite state products, it can realize the self-learning and self-evolution of command strategies, and support the exploration and optimization of combat command strategies in new combat scenarios without experience accumulation.
[0032] 2. Aiming at the mismatch between the single action sequence number output by the DQN algorithm and the multi-platform action decision of the aviation force, an action parsing method for the output sequence number of the DQN algorithm is proposed to achieve the goal of coordinating and commanding the joint operations of multi-platform forces through a DQN network.
[0033] 3. Adopt a target-oriented and global situation optimization reward value generation algorithm, design the components of the reward function for clear combat objectives, determine the weight coefficients of each component of the reward function according to the importance level of different force platforms to the final combat objectives, and finally use the reward function to drive the aviation multi-platform confrontation command decision-making model to complete strategy generation and optimization. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the present disclosure, the drawings required for use in the embodiments or prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1It is a flow chart of the DQN-based aviation multi-platform collaborative intelligent confrontation decision-making method disclosed in the present invention;
[0036] Figure 2 It is a flow chart of constructing a collaborative combat behavior model provided by the present disclosure;
[0037] Figure 3 is a flow chart of the scheduling layer state space provided by the present disclosure;
[0038] Figure 4 is a flowchart of constructing a scheduling layer behavior space provided by the present disclosure;
[0039] Figure 5 is a flow chart of the design reward function provided by the present disclosure;
[0040] Figure 6 It is a flow chart of obtaining aviation multi-platform collaborative intelligent confrontation strategy provided by the present disclosure;
[0041] Figure 7 It is a schematic diagram of the command decision-making hierarchical framework of the Red Army's multi-force platform provided by the present disclosure;
[0042] Figure 8 It is a flow chart of the aviation multi-platform intelligent confrontation simulation command decision-making behavior model training provided by the present disclosure;
[0043] Fig. 9 is a schematic diagram of the target allocation principle within a fighter formation provided by the present disclosure;
[0044] Fig.10 It is a schematic diagram of the combat rules of a single platform in a fighter formation provided by the present disclosure;
[0045] Fig.11 It is a schematic diagram of calculating the information domain coverage ratio provided by the present disclosure;
[0046] Fig.12 is a schematic diagram of the design of a non-sparse reward function provided by the present disclosure;
[0047] Fig.13 It is an aviation multi-platform adversarial reinforcement learning framework based on DQN provided by the present invention. DETAILED DESCRIPTION
[0048] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.
[0049] The following will describe in detail the aviation multi-platform intelligent collaborative behavior modeling method according to the present disclosure in conjunction with the accompanying drawings.
[0050] Figure 1 It is a flow chart of the DQN-based aviation multi-platform collaborative intelligent confrontation decision-making method provided by the present invention.
[0051] like Figure 1 As shown, the aviation multi-platform intelligent collaborative behavior modeling method includes:
[0052] S1, based on the DQN algorithm and rule set, constructs a collaborative combat behavior model of the scheduling layer, formation layer, and single-machine layer as a command and decision-making platform for the scheduling layer, formation layer, and single-machine layer.
[0053] Figure 2 It is a flow chart of constructing a collaborative combat behavior model provided by the present disclosure.
[0054] like Figure 2 As shown, S1 specifically includes:
[0055] S11, based on the DQN algorithm, build a scheduling layer behavior decision model;
[0056] The scheduling layer is the command and decision-making layer for aviation multi-platform operations. Based on the DQN algorithm, the scheduling layer state space is constructed.
[0057] Among them, DQN (Deep Q Network, deep reinforcement learning) is a type of reinforcement learning.
[0058] S12, based on the rule set, construct the formation layer and single-aircraft layer combat behavior models respectively.
[0059] The formation layer is a collection of formations organized according to different forces, and the single-aircraft layer is a collection of aircraft with specific numbers in each formation layer. The formation layer and single-aircraft layer of different force platforms have their own corresponding combat behaviors. Based on the rule set behavior modeling method, the specific combat behaviors of the formation layer and single-aircraft layer of different force platforms are modeled separately. For example, the allocation of fighter jet air interception targets, the fighter jet air interception behavior, and the allocation of bomber assault targets are modeled separately.
[0060] S13, based on the scheduling layer behavior decision model, the formation layer combat behavior model and the single-machine layer combat behavior model, constitutes a collaborative combat behavior model.
[0061] The scheduling layer behavior decision model based on the DQN algorithm, the formation layer combat behavior model based on the rule set, and the single-aircraft layer combat behavior model are integrated together. The scheduling layer is responsible for scheduling and command, the formation layer is responsible for selecting the target allocation of different types of formations, and the single-aircraft layer is responsible for dispatching specific aircraft to perform the task of fighting against enemy aircraft through the assignment of the formation layer. The three layers constitute a collaborative combat behavior model to form a collaborative combat behavior model.
[0062] Exemplarily, the scheduling layer includes three patrol area selections: fighter formation patrol area selection, bomber formation patrol area selection and early warning aircraft formation patrol area selection. Based on the output of each frame of the DQN network, it is area A in the fighter formation patrol area, area B in the bomber formation patrol area and area C in the early warning aircraft formation patrol area. At the same time, the formation layer will perform target allocation based on the rule set. For example, the fighter formation will allocate air interception instructions to some fighters in the formation, and the bomber formation will allocate target assault instructions to some bombers in the formation. During the simulation operation, if a single machine in the formation receives a target allocation instruction, it will perform the corresponding air interception task or target assault task, otherwise it will go to the patrol area output by the DQN network to perform the regional patrol task. Among them, the tasks that can be performed by a single fighter include: air interception of fighters, regional patrol of fighters, route planning of fighters, patrol of fighter takeoff areas and return of fighters. The tasks that a single bomber can perform include: bomber target assault, bomber area patrol, bomber route planning, and bomber take-off area patrol.
[0063] In addition, the early warning aircraft also plays an important role in the overall mission execution. The tasks that the early warning aircraft can perform include: early warning aircraft area patrol detection, early warning aircraft take-off area patrol detection, and early warning aircraft return.
[0064] S2, based on the characteristics of aviation multi-platform air combat and the importance level of related factors affecting the results of multi-platform joint operations, construct the scheduling layer state space.
[0065] Figure 3 It is a flow chart of the scheduling layer state space provided by the present disclosure.
[0066] like Figure 3 As shown, S2 specifically includes:
[0067] S21, according to the characteristics of multi-platform air combat of aviation forces and the importance level of relevant factors affecting the results of multi-platform joint operations, extract key elements to construct the state space vector of the scheduling layer DQN algorithm, among which the key elements include the coordinate positions of enemy and friendly aircraft, the types of enemy and friendly forces, the headings of both sides, the amount of our bombs carried, and the coverage ratio of the enemy and friendly detection information domain.
[0068] Air Force operations are different from those of the Navy and the Army, and have a unique combat environment. Aviation multi-platform air combat has relatively complex combat characteristics, involving not only its own factors, but also environmental factors such as weather factors. Therefore, there are many related factors that affect the results of multi-platform joint operations, and the importance of these related factors is also different. According to the characteristics of aviation multi-platform air combat and the importance level of related factors affecting the results of multi-platform joint operations, key elements are extracted from them to construct the state space vector of the scheduling layer DQN algorithm, so as to provide a parameter basis for constructing the state space of the scheduling layer.
[0069] S22, based on the state space vector, mathematically represents the battlefield situation of the aviation multi-platform joint confrontation and constructs the scheduling layer state space.
[0070] Under different circumstances, the battlefield situation of the multi-platform joint confrontation of air forces is different. The mathematical representation of the specific battlefield situation of the multi-platform joint confrontation of air forces based on the state space vector is equivalent to using the state space vector to quantitatively represent the specific battlefield situation of the multi-platform joint confrontation of air forces, and thereby constructing the scheduling layer state space.
[0071] S3 constructs the dispatch layer behavior space based on the combat characteristics of different force formations in the aviation system.
[0072] Figure 4 It is a flowchart of constructing the scheduling layer behavior space provided by the present disclosure.
[0073] like Figure 4 As shown, S3 specifically includes:
[0074] Based on the combat characteristics of different force formations in the aviation system, a set of executable tasks for different force platform formations is formed. Based on the executable task sets for different force platform formations, the scheduling layer behavior space of the aviation multi-platform equipment is constructed.
[0075] Different force formations in the aviation system have corresponding combat characteristics. According to the combat characteristics of different force formations in the aviation system, executable task sets of different force platform formations are formed. The executable task sets of different formations together constitute the behavior space of the scheduling layer.
[0076] S3 also includes: designing an action parsing algorithm output by DQN, and based on the action parsing algorithm output by DQN, establishing a mapping relationship between a single action sequence number output by the DQN algorithm and an action space of multi-platform aviation equipment.
[0077] According to the action parsing algorithm of DQN output, a DQN output parsing structure is established to map the single action sequence number output by the DQN algorithm to the action space of multi-platform aviation equipment, thereby solving the problem that the single action sequence number output of DQN does not match the action space of multiple force platforms.
[0078] The action number output by the DQN network is mapped to the sub-action number of different formations according to the analytical algorithm. Each sub-action number corresponds to a patrol area in different aviation formations at the scheduling layer.
[0079] S4, design the reward function in the DQN algorithm based on specific combat objectives.
[0080] The design of reward components and corresponding weight values will vary depending on the combat objectives. The design of reward function elements and weights for specific combat objectives should be targeted and optimized for the overall situation.
[0081] Figure 5 It is a flow chart of the design reward function provided by the present disclosure.
[0082] like Figure 5 As shown, S4 specifically includes:
[0083] S41, designing the components of the DQN algorithm reward function for specific combat objectives, wherein the components may generally include the number of losses of different force entities and the ratio of our information detection range to the enemy's information detection range;
[0084] In actual aviation combat scenarios, in order to achieve our combat objectives, the design of the single-step reward function usually takes into account the following elements: the advantages of fighters in air combat, the advantages of bombers in surface combat, and the advantages of bombers in combat against surface ships. In addition, since early warning aircraft are the most important platform for detecting intelligence information, they have a huge impact on the combat effectiveness of the aviation forces of both sides. In order to make the decisions output by the DQN network guide our aviation forces to victory as quickly as possible, the fuel loss term element can also be added to the reward function to make the combat duration as short as possible.
[0085] S42, based on the components of the reward function, and according to the importance of different force platforms to the final combat objective, determine the weight coefficient of each component of the reward function.
[0086] Based on the component functions of the reward function, the weight coefficients of each component element of the single-step reward function are determined according to the importance of different force platforms to the final combat goal. For example, in a certain confrontation training, the combat goal is to bomb the enemy's ground command post. 8 fighters are responsible for air assault and escort missions, and 2 bombers are responsible for ground bombing missions. In this confrontation mission, the importance of our bombers is obviously greater than that of fighters. The loss penalty weight of bombers in the reward function should be multiple times that of fighters, so as to guide fighters to form an escort strategy for bombers.
[0087] S43, constructing a reward function based on the reward element and the weight coefficient.
[0088] Based on the reward element and weight coefficient, a reward function is constructed according to the single-step reward and the settlement reward. The reward function includes two parts: the reward value and the penalty value. The reward function can provide a reward value or a penalty value according to the decision action.
[0089] S5, based on the DQN algorithm, forms a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation, and conducts iterative learning training of multi-platform joint command decision-making based on the reinforcement learning framework to obtain multi-platform collaborative combat strategies.
[0090] Based on the DQN algorithm, the dispatching layer state space, formation layer combat behavior model and single-aircraft layer combat behavior model jointly form a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation. Based on the above reinforcement learning framework, iterative learning and training of multi-platform joint command decision-making are carried out to obtain the command decision-making framework of the collaborative combat behavior model, and finally the aviation multi-platform collaborative intelligent confrontation decision-making is obtained.
[0091] Figure 6 It is a flow chart of obtaining the aviation multi-platform collaborative intelligent confrontation strategy provided by the present invention.
[0092] like Figure 6 As shown, S5 includes:
[0093] S51, based on the DQN algorithm, integrates the scheduling layer state space, scheduling layer behavior space and reward function to construct a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation.
[0094] Based on the DQN algorithm, the scheduling layer state space, scheduling layer behavior space and reward function are integrated to form a complete aviation multi-platform joint combat simulation confrontation model, forming a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation.
[0095] S52, based on the reinforcement learning framework, conducts iterative learning and training of multi-platform joint command decision-making to obtain the aviation force's multi-platform collaborative intelligent confrontation strategy.
[0096] Based on the DQN reinforcement learning framework, in a simulated confrontation scenario, driven by the above-mentioned reward function, the DQN network parameters are iteratively learned and trained, so that the multi-platform intelligent collaborative behavior decision-making can iteratively evolve towards the confrontation strategy with the optimal return, and finally the aviation multi-platform collaborative intelligent confrontation strategy is obtained.
[0097] Example 1
[0098] Figure 7 It is a schematic diagram of the command decision-making hierarchical framework of the Red Army's multi-force platform provided by the present invention.
[0099] like Figure 7 The hierarchical framework of aviation multi-platform command decision-making is shown in the figure.
[0100] Step 1: In a typical aviation multi-platform confrontation simulation scenario, the blue side is used as the enemy's confrontation force. The blue side's forces include but are not limited to typical equipment such as ships, fighter formations and early warning aircraft. The combat decision-making method is completely based on the finite state product and rule set. The red side is used as our force. The red side's forces include but are not limited to fighter assault formations, bomber formations, fighter escort formations and early warning aircraft. The joint command decision modeling of the red side's aviation system is transformed into a three-layer task decision modeling problem of scheduling-formation-single aircraft. The red and blue confrontation airspace is divided into pn different patrol areas. The patrol areas of different aviation formations in the scheduling layer are decided by the DQN algorithm to guide the red side's forces to different combat airspaces to confront the blue side's forces. In the red side's aviation force, the air interception behavior of a single fighter, the target assault behavior of a single bomber, and the target detection behavior of the early warning aircraft adopt a behavior modeling method based on a rule set. The target allocation behavior in the fighter formation and bomber formation is also modeled based on the rule set. At this point, the Red Army aviation system has formed a hierarchical structure framework for command and decision-making based on a mixture of rule sets and reinforcement learning algorithms.
[0101] Specific steps:
[0102] Figure 8 It is a flow chart of the aviation multi-platform intelligent confrontation simulation command decision-making behavior model training provided by the present invention.
[0103] Step 101: First, establish the target attack principle of the Red and Blue forces. Take ships as an example. With multiple launch channels, the effective range of air attack can reach more than 100 kilometers, which can achieve all-round attack. The attack principle of ships is designed as the principle of attacking at the nearest distance. For example, the Blue command center commands ships to attack enemy air targets close to the ships first based on the fused intelligence information.
[0104] Step 102: Determine the command decision variables of the Red Army command post at the scheduling layer based on the DQN algorithm. Taking the patrol area selection of different formations of the Red Army aviation as an example, in each frame of the simulation confrontation operation, the Red Army command center determines the patrol area positions of the assault fighter formation, bomber formation, and early warning aircraft formation based on the DQN network. The patrol area position divides the combat space into pn different areas.
[0105] Fig. 9 It is a schematic diagram of the target allocation principle within a fighter formation provided by the present invention.
[0106] Step 103: Taking the fighter formation as an example, a target allocation rule set within the red and blue combat aircraft formations is established as shown in Table 1. The target allocation process formed based on this rule set is as follows: Figure 3 As shown. A strike target list is designed for each surviving combat force in the formation. The strike target list arranges the blue combat forces discovered by the intelligence detection system in order of distance. Each combat aircraft in the formation will give priority to attacking the nearest enemy target, and will only lock onto the enemy target aircraft once.
[0107] Table 1 Fighter formation target allocation rule set
[0108]
[0109] Fig.10 It is a schematic diagram of the combat rules of a single platform in a fighter formation provided by the present invention.
[0110] Step 104: Taking fighter jets as an example, the combat rule set for the single-machine platform in the combat formation of the red and blue sides is established as shown in Table 2 below. The single-machine combat finite state transition diagram formed according to the rule set is shown in Figure 10. It includes three states: air interception, patrol to the designated area, and return. When the patrol fighter receives the interception order from the red command center, it switches from the regional patrol state to the air interception state. When the designated enemy aircraft is destroyed, the fighter will return to the designated area for regional patrol. When the fighter is short of fuel while performing an air interception mission or a patrol mission to the designated area, the fighter will return to the airport to replenish fuel.
[0111] Table 2 Combat rules set for single platform in fighter formation
[0112]
[0113] Step 2: Based on the characteristics of aviation multi-platform air combat and the importance of relevant factors affecting the results of multi-platform joint operations, key elements are extracted to construct state space vectors to effectively represent the battlefield situation of aviation multi-platform joint confrontation.
[0114] Specific steps:
[0115] Step 201: In a typical scenario, the Red side's aviation system usually includes but is not limited to fighter formations, bomber formations, and early warning aircraft; the Blue side's forces generally include but are not limited to fighter formations, early warning aircraft, and ships. According to the importance of the impact on the outcome of the air battle, the coordinate positions (X r ,Y r ), Force Type LX r , heading H r 、Ammunition capacity D_num r And the red side detection information domain coverage ratio I r etc., select the coordinate positions of all the blue team's troop platforms (X b ,Y b ), Force Type LX b , heading H r And the blue team's detection information domain coverage ratio I b Etc., together constitute the state input space of the red team command decision model based on the DQN network, for example: S = <X r ,Y r ,LX r ,H r ,D_num r ,I r ,X b ,Y b ,LX b ,H b ,I b >.
[0116] Fig.11 It is a schematic diagram of the information domain coverage ratio calculation provided by the present disclosure.
[0117] Step 202: For the calculation method of battlefield situation information, the ratio of the detection information domain coverage range of the red and blue sides I r and I b For example, the calculation process is as follows Figure 5 As shown in the figure. First, from the situation information returned in each frame, the current positions of all surviving aircraft on both sides are captured, and the information detection range of different types of aircraft is obtained according to the attributes of different aircraft types. Secondly, according to the aircraft position and information domain detection range, the information domain coverage range of all aircraft on the red side and the blue side is drawn on the white canvas respectively to form a fused information domain situation map. Finally, the area ratio of the information domain coverage range of the red side and the blue side is counted on the white canvas respectively, and the quantitative representation of the relative situation of the information domain can be obtained. r and I b .
[0118] Step 3: Design the behavior space of the scheduling layer of the multi-platform aviation system based on the DQN algorithm. According to the combat characteristics of different force formations in the aviation system, a set of executable tasks for different force formations is formed, which together constitute the behavior space of multi-platform joint command decision-making. Design a DQN output parsing dictionary and establish a mapping relationship between the action sequence number output by the DQN algorithm and the action space of multiple aviation platforms. Taking the command decision variable of the scheduling layer as the patrol area selection of different formations as an example, the specific steps of designing the behavior space of the scheduling layer are explained.
[0119] Step 301: First, divide the combat area into pn different patrol areas.
[0120] Step 302: When each frame of simulation is running, it is assumed that the combat command decision model based on the DQN network outputs three area patrol positions pos r =<pj,ph,py> , where pj∈{P1,P2,…,P jn}(jn≤pn), ph∈ {P1,P2,…,P hn}(hn≤pn), py∈{P 45 ,P2,…,P yn}(yn≤pn)pj represents the regional patrol position of the fighter formation, ph represents the regional patrol position of the bomber formation, and py represents the regional patrol position of the early warning aircraft formation.
[0121] Step 303: Calculate the range of the DQN network output value. If the patrol area selection space of the fighter formation is jn; the patrol area selection space of the bomber formation is hn; the early warning aircraft is the command platform of the red side, in order to keep it away from the firepower range of enemy fighters and ships, its patrol area is yn different areas behind the battlefield. Then the integer sequence number a output by the DQN network should be in the range of [0, an], where an = jn×hn×yn-1.
[0122] Step 304: Output a according to the integer of the DQN network i Analyze the patrol area location of the Red Army fighter jets i The formula is as follows:
[0123] pj i =a i / / (hn×yn)
[0124] Here, / / indicates integer division and taking the quotient.
[0125] Step 305: Output a according to the integer of the DQN network i Analyze the patrol area location of the Red Army bomber i The formula is as follows:
[0126] phi =(a i %(hn×yn)) / / yn
[0127] Among them, / / means taking the quotient when dividing by an integer, and % means taking the remainder when dividing by an integer.
[0128] Step 306: Output a according to the integer of the DQN network i Analyze the patrol area location of the red fighter jets i The formula is as follows:
[0129] py i =(a i %(hn×yn))%hn+(pn-yn+1)
[0130] Among them, / / means taking the quotient when dividing by an integer, and % means taking the remainder when dividing by an integer.
[0131] Fig.12 It is a schematic diagram of the non-sparse reward function design provided by the present disclosure.
[0132] Step 4: Adopt a target-oriented and global situation optimization reward value generation algorithm, design the components of the reward function for clear combat objectives, including the number of losses of different force entities, the ratio of our information detection coverage to the enemy's information detection coverage, etc., and determine the weight coefficients of each component of the reward function according to the importance of different force platforms to the final combat objectives. Provide positive and negative real-time reward feedback for each command decision of the combat simulation, so as to drive the aviation multi-platform intelligent confrontation command decision model to iteratively evolve towards a higher reward confrontation strategy during multiple rounds of simulation training. The design process of the non-sparse reward function of the scheduling layer DQN algorithm is shown in 12.
[0133] Specific steps:
[0134] Step 401: Design the components of the reward function. In a typical aviation combat scenario, in order for the Red Army to ultimately achieve its combat objectives, the design of a single-step reward function usually takes into account the advantages of fighter jets in air combat and the advantages of bombers in surface combat (for example, the combat advantages of bombers over surface ships). In addition, as the most important platform for detecting intelligence information, early warning aircraft have a huge impact on the combat effectiveness of the aviation forces of both sides. In order to enable the decisions output by the DQN network to guide the Red Army aviation force to victory as quickly as possible, a fuel loss term can be added to the reward function to make the combat duration as short as possible. In summary, the single-step reward function can be written as:
[0135] reward step =W1×(Fight_num b -Fight_num r )-W2×Bom_numr +W3×Hit_num r +W4×Awacs_num b +W5×((I r +b / (I b +b) -1)+(-L oil )
[0136] Where b is a small positive number to avoid division by zero, and its value range can be any positive number between 0 and 0.5. r Fight_num indicates the number of fighter planes damaged by the Red Army. b Indicates the number of blue fighter planes damaged, Bom_num r Indicates the number of Red bombers damaged in battle, Hit_num r Indicates the number of times the Red bomber hits the ship, Awacs_num b Indicates the number of damages suffered by the blue side’s early warning aircraft, I r Indicates the red side information detection coverage, I b Indicates the blue team's information detection coverage, L oil Represents oil loss (fixed constant).
[0137] Step 402: Determine the weight coefficients of each component of the single-step reward function according to the importance of different force platforms to the final combat goal. Assume that the ultimate goal of the Red side to win is to sink the Blue side's ships, and only bombers can achieve this combat goal. In order to achieve the final combat goal, bombers should be attacked by enemy fighters as little as possible, so that more bombers have the opportunity to launch missiles to attack Blue side ships. If necessary, our fighters can be sacrificed to protect the safety of our bombers. Therefore, in the design of the single-step reward function, the penalty weight W2 for losing a bomber should be set to a multiple of the penalty weight W1 for losing a fighter. The Blue side's early warning aircraft, as an important information detection platform in the Blue side's aviation force system, has a high military value. However, since the Blue side's ships also have a fairly large range of detection capabilities, and the Blue side's early warning aircraft is deployed behind the ship and is difficult to approach, considering the above, the reward weight W4 for destroying the Blue side's early warning aircraft can be set between 1 and 1.5 times the value of W1. In order to encourage the Red bombers to launch missiles to attack the Blue ships, a certain reward value is given for each hit on the Blue ships. Under normal circumstances, if a ship is hit by a bomber k times before it is completely destroyed, considering that the Red bombers and the Blue ships are tied, the Red bombers should hit the ship at least once to damage hk, where hk = h num / k,h numIndicates the number of Red bombers. In order to guide the Red bombers to gain an advantage in confrontation, the reward W3 given for each hit on a ship should be less than hk×W2. The information domain coverage reflects the relative advantage of the overall combat situation. If the Red side has a larger information domain coverage than the Blue side, it means that the Red side's aviation system has an advantage in information acquisition and should be given a certain reward. The value of its weight coefficient W5 should also be a multiple of W1.
[0138] Step 403: Design a settlement reward based on the termination conditions of the simulation confrontation. In order to avoid the problem of sparse rewards in reinforcement learning, the design of the reward function not only considers the single-step reward, but also needs to set a settlement reward, that is, after each round of simulation confrontation, give the reinforcement learning network a larger terminal value reward. When the simulation confrontation ends with the victory of the red side, the network is given a larger positive feedback reward. F0 , otherwise give the network a larger negative feedback reward F1 .
[0139] Step 404: Determine the weight coefficient of the settlement reward function based on the average reward value of each round of simulation confrontation. Assuming that the average simulation step length of each round is about n steps, in order to make the reward obtained by the red side in the final victory of each round F0 , when the DQN network discount factor is α, the discounted reward for the first step of each game is red w Left and right, solve the equation reward F0 ×α n =red w , you can get the final winning reward value reward F0 The specific value of . If the red team fails in each round, the final penalty reward F1 The discount penalty for the first step is red f Left and right, the same solution equation reward F1 ×α n =red f , you can get reward F1 The specific value of . So far, the complete reward function can be expressed as the following formula:
[0140]
[0141] Among them, reward is the reward function, reward F0 The reward for the red team's final victory, red w is the discounted reward for the first step of each game when the DQN network discount factor is α, reward F1 The ultimate penalty for the Red Team's failure in each round.
[0142] Fig.13It is an aviation multi-platform adversarial reinforcement learning framework based on DQN provided by the present invention.
[0143] Step 5: Based on the DQN algorithm, the state space vector of the aviation multi-platform confrontation simulation, the command decision space and the reward value generation algorithm based on global situation optimization formed in the above steps are integrated to form a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation as shown in Figure 13, and based on this framework, iterative learning and training of multi-platform joint command decision-making are carried out.
[0144] Specific steps:
[0145] Step 501: According to the specific content of the first step above, extract the current state space information S of the DQN network from the battlefield situation, where
[0146] S= <X r ,Y r ,LX r ,H r ,D_num r ,I r ,X b ,Y b ,LX b ,H b ,I b >.
[0147] Step 502: Input the acquired current situation information S into the prediction network of the DQN network to obtain the predicted value of executing different actions under the current situation.
[0148] Step 503: Select action a according to the ε-greedy strategy i , a i After action analysis, it is mapped into actions that different force platform formations should perform, and corresponding instructions are issued to different force platforms of the Red Army Aviation.
[0149] Step 504: Each force platform completes the corresponding action a i After that, the new battlefield situation is obtained, and steps 501 to 502 are repeated to obtain the state space information S_ of the next frame, and the reward R obtained after the action of the previous frame is executed is calculated according to the specific steps in the fourth step. i ,S_,R} is stored in the sample pool as a sample.
[0150] Step 505: Determine whether the conditions for training the neural network are met. Generally, the neural network is trained once every x steps. If the training conditions are not met, restart from step 501, otherwise, execute steps 506-511.
[0151] Step 506: extract n samples from the sample pool. The following steps 507-511 assume that 1 sample {S, a i ,S_,R}, explain the training principle of DQN network.
[0152] Step 507: Determine whether the target network replacement condition is met. Generally, it is assumed that the target network is replaced once every y training times. If the target network replacement condition is met, the parameters in the prediction network are directly copied to the target network.
[0153] Step 508: Input the state space S into the prediction network of the DQN network, and input the state space S_ into the target network of the DQN network, and obtain the <state-action> value table act corresponding to all executable actions of state S and S_ respectively. v0 and act v1 .
[0154] Step 509: Set the <state-action> value table act v0 Make a copy and get actc v0 , select actc v0 The action with the largest action value is a maxv , and a maxv The corresponding value is according to the formula Update to get new table acttar v0 , where Q next for act v1 All <state-action> values in the table.
[0155] Step 510: Acttar v0 and act v0 Subtract the value of the corresponding action in the table to get the loss value table of <state-action> value estimate act v0loss .
[0156] Step 511: Update the prediction network parameters using a gradient descent optimization algorithm based on the estimated loss value.
[0157] According to the technical solution provided by the present disclosure, by designing the action parsing algorithm output by the DQN, the single action number output by the DQN algorithm is mapped to the action space of multi-platform aviation equipment, thereby solving the problem that the single action number output of the DQN does not match the action space of multiple force platforms; by designing the reward function, the overall combat effect is optimized; by integrating the DQN algorithm into the aviation multi-platform intelligent confrontation simulation system, the aviation multi-platform collaborative intelligent confrontation strategy is obtained, and the goal of coordinating and commanding multi-platform force joint operations through a DQN network is achieved.
[0158] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0160] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.
Claims
1. DQN-based aviation multi-platform collaborative intelligent confrontation decision-making method, It is characterized in that include: S1 builds a command decision-making framework based on the DQN algorithm and rule set, which integrates the collaborative combat behavior model of the scheduling layer, formation layer, and single-machine layer; S2 constructs the scheduling layer state space based on the characteristics of multi-platform air combat of aviation forces and the importance level of relevant factors affecting the results of multi-platform joint combat; S3 constructs the dispatching layer behavior space based on the combat characteristics of different force formations in the aviation system; S4 designs the reward function in the DQN algorithm according to the specific combat objectives; S5 forms a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation based on the DQN algorithm, and performs iterative learning training for multi-platform joint command decision-making based on the reinforcement learning framework to obtain a multi-platform collaborative combat strategy; Said S1 comprises: Based on the DQN algorithm, the scheduling layer behavior space is constructed; Based on the rule set, construct the combat behavior models of the formation layer and the single-machine layer respectively; Based on the scheduling layer behavior space, the formation layer combat behavior model and the single-machine layer combat behavior model, a collaborative combat behavior model is constructed; The S2 comprises: According to the characteristics of multi-platform air combat of aviation forces and the importance level of relevant factors affecting the results of multi-platform joint operations, key elements are extracted to construct the state space vector of the scheduling layer DQN algorithm. Among them, the key elements include the coordinate positions of the enemy and our aircraft, the types of enemy and our forces, the headings of the enemy and our aircraft, the amount of our bombs carried, and the coverage ratio of the enemy and our detection information domain; Based on the state space vector, the battlefield situation of the aviation multi-platform joint confrontation is mathematically represented to construct the scheduling layer state space; The S3 includes: Based on the operational characteristics of different force formations in the aviation system, a set of executable tasks for different force platform formations is formed, and based on the executable task set, a scheduling layer behavior space for aviation multi-platform equipment is constructed; The S3 further includes: Design an action parsing algorithm for the DQN output, and based on the action parsing algorithm for the DQN output, establish a mapping relationship between a single action sequence number output by the DQN algorithm and an action space of multi-platform aviation equipment.
2. The aviation multi-platform collaborative intelligent confrontation decision-making method according to claim 1, It is characterized in that The S4 comprises: Design the components of the reward function according to the specific combat objectives, where the components include the number of losses of different force entities and the ratio of our information detection range to the enemy's information detection range; Based on the components of the reward function, and according to the importance of different force platforms to the final combat goal, the weight coefficients of the components of the reward function are determined; A reward function is constructed based on the component elements and the weight coefficients.
3. The aviation multi-platform collaborative intelligent confrontation decision-making method according to claim 1, It is characterized in that The reward function consists of two parts: a reward value and a penalty value.
4. The aviation multi-platform collaborative intelligent confrontation decision-making method according to claim 1, It is characterized in that The S5 comprises: Based on the DQN algorithm, the scheduling layer state space, the scheduling layer behavior space and the reward function are integrated to construct a reinforcement learning framework for aviation multi-platform joint combat simulation confrontation; Based on the framework, iterative learning and training of multi-platform joint command decision-making is carried out to obtain the aviation multi-platform collaborative intelligent confrontation strategy.
Citation Information
Patent Citations
Fight simulation system generating military strength based on computer and simulation method
CN106682351A
Aircraft soldier system intelligent behavior modeling method based on global situation information
CN112560332A
A virtual combat simulation system for combat effectiveness analysis of weapon system and method thereof
KR1020140145446A
