Intelligent decision-making method for air defense combat based on potential game and deep reinforcement learning

Abstract

The application discloses an air defense combat intelligent decision-making method based on potential game and deep reinforcement learning, and belongs to the field of data security. The specific steps of the decision-making method are as follows: (1) collecting data required for air defense combat and performing pretreatment; (2) extracting air situation features through a Bi-GRU network; (3) modeling potential game according to an air defense combat environment and participants; and (4) establishing a Markov process model to make an optimized decision. The application can improve the rationality of target distribution, realize efficient distribution, save fire resources, improve the exploration efficiency of the environment, improve the utilization rate of computing resources, realize a better strategy, and thus achieve a higher winning rate.

Description

Technical Field

[0001] This invention relates to the field of data security, and in particular to an intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning. Background Technology

[0002] Modern warfare is high-tech warfare, and high-tech weaponry is a crucial factor in determining victory or defeat. Decision-making is the core of combat, encompassing the efficient use of multiple types and platforms of weaponry based on enemy information, the rational allocation of incoming targets, and the successful counterattack. With the development of unmanned aerial vehicle (UAV) swarms and other air-attack weapons, the battlefield situation in air defense operations has become increasingly complex and volatile, posing new challenges to operational decision-making systems. A schematic diagram of the air defense combat process is shown below. Figure 1 As shown, in air defense operations, efficient decision-making improves combat effectiveness by more than three times compared to free decision-making. Rational allocation of combat resources and enhanced interception efficiency have become urgent problems for combat decision-making systems; therefore, inventing intelligent decision-making methods for air defense operations based on potential game theory and deep reinforcement learning has become particularly important.

[0003] Existing intelligent decision-making methods for air defense operations suffer from low target allocation rationality, inefficient target allocation, serious waste of firepower resources, and low efficiency in environmental exploration. To address this, we propose an intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning. The specific steps of this decision-making method are as follows:

[0007] (1) Collect and preprocess the data required for air defense operations;

[0008] (2) Extracting air situation features using a Bi-GRU network;

[0009] (3) Based on the air defense combat environment and the participants, conduct a game-theoretic modeling;

[0010] (4) Establish Markov process model to formulate optimal decision.

[0011] As a further aspect of the present invention, the air defense combat data required in step (1) specifically includes status data and action space;

[0012] The status data is specifically divided into the status of our defended key locations, the status of our firepower units, the status of detected enemy targets, and the status of attackable enemy targets.

[0013] The status of our protected key locations includes the key location number, location, type, and under attack status; the status of our fire unit includes the fire unit target number, location, number of remaining missiles, availability of remaining missiles, target number that this unit can attack, and under attack status; the status of detected enemy targets includes the target number, location, type, movement status, and under attack status.

[0014] The action space specifically includes target selection, target threat level, radar selection, and radar actions.

[0015] As a further aspect of the present invention, the data preprocessing steps required for air defense operations in step (1) are as follows:

[0016] Step 1: After classifying the collected data required for air defense operations, standardize each group of data. Based on the processed data, construct a vocabulary and assign a unique index to each different word.

[0017] Step 2: Convert the data required for air defense operations into a serialized representation, segment each sentence into words, map each word to an index in the vocabulary, and pad or truncate the sequence to make them all the same length.

[0018] As a further aspect of the present invention, the specific steps for Bi-GRU network feature extraction in step (2) are as follows:

[0019] Step 1: Map the serialized air defense combat data into a fixed-dimensional vector, then construct a Bi-GRU network using a set of forward GRUs and a set of reverse GRUs, add multiple attention heads to the Bi-GRU network, and determine the Bi-GRU network parameter information.

[0020] Step 2: Concatenate multiple self-attention layers to obtain the corresponding multi-head attention mechanism, and perform linear transformation on the same input from different angles through multiple attention layers to extract important features from the data required for air defense operations. Then, fuse the outputs of multiple attention heads and input the fused feature vector into subsequent layers.

[0021] As a further aspect of the present invention, the specific calculation formula for the linear transformation in step ② is as follows:

[0022] P = W p x,P∈(Q,K,V) (1)

[0024] Among them, W pThe linear transformation matrix representing the corresponding vector;

[0025] The specific calculation formula for the multi-head attention mechanism is as follows:

[0026]

[0027] MultiHead(Q,K,V)=Concat(head1,…,head h W O (3)

[0029] wherehead = Attention(QW i Q ,KW i K VW i V ) (4)

[0031] Among them, W O W i Q W i K and W i V represents the learnable parameter matrix in the data projection, and h represents the number of attention heads.

[0032] As a further aspect of the present invention, the specific steps of the potential game modeling in step (3) are as follows:

[0033] Step 1: Collect the targets to be intercepted and the fire units whose guidance radars are already activated, as provided by DRL-TSRC, and define the target set N. t = {1,2,…,T}, where the elements are the targets to be intercepted. The fire units whose guidance radars are already activated in air defense operations are considered as stakeholders, and the set of stakeholders N is defined. m ={LM1,…,LM s ,SM1,…SM k}, where the set of long-range fire units is LM = {LM1, ..., LM}. s The set of close-range firepower units is SM = {SM1, ... SM}. k};

[0034] Step II: Collect the strategy set of each player and define the shooting strategy of each player. At the same time, confirm the constraints of the strategy by the number of long-range fire units and short-range fire units. Then, design the penalty function based on minimizing the interception cost.

[0035] Step 3: Add the penalty function to each player to obtain the corresponding utility function, then set the maximum waiting time and iterate. Each group of players randomly generates a waiting time within the maximum waiting time and starts timing.

[0036] Step IV: If the player receives a DR signal from another player during the waiting time before the iteration begins, the timer stops and the player abandons the nomination of the decision-maker; otherwise, the player sends a DR signal to other players and is determined to be the decision-maker for this round.

[0037] Step V: Initialize the target allocation game, determine the number of fire units and the number of targets respectively, then randomly initialize the strategy combination and the number of iterations, treat the decision-maker as a player who changes the strategy in the iteration, obtain the available strategy set of random fire units and the strategy of the fire units, and calculate their strategy utility function.

[0038] Step VI: Select any set of update strategies from the firepower unit's strategies, temporarily update the strategies, and recalculate their utility functions. If the utility function is better than the original utility function, then update the strategy combination.

[0039] As a further aspect of the present invention, the player-led shooting strategy described in step II is specifically represented as follows:

[0040]

[0041] Where, if target j satisfies the interception condition of player i, it is denoted as C. ij =1, otherwise it is written as C ij =0;

[0042] The specific formula for calculating the utility function mentioned in step III is as follows:

[0043]

[0044]

[0045]

[0046] Where, α j The threat level of target j is represented by t, the target type is represented by f. t i J represents the reward value obtained by fire unit i for intercepting 1 target of type t. i ={i′|i′∈N m C ij =1,C i′j =1, i′≠i} represents the set of neighboring fire units of i. This represents the shortcut between fire unit i and target j. β represents the maximum shortcut value of the flight path from target j to the fire unit that can intercept the target, and β represents the penalty coefficient.

[0047] As a further aspect of the present invention, the specific steps for formulating the optimization decision in step (4) are as follows;

[0048] Step 1: Construct a Markov process model based on the collected data sets, and collect the state set S = [s1, s2, ..., s...]. n And the action set A = [a1, a2, ..., a n Then the agent uses the policy π: S×A → [0, 1] in state s i Select action a from the action set i Meanwhile, the battlefield environment transitions to the next state according to the state transition function P:S×A×S→[0,1];

[0049] Step 2: Calculate the transition probability between each pair of states using the state transition matrix, then define the state distribution probability at the initial time, and calculate the action 'a' taken by the agent. i The expected reward after the transition, and the strategy π in state s. i The expected cumulative reward at that time is then used to obtain a larger expected reward function through the policy gradient method;

[0050] The third step is to assign different reward values ​​to different types of fire units. Based on the type of fire unit, different rewards are given for radar being attacked, missile launch, and target interception. Then, the constructed Markov model is verified to see if it conforms to the characteristics and behavior of the actual system. Based on actual data and observation results, the model is adjusted and optimized to improve its accuracy and adaptability. Finally, the Markov model is used for decision-making.

[0051] As a further aspect of the present invention, the specific formula for calculating the transition probability in the second step is as follows:

[0052] P ss′ =P[s t+1 =s′|s t =s] (9)

[0054] Where s represents the current state and s′ represents the subsequent state;

[0055] The specific formula for calculating the expected cumulative reward mentioned in step two is as follows:

[0056] V π (s)=E[R t |s t =s,π] (10)

[0058] π(a|s)=P[a t =a|s t =s] (11)

[0060] Q π (s,a)=E[R t |s t =s,a t =a,π] (12)

[0062] Among them, V π (s) represents the state-value function, E represents the expectation operator, and Q represents the state-value function. π (s,a) represents the action state value function.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] This intelligent decision-making method for air defense operations, based on game theory and deep reinforcement learning, preprocesses the collected data required for air defense operations into a unified format, maps it to fixed-dimensional vectors, and adds a multi-head attention mechanism to the Bi-GRU network. Then, multiple attention layers perform linear transformations on the same input from different perspectives to extract important features from the data required for air defense operations. It collects information on targets to be intercepted and fire units whose guidance radars are already activated. A penalty function is designed based on minimizing interception costs and added to each player to obtain the corresponding utility function. A maximum waiting time is then set, and iterations are performed to determine the decision-maker. The target allocation game is initialized, and the number of fire units and targets are determined respectively. After randomly initializing the strategy combination and iteration count to update the strategy combination, a Markov process model is constructed, and different reward values ​​are given to different types of fire units. Different rewards are given for radar being attacked, missile launch, and target interception based on the type of fire unit. The constructed Markov model is then verified to ensure that it conforms to the characteristics and behavior of the actual system. Based on actual data and observation results, the model is adjusted and optimized to improve its accuracy and adaptability. Using the Markov model for decision-making can improve the rationality of target allocation, achieve efficient allocation, save fire resources, improve the efficiency of environmental exploration, and increase the utilization rate of computing resources. At the same time, it can realize better strategies, thereby achieving a higher win rate. Attached Figure Description

[0065] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0066] Figure 1 This is a flowchart of the intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning proposed in this invention. Detailed Implementation

[0067] Reference Figure 1 A smart decision-making method for air defense operations based on potential game theory and deep reinforcement learning is proposed. The specific steps of this decision-making method are as follows:

[0068] Collect and preprocess the data required for air defense operations.

[0069] Specifically, after the collected data required for air defense operations are classified, each group of data is standardized. Based on the processed data, a vocabulary is constructed, each different word is assigned a unique index, the data required for air defense operations is converted into a serialized representation, each sentence is segmented into words, and each word is mapped to an index in the vocabulary. At the same time, the sequences are padded or truncated to make them have the same length.

[0070] It should be further explained that the data required for air defense operations specifically includes status data and operational space;

[0071] The status data is specifically divided into the status of our defended key locations, the status of our firepower units, the status of detected enemy targets, and the status of attackable enemy targets; the action space specifically includes target selection, target threat level, radar selection, and radar actions.

[0072] The status of our protected key locations includes the key location number, location, type, and under attack status; the status of our fire unit includes the fire unit target number, location, number of remaining missiles, availability of remaining missiles, target number that this unit can attack, and under attack status; the status of detected enemy targets includes the target number, location, type, movement status, and under attack status.

[0073] Air situation features were extracted using a Bi-GRU network.

[0074] Specifically, the serialized data required for air defense operations is mapped into a fixed-dimensional vector. Then, a Bi-GRU network is constructed using a set of forward GRUs and a set of backward GRUs. Multiple attention heads are added to the Bi-GRU network, and the parameters of the Bi-GRU network are determined. Multiple self-attention layers are concatenated to obtain the corresponding multi-head attention mechanism. The same input is linearly transformed from different angles through multiple attention layers to extract important features from the data required for air defense operations. The outputs of multiple attention heads are fused, and the fused feature vector is input into subsequent layers.

[0075] It should be further explained that the specific calculation formula for the linear transformation is as follows:

[0076] P = W p x,P∈(Q,K,V) (1)

[0078] Among them, W p The linear transformation matrix representing the corresponding vector;

[0079] The specific calculation formula for the multi-head attention mechanism is as follows:

[0080]

[0081] MultiHead(Q,K,V)=Concat(head1,…,head h W O (3)

[0083] wherehead = Attention(QW i Q ,KW i K VW i V ) (4)

[0085] Among them, W O W i Q W i K and W i V represents the learnable parameter matrix in the data projection, and h represents the number of attention heads.

[0086] Based on the air defense combat environment and the participants, a game-theoretic model is constructed.

[0087] Specifically, the system collects the targets to be intercepted and the fire units whose guidance radars are already activated, as provided by DRL-TSRC, and defines the target set N. t = {1,2,…,T}, where the elements are the targets to be intercepted. The fire units whose guidance radars are already activated in air defense operations are considered as stakeholders, and the set of stakeholders N is defined. m ={LM1,…,LM s ,SM1,…SM k}, where the set of long-range fire units is LM = {LM1, ..., LM}. s The set of close-range firepower units is SM = {SM1, ... SM}. kThe process involves collecting the strategy sets of each player and defining their firing strategies. The constraints of these strategies are confirmed by the number of long-range and short-range fire units. A penalty function is then designed based on minimizing interception costs and applied to each player to obtain their corresponding utility function. A maximum waiting time is then set, and the process iterates. Each group of players randomly generates a waiting time within the maximum waiting time and begins timing. If a player receives a DR signal from another player during the waiting time before the iteration begins, the timing stops, and the player relinquishes their decision-maker nomination. Otherwise, the player sends a DR signal to the other players and is identified as the decision-maker for this round. The target allocation game is initialized, and the number of fire units and the number of targets are determined respectively. Then, the strategy combination and the number of iterations are randomly initialized. The decision-maker is regarded as the player who changes the strategy in the iteration. The available strategy set of random fire units and the strategy of the fire units are obtained, and their strategy utility function is calculated. A set of update strategies is randomly selected from the fire unit strategies, the strategy is temporarily updated, and its utility function is recalculated. If the utility function is better than the original utility function, the strategy combination is updated.

[0088] In addition, it should be noted that the specific form of the player's shooting strategy is as follows:

[0089]

[0090] Where, if target j satisfies the interception condition of player i, it is denoted as C. ij =1, otherwise it is written as C ij =0;

[0091] The specific formula for calculating the utility function is as follows:

[0092]

[0093]

[0094]

[0095] Where, α j The threat level of target j is represented by t, the target type is represented by f. t i J represents the reward value obtained by fire unit i for intercepting 1 target of type t. i ={i′|i′∈N m C ij =1,C i′j =1, i′≠i} represents the set of neighboring fire units of i. This represents the shortcut between fire unit i and target j. β represents the maximum shortcut value of the flight path from target j to the fire unit that can intercept the target, and β represents the penalty coefficient.

[0096] Establish a Markov process model to formulate optimal decisions.

[0097] Specifically, a Markov process model is constructed based on the collected data sets, and a state set S = [s1, s2, ..., s...] is collected. n And the action set A = [a1, a2, ..., a n Then the agent uses the policy π: S×A → [0, 1] in state s i Select action a from the action set i Simultaneously, the battlefield environment transitions to the next state according to the state transition function P: S×A×S→[0,1]. The transition probability between each pair of states is calculated using the state transition matrix. Then, the state distribution probability at the initial moment is defined, and the action a taken by the agent is calculated. i The expected reward after the transition, and the strategy π in state s. i The expected cumulative reward is calculated, and then a larger expected reward function is obtained through the policy gradient method. Different reward values ​​are given for different types of fire units. Different rewards are given for radar being attacked, missile launching, and target interception based on the type of fire unit. Then, it is verified whether the constructed Markov model conforms to the characteristics and behavior of the actual system. Based on actual data and observation results, the model is adjusted and optimized to improve its accuracy and adaptability. Finally, the Markov model is used for decision-making.

[0098] It should be further explained that the specific formula for calculating the transition probability is as follows:

[0099] P ss′ =P[s t+1 =s′|s t =s] (9)

[0101] Where s represents the current state and s′ represents the subsequent state;

[0102] The specific formula for calculating the expected cumulative reward is as follows:

[0103] V π (s)=E[R t |s t =s,π] (10)

[0105] π(a|s)=P[a t =a|s t =s] (11)

[0107] Q π (s,a)=E[R t |s t=s,a t =a,π] (12)

[0109] Among them, V π (s) represents the state-value function, E represents the expectation operator, and Q represents the state-value function. π (s,a) represents the action state value function.

Claims

1. An intelligent decision-making method for air defense combat based on potential game and deep reinforcement learning, characterized in that, The specific steps of this decision-making method are as follows: (1) Collect and preprocess the data required for air defense operations; (2) Extract air situation features using a Bi-GRU network; (3) Based on the air defense combat environment and the participants, conduct a game-theoretic model; (4) Establish a Markov process model to formulate optimal decisions; The specific steps for potential game modeling described in step (3) are as follows: Step I: Collect the targets to be intercepted given by DRL-TSRC and the fire units whose guidance radars have been turned on, and define the target set where element is the target to be intercepted, the fire units whose guidance radars have been turned on in air defense combat are regarded as players, and the player set is defined as where the set of long-range fire units is , and the set of short-range fire units is ; Step II: Collect the strategy set of each player and define the shooting strategy of each player. At the same time, confirm the constraints of the strategy by the number of long-range fire units and short-range fire units. Then, design the penalty function based on minimizing the interception cost. Step 3: Add the penalty function to each player to obtain the corresponding utility function, then set the maximum waiting time and iterate. Each group of players randomly generates a waiting time within the maximum waiting time and starts timing. Step IV: If the player receives a DR signal from another player during the waiting time before the iteration begins, the timer stops and the player abandons the nomination of the decision-maker; otherwise, the player sends a DR signal to other players and is determined to be the decision-maker for this round. Step V: Initialize the target allocation game, determine the number of fire units and the number of targets respectively, then randomly initialize the strategy combination and the number of iterations, treat the decision-maker as a player who changes the strategy in the iteration, obtain the available strategy set of random fire units and the strategy of the fire units, and calculate their strategy utility function. Step VI: Select any set of update strategies from the fire unit's strategies, temporarily update the strategies, and recalculate their utility functions. If the utility function is better than the original utility function, then update the strategy combination. The specific form of the player-led shooting strategy described in step II is as follows: （5） Wherein, if the target satisfies the interceptor interception condition, denoted as = 1, otherwise denoted as = 0; The specific formula for calculating the utility function mentioned in step III is as follows: （6） （7） （8） wherein, representing the target threat degree, representing the target type, representing the fire unit intercept 1 aircraft / missile reward value obtained by the type target, representing the set of adjacent fire units, representing the fire unit with the target the route shortcut, representing the target to the maximum value of the route shortcut among the fire units that can intercept the target, representing the penalty coefficient.

2. The intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning according to claim 1, characterized in that, The data required for air defense operations mentioned in step (1) specifically includes status data and action space; The status data is specifically divided into the status of our defended key locations, the status of our firepower units, the status of detected enemy targets, and the status of attackable enemy targets. The status of our protected key locations includes the key location number, location, type, and under attack status; the status of our fire unit includes the fire unit target number, location, number of remaining missiles, availability of remaining missiles, target number that this unit can attack, and under attack status; the status of detected enemy targets includes the target number, location, type, movement status, and under attack status. The action space specifically includes target selection, target threat level, radar selection, and radar actions.

3. The intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning according to claim 2, characterized in that, The data preprocessing steps required for air defense operations described in step (1) are as follows: Step 1: After classifying the collected data required for air defense operations, standardize each group of data. Based on the processed data, construct a vocabulary and assign a unique index to each different word. Step 2: Convert the data required for air defense operations into a serialized representation, segment each sentence into words, map each word to an index in the vocabulary, and pad or truncate the sequence to make them all the same length.

4. The intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning according to claim 3, characterized in that, The specific steps for Bi-GRU network feature extraction in step (2) are as follows: Step 1: Map the serialized air defense combat data into a fixed-dimensional vector, then construct a Bi-GRU network using a set of forward GRUs and a set of reverse GRUs, add multiple attention heads to the Bi-GRU network, and determine the Bi-GRU network parameter information. Step 2: Concatenate multiple self-attention layers to obtain the corresponding multi-head attention mechanism, and perform linear transformation on the same input from different angles through multiple attention layers to extract important features from the data required for air defense operations. Then, fuse the outputs of multiple attention heads and input the fused feature vector into subsequent layers.

5. The intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning according to claim 4, characterized in that, The specific calculation formula for the linear transformation mentioned in step ② is as follows: （1） in, The linear transformation matrix representing the corresponding vector; The specific calculation formula for the multi-head attention mechanism is as follows: （2） （3） （4） in, , , as well as The matrix representing the learnable parameters in the data projection. This represents the number of attention heads.

6. The intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning according to claim 1, characterized in that, The specific steps for formulating the optimization decision in step (4) are as follows; Step 1: Construct a Markov process model based on the collected data sets and collect the state sets. and action set Then the agent uses a policy In state Selecting actions from the action set Meanwhile, the battlefield environment follows the state transition function. Transition to the next state; Step 2: Calculate the transition probability between each pair of states using the state transition matrix, then define the state distribution probability at the initial time, and calculate the action to be taken by the agent. The expected reward after the transfer, and the strategy is calculated simultaneously. In state The expected cumulative reward at that time is then used to obtain a larger expected reward function through the policy gradient method; The third step is to assign different reward values ​​to different types of fire units. Based on the type of fire unit, different rewards are given for radar being attacked, missile launch, and target interception. Then, the constructed Markov model is verified to see if it conforms to the characteristics and behavior of the actual system. Based on actual data and observation results, the model is adjusted and optimized to improve its accuracy and adaptability. Finally, the Markov model is used for decision-making.

7. The intelligent decision-making method for air defense operations based on potential game theory and deep reinforcement learning according to claim 6, characterized in that, The specific formula for calculating the transition probability in the second step is as follows: P ss′ = P[s t+1 = s' | s t = s] (9) in, Represents the current state. Represents the subsequent state; The specific formula for calculating the expected cumulative reward mentioned in step two is as follows: （10） （11） （12） in, Represents the state value function. Represents the expectation operator. This represents the action state value function.

Images