Equipment dynamic confrontation game decision-making method and device based on knowledge graph

Abstract

The invention provides an equipment dynamic confrontation game decision-making method and device based on a knowledge graph, belongs to the technical field of behavior decision-making, and aims to solve the problem of lack of effectiveness and high efficiency in dynamic game strategy selection in the prior art. Firstly, a target is recognized through a knowledge graph, and the end point of an inference strategy is determined; and then, converting the game strategy reasoning path into a sequence decision problem by using a Markov decision process model, and solving the problem through an Actor-Critic deep reinforcement learning method. And finally, obtaining an optimal game strategy through path search and multi-path synthesis. According to the invention, the efficiency and efficiency of equipment confrontation decision making are improved, and the method has important application value.

Description

Technical Field

[0001] The present invention belongs to the field of decision-making technology and relates to a knowledge graph-based equipment dynamic confrontation game decision-making method and device. Background Art

[0002] Solving the confrontation game strategy between equipment is one of the key technologies for equipment confrontation decision-making. Its core lies in how to select the optimal confrontation strategy based on the environment and the dynamic performance characteristics of the equipment of both sides.

[0003] At present, the main problems in solving dynamic game strategies are: the existing game decision-making methods often consider the single performance characteristics of equipment, lack of systematic analysis of the interaction and correlation between the equipment of both parties, lack of comprehensiveness. The existing methods are difficult to effectively describe the dynamics of the game process between the confrontation equipment and the target, and difficult to adapt to the ever-changing environment.

[0004] In order to overcome the above shortcomings, the present invention proposes a game decision method and device for confrontation equipment and targets based on knowledge graph. By constructing a dynamic knowledge graph of confrontation experience, deep reinforcement learning is used to solve the confrontation game strategy path, and then the swarm intelligence algorithm is used to select the optimal strategy combination to achieve the optimal selection of confrontation game strategy. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method and device for equipment dynamic confrontation game decision-making based on knowledge graph. According to the interaction and correlation between equipment confrontation, the present invention integrates multi-dimensional game strategy characteristics and their associated game strategies and experience to construct a game confrontation strategy knowledge graph. On this basis, the confrontation game strategy path is converted into a confrontation game strategy action sequence. By constructing a Markov decision process model for dynamic path decision-making, the game decision path action value function is obtained, and deep reinforcement learning is used to solve the game strategy path. Finally, the evidence theory and swarm intelligence algorithm are integrated to determine the optimal confrontation game strategy.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The equipment dynamic confrontation game decision-making method based on knowledge graph includes the following steps:

[0008] S1: Determine the end point of the reasoning strategy based on the knowledge graph, including:

[0009] Using the knowledge of target attribute characteristics in the knowledge graph and the various technical parameters of the target obtained and analyzed in real time, the following reasoning analysis is performed from bottom to top based on knowledge-driven:

[0010] Taking various attribute parameters, real-time parameters and parameter change characteristics that reflect target characteristics as underlying driving data, extract the similarity, change rate and characteristic value of feature points of target type and state characteristics;

[0011] Further through the feature value analysis, the feature facts are obtained and matched with the target feature knowledge given by the knowledge graph; the matching result is the target recognition result of the target type and working mode, that is, the end point of the adversarial game strategy path;

[0012] S2: Construct a Markov decision process model (MDP) for adversarial game strategy path search, including:

[0013] Transform the strategy path solution of adversarial game into a sequential decision problem;

[0014] Construct a Markov decision process model (S, A, P, R) for adversarial game strategy reasoning path search, where:

[0015] S: represents the state, which is defined as the set of nodes on the adversarial game strategy path obtained from the knowledge graph, that is, the set of adversarial game operations;

[0016] A: indicates the action, that is, whether each node in the knowledge graph is selected;

[0017] P: represents the transition probability, which is the probability that the state changes to the next state S' after executing action A in the current state S;

[0018] R: represents the reward function R(S,A), which is the immediate reward obtained after selecting action A at the current state S, that is, the corresponding node on the adversarial game strategy path, that is, the evaluation of the adversarial effect after selecting a certain adversarial strategy operation;

[0019] Through the above Markov decision process model, the strategy path node selection action value function Q(A,S) is obtained;

[0020] S3: Based on the Actor-Critic deep reinforcement learning method, solve the action sequence that maximizes the value of Q(A,S). This sequence corresponds to the optimal adversarial game strategy reasoning path in the adversarial experience knowledge graph, including:

[0021] The reasoning decision network training is completed by the interaction and update between the Actor network and the Critic network and the environment; the environment is a confrontational game situation;

[0022] The optimal path node sequence decision depends on the strategic path solution obtained by using the environment and state as the input of the Actor network; where the state is the current path selection;

[0023] S4: Dynamic knowledge graph multi-adversarial game strategy path comprehensive strategy, including:

[0024] Delete nodes from the acquired strategy path, and then use the above method to complete the discovery and search of other strategy paths to obtain multiple adversarial game strategy paths;

[0025] Based on the adversarial game strategy path given by the adversarial game strategy knowledge graph and its adversarial game effect evaluation, the optimal adversarial game strategy fusion method is used to combine multiple adversarial interference strategy paths to obtain the optimal strategy for the target adversarial game.

[0026] Furthermore, S4 is obtained by fusing the strategies corresponding to multiple adversarial game strategy paths; firstly, based on the evidence theory method, multiple evaluation indicators describing the effectiveness of the fusion strategy are used as evidence focal elements to obtain the basic probability assignment of the strategy, and the effectiveness of the fusion strategy is calculated using the strategy combination rule method of evidence theory; then, the swarm optimization algorithm is used, combined with the effectiveness of the fusion strategy, to obtain the best strategy solution for the adversarial game to achieve strategy fusion.

[0027] A knowledge graph-based equipment dynamic confrontation game decision-making device, the device comprising:

[0028] The knowledge graph reasoning module is used to determine the adversarial strategy endpoint and strategy reasoning solution path based on the knowledge graph;

[0029] Markov decision process model building module, used to build the Markov decision process model for strategy path search;

[0030] Deep reinforcement learning module, using Actor-Critic based deep reinforcement learning method to solve dynamic confrontation game strategy path;

[0031] Multi-path confrontation game strategy synthesis module, used to synthesize multiple dynamic game decision strategies based on knowledge graph;

[0032] The output of the knowledge graph reasoning module, i.e., the target recognition result and the adversarial game reasoning path node, is directly connected to the input of the adversarial game-oriented Markov decision process model construction module of the present invention as the basis for state definition;

[0033] The output of the Markov decision process model building module, i.e., the path node selection action value function Q(A,S), is connected to the input of the deep reinforcement learning module to solve the adversarial game reasoning strategy;

[0034] The output of the deep reinforcement learning module, i.e., the optimal adversarial game strategy reasoning path, is connected to the input of the multi-path reasoning strategy synthesis module as the basis for strategy fusion;

[0035] The output of the multi-adversarial game strategy path synthesis module, namely the optimal target adversarial game strategy, is output as the final decision result.

[0036] Furthermore, the knowledge graph reasoning module includes:

[0037] Target recognition module, used to identify target types and working modes based on knowledge graphs;

[0038] A feature extraction module is used to extract the feature values ​​of the similarity, change rate and feature points that reflect the target type and state characteristics;

[0039] The matching module is used to match the feature facts with the target feature knowledge given by the knowledge graph;

[0040] A search module, used to search and obtain target recognition results of target type and working mode based on matching results;

[0041] The target recognition module is responsible for receiving target data and identifying the target using the target attribute feature knowledge stored in the knowledge graph; its output is the identified target type and working mode;

[0042] The feature extraction module receives the output from the target recognition module, i.e., the target type and working mode, and extracts feature values ​​such as similarity, change rate, and feature points reflecting the target type and state characteristics in combination with the target technical parameters acquired in real time;

[0043] The matching module receives the output from the feature extraction module, i.e., the feature value, and matches it with the target feature knowledge stored in the knowledge graph to determine the matching degree between the target feature and the target knowledge in the knowledge graph;

[0044] The search module receives the output from the matching module, i.e., the matching result, and searches the knowledge graph to obtain a structure that matches the target feature, and finally outputs the target recognition results such as the target type and working mode;

[0045] The output of the state definition module, i.e., the state space, is used as the input of the action definition module and the transition probability calculation module to define the action space and calculate the transition probability;

[0046] The output of the action definition module, i.e., the action space, is used to select appropriate nodes from the knowledge graph module as candidate adversarial game strategy path nodes, and is used as the input of the transition probability calculation module to calculate the transition probability;

[0047] The output of the transition probability calculation module, i.e., the transition probability matrix, together with the output of the state definition module, is used as the input of the reward function definition module to define the reward function;

[0048] The output of the reward function definition module, namely the reward function, is used to evaluate the rewards obtained after performing actions in each state and is an important part of the Markov decision process model.

[0049] Furthermore, the Markov decision process model building module includes:

[0050] A state definition module, used to define each node on the adversarial game strategy path, wherein the node describes an alternative adversarial game strategy;

[0051] An action definition module is used to define whether each node in the knowledge graph is selected. The action essentially corresponds to the selection of a certain interference countermeasure operation;

[0052] The transition probability calculation module is used to calculate the probability that the environment state will change to the next state after performing an action in the current state;

[0053] The reward function definition module is used to define the reward function R(S,A), where S represents the current state and A represents the execution action. The reward function is a scalar function of the current state and action, which represents the immediate reward obtained after selecting the execution action in the current state. The reward can essentially be regarded as an evaluation of the interference effect after the action selects a certain interference countermeasure operation.

[0054] Furthermore, the deep reinforcement learning module includes:

[0055] Actor network, used to generate adversarial game strategy reasoning paths;

[0056] Critic network, used to evaluate the value of the adversarial game strategy reasoning path;

[0057] The interactive update module is used to interactively update the Actor network and the Critic network with the environment, where the environment represents the knowledge graph;

[0058] The output of the Actor network, i.e., the adversarial game strategy reasoning path, is connected to the input of the Critic network to evaluate the path value;

[0059] The output of the Critic network, i.e. the path value evaluation result, is connected to the input of the Actor network and used to update the adversarial game strategy reasoning path generation strategy;

[0060] The outputs of the Actor network and the Critic network are connected to the input of the interactive update module, which is used to apply the adversarial game strategy reasoning path to the environment and obtain feedback information;

[0061] The output of the interactive update module, i.e., the environment state change and the reward information obtained, is connected to the input of the Actor network and the Critic network to update their parameters.

[0062] Furthermore, the multi-path confrontation game strategy integration module includes:

[0063] A path search module is used to delete nodes in the acquired adversarial game strategy path, and then use the above method to complete the adversarial game strategy path discovery search to obtain multiple adversarial game strategy paths;

[0064] The strategy fusion module is used to use the adversarial game strategy path and its adversarial game effect evaluation given by the adversarial game strategy knowledge graph, realize the integration of multiple adversarial game strategy paths through strategy comprehensive optimization, and obtain the best target adversarial game strategy;

[0065] The output of the path search module, i.e., multiple adversarial game strategy paths, is connected to the input of the strategy fusion module as the basis for strategy fusion;

[0066] The output of the strategy fusion module, i.e., the optimal target adversarial game strategy, is output as the final decision result.

[0067] The beneficial effects of the present invention are:

[0068] (1) By constructing a dynamic knowledge graph of confrontation experience, the present invention can comprehensively analyze the interactions and correlations between confrontation equipment and targets, and integrate multi-dimensional game strategy characteristics, thereby achieving a more systematic and scientific strategy selection.

[0069] (2) The present invention uses deep reinforcement learning to solve the adversarial game strategy, which can effectively describe the dynamics and complexity of the game process between the adversarial equipment and the target, and adapt to the ever-changing environment.

[0070] (3) The present invention utilizes the advantages of deep reinforcement learning to quickly find the optimal strategy, thereby improving the efficiency of strategy reasoning and meeting the needs of real-time decision-making.

[0071] (4) The present invention can help the countermeasure equipment select the optimal countermeasure strategy, thereby giving full play to the effectiveness of the countermeasure equipment and enhancing its advantages.

[0072] (5) The present invention can automatically generate countermeasure strategies, reduce the operating difficulty of countermeasure equipment, and improve efficiency.

[0073] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0075] Figure 1 Identify target types and working mode states based on knowledge graphs;

[0076] Figure 2 Deep reinforcement learning path discovery for adversarial empirical knowledge graphs;

[0077] Figure 3 MDP model for strategy path discovery in adversarial games on dynamic knowledge graphs;

[0078] Figure 4 Solve the reasoning path of adversarial game strategy based on Actor-Critic deep reinforcement learning;

[0079] Figure 5 It is a multi-path strategy synthesis based on adversarial experience knowledge graph;

[0080] Figure 6 Identify implementation examples for target types and working modes based on knowledge graphs;

[0081] Figure 7 An MDP model implementation for dynamic knowledge graph adversarial game strategy path discovery;

[0082] Figure 8 A comprehensive implementation example of a multi-path strategy based on adversarial experience knowledge graph;

[0083] Fig. 9 A multi-path strategy synthesis based on adversarial empirical knowledge graph. DETAILED DESCRIPTION

[0084] The following describes the implementation of the present invention by using a specific example of a radar as a specific target. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0085] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0086] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0087] The present invention is aimed at equipment anti-game applications. The present invention uses knowledge graphs and actual anti-game data to determine the anti-game strategy endpoint. On this basis, the anti-game strategy path is converted into an anti-game strategy action sequence. By constructing a Markov decision process model for dynamic path decision-making, the game decision path action value function is obtained, and deep reinforcement learning is used to solve the game strategy path. Finally, the evidence theory and swarm intelligence algorithm are combined to determine the optimal anti-game strategy. This method includes the following steps:

[0088] S1: Determine the end point of the adversarial game strategy based on the knowledge graph

[0089] In the confrontation game, the identification of radar targets is the first problem that needs to be solved. By identifying the target radar position, status and working mode, it can provide support for the determination of the end point of the subsequent confrontation game strategy path.

[0090] like Figure 1 As shown, the present invention utilizes the attribute feature knowledge of radiation sources such as radar targets in the constructed knowledge graph to obtain various technical parameters of the radar targets for analysis in real time, and adopts knowledge-driven bottom-up reasoning analysis to use various attribute parameters, real-time parameters and parameter change characteristics as underlying driving data, and extracts characteristic values ​​such as similarity, change rate and characteristic points that reflect target type and state characteristics. Further, through characteristic value analysis, characteristic facts are obtained and matched with the target feature knowledge given by the knowledge graph. According to the matching results, the matching structure results are obtained based on the knowledge graph search, that is, the target identification results such as target type and working mode, which serve as the end point of the adversarial game strategy of the knowledge graph.

[0091] S2: Constructing a Markov decision process model for adversarial game strategy path search

[0092] According to the above idea of ​​transforming the adversarial game strategy path into a sequential decision problem, such as Figure 2 As shown in the figure, we will take advantage of the strong ability of deep reinforcement learning in solving dynamic optimal sequence decision strategies, build a Markov decision process model (MDP) for strategy path search, obtain the path node selection action value function Q(A,S), and then use the deep reinforcement learning method to solve the action sequence A that maximizes the value of Q(A,S). x , this sequence corresponds to the optimal adversarial game strategy in the adversarial experience knowledge graph.

[0093] Specifically, if Figure 3 As shown, the dynamic knowledge graph is used to fight against the game strategy path R i Find the optimal node sequence A x To solve this problem, we need to build a Markov decision process model (MDP) for strategy path search. The four-tuple (S, A, P, R) elements of the Markov decision process are defined as follows:

[0094] S: represents the state, which is defined as the set of nodes (i.e., adversarial game operations) on the adversarial game strategy path obtained from the knowledge graph.

[0095] A: It represents an action, that is, whether each node in the knowledge graph is selected. In essence, selecting a node in the knowledge graph is to select the corresponding adversarial game operation.

[0096] P: represents the transition probability, which is the probability that the environment state changes to the next state S' after executing action A in the current state S;

[0097] R: represents the reward function R(S,A), which is the immediate reward obtained after selecting the adversarial execution action A in the current state S (i.e., the corresponding node on the adversarial game strategy path), that is, the evaluation of the adversarial interference effect on the radar target after selecting a certain adversarial strategy operation.

[0098] Through the above Markov decision process model, the path node selection action value function Q(A,S) is obtained.

[0099] S3: Solving the strategy path of adversarial game based on Actor-Critic deep reinforcement learning

[0100] like Figure 4As shown in Figure 1, the strategy path solution for the Actor-Critic deep reinforcement learning adversarial game consists of two parts: one is the reasoning decision network training, and the other is the optimal path sequence decision. The reasoning decision network training is completed by the interactive update between the Actor network and the Critic network and the environment (i.e., the knowledge graph); the optimal path sequence decision depends on the strategy path solution obtained by taking the environment and its state (current path selection) as the input of the Actor network.

[0101] S4: Comprehensive dynamic knowledge graph multi-adversarial game strategy path fusion strategy

[0102] It can be known that the common interference confrontation against the same target is completed by multiple devices in multiple ways. As far as the present invention is concerned, the above collaborative confrontation can be regarded as a multi-strategy path synthesis process. Based on the above path discovery synthesis method, by deleting some nodes in the acquired strategy path, and then using the above method to complete the strategy path discovery search, multiple confrontation game strategy paths can be obtained. Therefore, through the comprehensive strategy path, the coordinated confrontation game against the target can be realized.

[0103] Figure 5 The decision-level fusion process of the synthesis of multiple adversarial game strategy paths based on the adversarial game knowledge graph is given. The adversarial game strategy paths (knowledge graph nodes and connections) given by the aforementioned adversarial game strategy knowledge graph and their adversarial game effect evaluation are used to realize the synthesis of multiple adversarial interference strategy paths for target radars and obtain the best target adversarial game strategy. To illustrate the comprehensive decision-making strategy process, the present invention uses the system strategy effectiveness calculation based on evidence theory and the optimal strategy combination decision based on the particle swarm optimization algorithm to illustrate the comprehensive decision-making method.

[0104] (1) Calculation of the effectiveness of fusion strategy based on evidence theory

[0105] For the calculation of the effectiveness of the fusion strategy, the present invention converts the effectiveness of the strategy into the probability assignment corresponding to the evidence, and uses the combination rules of the evidence theory to complete the calculation of the effectiveness of the fusion strategy. The following is a technical route for the probability assignment fusion of two strategies, and multiple strategies are obtained by multiple pairwise fusions.

[0106] For two strategies with probability assignments m1 and m2, the rules for combining the two strategies. Let m1 and m2 be two independent basic probability assignments on U, and the basic probability assignment m after the combination of the two strategies c for:

[0107]

[0108] in, A1,A2,…,A k and B1,B2,…,Bk There are two evidence (strategy) focal elements, which are multiple evaluation indicators that describe the effectiveness of the fusion strategy.

[0109] (2) Optimal strategy decision based on swarm optimization method

[0110] In essence, the optimal strategy combination decision can be regarded as an optimization problem. The method of the present invention adopts a swarm optimization algorithm to realize the optimal fusion strategy combination selection solution. Among the swarm optimization algorithms, the particle swarm algorithm is more typical, which is simple and easy to implement, and has a fast convergence speed. The algorithm updates the memory and speed of each particle in the swarm, and uses the calculation result of the fusion strategy effectiveness based on the evidence theory described in this embodiment as the fusion strategy combination evaluation index. It uses swarm iteration and particles follow the optimal particles in the solution space to search and find the optimal solution of the adversarial game strategy path combination.

[0111] The examples of the present invention will be described in detail below with reference to the accompanying drawings.

[0112] 1. Determine the end point of the adversarial game strategy based on the knowledge graph

[0113] like Figure 6 As shown in the figure, various attribute parameters, real-time parameters and parameter change characteristics of radar targets are used as the underlying driving data, and characteristic values ​​such as similarity, change rate and characteristic points that reflect the radar target type and state characteristics are extracted. Through characteristic value analysis, characteristic facts are obtained and matched with the target feature knowledge given by the knowledge graph. According to the matching results, the matching results, namely the target recognition results such as radar target type and working mode, are obtained based on the knowledge graph search, which serves as the end point of the adversarial game strategy of the knowledge graph.

[0114] 2. Construction of Markov decision process model for strategy path search

[0115] Figure 7 Medium T j The i-th confrontation game strategy path R at time i (T j ) on the node set (P i , P1, P 12 , ...P i-3 , P n-1 , P n ). It can be seen that the alternative adversarial game strategy described by S (state). The cumulative reward obtained after selecting action A according to strategy π in state S0 can be expressed by the following state-action value function Q π (S,A) means:

[0116]

[0117]

[0118] In the above formula, the state-action value function Q π (S,A) The largest optimal strategy π * , recorded as:

[0119]

[0120] π * This is the optimal action sequence, which is the optimal strategy path for the game. Among them: γ is the discount rate, r is the immediate reward, t+k is the subscript indicating a certain moment in the future, T is the time when the reward is finally obtained, and s t is the state at time t, a t For state s t The action taken when

[0121] 3. Solving the strategy path of adversarial game with deep reinforcement learning

[0122] The strategy path solution of the Actor-Critic deep reinforcement learning adversarial game consists of two parts: one is the inference decision network training, and the other is the optimal path sequence decision. The inference decision network training is completed by the interactive update between the Actor network and the Critic network and the environment (i.e., the knowledge graph); the optimal path sequence decision depends on the strategy path solution obtained by taking the environment and its state (current path selection) as the input of the Actor network.

[0123] 4. Dynamic knowledge graph multi-adversarial game strategy path strategy synthesis

[0124] (1) Calculation of effectiveness of confrontation strategy based on evidence theory

[0125] like Figure 8 As shown in the figure, assume that m1 and m2 are the basic probability assignments of two independent adversarial game strategies on U, corresponding to strategy 1 and strategy 2 respectively. The basic probability assignment m after the combination of the two strategies 12 It can be calculated using the following formula (assuming ):

[0126]

[0127] The above formula A1, A2, ..., A k and B1,B2,…,B k There are two evidence (strategy) focal elements, which are multiple evaluation indicators that describe the effectiveness of the strategy.

[0128] (2) Optimal adversarial game strategy combination decision based on particle swarm optimization algorithm

[0129] The particle swarm algorithm is used to solve the optimal strategy combination, such as Fig. 9As shown in the figure, the basic idea is that for a component of each strategy particle, if the strategy is selected, the corresponding component value is 1, otherwise it is 0. The particle searches in the multidimensional binary space formed by the combination of various strategies. After each search, the above-mentioned system strategy effectiveness calculation results are used to evaluate the effect of the current search, that is, the efficiency of the combined interference game strategy, and then the particles are updated and iterated according to the evaluation results until the optimal combined confrontation game strategy is obtained.

[0130] A knowledge graph-based equipment dynamic confrontation game decision-making device, the device comprising:

[0131] The knowledge graph reasoning module is used to determine the radar countermeasure strategy endpoint and strategy reasoning solution path based on the knowledge graph;

[0132] Markov decision process model building module, used to build the Markov decision process model for strategy path search;

[0133] Deep reinforcement learning module, used to solve the adversarial game strategy path based on Actor-Critic deep reinforcement learning;

[0134] Multi-path strategy synthesis module, used to synthesize multiple dynamic game decision strategies based on knowledge graph;

[0135] The output of the knowledge graph module, i.e., the radar target recognition result, is directly connected to the input of the Markov decision process model building module as the basis for state definition.

[0136] The output of the Markov decision process model building module, namely the path node selection action value function Q(A,S), is connected to the input of the deep reinforcement learning module for training the Actor and Critic deep reinforcement learning networks.

[0137] The output of the deep reinforcement learning module, i.e., the optimal adversarial game strategy path, is connected to the input of the multi-adversarial game strategy path reasoning strategy synthesis module as the basis for strategy fusion.

[0138] The output of the multi-path adversarial game path strategy synthesis module, namely the optimal target adversarial game strategy, is output as the final decision result.

[0139] This connection forms a closed-loop process, from target recognition to policy reasoning, and then to policy fusion, and finally to the optimal decision result. The output of each module is the basis for the input of the next module, ensuring the coherence and effectiveness of the entire decision-making process.

[0140] The knowledge graph reasoning module includes:

[0141] Target recognition module, used to identify radar target types and working modes based on knowledge graph;

[0142] A feature extraction module is used to extract the feature values ​​of the similarity, change rate and feature points that reflect the target type and state characteristics;

[0143] The matching module is used to match the feature facts with the target feature knowledge given by the knowledge graph;

[0144] The search module is used to search and obtain target recognition results of target type and working mode based on the matching results.

[0145] The target recognition module is responsible for receiving radar target characteristic data and identifying the target using the radar target attribute feature knowledge stored in the knowledge graph. Its output is the identified target type and working mode.

[0146] The feature extraction module receives the output from the target recognition module, namely the radar target type and working mode, and extracts feature values ​​such as similarity, change rate and feature points that reflect the target type and state characteristics in combination with the target technical parameters acquired in real time.

[0147] The matching module receives the output from the feature extraction module, i.e., the radar target type and working mode feature values, and matches them with the target feature knowledge stored in the knowledge graph to determine the degree of matching between the target features and the target knowledge in the knowledge graph.

[0148] The search module receives the output from the matching module, i.e., the matching result, and searches the knowledge graph to obtain the structure that matches the radar target features, and finally outputs the radar target recognition results such as the target type and working mode.

[0149] This connection relationship forms a serial process, from radar target data acquisition to feature extraction, and then to feature matching, and finally target recognition. The output of each module is the basis for the input of the next module, ensuring the integrity and accuracy of the target recognition process.

[0150] The state definition module is responsible for defining the state space in the Markov decision process model, that is, the nodes on the adversarial game strategy path. Each node represents an adversarial game operation.

[0151] The action definition module is responsible for defining the selection actions in the Markov decision process model, that is, the actions that can be selected in each state. For each node, you can choose to execute or not execute the adversarial game operation represented by the node.

[0152] The transition probability calculation module is responsible for calculating the probability that the environment state will change to the next state after performing an action in the current state. For example, after executing a certain interference strategy, the state of the target may change, leading to entering the next state.

[0153] The reward function definition module is responsible for defining the reward function, which is used to evaluate the return obtained after selecting the adversarial game action in each state. The value of the reward function depends on the impact of the selected action on the radar target jamming effect, such as the strength of the jamming effect.

[0154] The output of the state definition module, namely the state space, is used as the input of the action definition module and the transition probability calculation module to define the action space and calculate the transition probability.

[0155] The output of the action definition module, namely the action space, is used as the input of the transition probability calculation module to calculate the transition probability.

[0156] The output of the transition probability calculation module, i.e., the transition probability matrix, together with the output of the state definition module, is used as the input of the reward function definition module to define the reward function.

[0157] The output of the reward function definition module, namely the reward function, is used to evaluate the rewards obtained after performing actions in each state and is an important part of the Markov decision process model.

[0158] The four modules, namely the state definition module, action definition module, transition probability calculation module and reward function definition module, are interconnected to jointly construct the Markov decision process model, providing a foundation for subsequent deep reinforcement learning.

[0159] The Actor network is responsible for generating a strategy path based on the current state information, that is, selecting which nodes represent the adversarial game strategy.

[0160] The Critic network is responsible for evaluating the value of the policy path generated by the Actor network, that is, evaluating the rewards that can be obtained by the path.

[0161] The interactive update module is responsible for interactively updating the Actor network and the Critic network with the environment (i.e., the knowledge graph). Specifically, it applies the policy path generated by the Actor network to the environment, observes the state changes of the environment and the rewards obtained, and then feeds this information back to the Actor network and the Critic network to update their parameters, thereby improving the generation quality and evaluation accuracy of the policy path.

[0162] The output of the Actor network, the policy path, is connected to the input of the Critic network to evaluate the path value.

[0163] The output of the Critic network, i.e., the path value evaluation result, is connected to the input of the Actor network to update the strategic path generation strategy.

[0164] The outputs of both the Actor network and the Critic network are connected to the input of the interaction update module to apply the policy path to the environment and obtain feedback information.

[0165] The output of the interactive update module, i.e., the environment state change and the reward information obtained, is connected to the input of the Actor network and the Critic network to update their parameters.

[0166] The three parts of the Actor network, Critic network and interactive update module are interconnected to form a closed-loop learning process. By constantly interacting with the environment and updating the parameters of the Actor network and Critic network, the Actor network can eventually generate the optimal strategy path.

[0167] The path search module is responsible for deleting nodes in the acquired strategy path, and then using the aforementioned method to complete the path discovery search to obtain multiple adversarial game strategy paths.

[0168] The strategy fusion module is responsible for evaluating the effectiveness of the adversarial game based on the adversarial game strategy path given by the adversarial game strategy knowledge graph and the adversarial game strategy effectiveness obtained by the evidence combination theory. It uses the particle swarm algorithm to obtain the optimal adversarial game strategy fusion result for radar targets and obtain the best target adversarial game strategy.

[0169] The output of the path search module, i.e., multiple adversarial game strategy paths, is connected to the input of the strategy fusion module as the basis for strategy fusion.

[0170] The output of the strategy fusion module, namely the optimal target adversarial game strategy, is the final decision result output.

[0171] The two modules, path search module and strategy fusion module, are interconnected to form a strategy fusion process, from the acquisition of multiple strategy paths to the fusion of strategy paths, and finally the best decision result is obtained. The output of each module is the basis for the input of the next module, ensuring the integrity and effectiveness of the strategy fusion process.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. The equipment dynamic confrontation game decision-making method based on knowledge graph is characterized by: The method comprises the following steps: S1: Determine the end point of the reasoning strategy based on the knowledge graph, including: Using the knowledge of target attribute characteristics in the knowledge graph and the various technical parameters of the target obtained and analyzed in real time, the following reasoning analysis is performed from bottom to top based on knowledge-driven: Taking various attribute parameters, real-time parameters and parameter change characteristics that reflect target characteristics as underlying driving data, extract the similarity, change rate and characteristic value of feature points of target type and state characteristics; Further through the feature value analysis, the feature facts are obtained and matched with the target feature knowledge given by the knowledge graph; the matching result is the target recognition result of the target type and working mode, that is, the end point of the adversarial game strategy path; S2: Construct a Markov decision process model (MDP) for adversarial game strategy path search, including: Transform the strategy path solution of adversarial game into a sequential decision problem; Construct a Markov decision process model (S, A, P, R) for adversarial game strategy reasoning path search, where: S: represents the state, which is defined as the set of nodes on the adversarial game strategy path obtained from the knowledge graph, that is, the set of adversarial game operations; A: indicates the action, that is, whether each node in the knowledge graph is selected; P: represents the transition probability, which is the probability that the state changes to the next state S' after executing action A in the current state S; R: represents the reward function R(S,A), which is the immediate reward obtained after selecting action A at the current state S, that is, the corresponding node on the adversarial game strategy path, that is, the evaluation of the adversarial effect after selecting a certain adversarial strategy operation; Through the above Markov decision process model, the strategy path node selection action value function Q(A,S) is obtained; S3: Based on the Actor-Critic deep reinforcement learning method, solve the action sequence that maximizes the value of Q(A,S). This sequence corresponds to the optimal adversarial game strategy reasoning path in the adversarial experience knowledge graph, including: The reasoning decision network training is completed by the interaction and update between the Actor network and the Critic network and the environment; the environment is a confrontational game situation; The optimal path node sequence decision depends on the strategic path solution obtained by using the environment and state as the input of the Actor network; where the state is the current path selection; S4: Dynamic knowledge graph multi-adversarial game strategy path comprehensive strategy, including: Delete nodes from the acquired strategy path, and then use the above method to complete the discovery and search of other strategy paths to obtain multiple adversarial game strategy paths; Based on the adversarial game strategy path given by the adversarial game strategy knowledge graph and its adversarial game effect evaluation, the optimal adversarial game strategy fusion method is used to combine multiple adversarial interference strategy paths to obtain the optimal strategy for the target adversarial game.

2. The equipment dynamic confrontation game decision-making method based on knowledge graph according to claim 1 is characterized in that: The S4 is obtained by fusing the strategies corresponding to multiple adversarial game strategy paths; firstly, based on the evidence theory method, multiple evaluation indicators describing the effectiveness of the fusion strategy are used as evidence focal elements to obtain the basic probability assignment of the strategy, and the effectiveness of the fusion strategy is calculated using the strategy combination rule method of evidence theory; then, the swarm optimization algorithm is used, combined with the effectiveness of the fusion strategy, to obtain the best strategy solution for the adversarial game to achieve strategy fusion.

3. The equipment dynamic confrontation game decision-making device based on knowledge graph is characterized by: The device includes: The knowledge graph reasoning module is used to determine the adversarial strategy endpoint and strategy reasoning solution path based on the knowledge graph; Markov decision process model building module, used to build the Markov decision process model for strategy path search; Deep reinforcement learning module, using Actor-Critic based deep reinforcement learning method to solve dynamic confrontation game strategy path; Multi-path confrontation game strategy synthesis module, used to synthesize multiple dynamic game decision strategies based on knowledge graph; The output of the knowledge graph reasoning module, i.e., the target recognition result and the adversarial game reasoning path node, is directly connected to the input of the adversarial game-oriented Markov decision process model construction module of the present invention as the basis for state definition; The output of the Markov decision process model building module, i.e., the path node selection action value function Q(A,S), is connected to the input of the deep reinforcement learning module to solve the adversarial game reasoning strategy; The output of the deep reinforcement learning module, i.e., the optimal adversarial game strategy reasoning path, is connected to the input of the multi-path reasoning strategy synthesis module as the basis for strategy fusion; The output of the multi-adversarial game strategy path synthesis module, namely the optimal target adversarial game strategy, is output as the final decision result.

4. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 3 is characterized by: The knowledge graph reasoning module includes: Target recognition module, used to identify target types and working modes based on knowledge graphs; A feature extraction module is used to extract the feature values ​​of the similarity, change rate and feature points that reflect the target type and state characteristics; The matching module is used to match the feature facts with the target feature knowledge given by the knowledge graph; A search module, used to search and obtain target recognition results of target type and working mode based on matching results; The target recognition module is responsible for receiving target data and identifying the target using the target attribute feature knowledge stored in the knowledge graph; its output is the identified target type and working mode; The feature extraction module receives the output from the target recognition module, i.e., the target type and working mode, and extracts feature values ​​such as similarity, change rate, and feature points reflecting the target type and state characteristics in combination with the target technical parameters acquired in real time; The matching module receives the output from the feature extraction module, i.e., the feature value, and matches it with the target feature knowledge stored in the knowledge graph to determine the matching degree between the target feature and the target knowledge in the knowledge graph; The search module receives the output from the matching module, i.e., the matching result, and searches the knowledge graph to obtain a structure that matches the target feature, and finally outputs the target recognition results such as the target type and working mode; The output of the state definition module, i.e., the state space, is used as the input of the action definition module and the transition probability calculation module to define the action space and calculate the transition probability; The output of the action definition module, i.e., the action space, is used to select appropriate nodes from the knowledge graph module as candidate adversarial game strategy path nodes, and is used as the input of the transition probability calculation module to calculate the transition probability; The output of the transition probability calculation module, i.e., the transition probability matrix, together with the output of the state definition module, is used as the input of the reward function definition module to define the reward function; The output of the reward function definition module, namely the reward function, is used to evaluate the rewards obtained after performing actions in each state and is an important part of the Markov decision process model.

5. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 3 is characterized by: The Markov decision process model building module includes: A state definition module, used to define each node on the adversarial game strategy path, wherein the node describes an alternative adversarial game strategy; An action definition module is used to define whether each node in the knowledge graph is selected. The action essentially corresponds to the selection of a certain interference countermeasure operation; The transition probability calculation module is used to calculate the probability that the environment state will change to the next state after performing an action in the current state; The reward function definition module is used to define the reward function R(S,A), where S represents the current state and A represents the execution action. The reward function is a scalar function of the current state and action, which represents the immediate reward obtained after selecting the execution action in the current state. The reward can essentially be regarded as an evaluation of the interference effect after the action selects a certain interference countermeasure operation.

6. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 3 is characterized by: The deep reinforcement learning module includes: Actor network, used to generate adversarial game strategy reasoning paths; Critic network, used to evaluate the value of the adversarial game strategy reasoning path; The interactive update module is used to interactively update the Actor network and the Critic network with the environment, where the environment represents the knowledge graph; The output of the Actor network, i.e., the adversarial game strategy reasoning path, is connected to the input of the Critic network to evaluate the path value; The output of the Critic network, i.e. the path value evaluation result, is connected to the input of the Actor network and used to update the adversarial game strategy reasoning path generation strategy; The outputs of the Actor network and the Critic network are connected to the input of the interactive update module, which is used to apply the adversarial game strategy reasoning path to the environment and obtain feedback information; The output of the interactive update module, i.e., the environment state change and the reward information obtained, is connected to the input of the Actor network and the Critic network to update their parameters.

7. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 4 is characterized in that: The multi-path confrontation game strategy integration module includes: A path search module is used to delete nodes in the acquired adversarial game strategy path, and then use the above method to complete the adversarial game strategy path discovery search to obtain multiple adversarial game strategy paths; The strategy fusion module is used to use the adversarial game strategy path and its adversarial game effect evaluation given by the adversarial game strategy knowledge graph, realize the integration of multiple adversarial game strategy paths through strategy comprehensive optimization, and obtain the best target adversarial game strategy; The output of the path search module, i.e., multiple adversarial game strategy paths, is connected to the input of the strategy fusion module as the basis for strategy fusion; The output of the strategy fusion module, i.e., the optimal target adversarial game strategy, is output as the final decision result.

Images