Equipment dynamic confrontation game decision-making method and device based on knowledge graph

Abstract

The present invention proposes a method and device for equipment dynamic confrontation game decision-making based on knowledge graph, which belongs to the field of behavioral decision-making technology and aims to solve the problem that the existing technology lacks effectiveness and efficiency in dynamic game strategy selection. First, the present invention identifies the target and determines the end point of the reasoning strategy through the knowledge graph. Then, using the Markov decision process model, the game strategy reasoning path is converted into a sequential decision problem, and solved by the Actor-Critic deep reinforcement learning method. Finally, the optimal game strategy is obtained through path search and multi-path integration. The present invention improves the efficiency and effectiveness of equipment confrontation decision-making and has important application value.

Description

Technical Field

[0001] The present invention belongs to the field of decision-making technology and relates to a knowledge graph-based equipment dynamic confrontation game decision-making method and device. Background Art

[0002] Solving the confrontation game strategy between equipment is one of the key technologies for equipment confrontation decision-making. Its core lies in how to select the optimal confrontation strategy based on the environment and the dynamic performance characteristics of the equipment of both sides.

[0003] Currently, the main challenges in solving dynamic game strategies are: existing game decision-making methods often consider single equipment performance characteristics, lack a systematic analysis of the interactions and relationships between equipment on both sides, and lack comprehensiveness. Existing methods struggle to effectively describe the dynamic nature of the game between opposing equipment and targets, making them difficult to adapt to ever-changing environments.

[0004] In order to overcome the above shortcomings, the present invention proposes a game decision-making method and device for confrontation equipment and targets based on knowledge graph. By constructing a dynamic knowledge graph of confrontation experience, deep reinforcement learning is used to solve the confrontation game strategy path, and then the swarm intelligence algorithm is used to select the optimal strategy combination to achieve the optimal selection of confrontation game strategy. Summary of the Invention

[0005] In light of this, the present invention aims to provide a knowledge graph-based method and device for dynamic equipment adversarial game decision-making. Based on the interactions and correlations between equipment adversarial situations, the present invention integrates multidimensional game strategy characteristics and their associated game strategies and experience to construct a knowledge graph for game adversarial strategy. On this basis, the adversarial game strategy path is converted into an adversarial game strategy action sequence. By constructing a Markov decision process model for dynamic path decision-making, the action-value function of the game decision path is obtained. Deep reinforcement learning is used to solve the game strategy path. Finally, by integrating evidence theory and swarm intelligence algorithms, the optimal adversarial game strategy is determined.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The equipment dynamic confrontation game decision-making method based on knowledge graph includes the following steps:

[0008] S1: Determine the endpoint of the reasoning strategy based on the knowledge graph, including:

[0009] By utilizing the target attribute characteristics in the knowledge graph and the target technical parameters acquired and analyzed in real time, the following reasoning analysis is performed in a knowledge-driven bottom-up manner:

[0010] Using various attribute parameters, real-time parameters, and parameter change characteristics that reflect target characteristics as underlying driving data, the similarity, change rate, and characteristic values ​​of target types and state characteristics are extracted;

[0011] Further feature value analysis is performed to obtain feature facts, which are then matched with the target feature knowledge given by the knowledge graph. The matching result is the target identification result of the target type and working mode, that is, the end point of the adversarial game strategy path.

[0012] S2: Construct a Markov Decision Process (MDP) model for adversarial game strategy path search, including:

[0013] Convert the strategy path solution of the adversarial game into a sequential decision problem;

[0014] Construct a Markov decision process model (S, A, P, R) for adversarial game strategy reasoning path search, where:

[0015] S: represents the state, which is defined as the set of nodes on the adversarial game strategy path obtained from the knowledge graph, that is, the set of adversarial game operations;

[0016] A: Indicates the action, that is, whether each node in the knowledge graph is selected;

[0017] P: represents the transition probability, which is the probability that the state will change to the next state S' after executing action A in the current state S;

[0018] R: Reward function R(S,A), which is the immediate reward obtained after executing action A at the current state S, that is, the corresponding node on the adversarial game strategy path. It is the evaluation of the adversarial effect after selecting a certain adversarial strategy operation;

[0019] Through the above Markov decision process model, the strategy path node selection action value function Q(A,S) is obtained;

[0020] S3: Based on the Actor-Critic deep reinforcement learning method, we solve the action sequence that maximizes the value of Q(A,S). This sequence corresponds to the optimal adversarial game strategy reasoning path in the adversarial experience knowledge graph, including:

[0021] The reasoning decision network training is completed through the interactive update between the Actor network and the Critic network and the environment; the environment is an adversarial game situation;

[0022] The optimal path node sequence decision is based on solving the strategic path obtained by using the environment and state as inputs to the Actor network; where the state is the current path selection;

[0023] S4: Dynamic knowledge graph multi-adversarial game strategy path comprehensive strategy, including:

[0024] Delete nodes from the acquired strategy path, and then use the aforementioned method to complete the search for other strategy paths to obtain multiple adversarial game strategy paths;

[0025] Based on the adversarial game strategy path and adversarial game effect evaluation given by the adversarial game strategy knowledge graph, the optimal adversarial game strategy fusion method is used to combine multiple adversarial interference strategy paths to obtain the optimal strategy for the target adversarial game.

[0026] Furthermore, the S4 is obtained by fusing the strategies corresponding to multiple adversarial game strategy paths. First, based on the evidence theory method, multiple evaluation indicators describing the effectiveness of the fusion strategy are used as evidence focal elements to obtain the basic probability assignment of the strategy. The effectiveness of the fusion strategy is calculated using the strategy combination rule method of evidence theory. Then, the swarm optimization algorithm is used in combination with the effectiveness of the fusion strategy to obtain the optimal strategy solution for the adversarial game, thereby realizing strategy fusion.

[0027] A knowledge graph-based equipment dynamic confrontation game decision-making device, which includes:

[0028] The knowledge graph reasoning module is used to determine the endpoint of the adversarial strategy and the strategy reasoning solution path based on the knowledge graph;

[0029] Markov decision process model building module, used to build a Markov decision process model for strategic path search;

[0030] Deep reinforcement learning module, which uses the actor-critic based deep reinforcement learning method to solve the dynamic confrontation game strategy path;

[0031] Multi-path adversarial game strategy synthesis module, used to synthesize multiple dynamic game decision strategies based on knowledge graphs;

[0032] The output of the knowledge graph reasoning module, i.e., the target recognition result and the adversarial game reasoning path node, is directly connected to the input of the adversarial game-oriented Markov decision process model construction module of the present invention as the basis for state definition;

[0033] The output of the Markov decision process model building module, namely the path node selection action value function Q(A,S), is connected to the input of the deep reinforcement learning module to solve the adversarial game reasoning strategy;

[0034] The output of the deep reinforcement learning module, i.e., the optimal adversarial game strategy reasoning path, is connected to the input of the multi-path reasoning strategy synthesis module as the basis for strategy fusion;

[0035] The output of the multi-adversarial game strategy path synthesis module, namely the optimal target adversarial game strategy, is output as the final decision result.

[0036] Furthermore, the knowledge graph reasoning module includes:

[0037] Target recognition module, used to identify target types and working modes based on knowledge graphs;

[0038] Feature extraction module, used to extract the similarity, change rate and feature values ​​of feature points that reflect the target type and state characteristics;

[0039] The matching module is used to match the feature facts with the target feature knowledge given by the knowledge graph;

[0040] A search module is used to search and obtain target recognition results of target type and working mode based on the matching results;

[0041] The target recognition module is responsible for receiving target data and identifying the target using the target attribute feature knowledge stored in the knowledge graph; its output is the identified target type and working mode;

[0042] The feature extraction module receives the output from the target recognition module, namely the target type and working mode, and combines it with the target technical parameters obtained in real time to extract feature values ​​such as similarity, change rate and feature points that reflect the target type and state characteristics;

[0043] The matching module receives the output from the feature extraction module, i.e., the feature value, and matches it with the target feature knowledge stored in the knowledge graph to determine the matching degree between the target feature and the target knowledge in the knowledge graph;

[0044] The search module receives the output from the matching module, i.e., the matching result, and searches the knowledge graph to obtain a structure that matches the target features. It finally outputs the target recognition results such as the target type and working mode.

[0045] The output of the state definition module, i.e., the state space, is used as the input of the action definition module and the transition probability calculation module to define the action space and calculate the transition probability;

[0046] The output of the action definition module, i.e., the action space, is used to select appropriate nodes from the knowledge graph module as candidate adversarial game strategy path nodes, and is also used as the input of the transition probability calculation module to calculate the transition probability.

[0047] The output of the transition probability calculation module, that is, the transition probability matrix, together with the output of the state definition module, is used as the input of the reward function definition module to define the reward function;

[0048] The output of the reward function definition module, namely the reward function, is used to evaluate the rewards obtained after performing an action in each state and is an important part of the Markov decision process model.

[0049] Furthermore, the Markov decision process model building module includes:

[0050] A state definition module, used to define each node on the adversarial game strategy path, wherein the node describes an alternative adversarial game strategy;

[0051] The action definition module is used to define whether each node in the knowledge graph is selected. The action actually corresponds to the selection of a certain interference countermeasure operation;

[0052] The transition probability calculation module is used to calculate the probability that the environment state will change to the next state after performing an action in the current state;

[0053] The reward function definition module is used to define the reward function R(S,A), where S represents the current state and A represents the executed action. The reward function is a scalar function of the current state and action, representing the immediate reward obtained after selecting the action to be executed in the current state. The reward can essentially be regarded as an evaluation of the interference effect after the action selects a certain interference countermeasure operation.

[0054] Furthermore, the deep reinforcement learning module includes:

[0055] Actor network, used to generate adversarial game strategy reasoning paths;

[0056] Critic network, used to evaluate the value of the adversarial game strategy reasoning path;

[0057] The interactive update module is used to interactively update the Actor network and the Critic network with the environment, where the environment represents the knowledge graph;

[0058] The output of the Actor network, i.e., the adversarial game strategy reasoning path, is connected to the input of the Critic network to evaluate the path value;

[0059] The output of the Critic network, i.e. the path value evaluation result, is connected to the input of the Actor network and is used to update the adversarial game strategy reasoning path generation strategy;

[0060] The outputs of the Actor and Critic networks are connected to the input of the interactive update module, which is used to apply the adversarial game strategy reasoning path to the environment and obtain feedback information;

[0061] The output of the interaction update module, namely the environment state change and the reward information obtained, is connected to the input of the Actor network and the Critic network to update their parameters.

[0062] Furthermore, the multi-path confrontation game strategy integration module includes:

[0063] A path search module is used to delete nodes from the acquired adversarial game strategy path, and then use the aforementioned method to complete the adversarial game strategy path discovery search to obtain multiple adversarial game strategy paths;

[0064] The strategy fusion module is used to use the adversarial game strategy path and its adversarial game effect evaluation given by the adversarial game strategy knowledge graph, realize the integration of multiple adversarial game strategy paths through strategy comprehensive optimization, and obtain the optimal target adversarial game strategy;

[0065] The output of the path search module, i.e., multiple adversarial game strategy paths, is connected to the input of the strategy fusion module as the basis for strategy fusion;

[0066] The output of the strategy fusion module, that is, the optimal target adversarial game strategy, is output as the final decision result.

[0067] The beneficial effects of the present invention are:

[0068] (1) By constructing a dynamic knowledge graph of adversarial experience, the present invention can comprehensively analyze the interactions and correlations between adversarial equipment and targets, and integrate multi-dimensional game strategy characteristics, thereby achieving a more systematic and scientific strategy selection.

[0069] (2) The present invention uses deep reinforcement learning to solve the adversarial game strategy, which can effectively describe the dynamics and complexity of the game process between the adversarial equipment and the target, and adapt to the ever-changing environment.

[0070] (3) The present invention utilizes the advantages of deep reinforcement learning to quickly find the optimal strategy, thereby improving the efficiency of strategy reasoning and meeting the needs of real-time decision-making.

[0071] (4) The present invention can help the counter-equipment select the optimal counter-equipment strategy, thereby giving full play to the effectiveness of the counter-equipment and enhancing its advantages.

[0072] (5) The present invention can automatically generate countermeasure strategies, reduce the difficulty of operating countermeasure equipment, and improve efficiency.

[0073] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0075] Figure 1 Identify target types and working mode states based on knowledge graphs;

[0076] Figure 2 Deep reinforcement learning path discovery for adversarial empirical knowledge graphs;

[0077] Figure 3 MDP model for strategy path discovery in dynamic knowledge graph adversarial games;

[0078] Figure 4 Solve the reasoning path for adversarial game strategies based on Actor-Critic deep reinforcement learning;

[0079] Figure 5 It is a multi-path strategy synthesis based on adversarial experience knowledge graph;

[0080] Figure 6 Identify implementation examples for target types and working modes based on knowledge graphs;

[0081] Figure 7 An MDP model implementation for dynamic knowledge graph adversarial game strategy path discovery;

[0082] Figure 8 A comprehensive implementation example of a multi-path strategy based on adversarial experience knowledge graph;

[0083] Figure 9 A multi-path strategy synthesis based on adversarial experience knowledge graph. DETAILED DESCRIPTION

[0084] The following describes the embodiments of the present invention using a specific example of a radar as a specific target. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different perspectives and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in the following embodiments are merely schematic illustrations of the basic concepts of the present invention, and the following embodiments and features therein may be combined with one another unless there is a conflict.

[0085] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0086] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0087] This invention is aimed at equipment anti-game applications. It uses knowledge graphs and actual anti-game data to determine the anti-game strategy endpoint. On this basis, the anti-game strategy path is converted into an anti-game strategy action sequence. By constructing a Markov decision process model for dynamic path decision-making, the action value function of the game decision path is obtained, and deep reinforcement learning is used to solve the game strategy path. Finally, the optimal anti-game strategy is determined by integrating evidence theory and swarm intelligence algorithms. This method includes the following steps:

[0088] S1: Determine the endpoint of the adversarial game strategy based on the knowledge graph

[0089] In a competitive game, radar target identification is the first issue that needs to be addressed. By identifying the target radar's location, status, and operating mode, we can support the subsequent determination of the endpoint of the competitive game strategy path.

[0090] like Figure 1 As shown, the present invention utilizes the attribute feature knowledge of radiation sources such as radar targets in the constructed knowledge graph to obtain the various technical parameters of the analyzed radar targets in real time, adopts knowledge-driven bottom-up reasoning analysis, and uses various attribute parameters, real-time parameters and parameter change characteristics as the underlying driving data. It extracts characteristic values ​​such as similarity, change rate and characteristic points that reflect the target type and state characteristics, and further obtains characteristic facts through characteristic value analysis. It matches the target feature knowledge given by the knowledge graph, and obtains the matching structure results based on the knowledge graph search according to the matching results, that is, the target identification results such as target type and working mode, as the end point of the adversarial game strategy of the knowledge graph.

[0091] S2: Constructing a Markov decision process model for adversarial game strategy path search

[0092] According to the above idea of ​​transforming the adversarial game strategy path into a sequential decision problem, such as Figure 2 As shown in the figure, we will take advantage of the strong ability of deep reinforcement learning in solving dynamic optimal sequence decision strategies. By constructing a Markov decision process model (MDP) for strategy path search, we can obtain the path node selection action value function Q(A, S). Then, we use the deep reinforcement learning method to solve the action sequence A that maximizes the value of Q(A, S). x , this sequence corresponds to the optimal adversarial game strategy in the adversarial experience knowledge graph.

[0093] Specifically, if Figure 3 As shown, the dynamic knowledge graph is used to fight against the game strategy path R i Found to be transformed into the optimal node sequence A x To solve this problem, we need to build a Markov decision process model (MDP) for strategic path search. The four-tuple (S, A, P, R) elements of the Markov decision process are defined as follows:

[0094] S: represents the state, which is defined as the set of nodes (i.e., adversarial game operations) on the adversarial game strategy path obtained from the knowledge graph.

[0095] A: Indicates the action, that is, whether each node in the knowledge graph is selected. In fact, selecting a node in the knowledge graph is to select the corresponding adversarial game operation.

[0096] P: represents the transition probability, which is the probability that the environment state will change to the next state S' after executing action A in the current state S;

[0097] R: Represents the reward function R(S,A), which is the immediate reward obtained after selecting the adversarial execution action A in the current state S (i.e., the corresponding node on the adversarial game strategy path). In other words, it is the evaluation of the adversarial interference effect on the radar target after selecting a certain adversarial strategy operation.

[0098] Through the above Markov decision process model modeling, the path node selection action value function Q(A,S) is obtained.

[0099] S3: Solving the strategy path of adversarial game based on Actor-Critic deep reinforcement learning

[0100] like Figure 4As shown in Figure 1, solving the strategic path for adversarial games based on actor-critic deep reinforcement learning consists of two parts: first, training the inference decision network, and second, determining the optimal path sequence. The inference decision network training relies on interactive updates between the actor and critic networks and the environment (i.e., the knowledge graph). The optimal path sequence decision is based on solving the strategic path using the environment and its state (the current path choice) as input to the actor network.

[0101] S4: Comprehensive dynamic knowledge graph multi-adversarial game strategy path fusion strategy

[0102] As we know, interference countermeasures against a common target are often accomplished through the coordinated efforts of multiple devices in various ways. In the context of the present invention, this coordinated countermeasure can be viewed as a multi-strategy path synthesis process. Based on the above path discovery and synthesis method, by removing some nodes from the acquired strategy path and then using the aforementioned method to complete the strategy path discovery search, multiple strategy paths for countermeasures can be obtained. Thus, by synthesizing the strategy paths, a coordinated countermeasure against the target can be achieved.

[0103] Figure 5 A decision-level fusion process for synthesizing multiple adversarial game strategy paths based on the adversarial game knowledge graph is presented. Utilizing the adversarial game strategy paths (knowledge graph nodes and connections) and their effectiveness evaluations provided by the aforementioned adversarial game strategy knowledge graph, multiple adversarial jamming strategy paths are synthesized for target radars to obtain the optimal target adversarial game strategy. To illustrate the integrated decision-making process, this paper uses the calculation of system strategy effectiveness based on evidence theory and the optimal strategy combination decision-making based on the particle swarm optimization algorithm to illustrate the integrated decision-making method.

[0104] (1) Calculation of the effectiveness of fusion strategies based on evidence theory

[0105] To calculate the effectiveness of fusion strategies, this paper converts the effectiveness of the strategies into probability assignments corresponding to the evidence. Using the combinational rules of evidence theory, this method calculates the effectiveness of the fusion strategy. The following describes a technical approach for fusion of probability assignments corresponding to two strategies. Multiple strategies are obtained by multiple pairwise fusions.

[0106] For two strategies with probability assignments m1 and m2, the rules for combining the two strategies. Let m1 and m2 be two independent basic probability assignments on U, and the basic probability assignment m after the two strategies are combined c for:

[0107]

[0108] in, A1,A2,…,A k and B1,B2,…,Bk It is two evidence (strategy) focal elements and multiple evaluation indicators that describe the effectiveness of the fusion strategy.

[0109] (2) Optimal strategy decision-making based on swarm optimization method

[0110] Essentially, the optimal strategy combination decision can be considered an optimization problem. The method of the present invention uses a swarm optimization algorithm to achieve the optimal fusion strategy combination selection solution. Among swarm optimization algorithms, the particle swarm algorithm is relatively typical. It is simple and easy to implement, and has a fast convergence speed. The algorithm uses the memory and speed update of each particle in the swarm, and uses the fusion strategy effectiveness calculation results based on evidence theory described in this embodiment as the fusion strategy combination evaluation index. It uses swarm iteration and particles to search in the solution space following the optimal particle to find the optimal solution for the adversarial game strategy path combination.

[0111] The following is a detailed description of examples of the present invention with reference to the accompanying drawings.

[0112] 1. Determine the endpoint of the adversarial game strategy based on the knowledge graph

[0113] like Figure 6 As shown in the figure, various attribute parameters, real-time parameters and parameter change characteristics of radar targets are used as the underlying driving data, and characteristic values ​​such as similarity, change rate and characteristic points reflecting the radar target type and state characteristics are extracted. Through characteristic value analysis, characteristic facts are obtained and matched with the target feature knowledge given by the knowledge graph. According to the matching results, the matching results, namely the target recognition results such as radar target type and working mode, are obtained based on the knowledge graph search, which serves as the end point of the adversarial game strategy of the knowledge graph.

[0114] 2. Construction of Markov Decision Process Model for Strategy Path Search

[0115] Figure 7 Middle T j The i-th adversarial game strategy path R at time i (T j ) on the node set (P i , P1, P 12 ,...P i-3 , P n-1 , P n ). It can be seen that the alternative adversarial game strategy described by S (state). The cumulative reward obtained after selecting action A according to strategy π in state S0 can be expressed using the following state-action value function Q π (S,A) means:

[0116]

[0117]

[0118] In the above formula, the state-action value function Q π (S,A) the largest optimal strategy π * , recorded as:

[0119]

[0120] π * This is the optimal action sequence, which is the optimal strategy path for the adversarial game. Here, γ is the discount rate, r is the immediate reward, t+k is the subscript representing a certain moment in the future, T is the time when the reward is finally obtained, and s t is the state at time t, a t For state s t Action taken when

[0121] 3. Solving the strategy path of adversarial game with deep reinforcement learning

[0122] Solving the strategic path for adversarial games based on actor-critic deep reinforcement learning consists of two parts: first, training the inference decision network, and second, determining the optimal path sequence. The inference decision network training relies on interactive updates between the actor and critic networks and the environment (i.e., the knowledge graph). Optimal path sequence determination relies on solving the strategic path using the environment and its state (the current path choice) as input to the actor network.

[0123] 4. Dynamic Knowledge Graph Multi-Adversarial Game Strategy Path Strategy Synthesis

[0124] (1) Calculation of the effectiveness of adversarial strategies based on evidence theory

[0125] like Figure 8 As shown, assuming that m1 and m2 are the basic probability assignments of two independent adversarial game strategies on U, corresponding to strategy 1 and strategy 2 respectively, the basic probability assignment m after the combination of the two strategies 12 It can be calculated using the following formula (assuming ):

[0126]

[0127] The above formula A1, A2, ..., A k and B1,B2,…,B k There are two focal elements of evidence (strategy), which are multiple evaluation indicators that describe the effectiveness of the strategy.

[0128] (2) Optimal adversarial game strategy combination decision-making based on particle swarm optimization algorithm

[0129] The particle swarm algorithm is used to solve the optimal strategy combination, such as Figure 9As shown in Figure 1, the basic idea is that for each strategy particle, if the strategy is selected, the corresponding component value is 1, otherwise it is 0. The particle searches in the multidimensional binary space formed by the combination of various strategies. After each search, the results of the aforementioned system strategy effectiveness calculation are used to evaluate the effect of the current search, that is, the efficiency of the combined interference game strategy. The particles are then updated and iterated based on the evaluation results until the optimal combined interference game strategy is obtained.

[0130] A knowledge graph-based equipment dynamic confrontation game decision-making device, which includes:

[0131] The knowledge graph reasoning module is used to determine the radar countermeasure strategy endpoint and strategy reasoning solution path based on the knowledge graph;

[0132] Markov decision process model building module, used to build a Markov decision process model for strategic path search;

[0133] Deep reinforcement learning module, used to solve the strategy path of adversarial game based on actor-critic deep reinforcement learning;

[0134] Multi-path strategy synthesis module, used to synthesize multiple dynamic game decision strategies based on knowledge graph;

[0135] The output of the knowledge graph module, i.e., the radar target recognition result, is directly connected to the input of the Markov decision process model building module as the basis for state definition.

[0136] The output of the Markov decision process model building module, namely the path node selection action value function Q(A,S), is connected to the input of the deep reinforcement learning module for training the Actor and Critic deep reinforcement learning networks.

[0137] The output of the deep reinforcement learning module, i.e., the optimal adversarial game strategy path, is connected to the input of the multi-adversarial game strategy path reasoning strategy synthesis module as the basis for strategy fusion.

[0138] The output of the multi-path adversarial game strategy synthesis module, that is, the optimal target adversarial game strategy, is output as the final decision result.

[0139] This connection forms a closed-loop process, from target identification to policy reasoning and then to policy fusion, ultimately achieving the optimal decision. The output of each module serves as the input for the next, ensuring the coherence and effectiveness of the entire decision-making process.

[0140] The knowledge graph reasoning module includes:

[0141] Target recognition module, used to identify radar target types and operating modes based on knowledge graphs;

[0142] Feature extraction module, used to extract the similarity, change rate and feature values ​​of feature points that reflect the target type and state characteristics;

[0143] The matching module is used to match the feature facts with the target feature knowledge given by the knowledge graph;

[0144] The search module is used to search and obtain target recognition results of target type and working mode based on the matching results.

[0145] The target recognition module is responsible for receiving radar target characteristic data and identifying the target using the radar target attribute feature knowledge stored in the knowledge graph. Its output is the identified target type and operating mode.

[0146] The feature extraction module receives the output from the target recognition module, namely the radar target type and working mode, and combines it with the target technical parameters obtained in real time to extract feature values ​​such as similarity, change rate and feature points that reflect the target type and state characteristics.

[0147] The matching module receives the output from the feature extraction module, namely the radar target type and working mode feature value, and matches it with the target feature knowledge stored in the knowledge graph to determine the degree of matching between the target feature and the target knowledge in the knowledge graph.

[0148] The search module receives the output from the matching module, i.e., the matching result, and searches the knowledge graph to obtain a structure that matches the radar target features, and finally outputs radar target recognition results such as target type and working mode.

[0149] This connection forms a serial process, from radar target data acquisition to feature extraction, feature matching, and ultimately target recognition. The output of each module serves as the input for the next module, ensuring the integrity and accuracy of the target recognition process.

[0150] The state definition module is responsible for defining the state space in the Markov decision process model, that is, the nodes on the adversarial game strategy path. Each node represents an adversarial game operation.

[0151] The action definition module is responsible for defining the selection actions in the Markov decision process model, that is, the actions that can be selected in each state. For each node, you can choose to execute or not execute the adversarial game operation represented by the node.

[0152] The transition probability calculation module is responsible for calculating the probability that the environment state will transition to the next state after executing an action in the current state. For example, after executing a certain interference strategy, the state of the target may change, leading to the next state.

[0153] The reward function definition module is responsible for defining the reward function, which is used to evaluate the reward obtained after selecting an action in the adversarial game in each state. The value of the reward function depends on the impact of the selected action on the radar target jamming effect, such as the strength of the jamming effect.

[0154] The output of the state definition module, namely the state space, is used as the input of the action definition module and the transition probability calculation module to define the action space and calculate the transition probability.

[0155] The output of the action definition module, i.e., the action space, is used as the input of the transition probability calculation module to calculate the transition probability.

[0156] The output of the transition probability calculation module, that is, the transition probability matrix, together with the output of the state definition module, is used as the input of the reward function definition module to define the reward function.

[0157] The output of the reward function definition module, namely the reward function, is used to evaluate the rewards obtained after performing an action in each state and is an important part of the Markov decision process model.

[0158] The four modules, namely the state definition module, action definition module, transition probability calculation module, and reward function definition module, are interconnected to jointly construct the Markov decision process model, providing a foundation for subsequent deep reinforcement learning.

[0159] The Actor network is responsible for generating a strategy path based on the current state information, that is, selecting which nodes represent the adversarial game strategy.

[0160] The Critic network is responsible for evaluating the value of the policy path generated by the Actor network, that is, evaluating the rewards that can be obtained by the path.

[0161] The interactive update module is responsible for interactively updating the Actor and Critic networks with the environment (i.e., the knowledge graph). Specifically, it applies the policy path generated by the Actor network to the environment, observes the changes in the environment's state and the rewards obtained, and then feeds this information back to the Actor and Critic networks to update their parameters, thereby improving the generation quality and evaluation accuracy of the policy path.

[0162] The output of the Actor network, i.e., the policy path, is connected to the input of the Critic network for evaluating the path value.

[0163] The output of the Critic network, i.e., the path value evaluation result, is connected to the input of the Actor network to update the strategic path generation strategy.

[0164] The outputs of both the Actor and Critic networks are connected to the input of the interaction update module to apply the policy path to the environment and obtain feedback information.

[0165] The output of the interaction update module, namely the environment state change and the reward information obtained, is connected to the input of the Actor network and the Critic network to update their parameters.

[0166] The three parts of the Actor network, Critic network and interactive update module are interconnected to form a closed-loop learning process. By continuously interacting with the environment, the parameters of the Actor network and Critic network are continuously updated, ultimately enabling the Actor network to generate the optimal strategy path.

[0167] The path search module is responsible for deleting nodes in the obtained strategy path, and then using the aforementioned method to complete the path discovery search to obtain multiple adversarial game strategy paths.

[0168] The strategy fusion module is responsible for evaluating the effectiveness of the confrontation game based on the confrontation game strategy path given by the confrontation game strategy knowledge graph and the evidence combination theory. It uses the particle swarm algorithm to obtain the optimal confrontation game strategy fusion result of the radar target and obtain the best target confrontation game strategy.

[0169] The output of the path search module, i.e., multiple adversarial game strategy paths, is connected to the input of the strategy fusion module as the basis for strategy fusion.

[0170] The output of the strategy fusion module, namely the optimal target adversarial game strategy, is the final decision result output.

[0171] The path search module and the policy fusion module are interconnected, forming a policy fusion process that starts with the acquisition of multiple policy paths and then fuses them together, ultimately achieving the optimal decision. The output of each module forms the basis for the input of the next module, ensuring the integrity and effectiveness of the policy fusion process.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A knowledge graph-based equipment dynamic confrontation game decision-making method, characterized by: The method comprises the following steps: S1: Determine the endpoint of the reasoning strategy based on the knowledge graph, including: By utilizing the knowledge of radar target attribute characteristics in the knowledge graph and the various technical parameters of radar targets acquired and analyzed in real time, the following reasoning analysis is performed based on knowledge-driven bottom-up analysis: Using various attribute parameters, real-time parameters, and parameter change characteristics that reflect radar target characteristics as underlying driving data, the similarity, change rate, and characteristic values ​​of radar target types and state characteristics are extracted; Further feature value analysis is performed to obtain feature facts, which are then matched with the radar target feature knowledge given by the knowledge graph. The matching result is the radar target recognition result of the radar target type and working mode, which is the end point of the adversarial game strategy path. S2: Construct a Markov Decision Process (MDP) model for adversarial game strategy path search, including: Convert the strategy path solution of the adversarial game into a sequential decision problem; Construct a Markov decision process model (S, A, P, R) for adversarial game strategy reasoning path search, where: S: represents the state, which is defined as the set of nodes on the adversarial game strategy path obtained from the knowledge graph, that is, the set of adversarial game operations; A: Indicates the action, that is, whether each node in the knowledge graph is selected; P: represents the transition probability, which is the probability that the state will change to the next state S' after executing action A in the current state S; R: Reward function R(S,A), which is the immediate reward obtained after selecting action A at the current state S, that is, the corresponding node on the adversarial game strategy path. It is the evaluation of the adversarial effect after selecting a certain adversarial strategy operation. Through the above Markov decision process model, the strategy path node selection action value function Q(A,S) is obtained; S3: Based on the Actor-Critic deep reinforcement learning method, we solve the action sequence that maximizes the value of Q(A,S). This sequence corresponds to the optimal adversarial game strategy reasoning path in the adversarial experience knowledge graph, including: The reasoning decision network training is completed through the interactive update between the Actor network and the Critic network and the environment; the environment is an adversarial game situation; The optimal path node sequence decision is based on solving the strategic path obtained by using the environment and state as inputs to the Actor network; where the state is the current path selection; S4: Dynamic knowledge graph multi-adversarial game strategy path comprehensive strategy, including: Delete nodes from the acquired strategy path, and then use the aforementioned method to complete the search for other strategy paths to obtain multiple adversarial game strategy paths; Based on the adversarial game strategy path and adversarial game effect evaluation given by the adversarial game strategy knowledge graph, the optimal adversarial game strategy fusion method is used to combine multiple adversarial interference strategy paths to obtain the optimal strategy for the target adversarial game.

2. The equipment dynamic confrontation game decision-making method based on knowledge graph according to claim 1 is characterized by: The S4 is obtained by fusing the strategies corresponding to multiple adversarial game strategy paths. First, based on the evidence theory method, multiple evaluation indicators describing the effectiveness of the fusion strategy are used as evidence focal elements to obtain the basic probability assignment of the strategy. The effectiveness of the fusion strategy is calculated using the strategy combination rule method of evidence theory. Then, a swarm optimization algorithm is used in combination with the effectiveness of the fusion strategy to obtain the optimal strategy solution for the adversarial game, thereby realizing strategy fusion.

3. A knowledge graph-based equipment dynamic confrontation game decision-making device that applies the equipment dynamic confrontation game decision-making method of claim 1, characterized in that: The device includes: The knowledge graph reasoning module is used to determine the endpoint of the adversarial strategy and the strategy reasoning solution path based on the knowledge graph; Markov decision process model building module, used to build a Markov decision process model for strategic path search; Deep reinforcement learning module, which uses the actor-critic based deep reinforcement learning method to solve the dynamic confrontation game strategy path; Multi-path adversarial game strategy synthesis module, used to synthesize multiple dynamic game decision strategies based on knowledge graphs; The output of the knowledge graph reasoning module, namely the target recognition results and the adversarial game reasoning path nodes, is directly connected to the input of the Markov decision process model construction module for adversarial games as the basis for state definition; The output of the Markov decision process model building module, namely the path node selection action value function Q(A,S), is connected to the input of the deep reinforcement learning module to solve the adversarial game reasoning strategy; The output of the deep reinforcement learning module, i.e., the optimal adversarial game strategy reasoning path, is connected to the input of the multi-path reasoning strategy synthesis module as the basis for strategy fusion; The output of the multi-adversarial game strategy path synthesis module, namely the optimal target adversarial game strategy, is output as the final decision result.

4. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 3 is characterized by: The knowledge graph reasoning module includes: Target recognition module, used to identify target types and working modes based on knowledge graphs; Feature extraction module, used to extract the similarity, change rate and feature values ​​of feature points that reflect the target type and state characteristics; The matching module is used to match the feature facts with the target feature knowledge given by the knowledge graph; A search module is used to search and obtain target recognition results of target type and working mode based on the matching results; The target recognition module is responsible for receiving target data and identifying the target using the target attribute feature knowledge stored in the knowledge graph; its output is the identified target type and working mode; The feature extraction module receives the output from the target recognition module, namely the target type and working mode, and extracts the feature values ​​reflecting the similarity, change rate and feature points of the target type and state characteristics in combination with the target technical parameters acquired in real time; The matching module receives the output from the feature extraction module, i.e., the feature value, and matches it with the target feature knowledge stored in the knowledge graph to determine the matching degree between the target feature and the target knowledge in the knowledge graph; The search module receives the output from the matching module, i.e., the matching result, and searches the knowledge graph to obtain a structure that matches the target features, and finally outputs the target recognition result of the target type and working mode; The output of the state definition module, i.e., the state space, is used as the input of the action definition module and the transition probability calculation module to define the action space and calculate the transition probability; The output of the action definition module, i.e., the action space, is used to select appropriate nodes from the knowledge graph module as candidate adversarial game strategy path nodes, and is also used as the input of the transition probability calculation module to calculate the transition probability. The output of the transition probability calculation module, that is, the transition probability matrix, together with the output of the state definition module, is used as the input of the reward function definition module to define the reward function; The output of the reward function definition module, namely the reward function, is used to evaluate the rewards obtained after performing an action in each state and is an important part of the Markov decision process model.

5. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 3 is characterized by: The Markov decision process model building module includes: A state definition module, used to define each node on the adversarial game strategy path, wherein the node describes an alternative adversarial game strategy; The action definition module is used to define whether each node in the knowledge graph is selected. The action actually corresponds to the selection of a certain interference countermeasure operation; The transition probability calculation module is used to calculate the probability that the environment state will change to the next state after performing an action in the current state; The reward function definition module is used to define the reward function R(S,A), where S represents the current state and A represents the executed action. The reward function is a scalar function of the current state and action, representing the immediate reward obtained after selecting the action to be executed in the current state. The reward can essentially be regarded as an evaluation of the interference effect after the action selects a certain interference countermeasure operation.

6. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 3 is characterized by: The Deep Reinforcement Learning module includes: Actor network, used to generate adversarial game strategy reasoning paths; Critic network, used to evaluate the value of the adversarial game strategy reasoning path; The interactive update module is used to interactively update the Actor network and the Critic network with the environment, where the environment represents the knowledge graph; The output of the Actor network, i.e., the adversarial game strategy reasoning path, is connected to the input of the Critic network to evaluate the path value; The output of the Critic network, i.e. the path value evaluation result, is connected to the input of the Actor network and is used to update the adversarial game strategy reasoning path generation strategy; The outputs of the Actor and Critic networks are connected to the input of the interactive update module, which is used to apply the adversarial game strategy reasoning path to the environment and obtain feedback information; The output of the interaction update module, namely the environment state change and the reward information obtained, is connected to the input of the Actor network and the Critic network to update their parameters.

7. The equipment dynamic confrontation game decision-making device based on knowledge graph according to claim 4 is characterized by: The multi-path confrontation game strategy integration module includes: A path search module is used to delete nodes from the acquired adversarial game strategy path, and then use the aforementioned method to complete the adversarial game strategy path discovery search to obtain multiple adversarial game strategy paths; The strategy fusion module is used to use the adversarial game strategy path and its adversarial game effect evaluation given by the adversarial game strategy knowledge graph, realize the integration of multiple adversarial game strategy paths through strategy comprehensive optimization, and obtain the optimal target adversarial game strategy; The output of the path search module, i.e., multiple adversarial game strategy paths, is connected to the input of the strategy fusion module as the basis for strategy fusion; The output of the strategy fusion module, that is, the optimal target adversarial game strategy, is output as the final decision result.

Images