Construction method of killing chain generation model

Abstract

The invention discloses a method for constructing a killing chain generation model, which comprises the following steps of: S1, defining an entity, instantiating attributes of the entity, acquiring artificial experience and battlefield information, and constructing a constraint relation knowledge graph according to the artificial experience and the battlefield information; s2, constructing a killing chain generation agent according to the attribute of the entity; s3, abstracting the constraint relation knowledge graph into an action space mask; and S4, training a killing chain generation agent through a deep reinforcement learning algorithm and an action space mask, and obtaining a killing chain generation model. According to the method, the problems of complex selectable actions, more selected branch networking and huge action space in a large-scale system combat condition are solved, the volume of an intelligent agent model is reduced, the model training and convergence speed is accelerated, and an intelligent agent continuous evolution mode is provided; the killing chain is generated through the killing chain generation model, the problems that a traditional method is long in calculation time and slow in decision making are solved, and battle tasks with complex and changeable battlefield situations can be achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of equipment command and control, and in particular relates to a method for constructing a kill chain generation model. Background Art

[0002] Kill chain generation technology aims to form a combat system with multi-domain aggregation and cross-domain coordination as the core by integrating detection resources, weapon platforms and command and control nodes on the battlefield. It is one of the main research difficulties of modern system defense operations. Its research content mainly targets large-scale incoming targets, and can quickly match detection, interception, and command and control equipment to different targets to form combat capabilities and eliminate target threats. As future wars place increasingly higher demands on combat systems, the linear and fixed matching mode of traditional kill chains can no longer meet combat needs. The construction of a dynamically variable kill chain makes combat more flexible and changeable, thereby improving overall combat capabilities. Intelligent methods can quickly process massive amounts of data, discover threatening targets, and synchronize firepower through intelligent task allocation, optimize combat coordination, and significantly improve the response speed and efficiency of combat.

[0003] However, how to build a reasonable and effective kill chain generation intelligent model that can not only meet the complexity and adaptability of different threat scenarios, but also meet the requirements of combat efficiency and have the ability to continuously evolve to cope with the challenges of future large-scale system warfare is the difficulty and pain point of current kill chain generation technology research. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a.

[0005] The present invention is achieved through the following technical solutions.

[0006] The present invention provides a method for constructing a kill chain generation model, comprising the following steps:

[0007] S1. Define entities, instantiate the attributes of entities, obtain artificial experience and battlefield intelligence, and build a constraint relationship knowledge graph based on artificial experience and battlefield intelligence. The entities include targets, detection equipment, interception equipment, and command and control equipment.

[0008] S2, build a kill chain to generate an intelligent agent based on the attributes of the entity;

[0009] S3, abstract the constraint relationship knowledge graph into an action space mask;

[0010] S4. Train the kill chain generation agent through deep reinforcement learning algorithm and action space mask to obtain the kill chain generation model.

[0011] Preferably, the attributes of the entity include: target type, flight speed, target RCS, target threat level;

[0012] The properties of the detection equipment include: detection range, detection accuracy, detectable target type, and number of detection channels;

[0013] The properties of the interception equipment include: interception area, interceptable target type, number of ammunition, number of interception channels, and single-shot kill probability;

[0014] The attributes of the command equipment include: response time to different equipment instructions.

[0015] Preferably, the step S2, constructing a kill chain according to the attributes of the entity to generate an intelligent agent, comprises the following steps:

[0016] Construct state space based on the attributes of entities;

[0017] Obtain equipment classification and construct action space based on the equipment classification;

[0018] Setting up decision-making space;

[0019] Construct a kill chain to generate an intelligent agent based on the state space, action space, and decision space.

[0020] Preferably, the state space includes a target state space, a detection equipment state space, an interception equipment state space and a command equipment state space;

[0021] The target state space expression is as follows:

[0022] <Target 1: N1: DE1><Target 2: N2: DE2>……<Target n: N n :DE n >

[0023] Among them, N n Indicates the number of n-type targets, DE n Indicates the threat level of type n target;

[0024] The state space expression of the detection equipment is as follows:

[0025] <Detection 1: C1: Dis1: P1><Detection 2: C2: Dis2: P2>……<Detection d: C d :Dis d :P d >

[0026] Among them, C d Indicates the number of channels of the d-type detection equipment, Dis d Indicates the detection distance of type d detection equipment, P d Indicates the detection accuracy of type D detection equipment;

[0027] The interception equipment state space expression is as follows:

[0028] <Intercept 1: HN1: HC1><Intercept 2: HN2: HC2>……<Intercept h: HN h :HC h >

[0029] Among them, HN h Indicates the number of h-type interception equipment, HC h Indicates the number of channels of the h-type interception equipment;

[0030] The state space expression of the command equipment is as follows:

[0031] <Allegation 1: GN1><Allegation 2: GN2>……<Allegation g: GN g >

[0032] Among them, GN g Indicates the number of G-type accusation equipment.

[0033] Preferably, the expression of the action space is as follows:

[0034] {A nd =(i,DY ij )|i=1,2,...,n,j=1,2,...,d}

[0035] ∩{A nh =(i,HY ij )|i=1,2,...,n,j=1,2,...,h}

[0036] ∩{A dh =(i,HY ij )|i=1,2,...,d,j=1,2,...,h}

[0037] Among them, DY ij Indicates the selection status of the detection equipment of type j for target i, HY ij Indicates the selection status of the detection equipment of type j for target i, DH ij Indicates the selection status of interception equipment j for detection equipment i;

[0038] If DY ij =1 means the target-detection action is selected, DY ij =0 means the action is not selected; if HY ij =1 means the target-interception action is selected, HY ij =0 means the action is not selected; if DH ij =1 means the detection-interception action is selected, DH ij =0 means the action is not selected.

[0039] Preferably, the decision space includes positive reward decisions and negative reward decisions.

[0040] Preferably, the positive reward decision includes: when the target has been detected, tracked and is being intercepted, a positive reward is given with the threat level and the strike probability as coefficient factors; when the target threat level drops below a threshold, a positive reward is given that is positively correlated with the original threat level of the target.

[0041] Preferably, the negative reward decision includes: if the threat level of the target is lower than the set threshold and interception weapons are still assigned, a negative reward positively correlated with the original threat level of the target is given; if the threat level of the target is lower than the set threshold and detection weapons are still assigned, a negative reward positively correlated with the original threat level of the target is given; if both detection equipment and interception equipment are remaining and there are unassigned targets, a negative reward with a larger weight is given; if the number of remaining channels of the detection equipment and interception is 0 or the number of remaining ammunition of the interception equipment is 0, a negative reward is given.

[0042] Preferably, the constraint relationship knowledge graph includes a target-detection relationship, a target-interception relationship and a detection-interception relationship.

[0043] Preferably, the step S4, training a kill chain generation agent through a deep reinforcement learning algorithm and an action space mask, and obtaining a kill chain generation model comprises the following steps:

[0044] Get the SAC network architecture;

[0045] Introduce value distribution learning theory on the SAC network architecture to obtain the DSAC framework, and use the DSAC framework as the reinforcement learning algorithm framework;

[0046] The action space mask is embedded in the reinforcement learning algorithm framework to train the kill chain generation agent and obtain the kill chain generation model.

[0047] The beneficial effects of the present invention are:

[0048] 1. The present invention converts artificial experience and intelligence information into an action space mask by constructing a knowledge graph based on rule constraints, so as to embed rule constraints into the training process of the deep reinforcement learning algorithm. This solves the problems of complex optional actions, large number of branch networking options, and huge action space in large-scale system combat situations, reduces the size of the intelligent agent model, speeds up model training and convergence, and provides a continuous evolution mode for the intelligent agent.

[0049] 2. Generating the kill chain through the kill chain generation model solves the problems of long calculation time and slow decision-making in traditional methods, which is conducive to achieving combat tasks with complex and changeable battlefield situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a method provided by an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of a model structure provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of the SAC network architecture structure provided by an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of a kill chain provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described below, but the scope of protection claimed is not limited to the description.

[0055] like Figure 1 As shown, a method for constructing a kill chain generation model includes the following steps:

[0056] S1. Define entities, instantiate the attributes of entities, obtain artificial experience and battlefield intelligence, and build a constraint relationship knowledge graph based on artificial experience and battlefield intelligence. The entities include targets, detection equipment, interception equipment, and command and control equipment.

[0057] The entities are defined to include four types of entities, namely targets, detection equipment, interception equipment and command and control equipment.

[0058] The instantiation of the attributes of the entity specifically refers to: instantiation of the attributes of the target, the attributes of the detection equipment, the attributes of the interception equipment and the attributes of the command and control equipment;

[0059] The attributes of the entity include: target type, flight speed, target RCS, and target threat level;

[0060] The properties of the detection equipment include: detection range, detection accuracy, detectable target type, and number of detection channels;

[0061] The properties of the interception equipment include: interception area, interceptable target type, number of ammunition, number of interception channels, and single-shot kill probability;

[0062] The attributes of the command equipment include: response time to different equipment instructions.

[0063] like Figure 2 As shown, artificial experience information and battlefield intelligence information are extracted, the relationships between entities are connected, edge relationships and relationship weights are assigned, and a constraint relationship knowledge graph is formed.

[0064] S2, build a kill chain to generate an intelligent agent based on the attributes of the entity;

[0065] The step S2, constructing a kill chain according to the attributes of the entity to generate an intelligent agent, comprises the following steps:

[0066] Construct state space based on the attributes of entities;

[0067] In this embodiment, four types of entities, namely, target, detection equipment, interception equipment and command and control equipment, are modeled to construct a state space.

[0068] The state space includes target state space, detection equipment state space, interception equipment state space and command equipment state space;

[0069] For the target entity, assuming that there are n types of enemy incoming threat targets, the threat situation of the enemy targets is represented by the triple of <type: quantity: threat degree>. Different targets have different threat degrees [0,1], and the larger the value, the higher the threat degree.

[0070] The target state space expression is as follows:

[0071] <Target 1: N1: DE1><Target 2: N2: DE2>……<Target n: N n :DE n >

[0072] Among them, N n express n The number of type targets, DE n Indicates the threat level of type n target;

[0073] Regarding the detection equipment entity, assuming that our defense system has a total of D types of detection equipment, the detection equipment perception capability is represented by the four-tuple of <type: number of channels: detection distance: detection accuracy>.

[0074] The state space expression of the detection equipment is as follows:

[0075] <Detection 1: C1: Dis1: P1><Detection 2: C2: Dis2: P2>……<Detection d: C d :Dis d :P d >

[0076] Among them, C d Indicates the number of channels of the d-type detection equipment, Dis d Indicates the detection distance of type d detection equipment, P d Indicates the detection accuracy of type D detection equipment;

[0077] For the interception equipment entity, assuming that there are h types of interception equipment in our defense system, the interception equipment status is represented by the triple of <type: quantity: number of channels>.

[0078] The interception equipment state space expression is as follows:

[0079] <Intercept 1: HN1: HC1><Intercept 2: HN2: HC2>……<Intercept h: HN h :HC h >

[0080] Among them, HN h Indicates the number of h-type interception equipment, HC h Indicates the number of channels of the h-type interception equipment;

[0081] The single-shot kill probability of different interception equipment against different targets is expressed by matrix P:

[0082]

[0083] Among them, p hn represents the probability of a single shot kill of target n by interception weapon h. If p hn =0 means that the interception weapon h does not have the ability to intercept the target n.

[0084] For the C&D equipment entity, assuming that our defense system has a total of g types of C&D equipment, the C&D equipment status is represented by a <type: quantity> tuple.

[0085] The state space expression of the command equipment is as follows:

[0086] <Charge 1: GN1><Charge 2: GN2>……<Charge g: GN g >

[0087] Among them, GN g Indicates the number of G-type accusation equipment.

[0088] Obtain equipment classification and construct action space based on the equipment classification;

[0089] In this embodiment, we obtain that our defense system has detection equipment of category D, interception equipment of category H, and accusation equipment of category G.

[0090] Construct action space according to equipment classification and use DY ij Indicates the selection status of the detection equipment of type j for target i, using HY ij Indicates the selection status of the detection equipment of type j for target i, using DH ij Indicates the selection status of interception equipment j for detection equipment i.

[0091] By DY ij HY ij DH ijConstruct a ternary selectable relationship matrix and then select the accusation to obtain the action space.

[0092] The expression of the action space is as follows:

[0093] {A nd =(i,DY ij )|i=1,2,...,n,j=1,2,...,d}

[0094] ∩{A nh =(i,HY ij )|i=1,2,...,n,j=1,2,...,h}

[0095] ∩{A dh =(i,HY ij )|i=1,2,...,d,j=1,2,...,h}

[0096] Among them, DY ij Indicates the selection status of the detection equipment of type j for target i, HY ij Indicates the selection status of the detection equipment of type j for target i, DH ij Indicates the selection status of interception equipment j for detection equipment i;

[0097] If DY ij =1 means the target-detection action is selected, DY ij =0 means the action is not selected; if HY ij =1 means the target-interception action is selected, HY ij =0 means the action is not selected; if DH ij =1 means the detection-interception action is selected, DH ij =0 means the action is not selected.

[0098] It also includes the detection and interception equipment that matches each target, the matching of detection and interception equipment, etc. Select any action from the action space to execute.

[0099] Setting up decision-making space;

[0100] The decision space includes positive reward decisions and negative reward decisions.

[0101] The positive reward decision includes: if the target has been detected, tracked and is being intercepted, a positive reward with threat level and strike probability as coefficient factors is given; if the target threat level drops below a threshold, a positive reward positively correlated with the original threat level of the target is given.

[0102] The negative reward decision includes: if the threat level of the target is lower than the set threshold and interception weapons are still assigned, a negative reward positively correlated with the original threat level of the target is given; if the threat level of the target is lower than the set threshold and detection weapons are still assigned, a negative reward positively correlated with the original threat level of the target is given; if both detection equipment and interception equipment are remaining and there are unassigned targets, a negative reward with a larger weight is given; if the number of remaining channels of detection equipment and interception is 0 or the number of remaining ammunition of interception equipment is 0, a negative reward is given.

[0103] Construct a kill chain to generate an intelligent agent based on the state space, action space, and decision space.

[0104] That is, the state space, action space and decision space are connected to obtain the kill chain generation agent. The state space S is converted into the action space A through the Actor network. In the process of calculating the action space A, the A-mask mask model formed by the knowledge graph modeling based on constraint rules is introduced. The filtering of the inaction action is completed in the action space A. The generated target, detection, interception, and accusation vectors constitute the action selection result. The matching relationship is the equipment ID corresponding to each column element, thus forming a target-detection-interception-accusation link. The state space S and the action space A are input into the Critic network for decision space update and drive model training.

[0105] S3, abstract the constraint relationship knowledge graph into an action space mask;

[0106] The constraint relationship knowledge graph includes a target-detection relationship, a target-interception relationship and a detection-interception relationship.

[0107] In this embodiment, abstracting the constraint relationship knowledge graph into an action space mask includes the following steps:

[0108] Extract the target-detection relationship and model it as a matrix Wtd ij ,

[0109]

[0110] Among them, w ij =1 means the action is feasible, w ij =0 means the action is not feasible.

[0111] The target-interception relationship is extracted and modeled as the matrix Wth ij ,

[0112]

[0113] Among them, w ij =1 means the action is feasible, w ij =0 means the action is not feasible.

[0114] Extract the detection-interception relationship and model it as the matrix Wdh ij ,

[0115]

[0116] in, The value range of is [0,1], Indicates that the action is not possible. Indicates that the action is completely feasible, and other cases are feasible according to probability. ξ represents the matching factor of different types of detection equipment and interception equipment.

[0117] That is, the action space mask is obtained.

[0118] S4. Train the kill chain generation agent through deep reinforcement learning algorithm and action space mask to obtain the kill chain generation model.

[0119] S4, training the kill chain generation agent through the deep reinforcement learning algorithm and the action space mask, and obtaining the kill chain generation model includes the following steps:

[0120] In this embodiment, the deep reinforcement learning algorithm used is the DSAC framework. The DSAC framework is based on the SAC network architecture and introduces the value distribution learning theory.

[0121] Get the SAC network architecture;

[0122] The obtained SAC network architecture is as follows: Figure 3 As shown in the figure, the Actor uses the behavior strategy a generated by the strategy network to act on the environment, generate a new state s', and put this group of states into the experience replay pool; the Critic puts the original state s, current state s' and action a into the estimation network for calculation, and updates the network parameters through the loss function, and the target network completes the calculation of this round of action execution.

[0123] The value distribution learning theory is introduced on the SAC network architecture to obtain the DSAC framework, which is used as the reinforcement learning algorithm framework.

[0124] The value distribution function is used to represent the reward distribution of the intelligent agent in different states or state-action pairs, which takes uncertainty into account more comprehensively. It can accurately reflect the probability distribution of different reward values ​​in tasks dealing with noise and randomness, and also significantly improves the over-estimation problem of the value function, reduces the phenomenon of unstable strategy performance during the learning process, and improves the strategy performance after convergence.

[0125] The action space mask is embedded in the reinforcement learning algorithm framework to train the kill chain generation agent and obtain the kill chain generation model.

[0126] In this embodiment, the Actor part in the DSAC framework embeds the action space mask matrix, effectively shielding the infeasible space and reducing the complexity of action selection. In the case of no less than 60,000 iterations, the optimal dynamic matching of target-detection-interception-accusation can be achieved.

[0127] In this embodiment, the kill chain is generated by the acquired kill chain generation model to obtain the target-detection-interception-command kill chain solution generation result, such as Figure 4 As shown in the figure, detection is represented by the letter S, interception is represented by the letter D, and accusation is represented by the letter I. For example, the kill chain for target 1 is target 1-detection 2-interception 2-accusation 3-target 1.

[0128] The present invention converts artificial experience and intelligence information into action space mask by constructing a knowledge graph based on rule constraints, so as to embed rule constraints into the training process of deep reinforcement learning algorithm, solve the problems of complex optional actions, more branch networking, and huge action space in large-scale system combat situations, reduce the volume of intelligent agent model, speed up model training and convergence, and provide a continuous evolution mode of intelligent agent. The kill chain generation model is used to generate the kill chain, which solves the problems of long calculation time and slow decision-making of traditional methods, and is conducive to realizing the combat tasks with complex and changeable battlefield situations.

Claims

1. A method for constructing a kill chain generation model, characterized in that: The following steps are involved: S1. Define entities, instantiate the attributes of entities, obtain artificial experience and battlefield intelligence, and build a constraint relationship knowledge graph based on artificial experience and battlefield intelligence. The entities include targets, detection equipment, interception equipment, and command and control equipment. S2, build a kill chain to generate an intelligent agent based on the attributes of the entity; S3, abstract the constraint relationship knowledge graph into an action space mask; S4. Train the kill chain generation agent through deep reinforcement learning algorithm and action space mask to obtain the kill chain generation model.

2. The construction method according to claim 1, characterized in that: The attributes of the target include: target type, flight speed, target RCS, and target threat level; The properties of the detection equipment include: detection range, detection accuracy, detectable target type, and number of detection channels; The properties of the interception equipment include: interception area, interceptable target type, number of ammunition, number of interception channels, and single-shot kill probability; The attributes of the command equipment include: response time to different equipment instructions.

3. The construction method according to claim 1, characterized in that: The step S2, constructing a kill chain according to the attributes of the entity to generate an intelligent agent, comprises the following steps: Construct a state space based on the attributes of the entity; Obtain equipment classification and construct action space based on equipment classification; Setting up decision-making space; Construct a kill chain to generate an intelligent agent based on the state space, action space, and decision space.

4. The construction method according to claim 3, characterized in that: The state space includes target state space, detection equipment state space, interception equipment state space and command equipment state space; The target state space expression is as follows: <Objective 1: N1: DR1><Objective 2: N2: DE2>……<Objective n: N n : DE n > Among them, N n Indicates the number of n-type targets, DE n Indicates the threat level of type n target; The state space expression of the detection equipment is as follows: < probed1:C1:Dis1:P1>< probed2:C2:Dis2:P2>……< probed:C d :Dis d :P d > Among them, C d Indicates the number of channels of the d-type detection equipment, Dis d Indicates the detection distance of type d detection equipment, P d Indicates the detection accuracy of type D detection equipment; The interception equipment state space expression is as follows: <Interception 1: HN1: HC1><Interception 2: HN2: HC2>……<Interception h: HN h : HC h > Among them, HN h Indicates the number of h-type interception equipment, HC h Indicates the number of channels of the h-type interception equipment; The state space expression of the command equipment is as follows: <Accusation 1: GN1><Accusation 2: GN2>……<Accusation g: GN g > Among them, GN g Indicates the number of G-type accusation equipment.

5. The construction method according to claim 3, characterized in that: The expression of the action space is as follows: {A nd (i,DY ij )|i=1,2,…,n,j=1,2,…,d}∩{A nh =(i,HY ij )|i=1,2,…,n,j=1,2,…,h}∩{A dh =(i,HY ij )|i=1,2,...,d,j=1,2,...,h} Among them, DY ij Indicates the selection status of the detection equipment of type j for target i, HY ij Indicates the selection status of the detection equipment of type j for target i, DH ij Indicates the selection status of interception equipment j for detection equipment i; If DY ij =1 means the target-detection action is selected, DY ij =0 means the action is not selected; if HY ij =1 means the target-interception action is selected, HY ij =0 means the action is not selected; if DH ij =1 means the detection-interception action is selected, DH ij =0 means the action is not selected.

6. The construction method according to claim 3, characterized in that: The decision space includes positive reward decisions and negative reward decisions.

7. The construction method according to claim 6, characterized in that: The positive reward decision includes: if the target has been detected, tracked and is being intercepted, a positive reward with threat level and strike probability as coefficient factors is given; if the target threat level drops below a threshold, a positive reward positively correlated with the original threat level of the target is given.

8. The construction method according to claim 6, characterized in that: The negative reward decision includes: if the threat level of the target is lower than the set threshold and interception weapons are still assigned, a negative reward positively correlated with the original threat level of the target is given; if the threat level of the target is lower than the set threshold and detection weapons are still assigned, a negative reward positively correlated with the original threat level of the target is given; if both detection equipment and interception equipment are remaining and there are unassigned targets, a negative reward with a larger weight is given; if the number of remaining channels of detection equipment and interception is 0 or the number of remaining ammunition of interception equipment is 0, a negative reward is given.

9. The construction method according to claim 1, characterized in that: The constraint relationship knowledge graph includes a target-detection relationship, a target-interception relationship and a detection-interception relationship.

10. The construction method according to claim 1, characterized in that: S4, training the kill chain generation agent through the deep reinforcement learning algorithm and the action space mask, and obtaining the kill chain generation model includes the following steps: Get the SAC network architecture; Introduce value distribution learning theory on the SAC network architecture to obtain the DSAC framework, and use the DSAC framework as the reinforcement learning algorithm framework; The action space mask is embedded in the reinforcement learning algorithm framework to train the kill chain generation agent and obtain the kill chain generation model.

Images