Network node security measure mitigation deployment optimization method and system based on deep reinforcement learning

Abstract

The present application belongs to the technical field of network security, and particularly relates to a network node security measure mitigation deployment optimization method and system based on deep reinforcement learning. The network node security measure mitigation deployment is converted into a multi-objective optimization problem by using a deep reinforcement learning mitigation measure model. The deep reinforcement learning mitigation measure model is based on a security knowledge base and establishes the relationship between attacks and defenses connected by vulnerabilities by using the security knowledge base. The problem space is constructed according to the action space, the state space and the reward function, and the mitigation deployment is optimized in the problem space by using the deep reinforcement learning method. The present application can comprehensively consider the importance weight of the deployment node, the deployment cost, the vulnerability repair and the attack effectiveness, establish a bridge between the vulnerability, the mitigation measure and the attack technology, and better maintain the decision of the mitigation action, and has potential in controlling the network defense cost of large organizations.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network security, and particularly relates to a network node security measure mitigation deployment optimization method and system based on deep reinforcement learning. BACKGROUND

[0002] Mitigation deployment for specific attacks has become a key problem in network security research and application. The mainstream mitigation measure deployment method is usually those relying on attack graphs and machine learning, which conventionally conducts risk assessment to obtain the most vulnerable device, and then deploys mitigation measures against network attacks to complete effective defense. Although great achievements have been made, how to minimize redundancy while ensuring effective defense against network attacks is still an open problem in mitigation measure deployment. For a long time, large organizations have been suffering from the inefficiency and high cost generated by this work, and even if excessive redundancy is paid at the cost, it cannot guarantee its effectiveness. The CVSS (Common Vulnerability Scoring System) mitigation deployment task workflow is shown in Figure 2 In the early stage, the attack path can be determined by estimating the probability of attacking network assets under the guidance of BAG (Bayesian Attack Graph). Appropriate mitigation measures can be deployed to vulnerable nodes with reference to the attack graph to improve the overall security level. Efforts have been made in this direction to improve risk assessment, thereby optimizing mitigation measures by finding the best attack path, and considering the possibility of system damage and protection of critical infrastructure. It should be noted that BAG is not enough for large dynamic networks, because the super-high complexity of constructing (updating) attack graphs makes it difficult to accurately deploy mitigation measures within time limits, and redundancy is inevitable. After that, this problem is considered as a decision problem between subjects with conflicting goals in the context of game theory. By balancing the Nash equilibrium of deployment cost and defense benefits, the "best" defense strategy can be obtained. For example, the purpose of Bayesian game is to find the best strategy by realizing the Bayesian Nash equilibrium point with the probability information in the scene under the uncertainty in the Internet. In this case, the optimal solution can be obtained by considering the cost and defense benefits through an abstract attack-defense level mathematical model. However, how to determine the exact location of defense deployment is still unclear, although it is a key factor for defense effect.

[0003] Recently, machine learning methods have been flourishing in cyber security applications, as these methods have been proven to be very powerful in exploring / implementing mitigation deployment strategies to achieve given security goals. Because the task of mitigation deployment is essentially to find solutions for multi-objective problems involving dynamic key factors in cyber security tasks, and machine learning is excellent in this regard. Mitigation deployment methods employing Q-learning and deep reinforcement learning have achieved great success in dealing with increasingly complex cyber attack problems. It is worth noting that it is still not a trivial matter to minimize the deployment cost, because the effectiveness of defense is usually guaranteed at the cost of excessive redundancy. Formulating the relationship between attack and defense is necessary for any mitigation deployment method to achieve its goals. Security knowledge bases like MITRE ATT&CK are often applicable, which can simulate the behavior of network adversaries, the attack life cycle of adversaries, known target platforms, and more importantly, the mitigation measures of existing defenses. Then, the defender can take actions to mitigate attacks by referring to such security knowledge bases. However, existing security knowledge bases mainly focus on attacks and defenses, and the role of vulnerabilities is not explicitly formulated, so that the role of vulnerabilities is not focused on. There is still an urgent need for mitigation measures in terms of deployment to address the need to narrow down the scope of vulnerable nodes and maintain a low level of redundancy while ensuring the effectiveness of mitigation measures. SUMMARY

[0004] To this end, the present application provides a network node security measure mitigation deployment optimization method and system based on deep reinforcement learning, which comprehensively considers the importance weight of deployed nodes, deployment cost, vulnerability repair and attack effectiveness, establishes a bridge between vulnerability, mitigation measure and attack technology, and can better maintain the decision-making of mitigation actions.

[0005] According to the design scheme provided by the present application, a network node security measure mitigation deployment optimization method based on deep reinforcement learning is provided, which comprises the following contents:

[0006] A deep reinforcement learning mitigation measure model is constructed, and the deep reinforcement learning mitigation measure model is used to convert network node security mitigation measure deployment into a multi-objective optimization problem, wherein the deep reinforcement learning mitigation measure model is run based on a security knowledge base and uses the security knowledge base to establish the relationship between attack and defense connected by vulnerabilities, constructs a problem space according to an action space, a state space and a reward function, and optimizes mitigation deployment in the problem space through a deep reinforcement learning method;

[0007] For the target network, the attack techniques and mitigation measures in the network are obtained by using the security knowledge base, and the node weights are allocated according to the node functions in the target network. The attack techniques, mitigation measures and node importance weights in the target network are used as inputs of the deep reinforcement learning mitigation measure model, and the best mitigation deployment strategy in the target network is obtained by solving the deep reinforcement learning mitigation measure model.

[0008] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning in the application, further, the security knowledge base is an open opponent tactics and technology knowledge base, which includes an ATT&CK knowledge base and a CVE knowledge base.

[0009] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning in the application, further, in the problem space, each mitigation action that may occur in the network at a time step is obtained according to the security knowledge base, and an action space is formed by each mitigation action. In each time step, a state space is formed by each attack technique blocking state. A mixed reward function is constructed by deploying node importance weights, rewards for preventing mitigation measures from affecting all attack techniques, rewards for repairing vulnerabilities by mitigation measures and deployment costs, and the mixed reward function is used to balance the deployment of node importance, the prevention of network attack effect, the repair of vulnerability effect and the deployment cost.

[0010] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning in the application, further, the mixed reward function is expressed as: Where w is the deployment node importance weight, r tech is the reward for preventing mitigation measures from affecting all attack techniques, r vul is the reward for repairing vulnerabilities by mitigation measures, c is the deployment cost, e techi is the influence of mitigation measures on the i th attack technique, N is the number of attack techniques, CVSS v is the vector representation of the CVSS score of the repaired vulnerability v.

[0011] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning in the application, further, in the node importance weight evaluation, first, the server, network connection weight and host node weight are preliminarily set according to the experience data of the attack possibility size; then, the host function triad is obtained according to the processing file importance degree factor, access database permission factor and publicity factor, and the host node weight is obtained by using the host function triad.

[0012] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning of the present application, further, the process of obtaining the host node weight W by using the host function triad is represented as: W=1-(1-X)(1-Y)(1-Z), wherein X, Y and Z represent the processing file importance degree factor, the access database permission factor and the publicity factor respectively.

[0013] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning of the present application, further, in the deep reinforcement learning mitigation measure model solving, by constructing a multi-objective optimization objective function, and using SAC algorithm to maximize the multi-objective optimization objective function in the problem space to learn mitigation deployment, the maximum process of the objective function is used to obtain the optimal mitigation deployment strategy, wherein the multi-objective optimization objective function is represented as: Wherein, π is a candidate strategy, π * is the output optimal strategy; is the expected function, r(s t ,a t ) is the reward function; γ∈[0,1] is the discount rate, s t is the attack technology hindering state at time step t, a t is the mitigation action at time step t; T π is the trajectory distribution caused by the strategy π; α is the temperature parameter; is the entropy value of the strategy π in state s t .

[0014] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning of the present application, further, in the maximum multi-objective optimization objective function, the state value function of the strategy is set as V(s t ):=π(s t ) T [Q(s t )-αlog(π(s t ))], and two Q networks are used to construct the state value network, the state value network is trained by using the experience replay of state and action and the Q function to generate the mitigation deployment strategy, and the action distribution and the expected estimate are obtained by maximizing the reward in the strategy network.

[0015] As the network node security measure mitigation deployment optimization method based on deep reinforcement learning of the application, further, the process of learning mitigation deployment by using SAC algorithm contains the following contents: first, select mitigation action from candidate strategy, and transfer attack technology blocking state in environment, store transition in replay buffer; then, at each time step gradient, optimize deep reinforcement learning mitigation measure model parameters through state value network objective function, policy network objective function and temperature network objective function, until deep reinforcement learning mitigation measure model reaches convergence state.

[0016] Further, the application also provides a network node security measure mitigation deployment optimization system based on deep reinforcement learning, containing: model construction module and deployment optimization module, wherein,

[0017] The model construction module is used to construct deep reinforcement learning mitigation measure model, and the deep reinforcement learning mitigation measure model is used to convert network node security mitigation measure deployment into a multi-objective optimization problem, wherein the deep reinforcement learning mitigation measure model is based on security knowledge base and uses security knowledge base to establish the relationship between attacks and defenses connected by vulnerabilities, constructs problem space according to action space, state space and reward function, and optimizes mitigation deployment in problem space through deep reinforcement learning method;

[0018] The deployment optimization module is used to obtain attack technology and mitigation measure in network by using security knowledge base according to target network, and allocate node weight according to node function in target network, and the attack technology, mitigation measure and node importance weight in target network are input into deep reinforcement learning mitigation measure model, and the best mitigation deployment strategy in target network is obtained by solving deep reinforcement learning mitigation measure model.

[0019] The application has the following beneficial effects:

[0020] The application builds deep reinforcement learning mitigation measure model, comprehensively considers deployment node importance weight, deployment cost, vulnerability repair and blocking attack effectiveness, establishes a bridge between vulnerability, mitigation measure and attack technology, and more effectively defends network attack persistence, greatly reduces redundancy. And through data simulation verification, in order to achieve the same security target, the mitigation measures needed to be deployed by the scheme of the application are much less than the most advanced peers, which further illustrates that the scheme of the application has potential in controlling network defense cost of large organizations. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 It is the network node security measure mitigation deployment optimization process based on deep reinforcement learning in the embodiment;

[0022] Figure 2 It is the CVSS mitigation deployment task workflow in the embodiment;

[0023] Figure 3 A schematic diagram of a deep reinforcement learning mitigation measure model architecture in an embodiment;

[0024] Figure 4 A schematic diagram of a V-ATT&CK relationship model in an embodiment;

[0025] Figure 5 A schematic diagram of a test platform in an embodiment;

[0026] Figure 6 A schematic diagram of a WannaCry knowledge graph of V-ATT&CK in an embodiment;

[0027] Figure 7 A schematic diagram of the performance of mitigation deployment using different reinforcement learning methods in an embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the drawings and technical solutions.

[0029] The embodiments of the present application are described below with reference to the drawings and technical solutions. Figure 1 As shown in the drawings, a network node security measure mitigation deployment optimization method based on deep reinforcement learning is provided, which comprises:

[0030] S101, a deep reinforcement learning mitigation measure model is constructed, and the deep reinforcement learning mitigation measure model is used to convert network node security mitigation measure deployment into a multi-objective optimization problem, wherein the deep reinforcement learning mitigation measure model is run based on a security knowledge base and uses the security knowledge base to establish the relationship between attacks and defenses connected by vulnerabilities, constructs a problem space according to an action space, a state space and a reward function, and optimizes mitigation deployment in the problem space through a deep reinforcement learning method;

[0031] S102, for a target network, the attack techniques and mitigation measures in the network are obtained using the security knowledge base, and the node weights are allocated according to the node functions in the target network, the attack techniques, mitigation measures and node importance weights in the target network are input into the deep reinforcement learning mitigation measure model, and the best mitigation deployment strategy in the target network is obtained by solving the deep reinforcement learning mitigation measure model.

[0032] The deep reinforcement learning mitigation measure (DRL-MD) model framework is as follows: Figure 3As shown, the mitigation deployment process is considered as finding a solution to a complex multi-objective optimization problem, and the focus of the problem-solving phase is to optimize the mitigation deployment under the defined inputs and constraints in the problem space by a deep reinforcement learning method. In the present embodiment, further, the DRL-MD model can be run on an enhanced ATT&CK model (labeled as "V-ATT&CK") and establish the relationship between attacks and defenses connected by vulnerabilities to formulate the action space, state space and reward function based on V-ATT&CK in the problem space

[0033] As a preferred embodiment, further, in the problem space, each mitigation action that the network can take at a time step is obtained according to the security knowledge base, and the action space is composed of each mitigation action; the state space is composed of each attack technique hindered state in each time step; and a hybrid reward function is constructed by the deployment node importance weight, the reward for preventing the mitigation measure from affecting all attack techniques, the reward for the mitigation measure repairing the vulnerability and the deployment cost, and the hybrid reward function is used to balance the deployment node importance, the prevention of network attack effect, the repair of vulnerability effect and the deployment cost.

[0034] The recommended security knowledge base, i.e. V-ATT&CK, is a seamless integration of ATT&CK and CVE (Common Vulnerabilities and Exposures). As a global accessible knowledge base of adversary tactics and techniques based on real-world observations, the former can provide a common taxonomy for both offense and defense, and the latter can refer to CVSS and published security vulnerabilities. V-ATT&CK facilitates better understanding of the actions of attackers and how mitigation measures prevent attacks under the explicit guidance of vulnerability knowledge, so that the decision problem of mitigation actions can be reduced.

[0035] V-ATT&CK classifies the adversary tactics, techniques and procedures (TTPs) that an attacker can perform to achieve the goal represented by the tactical category, and groups the corresponding techniques and sub-techniques, wherein the key high-level components can be described as follows: tactics represent the short-term tactical goals of the adversary in the attack process; techniques describe the means by which the adversary achieves the tactical goal; sub-techniques describe more specific means by which the adversary achieves the tactical goal at a lower level than the technique; attack groups can directly use techniques, or use software that implements techniques to complete tactics; mitigation measures represent security concepts and technology categories that can be used to prevent techniques or sub-techniques from being successfully executed, such as configurations, tools and processes; vulnerabilities are exploited by attack techniques to achieve the goal of the attacker; mitigations should repair them to resist attacks. For example, Figure 4The relationships between components are shown, and when dealing with an attack group / software covered by prior knowledge, reference can be made to V-ATT&CK to obtain: 1) attack techniques used by the attack group / software; 2) mitigation measures taken to fix the vulnerabilities exploited by the attack techniques. The decision process can then be guided to the best mitigation action.

[0036] Further, in the node importance weight evaluation, the server, network connection weight and host node weight can be initially set according to the experience data of the attack possibility size; then, the host function triad is obtained according to the processing file importance degree factor, access database permission factor and publicity factor, and the host node weight is obtained by using the host function triad.

[0037] The node importance estimation constitutes the standard for prioritizing the final mitigation measure deployment location. This problem is usually identified by macro centrality measures to identify influential nodes in complex networks. However, this index cannot measure the diversity of node functions, nor can it measure the servers that store core data, which are key factors for attracting attackers. Therefore, in the embodiments of the present case, alternative node importance estimation is used to emphasize the functions of the nodes, especially the importance of storing data. The importance of the stored data can be reflected by different weights.

[0038] The basic principle of the alternative node importance estimation is to allocate weights according to the functions of the network nodes. Table 1 lists the initial experience setting of the node weight (W).

[0039] Table 1: Weight allocation of different node types

[0040]

[0041] 1. Servers (WEB servers, DB servers, and FTP servers, etc.) and network connection devices (routers, switches, and hubs): the highest weight is allocated to these most likely targets.

[0042] 2. Ordinary hosts (desktops and laptops): due to the uncertainty of their functions, a uniform weight cannot be used, and further analysis is needed (Table 2).

[0043] 3. Firewalls: as network gateways that attackers must cross, they can be ignored for convenience.

[0044] Table 2: Customized weights for hosts

[0045]

[0046] The weights of a custom host can be associated with three different features: 1) handling important files, 2) having access to a database, 3) publicity. The host features can be represented as a triplet <X, Y, Z>, where X, Y, Z correspond to the three features defined in Table 2.

[0047] When the host satisfies these features, the three weights are set to <0.9, 0.8, 0.6> respectively, otherwise <0.1, 0.2, 0.4> applies. The weight of a host can be represented by Equation 1.

[0048] W = 1 - (1 - X)(1 - Y)(1 - Z) (1)

[0049] The importance of a deployment node is attempted to be estimated according to its features. It at least provides a quantitative index to determine the priority order of the target location of the mitigation measure deployment.

[0050] As a preferred embodiment, further, in the process of solving the deep reinforcement learning mitigation measure model, by constructing a multi-objective optimization objective function, and using the SAC algorithm to learn the mitigation deployment in the problem space by maximizing the multi-objective optimization objective function, the maximum process of the objective function is used to obtain the optimal mitigation deployment strategy.

[0051] Further, the process of learning mitigation deployment using the SAC algorithm includes the following contents: first, select mitigation actions from candidate strategies, and transition attack technology blocking states in the environment, and store the transition in the replay buffer; then, at each time step gradient, optimize the deep reinforcement learning mitigation measure model parameters through the state value network objective function, the policy network objective function and the temperature network objective function, until the deep reinforcement learning mitigation measure model reaches a state of convergence.

[0052] The core of the DRL-MD framework is the Soft Actor-Critic (SAC) algorithm to achieve the optimal mitigation deployment strategy, as shown in Figure 3 It includes two stages, namely problem space definition and problem solving. Among them, in the problem space, the action space A can be formulated on the basis of V-ATT&CK, which includes every possible mitigation action at time t. The state space S includes the observed attack technology blocking state s t = (tech1, tech2,..., tech i ), 0≤i≤N, where N represents the number of technologies at each time step t, tech i ∈ {0, 1}: 0 / 1 represents tech iUnhindered / hindered. The target state in the state space S is (1, 1,..., 1), i.e., it is desired that all attack techniques are hindered; the reward function r evaluates mitigation measures to achieve the multi-objective optimization, which includes 1) the importance of deploying a node w (Section 5.2), 2) the reward r for preventing the impact of mitigation measures on all attack techniques tech , 3) the reward r for fixing a vulnerability by a mitigation measure vul , and 4) the deployment cost c. Thus, the reward function can be further expressed as:

[0053]

[0054] where, represents the impact of a mitigation measure on the i-th attack technique; N represents the number of attack techniques; CVSS v is the CVSS score of fixing a vulnerability v; for convenience of calculation, the deployment cost c can be set to 0.1.

[0055] SAC works in the problem space to learn the best strategy for mitigation deployment by maximizing the objective function of multi-objective optimization shown in equation (3), i.e., problem solving. In this context, the optimal strategy, state value function, Q-network, policy network, and temperature network can be described as follows:

[0056] The problem of finding the optimal strategy can be formulated as maximizing the maximum entropy objective with SAC:

[0057]

[0058] where, π is a candidate strategy, π * is the best strategy to be derived; is the reward function, r ∈ [0, 1] is the discount rate, s t ∈ S is the state at time point t, a t ∈ A is the action at time point t; T π is the trajectory distribution caused by the policy π; α determines the relative importance of the entropy term and the reward, which is called the temperature parameter; is the entropy value of the policy π in state s t :

[0059]

[0060] To maximize the objective, the state value function of the policy π can be defined as:

[0061] V(s t ): = π(s t ) T [Q(s t )-αlog(π(s t ))] (4)

[0062] Two Q networks are trained according to the Q function respectively, and the minimum value of the two outputs is applied. This is done to avoid overestimating state values and minimize the Bellman residual, so the resulting policy can be improved.

[0063]

[0064] Where D is the buffer of past experience replay; is the Monte Carlo estimate of formula 4 after sampling experience from D using the target network of Q.

[0065] In the policy network, the target policy can be improved by maximizing the reward. The Q function guides any changes to the policy with the new policy target.

[0066]

[0067] Where, the policy π φ (s t ) uses a neural network with parameters φ to provide an accurate action distribution to directly derive the expectation.

[0068] In the temperature network, the variance of the temperature loss estimate is reduced by learning the temperature:

[0069]

[0070] Where, is a constant vector representing the hyperparameter of the target entropy.

[0071] The training of each iteration can include: 1) sampling an action a t from the policy π t , transitioning the state s t+1 from the environment, and storing the transition in the replay buffer D; 2) for each gradient step, updating the Q function parameters θ, the policy weights φ and the temperature α to optimize the parameters.

[0072] To this end, after obtaining the mitigation measures for the attack techniques launched by the attack group and completing the training of the DRL-MD model, the optimal mitigation deployment strategy can be determined through multi-objective optimization.

[0073] Further, based on the above method, the embodiment of the present application also provides a network node security measure mitigation deployment optimization system based on deep reinforcement learning, comprising: a model construction module and a deployment optimization module, wherein,

[0074] The model building module is used to construct a deep reinforcement learning mitigation model. The deep reinforcement learning mitigation model is used to transform the deployment of network node security mitigation measures into a multi-objective optimization problem. The deep reinforcement learning mitigation model runs on a security knowledge base and uses the security knowledge base to establish the relationship between attacks and defenses connected by vulnerabilities. The problem space is constructed based on the action space, state space and reward function, and the mitigation deployment is optimized in the problem space through deep reinforcement learning methods.

[0075] The deployment optimization module is used to acquire attack techniques and mitigation measures in the target network using a security knowledge base, and to allocate node weights based on the functions of nodes in the target network. The attack techniques, mitigation measures, and node importance weights in the target network are used as inputs to the deep reinforcement learning mitigation measure model. The optimal mitigation deployment strategy in the target network is obtained by solving the deep reinforcement learning mitigation measure model.

[0076] To verify the effectiveness of the proposed solution, the following explanation is provided in conjunction with simulation data:

[0077] To test the mitigation and deployment capabilities of DRL-MD. This ransomware has become one of the largest internet attacks, maliciously blocking the data or devices of victims worldwide in exchange for ransom. The test platform includes... Figure 5 As shown, the system consists of 11 nodes, including 2 hardware firewalls, 1 router, 1 switch, 1 FTP server, 1 DB server, and 5 hosts. The external firewall isolates the internet from the internal router, while the internal firewall separates Host 1, Host 2, and the server storing important files. Host 1 and Host 2 can communicate with each other, but only Host 2 can access the FTP server and DB server. The DB server is connected to the FTP server to receive and respond to its requests. The switch is directly connected to the router, which in turn connects to Host 3, Host 4, and Host 5. These three hosts are for public use and do not contain important files.

[0078] A problem space was constructed to test attacks against the WannaCry ransomware, where the attack techniques and mitigation measures for WannaCry were initially obtained through V-ATT&CK. Table 3 lists the details of the attack techniques, including the attack technique ID, attack technique name, tactical objectives to be achieved, and the completion of those objectives, as well as a description of the technique.

[0079] Table 3: Attack Technique Information

[0080]

[0081] Table 4 lists the information of mitigation measures, including the ID, name, and description of the mitigation measures. Table 5 illustrates 1) the relationship between attack techniques and mitigation measures, 2) attack techniques that can be impeded by mitigation measures, and 3) how effective the mitigation measures are according to past records. It is worth noting that it is a challenge to mitigate attack techniques related to the “discovery tactics” category with preventive controls, as these techniques are based on the misuse of system functionality.

[0082] Table 4: Information of Mitigation Measures

[0083]

[0084]

[0085] Table 5: Relationship between Attack Techniques and Mitigation Measures

[0086]

[0087]

[0088] The vulnerability assessment tool Nessus detects relevant vulnerabilities in the network. The scores in CVSS 3.0 are listed in Table 6. Figure 6 The connections between vulnerabilities and corresponding techniques and mitigation measures are shown, referring to V-ATT&CK. The knowledge base graph is visualized through Neo4j, with each type of entity marked in a separate color as attack organization, tactic, technique, mitigation, and vulnerability. The relationships between entities are represented with directed edges.

[0089] Table 6: Information of Relevant Vulnerabilities

[0090]

[0091] The importance of a node can be estimated by Formula 1, as shown in Table 7, according to the settings of the test platform.

[0092] Table 7: Node Importance of Test Platform

[0093]

[0094]

[0095] 1. Host 1 handles important files, cannot access the database, and is private, with a node importance of 1-(1-0.9) x (1-0.2) x (1-0.4) = 0.952.

[0096] 2. Host 2 handles important files, has access to the database, and is private, with a node importance of 1-(1-0.9) x (1-0.8) x (1-0.4) = 0.988.

[0097] 3. Host 3, Host 4 and Host 5 do not handle important files, cannot access the database, and are public, the node importance is 1-(1-0.1) x (1-0.2) x (1-0.6) = 0.712.

[0098] 4. The node importance of the router, switch and two servers are set to 0.9.

[0099] To find the optimal mitigation deployment strategy π * , DRL-MD needs to derive the hybrid reward function r according to Equation 2. This task relies on: 1) the node importance w in Table 7; 2) the reward r tech , i.e., the blocking effect of the mitigation on all attack techniques; 3) the reward r vul , i.e., the repair of the vulnerability by the mitigation.

[0100] The blocking effect of the mitigation on the attack techniques can refer to the mitigation effect in Table 5, with the total number of attack techniques N = 10. The reward of repairing the vulnerability by the mitigation can be obtained by the relationship between the vulnerability and the mitigation in Figure 6 , where the straight line between the two means that the mitigation has a repair effect on the vulnerability, and vice versa.

[0101] So far, in order to find the optimal mitigation deployment strategy, the problem space has been properly initialized, that is, the initial values / settings of the key factors are obtained by the proposed DRL-MD method.

[0102] In the performance evaluation, first, the effectiveness of V-ATT&CK is evaluated compared with the existing ATT&CK model. After that, the overall performance of DRL-MD is compared with the state-of-the-art corresponding model.

[0103] 1. Effectiveness of V-ATT&CK

[0104] The proposed V-ATT&CK knowledge base is compared with ATT&CK (without vulnerability knowledge) in terms of deployment cost and attack technique coverage using DRL-MD. Attack technique coverage is a common indicator to evaluate the effect of mitigation deployment, while deployment cost is an indicator to measure the redundancy. The results are listed in Table 8.

[0105] It is clear that the mitigation strategy derived with V-ATT&CK is superior to the strategy derived with ATT&CK. The former can achieve the same security goal of preventing attack technique coverage with fewer mitigations and lower deployment cost.

[0106] Table 8: Attack technique coverage and deployment cost of deploying mitigations using different knowledge bases (average and standard deviation of 50 experiments).

[0107]

[0108] The results show that: 1) V-ATT&CK can be used as an effective cyber security knowledge base to find mitigation deployment strategies; 2) using V-ATT&CK can find more optimal strategies than using ATT&CK, while ensuring effective defense, greatly reducing redundancy.

[0109] 2. SAC performance

[0110] The SAC algorithm in the DRL-MD model is tested against state-of-the-art reinforcement learning counterpart methods, including: 1) DQN, 2) DDQN, 3) Dueling DDQN, and 4) A3C. Figure 7 The performance of the mitigation deployment to defend against the WannaCry attack is described. Performance is measured in terms of 200 rolling episode scores. All candidate methods are able to reach the optimal strategy. The performance metrics of the other counterpart methods are close to each other, while DRL-MD is always superior to the other methods. DRL-MD reaches the highest score of 31.42 at episode 160, and its best strategy sequence for mitigation deployment is superior to the other methods.

[0111] The results show that, compared with state-of-the-art methods, DRL-MD can find the best mitigation deployment strategy with high performance in fewer training episodes using SAC.

[0112] 3. Overall performance of DRL-MD

[0113] To evaluate the overall performance of DRL-MD in terms of mitigation deployment compared with state-of-the-art solutions for the WannaCry attack, including BAG and game theory methods, experiments are conducted, and the results are listed in Table 9.

[0114] The deployment cost of DRL-MD is obviously lower than that of other solutions when achieving the same security goal (attack technique coverage). In addition, the mitigation measures obtained from DRL-MD can cover all vulnerabilities, while 1) the BAG solution has a higher average CVSS score but lower vulnerability coverage; 2) the game theory solution has a lower average CVSS score and lower vulnerability coverage.

[0115] Table 9: Comparison of mitigation deployment methods in terms of deployment cost, attack technique coverage, vulnerability coverage, and average CVSS score.

[0116]

[0117]

[0118] In general, the DRL-MD proposed in the embodiments can greatly reduce the redundancy of mitigation deployment in the case of effective defense against network attacks, balance the importance of the deployment node, the effect of preventing network attacks, vulnerability repair and deployment cost, which has potential in controlling the network defense cost in large-scale scenarios.

[0119] Unless otherwise specified, the relative steps, numerical expressions and numerical values of the components and steps set forth in the embodiments are not intended to limit the scope of the present application.

[0120] The various embodiments are described in the specification by way of progression, each building on the last, to convey the most complete understanding of the present application. The embodiments disclosed herein are not exhaustive of the scope of the application. The same or similar parts and / or functions between embodiments exchange reference numerals across all figures and / or embodiments.

[0121] The units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation does not exceed the scope of the present application.

[0122] A person of ordinary skill in the art can understand that all or part of the steps in the above method can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. Alternatively, all or part of the steps of the above embodiments can also be implemented using one or more integrated circuits, accordingly, each module / unit in the above embodiments can be implemented in the form of hardware or in the form of a software function module. The present application is not limited to any specific form of combination of hardware and software.

[0123] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for mitigating and optimizing the deployment of network node security measures based on deep reinforcement learning, characterized in that, Includes the following content: A deep reinforcement learning mitigation model is constructed to transform the deployment of network node security mitigation measures into a multi-objective optimization problem. The deep reinforcement learning mitigation model operates based on a security knowledge base and uses the security knowledge base to establish the relationship between attacks and defenses connected by vulnerabilities. The problem space is constructed based on the action space, state space, and reward function, and the mitigation deployment is optimized in the problem space through deep reinforcement learning methods. For the target network, the attack techniques and mitigation measures in the network are obtained by using a security knowledge base, and the node weights are assigned according to the node functions in the target network. The attack techniques, mitigation measures and node importance weights in the target network are used as inputs to the deep reinforcement learning mitigation measure model. The optimal mitigation deployment strategy in the target network is obtained by solving the deep reinforcement learning mitigation measure model. In the problem space, each possible mitigation action of the network at a time step is obtained based on the security knowledge base, and each mitigation action constitutes the action space; in each time step, each attack technique blocking state constitutes the state space; and a hybrid reward function is constructed by deploying node importance weights, rewards for preventing mitigation measures from affecting all attack techniques, rewards for mitigation measures to fix vulnerabilities, and deployment costs. This hybrid reward function is used to balance the importance of deploying nodes, the effectiveness of preventing network attacks, the effectiveness of fixing vulnerabilities, and the deployment costs.

2. The deep reinforcement learning based network node security measure mitigation deployment optimization method of claim 1, wherein, The security knowledge base is an open adversary tactics and technology knowledge base, which includes the ATT&CK knowledge base and the CVE knowledge base. 3.The deep reinforcement learning based network node security measure mitigation deployment optimization method of claim 1, wherein, The hybrid reward function is expressed as: Where w is the importance weight of the deployment node, r tech To prevent the mitigation measures from affecting all attack techniques, r vul The reward for mitigation measures to fix the vulnerability, where c represents the deployment cost. To mitigate the impact of the measures on the i-th attack technique, where N is the number of attack techniques, CVSS v CVSS score for fixing vulnerability v.

4. The deep reinforcement learning based network node security measure mitigation deployment optimization method of claim 1, wherein, In the node importance weight assessment, firstly, the node weights of servers, network connections, and hosts are initially set based on empirical data on the likelihood of being attacked. Then, the host function tripartite is obtained based on the importance factor of the processed file, the database access permission factor, and the public access factor, and the host node weight is obtained using this host function tripartite.

5. The deep reinforcement learning based network node security measure mitigation deployment optimization method of claim 1, wherein, The process of obtaining the host node weight W using the host function triad is expressed as: W = 1 - (1 - X)(1 - Y)(1 - Z), where X, Y, and Z represent the importance factor of the processed file, the access permission factor to the database, and the public access factor, respectively.

6. The deep reinforcement learning based network node security measure mitigation deployment optimization method of claim 1, wherein, In the deep reinforcement learning mitigation measure model solving, a multi-objective optimization objective function is constructed, and the SAC algorithm is used to maximize the multi-objective optimization objective function in the problem space to learn the mitigation deployment, and the optimal mitigation deployment strategy is obtained by using the maximization process of the objective function, wherein the multi-objective optimization objective function is represented as: Wherein, π is a candidate strategy, and π* is the output optimal strategy; is an expected function, r(s t ,a t ) is a reward function; γ is a discount rate, and γ ∈ [0, 1], s t is the attack technology hindering state of time step t, a t is the mitigation action of time step t; T π is the trajectory distribution caused by the strategy π; and α is a temperature parameter; is the entropy value of the strategy π in the state s t .

7. The deep reinforcement learning based network node security measure mitigation deployment optimization method of claim 6, wherein, In maximizing the objective function of multi-objective optimization, the state value function of the setting strategy is V(s t ):=π(s t ) T [Q(s t )-αlog(π(s t ))] and a state value network is constructed by using two Q networks, the state value network is trained by using experience replay of state and action and Q function to generate a mitigation deployment strategy, and the action distribution and expected estimate are obtained by maximizing the reward in the strategy network.

8. The deep reinforcement learning based network node security measure mitigation deployment optimization method according to claim 7, characterized by, The process of learning mitigation deployment using the SAC algorithm includes the following: First, mitigation actions are selected from candidate policies, and the attack technique blocking state is transferred in the environment, with the transition stored in a replay buffer; then, at each time step gradient, the parameters of the deep reinforcement learning mitigation model are optimized through the objective functions of the state value network, the policy network, and the temperature network until the deep reinforcement learning mitigation model reaches a convergent state.

9. A deep reinforcement learning based network node security measure mitigation deployment optimization system, characterized in that, It includes: a model building module and a deployment optimization module, wherein, A model construction module is configured to construct a deep reinforcement learning mitigation model, and the deep reinforcement learning mitigation model is used to convert network node security mitigation deployment into a multi-objective optimization problem, wherein the deep reinforcement learning mitigation model is run based on a security knowledge base and establishes the relationship between attacks and defenses connected by vulnerabilities by using the security knowledge base, a problem space is constructed according to an action space, a state space and a reward function, and the mitigation deployment is optimized in the problem space by using a deep reinforcement learning method. A deployment optimization module is configured to obtain attack techniques and mitigation measures in a network by using the security knowledge base, and to allocate node weights according to the node functions in a target network, wherein the attack techniques, the mitigation measures and the node importance weights in the target network are used as inputs of the deep reinforcement learning mitigation model, and the best mitigation deployment strategy in the target network is obtained by solving the deep reinforcement learning mitigation model. In the problem space, each mitigation action that can occur at a time step of the network is obtained from the security knowledge base, and the action space is formed by each mitigation action; the state space is formed by each attack technology at each time step; and a mixed reward function is constructed by using the node importance weights, the rewards for preventing the influence of mitigation measures on all attack techniques, the rewards for repairing vulnerabilities by mitigation measures and the deployment costs, and the mixed reward function is used to balance the node importance, the effect of preventing network attacks, the effect of repairing vulnerabilities and the deployment costs.

Images