Honeynet deployment strategy generation method based on reinforcement learning in intranet environment

Abstract

The invention discloses a method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment, and the method comprises the steps: randomly generating a plurality of nodes, and constructing the intranet environment; constructing an attack agent, and carrying out staged interaction on the attack agent and the intranet environment to pre-train the attack agent; pre-training attack agents with different attack intentions are distributed to each honeypot agent, and the honeypot agents are pre-trained in a network environment; initializing a pre-training honeypot agent, and constructing a honeynet environment by using the pre-training honeypot agent; in the honeynet environment, the pre-training attack agent interacts with the pre-training honeypot agent, and the decision agent is pre-trained; and training all agents by using an MADDPG algorithm, constructing a hierarchical defense agent by using a pre-training honeypot agent and a pre-training decision agent, and updating a strategy network and a value network of each agent. Through the dynamic deployment strategy, the adaptive capacity of the honeynet system is improved, and the safety protection level of the intranet environment is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment. Background Art

[0002] Traditional defense mainly formulates corresponding rules based on existing attacks and cannot resist unknown attacks. There is a lack of linkage between different network security protection devices. Most of the existing detection mechanisms are identified through abnormal representation, which has a lag. Different from the defense method of directly resisting network attacks, honeypot technology reduces the threat of network intrusion to the target object by deceiving attackers and significantly consuming attack resources, and obtains information such as the attack method, attack tools and attack intention used by attackers by analyzing the trapped attack traffic and attack behavior. However, when deploying traditional honeypots, it is necessary to determine the network location of the honeypot system. It is usually deployed in the DMZ (Demilitarized Zone) area, and the network switches and routers are configured to assign IP addresses to the honeypot system. Therefore, there is a problem that the static configuration and fixed location of traditional honeypot deployment applications are easily identified, marked and bypassed by attackers. It can be seen that traditional honeypots are a means of "passive active defense". In addition, honeypots only have a limited perspective and usually only simulate the behavior of a single system or service. In contrast, honeynets are configured with multiple honeypots with vulnerabilities as traps, thereby providing more dimensional attack data. These data include information such as the attacker's lateral movement in the intranet and the overall picture of the attack chain, which helps to better understand and respond to network threats.

[0003] Research related to honeynet deployment strategy generation can be divided into two categories. One is to dynamically deploy honeynets by analyzing the attacker's attack information. Bartwal et al. analyzed the information in the attacker's pcap package, pulled the honeypot image file corresponding to the malicious traffic attack category on the virtual machine, and linked the image file with the destination IP in the pcap package to achieve dynamic honeypot deployment, solving the problem of what type of honeypot to deploy at what time. Huang et al. focused on the harm caused by internal threats to the network. By modeling defenders and users, they proposed a new game model, combined mathematical programming and graphics methods to calculate the optimal solution, and obtained a systematic solution under the network deception defense mechanism. A two-person zero-sum game model was modeled between network defenders and attackers, and a reward function was constructed. Different calculation methods were used to solve it and obtain the deployment location of the honeypot. In terms of the application of dynamic deployment of honeypots, Gao et al. obtained the attacker's behavior data by modeling the attacker, used neural networks to learn the data, predicted the attacker's intentions, and adjusted the deployment strategy. Wang et al. combined reinforcement learning methods to model attack and defense agents. The defender predicted the attacker's attack nodes, placed honeypots, and designed a two-layer attack graph to narrow the range of honeypot deployment locations.

[0004] The other type is to model the network environment and dynamically deploy the honeynet by analyzing the characteristics of the network environment itself. Kong et al. aimed to realize the automated honeynet deployment strategy in the cloud environment. By monitoring the changes of the cloud system in real time, updating the attack graph, identifying the key nodes in the attack graph, and abstracting the deployment of the honeynet into a set coverage problem, they finally realized the dynamic deployment of the honeynet. Yang et al. established an attack graph to analyze the vulnerability of each node in the network and formulated the deployment strategy of the honeynet. They used the ant colony algorithm to simulate the retention decision of the honeypot in the honeynet and used the wolf pack algorithm to determine the value of the honeypot based on the information collected by the honeypot, thus realizing the dynamic deployment of the honeynet. In order to prevent the fingerprint of the honeypot from being identified by the attacker, Fraunholz et al. proposed a method of context-awareness to correctly deploy the honeypot with the correct content configuration. The configuration information of the intranet environment was used as training data, and the k-means algorithm was used for clustering to automatically generate configuration information based on entity features.

[0005] Methods that model attackers or attack information and analyze them to dynamically generate honeynet deployment strategies require a deep understanding of the attacker's behavioral characteristics and attack methods. This modeling process not only involves the abstract representation of factors such as attacker motivation, target selection, and attack methods, but also includes the dynamic characteristics of behavior evolution over time. By modeling the game of the attacker and defender models, the deployment of honeypots or honeynets is achieved. These methods focus more on the deployment location of each honeypot, without considering the linkage between honeypots.

[0006] A method for generating honeynet deployment and configuration based on the intranet environment identifies key nodes in the intranet by mapping the host's own configuration information and the connection relationship between hosts. Honeypots are deployed on key nodes or near each key node to enhance the effective detection and defense of known threats. However, in terms of the selection of key nodes, this method mainly relies on the modeling and reasoning of known network vulnerabilities, which limits its ability to identify and defend against unknown attack patterns. Summary of the invention

[0007] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment. Aiming at the problem that static configuration and fixed position in traditional honeypot deployment applications are easily identified, labeled, and bypassed by attackers, a reinforcement learning framework is used to model the interaction process between the attacking intelligent agent and the enterprise intranet environment simulated by the honeynet, and to realize the dynamic transformation of the honeynet environment. At the same time, by dynamically transforming the IP, service, vulnerability and other information of the honeypot, the problem that the honeypot is easily circumvented by attackers, the attack behavior is delayed, the attack characteristics are identified, and the attack intention is assisted in judging, thereby realizing "automated active defense", improving defense efficiency, and effectively protecting service resources.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] The present invention provides a method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment, comprising the following steps:

[0010] Multiple nodes are randomly generated, and each node is assigned a honeypot agent. Each honeypot agent has at least one service and corresponding vulnerability. The connectivity between honeypot agents is established based on the service type, and the intranet environment is constructed.

[0011] Construct an attack agent, which interacts with the intranet environment in stages. In the preparation stage, a pre-trained attack agent is obtained, and in the hybrid training stage, an attack agent with judgment ability is obtained.

[0012] According to the service type, each honeypot agent is assigned a pre-trained attack agent with different attack intentions. The pre-trained attack agent selects attack actions to change the intranet environment. The honeypot agent deploys appropriate vulnerabilities according to the intranet environment. After multiple interactions, the pre-trained honeypot agent is obtained.

[0013] Initialize the pre-trained honeypot agent and use the pre-trained honeypot agent to build the honeynet environment;

[0014] In the honeynet environment, the pre-trained attack agent interacts with the pre-trained honeypot agent to construct a decision agent. The decision agent predicts the attack intention of the pre-trained attack agent based on the interaction information, dynamically configures and adjusts the pre-trained honeypot agent and its services, and obtains the pre-trained decision agent.

[0015] A layered defense agent is constructed using pre-trained honeypot agents and pre-trained decision agents. The pre-trained attack agent attacks the layered defense agent. The pre-trained decision agent obtains the information of the pre-trained attack agent and determines the type and attack intention, and sends out the defense target. The pre-trained honeypot agent accepts the defense target and automatically generates a service type vulnerability that matches the target. The MADDPG algorithm is used to train all agents, and the policy network and value network of each agent are updated to obtain the final honeypot agent, decision agent and attack agent.

[0016] As a preferred technical solution, the attacking agent interacts with the intranet environment in stages, including a preparation stage and a training stage;

[0017] Preparatory stage: Create attack action set A pre ={a1,a2,……,a n},a i represents the attack action i, n is the number of attack actions, and the attack action types include remote connection attacks, local attacks, and privilege escalation attacks. The attack agent is trained using the DDPG algorithm. The attack agent selects an attack action to attack the intranet environment. If the attack is successful, a reward is obtained;

[0018] Hybrid training phase: add recognition honeypot agent actions and train attack action set A trn ={a1,a2,……,a n ,b1,b2,……,b m}, b j represents the action j of identifying the honeypot, and m is the number of identified actions. The MADDPG algorithm is used to mix the attack agent with all other agents. The attack agent is trained in the attack action set A. trn Select actions to attack the intranet environment, and get rewards when the attacker agent successfully identifies the honeypot and exits.

[0019] As a preferred technical solution, the method of allocating pre-trained attack agents with different attack intentions to each honeypot agent according to the service type is as follows:

[0020] Deploy the pre-trained attack agent in the intranet environment and set the attack intention set Des = {d1, d2, ..., d l}, l is the total number of attack intentions, and the honeypot agent is randomly assigned a service type it carries.

[0021] As a preferred technical solution, the following steps are included during the pre-training of the honeypot agent:

[0022] When the pre-trained honeypot agent chooses to add a vulnerability, it determines whether the pre-trained attack agent successfully exploits the vulnerability. Whenever the attack agent successfully exploits a vulnerability, the honeypot agent will receive a reward α·U(v t ); When the attacker successfully achieves his attack goal, the honeypot agent will receive a reward β·G, inducing the attacker to reveal his attack intention; the pre-trained reward function of the honeypot agent is as follows:

[0023]

[0024] Among them, U(v t ) indicates that the attacker exploits the vulnerability v t The number of times, G represents the score of the attacker achieving the attack goal, α and β are the weight coefficients of the reward, and c represents the number of vulnerabilities in the honeypot intelligent body;

[0025] The four-tuple (the state of the honeypot agent, the selected action, the generated and deployed vulnerability, and the state of the next time step) is stored in the buffer. After a batch, the advantage function and value function are calculated and updated. After multiple iterations, the pre-trained honeypot agent is finally obtained.

[0026] As a preferred technical solution, the initialization of the pre-trained honeypot agent includes:

[0027] Set the number of honeypot agents, set the service set and vulnerability set of the honeypot agents; the number of honeypot agents V (hp i ), as follows:

[0028] V(hp i )={v d ∣v d ∈Vul,P(v d )>P threshold ||V(hp i )∣≤ρ×q}

[0029] Among them, hp i represents the i-th honeypot, v d Indicates service j The vulnerability in P(v d ) represents the probability of the vulnerability being selected, P threshold is the threshold of the probability of selection, |V(hp i )| represents the number of vulnerabilities. The number of honeypot vulnerabilities generated during initialization is less than ρ times the total number of service vulnerabilities.

[0030] Set the service type set, vulnerability set, total vulnerability set, and the method of randomly generating service types for each honeypot;

[0031] Build connectivity between honeypot agents based on a set of service types.

[0032] As a preferred technical solution, during the pre-training of the decision agent, the following steps are included:

[0033] S51. At time t, the pre-trained attack agent selects an attack action, the pre-trained honeypot agent interacting with the pre-trained attack agent selects a defense action, and the other pre-trained honeypot agents remain stationary; the defense action includes adding vulnerabilities and selecting vulnerability types;

[0034] S52, recording training process information;

[0035] S53, after τ steps, the decision agent reallocates services to other pre-trained honeypot agents;

[0036] S54. When the pre-trained attack agent successfully exits or completes the attack intention, a reward is obtained;

[0037] If the pre-trained attack agent exits successfully and completes the attack intention, the reward of the decision agent is the ratio of the attack action entries of the attack agent after deduplication to the total number of time steps. In other cases, the reward is zero.

[0038] S55. After multiple rounds of iterations, the pre-training of the decision-making agent is completed.

[0039] As a preferred technical solution, the method of using a pre-trained honeypot agent and a pre-trained decision agent to construct a layered defense agent includes:

[0040] The initial states of the decision agent and each honeypot agent in the layered defense agent are and

[0041] The decision agent sends a target to each honeypot agent. Each target defines the type and location of each honeypot; each honeypot agent selects an action based on the sent target and initial state.

[0042] Each honeypot agent The probability of transferring to state And obtain the corresponding reward value according to the reward function Each honeypot agent interacts with the environment until, after τ steps, the decision agent The probability of transferring to state and receive external rewards

[0043] The layered defense agent generates an optimal action strategy that maximizes the reward value.

[0044] As a preferred technical solution, the dynamic configuration and adjustment of the pre-trained honeypot agent and its services are specifically as follows: according to the pre-trained attack agent information, the decision agent dynamically allocates the location and service type of each honeypot agent through actions.

[0045] As a preferred technical solution, the strategy network θ i The update is as follows:

[0046]

[0047] Among them, μ i (s|θ i ) represents a deterministic strategy, Q i (·) represents the Q function, a hp1 ,a hp2 ,……,a hpn are the actions of all pre-trained honeypot agents, a attacker Indicates the current attacker's action, a decision represents the action of the decision-making agent, φ i represents the value network, s i Represents the state of the i-th agent, and i represents the i-th agent.

[0048] As a preferred technical solution, the value network update formula of the decision agent and the honeypot agent is as follows:

[0049]

[0050] Among them, Critic decision′ Represents the value network value of the decision-making agent after secondary distribution, Critic decision represents the value of the critic network of the policy agent after being updated by MADDPG, n is the total number of honeypot agents, HP i is the i-th honeypot agent, HP j A honeypot agent that interacts with the attacker.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] (1) The present invention models the interaction process between the attacking intelligent agent and the enterprise intranet environment simulated by the honeynet, realizes the dynamic transformation of the honeynet environment, and dynamically transforms the IP, service, vulnerability and other information of the honeypot. It alleviates the problem that the honeypot is easily circumvented by attackers, delays attack behaviors, identifies attack features and assists in determining attack intentions, thereby realizing "automated active defense", improving defense efficiency, and effectively protecting service resources.

[0053] (2) By modeling the attacker, the present invention allows the defender to adopt an active defense strategy, that is, predict and prepare corresponding defense measures before the attack occurs, rather than just passively responding to the attack. Compared with the original work, in the attack agent modeling process of the present invention, the corresponding actions at different stages are different, thereby achieving fine control and adjustment of the behaviors at different stages.

[0054] (3) The present invention draws on the idea of ​​layered reinforcement agents. The deployment of the honeynet is modeled as a layered defense agent under a multi-agent system. A honeynet environment suitable for attackers is constructed by high-level decision-making agents and low-level honeypot agents. After an attack is detected, the attacker's traffic will be redirected to the honeynet environment. Since the decision-making agent in the layered defense agent aggregates information and uniformly schedules each honeypot in the honeynet, the linkage between honeypots is increased.

[0055] (4) The present invention adopts a customized virtual honeynet scenario to adaptively configure the host information in the intranet according to the scenario requirements, and uses a reinforcement learning algorithm to train the layered defense agent to control the deployment location of each honeypot in the honeynet and the type of vulnerability carried by each honeypot; through a reasonable honeypot deployment strategy, the attacker's attack intention is inferred and verified to achieve delayed attack or even tracing countermeasures.

[0056] (5) Through this intelligent dynamic deployment strategy, the present invention not only improves the adaptive capability of the honeynet system, but also significantly enhances the security protection level of the intranet environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a flow chart of a method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment according to an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of dynamic deployment of a honeynet in an intranet scenario according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the interaction process between an attacker and a honeypot agent according to an embodiment of the present invention;

[0061] Figure 4 It is a structural diagram of the layered defense agent decision model of an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0063] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0064] See also Figure 1 This embodiment provides a method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment, comprising the following steps:

[0065] S1. Randomly create an intranet environment.

[0066] Randomly create an intranet environment, randomly generate N host nodes, each host node has at least one service, randomly select vulnerabilities from the vulnerability set and generate them, see Figure 2 .

[0067] Host node set (N), N = {h1,h2,……,h n},h i Represents the i-th host, and n is the number of hosts.

[0068] The service type set (Services) is a set of service types that can be run on the host, Services = {ser1, ser2, ..., ser m},ser j represents service type j, and m is the number of service types.

[0069] Vulnerability collection (Vul) is a service ser jThe corresponding set of existing vulnerabilities. Vul = {v1, v2, ..., v q},v d Indicates vulnerability d, q is ser j Therefore, the total vulnerability set Vuln = {Vul1, Vul2, ..., Vul m}, the total vulnerability set is the set of all vulnerabilities.

[0070] Randomly assign one or more services to each host. i )={ser j ∣ser j ∈Ser,P(ser j )>P threshold} Among them, h i Indicates the i-th host, ser j Indicates the service type, P(ser j ) indicates service ser j The probability of being selected, P threshold is the threshold of the probability of selection.

[0071] Then select vulnerabilities from the vulnerability set corresponding to the service. V(h i )={v d ∣v d ∈Vul,P(v d )>P threshold} Among them, h i represents the i-th host, v d Indicates the vulnerability on the host, P(v d ) represents the probability of the vulnerability being selected, P threshold is the threshold of the probability of selection.

[0072] Based on the service type, the connectivity between hosts is constructed, that is, hosts of the same service type have connectivity.

[0073] S2. Pre-trained attack agent.

[0074] This embodiment uses a randomly created intranet environment as the training environment for the attacking agent, and trains the agent by interacting with the intranet environment in stages. The staged attacking agent is designed to simulate potential threats in the real world to achieve comprehensive training and evaluation of the defending agent. Specifically, the pre-training of the attacking agent includes a preparation stage and a hybrid training stage;

[0075] Use the DDPG algorithm to train the attack agent. The attack agent selects an attack action to attack the intranet environment. If the attack is successful, it will get a reward.

[0076] In the preparation stage, the attacking agent aims to attack efficiently and achieve the goal. The attack action set A in this stage is pre ={a1,a2,……,a n},a i represents the attack action i, and n is the number of attack actions. The attack action type can be a att ={Remote, local, privilege}, corresponding to remote connection attacks, local attacks and privilege escalation attacks respectively. In actual training, one or more of them are selected from the set to attack the target.

[0077] In the hybrid training stage, the role of the attacking agent is to achieve the attack goal without stepping on the honeypot. The attack action set A in this stage is trn ={a1,a2,……,a n ,b1,b2,……,b m},a i Indicates attack action i, b j represents the action j of identifying the honeypot, n is the number of attack actions, and m is the number of identification actions.

[0078] After multiple iterations of training, we finally obtain a pre-trained attack agent that can attack efficiently in an intranet environment. The modeling process of the attack agent in stages is shown in Table 1.

[0079] Table 1

[0080]

[0081] The advantage of adopting a staged approach is that the preparation stage only involves the attack agent learning how to attack in an intranet environment. After the preparation stage, an attack agent that can attack efficiently is obtained. Compared with directly training attackers and layered defense agents, this staged approach improves the efficiency of the confrontation between the attack agent and the layered defense agent in the training stage and saves training resources.

[0082] S3. Train each underlying honeypot agent.

[0083] To each bottom honeypot agent, pre-trained attack agents with different attack intentions are assigned in sequence, i.e., the attack intention set Des = {d1, d2, ..., d l}, l is the total number of attack intentions; and randomly assigns a service type to it, Services(h i )={ser j ∣ser j ∈Random_Select(Ser)}. Pre-train the attack agent to select the attack action, change the environment, and use the DDPG algorithm to train the honeypot intelligence so that it can select the appropriate action according to the environment.t ={add_vulnerability,static}

[0084] Assume that at time step t, the state of the honeypot agent is s t , the action it chooses is a t , the vulnerability of honeypot agent generation and deployment is v t , the attacker exploits the vulnerability v t The number of times is U(v t ), the score G (G = 100) at which the attacker achieves the attack goal, and α and β are the weight coefficients of the reward, reflecting the influence of each behavior on the overall reward.

[0085] Whenever the attacking agent successfully exploits a vulnerability, the honeypot agent will gain α·U(v t ) reward. This reward mechanism is designed to ensure that the vulnerabilities generated by the honeypot agent are exploitable, thereby attracting attackers to interact. When the attacker successfully achieves his attack goal, the honeypot agent will receive a reward of β·G. This reward is designed to encourage the honeypot agent to induce the attacker to reveal his true purpose, thereby obtaining valuable attack data. Therefore, if there are c vulnerabilities in a honeypot agent, the reward function R(s t ,a t ) can be expressed as:

[0086] Will (s t ,a t ,r t ,s t+1 ) is stored in the buffer. After a batch, the advantage function and value function are calculated and updated. After multiple iterations, the pre-trained honeypot agent is finally obtained.

[0087] S4, initialize the honeynet environment.

[0088] Use pre-trained honeypot agents to form a honeynet environment. Randomly generate the services run by N honeypots in the honeynet environment and the connectivity between the honeypots.

[0089] First, define the parameters in the honeynet environment:

[0090] Number of honeypots (N): The number of honeypots contained in the honeynet, N = {hp1, hp2, ..., hp n},hp i represents the i-th honeypot, and n is the number of honeypots.

[0091] The service type set (Services), vulnerability set (Vul), total vulnerability set (Vuln), and the method of randomly generating service types for each honeypot are consistent with those in S1.

[0092] Limitation on the number of honeypot agents: Since subsequent interactions also require the addition of vulnerabilities, the number of vulnerabilities in the initialization of the honeypot must be strictly limited. i )={v d ∣v d ∈Vul,P(v d )>P threshold ||V(hp i )∣≤0.2×q}, where hp i represents the i-th honeypot, v d Indicates service j The vulnerability in P(v d ) represents the probability of the vulnerability being selected, P threshold is the threshold of the probability of selection. |V(hp i )| represents the number of vulnerabilities. The number of honeypot vulnerabilities generated during initialization is less than 20% of the total number of vulnerabilities of the service.

[0093] The connectivity between honeypot agents is the same as S1.

[0094] Secondly, the attacker's access traffic is redirected to one of the honeypots in the virtual honeynet environment. During the interaction, the honeypot agent dynamically deploys the vulnerabilities of the current environment, collects information, and maintains the interaction state with the attacker. After τ steps, the decision agent summarizes the attacker information in the honeynet environment, determines the attacker type, deduces the attacker's intention, and republishes the location and service type of each honeypot except the current interaction according to the attacker type. The honeypot agent accepts the target issued by the decision agent and automatically generates a service type vulnerability that matches the target.

[0095] like Figure 2 and Figure 3 The following is an example of a dynamic honeynet deployment scenario. The initial honeynet environment consists of 1 router (router), 2 switches (switch1, switch2), and 5 hosts (client1 hp 、webserver1 hp 、webserver2 hp 、mysql1 hp 、mysql2 hp ) and 1 decision-making agent. The 5 hosts are simulated by honeypot intelligence, and the categories of simulation services are as follows:

[0096] client hp : The developer's host.

[0097] webserver hp : The server that hosts the portal website class services.

[0098] mysql hp / sqlserver hp : A server that hosts data storage services.

[0099] The attacker's traffic is redirected to webserver1 hp At this time, webserver1 hp The running service is owncloud, which is a website focused on file services. The website contains two vulnerabilities, CVE1 and CVE2.

[0100] S41, at time t, the attacker first executes action 1 (attacker_action1) to scan for vulnerability information. The defender chooses action 1 (defender_action1) to remain unchanged. At this time, webserver1 hp Contains two vulnerabilities and displays their vulnerability information.

[0101] At S42, t+1, the attacker chooses to execute action 2 (attacker_action2) to exploit CVE1, and the defender chooses action 2 (defender_action2) to add CVE3 vulnerability. At this time, webserver1 hp It contains three vulnerabilities and shows that after CVE1 is successfully executed, it will enter the interface inside owncloud.

[0102] Step S43, at time t+2, the attacker chooses to perform action 3 (attacker_action3) to collect information, and the defender's third action is to choose action 1 (defender_action1) to remain unchanged. At this time, webserver1 hp It contains three vulnerabilities and shows that after CVE1 is successfully executed, it will enter the interface inside owncloud.

[0103] Step S44: At time t+3, the attacker chooses to execute action 4 (attacker_action4) to exploit CVE3, and the defender chooses action 3 (defender_action3) to add the privilege escalation vulnerability CVE4. At this time, webserver1 hp It contains four vulnerabilities and returns the mysql password obtained after CVE3 is successfully executed.

[0104] Step S45, at time t+4, the attacker may exit, and may move horizontally to other hosts simulated by honeypots according to the mysql password, or may hp Continue the attack. Strategy Agent Summary webserver1 hpBased on the attack information in the , we can judge that the attacker’s attack intention is to steal data. Therefore, after step 4, we reduce the number of honeypots carrying portal websites and increase the number of honeypots carrying databases, that is, Figure 3 Added sqlserver hp If the attacker continues to hp If the attacker moves laterally to another host or exits, the attacker remains stationary.

[0105] S5. Train decision-making agents.

[0106] The obtained pre-trained attack agent and N pre-trained honeypot agents are used as part of the environment to obtain a pre-trained decision agent. Specifically, the following steps are included:

[0107] S51. At time t, the pre-trained attack agent selects an attack action, the pre-trained honeypot agent interacting with the pre-trained attack agent selects a defense action, and the other pre-trained honeypot agents remain stationary; the defense action includes adding vulnerabilities and selecting vulnerability types;

[0108] S52, record the training process information and train the decision-making agent;

[0109] S53, the policy agent reallocates service ser to other pre-trained honeypot agents h , the assigned action is a t (hp i )={select(ser j )|ser j ∈Servies};

[0110] S54. When the pre-trained attack agent successfully exits or completes the attack intention, it obtains a reward and completes the pre-training.

[0111] If the pre-trained attack agent exits successfully and completes the attack intention, the reward of the decision agent is the ratio of the attack action entries of the attack agent after deduplication to the total number of time steps. In other cases, the reward is zero.

[0112] The decision agent is trained using DDPG to obtain a decision agent that can dynamically adjust the honeypot type according to the interaction between the attack agent and the honeypot agent.

[0113] After completing steps S3-S5, this embodiment obtains the pre-trained honeypot agent and decision agent, and constructs a layered defense agent model in an attack-defense interactive environment, such as Figure 4 As shown in the figure, the decision agent and each honeypot agent in the layered defense agent are in the initial state. and Decision-making agent performs actions That is, send the target to each honeypot agent The goal defines the type and location of each honeypot. Each honeypot agent chooses an action based on the goal and initial state. That is, deploy vulnerabilities. Each honeypot has The probability of transferring to state And obtain the corresponding reward value according to the reward function Each honeypot agent interacts with the environment until, after τ steps, the decision agent The probability of transferring to state and receive external rewards During the entire learning process, the layered defense agent generates the optimal action strategy that maximizes the reward value through high-iteration and high-interaction self-learning, thus achieving the final honeynet deployment. Table 2 shows the modeling process of the layered defense agent.

[0114] Table 2

[0115]

[0116] Since it is not easy to converge directly with the attacker and the layered defense agent, the layered defense agent is trained in stages. In the preparatory stage, the DDPG algorithm in reinforcement learning is used to train a pre-trained attack agent based on the optimal action strategy, so that it can attack efficiently in the intranet environment. In the training stage, first, the pre-trained attack agent is modeled into the environment, and an environment containing the attack agent is created for each honeypot agent. Each honeypot agent adjusts the number of types of vulnerabilities in the environment by selecting actions. The DDPG algorithm is used to obtain a pre-trained honeypot agent that dynamically adjusts the environment when the attacker attacks. Secondly, the pre-trained attack agent and n pre-trained honeypot agents are used as the environment for training the decision agent. The decision agent assigns the position and type of each honeypot agent through actions. The DDPG algorithm is used to train a decision agent that dynamically changes the service type of the honeypot agent according to the attacker's information, that is, a decision agent that dynamically changes the state of the honeynet. In this way, a pre-trained layered defense agent is obtained. Finally, the improved MADDPG is used to train 1 pre-trained attack agent, n pre-trained honeypot agents, and 1 pre-trained strategy agent obtained in the aforementioned training process. The agents learn through interaction to promote the development of cooperative or confrontational strategies.

[0117] In MADDPG, there are multiple agents, each of which has a policy network, which inputs the current state and outputs the action. The policy network uses parameters θ i , where i represents the i-th agent. The policy network uses a deterministic policy μi (s|θ i ) to select actions; each agent also has a value network (critic network), which inputs the current state and the actions of all agents and outputs a Q value, which represents the value of the current state and action. The value network uses the parameter φ i Indicates that the Q function is Q i (s i ,a attackrr ,a hp1 ,……,a hpn ,a decision |φ i ), where a hp1 ,a hp2 ,……,a hpn are the actions of all underlying honeypot agents, a attacker is the current attacker's action, a decision is the action of the decision agent, s i is the state of agent i.

[0118] The principle of the MADDPG algorithm itself is to use the Q value of the joint action of all agents as part of the input to update the actor and critic networks. The update of the actor network of all agents i is shown in formula (1), and the update formula of the critic network is shown in formula (2) and formula (3).

[0119]

[0120] y=r+γQ i ′(s′,a′ attacker ,a′ hp1 ,……,a′ hpn ,a′ desicion |φ i ′) (2)

[0121]

[0122] Among them, μ i (s|θ i ) represents a deterministic strategy, Q i (·) represents the Q function, a hp1 ,a hp2 ,……,a hpn are the actions of all pre-trained honeypot agents, a attacker Indicates the current attacker's action, a decision represents the action of the decision-making agent, φ i represents the value network, s i Represents the state of the i-th agent, and i represents the i-th agent.

[0123] However, in the actual interaction process with the attacker, not every honeypot agent will interact with the attacker. Only the information collected by the honeypot agents that interact with the attacker is more valuable. For example: honeypot agent A and honeypot agent B interact with the attacker, and honeypot agent C does not interact with the attacker. Then, the network update direction of honeypot agents A and B is more meaningful to the network update direction of the decision-making agent. Therefore, the actor network update of the decision-making agent is consistent with formula (1), while the critic network update of the decision-making agent is based on formula (3) plus the loss mean of the critic network of each honeypot agent that interacts with the attacker. Critic decision Represents the value of the critic network of the policy agent after being updated by MADDPG. decision′ represents the critic network value of the decision agent after the secondary distribution. The same is true for the honeypot agent. In each τ step, the number of interactions between the attacker and the honeypot agent is uncertain. The critic network update formula for the decision agent and the honeypot agent is as follows:

[0124]

[0125] Where n is the total number of honeypot agents, HP i For honeypot agent i, HP j A honeypot agent that interacts with the attacker.

[0126] When the strategy update reaches the maximum convergence times, the best honeynet deployment strategy is obtained.

[0127] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment, characterized in that: The steps include: Multiple nodes are randomly generated, and each node is assigned a honeypot agent. Each honeypot agent has at least one service and corresponding vulnerability. The connectivity between honeypot agents is established based on the service type, and the intranet environment is constructed. Construct an attack agent, which interacts with the intranet environment in stages. In the preparation stage, a pre-trained attack agent is obtained, and in the hybrid training stage, an attack agent with judgment ability is obtained. According to the service type, each honeypot agent is assigned a pre-trained attack agent with different attack intentions. The pre-trained attack agent selects attack actions to change the intranet environment. The honeypot agent deploys appropriate vulnerabilities according to the intranet environment. After multiple interactions, the pre-trained honeypot agent is obtained. Initialize the pre-trained honeypot agent and use the pre-trained honeypot agent to build the honeynet environment; In the honeynet environment, the pre-trained attack agent interacts with the pre-trained honeypot agent to construct a decision agent. The decision agent predicts the attack intention of the pre-trained attack agent based on the interaction information, dynamically configures and adjusts the pre-trained honeypot agent and its services, and obtains the pre-trained decision agent. A layered defense agent is constructed using pre-trained honeypot agents and pre-trained decision agents. The pre-trained attack agent attacks the layered defense agent. The pre-trained decision agent obtains the information of the pre-trained attack agent and determines the type and attack intention, and sends out the defense target. The pre-trained honeypot agent accepts the defense target and automatically generates a service type vulnerability that matches the target. The MADDPG algorithm is used to train all agents, and the policy network and value network of each agent are updated to obtain the final honeypot agent, decision agent and attack agent.

2. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: The attack agent interacts with the intranet environment in stages, including a preparation stage and a training stage; Preparatory stage: Create attack action set A pre ={a1,a2,……,a n },a i represents the attack action i, n is the number of attack actions, and the attack action types include remote connection attacks, local attacks, and privilege escalation attacks. The attack agent is trained using the DDPG algorithm. The attack agent selects an attack action to attack the intranet environment. If the attack is successful, a reward is obtained; Hybrid training phase: add recognition honeypot agent actions and train attack action set A trn ={a1,a2,……,a n ,b1,b2,……,b m }, b j represents the action j of identifying the honeypot, and m is the number of identified actions. The MADDPG algorithm is used to mix the attack agent with all other agents. The attack agent is trained in the attack action set A. trn Select actions to attack the intranet environment, and get rewards when the attacker agent successfully identifies the honeypot and exits.

3. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: The method of allocating pre-trained attack agents with different attack intentions to each honeypot agent according to the service type is as follows: Deploy the pre-trained attack agent in the intranet environment and set the attack intention set Des = {d1, d2, ..., d l }, l is the total number of attack intentions, and the honeypot agent is randomly assigned a service type it carries.

4. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: When pre-training the honeypot agent, the following steps are included: When the pre-trained honeypot agent chooses to add a vulnerability, it determines whether the pre-trained attack agent successfully exploits the vulnerability. Whenever the attack agent successfully exploits a vulnerability, the honeypot agent will receive a reward α·U(v t ); When the attacker successfully achieves his attack goal, the honeypot agent will receive a reward β·G, inducing the attacker to reveal his attack intention; the pre-trained reward function of the honeypot agent is as follows: Among them, U(v t ) indicates that the attacker exploits the vulnerability v t The number of times, G represents the score of the attacker achieving the attack goal, α and β are the weight coefficients of the reward, and c represents the number of vulnerabilities in the honeypot intelligent body; The four-tuple (the state of the honeypot agent, the selected action, the generated and deployed vulnerability, and the state of the next time step) is stored in the buffer. After a batch, the advantage function and value function are calculated and updated. After multiple iterations, the pre-trained honeypot agent is finally obtained.

5. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: The initialization of the pre-trained honeypot agent includes: Set the number of honeypot agents, set the service set and vulnerability set of the honeypot agents; the number of honeypot agents V (hp i ), as follows: V(hp i )={v d ∣v d ∈Vul,P(v d )>P threshold |∣V(hp i )∣≤ρ×q} Among them, hp i represents the i-th honeypot, v d Indicates service j The vulnerability in P(v d ) represents the probability of the vulnerability being selected, P threshold is the threshold of the probability of selection, |V(hp i )| represents the number of vulnerabilities. The number of honeypot vulnerabilities generated during initialization is less than ρ times the total number of service vulnerabilities. Set the service type set, vulnerability set, total vulnerability set, and the method of randomly generating service types for each honeypot; Build connectivity between honeypot agents based on a set of service types.

6. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: When pre-training the decision agent, the following steps are included: S51. At time t, the pre-trained attack agent selects an attack action, the pre-trained honeypot agent interacting with the pre-trained attack agent selects a defense action, and the other pre-trained honeypot agents remain stationary; the defense action includes adding vulnerabilities and selecting vulnerability types; S52, recording training process information; S53, after τ steps, the decision agent reallocates services to other pre-trained honeypot agents; S54. When the pre-trained attack agent successfully exits or completes the attack intention, a reward is obtained; If the pre-trained attack agent exits successfully and completes the attack intention, the reward of the decision agent is the ratio of the attack action entries of the attack agent after deduplication to the total number of time steps. In other cases, the reward is zero. S55. After multiple rounds of iterations, the pre-training of the decision-making agent is completed.

7. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: The method of using a pre-trained honeypot agent and a pre-trained decision agent to construct a layered defense agent includes: The initial states of the decision agent and each honeypot agent in the layered defense agent are and The decision agent sends a target to each honeypot agent. Each target defines the type and location of each honeypot; each honeypot agent selects an action based on the sent target and initial state. Each honeypot agent The probability of transferring to state And obtain the corresponding reward value according to the reward function Each honeypot agent interacts with the environment until, after τ steps, the decision agent The probability of transferring to state and receive external rewards The layered defense agent generates an optimal action strategy that maximizes the reward value.

8. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: The dynamic configuration and adjustment of the pre-trained honeypot agent and its services are specifically as follows: according to the pre-trained attack agent information, the decision agent dynamically allocates the position and service type of each honeypot agent through actions.

9. According to claim 1, the method for generating a honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: The policy network θ i The update is as follows: Among them, μ i (s|θ i ) represents a deterministic strategy, Q i (·) represents the Q function, a hp1 ,a hp2 ,……,a hpn are the actions of all pre-trained honeypot agents, a attacker Indicates the current attacker's action, a decision represents the action of the decision-making agent, φ i represents the value network, s i Represents the state of the i-th agent, and i represents the i-th agent.

10. According to claim 1, the method for generating honeynet deployment strategy based on reinforcement learning in an intranet environment is characterized in that: The value network update formula of the decision agent and the honeypot agent is as follows: Among them, Critic decision′ Represents the value network value of the decision-making agent after secondary distribution, Critic decision represents the value of the critic network of the policy agent after being updated by MADDPG, n is the total number of honeypot agents, HP i is the i-th honeypot agent, HP j A honeypot agent that interacts with the attacker.

Images