A Reinforcement Learning-Based Method for Generating Honeynet Deployment Strategies in an Intranet Environment

Abstract

This invention discloses a method for generating honeypot deployment strategies based on reinforcement learning in an intranet environment. The method includes: randomly generating multiple nodes to construct an intranet environment; constructing attack agents, which interact with the intranet environment in stages to pre-train the attack agents; assigning pre-trained attack agents with different attack intentions to each honeypot agent, and pre-training the honeypot agents in the intranet environment; initializing the pre-trained honeypot agents and constructing the honeypot environment using them; in the honeypot environment, the pre-trained attack agents interact with the pre-trained honeypot agents to pre-train decision agents; training all agents using the MADDPG algorithm, constructing a hierarchical defense agent using the pre-trained honeypot agents and the pre-trained decision agents, and updating the policy network and value network of each agent. This invention improves the adaptability of the honeypot system and enhances the security protection level of the intranet environment through dynamic deployment strategies.

Description

Technical Field

[0001] This invention belongs to the technical field of network security, specifically relating to a method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment. Background Technology

[0002] Traditional defenses primarily rely on existing attacks to formulate corresponding rules, making them ineffective against unknown attacks. There is a lack of coordination between different network security protection devices. Existing detection mechanisms mostly rely on anomaly indicators for identification, which is inherently lagging. Unlike direct defense against network attacks, honeypot technology reduces the threat of network intrusions to targets by deceiving attackers and significantly depleting their resources. It also analyzes captured attack traffic and behaviors to obtain information such as the attacker's methods, tools, and intent. However, traditional honeypots require determining their network location during deployment, typically in a DMZ (Demilitarized Zone), and configuring network switches and routers to assign IP addresses. This results in static configurations and fixed locations, making them easily identifiable, labeled, and bypassed by attackers. Therefore, traditional honeypots represent a "passive-active defense" approach. Furthermore, honeypots offer only a limited perspective, usually simulating the behavior of a single system or service. In contrast, honeynets configure multiple vulnerable honeypots as traps, providing more comprehensive attack data. This data includes information such as attackers' lateral movement within the internal network and the full picture of the attack chain, which helps to better understand and respond to cyber threats.

[0003] Research on honeynet deployment strategy generation can be broadly categorized into two types. One type involves dynamically deploying honeynets based on attacker attack information analysis. Bartwal et al. analyzed information from attacker pcap packets, retrieved honeypot image files corresponding to malicious traffic attack categories on virtual machines, and linked these image files with the destination IPs in the pcap packets, thus achieving dynamic honeypot deployment and solving the problem of when to deploy which type of honeypot. Huang et al. focused on the harm caused to networks by internal threats. By modeling defenders and users, they proposed a new game theory model and combined mathematical programming and graphical methods to calculate the optimal solution, obtaining a systematic solution for network deception defense mechanisms. They modeled a two-person zero-sum game between network defenders and attackers, constructed a reward function, and solved it using different calculation methods to obtain the honeypot deployment location. Regarding the application of dynamic honeypot deployment, Gao et al. modeled attackers to obtain attacker behavior data, used neural networks to learn from the data, predicted attacker intentions, and adjusted deployment strategies. Wang et al. combined reinforcement learning methods to model offensive and defensive intelligent agents. The defender predicts the attacker's attack nodes, places honeypots, and designs a two-layer attack graph to narrow the range of honeypot deployment locations.

[0004] Another approach focuses on modeling the network environment and dynamically deploying honeynets by analyzing its characteristics. Kong et al. aimed to automate honeynet deployment strategies in cloud environments by monitoring changes in the cloud system in real time, updating attack graphs, identifying key nodes in the attack graphs, and abstracting honeynet deployment into a set coverage problem, ultimately achieving dynamic honeynet deployment. Yang et al. developed honeynet deployment strategies by analyzing the vulnerability of each node in the network through attack graph analysis. They used ant colony optimization to simulate honeypot retention decisions and wolf pack optimization to determine the value of honeypots based on information collected by the honeypots, thus achieving dynamic honeynet deployment. Fraunholz et al. proposed a context-aware method to correctly deploy and configure honeypots to avoid attackers identifying honeypot fingerprints. They used internal network configuration information as training data, clustered it using the k-means algorithm, and automatically generated configuration information based on entity characteristics.

[0005] Methods that model attackers or attack information and analyze it to dynamically generate honeynet deployment strategies require a deep understanding of attacker behavior and attack methods. This modeling process involves not only abstract representations of attacker motivations, target selection, and attack methods, but also the dynamic characteristics of behavior evolving over time. While these methods deploy honeypots or honeynets by modeling the game between attackers and defenders, they primarily consider the deployment location of each honeypot and neglect the interrelationships between them.

[0006] This method for generating and configuring honeynets within an intranet environment identifies key nodes by mapping host configuration information to connectivity relationships between hosts. Honeypots are then deployed on or near these key nodes to enhance the effective detection and defense against known threats. However, the method relies heavily on modeling and reasoning about known network vulnerabilities in selecting key nodes, which limits its ability to identify and defend against unknown attack patterns. Summary of the Invention

[0007] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a method for generating honeypot deployment strategies based on reinforcement learning in an intranet environment. Addressing the problems of static configuration and fixed location in traditional honeypot deployment applications, which are easily identified, labeled, and bypassed by attackers, this invention utilizes a reinforcement learning framework to model the interaction process between the attacking agent and the simulated enterprise intranet environment, and to achieve dynamic changes in the honeypot environment. Simultaneously, by dynamically changing the honeypot's IP, services, vulnerabilities, and other information, it aims to alleviate the problem of honeypots being easily circumvented by attackers, delay attack behavior, identify attack characteristics, and assist in judging attack intent, thereby achieving "automated proactive defense," improving defense efficiency, and effectively protecting service resources.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] This invention provides a method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment, comprising the following steps:

[0010] Multiple nodes are randomly generated, and each node is assigned a honeypot agent. Each honeypot agent has at least one service and a corresponding vulnerability. The connectivity between honeypot agents is built based on the service type to construct the intranet environment.

[0011] An attack agent is constructed, and the attack agent interacts with the intranet environment in stages. In the preparatory stage, a pre-trained attack agent is obtained, and in the mixed training stage, an attack agent with judgment capabilities is obtained.

[0012] Based on the service type, each honeypot agent is assigned a pre-trained attack agent with different attack intentions. The pre-trained attack agent selects attack actions to change the internal network environment. The honeypot agent deploys appropriate vulnerabilities according to the internal network environment. After multiple interactions, the pre-trained honeypot agent is obtained.

[0013] Initialize the pre-trained honeypot agent and use the pre-trained honeypot agent to build a honeycomb environment;

[0014] In a honeynet environment, pre-trained attack agents interact with pre-trained honeypot agents to build decision agents. The decision agents infer and predict the attack intentions of the pre-trained attack agents based on the interaction information, dynamically configure and adjust the pre-trained honeypot agents and their services, and obtain the pre-trained decision agents.

[0015] A hierarchical defense agent is constructed using pre-trained honeypot agents and pre-trained decision agents. A pre-trained attack agent attacks the hierarchical defense agent. The pre-trained decision agent obtains information from the pre-trained attack agent, determines the type and attack intent, and issues defense targets. The pre-trained honeypot agent accepts the defense targets and automatically generates service type vulnerabilities matching the targets. The MADDPG algorithm is used to train all agents, updating the policy network and value network of each agent to obtain the final honeypot agent, decision agent, and attack agent.

[0016] As a preferred technical solution, the attacking agent interacts with the intranet environment in stages, including a preparation stage and a training stage.

[0017] Preparation phase: Create attack action set A pre ={a1,a2,……,a n},a i Let i represent the attack action, and n represent the number of attack actions. The types of attack actions include remote connection attacks, local attacks, and privilege escalation attacks. The attack agent is trained using the DDPG algorithm. The attack agent selects attack actions to attack the intranet environment. If the attack is successful, it receives a reward.

[0018] Hybrid training phase: Incorporate the recognition of honeypot agent actions to train the attack action set A. trn ={a1,a2,……,a n ,b1,b2,……,b m}, b j Let j represent the action j for identifying honeypots, and m represent the number of actions to be identified. The MADDPG algorithm is used to train the attack agent in a hybrid training set A with all other agents. trn The attacker chooses an action to attack the internal network environment. When the attacker's agent successfully identifies the honeypot and exits, it receives a reward.

[0019] As a preferred technical solution, the step of assigning pre-trained attack agents with different attack intentions to each honeypot agent according to the service type specifically includes:

[0020] Deploy the pre-trained attack agent in the intranet environment and set the attack intent set Des = {d1, d2, ..., d...} l}, l represents the total number of attack intentions, and a service type is randomly assigned to the honeypot agent.

[0021] As a preferred technical solution, the pre-training of the honeypot agent includes the following steps:

[0022] When the pre-trained honeypot agent chooses to add vulnerabilities, it determines whether the pre-trained attack agent successfully exploits the vulnerability. Each time the attack agent successfully exploits a vulnerability, the honeypot agent receives a reward α·U(v). t When the attacker successfully achieves their attack objective, the honeypot agent will receive a reward β·G, inducing the attacker to reveal their attack intent; the reward function of the honeypot agent's pre-training is as follows:

[0023]

[0024] Among them, U(v) t This indicates that the attacker exploited the vulnerability v. t The number of times, G represents the score of the attacker achieving the attack objective, α and β are the weighting coefficients of the reward, and c represents the number of vulnerabilities in the honeypot intelligent body;

[0025] The quadruple (the state of the honeypot agent, the selected action, the vulnerabilities generated and deployed, and the state at the next time step) is stored in a buffer. After a batch, the advantage function and value function are calculated and updated. After multiple iterations, the pre-trained honeypot agent is finally obtained.

[0026] As a preferred technical solution, the initialization of the pre-trained honeypot agent includes:

[0027] The number of honeypot agents is set, and the service set and vulnerability set of the honeypot agents are set; the number of honeypot agents V(hp) i ), as shown in the following formula:

[0028] V(hp i )={v d |v d ∈Vul,P(v d )>P threshold |∣V(hp i )∣≤ρ×q}

[0029] Among them, hp i Let v represent the i-th honeypot. d Indicates service ser j The vulnerability exists in P(v) d P represents the probability that a vulnerability is selected. threshold It is the threshold for the selection probability, |V(hp) i The number of vulnerabilities is represented by |. The number of honeypot vulnerabilities generated during initialization is less than ρ times the total number of vulnerabilities in the service.

[0030] Configure the service type set, vulnerability set, total vulnerability set, and the method for randomly generating service types for each honeypot;

[0031] Connectivity between honeypot agents is constructed based on the set of service types.

[0032] As a preferred technical solution, the pre-training of the decision-making agent includes the following steps:

[0033] S51. At time t, the pre-trained attack agent selects an attack action, the pre-trained honeypot agent that interacts with the pre-trained attack agent selects a defense action, and other pre-trained honeypot agents remain stationary; the defense action includes adding vulnerabilities and selecting vulnerability types.

[0034] S52. Record training process information;

[0035] S53. After τ steps, the decision-making agent reallocates services to other pre-trained honeypot agents;

[0036] S54. When the pre-trained attack agent successfully exits or completes its attack intent, it receives a reward.

[0037] If the pre-trained attack agent successfully exits and completes its attack intent, the reward for the decision agent is the ratio of the number of attack action entries of the deduplicated attack agent to the total number of time steps; otherwise, the reward is zero.

[0038] S55. After multiple iterations, the pre-training of the decision-making agent is completed.

[0039] As a preferred technical solution, the construction of a layered defense agent using pre-trained honeypot agents and pre-trained decision agents includes:

[0040] The initial states of the decision-making agent and each honeypot agent in the layered defense agent are as follows: and

[0041] The decision-making agent sends the target to each honeypot agent. Each objective defines the type and location of each honeypot; each honeypot agent selects its action based on the issued objective and initial state.

[0042] Each honeypot agent uses Probability transition to state And obtain the corresponding reward value according to the reward function. Each honeypot agent interacts with the environment until step τ, at which point the decision-making agent... Probability transition to state And receive external rewards

[0043] The layered defense agent generates the optimal action strategy that maximizes the reward value.

[0044] As a preferred technical solution, the dynamic configuration and adjustment of pre-trained honeypot agents and their services specifically involves: based on the pre-trained attack agent information, the decision agent dynamically allocates the position and service type of each honeypot agent through actions.

[0045] As a preferred technical solution, the policy network θ i The update is as follows:

[0046]

[0047] Where, μ i (s∣θ i ) represents a deterministic strategy, Q i (·) represents the Q function, a hp1 ,a hp2 ,……,a hpn It refers to the actions of all pre-trained honeypot agents, a attacker Indicates the current action of the attacker, a decision φ represents the action of the decision-making agent. i Represents the value network, s i Let i represent the state of the i-th agent, where i represents the i-th agent.

[0048] As a preferred technical solution, the value network update formula for the decision-making agent and the honeypot agent is as follows:

[0049]

[0050] Among them, Critic decision′ Critic represents the value network value of the decision-making agent after secondary allocation. decision This represents the value of the critic network of the policy agent after being updated by MADDPG, where n is the total number of honeypot agents, and HP i For the i-th honeypot agent, HP j A honeypot agent that interacts with the attacker.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] (1) This invention models the interaction process between the attack agent and the enterprise intranet environment simulated by the honeynet, and realizes the dynamic transformation of the honeynet environment. By dynamically changing the IP, services, vulnerabilities and other information of the honeypot, it alleviates the problem that the honeypot is easily evaded by attackers, delays attack behavior, identifies attack characteristics and assists in judging attack intentions, thereby realizing "automated active defense", improving defense efficiency and effectively protecting service resources.

[0053] (2) By modeling the attacker, the defender can take an active defense strategy, that is, predict and prepare corresponding defense measures before the attack occurs, instead of just passively responding to the attack. Compared with the original work, in the process of modeling the attack agent in this invention, the actions corresponding to different stages are different, thereby realizing fine control and adjustment of behavior at different stages.

[0054] (3) This invention draws on the idea of ​​layered reinforcement agents. The deployment of honeynet is modeled as a layered defense agent under a multi-agent system. The honeynet environment suitable for attackers is constructed by the high-level decision agent and the low-level honeypot agent together. After an attack is detected, the attacker's traffic will be redirected to the honeynet environment. Since the decision agent in the layered defense agent summarizes information and uniformly schedules each honeypot in the honeynet, this increases the linkage between honeypots.

[0055] (4) The present invention adopts a custom virtual honeynet scenario to adaptively configure the host information in the intranet according to the scenario requirements, uses reinforcement learning algorithm to train the layered defense agent, controls the deployment location of each honeypot in the honeynet, and the vulnerability type carried in each honeypot; through reasonable honeypot deployment strategy, it infers and verifies the attacker's attack intention, realizes delayed attack, or even source tracing and countermeasure.

[0056] (5) Through this intelligent dynamic deployment strategy, the present invention not only improves the self-adaptive capability of the honeynet system, but also significantly enhances the security protection level of the intranet environment. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram illustrating the dynamic deployment of a honeynet in an intranet scenario according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram illustrating the interaction process between the attacker and the honeypot agent in an embodiment of the present invention.

[0061] Figure 4 This is a structural diagram of the hierarchical defense agent decision-making model according to an embodiment of the present invention. Detailed Implementation

[0062] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0063] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0064] Please see Figure 1 This embodiment provides a method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment, including the following steps:

[0065] S1. Randomly create an intranet environment.

[0066] A random intranet environment is created, with N host nodes randomly generated. Each host node hosts at least one service. Vulnerabilities are randomly selected from a vulnerability set and generated. Please refer to [link / reference]. Figure 2 .

[0067] The set of host nodes (N), N = {h1, h2, ..., h n},h i Let represent the i-th host, and n be the number of hosts.

[0068] The service type set (Services) is a collection of service types that can run on the host, Services = {ser1, ser2, ..., ser...} m},ser j Let j represent the service type and m be the number of service types.

[0069] A vulnerability set (Vul) is a set of vulnerabilities for a certain service. jThe corresponding set of existing vulnerabilities. Vul = {v1, v2, ..., v...} q},v d Indicates vulnerability d, where q represents the vulnerability ser. j The number of vulnerabilities. Therefore, the total vulnerability set Vuln = {Vul1, Vul2, ..., Vuln} m The total set of vulnerabilities is the sum of all vulnerabilities.

[0070] Randomly assign one or more services to each host. Services(h i )={ser j ∣ser j ∈Ser,P(ser j )>P threshold} where h i Represents the i-th host, ser j Indicates the service type, P(ser) j ) indicates service ser j The probability of being selected, P threshold It is the threshold for the selection probability.

[0071] Then select vulnerabilities from the vulnerability set corresponding to the service. V(h) i )={v d |v d ∈Vul,P(v d )>P threshold} where h i Let v represent the i-th host. d This indicates a vulnerability present on the host, P(v d P represents the probability that a vulnerability is selected. threshold It is the threshold for the selection probability.

[0072] Based on the service type, connectivity between hosts is constructed, that is, hosts of the same service type have connectivity.

[0073] S2, pre-trained attack agent.

[0074] This embodiment uses a randomly created intranet environment as the training environment for the attack agent, and trains it through phased interactions with the intranet environment. This phased attack agent aims to simulate potential threats in the real world to achieve comprehensive training and evaluation of the defense agent. Specifically, the pre-training of the attack agent includes a preparatory phase and a mixed training phase.

[0075] The attack agent is trained using the DDPG algorithm. The attack agent selects attack actions to attack the internal network environment. If the attack is successful, it receives a reward.

[0076] In the preparation phase, the attacking agent's primary objective is to launch an efficient attack and achieve its goal. The set of attack actions in this phase is A. pre ={a1,a2,……,a n},a i Let i represent the attack action, and n represent the number of attack actions. The attack action type can be a. att ={Remote, local, privilege}, which correspond to remote connection attacks, local attacks, and privilege escalation attacks, respectively. During actual training, one or more of these can be selected from the set to attack the target.

[0077] During the hybrid training phase, the attacking agent's role is to achieve its attack objective while avoiding triggering honeypots. The set of attack actions for this phase is A. trn ={a1,a2,……,a n ,b1,b2,……,b m},a i Indicates attack actions i, b j Let j represent the action j for identifying honeypots, n represent the number of attack actions, and m represent the number of actions to be identified.

[0078] After multiple iterations of training, a pre-trained attack agent capable of efficiently attacking within an intranet environment was finally obtained. The modeling process of the phased attack agent is shown in Table 1.

[0079] Table 1

[0080]

[0081] The advantage of adopting a phased approach is that the preparatory phase only involves the attack agent learning how to attack in the intranet environment. After the preparatory phase, an attack agent that can attack efficiently is obtained. Compared with directly training the attacker and the layered defense agent, this phased approach improves the efficiency of the adversarial process between the attack agent and the layered defense agent in the training phase and saves training resources.

[0082] S3, train each underlying honeypot agent.

[0083] Each underlying honeypot agent is sequentially assigned a pre-trained attack agent with different attack intentions, i.e., the attack intention set Des = {d1, d2, ..., d...} l}, l represents the total number of attack intentions; and a service type is randomly assigned to each service it carries, Services(h i )={ser j ∣ser j ∈Random_Select(Ser)}. The pre-trained attack agent selects attack actions, changes the environment, and uses the DDPG algorithm to train the honeypot intelligence so that it can select appropriate actions based on the environment.t ={add_vulnerability,static}

[0084] Suppose that at time step t, the state of the honeypot agent is s. t The action it chooses is a t The vulnerability in honeypot agent generation and deployment is v. t Attackers exploited vulnerability v t The number of times is U(v) t ), where G is the score G (G = 100) for the attacker to achieve the attack objective, and α and β are the weighting coefficients of the reward, reflecting the degree of influence of each behavior on the overall reward.

[0085] Whenever an attacking agent successfully exploits a vulnerability, the honeypot agent will receive α·U(v t The reward mechanism is designed to ensure that vulnerabilities generated by the honeypot agent are exploitable, thereby attracting attackers to interact. When an attacker successfully achieves their attack objective, the honeypot agent receives a reward of β·G. This reward is designed to encourage the honeypot agent to induce attackers to reveal their true purpose, thereby obtaining valuable attack data. Therefore, if a honeypot agent has c vulnerabilities, the reward function R(s) is... t ,a t This can be represented as:

[0086] (s) t ,a t ,r t ,s t+1 The data is stored in a buffer. After each batch, the advantage function and value function are calculated and updated. After multiple iterations, the pre-trained honeypot agent is finally obtained.

[0087] S4. Initialize the honeynet environment.

[0088] A honeycomb environment is constructed using pre-trained honeypot agents. Services running in N honeypots within the honeycomb environment and the connectivity between the honeypots are randomly generated.

[0089] First, define the parameters in the honeynet environment:

[0090] Number of honeypots (N): The number of honeypots contained in a honey web, N = {hp1, hp2, ..., hp} n},hp i Let i represent the i-th honeypot, and n be the number of honeypots.

[0091] The service type set (Services), vulnerability set (Vul), total vulnerability set (Vuln), and the method of randomly generating service types for each honeypot are consistent with those in S1.

[0092] Limitation on the number of honeypot agents: Since subsequent interactions also require adding vulnerabilities, the number of vulnerabilities used to initialize a honeypot must be strictly limited. V(hp) i )={v d |v d ∈Vul,P(v d )>P threshold |∣V(hp i ) |≤0.2×q} where, hp i Let v represent the i-th honeypot. d Indicates service ser j The vulnerability exists in P(v) d P represents the probability that a vulnerability is selected. threshold This is the threshold for the selection probability. |V(hp) i The )| indicates the number of vulnerabilities. The number of honeypot vulnerabilities generated during initialization is less than 20% of the total number of vulnerabilities in the service.

[0093] The connectivity between honeypot agents is the same as in S1.

[0094] Secondly, the attacker's traffic is redirected to one of the honeypots in the virtual honeynet environment. During the interaction, the honeypot agent dynamically deploys vulnerabilities in the current environment, collects information, and maintains the interaction with the attacker. After τ steps, the decision agent summarizes the attacker information in the honeynet environment, determines the attacker type, infers the attacker's intent, and redistributes the location and service type of each honeypot except for the one currently interacting, based on the attacker type. The honeypot agent receives the target issued by the decision agent and automatically generates service type vulnerabilities that match the target.

[0095] like Figure 2 and Figure 3 The example of a dynamic honeynet deployment scenario shown initially consists of 1 router, 2 switches (switch1 and switch2), and 5 hosts (client1, client2, and client3). hp webserver1 hp webserver2 hp mysql1 hp mysql2 hp It consists of one mainframe and one decision-making agent. The five mainframes are simulated by Honeypot Intelligence, and the simulation services are categorized as follows:

[0096] client hp : The developer's host.

[0097] webserver hp Servers that host portal website services.

[0098] mysql hp SQL Server hp Servers that host data storage services.

[0099] The attacker's traffic was redirected to webserver1. hp In the middle, at this time webserver1 hp The service being run is owncloud, a website focused on file services, which contains two vulnerabilities, CVE1 and CVE2.

[0100] S41. At time t, the attacker first performs action 1 (attacker_action1) to scan for vulnerability information. The defender chooses action 1 (defender_action1) to remain unchanged. At this time, webserver1... hp It contains two vulnerabilities and displays their vulnerability information.

[0101] At time S42 and t+1, the attacker chooses to execute action 2 (attacker_action2) to exploit CVE1, and the defender chooses action 2 (defender_action2) to add CVE3 vulnerability. At this time, webserver1... hp It contains three vulnerabilities and shows that after CVE1 is successfully executed, it enters the interface inside owncloud.

[0102] At step S43, time t+2, the attacker chooses to perform action 3 (attacker_action3) to collect information, while the defender chooses action 1 (defender_action1) to remain unchanged. At this time, webserver1... hp It contains three vulnerabilities and shows that after CVE1 is successfully executed, it enters the interface inside owncloud.

[0103] Step S44: At time t+3, the attacker chooses to execute action 4 (attacker_action4) to exploit CVE3, and the defender chooses action 3 (defender_action3) to add privilege escalation vulnerability CVE4. At this time, webserver1... hp It contains four vulnerabilities and returns the MySQL password obtained after CVE3 is successfully executed.

[0104] At step S45, t+4, the attacker may exit, move laterally to another host simulated by the honeypot based on the MySQL password, or remain on webserver1. hp The attack continues. A summary of the policy-based intelligent agents is provided for webserver1. hpBased on the attack information, it was determined that the attacker's intent was to steal data. Therefore, after step four, the number of honeypots hosting portal websites was reduced, and the number of honeypots hosting databases was increased. Figure 3 SQL Server has been added. hp If the attacker continues on webserver1 hp If the attacker continues to exploit vulnerabilities and interact with the system, they will remain stationary if they move laterally to another host or exit the system.

[0105] S5, Training the decision-making agent.

[0106] The obtained pre-trained attack agent and N pre-trained honeypot agents are used as part of the environment to obtain the pre-trained decision agent. Specifically, this includes the following steps:

[0107] S51. At time t, the pre-trained attack agent selects an attack action, the pre-trained honeypot agent that interacts with the pre-trained attack agent selects a defense action, and other pre-trained honeypot agents remain stationary; the defense action includes adding vulnerabilities and selecting vulnerability types.

[0108] S52. Record training process information and train the decision-making agent;

[0109] S53, the agent reassigns services to other pre-trained honeypot agents. h The assigned action is a t (hp i )={select(ser j )∣ser j ∈Servies};

[0110] S54. When the pre-trained attack agent successfully exits or completes its attack intent, it receives a reward and completes the pre-training.

[0111] If the pre-trained attack agent successfully exits and completes its attack intent, the reward for the decision agent is the ratio of the deduplicated attack agent's attack action entries to the total number of time steps; otherwise, the reward is zero.

[0112] DDPG was used to train the decision agent, resulting in a decision agent that can dynamically adjust the honeypot type based on the interaction between the attack agent and the honeypot agent.

[0113] After completing steps S3-S5, this embodiment obtains a pre-trained honeypot agent and a decision agent, and constructs a layered defense agent model in an attack-defense interactive environment, such as... Figure 4 As shown, the decision-making agent and each honeypot agent in the hierarchical defense agent are each in their initial state. and Decision-making agents execute actions That is, to issue a target to each honeypot agent. The objective defines the type and location of each honeypot. Each honeypot agent selects an action based on the issued objective and initial state. That is, deploying vulnerabilities. Each honeypot uses... Probability transition to state And obtain the corresponding reward value according to the reward function. Each honeypot agent interacts with the environment until step τ, at which point the decision-making agent... Probability transition to state And receive external rewards Throughout the learning process, the hierarchical defense agent generates the optimal action strategy that maximizes the reward value through high-iteration and high-interaction self-learning, thus achieving the final honeynet deployment. Table 2 shows the modeling process of the hierarchical defense agent.

[0114] Table 2

[0115]

[0116] Because directly training attackers and layered defense agents is difficult to converge, the layered defense agents are trained in stages. In the preparatory stage, the DDPG algorithm from reinforcement learning is used to train a pre-trained attack agent based on optimal action policies, enabling it to efficiently attack within the internal network environment. In the training stage, firstly, the pre-trained attack agents are modeled into the environment. An environment containing the attack agent is created for each honeypot agent. Each honeypot agent adjusts the types and number of vulnerabilities in the environment by selecting actions. The DDPG algorithm is used to obtain a pre-trained honeypot agent that dynamically adjusts the environment in response to attacker attacks. Secondly, the pre-trained attack agents and n pre-trained honeypot agents are used together as the environment for training the decision agent. The decision agent assigns the position and type of each honeypot agent through actions. The DDPG algorithm is used to train a decision agent that dynamically changes the service type of the honeypot agents based on attacker information, i.e., dynamically changes the honeynet state. This results in a pre-trained layered defense agent. Finally, using the improved MADDPG, we train one pre-trained attack agent, n pre-trained honeypot agents, and one pre-trained policy agent obtained in the aforementioned training process. The agents learn through interaction, promoting the development of cooperative or adversarial strategies.

[0117] In MADDPG, there are multiple agents, each with its own policy network. This network takes the current state as input and outputs an action. The policy network uses parameters θ. i Let represent the i-th agent. The policy network uses a deterministic policy μ.i (s∣θ i Each agent selects actions using a parameter φ; each agent also has a critical network, which takes the current state and all actions of all agents as input and outputs a Q-value, representing the value of the current state and actions. i This indicates that the Q function is Q. i (s i ,a attackrr ,a hp1 ,……,a hpn ,a decision |φ i ), where a hp1 ,a hp2 ,……,a hpn It refers to the actions of all underlying honeypot agents, a attacker This refers to the current attacker's action, a. decision It refers to the actions of the decision-making agent, s i It is the state of agent i.

[0118] The principle of the MADDPG algorithm is to use the Q value of the joint action of all agents as part of the input for updating the actor and critic networks. The update of the actor network of all agents i is shown in formula (1), and the update formula of the critic network is shown in formula (2) and formula (3).

[0119]

[0120] y = r + γQ i ′(s′,a′ attacker ,a′ hp1 ,……,a′ hpn ,a′ desicion |φ i ′) (2)

[0121]

[0122] Where, μ i (s∣θ i ) represents a deterministic strategy, Q i (·) represents the Q function, a hp1 ,a hp2 ,……,a hpn It refers to the actions of all pre-trained honeypot agents, a attacker Indicates the current action of the attacker, a decision φ represents the action of the decision-making agent. i Represents the value network, s i Let i represent the state of the i-th agent, where i represents the i-th agent.

[0123] However, in actual interactions with attackers, not every honeypot agent necessarily interacts with the attacker. Only the information collected by honeypot agents that interact with the attacker is more valuable. For example, if honeypot agents A and B interact with the attacker, but honeypot agent C does not, then the network update directions of honeypot agents A and B are more meaningful for the network update direction of the decision agent. Therefore, the actor network update of the decision agent is consistent with formula (1), while the critic network update of the decision agent is based on formula (3), plus the average loss of the critic network of each honeypot agent that interacts with the attacker. decision This represents the value of the critic network of the policy agent after being updated by MADDPG. decision′ This represents the critic network value of the decision-making agent after secondary allocation. The same logic applies to the honeypot agent. The number of interactions between the attacker and the honeypot agent in each τ-step is uncertain. The update formulas for the critic networks of the decision-making agent and the honeypot agent are as follows:

[0124]

[0125] Where n is the total number of honeypot agents, HP i For honeypot agent i, HP j A honeypot agent that interacts with the attacker.

[0126] When the strategy update reaches the maximum number of convergences, the optimal honeynet deployment strategy is obtained.

[0127] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment, characterized in that, Includes the following steps: Multiple nodes are randomly generated, and each node is assigned a honeypot agent. Each honeypot agent has at least one service and a corresponding vulnerability. The connectivity between honeypot agents is built based on the service type to construct the intranet environment. An attack agent is constructed, and the attack agent interacts with the intranet environment in stages. In the preparatory stage, a pre-trained attack agent is obtained, and in the mixed training stage, an attack agent with judgment capabilities is obtained. Based on the service type, each honeypot agent is assigned a pre-trained attack agent with different attack intentions. The pre-trained attack agent selects attack actions to change the internal network environment. The honeypot agent deploys appropriate vulnerabilities according to the internal network environment. After multiple interactions, the pre-trained honeypot agent is obtained. Initialize the pre-trained honeypot agent and use the pre-trained honeypot agent to build a honeycomb environment; In a honeynet environment, pre-trained attack agents interact with pre-trained honeypot agents to build decision agents. The decision agents infer and predict the attack intentions of the pre-trained attack agents based on the interaction information, dynamically configure and adjust the pre-trained honeypot agents and their services, and obtain the pre-trained decision agents. A hierarchical defense agent is constructed using pre-trained honeypot agents and pre-trained decision agents. A pre-trained attack agent attacks the hierarchical defense agent. The pre-trained decision agent obtains information from the pre-trained attack agent, determines the type and attack intent, and issues defense targets. The pre-trained honeypot agent accepts the defense targets and automatically generates service type vulnerabilities matching the targets. The MADDPG algorithm is used to train all agents, updating the policy network and value network of each agent to obtain the final honeypot agent, decision agent, and attack agent.

2. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The attack agent interacts with the intranet environment in stages, including a preparation stage and a training stage. Preparation phase: Create attack action set A prer ={a1,a2,……,a i ,……,a n },a i Let i represent the attack action, and n represent the number of attack actions. The types of attack actions include remote connection attacks, local attacks, and privilege escalation attacks. The attack agent is trained using the DDPG algorithm. The attack agent selects attack actions to attack the intranet environment. If the attack is successful, it receives a reward. Hybrid training phase: Incorporate the recognition of honeypot agent actions to train the attack action set A. trn ={a1,a2,……,a i ,……,a n ,b1,b2,……,b j ,……,b m }, b j Let j represent the action j for identifying honeypots, and m represent the number of actions to be identified. The MADDPG algorithm is used to train the attack agent in a hybrid training set A with all other agents. trn The attacker chooses an action to attack the internal network environment. When the attacker's agent successfully identifies the honeypot and exits, it receives a reward.

3. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The process of assigning pre-trained attack agents with different attack intentions to each honeypot agent based on the service type is as follows: Deploy the pre-trained attack agent in the intranet environment and set the attack intent set Des = {d1, d2, ..., d...} l }, l represents the total number of attack intentions, and a service type is randomly assigned to the honeypot agent.

4. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The pre-training of honeypot agents includes the following steps: When the pre-trained honeypot agent chooses to add vulnerabilities, it determines whether the pre-trained attack agent successfully exploits the vulnerability. Each time the attack agent successfully exploits a vulnerability, the honeypot agent receives a reward α·U(v). t When the attacker successfully achieves their attack objective, the honeypot agent will receive a reward β·G, inducing the attacker to reveal their attack intent; the reward function of the honeypot agent's pre-training is as follows: Among them, U(v) t This indicates that the attacker exploited the vulnerability v. t The number of times, G represents the score of the attacker achieving the attack objective, α and β are the weighting coefficients of the reward, and c represents the number of vulnerabilities in the honeypot intelligent body; The quadruple {state of the honeypot agent, selected action, vulnerabilities generated and deployed, state of the next time step} is stored in a buffer. After a batch, the advantage function and value function are calculated and updated. After multiple iterations, the pre-trained honeypot agent is finally obtained.

5. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The initialization of the pre-trained honeypot agent includes: The number of honeypot agents is set, and the service set and vulnerability set of the honeypot agents are set; the number of honeypot agents V(hp) i ), as shown in the following formula: V(hp i )={v d ∣v d ∈Vul,P(v d )>P threshold |∣V(hp i )∣≤ρ×q} Among them, hp i Let v represent the i-th honeypot. d P(v) represents a vulnerability that exists in a certain service, Vul represents the set of vulnerabilities that exist in a certain service, and P(v) represents the set of vulnerabilities that exist in a certain service. d P represents the probability that a vulnerability is selected. threshold It is the threshold for the selection probability, |V(hp) i The number of vulnerabilities is represented by |. The number of honeypot vulnerabilities generated during initialization is less than or equal to ρ times the total number of vulnerabilities in the service, q. Configure the service type set, vulnerability set, total vulnerability set, and the method for randomly generating service types for each honeypot; Connectivity between honeypot agents is constructed based on the set of service types.

6. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The pre-training of the decision-making agent includes the following steps: S51. At time t, the pre-trained attack agent selects an attack action, the pre-trained honeypot agent that interacts with the pre-trained attack agent selects a defense action, and other pre-trained honeypot agents remain stationary; the defense action includes adding vulnerabilities and selecting vulnerability types. S52. Record training process information; S53. After τ steps, the decision-making agent reallocates services to other pre-trained honeypot agents; S54. When the pre-trained attack agent successfully exits or completes its attack intent, it receives a reward. If the pre-trained attack agent successfully exits and completes its attack intent, the reward for the decision agent is the ratio of the number of attack action entries of the deduplicated attack agent to the total number of time steps; otherwise, the reward is zero. S55. After multiple iterations, the pre-training of the decision-making agent is completed.

7. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The construction of a layered defense agent using pre-trained honeypot agents and pre-trained decision agents includes: The initial states of the decision-making agent and each honeypot agent in the layered defense agent are as follows: and The decision-making agent sends the target to each honeypot agent. Each objective defines the type and location of each honeypot; each honeypot agent selects its action based on the issued objective and initial state. Each honeypot agent uses Probability transition to state And obtain the corresponding reward value according to the reward function. Each honeypot agent interacts with the environment until step τ, at which point the decision-making agent... Probability transition to state And receive external rewards The layered defense agent generates the optimal action strategy that maximizes the reward value.

8. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The dynamic configuration and adjustment of pre-trained honeypot agents and their services specifically involves: based on the pre-trained attack agent information, the decision agent dynamically assigns the location and service type of each honeypot agent through actions.

9. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The policy network θ i The update is as follows: Where, μ i (s∣θ i ) represents a deterministic strategy, Q i (·) represents the Q function, a hp1 ,a hp2 ,……,a hpn It refers to the actions of all pre-trained honeypot agents, a attacker Indicates the current action of the attacker, a decision φ represents the action of the decision-making agent. i Represents the value network, s i Let a represent the state of the i-th agent, where i represents the i-th agent. i This indicates the current action of honeypot agent i.

10. The method for generating honeynet deployment strategies based on reinforcement learning in an intranet environment according to claim 1, characterized in that, The value network update formulas for the decision-making agent and the honeypot agent are as follows: Among them, Critic decision′ Critic represents the value network value of the decision-making agent after secondary allocation. decison This represents the value of the critic network of the policy agent after being updated by MADDPG, where n is the total number of honeypot agents, and HP i For the i-th honeypot agent, HP j A honeypot agent that interacts with the attacker.

Images