Intranet penetration environment method and device based on reinforcement learning and virtualization container

By using reinforcement learning and virtualized container technology in a virtual environment, the problem of verifying path validity in the existing technology is solved, efficiency and accuracy are improved, and attack paths are quickly acquired and verified. method.

CN118944923APending Publication Date: 2024-11-12GUANGZHOU UNIVERSITY
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410971473.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art cannot verify the effectiveness of the predicted attack path in simulated intranet penetration environments, and the verification time period in the actual environment is long, making it difficult to take into account both speed and accuracy, and has low efficiency.

Method used

Using reinforcement learning and virtualized container methods, Docker-compose files are adaptively generated in the virtual environment, and through the exploration and exploitation of attacker agents in the simulated environment, the optimal intranet horizontal mobile attack path and possible exploit vulnerabilities and verify their feasibility in a virtualized simulation network environment, and finally add viable paths and vulnerabilities to the firewall.

Benefits of technology

It greatly saves training time and cost, improves the adaptability of the intranet penetration environment and the accuracy of attack characteristics, quickly obtains attack paths, and reduces verification time and cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118944923A_ABST
    Figure CN118944923A_ABST
Patent Text Reader

Abstract

The invention discloses an intranet penetration environment method and device based on reinforcement learning and a virtualization container, and the method comprises the steps: firstly simulating the intranet environment of an enterprise on software, and setting a high-value target node according to the reality, and training an attack agent to explore and utilize vulnerabilities in the environment by using a PPO algorithm with the best reinforcement learning effect at present, so as to output a possible optimal attack path and utilization method. According to the method, the problems that an intranet attack path is complex and verification is difficult are solved, the training time and cost are greatly saved, and meanwhile, the adaptability of an intranet permeation environment and the accuracy of attack characteristics are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of intranet security, and specifically relates to an intranet environment penetration method and device based on reinforcement learning and virtualized containers. Background Art

[0002] The Microsoft Defend365 team has open-sourced a tool called CyberbattleSim, which focuses on threat modeling for the lateral movement phase after a cyberattack intrusion, and is used to study the interaction of automated agents running in a simulated abstract enterprise network environment. Its Python-based Open AI Gym interface uses reinforcement learning algorithms to train automated agents. The simulation environment parameterization under this project defines fixed network topologies and corresponding vulnerabilities, which agents can exploit to move laterally in the network. The attacker's goal is to gain ownership of the network by exploiting parameterized vulnerabilities implanted in computer nodes. The entire process is abstractly modeled as an operation across multiple simulation steps. This project is used to simulate the intranet network topology environment to find attack paths. Since it is a simulated environment and there is no real data interaction, the training speed is faster than in the traditional real topology. Although this method improves the speed of attack path prediction, it is difficult to verify whether the key attack paths found from this simulated network topology are effective.

[0003] Docker is an open platform for developing, publishing, and running applications. It provides the ability to package and run applications in a loosely isolated environment called a container environment. Containers are lightweight and contain everything needed to run an application. Their isolation and security allow many containers to run on a given host at the same time. Docker-Compose is a tool for defining and running multi-container Docker applications. Docker-Compose can be used to easily define and share multi-container applications. With Compose, users can create YAML files to define services. With just one command, you can start or delete everything, and it is convenient for multiple people to collaborate in the file to build a project environment.

[0004] Grant Ho et al. proposed a device for lateral movement detection in corporate intranets. By building a log of internal machine login activities, the suspicious login sequence corresponding to lateral movement is identified, thereby identifying the path sequence corresponding to lateral movement for tracing abnormal users. This method achieves a high lateral movement detection rate and reduces false positives by nearly eight times compared to previous devices. However, this method requires a large amount of internal login data set training, and the verification cycle is long. The verification method still uses traditional expert experience to assist.

[0005] In summary, the existing technology cannot verify the effectiveness of the predicted attack path in a simulation environment, and the time period for verifying the effectiveness of the attack path in the actual environment is long. In addition, it still uses traditional expert experience to assist, which makes it difficult to balance speed and accuracy and has low efficiency. Summary of the invention

[0006] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide an intranet penetration environment method and device based on reinforcement learning and virtualized containers. By adopting reinforcement learning in a virtual environment and adaptively generating a Docker-compose file to realize a virtualized environment, the output results of the intelligent agent in a simulated environment are verified, which greatly saves training time and cost, and at the same time improves the adaptability of the intranet penetration environment and the accuracy of the attack characteristics.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides an intranet penetration environment method based on reinforcement learning and virtualized containers, comprising the following steps:

[0009] S1. Build a simulated network topology environment, set target nodes according to actual conditions, and mark the value of target nodes; set rewards based on the value of target nodes, and set vulnerability dictionaries for target nodes; set exploration and exploitation as action spaces, and set the penetration process as observation space; set the agent in the environment as the attacker agent, and set the state to the current simulated network state;

[0010] S2. Use the attacker agent to use the ε greedy strategy to explore and exploit vulnerabilities in the environment. When the attacker agent explores a machine, it exploits the vulnerability to attack the machine. When the machine completes the specified operation, it gives a corresponding reward and outputs the current path and vulnerability type in the topology. After iteration, the optimal intranet lateral movement attack path and exploitable vulnerabilities explored by the attacker agent are output;

[0011] S3, reads the vulnerabilities set in the target node in the environment and adaptively generates a container configuration file;

[0012] S4. Use the container configuration file to form a virtualized simulated network environment, use the optimal intranet lateral movement attack path and exploitable vulnerabilities to verify the feasibility of the attacker's intelligent agent output content, and add the feasible intranet lateral movement attack path and exploitable vulnerabilities to the firewall.

[0013] As a preferred technical solution, in the simulated network topology environment, the target nodes represent computers, and the edges represent the connectivity between computers.

[0014] As a preferred technical solution, the penetration process includes discovering nodes and capturing nodes.

[0015] As a preferred technical solution, the ε-greedy strategy is used for exploration and vulnerability exploitation, specifically:

[0016] Exploration in the environment: The attacker agent adopts a certain proportion of exploration strategies to gain environmental awareness, where the proportion of exploration strategies is greater than the proportion of exploitation strategies;

[0017] Exploitation of vulnerabilities in the environment: The attacker agent adopts a certain proportion of exploitation strategies to obtain decision information, among which the proportion of exploration strategies is smaller than that of exploitation strategies.

[0018] As a preferred technical solution, the reward mechanism is specifically as follows: the fewer the attempts, the more serious the consequences of the attack, the higher the score, a certain number of points will be deducted if the exploit fails, and the more times the wrong methods and repeated methods are tried, the more points will be deducted.

[0019] As a preferred technical solution, if the output content of the attacker agent is not feasible, the attacker agent is informed in the form of manual feedback, a low score is given to the infeasible steps or paths, and the result of the manual feedback is returned to step S2;

[0020] If the output of the attacker’s agent is feasible, it is applied to the production environment to reproduce the intranet lateral movement attack path and exploitable vulnerabilities in the virtualized simulation network environment.

[0021] As a preferred technical solution, if the reproduction fails in the production environment, a low score is given to the infeasible steps or paths, and the result of manual feedback is returned to step S2.

[0022] As a preferred technical solution, the network topology environment is updated, and steps S1 to S4 are executed in sequence to add the predicted intranet lateral movement attack path and exploitable vulnerabilities to the firewall.

[0023] In a second aspect, the present invention further provides an intranet penetration environment device based on reinforcement learning and virtualized containers, which is applied to the intranet penetration environment method based on reinforcement learning and virtualized containers, including an environment construction module, a model training module, a file configuration module and a verification and update module;

[0024] The environment construction module is used to construct a simulated network topology environment, set target nodes according to actual conditions, and mark the value of target nodes; set rewards based on the value of target nodes, and set vulnerability dictionaries for target nodes; set exploration and exploitation as action spaces, and set the penetration process as observation space; set the agent in the environment as the attacker agent, and set the state to the current simulated network state;

[0025] The model training module is used to use the attacker agent to use the ε greedy strategy to explore and exploit vulnerabilities in the environment. When the attacker agent explores a machine, it exploits the vulnerability to attack the machine. When the machine completes the specified operation, it gives a corresponding reward and outputs the current path and vulnerability type in the topology. After iteration, the optimal intranet lateral movement attack path and exploitable vulnerabilities explored by the attacker agent are output;

[0026] The file configuration module is used to read the vulnerabilities set in the target node in the environment and adaptively generate container configuration files;

[0027] The verification and update module is used to use the container configuration file to form a virtualized simulation network environment, use the optimal intranet lateral movement attack path and exploitable vulnerabilities to verify the feasibility of the attacker's intelligent agent output content, and add feasible intranet lateral movement attack paths and exploitable vulnerabilities to the firewall.

[0028] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0029] at least one processor; and,

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the intranet penetration environment method based on reinforcement learning and virtualized containers.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] (1) The present invention customizes the simulation environment on CyberbattleSim according to the environment required by the user, and uses the reinforcement learning PPO algorithm to train the intelligent agent in the environment. By utilizing the attacker intelligent agent to explore and exploit vulnerabilities in the environment and setting a corresponding reward mechanism, there is no need to collect a large amount of login logs as a data set, nor is there any need to generate traffic interactions in the actual environment, which greatly saves training time and cost.

[0034] (2) The present invention adopts the ε-greedy strategy to enable the intelligent agent to quickly converge to the specified network topology in the program, and can quickly obtain the attack path and method in the simulated topology, thereby improving the model training efficiency.

[0035] (3) The present invention adopts natural language analysis technology, establishes a threat intelligence entity, and uses deep reinforcement learning technology to select and manually enhance each feature, so that the extracted features are more accurate and compatible with heterogeneous data formats. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a flow chart of a method for penetrating an intranet environment based on reinforcement learning and a virtualized container according to an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of attacker agent exploration and vulnerability exploitation according to an embodiment of the present invention;

[0039] Figure 3 It is a structural schematic diagram of an intranet penetration environment device based on reinforcement learning and virtualized container according to an embodiment of the present invention;

[0040] Figure 4 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0042] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0043] CyberbattleSim: A Python project developed by the Microsoft team that uses reinforcement learning to simulate intranet attack and defense. The network topology is simulated with the help of the Networkx third-party library, and the host nodes are simulated in the form of arrays and lists. It does not involve real network content, so the reinforcement learning training process is much faster than in the real environment.

[0044] Docker: Docker is a lightweight virtualization container project that can package programs and all dependencies of the program into the Docker environment, which can shield the environmental differences between operating devices and ensure that the program behaves consistently in any environment. It is often used to quickly deploy the required environment.

[0045] Docker-Compose: Docker-Compose calls the Docker-py library to interact with the Docker engine to build Docker images, start and stop Docker containers, and implement container orchestration.

[0046] RLHF, or Reinforcement Learning from Human Feedback, is a technical approach that combines reinforcement learning (RL) and human feedback. This approach aims to guide and optimize the behavior of reinforcement learning models by introducing human evaluation or preference information, so that they can learn and exhibit behaviors that are more in line with human expectations and ethical standards.

[0047] See also Figure 1 , a method for penetrating an intranet environment based on reinforcement learning and a virtualized container provided in this embodiment includes the following steps:

[0048] S1. Implement the required simulated network topology environment into a simulation environment in the CyberbattleSim project, set the target nodes according to the actual situation, and mark the value of the target nodes; set rewards based on the value of the target nodes, and set the vulnerability dictionary for the target nodes; set exploration and exploitation as the action space, and set the penetration process as the observation space; set the agent in the environment as the attacker agent, and set the state to the current simulated network state.

[0049] The reinforcement learning modeling is as follows: the target node in the project represents the computer, and the edge represents the connectivity between computers; exploration and utilization are set as the action space; rewards are set based on the intrinsic value of the target node, such as the reward value of the database machine is higher than that of the ordinary machine, and a series of vulnerability dictionaries simulating the real environment are set for the target node, simulating the real environment as much as possible, such as ports, open services, privilege escalation vulnerabilities, etc.; the penetration process such as discovering nodes and capturing nodes is set as the observation space; the agent in the environment is the attacker intelligent agent, and the state is the current simulated network state. For the attacker, the current network state is a static, partially observable environment.

[0050] S2. Use the attacker agent to explore and exploit vulnerabilities in the environment. When the attacker agent discovers a machine, it exploits the vulnerability to attack the machine. When the machine completes the specified operation, it gives a corresponding reward and outputs the current path and type of vulnerability in the topology. After iteration, the optimal intranet lateral movement attack path and exploitable vulnerabilities explored by the attacker agent are output.

[0051] See also Figure 2 In this embodiment, first, exploration and vulnerability exploitation actually use the reinforcement learning PPO algorithm to train the attack agent in the environment. In the current simulated topology, the attacker agent has an initial node, which only simulates the environment in the intranet penetration. It adopts two methods of exploration and utilization for the environment, mainly using the ε greedy strategy. At the beginning of training, the agent adopts more exploration strategies and has more perception of the environment. In the later stage of training, more utilization strategies are adopted. Such a setting is more in line with the process of penetration testing, and the agent can also make better decisions with more information in the later stage of training. When the agent explores the machine, exploits the vulnerability to capture the machine, and performs operations such as authority escalation on the machine, it will give corresponding rewards and output the current path in the topology and the method of exploiting the vulnerability. The fewer the number of attempts, the more serious the consequences of the attack, and the higher the score. If the exploitation fails, a certain score will be deducted. The more times the wrong method and repeated method are tried, the more points will be deducted. In this way, the agent reaches a convergence value in the program on how to explore and utilize the specified network topology, which means that the attack agent knows how to attack in the simulated topology, and the output path and method are the attack path and method.

[0052] It should be noted here that the "more" mentioned here means that the proportion of adopting this strategy is high. In actual operation, the exploration strategy or exploitation strategy adopted in training is parallel, but the focus is on the exploration strategy at the beginning of training, so that it can have more perception of the environment, and in the later stage of training, it is more focused on the exploitation strategy. This setting is more in line with the process of penetration testing, and the later stage of training can also allow the agent to make better decisions with more information.

[0053] Then, when the agent explores the machine, exploits the vulnerability to attack the machine, and elevates permissions on the machine, it will be rewarded accordingly and output the current path in the topology and the method of exploiting the vulnerability. The fewer attempts, the more serious the consequences of the attack and the higher the score. If the exploit fails, a certain number of points will be deducted. The more times the wrong method is tried and the more repeated methods are tried, the more points will be deducted. The result of a round of iteration is the process in which the agent takes actions to explore in the environment and exploits the vulnerability to move laterally to the target node. After multiple iterations and optimizations, the attacker agent reaches a convergence value in how to explore and exploit the specified network topology in the program, which means that the attacker agent knows how to attack in the simulated topology, and the output path and method are the optimal intranet lateral movement attack path and exploitable vulnerabilities.

[0054] S3, reads the vulnerabilities set in the target node in the environment and adaptively generates a container configuration file.

[0055] According to the environment simulated by CyberbattleSim, by reading the vulnerabilities of the target nodes set in the simulation environment, the Docker-Compose file is adaptively generated to quickly form a virtualized simulation network environment.

[0056] This embodiment uses the PPO algorithm for training. After the experimental results converge, the program outputs the attack path and the method of use. Then the verification part of the experimental results is migrated to the virtualized container environment to verify whether the lateral movement path and the method of use predicted in the simulation environment are effective. If it cannot be verified, it is retrained through manual feedback until the verification is successful. The whole process is much faster than the traditional method because there is no actual traffic interaction in the number of iterations in the simulation environment, and the key path and the method of use are indicated for professionals to use for verification, which greatly reduces the verification time. See step S4 for details.

[0057] S4. Use the container configuration file to form a virtualized simulated network environment, use the optimal intranet lateral movement attack path and exploitable vulnerabilities to verify the feasibility of the attacker's intelligent agent output content, and add the feasible intranet lateral movement attack path and exploitable vulnerabilities to the firewall.

[0058] In particular, if the output of the attacker agent is not feasible, the attacker agent is informed in the form of manual feedback, a low score is given to the infeasible step or path, and the result of the manual feedback is returned to step S2;

[0059] If the attacker’s agent output is feasible, it is applied to the production environment to reproduce the intranet lateral movement attack path and exploitable vulnerabilities in the virtualized simulation network environment;

[0060] If the reproduction fails in the production environment, a low score is given to the infeasible step or path, and the result of the manual feedback is returned to step S2.

[0061] In some embodiments, step S5 is also included for the subsequent device use stage. For the continuously updated network topology, after adding the updated topology to the simulation environment, steps S1-S4 are executed in sequence, so that the intelligent agent can update the results after training to output the lateral movement path and attack method, and add it to the security device to ensure that the program operation does not occupy a large amount of resources and can quickly respond to threats.

[0062] It should be explained that the above-mentioned security devices include Web Application Firewalls. Web Application Firewall (WAF) is a product that specifically provides protection for Web applications by executing a series of security policies for HTTP / HTTPS. It is used to identify and protect against Web attacks, such as SQL injection and cross-site scripting (XSS). WAF products are usually deployed at the front end of Web applications and have the ability to intercept illegal access and malicious attacks without making any modifications to the Web application itself.

[0063] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0064] Based on the same idea as the intranet penetration environment method based on reinforcement learning and virtualized container in the above-mentioned embodiment, the present invention also provides an intranet penetration environment device based on reinforcement learning and virtualized container, which can be used to execute the above-mentioned intranet penetration environment method based on reinforcement learning and virtualized container. For ease of explanation, in the structural schematic diagram of the embodiment of the intranet penetration environment device based on reinforcement learning and virtualized container, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0065] See also Figure 3 In another embodiment of the present application, an intranet penetration environment device 10 based on reinforcement learning and virtualized container is provided, the device comprising an environment construction module 11, a model training module 12, a file configuration module 13 and a verification update module 14;

[0066] The environment construction module 11 is used to construct a simulated network topology environment, set target nodes according to actual conditions, and mark the value of the target nodes; set rewards based on the value of the target nodes, and set vulnerability dictionaries for the target nodes; set exploration and utilization as action spaces, and set the penetration process as observation space; set the agent in the environment as an attacker agent, and set the state to the current simulated network state;

[0067] The model training module 12 is used to use the attacker agent to use the ε greedy strategy to explore and exploit vulnerabilities in the environment. When the attacker agent explores a machine, it exploits the vulnerability to attack the machine. When the machine completes the specified operation, it gives a corresponding reward and outputs the current path and vulnerability type in the topology. After iteration, the optimal intranet lateral movement attack path and exploitable vulnerability explored by the attacker agent are output;

[0068] The file configuration module 13 is used to read the vulnerabilities set in the target node in the environment and adaptively generate a container configuration file;

[0069] The verification and update module 14 is used to use the container configuration file to form a virtualized simulation network environment, use the optimal intranet lateral movement attack path and exploitable vulnerabilities to verify the feasibility of the attacker's intelligent agent output content, and add the feasible intranet lateral movement attack path and exploitable vulnerabilities to the firewall.

[0070] It should be noted that the intranet penetration environment device based on reinforcement learning and virtualized containers of the present invention corresponds one to one with the intranet penetration environment method based on reinforcement learning and virtualized containers of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the intranet penetration environment method based on reinforcement learning and virtualized containers are applicable to the embodiment of the intranet penetration environment method based on reinforcement learning and virtualized containers. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0071] In addition, in the implementation of the intranet penetration environment device based on reinforcement learning and virtualized containers in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the intranet penetration environment device based on reinforcement learning and virtualized containers is divided into different program modules to complete all or part of the functions described above.

[0072] See also Figure 4In one embodiment, an electronic device for implementing an intranet penetration environment method based on reinforcement learning and virtualized containers is provided. The electronic device 20 may include a first processor 21, a first memory 22 and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as an intranet penetration environment program 23 based on reinforcement learning and virtualized containers.

[0073] Among them, the first memory 22 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as a mobile hard disk of the electronic device 20. In other embodiments, the first memory 22 can also be an external storage device of the electronic device 20, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 20. Further, the first memory 22 can also include both an internal storage unit of the electronic device 20 and an external storage device. The first memory 22 can not only be used to store application software and various types of data installed in the electronic device 20, such as the code of the intranet penetration environment program 23 based on reinforcement learning and virtualized containers, but also can be used to temporarily store data that has been output or is to be output.

[0074] In some embodiments, the first processor 21 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The first processor 21 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 20 by running or executing programs or modules stored in the first memory 22, and calling data stored in the first memory 22.

[0075] Figure 4 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device 20 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0076] The intranet penetration environment program 23 based on reinforcement learning and virtualized container stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions. When running in the first processor 21, it can achieve:

[0077] S1. Build a simulated network topology environment, set target nodes according to actual conditions, and mark the value of target nodes; set rewards based on the value of target nodes, and set vulnerability dictionaries for target nodes; set exploration and exploitation as action spaces, and set the penetration process as observation space; set the agent in the environment as the attacker agent, and set the state to the current simulated network state;

[0078] S2. Use the attacker agent to explore and exploit vulnerabilities in the environment using the ε greedy strategy. When the attacker agent explores a machine, it exploits the vulnerability to attack the machine. When the machine completes the specified operation, it gives a corresponding reward and outputs the current path and vulnerability type in the topology. After iteration, the optimal intranet lateral movement attack path and exploitable vulnerabilities explored by the attacker agent are output;

[0079] S3, reads the vulnerabilities set in the target node in the environment and adaptively generates a container configuration file;

[0080] S4. Use the container configuration file to form a virtualized simulated network environment, use the optimal intranet lateral movement attack path and exploitable vulnerabilities to verify the feasibility of the attacker's intelligent agent output content, and add the feasible intranet lateral movement attack path and exploitable vulnerabilities to the firewall.

[0081] Furthermore, if the module / unit integrated in the electronic device 20 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0082] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0083] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An intranet penetration environment method based on reinforcement learning and virtualized containers, characterized in that: The steps include: S1. Build a simulated network topology environment, set target nodes according to actual conditions, and mark the value of target nodes; set rewards based on the value of target nodes, and set vulnerability dictionaries for target nodes; set exploration and exploitation as action spaces, and set the penetration process as observation space; set the agent in the environment as the attacker agent, and set the state to the current simulated network state; S2, using the attacker agent to adopt the ε greedy strategy to explore and exploit vulnerabilities in the environment. When the attacker agent explores a machine, it exploits the vulnerability to attack the machine. When the machine completes the specified operation, it gives a corresponding reward and outputs the current path in the topology and the type of vulnerability; After iteration, the optimal intranet lateral movement attack path and exploitable vulnerabilities explored by the attacker agent are output; S3, reads the vulnerabilities set in the target node in the environment and adaptively generates a container configuration file; S4. Use the container configuration file to form a virtualized simulated network environment, use the optimal intranet lateral movement attack path and exploitable vulnerabilities to verify the feasibility of the attacker's intelligent agent output content, and add the feasible intranet lateral movement attack path and exploitable vulnerabilities to the firewall.

2. According to claim 1, the method for intranet penetration environment based on reinforcement learning and virtualized container is characterized in that: In the simulated network topology environment, the target nodes represent computers, and the edges represent the connectivity between computers.

3. According to claim 1, the method for intranet penetration environment based on reinforcement learning and virtualized container is characterized in that: The infiltration process includes discovering nodes and capturing nodes.

4. According to claim 1, the method for intranet penetration environment based on reinforcement learning and virtualized container is characterized in that: The ε greedy strategy is used for exploration and vulnerability exploitation, specifically: Exploration in the environment: The attacker agent adopts a certain proportion of exploration strategies to gain environmental awareness, where the proportion of exploration strategies is greater than the proportion of exploitation strategies; Exploitation of vulnerabilities in the environment: The attacker agent adopts a certain proportion of exploitation strategies to obtain decision information, among which the proportion of exploration strategies is smaller than that of exploitation strategies.

5. According to claim 1, the method for intranet penetration environment based on reinforcement learning and virtualized container is characterized in that: The reward mechanism is specifically as follows: the fewer the attempts, the more serious the consequences of the attack, the higher the score; a certain number of points will be deducted if the exploit fails; the more wrong methods and repeated methods are tried, the more points will be deducted.

6. The intranet penetration environment method based on reinforcement learning and virtualized container according to claim 1 is characterized in that: If the output of the attacker agent is not feasible, the attacker agent is informed by manual feedback, a low score is given to the infeasible steps or paths, and the result of the manual feedback is returned to step S2; If the output of the attacker’s agent is feasible, it is applied to the production environment to reproduce the intranet lateral movement attack path and exploitable vulnerabilities in the virtualized simulation network environment.

7. The method for intranet penetration environment based on reinforcement learning and virtualized container according to claim 6 is characterized in that: If the reproduction fails in the production environment, a low score is given to the infeasible step or path, and the result of the manual feedback is returned to step S2.

8. The method for intranet penetration environment based on reinforcement learning and virtualized container according to claim 1 is characterized in that: The step S5 is further included, which is specifically: Update the network topology environment, execute steps S1 to S4 in sequence, and add the predicted intranet lateral movement attack path and exploitable vulnerabilities to the firewall.

9. An intranet penetration environment device based on reinforcement learning and virtualized container, characterized in that: An intranet penetration environment method based on reinforcement learning and virtualized containers applied to any one of claims 1-8, comprising an environment construction module, a model training module, a file configuration module and a verification and update module; The environment construction module is used to construct a simulated network topology environment, set target nodes according to actual conditions, and mark the value of target nodes; set rewards based on the value of target nodes, and set vulnerability dictionaries for target nodes; set exploration and exploitation as action spaces, and set the penetration process as observation space; set the agent in the environment as the attacker agent, and set the state to the current simulated network state; The model training module is used to use the attacker agent to use the ε greedy strategy to explore and exploit vulnerabilities in the environment. When the attacker agent explores a machine, it exploits the vulnerability to attack the machine. When the machine completes the specified operation, it gives a corresponding reward and outputs the current path in the topology and the type of vulnerability. After iteration, the optimal intranet lateral movement attack path and exploitable vulnerabilities explored by the attacker agent are output; The file configuration module is used to read the vulnerabilities set in the target node in the environment and adaptively generate container configuration files; The verification and update module is used to use the container configuration file to form a virtualized simulation network environment, use the optimal intranet lateral movement attack path and exploitable vulnerabilities to verify the feasibility of the attacker's intelligent agent output content, and add feasible intranet lateral movement attack paths and exploitable vulnerabilities to the firewall.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the intranet penetration environment method based on reinforcement learning and virtualized containers as described in any one of claims 1-8.

Citation Information

Cited By

  • Intelligent network vulnerability repairing method and system based on penetration testing technology

    CN120090872A

  • Network penetration data generation method and device, electronic equipment and program product

    CN121037109A