Automatic penetration method and system based on generative adversarial imitation learning in cloud environment

Abstract

The invention discloses an automatic penetration method and system based on generative adversarial imitation learning in a cloud environment, and the method comprises the steps: storing an execution action-state pair when a reinforcement learning model successfully penetrates a target drone through an Exploit open source tool and a Metasploit open source penetration tool, so as to construct a penetration expert knowledge base; inputting the content in the expert knowledge base into a discriminator in the imitation learning framework for training; training actions output by the rear discriminator by using an imitation learning framework, wherein the actions can be used for automatically infiltrating a specified host; and deploying the whole automatic penetration framework to a cloud environment through containerization and arrangement to simulate external network attacks. According to the method, imitation learning is carried out in the cloud environment, resources can be efficiently utilized, container resource allocation can be automatically adjusted by enhancing the GAIL network and penetration test tool containerization, and therefore the overall performance is improved, and resource use and response time is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security attack and defense, and specifically relates to an automated penetration method and system based on generative adversarial imitation learning in a cloud environment. Background Art

[0002] In today's digital age, the global cybersecurity situation is facing unprecedented complexity and challenges. With the rapid development of Internet technology, cyberspace has become an important part of the national economy, social operations and citizens' lives. However, the resulting cybersecurity risks are also rising sharply, posing a serious threat to personal privacy, corporate operations and even national security.

[0003] Nowadays, when dealing with network attacks, enterprises mainly focus on using firewalls, WAFs, and other application scenarios during and after attacks. For prediction and prevention before attacks, enterprises generally use manual penetration testing and vulnerability scanning tools to detect and evaluate network vulnerabilities. Penetration testing is an evaluation method that evaluates the security of computer network systems by simulating the attack methods of malicious hackers. The cost from the discovery of security vulnerabilities to their repair can reach millions of dollars. Penetration testing can effectively identify and repair these security vulnerabilities before they are exploited, thus avoiding attacks by hackers that exploit security vulnerabilities.

[0004] However, as business becomes more globalized and complex, the limitations of manual penetration testing are becoming more and more obvious. For example, it is difficult to implement and has a great impact on business systems. Sometimes, system testing needs to be stopped, and the testing cost is too high. It lacks integration with corporate business and is difficult to adapt to business cloudification and virtualization. It is difficult to predict key asset risks in real time. Manual penetration testing that is implemented once a year and requires downtime provides insufficient and outdated test reports. Enterprises are increasingly in urgent need of continuous, automated, and intelligent penetration testing.

[0005] At present, there are several solutions for automated penetration: First, the rule-matching-based method, which is a traditional automated penetration testing technology that relies on predefined rules and signatures to detect and exploit vulnerabilities. However, it is usually dependent on known vulnerability signatures and attack patterns and the update of the rule library, which is limited by the coverage of known rules and lacks adaptability and flexibility. Second, the machine learning-based method, which has been a research hotspot in recent years, uses intelligent algorithms to automatically identify and exploit vulnerabilities, but is limited by data quality and model interpretability. Third, the automated penetration method based on reinforcement learning faces the challenges of the trade-off between exploration and exploitation, training time, and environmental stability.

[0006] Therefore, the high cost, time-consuming, lack of automation and intelligence of manual penetration testing, as well as the shortcomings of existing automated penetration technology in new vulnerability identification, environmental adaptability and policy generalization capabilities are currently in urgent need of resolution. Summary of the invention

[0007] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide an automated penetration method and system based on generative adversarial imitation learning in a cloud environment. Imitation learning is performed in a cloud environment, and resources can be efficiently utilized. At the same time, by enhancing the containerization of the GAIL network and penetration testing tools, container resource allocation can be automatically adjusted, thereby improving overall performance and optimizing resource utilization and response time.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides an automated penetration method based on generative adversarial imitation learning in a cloud environment, comprising the following steps:

[0010] Collect expert knowledge data, build an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration testing;

[0011] Set up asynchronous multi-environment to allow pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments;

[0012] The current state observed by the agent during the training process is used to generate predicted state-action pairs online, and the predicted state-action pairs are used to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through an adaptive learning rate and a multi-layer perception mechanism to obtain high-quality state-action pairs; the enhanced GAIL network includes a policy network, a value network, and a discriminator;

[0013] Calculate the discounted rewards for high-quality state-action pairs, and use the discounted rewards as feedback to optimize the policy network; use security events to evaluate the policy network and provide real-time security feedback to the policy network; complete the training of the policy network after multiple rounds of training;

[0014] The penetration testing tools are containerized and orchestrated using k8s. High-quality state actions are used to perform penetration tests on the trained policy network. Experiments are repeated on specific target machines, and the policy network is adjusted based on the experimental results. The performance of the intelligent agent is continuously evaluated, and the parameters of the enhanced GAIL network are recorded and feedback is provided to generate monitoring logs.

[0015] As a preferred technical solution, the method of enhancing expert knowledge data by using a natural language processing algorithm includes:

[0016] Automatically collect expert reports, use natural language processing algorithms to analyze the content of expert reports, label the expert knowledge data, and add the labeled expert knowledge data to the expert knowledge base.

[0017] As a preferred technical solution, the asynchronous multi-environment is set up to allow the pre-trained agent to perform parallel generative imitation learning in multiple sandbox environments, including:

[0018] S21. Use large batches to accumulate gradients for the agent in a multi-sandbox environment, and update gradients using multiple small batches of expert knowledge data.

[0019] S22, using the parameter server architecture to coordinate multiple computing nodes and update computing node parameters in parallel;

[0020] S23, using synchronous SGD to obtain the distributed average gradient of each computing node, and broadcasting the distributed average gradient to each computing node;

[0021] S24. Perform mixed precision training on the agent and scale the loss and gradients of the computing nodes. The mixed precision training includes low-precision floating-point operations and high-precision operations.

[0022] As a preferred technical solution, the gradient update is as follows:

[0023]

[0024] Where N is the number of accumulated batches, is the loss function, represents the gradient with respect to the parameter θ.

[0025] As a preferred technical solution, the parallel update calculation node parameter θ t , as follows:

[0026]

[0027] Where K is the number of parallel computing nodes, g k is the gradient calculated at node k, and η is the learning rate.

[0028] As a preferred technical solution, step S24 is specifically as follows:

[0029] Perform low-precision floating-point operations on the agent. When back-propagating the gradient, a scaling factor is used to amplify the loss and increase the gradient, as shown in the following formula:

[0030] scaled_loss = loss × scale

[0031] Among them, scaled_loss represents the loss after amplification, loss represents the loss, and scale represents the scaling factor;

[0032] The gradient is calculated during back propagation as follows:

[0033]

[0034] Among them, scaled_gradients represents the gradient;

[0035] Scaling back to the original gradient is as follows:

[0036]

[0037] Among them, gradient represents the scaled gradient;

[0038] Use the scale factor to update the parameters as follows:

[0039]

[0040] Among them, η is the scale factor.

[0041] In a second aspect, the present invention provides an automated penetration system based on generative adversarial imitation learning in a cloud environment, which is applied to the automated penetration method based on generative adversarial imitation learning in the cloud environment, including: a knowledge base construction and management module, an imitation learning training module, an adversarial training module, an evaluation and adjustment module, and a penetration testing module;

[0042] A knowledge base construction and management module is used to collect expert knowledge data, construct an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration testing;

[0043] The imitation learning training module is used to set up asynchronous multi-environment and enable pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments;

[0044] The adversarial training module is used to generate predicted state-action pairs online from the state currently observed by the agent during the training process, and use the predicted state-action pairs to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through an adaptive learning rate and a multi-layer perception mechanism to obtain high-quality state-action pairs; the enhanced GAIL network includes a policy network, a value network and a discriminator;

[0045] The evaluation and adjustment module is used to calculate the discounted rewards for high-quality state-action pairs and use the discounted rewards as feedback to optimize the policy network. The policy network is evaluated using security events and given real-time security feedback. After multiple rounds of training, the policy network training is completed.

[0046] The penetration testing module is used to containerize the penetration testing tools and orchestrate them using k8s. It uses high-quality state actions to perform penetration tests on the trained policy network, repeats experiments on specific target machines, and adjusts the policy network based on the experimental results. It continuously evaluates the performance of the intelligent agent, records and provides feedback to adjust the parameters of the enhanced GAIL network, and generates monitoring logs.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] (1) The present invention enables pre-trained agents to perform parallel generative imitation learning in a multi-sandbox environment. On the one hand, it can improve the training efficiency and model generalization ability. On the other hand, through mixed precision training, it can reduce the computing cost, ensure the quality of model training, and improve the model training speed, thereby efficiently utilizing resources.

[0049] (2) The present invention introduces adaptive learning rate and multi-layer perception mechanism by enhancing the GAIL network, which can optimize the accuracy of strategy generation and value assessment, and dynamically adjust the state representation to improve the generation quality of state-action pairs. At the same time, through the containerization of the penetration testing tool, the container resource allocation can be automatically adjusted, thereby improving the overall performance and optimizing resource utilization and response time.

[0050] (3) The present invention optimizes the automated penetration framework and deploys it in a cloud environment, which can quickly adapt to dynamic changes in the cloud environment, such as elastic scaling of resources and instant updates of configurations. This is more flexible and responsive than locally deployed penetration testing tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a flow chart of an automated penetration method based on generative adversarial imitation learning in a cloud environment according to an embodiment of the present invention;

[0053] Figure 2 This is a structural diagram of an automated penetration system based on generative adversarial imitation learning in a cloud environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0055] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0056] See also Figure 1 , this embodiment provides an automated penetration method based on generative adversarial imitation learning in a cloud environment, comprising the following steps:

[0057] S1. Collect expert knowledge data, build an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration tests.

[0058] In this embodiment, the penetration expert knowledge base involves combining a penetration testing automation tool (such as DeepExploit) with a storage mechanism for state-action pairs, thereby building a dynamic expert knowledge base based on actual penetration testing results.

[0059] First, define the states and actions.

[0060] 1. Define the state:

[0061] (1)Host information.

[0062] Host operating system: such as Linux, Windows, macOS, etc.

[0063] Operating system patch status: includes installed patches and uninstalled patches.

[0064] Firewall Status: The host's firewall configuration and its rules.

[0065] Anti-virus software: whether anti-virus software is installed and its version.

[0066] (2)Service information.

[0067] Security configuration of the service: for example, whether secure communication (HTTPS) is enabled, whether authentication is enabled, etc.

[0068] Service port opening status: Which ports are opened, especially sensitive ports: 21 (FTP), 22 (SSH), 23 (Telnet), 25 (SMTP), 53 (DNS), 80 (HTTP), 110 (POP3), 123 (NTP), 137-139 (NetBIOS), 443 (HTTPS), 445 (SMB), 3389 (RDP), 1433 (MSSQL), 3306 (MySQL), 8080 (custom HTTP service).

[0069] The status of the service when it is running: whether the service is running, the running time, etc.

[0070] (3) Application information.

[0071] Installed Software: The software installed on the host and its versions.

[0072] Web application information: Web applications running on the host, including web server type, web framework, API endpoints, etc.

[0073] Configuration file information: The content of configuration files of applications and services and their sensitive information (such as default passwords).

[0074] (4) User and authentication information.

[0075] User Accounts: Account list, including account type (administrator, ordinary user, etc.) and account status (enabled or disabled).

[0076] Authentication method: The authentication method used by the host and service, including password, two-factor authentication, Kerberos, etc.

[0077] Access Log: Recent login records and attempts, including successful and failed login attempts.

[0078] (5) Network information.

[0079] Subnet and network segment: subnet and network segment information where the host is located.

[0080] Routing information: Router configuration information and connections to other subnets.

[0081] Network protocol: The network protocol used by the host (such as IPv4, IPv6, etc.).

[0082] (6) Environmental information.

[0083] Virtualization status: Whether the host is running in a virtualization environment, such as VMware, VirtualBox, KVM, etc.

[0084] Cloud environment information: If the host runs on the cloud, provide information about the cloud provider and related configurations (such as AWS, Azure, Google Cloud, etc.).

[0085] These states will be mapped one-to-one into numbers and stored, such as Windows 7 system mapping is 1, Windows 8 system mapping is 2, Windows 10 system mapping is 3, and so on. If a system mapping that does not exist in the database is encountered, a new mapping will be automatically added and recorded.

[0086] The same goes for other states such as port numbers. Manually specify an initial number and then increment it according to certain rules.

[0087] 2. Define actions:

[0088] Including but not limited to the following information:

[0089] (1) Specific attack payload;

[0090] (2) Metasploit modules used;

[0091] (3) Attack target port;

[0092] (4) Attack parameters.

[0093] Assume that the IP address of the target host is 192.168.1.100, which is a server running Windows Server 2016. It has some common services and ports open and has known vulnerabilities.

[0094] 3. Current status:

[0095] (1) Host information:

[0096] Host IP: 192.168.1.100

[0097] Operating system: Windows Server 2016

[0098] Patch status: Some patches have been installed, KB5001402 has not been installed

[0099] Firewall Status: Enabled

[0100] Antivirus software: Windows Defender

[0101] (2) Service information:

[0102] Open Ports:

[0103] 80 (HTTP) - Service name: IIS, version: 10.0

[0104] 445(SMB)-Service name: File Sharing 3389(RDP)-Service name: Remote Desktop

[0105] Service security configuration: HTTP does not enable HTTPS, and the SMB service has a weak password

[0106] Service dependencies: None

[0107] Service running status: All running

[0108] (3) Application information:

[0109] Installed software: IIS, SQL Server 2016

[0110] Web application information: Run an application called "WebApp"

[0111] Configuration file information: No particularly sensitive information is exposed

[0112] (4) User and authentication information:

[0113] User accounts: Administrator account Admin01 is enabled, password policy is weak

[0114] Authentication method: Password authentication only

[0115] Access log: No unusual login attempts

[0116] (5) Network information:

[0117] Subnet information: 192.168.1.0 / 24

[0118] Routing information: default gateway 192.168.1.1

[0119] Network protocols: IPv4, IPv6

[0120] (6) Environmental information:

[0121] Virtualization status: Running on Hyper-V

[0122] Cloud environment information: None

[0123] Vulnerability Information: Known SMB v1 vulnerability (MS17-010, EternalBlue) is not patched

[0124] 4. Execution action information:

[0125] Assume that the penetration tester decides to exploit the known EternalBlue vulnerability (MS17-010) and uses the exploit / windows / smb / ms17_010_eternalblue module in Metasploit to attempt to execute remote code on the target host and obtain a Meterpreter session.

[0126] (1)Payload: windows / x64 / meterpreter / reverse_tcp

[0127] (2) Metasploit module: exploit / windows / smb / ms17_010_eternalblue

[0128] (3) Destination port: 445

[0129] (4) Attack parameters:

[0130] RHOST: 192.168.1.100

[0131] RPORT::445

[0132] LHOST: The attacker's IP address, for example 192.168.1.10

[0133] LPORT: The attacker's listening port, for example: 4444

[0134] The action payload is number 137 in the specified order, so the action sequence is recorded as 137 and stored in the database. So the state-action pair created in this example is:

[0135] State: [198,34,105,76,142,23,210,67,89150,3,201,56,123,90,4,254,18,72,113…]

[0136] Action:

[137]

[0137] Next, to increase the richness of state-action pairs, this embodiment improves the expert knowledge base.

[0138] S11. Introduce automated data collection and labeling technology and use natural language processing (NLP) algorithms to analyze expert reports;

[0139] S12. Use open source security intelligence to regularly update the expert knowledge base to ensure the timeliness of training data and coverage of new threats.

[0140] Before automated penetration testing, the agents are pre-trained so that they can perform basic penetration tasks in a simulated environment. The agents can adopt reinforcement learning (RL) or deep reinforcement learning (DRL) models.

[0141] S2. Set up asynchronous multi-environment to enable pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments.

[0142] In this embodiment, an asynchronous multi-environment training framework is designed to allow the RL / DRL model to learn in parallel in multiple sandbox environments to improve training efficiency and model generalization ability.

[0143] Specifically, in a cloud environment, targeted optimization of generative imitation learning can be performed, mainly with the goal of efficient resource utilization, including:

[0144] S21. Use large batches to accumulate gradients for the agent in a multi-sandbox environment, and update gradients using multiple small batches of expert knowledge data.

[0145] In multi-sandbox environments, use larger batch sizes for gradient accumulation to reduce communication latency. By combining data from multiple small batches to compute a large update, GPUs / TPUs can be used efficiently.

[0146] The optimized gradient calculation formula is:

[0147] Where N is the number of accumulated batches, is the loss function, represents the gradient with respect to the parameter θ.

[0148] S22, using the parameter server architecture to coordinate multiple computing nodes and update computing node parameters in parallel;

[0149] Using the parameter server architecture, the computing nodes can update parameters in parallel. The optimized model update formula is coordinated by multiple nodes through the parameter server. The parallel update of the computing node parameters θ t , as follows:

[0150]

[0151] Where K is the number of parallel computing nodes, g k is the gradient calculated at node k, and η is the learning rate.

[0152] S23, using synchronous SGD to obtain the distributed average gradient of each computing node, and broadcasting the distributed average gradient to each computing node;

[0153] The synchronous SGD algorithm combines the stochastic gradient descent (SGD) algorithm with the synchronous bulk processing (BSP) framework. In a distributed system, multiple computing nodes (worker nodes) each calculate gradients based on a local subset of data, and then aggregate these gradients to the master node (or parameter server). The master node averages these gradients to form a new global gradient for updating the model parameters. This process is synchronized after each mini-batch data update to ensure that all computing nodes are trained with the latest model parameters.

[0154] In the meantime, the use of distributed averaging can ensure that the parameters of all nodes are updated in step with each other, and the gradient is as follows:

[0155]

[0156] S24. Perform mixed precision training on the agent and scale the loss and gradients of the computing nodes. The mixed precision training includes low-precision floating-point operations and high-precision operations.

[0157] Perform low-precision floating-point operations on the agent. When back-propagating the gradient, a scaling factor is used to amplify the loss and increase the gradient, as shown in the following formula:

[0158] scaled_loss = loss × scale

[0159] Among them, scaled_loss represents the loss after amplification, loss represents the loss, and scale represents the scaling factor;

[0160] The gradient is calculated during back propagation as follows:

[0161]

[0162] Among them, scaled_gradients represents the gradient;

[0163] Scaling back to the original gradient is as follows:

[0164]

[0165] Among them, gradient represents the scaled gradient;

[0166] Use the scale factor to update the parameters as follows:

[0167]

[0168] Where η is the scale factor, which can be calculated using standard optimization algorithms (such as SGD, Adam). The scale value is not fixed, and the standard selection method is usually to use a larger scale at the beginning (for example, 2 8 ~2 15 ), and then dynamically adjust during training to avoid gradient overflow (for example, reduce the scale if overflow is detected). Since mixed precision training, especially when applying loss scaling, involves amplifying the loss value to avoid numerical precision issues, it is necessary to adjust the loss value so that the gradient of back propagation in FP16 does not lose information because it is too small. After the adjustment is completed, the original loss is restored. This ensures the stability of FP16 in back propagation and the final parameter update process is indeed correctly updated.

[0169] In mixed precision training, while reducing computational costs, the training quality of the model will not be significantly affected. By leveraging the optimized performance of FP16 on modern GPUs, mixed precision training can achieve significant acceleration.

[0170] S3. Generate predicted state-action pairs online from the state currently observed by the agent during the training process, and use the predicted state-action pairs to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through adaptive learning rate and multi-layer perception mechanism to obtain high-quality state-action pairs; the enhanced GAIL network includes a policy network, a value network and a discriminator.

[0171] This example uses the Generative Adversarial Imitation Learning (GAIL) algorithm, which is a method that combines Generative Adversarial Networks (GANs) and imitation learning. The core idea of ​​GAIL is to optimize the strategy of the agent through adversarial means in the process of imitation learning, so that its behavior in a given state is as close as possible to that of the expert.

[0172] Generative adversarial imitation learning consists of two main components:

[0173] Generator: The generator is our policy that tries to generate a sequence of actions such that they look like they came from an expert.

[0174] Discriminator: The discriminator attempts to distinguish between real expert data and data generated by the generator.

[0175] These two components form an adversarial network. The more realistic the data generated by the generator, the more difficult it is for the discriminator to distinguish between expert data and generator data.

[0176] Then, the training process of GAIL can be expressed as the following optimization problem:

[0177]

[0178] Where: π expert represents the expert’s strategy, π θ is a parameterized generation strategy, D w is a discriminator with parameter w.

[0179] The specific steps are as follows:

[0180] Initialization: Initialize the generator (strategy) Π θ and the discriminator D w Parameters.

[0181] Discriminator training: Discriminator D w It tries to maximize the following objective, which distinguishes between expert and generative strategies:

[0182]

[0183] Generator Training: Generator Π θ We try to minimize the following objective, that is, the generated data becomes more and more like the real expert data:

[0184]

[0185] Iteration: Repeat the above steps, alternately optimizing the generator and discriminator until convergence.

[0186] In order to optimize the accuracy of strategy generation and value assessment, this embodiment introduces adaptive learning rate and multi-layer perception mechanism in the GAIL network, and develops an intelligent state perception module based on sensor data fusion, which can dynamically adjust the state representation and improve the generation quality of state-action pairs.

[0187] After completing the deployment of the enhanced GAIL network, the discriminator training, adversarial training, and policy network optimization are performed respectively.

[0188] (1) Discriminator training: During the training process, the agent inputs the first state-action pair generated by the policy network (Actor) and the second state-action pair generated by the value network (Critic) into the discriminator. The discriminator dynamically adjusts the state through adaptive learning rate and multi-layer perception mechanism to obtain high-quality state-action pairs. In this process, the goal of the discriminator is to distinguish between expert samples and samples generated by the agent.

[0189] Among them, the "adaptive learning rate" is mainly reflected in the training process of the discriminator in the enhanced generative adversarial imitation learning (GAIL) network. The learning rate of the discriminator is not fixed, but dynamically adjusted according to the progress of training and the feedback of the model. The adaptive learning rate algorithm can adjust the learning rate according to the changes in gradients and its own performance indicators (such as changes in loss) to adapt to the training needs of different stages. The "adaptive learning rate" can also be used to improve training efficiency and help the discriminator converge to a stable state faster, because it can increase the learning rate when rapid learning is required and reduce the learning rate when fine-tuning is required. Specifically in automated penetration, the role of the "adaptive learning rate" is generally to ensure that the discriminator can effectively learn and adjust in a constantly changing training environment, thereby obtaining high-quality state-action pairs, thereby optimizing the performance of the entire GAIL network.

[0190] The multi-layer perception mechanism can be used to build intelligent state-aware systems because it provides a more detailed and layered analysis perspective, helping the system better understand and process complex input information.

[0191] (2) Adversarial training: Through an adversarial process, the actions generated by the policy network (Actor) attempt to deceive the discriminator network into thinking that these actions are expert-level. The discriminator network then attempts to correctly classify these actions.

[0192] (3) Optimization of the policy network. The discounted reward output by the discriminator network is used as feedback to guide the training of the policy network. This process optimizes the agent’s strategy by imitating the behavior of the expert. See step S4 for details.

[0193] S4. Calculate the discounted rewards for high-quality state-action pairs, and use the discounted rewards as feedback to optimize the policy network. Use security events to evaluate the policy network and provide real-time security feedback to the policy network. After multiple rounds of training, complete the training of the policy network.

[0194] In the process of optimizing the policy network, in addition to the feedback of the training itself, in order to improve the dynamic performance of the policy, this embodiment also adds a real-time security feedback mechanism to the imitation learning, and dynamically adjusts and optimizes the policy in combination with the post-evaluation of security events. Then, the test results are fed back to the training process in a loop, and the expert knowledge base and GAIL network are continuously optimized iteratively to improve the performance and accuracy of the automated penetration test.

[0195] In addition, this embodiment can also combine self-adversarial training strategies to allow agents to challenge each other, further improving the intelligence and adaptability of automated penetration.

[0196] In the automated penetration method, the self-adversarial training strategy introduces multiple agents to compete with each other in the same environment, prompting them to optimize their strategies in simulated adversarial scenarios. Each agent can play a different role, for example, one agent performs attack operations to simulate penetration testing, while another agent is responsible for defense or detection of attacks. Through this mutual challenge mechanism, the agent is forced to continuously improve its strategy to effectively respond to the opponent's actions. This competitive environment not only improves the learning depth and flexibility of the agent, but also enhances its ability to adapt to unknown threats and environmental changes.

[0197] In this multi-agent environment, agents continuously improve themselves through repeated iterative training and real-world testing. In each training cycle, they adjust their strategies based on the behavior and strategies of their opponents. This dynamic feedback loop allows agents to quickly adapt to new challenges and changes, while also constantly refreshing the system's overall strategy library to form a self-learning and evolving system. Therefore, under this self-confrontation training, the entire automated penetration system can not only handle complex penetration tasks, but also demonstrate superior intelligence and adaptability in the security field.

[0198] S5. Containerize the penetration testing tools and use k8s for orchestration. Use high-quality state actions to perform penetration tests on the trained policy network, repeat the experiment on a specific target machine, and adjust the policy network based on the experimental results. Continuously evaluate the performance of the intelligent agent, record and feedback the parameters of the enhanced GAIL network, and generate monitoring logs.

[0199] In this embodiment, containerization and orchestration are used, that is, the penetration testing tool is containerized and orchestrated using k8s to ensure the portability and scalability of the system. An intelligent load monitoring and prediction module is developed, and machine learning algorithms are used to predict workload changes and automatically adjust container resource allocation to improve overall performance. According to the real-time needs and priorities of the tasks, an adaptive expansion strategy is designed to dynamically adjust the number and distribution of containers to optimize resource usage and response time. The penetration testing process is split into microservice modules, each of which completes only a single task and communicates through lightweight APIs to improve system maintainability and scalability. Using technologies such as ELK (Elasticsearch, Logstash, Kibana) or Prometheus, a distributed log and monitoring system is built to achieve centralized management and real-time analysis of logs.

[0200] After completing container orchestration, set up security for cloud services and k8s, including network security, access control, and permission minimization. Introduce replication and snapshot functions, regularly generate copies of systems and data, and configure automatic failover mechanisms to ensure high availability and rapid recovery capabilities of the system.

[0201] Next, an automated penetration test execution is required. Using the trained policy network to automatically execute penetration tests, repeated experiments can be performed on specific targets. The agent selects actions based on the input state, performs penetration tasks, and adjusts the policy based on feedback. During the penetration test, the performance of the agent is continuously evaluated, including the penetration success rate, the diversity and efficiency of action selection, etc. The parameters and training process of the enhanced GAIL network are adjusted based on the evaluation results. At the same time, the test results are fed back into the training process, and the expert knowledge base and the enhanced GAIL network are continuously iterated to improve the performance and accuracy of automated penetration testing.

[0202] In order to further improve the rapid response capability and security of automated penetration, this embodiment also includes the following steps:

[0203] Automated deployment and management: Use the cloud platform's automation tools (such as CI / CD pipelines) for deployment to ensure configuration consistency and rapid iteration.

[0204] Monitoring and logging: Integrate cloud monitoring services to achieve real-time monitoring, log collection and analysis to facilitate rapid response to security incidents.

[0205] Data protection: Implement data encryption, secure storage, and backup strategies to ensure the security and recoverability of test data.

[0206] Compliance and Auditing: Comply with the compliance requirements of cloud services and k8s, record audit logs, and ensure transparency and traceability of operations.

[0207] Resilience and Disaster Recovery: Configure elastic resources and disaster recovery strategies to address high availability and business continuity requirements in cloud environments.

[0208] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0209] Based on the same idea as the automated penetration method based on generative adversarial imitation learning in the cloud environment in the above-mentioned embodiment, the present invention also provides an automated penetration system based on generative adversarial imitation learning in the cloud environment, which can be used to execute the automated penetration method based on generative adversarial imitation learning in the above-mentioned cloud environment. For ease of explanation, the structural schematic diagram of the embodiment of the automated penetration system based on generative adversarial imitation learning in the cloud environment only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0210] See also Figure 2 In another embodiment of the present application, an automated penetration system 10 based on generative adversarial imitation learning in a cloud environment is provided, the system comprising a knowledge base construction and management module 11, an imitation learning training module 12, an adversarial training module 13, an evaluation and adjustment module 14 and a penetration testing module 15;

[0211] The knowledge base construction and management module 11 is used to collect expert knowledge data, construct an expert knowledge base, enhance the expert knowledge data using a natural language processing algorithm, and regularly update the expert knowledge base using open source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration testing;

[0212] An imitation learning training module 12 is used to set up asynchronous multi-environment, so that the pre-trained intelligent agent can perform parallel generative imitation learning in multiple sandbox environments;

[0213] The adversarial training module 13 is used to generate a predicted state-action pair online from the state currently observed by the agent during the training process, and use the predicted state-action pair to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through an adaptive learning rate and a multi-layer perception mechanism to obtain high-quality state-action pairs; the enhanced GAIL network includes a policy network, a value network and a discriminator;

[0214] The evaluation and adjustment module 14 is used to calculate the discounted rewards of high-quality state-action pairs, and use the discounted rewards as feedback of the policy network to optimize the policy network; use security events to evaluate the policy network, and provide real-time security feedback to the policy network; after multiple rounds of training, the training of the policy network is completed;

[0215] The penetration testing module 15 is used to containerize the penetration testing tools and use k8s for orchestration. It uses high-quality state actions to perform penetration tests on the trained policy network, repeat experiments on specific target machines, and adjust the policy network based on the experimental results; continuously evaluate the performance of the intelligent agent, record and feedback the parameters of the enhanced GAIL network, and generate monitoring logs.

[0216] It should be noted that the automated penetration system based on generative adversarial imitation learning in the cloud environment of the present invention corresponds one-to-one to the automated penetration method based on generative adversarial imitation learning in the cloud environment of the present invention. The technical features and beneficial effects described in the embodiment of the automated penetration method based on generative adversarial imitation learning in the above-mentioned cloud environment are all applicable to the embodiment of the automated penetration method based on generative adversarial imitation learning in the cloud environment. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0217] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An automated penetration method based on generative adversarial imitation learning in a cloud environment, characterized in that: The steps include: Collect expert knowledge data, build an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration testing; Set up asynchronous multi-environment to allow pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments; The current state observed by the agent during the training process is used to generate predicted state-action pairs online, and the predicted state-action pairs are used to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through an adaptive learning rate and a multi-layer perception mechanism to obtain high-quality state-action pairs; the enhanced GAIL network includes a policy network, a value network, and a discriminator; Calculate the discounted rewards for high-quality state-action pairs, and use the discounted rewards as feedback to optimize the policy network; use security events to evaluate the policy network and provide real-time security feedback to the policy network; complete the training of the policy network after multiple rounds of training; The penetration testing tools are containerized and orchestrated using k8s. High-quality state actions are used to perform penetration tests on the trained policy network. Experiments are repeated on specific target machines, and the policy network is adjusted based on the experimental results. The performance of the intelligent agent is continuously evaluated, and the parameters of the enhanced GAIL network are recorded and feedback is provided to generate monitoring logs.

2. According to claim 1, the automated penetration method based on generative adversarial imitation learning in a cloud environment is characterized in that: The method of enhancing expert knowledge data by using a natural language processing algorithm includes: Automatically collect expert reports, use natural language processing algorithms to analyze the content of expert reports, label the expert knowledge data, and add the labeled expert knowledge data to the expert knowledge base.

3. According to claim 1, the automated penetration method based on generative adversarial imitation learning in a cloud environment is characterized in that: The asynchronous multi-environment setting enables the pre-trained agent to perform parallel generative imitation learning in multiple sandbox environments, including: S21. Use large batches to accumulate gradients for the agent in a multi-sandbox environment, and update gradients using multiple small batches of expert knowledge data. S22, using the parameter server architecture to coordinate multiple computing nodes and update computing node parameters in parallel; S23, using synchronous SGD to obtain the distributed average gradient of each computing node, and broadcasting the distributed average gradient to each computing node; S24. Perform mixed precision training on the agent and scale the loss and gradients of the computing nodes. The mixed precision training includes low-precision floating-point operations and high-precision operations.

4. According to claim 3, the automated penetration method based on generative adversarial imitation learning in a cloud environment is characterized in that: The gradient update is as follows: Where N is the number of accumulated batches, is the loss function, represents the gradient with respect to the parameter θ.

5. According to claim 3, the automated penetration method based on generative adversarial imitation learning in a cloud environment is characterized in that: The parallel update calculation node parameters θ t , as follows: Where K is the number of parallel computing nodes, g k is the gradient calculated at node k, and η is the learning rate.

6. The automated penetration method based on generative adversarial imitation learning in a cloud environment according to claim 3 is characterized in that: The step S24 is specifically as follows: Perform low-precision floating-point operations on the agent. When back-propagating the gradient, a scaling factor is used to amplify the loss and increase the gradient, as shown in the following formula: scaled_loss = loss × scale Among them, scaled_loss represents the loss after amplification, loss represents the loss, and scale represents the scaling factor; The gradient is calculated during back propagation as follows: Among them, scaled_gradients represents the gradient; Scaling back to the original gradient is as follows: Among them, gradient represents the scaled gradient; Use the scale factor to update the parameters as follows: Among them, η is the scale factor.

7. The automated penetration method based on generative adversarial imitation learning in a cloud environment according to claim 1 is characterized in that: The penetration testing tools are containerized and orchestrated using k8s, specifically: Set up an intelligent load monitoring and prediction module, use machine learning algorithms to predict workload changes, automatically adjust container resource allocation, design adaptive expansion strategies based on the real-time needs and priorities of tasks, and dynamically adjust the number and distribution of containers; The penetration testing process is split into microservice modules. Each module completes only a single task and communicates through lightweight APIs. A distributed log and monitoring system is built to achieve centralized log management and real-time analysis.

8. The automated penetration method based on generative adversarial imitation learning in a cloud environment according to claim 7 is characterized in that: This includes strengthening the security settings of cloud services and K8s, specifically: Introduce replication and snapshot functions to regularly generate copies of systems and data, and configure automatic failover mechanisms; The security settings include network security, access control, and privilege minimization.

9. The automated penetration method based on generative adversarial imitation learning in a cloud environment according to claim 1, characterized in that: include: Self-adversarial training: Combined with self-adversarial training strategies, allowing agents to challenge each other; Automated deployment and management: Use the cloud platform's automated tools for deployment to ensure configuration consistency and rapid iteration; Monitoring and logs: Integrate cloud monitoring services to achieve real-time monitoring, log collection and analysis for rapid response to security incidents; Data protection: Implement data encryption, secure storage, and backup strategies to ensure the security and recoverability of test data; Compliance and auditing: Comply with the compliance requirements of cloud services and k8s, and record audit logs to ensure transparency and traceability of operations; Resilience and Disaster Recovery: Configure elastic resources and disaster recovery strategies to address high availability and business continuity requirements in cloud environments.

10. An automated penetration system based on generative adversarial imitation learning in a cloud environment, characterized in that: An automated penetration method based on generative adversarial imitation learning in a cloud environment as described in any one of claims 1 to 9, comprising a knowledge base construction and management module, an imitation learning training module, an adversarial training module, an evaluation and adjustment module, and a penetration testing module; A knowledge base construction and management module is used to collect expert knowledge data, construct an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration testing; The imitation learning training module is used to set up asynchronous multi-environment and enable pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments; The adversarial training module is used to generate predicted state-action pairs online from the state currently observed by the agent during the training process, and use the predicted state-action pairs to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through an adaptive learning rate and a multi-layer perception mechanism to obtain high-quality state-action pairs; the enhanced GAIL network includes a policy network, a value network and a discriminator; The evaluation and adjustment module is used to calculate the discounted rewards for high-quality state-action pairs and use the discounted rewards as feedback to optimize the policy network. The policy network is evaluated using security events and given real-time security feedback. After multiple rounds of training, the policy network training is completed. The penetration testing module is used to containerize the penetration testing tools and orchestrate them using k8s. It uses high-quality state actions to perform penetration tests on the trained policy network, repeats experiments on specific target machines, and adjusts the policy network based on the experimental results. It continuously evaluates the performance of the intelligent agent, records and provides feedback to adjust the parameters of the enhanced GAIL network, and generates monitoring logs.

Images