An automatic penetration method and system based on generative adversarial imitation learning in a cloud environment

Abstract

The application discloses an automatic penetration method and system based on generative adversarial imitation learning in a cloud environment, which comprises the following steps: using the Exploit open source tool and the Metasploit open source penetration tool to store the action-state pairs executed when the target machine is successfully penetrated by using the reinforcement learning model to construct a penetration expert knowledge base; inputting the content in the expert knowledge base into a discriminator in an imitation learning framework for training; using the actions output by the discriminator after training the imitation learning framework, which can be used for automatic penetration of specified hosts; and deploying the entire automatic penetration framework to the cloud environment to simulate external network attacks through containerization and orchestration. The application can efficiently utilize resources by performing imitation learning in the cloud environment, and can automatically adjust container resource allocation by enhancing the GAIL network and containerizing the penetration testing tool, thereby improving overall performance and optimizing resource utilization and response time.

Description

Technical Field

[0001] This invention belongs to the technical field of network security attack and defense, specifically relating to an automated penetration method and system based on generative adversarial learning in a cloud environment. Background Technology

[0002] In today's digital age, the global cybersecurity landscape is facing unprecedented complexity and challenges. With the rapid development of internet technology, cyberspace has become an integral part of national economies, social operations, and citizens' lives. However, the accompanying cybersecurity risks are also rising sharply, posing a serious threat to personal privacy, business operations, and even national security.

[0003] Currently, enterprises primarily focus on using firewalls and WAFs to address network attacks during and after the attack. For pre-attack prediction and prevention, enterprises generally rely on manual penetration testing and vulnerability scanning tools to detect and assess network vulnerabilities. Penetration testing is an assessment method that simulates malicious hacker attack methods to evaluate the security of computer network systems. The cost from discovering to patching security vulnerabilities can reach millions of dollars. Penetration testing can effectively identify and fix these security vulnerabilities proactively before they are exploited, preventing hackers from using them to launch attacks.

[0004] However, as businesses become increasingly globalized and complex, the limitations of manual penetration testing are becoming increasingly apparent. For example, it is difficult to implement, has a significant impact on business systems, sometimes requires system shutdowns, and is prohibitively costly; it lacks integration with business operations and struggles to adapt to cloud-based and virtualized business environments; and it is difficult to predict risks to critical assets in real time. Annual manual penetration testing, which requires system downtime, provides insufficient and outdated test reports. Enterprises have an increasingly urgent need for penetration testing that provides continuous, automated, and intelligent testing.

[0005] Currently, there are several approaches to automated penetration testing: First, rule-matching methods, which are traditional automated penetration testing techniques that rely on predefined rules and signatures to detect and exploit vulnerabilities. However, they are limited by their reliance on known vulnerability signatures and attack patterns, as well as the updating of the rule base, resulting in a lack of adaptability and flexibility due to the limited coverage of known rules. Second, machine learning-based methods, which have been a research hotspot in recent years, utilize intelligent algorithms to automatically identify and exploit vulnerabilities. However, they are limited by data quality and model interpretability. Third, reinforcement learning-based automated penetration testing methods face challenges in balancing exploration and exploitation, as well as training time and environmental stability.

[0006] Therefore, the high cost, time consumption, lack of automation and intelligence of manual penetration testing, as well as the shortcomings of existing automated penetration testing technologies in identifying new vulnerabilities, adapting to the environment, and generalizing strategies, are issues that urgently need to be addressed. Summary of the Invention

[0007] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an automated penetration testing method and system based on generative adversarial learning in a cloud environment. By performing imitation learning in the cloud environment, resources can be utilized efficiently. At the same time, by enhancing the GAIL network and containerizing penetration testing tools, container resource allocation can be automatically adjusted, thereby improving overall performance and optimizing resource usage and response time.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides an automated penetration testing method based on generative adversarial learning in a cloud environment, comprising the following steps:

[0010] Collect expert knowledge data, construct an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open-source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration tests.

[0011] Set up asynchronous multi-environment to enable pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments;

[0012] The agent generates predicted state-action pairs online from the current observed state during training. These predicted state-action pairs are then used to train the discriminator in the augmented GAIL network. The discriminator dynamically adjusts the state through an adaptive learning rate and a multilayer perceptron mechanism to obtain high-quality state-action pairs. The augmented GAIL network includes a policy network, a value network, and a discriminator.

[0013] Calculate the discounted reward for high-quality state-action pairs, and use the discounted reward as feedback to optimize the policy network; evaluate the policy network using security events and provide real-time security feedback to the policy network; complete the training of the policy network after multiple rounds of training.

[0014] The penetration testing tool is containerized and orchestrated using Kubernetes. High-quality state-action pairs are used to perform penetration testing on the trained policy network. Repeated experiments are conducted on specific target machines, and the policy network is adjusted based on the experimental results. The performance of the agent is continuously evaluated, and the parameters for adjusting and enhancing the GAIL network are recorded and fed back, generating monitoring logs.

[0015] As a preferred technical solution, the enhancement of expert knowledge data using natural language processing algorithms includes:

[0016] The system automatically collects expert reports, analyzes the content of these reports using natural language processing algorithms, tags the expert knowledge data, and adds the tagged expert knowledge data to the expert knowledge base.

[0017] As a preferred technical solution, the setting of asynchronous multi-environment, enabling the pre-trained agent to perform parallel generative imitation learning in multiple sandbox environments, includes:

[0018] S21. In a multi-sandbox environment, gradient accumulation is performed on the agent using large batches of expert knowledge data, and gradient updates are performed using multiple small batches of expert knowledge data.

[0019] S22. Use a parameter server architecture to coordinate multiple compute nodes and update compute node parameters in parallel.

[0020] S23. Use synchronous SGD to obtain the distributed average gradient of each computing node, and broadcast the distributed average gradient to each computing node.

[0021] S24. Perform mixed-precision training on the agent, and scale the loss and gradient of the computation node. The mixed-precision training includes low-precision floating-point operations and high-precision operations.

[0022] As a preferred technical solution, the gradient update is as follows:

[0023]

[0024] Where N is the cumulative number of batches, It is a loss function. This represents the gradient with respect to the parameter θ.

[0025] As a preferred technical solution, the parallel update of computing node parameters θ t As shown in the following formula:

[0026]

[0027] Where K is the number of parallel computing nodes, g k η is the gradient calculated at node k, and η is the learning rate.

[0028] As a preferred technical solution, step S24 specifically includes:

[0029] Low-precision floating-point operations are performed on the agent. During backpropagation gradient calculation, a scaling factor is used to amplify the loss and increase the gradient, as shown in the following equation:

[0030] scaled_loss = loss × scale

[0031] Where scaled_loss represents the amplified loss, loss represents the loss, and scale represents the scaling factor;

[0032] The gradient is calculated during backpropagation, as shown in the following equation:

[0033]

[0034] Where scaled_gradients represents the gradient;

[0035] Scaling back to the original gradient, as shown in the following equation:

[0036]

[0037] Where gradient represents the scaled gradient;

[0038] The parameters are updated using the scaling factor, as shown in the following equation:

[0039]

[0040] Where η is the scale factor.

[0041] Secondly, the present invention provides an automated penetration testing system based on generative adversarial learning in a cloud environment, which is applied to the automated penetration testing method based on generative adversarial learning in the cloud environment, including: a knowledge base construction and management module, an imitation learning training module, an adversarial training module, an evaluation and adjustment module, and a penetration testing module.

[0042] The knowledge base construction and management module is used to collect expert knowledge data, build an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open-source security intelligence; the expert knowledge data includes expert status action pairs of security experts in successful penetration testing.

[0043] The imitation learning training module is used to set up asynchronous multi-environment environments, enabling pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments.

[0044] The adversarial training module is used to generate predicted state-action pairs online from the current observed state of the agent during training. The predicted state-action pairs are used to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through adaptive learning rate and multilayer perception mechanism to obtain high-quality state-action pairs. The enhanced GAIL network includes a policy network, a value network, and a discriminator.

[0045] The evaluation and adjustment module is used to calculate the discounted reward for high-quality state-action pairs, and uses the discounted reward as feedback to optimize the policy network; it evaluates the policy network using security events and provides real-time security feedback to the policy network; after multiple rounds of training, the training of the policy network is completed.

[0046] The penetration testing module is used to containerize penetration testing tools and orchestrate them using Kubernetes. It uses high-quality state-action pairs to perform penetration testing on the trained policy network, repeats experiments on specific target machines, adjusts the policy network based on the experimental results, continuously evaluates agent performance, records and feeds back the parameters for adjusting and enhancing the GAIL network, and generates monitoring logs.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] (1) This invention enables pre-trained agents to perform parallel generative imitation learning in a multi-sandbox environment. On the one hand, it can improve training efficiency and model generalization ability. On the other hand, through mixed precision training, it can reduce computational costs, ensure model training quality, and improve model training speed, thereby making efficient use of resources.

[0049] (2) This invention introduces an adaptive learning rate and a multi-layer perception mechanism by enhancing the GAIL network. This can optimize the accuracy of policy generation and value assessment, dynamically adjust the state representation, improve the generation quality of state-action pairs, and automatically adjust container resource allocation by containerizing penetration testing tools, thereby improving overall performance and optimizing resource usage and response time.

[0050] (3) This invention optimizes the automated penetration testing framework and deploys it in the cloud environment, which can quickly adapt to the dynamic changes in the cloud environment, such as the elastic scaling of resources and the instant update of configuration. Compared with locally deployed penetration testing tools, it is more flexible and responsive. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of an automated penetration testing method based on generative adversarial learning in a cloud environment, as described in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the structure of an automated penetration testing system based on generative adversarial learning in a cloud environment, according to an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0055] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0056] Please see Figure 1 This embodiment provides an automated penetration testing method based on generative adversarial learning in a cloud environment, including the following steps:

[0057] S1. Collect expert knowledge data, construct an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open-source security intelligence; the expert knowledge data includes expert status action pairs of security experts in successful penetration testing.

[0058] In this embodiment, the penetration testing expert knowledge base involves combining penetration testing automation tools (such as DeepExploit) with a state-action pair storage mechanism to build a dynamic expert knowledge base based on actual penetration testing results.

[0059] First, define the state and action.

[0060] 1. Define the state:

[0061] (1) Host information.

[0062] Host operating system: such as Linux, Windows, macOS, etc.

[0063] Operating system patch status: including installed patches and uninstalled patches.

[0064] Firewall status: The host's firewall configuration and rules.

[0065] Antivirus software: Whether antivirus software and its version are installed.

[0066] (2) Service information.

[0067] Service security configuration: such as whether secure communication (HTTPS) is enabled, whether authentication is enabled, etc.

[0068] Service port open status: Which ports are open, especially sensitive ports: 21 (FTP), 22 (SSH), 23 (Telnet), 25 (SMTP), 53 (DNS), 80 (HTTP), 110 (POP3), 123 (NTP), 137-139 (NetBIOS), 443 (HTTPS), 445 (SMB), 3389 (RDP), 1433 (MSSQL), 3306 (MySQL), 8080 (custom HTTP service).

[0069] Service runtime status: whether the service is running, runtime, etc.

[0070] (3) Application information.

[0071] Installed software: Software and its versions installed on the host computer.

[0072] Web application information: Web applications running on the host, including web server type, web framework, API endpoints, etc.

[0073] Configuration file information: The contents of the application and service configuration files and their sensitive information (such as default passwords).

[0074] (4) User and authentication information.

[0075] User Accounts: A list of accounts, including account type (administrator, regular user, etc.) and account status (enabled or disabled).

[0076] Authentication methods: The authentication methods used by the host and services, including password, two-factor authentication, Kerberos, etc.

[0077] Access logs: Recent login records and attempts, including successful and failed login attempts.

[0078] (5) Network information.

[0079] Subnet and network segment: Information about the subnet and network segment where the host is located.

[0080] Routing information: Router configuration information and connections to other subnets.

[0081] Network protocol: The network protocol used by the host (such as IPv4, IPv6, etc.).

[0082] (6) Environmental information.

[0083] Virtualization status: Whether the host is running in a virtualization environment, such as VMware, VirtualBox, KVM, etc.

[0084] Cloud environment information: If the host is running in the cloud, provide information about the cloud service provider and related configurations (such as AWS, Azure, Google Cloud, etc.).

[0085] These states will be mapped one-to-one into numbers and stored. For example, Windows 7 will be mapped to 1, Windows 8 to 2, Windows 10 to 3, and so on. If a system mapping that does not exist in the database is encountered, a new mapping will be automatically added and recorded.

[0086] The same applies to other states such as port numbers; simply specify an initial number manually, and then increment it according to certain rules.

[0087] 2. Define the action:

[0088] Including but not limited to the following information:

[0089] (1) Specific attack payload;

[0090] (2) The Metasploit module used;

[0091] (3) Attack the target port;

[0092] (4) Attack parameters.

[0093] Assume the target host's IP address is 192.168.1.100, which is a server running Windows Server 2016 with some common services and ports open, and that there are known vulnerabilities.

[0094] 3. Current status:

[0095] (1) Host information:

[0096] Host IP: 192.168.1.100

[0097] Operating System: Windows Server 2016

[0098] Patch status: Some patches are installed, but KB5001402 is not installed.

[0099] Firewall status: Enabled

[0100] Antivirus software: Windows Defender

[0101] (2) Service Information:

[0102] Open ports:

[0103] 80 (HTTP) - Service Name: IIS, Version: 10.0

[0104] 445 (SMB) - Service Name: File Sharing 3389 (RDP) - Service Name: Remote Desktop

[0105] Service security configuration: HTTPS is not enabled for HTTP, and the SMB service has weak passwords.

[0106] Service dependencies: None

[0107] Service status: All services are running.

[0108] (3) Application information:

[0109] Installed software: IIS, SQL Server 2016

[0110] Web application information: Running an application called "WebApp"

[0111] Configuration file information: No particularly sensitive information exposed.

[0112] (4) User and authentication information:

[0113] User account: Administrator account Admin01 is enabled, with a weak password policy.

[0114] Authentication method: Password authentication only

[0115] Access logs: No abnormal login attempts.

[0116] (5) Network information:

[0117] Subnet information: 192.168.1.0 / 24

[0118] Routing information: Default gateway 192.168.1.1

[0119] Network protocols: IPv4, IPv6

[0120] (6) Environmental information:

[0121] Virtualization status: Running on Hyper-V

[0122] Cloud environment information: None

[0123] Vulnerability Information: An unpatched SMB v1 vulnerability (MS17-010, EternalBlue) is known to exist.

[0124] 4. Execution action information:

[0125] Suppose a penetration tester decides to exploit the known EternalBlue vulnerability (MS17-010) and uses the exploit / windows / smb / ms17_010_eternalblue module in Metasploit to attempt to execute remote code on the target host and obtain a Meterpreter session.

[0126] (1)Payload: windows / x64 / meterpreter / reverse_tcp

[0127] (2) Metasploit module: exploit / windows / smb / ms17_010_eternalblue

[0128] (3) Target port: 445

[0129] (4) Attack parameters:

[0130] RHOST: 192.168.1.100

[0131] RPORT: 445

[0132] LHOST: The attacker's IP address, such as 192.168.1.10

[0133] LPORT: The attacker's listening port, for example: 4444

[0134] This action payload, numbered 137 in the specified sequence, is therefore recorded as action sequence 137 and stored in the database. Thus, the state-action pair created in this example is:

[0135] State: [198,34,105,76,142,23,210,67,89150,3,201,56,123,90,4,254,18,72,113…]

[0136] Action:

[137]

[0137] Next, to increase the richness of state-action pairs, this embodiment improves the expert knowledge base.

[0138] S11. Introduce automated data collection and labeling technologies, and use natural language processing (NLP) algorithms to analyze expert reports;

[0139] S12. Utilize open-source security intelligence and regularly update the expert knowledge base to ensure the timeliness of training data and coverage of new threats.

[0140] Before automating penetration testing, agents are pre-trained so that they can perform basic penetration tasks in a simulated environment. These agents can employ reinforcement learning (RL) or deep reinforcement learning (DRL) models.

[0141] S2. Set up asynchronous multi-environment to enable pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments.

[0142] In this embodiment, an asynchronous multi-environment training framework is designed to allow RL / DRL models to learn in parallel in multiple sandbox environments, thereby improving training efficiency and model generalization ability.

[0143] Specifically, in a cloud environment, generative imitation learning can be optimized in a targeted manner, primarily with the goal of efficient resource utilization, including:

[0144] S21. In a multi-sandbox environment, gradient accumulation is performed on the agent using large batches of expert knowledge data, and gradient updates are performed using multiple small batches of expert knowledge data.

[0145] In multi-sandbox environments, larger batch sizes are used for gradient accumulation to reduce communication latency. By combining multiple small batches of data to compute a large update, GPU / TPU can be effectively utilized.

[0146] Optimized gradient calculation formula:

[0147] Where N is the cumulative number of batches, It is a loss function. This represents the gradient with respect to the parameter θ.

[0148] S22. Use a parameter server architecture to coordinate multiple compute nodes and update compute node parameters in parallel.

[0149] A parameter server architecture is used, allowing compute nodes to update parameters in parallel. The optimized model update formula coordinates multiple nodes through the parameter server. The parallel update of compute node parameters θ... t As shown in the following formula:

[0150]

[0151] Where K is the number of parallel computing nodes, g k η is the gradient calculated at node k, and η is the learning rate.

[0152] S23. Use synchronous SGD to obtain the distributed average gradient of each computing node, and broadcast the distributed average gradient to each computing node.

[0153] The synchronous SGD algorithm combines stochastic gradient descent (SGD) with a synchronous bulk synchronous processing (BSP) framework. In a distributed system, multiple worker nodes each compute gradients based on their local subsets of data, and then these gradients are aggregated to the master node (or parameter server). The master node averages these gradients to form a new global gradient, which is used to update the model parameters. This process is synchronized after each mini-batch data update, ensuring that all worker nodes train with the latest model parameters.

[0154] In this process, using distributed averaging ensures that the parameter updates of all nodes are synchronized, and the gradient is as follows:

[0155]

[0156] S24. Perform mixed-precision training on the agent, and scale the loss and gradient of the computation node. The mixed-precision training includes low-precision floating-point operations and high-precision operations.

[0157] Low-precision floating-point operations are performed on the agent. During backpropagation gradient calculation, a scaling factor is used to amplify the loss and increase the gradient, as shown in the following equation:

[0158] scaled_loss = loss × scale

[0159] Where scaled_loss represents the amplified loss, loss represents the loss, and scale represents the scaling factor;

[0160] The gradient is calculated during backpropagation, as shown in the following equation:

[0161]

[0162] Where scaled_gradients represents the gradient;

[0163] Scaling back to the original gradient, as shown in the following equation:

[0164]

[0165] Where gradient represents the scaled gradient;

[0166] The parameters are updated using the scaling factor, as shown in the following equation:

[0167]

[0168] Here, η is the scale factor, which can be calculated using standard optimization algorithms (such as SGD, Adam). The scale value is not fixed; the standard approach is to start with a large scale (e.g., 2). 8 ~2 15 Then, during training, the loss is dynamically adjusted to avoid gradient overflow (e.g., the scale is reduced if overflow is detected). Since mixed-precision training, especially when applying loss scaling, involves amplifying the loss value to avoid numerical precision issues, it's necessary to adjust the loss value so that the gradients during backpropagation in FP16 are not too small and information is lost. After adjustment, the original loss is restored. This ensures the stability of FP16 during backpropagation and guarantees that the final parameter update process is indeed correct.

[0169] In mixed-precision training, computational costs are reduced without significantly impacting model training quality. Leveraging the optimized performance of modern GPUs for FP16, mixed-precision training can achieve significant acceleration.

[0170] S3. During the training process, the agent generates predicted state-action pairs online from the observed states. These predicted state-action pairs are then used to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts its state through an adaptive learning rate and a multilayer perceptron mechanism to obtain high-quality state-action pairs. The enhanced GAIL network includes a policy network, a value network, and a discriminator.

[0171] This embodiment employs Generative Adversarial Imitation Learning (GAIL), a method that combines Generative Adversarial Networks (GANs) and imitation learning. The core idea of ​​GAIL is to optimize the agent's policy through adversarial methods during the imitation learning process, making its behavior in a given state as close as possible to that of an expert.

[0172] Generative adversarial imitation learning comprises two main components:

[0173] Generator: The generator is our strategy that attempts to generate a sequence of actions that appear to be from an expert.

[0174] Discriminator: The discriminator attempts to distinguish between real expert data and data generated by the generator.

[0175] These two components form an adversarial network. The more realistic the data generated by the generator, the harder it is for the discriminator to distinguish between expert data and generator data.

[0176] Next, the training process of GAIL can be represented by the following optimization problem:

[0177]

[0178] Where: π expert This represents the expert's strategy, π θ It is a parameterized generation strategy, D w It is a discriminator with parameter w.

[0179] The specific steps are as follows:

[0180] Initialization: Initialize the generator (strategy) Π θ Discriminator D w The parameters.

[0181] Discriminator Training: Discriminator D w The aim is to maximize the following objective: distinguishing between expert and generative policies.

[0182]

[0183] Generator Training: Generator Π θ The goal is to minimize the generated data so that it increasingly resembles genuine expert data:

[0184]

[0185] Repeated iterations: Repeat the above steps, alternately optimizing the generator and discriminator until convergence.

[0186] To optimize the accuracy of policy generation and value assessment, this embodiment introduces an adaptive learning rate and a multi-layer perception mechanism into the GAIL network, and develops an intelligent state perception module based on sensor data fusion, which can dynamically adjust the state representation and improve the generation quality of state-action pairs.

[0187] After the deployment of the enhanced GAIL network is completed, discriminator training, adversarial training, and policy network optimization are carried out respectively.

[0188] (1) Discriminator Training: During training, the agent inputs the first state-action pair generated by the policy network (Actor) and the second state-action pair generated by the value network (Critic) into the discriminator. The discriminator dynamically adjusts the state through adaptive learning rate and multilayer perception mechanism to obtain high-quality state-action pairs. In this process, the discriminator's goal is to distinguish between expert samples and agent-generated samples.

[0189] In Enhanced Generative Adversarial Imitation Learning (GAIL) networks, the "adaptive learning rate" is primarily manifested in the discriminator's training process. The discriminator's learning rate is not fixed but dynamically adjusted based on training progress and model feedback. The adaptive learning rate algorithm can adjust the learning rate according to changes in gradients and its own performance metrics (such as changes in loss) to adapt to the training needs at different stages. The adaptive learning rate can also improve training efficiency, helping the discriminator converge to a stable state more quickly because it can increase the learning rate when rapid learning is needed and decrease it when fine-tuning is required. Specifically, in automated penetration testing, the role of the adaptive learning rate is generally to ensure that the discriminator can effectively learn and adjust in a constantly changing training environment, thereby acquiring high-quality state-action pairs and optimizing the overall performance of the GAIL network.

[0190] Multilayer sensing mechanisms can be used to build intelligent state-aware systems because they provide a more detailed and hierarchical analytical perspective, helping the system to better understand and process complex input information.

[0191] (2) Adversarial training: Through an adversarial process, the actions generated by the policy network (Actor) attempt to deceive the discriminator network into believing that these actions are expert-level. The discriminator network then attempts to correctly classify these actions.

[0192] (3) Optimization of the policy network. The discounted reward output by the discriminator network is used as feedback to guide the training of the policy network. This process optimizes the agent's policy by mimicking the behavior of experts. See step S4 for details.

[0193] S4. Calculate the discounted reward for high-quality state-action pairs, and use the discounted reward as feedback to optimize the policy network; evaluate the policy network using security events and provide real-time security feedback to the policy network; complete the training of the policy network after multiple rounds of training.

[0194] In addition to the feedback from the training process itself, this embodiment incorporates a real-time security feedback mechanism during the imitation learning process to improve the dynamic performance of the policy network. This mechanism combines post-evaluation of security events to dynamically adjust and optimize the policy. Subsequently, test results are iteratively fed back into the training process, continuously optimizing the expert knowledge base and the GAIL network to improve the performance and accuracy of automated penetration testing.

[0195] In addition, this embodiment can also incorporate a self-adversarial training strategy, allowing agents to challenge each other, further enhancing the intelligence and adaptability of automated penetration.

[0196] In automated penetration testing methods, self-adversarial training strategies introduce multiple agents to compete against each other in the same environment, prompting them to optimize their strategies in simulated adversarial scenarios. Each agent can play a different role; for example, one agent performs attack operations to simulate penetration testing, while another agent is responsible for defense or attack detection. Through this mutually challenging mechanism, agents are forced to continuously improve their strategies to effectively respond to adversary actions. This competitive environment not only enhances the learning depth and flexibility of the agents but also strengthens their adaptability to unknown threats and environmental changes.

[0197] In this multi-agent environment, agents continuously improve themselves through iterative training and real-world testing. During each training cycle, they adjust their strategies based on the opponent's behavior and tactics. This dynamic feedback loop allows agents to quickly adapt to new challenges and changes, while constantly updating the system's overall strategy library, forming a self-learning and evolving system. Therefore, under this self-adversarial training, the entire automated penetration testing system can not only handle complex penetration tasks but also demonstrate superior intelligence and adaptability in the security field.

[0198] S5. Containerize the penetration testing tool and orchestrate it using Kubernetes. Use high-quality state-action pairs to perform penetration testing on the trained policy network. Repeat the experiment on a specific target machine and adjust the policy network based on the experimental results. Continuously evaluate the agent's performance, record and provide feedback on the parameters for adjusting and enhancing the GAIL network, and generate monitoring logs.

[0199] In this embodiment, containerization and orchestration are employed. The penetration testing tools are containerized and orchestrated using Kubernetes to ensure system portability and scalability. An intelligent load monitoring and prediction module is developed, utilizing machine learning algorithms to predict workload changes and automatically adjust container resource allocation to improve overall performance. An adaptive scaling strategy is designed based on the real-time requirements and priorities of tasks, dynamically adjusting the number and distribution of containers to optimize resource usage and response time. The penetration testing process is broken down into microservice modules, each completing a single task and communicating through lightweight APIs to improve system maintainability and scalability. A distributed log and monitoring system is built using technologies such as ELK (Elasticsearch, Logstash, Kibana) or Prometheus to achieve centralized log management and real-time analysis.

[0200] After container orchestration is complete, security settings are configured for cloud services and Kubernetes, including network security, access control, and privilege quorum. Replication and snapshot capabilities are introduced to periodically generate copies of the system and data, and an automatic failover mechanism is configured to ensure high system availability and rapid recovery capabilities.

[0201] Next, automated penetration testing needs to be performed. Using a pre-trained policy network, penetration tests can be executed automatically, allowing for repeated experiments on specific target machines. The agent selects actions based on the input state, executes the penetration task, and adjusts the strategy based on feedback. During the penetration test, the agent's performance is continuously evaluated, including penetration success rate, action selection diversity, and efficiency. The parameters and training process of the enhanced GAIL network are adjusted based on the evaluation results. Simultaneously, test results are fed back into the training process, continuously iterating and optimizing the expert knowledge base and the enhanced GAIL network to improve the performance and accuracy of automated penetration testing.

[0202] To further improve the rapid response capability and security of automated penetration testing, this embodiment also includes the following steps:

[0203] Automated deployment and management: Utilize cloud platform automation tools (such as CI / CD pipelines) for deployment to ensure configuration consistency and rapid iteration.

[0204] Monitoring and logging: Integrates cloud monitoring services to enable real-time monitoring, log collection and analysis, facilitating rapid response to security incidents.

[0205] Data protection: Implement data encryption, secure storage, and backup strategies to ensure the security and recoverability of test data.

[0206] Compliance and Audit: Adhere to the compliance requirements of cloud services and Kubernetes, record audit logs, and ensure the transparency and traceability of operations.

[0207] Elasticity and Disaster Recovery: Configure elastic resources and disaster recovery strategies to address the high availability and business continuity requirements in cloud environments.

[0208] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0209] Based on the same idea as the automated penetration testing method based on generative adversarial learning in a cloud environment described in the above embodiments, this invention also provides an automated penetration testing system based on generative adversarial learning in a cloud environment. This system can be used to execute the aforementioned automated penetration testing method based on generative adversarial learning in a cloud environment. For ease of explanation, the structural diagram of the embodiment of the automated penetration testing system based on generative adversarial learning in a cloud environment only shows the parts related to the embodiments of this invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0210] Please see Figure 2 In another embodiment of this application, an automated penetration testing system 10 based on generative adversarial learning in a cloud environment is provided. The system includes a knowledge base construction and management module 11, an imitation learning training module 12, an adversarial training module 13, an evaluation and adjustment module 14, and a penetration testing module 15.

[0211] The knowledge base construction and management module 11 is used to collect expert knowledge data, construct an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open-source security intelligence; the expert knowledge data includes expert status action pairs of security experts in successful penetration testing.

[0212] The imitation learning training module 12 is used to set up asynchronous multi-environment, enabling the pre-trained agent to perform parallel generative imitation learning in multiple sandbox environments;

[0213] The adversarial training module 13 is used to generate predicted state-action pairs online from the current observed state of the agent during the training process, and to train the discriminator in the enhanced GAIL network using the predicted state-action pairs. The discriminator dynamically adjusts the state through adaptive learning rate and multilayer perception mechanism to obtain high-quality state-action pairs. The enhanced GAIL network includes a policy network, a value network and a discriminator.

[0214] Evaluation and adjustment module 14 is used to calculate the discounted reward for high-quality state-action pairs, and use the discounted reward as feedback to optimize the policy network; it evaluates the policy network using security events and provides real-time security feedback to the policy network; after multiple rounds of training, the training of the policy network is completed.

[0215] The penetration testing module 15 is used to containerize penetration testing tools and orchestrate them using Kubernetes. It uses high-quality state-action pairs to perform penetration testing on the trained policy network, repeats experiments on specific target machines, adjusts the policy network based on the experimental results, continuously evaluates agent performance, records and feeds back the parameters for adjusting and enhancing the GAIL network, and generates monitoring logs.

[0216] It should be noted that the automated penetration testing system based on generative adversarial learning in the cloud environment of the present invention corresponds one-to-one with the automated penetration testing method based on generative adversarial learning in the cloud environment of the present invention. The technical features and beneficial effects described in the above embodiments of the automated penetration testing method based on generative adversarial learning in the cloud environment are all applicable to the embodiments of the automated penetration testing method based on generative adversarial learning in the cloud environment. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0217] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An automated penetration testing method based on generative adversarial learning in a cloud environment, characterized in that, Includes the following steps: Collect expert knowledge data, construct an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open-source security intelligence; the expert knowledge data includes expert state-action pairs of security experts in successful penetration tests. Set up asynchronous multi-environment to enable pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments; The agent generates predicted state-action pairs online from the current observed state during training. These predicted state-action pairs are then used to train the discriminator in the augmented GAIL network. The discriminator dynamically adjusts the state through an adaptive learning rate and a multilayer perceptron mechanism to obtain high-quality state-action pairs. The augmented GAIL network includes a policy network, a value network, and a discriminator. Calculate the discounted reward for high-quality state-action pairs, and use the discounted reward as feedback to optimize the policy network; evaluate the policy network using security events and provide real-time security feedback to the policy network; complete the training of the policy network after multiple rounds of training. The penetration testing tool is containerized and orchestrated using Kubernetes. High-quality state-action pairs are used to perform penetration testing on the trained policy network. Repeated experiments are conducted on specific target machines. The experimental results are fed back and the policy network is adjusted. The performance of the agent is continuously evaluated, the parameters of the GAIL network are recorded and fed back to adjust and enhance the network, and monitoring logs are generated.

2. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 1, characterized in that, The enhancement of expert knowledge data using natural language processing algorithms includes: The system automatically collects expert reports, analyzes the content of these reports using natural language processing algorithms, tags the expert knowledge data, and adds the tagged expert knowledge data to the expert knowledge base.

3. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 1, characterized in that, The aforementioned asynchronous multi-environment setup, enabling pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments, includes: S21. In a multi-sandbox environment, gradient accumulation is performed on the agent using large batches of expert knowledge data, and gradient updates are performed using multiple small batches of expert knowledge data. S22. Use a parameter server architecture to coordinate multiple compute nodes and update compute node parameters in parallel. S23. Use synchronous SGD to obtain the distributed average gradient of each computing node, and broadcast the distributed average gradient to each computing node. S24. Perform mixed-precision training on the agent, and scale the loss and gradient of the computation node. The mixed-precision training includes low-precision floating-point operations and high-precision operations.

4. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 3, characterized in that, The gradient update is as follows: in, N It is the cumulative number of batches. It is a loss function. Relative to the parameter The gradient.

5. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 3, characterized in that, The parallel update of computing node parameters As shown in the following formula: in, It is the number of parallel computing nodes. It is a node The calculated gradient, It is the learning rate.

6. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 3, characterized in that, Step S24 specifically includes: Low-precision floating-point operations are performed on the agent. During backpropagation gradient calculation, a scaling factor is used to amplify the loss and increase the gradient, as shown in the following equation: scaled_loss = loss × scale Where scaled_loss represents the amplified loss, loss represents the loss, and scale represents the scaling factor; The gradient is calculated during backpropagation, as shown in the following equation: scaled_gradients =∇( scaled_loss ) Where scaled_gradients represents the gradient; Scaling back to the original gradient, as shown in the following equation: Where gradient represents the scaled gradient; The parameters are updated using the scaling factor, as shown in the following equation: in, is the scale factor.

7. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 1, characterized in that, The containerization of penetration testing tools and their orchestration using Kubernetes are specifically described as follows: Set up an intelligent load monitoring and prediction module, use machine learning algorithms to predict workload changes, automatically adjust container resource allocation, and design adaptive scaling strategies based on the real-time needs and priorities of tasks to dynamically adjust the number and distribution of containers. The penetration testing process is broken down into microservice modules, each of which performs a single task and communicates through lightweight APIs. This builds a distributed logging and monitoring system to achieve centralized management and real-time analysis of logs.

8. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 7, characterized in that, This includes strengthening the security settings for cloud services and Kubernetes, specifically: Introduce replication and snapshot capabilities to periodically generate copies of the system and data, and configure an automatic failover mechanism; The security settings include network security, access control, and privilege maximization.

9. The automated penetration testing method based on generative adversarial learning in a cloud environment according to claim 1, characterized in that, include: Self-adversarial training: Combining self-adversarial training strategies allows agents to challenge each other; Automated deployment and management: Utilize automated tools on the cloud platform for deployment, ensuring configuration consistency and rapid iteration; Monitoring and Logging: Integrates cloud monitoring services to enable real-time monitoring, log collection and analysis for rapid response to security incidents; Data protection: Implement data encryption, secure storage, and backup strategies to ensure the security and recoverability of test data; Compliance and Audit: Adhere to cloud service and Kubernetes compliance requirements, and keep audit logs to ensure the transparency and traceability of operations; Elasticity and Disaster Recovery: Configure elastic resources and disaster recovery strategies to address the high availability and business continuity requirements in cloud environments.

10. An automated penetration testing system based on generative adversarial learning in a cloud environment, characterized in that, An automated penetration testing method based on generative adversarial imitation learning, applicable to any one of claims 1-9 in a cloud environment, includes a knowledge base construction and management module, an imitation learning training module, an adversarial training module, an evaluation and adjustment module, and a penetration testing module; The knowledge base construction and management module is used to collect expert knowledge data, build an expert knowledge base, enhance the expert knowledge data using natural language processing algorithms, and regularly update the expert knowledge base using open-source security intelligence; the expert knowledge data includes expert status action pairs of security experts in successful penetration testing. The imitation learning training module is used to set up asynchronous multi-environment environments, enabling pre-trained agents to perform parallel generative imitation learning in multiple sandbox environments. The adversarial training module is used to generate predicted state-action pairs online from the current observed state of the agent during training. The predicted state-action pairs are used to train the discriminator in the enhanced GAIL network. The discriminator dynamically adjusts the state through adaptive learning rate and multilayer perception mechanism to obtain high-quality state-action pairs. The enhanced GAIL network includes a policy network, a value network, and a discriminator. The evaluation and adjustment module is used to calculate the discounted reward for high-quality state-action pairs, and uses the discounted reward as feedback to optimize the policy network; it evaluates the policy network using security events and provides real-time security feedback to the policy network; after multiple rounds of training, the training of the policy network is completed. The penetration testing module is used to containerize penetration testing tools and orchestrate them using Kubernetes. It uses high-quality state-action pairs to perform penetration testing on the trained policy network, repeats experiments on specific target machines, feeds back the experimental results and adjusts the policy network, continuously evaluates agent performance, records and feeds back the parameters for adjusting and enhancing the GAIL network, and generates monitoring logs.

Images