A distributed multi-agent network penetration method, device and storage medium

Abstract

This invention belongs to the technical fields of reinforcement learning and penetration testing. It discloses a distributed multi-agent network penetration method, device, and storage medium. Hosts on the same network segment but with different feature information are divided into different subnets. In a distributed causal reinforcement learning-based attack path discovery task, an agent is placed in each subnet domain to perform node scanning. The scanned information is input into a temporal convolutional causal model, which outputs spatiotemporal feature information representing vulnerability types. Network state information is aggregated to obtain a host state sequence. This host state sequence is input to each agent, causing the agent to perform a series of actions based on the host state sequence to acquire and share each agent's penetration experience. The method iterates until the causal reinforcement learning model converges, resulting in a trained causal reinforcement learning model. Advantages: This method effectively alleviates issues such as agent state and action space explosion, and accelerates training efficiency.

Description

Technical Field

[0001] The present invention relates to a distributed multi-agent network penetration method, device and storage medium, and belongs to the technical field of reinforcement learning and penetration testing. Background Art

[0002] In recent years, artificial intelligence has achieved remarkable success in many fields, gradually moving from perceptual intelligence to decision-making intelligence. Reinforcement learning is a key path to achieving decision-making intelligence. For example, AlphaGo's outstanding performance in the game of Go has sparked a surge in reinforcement learning research. The real world often involves interactions between multiple agents, including cooperative and competitive situations. Examples include robots assembling soccer balls, multi-vehicle avoidance in autonomous driving, and the various complex scenarios found in games, which are also driving the development of multi-agent reinforcement learning.

[0003] However, traditional data-driven reinforcement learning algorithms rely heavily on training data and are easily affected by environmental changes. Furthermore, they are unable to explore the essential relationships between variables or the associations between penetration targets, resulting in low learning efficiency and accuracy in the field of automated penetration testing.

[0004] Penetration testing based on traditional reinforcement learning is easily affected by environmental changes, and it is unable to explore the essential relationship between variables, resulting in low penetration efficiency. Moreover, when conducting large-scale network penetration testing, its attack actions take a long time to execute. The low penetration efficiency is also prone to the problem of exponential explosive growth in the state space. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a distributed multi-agent network penetration method.

[0006] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0007] In one aspect, the present invention discloses a distributed multi-agent network penetration method, comprising:

[0008] Based on the different characteristic information of each network in the target large network, hosts on the same network segment but with different characteristic information are divided into different subnets;

[0009] After subnet division, subnet domains are divided into different sub-areas. In the attack path discovery task based on distributed causal reinforcement learning, an agent is placed in each subnet domain to perform node scanning. Each agent includes a pre-trained temporal convolutional causal model. The scanned information is input into the pre-trained temporal convolutional causal model, and the spatiotemporal feature information representing the vulnerability type is output.

[0010] Aggregate the spatiotemporal feature information of vulnerability types of hosts scanned by each agent and the network status information of different subnets to obtain the host status sequence;

[0011] A causal reinforcement learning model is pre-built based on the interaction process between multiple agents and the network environment. The host state sequence is input to each agent so that the agent performs a series of actions based on the host state sequence to obtain and share the penetration experience of each agent. It is iterated until the causal reinforcement learning model converges to obtain a trained causal reinforcement learning model and complete the penetration of sensitive hosts.

[0012] Furthermore, the method of dividing hosts on the same network segment but with different characteristic information into different subnets based on different characteristic information of each network in the target large network includes:

[0013] The network status information of the target large network is collected based on the two-bit random walk method. The network status information is sequentially cleaned, factorized, and normalized, and then sequentially spliced ​​to obtain the feature vector of each host.

[0014] Randomly select k hosts as the central hosts of each sub-network, calculate the feature similarity between the remaining hosts and the central host of each sub-network based on the host's feature vector, and assign the remaining hosts to the sub-network to which the most similar sub-network central host belongs, where k is a parameter.

[0015] Furthermore, the similarity between each host is calculated as follows:

[0016] ;

[0017] Where, d ed 、 cosθ as well as D ( A , B ) are the Euclidean distance, cosine distance and Manhattan distance between each node respectively;

[0018] The calculation formulas for Euclidean distance, cosine similarity, and Manhattan distance are as follows:

[0019] ;

[0020] ;

[0021] ;

[0022] Where, a i and b i Represent the feature vectors of the two hosts respectively,n Represents the feature dimension.

[0023] Furthermore, the temporal convolutional causal model includes:

[0024] A temporal structure for extracting temporal features, a convolutional structure for extracting spatial features, and a fully connected network for merging temporal and spatial features to obtain spatiotemporal feature information.

[0025] Furthermore, the training process of the temporal convolutional causal model includes:

[0026] 1) Input the host feature information at different times in the target time period into the time series structure, and train the time series structure and the fully connected network together;

[0027] 2) The average features and comprehensive distance values ​​of adjacent hosts in the target time period are input into the convolutional structure, and the convolutional structure and fully connected network are trained together;

[0028] 3) Simultaneously train the temporal structure, convolutional structure, and fully connected structure to extract spatiotemporal feature information;

[0029] 4) Repeat steps 1) to 3) until the loss function L Convergence, complete alternating training, and obtain the trained temporal convolutional causal model;

[0030] The loss function L The expression is as follows:

[0031] ;

[0032] in, M Indicates the number of vulnerability categories; y ic Represents a sign function, if the sample i The true category is equal to c Then take 1, otherwise take 0; p ic Represents an observation sample i Belong to category c probability; N Replay memory D Initialization capacity.

[0033] Furthermore, the causal reinforcement learning model is expressed as:

[0034] ;

[0035] in, r represents the immediate reward value, γ represents the discount factor, Indicates the next state The maximum expected return under Indicates the action to be performed at the next moment. θ Represents model parameters.

[0036] Furthermore, the training of the causal reinforcement learning model includes:

[0037] Inputting the spatiotemporal feature information into an autoencoder and performing noise filtering to obtain purified spatiotemporal feature information;

[0038] The initial capacity is N Playback memory D , and the action-value function with random weights Q , and then execute the round iteration of the outer loop; in each round of the round iteration of the outer loop, first initialize the sequence s 1={ x 1}, where x 1 represents the initial network status information, vulnerability type, and purified spatiotemporal status information, and preprocesses it into a fixed length , the outer loop means looping according to the set number of rounds, represents the preprocessed sequence;

[0039] Execute the time step iteration of the inner loop, which means looping according to the set number of time steps; in each step of the time step iteration of the inner loop, the current time step is first selected based on the epsilon-Greedy algorithm. t Random action or current best action a t , and then execute it in the simulator a t Observation Rewards r and the Internet x t+1 ;set up s t+1 = s t , a t , x t+1 , s t express t The state of the moment, x t+1 express t +1 time network status information, vulnerability type and purified spatiotemporal status information, and perform preprocessing , s t+1 represents the state sequence at time t+1, express t+1 time step after preprocessing of the state sequence; transfer the state obtained in the current time step Store to playback memory D Among them r t express t The rewards of the moment, express t A state sequence that has been preprocessed at all times; based on replay memory D Randomly sample mini-batches of state transitions ,in a j express j The action of the moment, r j express j The rewards of the moment, express j +1 time after preprocessing state sequence, express j The state sequence after preprocessing at each moment; according to Is it in the termination state? Set y i for:

[0040] ;

[0041] in, Q ( ) represents a neural network function approximator;

[0042] With loss function Perform gradient descent, update network parameters, and adjust the model's structural parameters through forward and backward propagation of gradients until the loss function value converges to obtain a trained causal reinforcement learning model;

[0043] described Expressed as:

[0044] ;

[0045] Denotes the original loss, It's about sequence s and actions a The probability distribution of θ i represents the model parameters, MSE is the original data from the input layer of the autoencoder to the hidden layer of the autoencoder X And the decoding result X R The distance metric function is the mean square error.

[0046] Furthermore, the processing of the autoencoder includes:

[0047] The raw data from the input layer of the autoencoder to the hidden layer of the autoencoder X The encoding process is:

[0048] ;

[0049] in, h Indicates the encoding result, Represents the encoding process, σ is the parameter value, W 1 represents the autoencoder parameter, b 1 represents the bias of the autoencoder;

[0050] The decoding process from the hidden layer of the autoencoder to the output layer of the autoencoder is:

[0051] ;

[0052] in, X R Indicates the decoding result, Represents the decoding process, W 2 represents the decoder parameters, b 2 represents the decoder bias.

[0053] In a second aspect, the present invention provides a distributed multi-agent network penetration device, comprising:

[0054] A partitioning module is used to divide hosts on the same network segment but with different characteristic information into different subnets based on the different characteristic information of each network in the target large network;

[0055] The first model processing module is used to obtain subnet domains divided into different sub-areas after subnet division. In the attack path discovery task based on distributed causal reinforcement learning, an intelligent agent is placed in each subnet domain to perform node scanning. Each intelligent agent includes a pre-trained temporal convolutional causal model. The scanned information is input into the pre-trained temporal convolutional causal model and the spatiotemporal feature information representing the vulnerability type is output;

[0056] Aggregation module, used to aggregate the spatiotemporal feature information of vulnerability types of hosts scanned by each agent and the network status information of different subnets to obtain the host status sequence;

[0057] The second model processing module is used to obtain a causal reinforcement learning model pre-constructed based on the interaction process between multiple agents and the network environment, input the host state sequence into each agent, so that the agent performs a series of actions based on the host state sequence to obtain and share the penetration experience of each agent, iterate until the causal reinforcement learning model converges, obtain a trained causal reinforcement learning model, and complete the penetration of sensitive hosts.

[0058] In a third aspect, the present invention provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of one aspect.

[0059] The beneficial effects achieved by the present invention are:

[0060] 1) Penetration testing based on traditional reinforcement learning is easily affected by environmental changes, and it is unable to explore the essential relationship between variables, resulting in low penetration efficiency. This solution designs a method based on causal reinforcement learning, which uses the causal reinforcement learning algorithm model to train the intelligent agent to automatically generate attack paths according to the penetration environment, thereby achieving the purpose of intelligent penetration. 2) When conducting large-scale network penetration tests, the attack actions take a long time to execute, the penetration efficiency is low, and it is easy to encounter the problem of exponential explosive growth in the state space. In order to improve the penetration efficiency, the random walk method is used to divide the large network into subnets, and then multiple agents are used for scanning and penetration. The scanning penetration test problem is decomposed into subtasks, and the target sensitive host is found more quickly and the vulnerability information is scanned, so that the problems of intelligent agent state and action space explosion are effectively alleviated, thereby accelerating training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is the overall flow chart of the present invention;

[0062] Figure 2 It is a structural diagram of causal reinforcement learning;

[0063] Figure 3 It is a simulation scene graph. DETAILED DESCRIPTION

[0064] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0065] Example 1. This example introduces a distributed multi-agent network penetration method. Step 1 is subnet division based on random walks. It is used to segment the target large network. Based on the different characteristic information of the network, the clustering method is applied to divide the hosts on the same network segment into different subnets. Subnet division can reduce the network scale and the number of hosts, and explore the internal connections between subnets, thereby reducing the state space that each agent needs to process. Compared with directly processing large networks, after subnet division, agents in different subnet domains can share the experience learned in their respective subnet domains with agents in other areas, thereby accelerating the penetration process.

[0066] Step 2: Subnet scanning ports based on a time-series convolutional causal model. After subnetting, each agent plans attacks to penetrate hosts within its own subnet domain. Causal reinforcement learning determines the scope of the penetration test target through observation and scanning actions by agents in different subnet domains. Intelligence collection uncovers security vulnerabilities and weaknesses in the penetration target, laying the foundation for future penetration attacks. The scanning operation selects the next target host to attack based on the current state of the controlled host, allowing the agent to quickly penetrate all sensitive hosts and complete the path planning task.

[0067] Step 3: Host penetration based on causal reinforcement learning. Agents aggregate the state information of scanned hosts and execute a series of attack actions against sensitive hosts. To enable multi-agent collaboration and faster execution, this step also uses causal reinforcement learning. Each agent is fed the state information of the scanned subnet containing sensitive hosts, and after executing a series of actions, each agent shares its penetration experience to select the fastest penetration path and method to complete the penetration of the sensitive host.

[0068] The above steps are iterated until the causal reinforcement learning algorithm model converges, completing the training of the distributed multi-agent penetration testing model based on causal reinforcement learning. After training is complete, we will gain control of the sensitive host.

[0069] refer to Figure 1 and Figure 2 , the method comprises the following steps:

[0070] S1. Subnetting based on random walk.

[0071] S11. Network status information collection and preprocessing. First, various information about the target network is collected based on the two-bit random walk method, including:

[0072] Address information: server IP address, DNS domain name, open port number and corresponding service process, etc.

[0073] System information: operating system type and version, web server software type and version, web application and version, web application development tools and version, web application architecture, database management system type and version.

[0074] Account information: This includes operating system login accounts, database management system accounts, application system management accounts, etc. Configuration information: This includes network topology, address mapping table (when the web server is located in an internal LAN and uses private IP addresses), service configuration information, shared resources, firewall type and configuration information, identity authentication and access control methods, encryption and password management mechanisms.

[0075] Other information: security vulnerabilities (software vulnerabilities and management vulnerabilities), DNS registration information, network administrator contact information.

[0076] The collected data are preprocessed, including data cleaning (filling empty values, merging similar data, etc.), feature factorization, data standardization, etc., and then all the information features of each host are spliced ​​in order to obtain their respective feature vectors. ,in i Indicates the host number. t Indicates the sampling time.

[0077] S12. Similarity measurement: First, the Euclidean distance, cosine distance, and Manhattan distance between each node are calculated based on the node feature vector obtained in step S11, and then the three distances are multiplied together to obtain the result to measure the similarity of each node vector.

[0078] The calculation formula of Euclidean distance is as follows:

[0079] (1);

[0080] The formula for calculating cosine similarity is as follows:

[0081] (2);

[0082] The formula for calculating Manhattan distance is as follows:

[0083] (3);

[0084] Then multiply the three distance functions to calculate the similarity between each node vector:

[0085] (4);

[0086] Random selection kThe host is used as the central host of each sub-network. The feature similarity between the remaining hosts and the central host of each sub-network is calculated based on the host's feature vector, and the remaining hosts are assigned to the sub-network to which the most similar sub-network central host belongs. k As a parameter, it is determined according to the actual number of hosts and its own needs, and each intelligent agent is set to scan and penetrate, thereby speeding up the penetration efficiency.

[0087] S2. After subnetting the target network according to the method in step S1, subnet domains divided into different areas are obtained. In the attack path discovery task based on distributed causal reinforcement learning, an intelligent agent is placed in each subnet domain to perform port scanning, find sensitive hosts in the target network, and plan attacks to penetrate between hosts.

[0088] S21. Build a temporal convolution causal model. The input of the model includes the temporal features of the nodes collected at different times, the average features of the adjacent nodes in the time period, and the comprehensive distance value between the node and the adjacent nodes; the output of the model is the vulnerability type after scanning, such as ftp, ssh, etc. The structure of the model includes a temporal structure and a convolutional structure. The temporal structure extracts temporal features (i.e., node features at different times), and the convolutional structure extracts spatial features (i.e., the average features and comprehensive distance values ​​of the adjacent nodes in the time period). The temporal structure is implemented using the LSTM network and the convolutional structure is implemented using a general convolutional network. The LSTM and CNN are finally connected in parallel through full connection to realize the causal reasoning of temporal convolution. First, each intelligent agent collects the open information of the subnet hosts collected at different times to form a temporal information matrix, and then inputs this temporal information matrix into the LSTM long short-term memory network to extract the temporal information:

[0089] (5);

[0090] After extracting the time series information through the LSTM network, the time series information matrix is ​​input into the CNN convolutional network to extract spatial features:

[0091] (6);

[0092] Finally, the temporal feature information and spatial feature information extracted by the LSTM network and the CNN network are input into the fully connected network to merge and extract the spatiotemporal feature information as the input of the reinforcement learning penetration testing model.

[0093] S22. Train the temporal convolutional causal model. Achieve training convergence by alternating between temporal and convolutional structures. Use cross entropy (categorical_crossentropy) as the loss function for causal feature extraction network optimization, as expressed as follows:

[0094] ;

[0095] in, M Indicates the number of categories. This patent includes 11 vulnerabilities, including vnc, telent, ssh, rpc, proftdp, phb,mysql, irc, http, apache and prostgresql. y ic is a sign function (0 or 1), if the sample i The true category is equal to c If yes, it takes 1, otherwise it takes 0. p ic Represents an observation sample i Belong to category c probability; N Replay memory D Initialization capacity.

[0096] The training steps of the causal feature extraction network specifically include:

[0097] 1) First, the node features at different times are input into the time series structure, and the LSTM long short-term memory network and the fully connected neural network are trained together;

[0098] 2) Then the average features and comprehensive distance values ​​of adjacent nodes in the time period are input into the convolution structure to train the convolutional neural network and the fully connected neural network together;

[0099] 3) Simultaneously train the temporal structure, convolutional structure, and fully connected structure to extract spatiotemporal feature information;

[0100] 4) Repeat steps 1) to 3) until the loss function L Convergence, complete alternating training, and achieve comprehensive learning of temporal causality and spatial causality.

[0101] In this paper, we assume that future rewards decay over time steps, and the decay factor is γ , then in t The sum of rewards in the future is:

[0102] (7);

[0103] in, R t express t The total reward value at the moment, T Indicates the total time, express The reward value at that moment.

[0104] Optimal action-value function To observe a sequence s and perform actions a After that, the maximum expected reward that can be achieved through any subsequent strategy is:

[0105] (9);

[0106] in, and Respectively represent the state and action at the next moment, r Represents the reward value, and the entropy form indicates that the target expectation consists of the maximum expectation of the immediate reward and the discounted reward at the next time step. A function approximator is then used to estimate the action-value function:

[0107] (10);

[0108] The present invention uses a weight θ The neural network function approximator is called Q-network. i Minimize the loss function To train:

[0109] (11);

[0110] in, For the current iteration i The goal, It is a sequence s and actions a The probability distribution of , that is, the behavior distribution. The parameters from the previous iteration In optimizing the loss function , which remains unchanged and is used to calculate the optimal value function for the current iteration. The gradient of the loss function with respect to the weights is obtained as follows:

[0111] (12);

[0112] Compared to directly calculating the expectation in the gradient above, for computational efficiency we can optimize the loss function through stochastic gradient descent. Each time, a single sample is sampled based on the behavior distribution and the simulator as the expectation, which is used to update the weights.

[0113] S32. Train a causal reinforcement learning model to achieve rapid penetration of the target network.

[0114] like Figure 2 and Figure 3 As shown in the figure, the spatiotemporal feature information of the fully connected layer is first input into the autoencoder to remove noise and purify information. The encoding process of the original data from the autoencoder input layer to the hidden layer is as follows:

[0115] (13);

[0116] in, h Indicates the encoding result, Represents the encoding process, σ is the parameter value, W 1 represents the autoencoder parameter, b 1 represents the bias of the autoencoder;

[0117] Decoding process from hidden layer to output layer:

[0118] (14);

[0119] in, X R Indicates the decoding result, Represents the decoding process, W 2 represents the decoder parameters, b 2 represents the decoder bias.

[0120] Then the optimization objective function of the algorithm is written as:

[0121] (15);

[0122] Where is the mean square error of the distance metric function between the two.

[0123] Then initialize the capacity to N Playback memory D , and the action-value function with random weights Q ; Then perform round iteration (outer loop, total M rounds), in each round, first initialize the sequence ,in, Represents the purified network status, spatiotemporal status information, and vulnerability type, and preprocesses them into fixed-length ; Then execute the time step iteration (inner loop, total T Step), in each step, first select the action based on the epsilon-Greedy algorithm (random action or current best action), and then execute it in the simulator Observation Rewards r and the Internet ;set up And perform preprocessing , x t+1 express t +1 time point network status information, vulnerability type and purified spatiotemporal status information; transfer the current time step Save toD in; based on D Randomly sampled mini-batches of transfers ;according to Is it in the termination state? Set for:

[0124] (16);

[0125] in, Q ( ) represents a neural network function approximator.

[0126] Finally, a regularization term is added to the loss function of formula (11) to The loss function performs gradient descent to update the network parameters. The model's structural parameters are adjusted through forward and backward propagation of the gradient until the loss function converges, ultimately successfully penetrating sensitive hosts in the target network and obtaining a controlled state.

[0127] Based on this, the present invention applies causal reinforcement learning and clustering methods to penetration testing to improve the effect of penetration testing.

[0128] Causal reinforcement learning (CRL) is a learning approach that extends reinforcement learning algorithms to consider causal structures. While traditional reinforcement learning focuses on the correlation between actions and rewards, CRL takes a causal approach and considers the impact of actions on future states. This allows for better handling of environmental disturbances, thereby improving learning efficiency and accuracy. In CRL, an agent attempts to maximize future rewards by continuously experimenting with different behaviors. The agent typically perceives one or more states in the environment and the rewards it will receive after performing a certain action. In traditional CRL, the agent focuses solely on the correlation between actions and rewards, without considering causal relationships. However, in CRL, actions may have a causal impact on the next state. For example, one action may lead to the occurrence of a next state, which in turn leads to changes in the subsequent reward. Therefore, CRL algorithms consider the causal relationships between actions, states, and rewards. CRL focuses not only on the joint distribution between actions and rewards, but also on the causal impact of actions on future states and rewards, modeling the impact of actions as a causal intervention.

[0129] Fundamentally, causal reinforcement learning can better understand the causal structure of a problem through analysis and modeling of causal relationships, enabling more efficient learning of optimal strategies and broad application prospects. Applying clustering methods during the penetration process can explore the relationships between different penetration targets, dividing them into different categories based on their attributes for distributed penetration, and improving penetration efficiency and accuracy.

[0130] In summary, the distributed multi-agent penetration testing method based on causal reinforcement learning proposed in this patent can improve efficiency and success rate, focus on the intrinsic connection of penetration targets, consider the causal effect structure, not only focus on the joint distribution between behavior and rewards, but also focus on the causal impact of behavior on future states and rewards, and model the impact of behavior as a causal intervention, thereby completing the penetration of vulnerabilities, which has extremely important practical significance for improving penetration efficiency and accuracy.

[0131] Example 2. This example provides a distributed network penetration method based on causal reinforcement learning and its application in different networks, specifically including a target network model and a penetration environment. The network environment includes different configurations for the host, specifically including subnets, network topology, firewalls, routers, etc.; the penetration environment includes the settings of the target penetration network and the interaction between multiple agents and the network environment. The process of interaction with the environment is modeled as a Markov decision process based on distributed causal reinforcement learning of the DQN algorithm, thereby realizing the mutual integration of the network and the penetration environment, and achieving the purpose of distributed penetration in networks of different scales through this method.

[0132] Example 3, based on the same inventive concept as Example 1, introduces a distributed multi-agent network penetration device, including:

[0133] A partitioning module is used to divide hosts on the same network segment but with different characteristic information into different subnets based on the different characteristic information of each network in the target large network;

[0134] The first model processing module is used to obtain subnet domains divided into different sub-areas after subnet division. In the attack path discovery task based on distributed causal reinforcement learning, an intelligent agent is placed in each subnet domain to perform node scanning. Each intelligent agent includes a pre-trained temporal convolutional causal model. The scanned information is input into the pre-trained temporal convolutional causal model and the spatiotemporal feature information representing the vulnerability type is output;

[0135] Aggregation module, used to aggregate the spatiotemporal feature information of vulnerability types of hosts scanned by each agent and the network status information of different subnets to obtain the host status sequence;

[0136] The second model processing module is used to obtain a causal reinforcement learning model pre-constructed based on the interaction process between multiple agents and the network environment, input the host state sequence into each agent, so that the agent performs a series of actions based on the host state sequence to obtain and share the penetration experience of each agent, iterate until the causal reinforcement learning model converges, obtain a trained causal reinforcement learning model, and complete the penetration of sensitive hosts.

[0137] Based on the same inventive concept as Example 1, this embodiment introduces a computer-readable storage medium storing one or more programs, characterized in that the one or more programs include instructions, which, when executed by a computing device, enable the computing device to execute the method described in Example 1.

[0138] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0142] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A distributed multi-agent network penetration method, characterized in that: include: Based on the different characteristic information of each network in the target large network, hosts on the same network segment but with different characteristic information are divided into different subnets; After subnet division, subnet domains are divided into different sub-areas. In the attack path discovery task based on distributed causal reinforcement learning, an agent is placed in each subnet domain to perform node scanning. Each agent includes a pre-trained temporal convolutional causal model. The scanned information is input into the pre-trained temporal convolutional causal model, and the spatiotemporal feature information representing the vulnerability type is output. Aggregate the spatiotemporal feature information of vulnerability types of hosts scanned by each agent and the network status information of different subnets to obtain the host status sequence; A causal reinforcement learning model is pre-built based on the interaction process between multiple agents and the network environment. The host state sequence is input to each agent so that the agent performs a series of actions based on the host state sequence to obtain and share the penetration experience of each agent. It is iterated until the causal reinforcement learning model converges to obtain a trained causal reinforcement learning model and complete the penetration of sensitive hosts.

2. The distributed multi-agent network penetration method according to claim 1, characterized in that: The method of dividing hosts on the same network segment but with different characteristic information into different subnets based on the different characteristic information of each network in the target large network includes: The network status information of the target large network is collected based on the two-bit random walk method. The network status information is sequentially cleaned, factorized, and normalized, and then sequentially spliced ​​to obtain the feature vector of each host. Randomly select k hosts as the central hosts of each sub-network, calculate the feature similarity between the remaining hosts and the central host of each sub-network based on the host's feature vector, and assign the remaining hosts to the sub-network to which the most similar sub-network central host belongs, where k is a parameter.

3. The distributed multi-agent network penetration method according to claim 2, characterized in that: The calculation formula for the similarity between each host is: ； Where, d ed 、 cosθ as well as D ( A , B ) are the Euclidean distance, cosine distance and Manhattan distance between each node respectively; The calculation formulas for Euclidean distance, cosine similarity, and Manhattan distance are as follows: ； ； ； Where, a i and b i Represents the first i dimensional feature vector, n Represents the feature dimension.

4. The distributed multi-agent network penetration method according to claim 1, characterized in that: The temporal convolutional causal model includes: A temporal structure for extracting temporal features, a convolutional structure for extracting spatial features, and a fully connected network for merging temporal and spatial features to obtain spatiotemporal feature information.

5. The distributed multi-agent network penetration method according to claim 4, characterized in that: The training process of the temporal convolutional causal model includes: 1) Input the host feature information at different times in the target time period into the time series structure, and train the time series structure and the fully connected network together; 2) The average features and comprehensive distance values ​​of adjacent hosts in the target time period are input into the convolutional structure, and the convolutional structure and fully connected network are trained together; 3) Simultaneously train the temporal structure, convolutional structure, and fully connected structure to extract spatiotemporal feature information; 4) Repeat steps 1) to 3) until the loss function L Convergence, complete alternating training, and obtain the trained temporal convolutional causal model; The loss function L The expression is as follows: ； in, M Indicates the number of vulnerability categories; y ic Represents a sign function, if the sample i The true category is equal to c Then take 1, otherwise take 0; p ic Represents an observation sample i Belong to category c probability; N Replay memory D Initialization capacity.

6. The distributed multi-agent network penetration method according to claim 5, characterized in that: The causal reinforcement learning model is expressed as: ； in, r represents the immediate reward value, γ represents the discount factor, Indicates the next state The maximum expected return under Indicates the action to be performed at the next moment. θ Represents model parameters.

7. The distributed multi-agent network penetration method according to claim 6, characterized in that: The training of the causal reinforcement learning model includes: Inputting the spatiotemporal feature information into an autoencoder and performing noise filtering to obtain purified spatiotemporal feature information; The initial capacity is N Playback memory D , and the action-value function with random weights Q , and then execute the round iteration of the outer loop; in each round of the round iteration of the outer loop, first initialize the sequence s 1={ x 1}, where x 1 represents the initial network status information, vulnerability type, and purified spatiotemporal status information, and preprocesses it into a fixed length , the outer loop means looping according to the set number of rounds, represents the preprocessed sequence; Execute the time step iteration of the inner loop, which means looping according to the set number of time steps; in each step of the time step iteration of the inner loop, the current time step is first selected based on the epsilon-Greedy algorithm. t Random action or current best action a t , and then execute it in the simulator a t Observation Rewards r and the Internet x t+1 ;set up s t+1 = s t , a t , x t+1 , s t express t The state of the moment, x t+1 express t +1 time network status information, vulnerability type and purified spatiotemporal status information, and perform preprocessing , s t+1 represents the state sequence at time t+1, express t +1 time step after preprocessing of the state sequence; transfer the state obtained in the current time step Store to playback memory D Among them r t express t The rewards of the moment, express t A state sequence that has been preprocessed at all times; based on replay memory D Randomly sample mini-batches of state transitions ,in a j express j The action of the moment, r j express j The rewards of the moment, express j +1 time after preprocessing state sequence, express j The state sequence after preprocessing at each moment; according to Is it in the termination state? Set y i for: ； in, Q ( ) represents a neural network function approximator; With loss function Perform gradient descent, update network parameters, and adjust the model's structural parameters through forward and backward propagation of gradients until the loss function value converges to obtain a trained causal reinforcement learning model; described Expressed as: ； Denotes the original loss, It's about sequence s and actions a The probability distribution of θ i represents the model parameters, MSE is the original data from the input layer of the autoencoder to the hidden layer of the autoencoder X And the decoding result X R The distance metric function is the mean square error.

8. The distributed multi-agent network penetration method according to claim 7, characterized in that: The processing process of the autoencoder includes: The raw data from the input layer of the autoencoder to the hidden layer of the autoencoder X The encoding process is: ； in, h Indicates the encoding result, Represents the encoding process, σ is the parameter value, W 1 represents the autoencoder parameter, b 1 represents the bias of the autoencoder; The decoding process from the hidden layer of the autoencoder to the output layer of the autoencoder is: ； in, X R Indicates the decoding result, Represents the decoding process, W 2 represents the decoder parameters, b 2 represents the decoder bias.

9. A distributed multi-agent network penetration device, characterized in that: include: A partitioning module is used to divide hosts on the same network segment but with different characteristic information into different subnets based on the different characteristic information of each network in the target large network; The first model processing module is used to obtain subnet domains divided into different sub-areas after subnet division. In the attack path discovery task based on distributed causal reinforcement learning, an intelligent agent is placed in each subnet domain to perform node scanning. Each intelligent agent includes a pre-trained temporal convolutional causal model. The scanned information is input into the pre-trained temporal convolutional causal model and the spatiotemporal feature information representing the vulnerability type is output; Aggregation module, used to aggregate the spatiotemporal feature information of vulnerability types of hosts scanned by each agent and the network status information of different subnets to obtain the host status sequence; The second model processing module is used to obtain a causal reinforcement learning model pre-constructed based on the interaction process between multiple agents and the network environment, input the host state sequence into each agent, so that the agent performs a series of actions based on the host state sequence to obtain and share the penetration experience of each agent, iterate until the causal reinforcement learning model converges, obtain a trained causal reinforcement learning model, and complete the penetration of sensitive hosts.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 8 .

Images