5G-IoT network optimal path penetration method based on multi-agent reinforcement learning

Abstract

The invention discloses a 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning. The method comprises the following steps: constructing an attack graph oriented to a 5G-IoT network scene; and then planning an optimal penetration path in the attack graph. Each state node in the attack graph represents equipment in the network; directed edges among the state nodes represent state transition among the nodes, and corresponding vulnerability scores are marked on the directed edges; the optimal penetration path is a path with the maximum attack benefit between a starting node and a target node in the attack graph; the measurement index of the attack benefit comprises a vulnerability score and attack duration. Planning a penetration path by adopting a collaborative penetration model based on a multi-agent depth deterministic strategy gradient algorithm; the collaborative penetration model adopts a centralized training-decentralized decision-making MARL framework as a basic model; the input of the collaborative penetration model comprises current network environment parameters and attack graph information, and the output of the collaborative penetration model is an optimal task scheduling strategy in the current network environment, namely an optimal penetration path.

Description

Technical Field

[0001] The present invention relates to the application of artificial intelligence technology in the field of network security, and specifically to a 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning. Background Art

[0002] In addition to providing users with higher bandwidth, lower latency, and more connections, 5G networks also cover a large number of machine-type terminal devices (i.e., IoT devices). The resulting 5G IoT network is becoming increasingly complex and open, especially as 5G IoT devices gradually penetrate critical infrastructure, from smart homes to smart cities. Any security vulnerability may have serious economic and social impacts. Penetration testing can effectively predict, prevent, and respond to potential threats, ensure the security and reliability of 5G IoT networks, and ensure the steady development of the smart society. Considering the increasingly diverse means and forms of attackers using network security vulnerabilities to carry out malicious attacks [1], it is necessary and urgent to study 5G IoT network penetration technology.

[0003] As a technical means of analyzing attacker behavior, the penetration process can be divided into direct and indirect modes. The former refers to the process of launching an attack directly from one host and successfully penetrating another host. The latter refers to the attacker first obtaining and controlling the springboard host, and then using it as a secure channel or starting point to further penetrate the target host to conceal the real source and identity of the attack, thereby increasing the complexity and difficulty of the attack [2]. Based on the understanding of the penetration principle, network administrators can simulate real network attack and defense scenarios, discover system vulnerabilities or penetration paths that may be exploited by attackers, and thus design targeted defense strategies [3][4][5]. However, the uncertainty of attacker penetration behavior, coupled with the constant changes in network situation and the objective evaluation of the difficulty of host vulnerability exploitation, increase the difficulty of constructing penetration paths [6][7]. At the same time, the establishment and improvement of the network security defense system also directly affects the judgment of whether the penetration behavior is objective and efficient. Therefore, there are severe challenges in the reasonable and scientific design of the penetration path generation system.

[0004] Traditional penetration path planning methods consider the strategy exploration problem of a single agent in a simple environment. However, real network systems often encounter complex penetration attacks jointly launched by multiple attackers. Due to the limited perception range of a single agent, it is difficult to fully detect such complex collaborative attacks. To this end, it is necessary to analyze and consider the attack behavior patterns of multiple attackers (including: target selection, attack methods, path planning, etc.) in order to develop more comprehensive and targeted security protection measures. Despite this, the design of multi-attacker collaborative penetration solutions for 5G IoT networks faces the following technical difficulties:

[0005] (1) Collaborative environmental perception: Modern network environments are complex and include a variety of customized network protocols and services, different operating systems, configuration parameters, and time-varying network topologies[8][9]. A system is usually composed of multiple heterogeneous agents. Different agents have different perception capabilities, computing resources, and communication capabilities. How to effectively integrate the environmental information collected by all parties and overcome the negative impact of heterogeneity has become a major challenge

[10] .

[0006] (2) Information sharing: Multi-agent systems need to quickly share and update information to maintain collaboration, especially in adversarial or competitive environments, where the timeliness of information is crucial

[11] . Agents distributed in different locations can only obtain local environmental information. In order to obtain globally consistent environmental perception, agents need to share and interact with each other about environmental information and minimize communication delays and packet loss

[12] . However, when there are many agents, the real-time and reliability of information exchange and collaborative decision-making are inevitably limited due to resource bottlenecks

[13] .

[0007] (3) Coordination and consistency: Agents involved in collaborative infiltration have different capabilities and characteristics in terms of mobility, concealment, and aggressiveness

[14] . Some agents may focus on minimizing the infiltration time, while others may focus on maximizing the infiltration success rate. It can be seen that how to reasonably allocate infiltration tasks and achieve consistency in group decision-making needs further exploration

[15] . Summary of the invention

[0008] In view of the above limitations and challenges, this paper proposes a collaborative penetration method based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG), which is a multi-party collaborative penetration scenario for 5G-IoT scenarios. Different from traditional single-agent path optimization, this method improves the overall penetration path quality through multi-agent collaboration.

[0009] The 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning of the present invention comprises the following steps: step 1, constructing an attack graph for the 5G-IoT network scenario; step 2, planning the optimal penetration path in the attack graph;

[0010] In step 1, the attack graph is obtained based on the network topology and the vulnerability information of the devices in the network;

[0011] In step 2, each state node in the attack graph represents a device in the network; the directed edges between the state nodes represent the state transfer between the nodes, and the corresponding vulnerability scores are marked on the edges;

[0012] The optimal penetration path is the path between the starting node and the target node in the attack graph that has the greatest attack benefit; the attack benefit measurement indicators include vulnerability score and attack duration;

[0013] In step 1, there are multiple attackers;

[0014] In step 2, a collaborative penetration model based on the multi-agent deep deterministic policy gradient MADDPG algorithm is used to plan the penetration path;

[0015] The collaborative penetration model uses the MARL architecture of centralized training and decentralized decision-making as the basic model; the input of the collaborative penetration model includes the current network environment parameters and attack graph information, and the output of the model is the optimal task scheduling strategy π under the current network environment. * , i.e., the optimal permeation path;

[0016] The main contributions of the present invention are as follows:

[0017] 1. A multi-agent collaboration method integrating Contract Net protocol is developed to solve the task allocation problem in penetration testing scenarios. Agents bid for tasks based on their own capabilities, and the central controller assigns tasks to the best agents. During task execution, agents can share local penetration status and attack behavior information in real time to improve collaboration efficiency.

[0018] 2. In order to break through the limitation of single agent observation, the experience pool priority replay algorithm is introduced. A global experience pool is constructed to store the interactive experience trajectories of all agents. Different priorities are assigned to each sample according to the quality and rarity of the experience. High-quality and rare samples are given priority during training to accelerate strategy convergence and improve the stability of path generation.

[0019] 3. Multi-agent infiltration path planning experiments and ablation experiments under different relationship settings are designed. The results show that the proposed scheme is superior to the mainstream deep reinforcement learning benchmark method in terms of environmental adaptability, infiltration efficiency, decision stability, task completion rate and other indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A typical multi-party collaborative penetration scenario network topology diagram for 5G-IoT scenarios;

[0021] Figure 2 An attack graph corresponding to a local scenario of an embodiment is shown;

[0022] Figure 3 A diagram showing the centralized training-decentralized execution architecture;

[0023] Figure 4 Represents the MADDPG training and execution process;

[0024] Figure 5 Represents the CNP-based task allocation process;

[0025] Figure 6(a) and Figure 6(b) show the reward values ​​and probability density distribution of the four methods respectively, where:

[0026] Figure 6(a) shows the reward values ​​of different methods, and Figure 6(b) shows the probability density of the reward values;

[0027] Figure 7(a) and Figure 7(b) show the task failure rate and probability density distribution of different methods under three relationship settings, respectively;

[0028] Figure 7(a) shows the comparison of failure rates, and Figure 7(b) shows the probability density of failure rates;

[0029] Figure 8 It represents the corresponding attack benefit value when the multi-agent system generates a penetration path under three relationship settings with different penetration path lengths;

[0030] Fig. 9 Indicates the task completion rate under different numbers of vulnerabilities;

[0031] Fig.10 Represents the relationship between the number of exploration steps and the number of vulnerabilities of each method;

[0032] Figure 11(a) and Figure 11(b) respectively show the failure rate and probability density distribution of each method under the ablation experiment, where:

[0033] FIG11( a ) shows the comparison of failure rates, and FIG11( b ) shows the probability density of failure rates. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with specific implementation modes.

[0035] 1 Overview

[0036] Analyzing the attacker's intrusion intention and penetration behavior is crucial to guiding network security defense. However, the existing penetration path construction mostly relies on a single agent, which cannot comprehensively evaluate the impact of group attack behavior and system vulnerability combination on security. To address this problem, the present invention applies a collaborative multi-agent reinforcement learning method. The scheme uses network situation information to guide strategy learning to improve the overall path generation quality. A hierarchical structure of subject and object (master agent + object agent) is adopted between agents. The master agent can observe the entire environment state and formulate the team's top-level collaboration strategy, while the object agent learns individual strategies based on the tasks issued and team rewards and interacts with the environment. With this model, the agents coordinate with each other in the selection of penetration strategies (penetration paths) and continuously improve their respective strategies. Experimental results show that compared with the single path generation under a single agent, the collaborative penetration path generated by the proposed method has higher penetration efficiency, decision stability, and task completion rate.

[0037] 2 System Model and Problem Modeling

[0038] 2.1 Scenario Description

[0039] Figure 1 This is a typical multi-party collaborative penetration scenario for 5G-IoT scenarios, which includes the DMZ area (demilitarized zone), 5G access network (NG-RAN)

[16] , 5G core network (5GC)

[17] , and application core area. These areas include multiple roles such as attackers, hosts, servers, firewalls, 5GC network elements (AMF, SMF, UDF)

[18] , and 5GC servers. Firewalls are divided into boundary (Firewall1 in the DMZ area) and internal firewalls (Firewall2 and Firewall3) according to their functions. Different interfaces are divided into different security zones by these firewalls.

[0040] The DMZ area is a buffer zone for deploying servers, which plays a role in isolating risks, access control, server isolation, and external access. All traffic in and out of the internal network needs to pass through the DMZ area, which is convenient for traffic inspection, filtering, and monitoring to prevent illegal access. Web servers, file servers, etc. are placed in the DMZ area to provide services to the outside world. They are isolated from other servers in the intranet. Only legitimate external users can access public services in this area.

[0041] NG-RAN and 5GC are key components of the 5G-IoT network, responsible for wireless communication with terminal devices and processing wireless protocols. 5GC is the 5G core network, responsible for session management, mobility management, policy control and other functions. The access and mobility management function (AMF) performs operations such as registration, connection, mobility management, and provides a session management message transmission channel. The session management function (SMF) is responsible for tunnel maintenance, IP address allocation and management, user plane selection, etc. The user plane function (UPF) is responsible for routing forwarding, policy implementation, traffic reporting and other functions.

[0042] The application core area includes two types of control sites: wired and wireless. The former is connected to the application layer through a gateway (GW4) to provide integrated services such as communication and computing, while the latter is connected to mobile devices through a gateway (GW5) to provide virtual services.

[0043] Table 1. Detailed information of some vulnerabilities

[0044]

[0045]

[0046] In the scenario considered, the network security vulnerabilities are set with reference to the NVD database. Table 1 gives the detailed information of some vulnerabilities.

[0047] 2.2 Attack Graph Model

[0048] According to the experimental network topology and the collected host vulnerability information, the automatic generation tool MulVAL is used to generate the attack graph. For the convenience of display, this specific implementation method extracts part of the attack graph of the DMZ area and the application core area from a large-scale scenario, and the corresponding vulnerability scores are marked on the directed edges (state transitions).

[0049] 2.3 Problem Modeling

[0050] The optimal penetration path aims to find a path between the starting node and the target node in the attack graph model that maximizes the attack benefit. The measurement indicators of attack benefit are mainly reflected in:

[0051] Vulnerability score: Vulnerability score is usually evaluated based on the exploitability, impact, and impact range of the vulnerability. The vulnerability score reflects the difficulty of exploiting the vulnerability. The higher the score, the easier it is for attackers to exploit the vulnerability and the lower the attack difficulty. Attackers usually give priority to paths with high vulnerability scores to ensure the possibility of successful attacks.

[0052] Attack duration: The attacker must complete the penetration within a certain period of time, otherwise the defender may detect the attack and take corresponding repair measures, thereby reducing the success rate of the attack. When determining the optimal penetration path, the attacker needs to consider the time window of the attack and try to complete the attack in a shorter period of time to reduce the risk of being discovered and blocked.

[0053] P S,D is defined as the initial (start) node N in the attack graph (AG) S and destination node N D The path between. S,D By the state node sequence s 1 ,...s n Composition, meeting two conditions: 1) All i ,s i+1 )∈E; 2) Each node acts only once in the path. N represents the nodes in the attack graph AG, and E represents the set of edges.

[0054] remember and are the paths obtained for the first attack and the kth attack respectively, and is the path length obtained for the first attack and the path length obtained for the kth attack.

[0055] Define a 0-1 variable z i,j Indicates whether the attacker chooses to start from s i Transfer to status j .

[0056]

[0057] The length of a path is expressed as:

[0058]

[0059] The vulnerability score on the edge of the attack graph is calculated as d i,j =Score(e i,j ), where Score(e i,j ) is the vulnerability score. The attacker starts from state s i Transfer to state s j The attack cost is expressed as C(z i,j ), which is used to balance the attack duration and vulnerability score to prevent the system from falling into an infinite loop. i,j ) contains the following properties:

[0060] 1) Every time a state transition occurs, there is a unique C(z i,j ) corresponds to it;

[0061] 2) When , the attack cost is not considered;

[0062] 3) hour, represent The length of the partial path in N X =s m .

[0063] From the state node s m Start calculating the attack cost C(z m,j ). v represents the attack cost index, E * represents the total number of directed edges in the current attack graph, and H represents the vulnerability score (e i.j ). The attack cost is defined as

[0064]

[0065] Where v is the attack cost index, calculated as

[0066]

[0067] The first permeation path obtained is expressed as when When C(z i,j )=0. Each time the attacker successfully penetrates a node s j And a state transition occurs, namely z i,j =1, we will get the directed edge e i,j Annotated vulnerability score d i,j Gained as an attack. When the attacker continues to launch penetration attacks, Score(e i.j ) and C(z i,j ) need to be considered simultaneously. Therefore, the attack benefit of the kth penetration path is quantified as

[0068]

[0069] The essence of the optimal penetration path is given Find a path in the attack graph that maximizes the attack benefit Right now:

[0070]

[0071] subject to:

[0072]

[0073] When facing large-scale complex network scenarios, a single penetration path may not be able to fully expose all security vulnerabilities and weak links in the system (penetration test system). Constructing multiple penetration path sequences helps to comprehensively detect potential risks at all levels, thereby improving defense strategies. The combination of multi-path sequences corresponding to multiple attackers must ensure that the total attack benefit of the path sequence is maximized. Then the multi-path optimal penetration path sequence construction problem is modeled as:

[0074]

[0075] Subject to(5)and(6)

[0076] 3. Collaborative Multi-Agent Deep Deterministic Policy Gradient Method

[0077] Complex decision-making relies on multiple agents to learn and handle different tasks simultaneously. However, the increase in the number of agents will bring challenges to managing their interactions. In this regard, the concepts of coordination and balance are introduced to regulate the collaboration of multiple agents. Before designing the algorithm, the characteristics of different MARL architectures are analyzed first, and then the basic model suitable for the penetration testing task is selected. Then, the key links in the algorithm model are optimized to form an efficient and collaborative multi-agent system for 5G-IoT penetration testing.

[0078] MARL architectures

[13] can be divided into three categories: 1) centralized training-centralized decision-making; 2) decentralized training-decentralized decision-making; and 3) centralized training-decentralized decision-making. Table 2 summarizes the advantages and disadvantages of different categories. Under scheme 1, as the number of agents increases, the dimensionality of the network topology space and the penetration action will grow exponentially, directly leading to a significant increase in computational complexity. At the same time, all agents must update their own policy functions, which is not suitable for dynamic network environments. Under scheme 2, each agent makes decisions autonomously and independently of other agents. Agents do not share information and make decisions entirely based on their own local observations. Such uncoordinated behavior may lead to suboptimal penetration attack strategies.

[0079] Table 2 Vulnerability information

[0080]

[0081] In summary, the collaborative approach needs to consider: 1) Infiltration path planning usually requires highly coordinated actions between multiple agents, so it is necessary to use global information to learn how to coordinate during training; 2) Infiltration tasks require agents to remain concealed, while decentralized execution can reduce communication requirements and reduce the risk of detection, making it more suitable for simulating attack behaviors; 3) During the execution phase, attackers need to make attack decisions quickly based on real-time information, and decentralized execution allows for the diversity of attacker behaviors. From the above analysis, it can be seen that solution 3, centralized training-decentralized decision-making, is more suitable for the scenario in this article.

[0082] exist Figure 3 In the framework of Scheme 3 shown in Figure 1, the training of all agents is centrally scheduled by the central controller. During the training phase, the global attack graph state transition information or the penetration actions of each agent are shared. This sharing enables the agents to learn more effective strategies, thereby accelerating the training process. However, during the execution phase, each agent is decentralized and does not rely on the central controller. In this phase, each agent independently executes its own attack based on the observed o i Make local penetration actions without knowing the actions or states of other agents. This decentralization helps improve the robustness and scalability of penetration testing.

[0083] 3.1 Collaborative penetration method based on MADDPG

[0084] Following the centralized training-decentralized execution architecture, this section designs a collaborative penetration method based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm.

[0085] Consider a scenario where M agents jointly initiate infiltration, and the strategy parameter of agent i is expressed as θ i Agent i corresponds to a policy network π(o i ,θ i ) and a value network Q(s,a). The policy network is deterministic, that is, for the observed local network topology state o i , the output penetration action a i =π(o i ,θ i ) is certain. The input of the centralized value network is the global state of the current system s = [o 1 ,o 2 ,...,o M ] and all agents’ penetration actions a=[a 1 ,…,a M ], the output is the degree of success of the penetration action in the current state.

[0086] The following information is used in the collaborative penetration method / model training process based on MADDPG: the current record in the experience replay pool (s t ,a t ,s t+1 ,r i,t ), all M strategic networks and the i-th value network. The specific content is as follows Figure 4 As shown in Figure 2, in MADDPG, all agents share a centralized value network (Critic network), which can provide guidance to each agent's policy network (Actor network) during model training, while the Actor network of each agent is executed completely independently during execution

[20]

[21] .

[0087] In the scenario of multiple attackers' collaborative penetration, the central controller is responsible for training and deploying the policy networks and value networks of all attackers. During the training phase, each attacker's policy network will be deployed to the central controller and perform actions according to the controller's instructions. At the same time, the value network is used to evaluate the quality of each action. After the training is completed, the value network is no longer needed to make judgments, only the policy network is needed to make decisions and deploy the policy network to the corresponding agent. During penetration, attacker i only needs to make independent attack behavior decisions based on his local observations and using the locally deployed policy network without the need for instructions from the central controller.

[0088] For deterministic strategies, consider that there are now M consecutive strategies The centralized action value function can be updated according to the following loss function:

[0089]

[0090] in, is the set of target policies used in the update value function.

[0091] In summary, the specific algorithm flow of the MADDPG-based collaborative penetration method is as follows:

[0092] Algorithm 1: Collaborative penetration method based on MADDPG

[0093]

[0094] 3.2Contract Net Protocol

[0095] In the collaborative penetration method based on MADDPG, the central controller is responsible for mastering global information and formulating overall strategies. The central controller can be regarded as a master agent, and its value network (Critic network) needs to accept the state observations and behaviors of all sub-agents as input, so as to make a global value assessment of the state of the entire environment.

[0096] This section combines Contract Net Protocol (CNP) with MADDPG to solve the problem of penetration task allocation and attack action execution in distributed systems. All the roles of agents in CNP are divided into master attackers (central controllers) and collaborative attackers. CNP simulates a distributed task bidding and allocation process, allowing the most suitable executors to be dynamically selected to complete the task. Figure 5 The task allocation process of the algorithm is introduced, which mainly includes the following stages:

[0097] (1) Task publishing and bidding: The master attacker broadcasts the penetration task to be executed and the task description information to all collaborative attackers. Collaborative attackers bid according to their own capabilities and conditions, formulate penetration plans, and send bidding requests to the task publisher.

[0098] (2) Bid winning and contract signing: The master attacker collects all bids and selects the most suitable collaborative attacker to perform the task based on pre-set evaluation criteria (such as vulnerability score, system risk, time cost, etc.). The master attacker and the agent sign an attack contract to clarify the execution details and remuneration of the task.

[0099] (3) Task execution: The selected collaborative attacker starts to execute the task, plans the local penetration path, and adjusts its own strategy to maximize the task execution efficiency and global benefits. Other collaborators continue to bid for the next task to be executed.

[0100] (4) Task completion and result feedback: After the collaborative attacker completes the task, he / she returns the result to the task publisher and delivers the result in the manner required by the contract.

[0101] Applying the above algorithm to MADDPG, the master attacker can output the penetration task description through the policy network and use it as the input of CNP. All agents can participate in the bidding, and the winning agent can execute the corresponding policy network to select the penetration behavior according to the bidding results. This explicit division of labor not only improves the overall collaboration efficiency, but also enhances the flexibility and scalability of the system.

[0102] To sum up, the specific algorithm flow of CNP task allocation is as follows:

[0103] Algorithm 2: CNP task allocation algorithm

[0104]

[0105] CNP provides a standardized subject-object interaction framework for MADDPG. The combination of the two can optimize the path planning and task collaboration of agents in penetration testing. Specifically, CNP first subdivides the penetration task, and then distributes the subtasks to different individuals according to the matching degree of each agent. This task allocation method can give full play to the expertise of each agent and improve the penetration efficiency. The communication mechanism between agents in CNP can help coordinate the behavior of agents and avoid conflicts and waste of resources. For example, when an agent successfully penetrates a node, it notifies other agents to avoid the node and concentrate resources on the new target.

[0106] 3.3 Experience Pool Priority Replay Algorithm

[0107] In the MADDPG algorithm, the experience replay pool can break up the temporal correlation of samples and improve the independence and efficiency of sample utilization. This section will implement the Prioritized Experience Replay (PER) algorithm to improve the efficiency of percolation path strategy generation by extracting samples with more learning value. Prioritized replay can help agents learn more efficiently by giving different importance weights to experience samples to guide sample sampling.

[0108] In reinforcement learning, Temporal Difference (TD) is used to quantify the difference between the agent's predicted future rewards and the actual future rewards. In the experience pool priority replay algorithm, TD is often used to assign weights to experience samples so that learning from important (i.e., experience with larger errors) experiences is more frequent. The calculation methods of TD and priority are introduced below.

[0109] Assume that the agent is in state s t Next, select penetration action a t , get reward r t+1 And the state is transferred to s t+1 , then the error TD can be expressed as:

[0110] δ t =r t+1 +γmax a Q(s t+1 ,a)-Q(s t ,a t ) (10)

[0111] in:

[0112] γ is a discount factor that determines how important future rewards are relative to immediate rewards;

[0113] ·Q(s t ,a t ) is the current value function estimate for the attack behavior;

[0114] ·max a Q(s t+1 ,a) is the estimate of the maximum reward for the next state transition.

[0115] The priority is usually calculated by adding a small positive constant to the absolute value of TD to avoid the priority being zero. Samples with large weights will have a greater probability of being selected for learning. t It can be expressed as:

[0116] p t =|δ t |+ε (11)

[0117] Among them, ε is a very small positive integer, which can ensure that each experience sample has a certain probability of being drawn.

[0118] In PER, priority weights are used to change the probability of samples being extracted. Usually, priority weights are proportional to the size of TD, that is, samples with larger errors are considered to have higher learning value and are given higher priority. This will cause high-priority samples to be extracted frequently during model training. In response to this, Importance Sampling Weights (ISW) are introduced to adjust the priority experience playback together with priority weights to compensate for the model bias caused by priority sampling.

[0119] ISW is inversely proportional to priority. That is, if a sample has a high priority, it is more likely to be selected, but in order to avoid the sample having too much influence on the learning process, its ISW should be relatively small. On the contrary, if a sample has a low priority, it is less likely to be selected, and its ISW should be large to ensure that when the sample is occasionally selected, it can contribute enough to the learning process.

[0120] Before calculating the ISW, the sampling probability of each experience is expressed as

[0121]

[0122] Where α is a hyperparameter in priority replay, which is used to adjust the impact of priority. It determines how the priority of experience affects the sampling probability. When α = 0, sampling becomes a random pattern. The ISW of the sampled experience sample i is calculated as

[0123]

[0124] Where N is the total number of experiences in the replay pool, and β is another hyperparameter that controls the effect of the weights, which typically increases gradually to 1 over time.

[0125] To limit the impact of weights on learning, ISW is normalized in a batch, that is, each weight is divided by the maximum value of all weights in the batch. Such normalization ensures that even in the case of non-uniform sampling, the weights will not have too much impact on the learning update step. Importance sampling weights are used to adjust the loss function to update the network parameters during backpropagation. This approach reduces the estimation bias introduced by priority sampling by balancing the contribution of each sample to the gradient according to the probability of the sample being sampled, which helps stabilize the learning process.

[0126] To sum up, the algorithm flow of experience pool priority replay can be obtained as follows:

[0127] Algorithm 3: Experience Pool Priority Replay Algorithm

[0128]

[0129] There are two points to note during the algorithm execution process. First, after the experience replay pool is loaded, old experience needs to be eliminated in a timely manner. Second, the initial values ​​and subsequent updates of the hyperparameters α and β will affect the sampling probability distribution density and control weight of the algorithm.

[0130] 4 Experimental design and results analysis

[0131] This section examines the performance of multi-agent based penetration testing under different relationships (full cooperation, full competition, and egoism); at the same time, ablation experiments are designed to analyze the impact of each module on the overall performance. For a comprehensive and intuitive comparison, representative baseline methods are selected and developed, including:

[0132] Baseline-1, MADDPG method under the setting of egoistic relationship, each agent only cares about the best interests of itself.

[0133] Maximization, regardless of the impact of other intelligent agents' behavior on the overall interests.

[0134] Baseline-2, MADDPG method under competitive relationship setting, each agent only focuses on maximizing its own benefits

[0135] In the process of execution, the agents hinder each other from achieving their goals.

[0136] Baseline-3, A2C method in a cooperative relationship setting

[22] , also belongs to the Actor-Critic method, but does not use experience feedback

[0137] Release mechanism.

[0138] Baseline-4, MADDPG method without CNP and PER.

[0139] Baseline-5, MADDPG method with only CNP integration.

[0140] Baseline-6, MADDPG method that only integrates PER.

[0141] 4.1 Impact of different collaboration modes on performance

[0142] In the penetration test scenario, the goal of the agents is the same, that is, to successfully penetrate the target host and obtain permissions. Therefore, the multi-agent reinforcement learning penetration path planning methods mentioned above are all carried out under the assumption of a cooperative relationship. This section applies multi-agent reinforcement learning under the setting of competitive relationship and egoistic relationship to a controlled experiment on penetration path planning, and compares the A2C method under another cooperative relationship setting.

[0143] In the competitive relationship setting, agents will seize resources in the environment. For example, once agent i successfully infiltrates a certain springboard host in the environment, other agents will no longer attack the host. In the egoistic relationship setting, each agent tries to maximize its individual cumulative reward, and the actions of the agent will change the state of the network environment, thereby benefiting or harming other agents.

[0144] First, the effect of training rounds on the average reward value is investigated. As shown in Figure 6(a), the average reward value under the cooperative relationship setting shows a slow growth trend. As the number of training rounds increases, the strategy learning process tends to be stable. The reward value fluctuates within a small positive range. After integrating the CNP and PER algorithms, efficient task allocation and delivery operations can be performed between the main agent and the sub-agents. At the same time, the model can prioritize learning those experiences that are more critical to improving strategy performance, thereby accelerating the learning process. Therefore, the average reward value of the proposed method is higher than that of the A2C method. The reward value of the multi-agent system under the egoistic relationship setting shows a certain growth trend, but this growth is at a low level and unstable. The reward value of the multi-agent system under the competitive relationship setting has large jitters and unstable convergence.

[0145] Figure 6(b) shows the probability density distribution of the reward values ​​of the four methods. The data distribution of the proposed method is generally on the right, which means that the reward value of the proposed method is maintained at a relatively high level. This indirectly reflects that the proposed method can learn a relatively stable strategy.

[0146] Figure 7 shows the task failure rates and their probability density distributions of different methods under three relationship settings. By comparison, it is found that the failure rate of tasks executed under cooperative and egoistic relationship settings shows a slow downward trend with the increase of training rounds. Under the cooperative relationship setting, the task failure rates of the proposed method and baseline method 3 are generally at a low level. This finding confirms the effectiveness of the collaborative mechanism in dealing with complex penetration tasks. Agents under cooperative relationships can share information and learn coordination strategies, thereby learning and developing highly adaptable penetration paths. Under the competitive relationship setting, large fluctuations in the failure rate are observed, which is caused by the resource grabbing behavior between agents. Under this relationship, agents tend to compete with each other to maximize individual interests, resulting in an unreasonable allocation of system resources. Subsequently, in some cases, the behaviors of the agents conflict with each other, making it impossible for the penetration strategy to converge to the overall optimum.

[0147] Based on the above analysis, when designing a multi-agent system to deal with network penetration tasks, we should tend to encourage cooperation between agents rather than pure self-interest or competition. Promoting collaboration between agents can improve the adaptability and robustness of the system and reduce the failure rate during task execution.

[0148] Figure 8 It reflects the corresponding attack benefit values ​​when the multi-agent system generates penetration paths under three relationship settings with different penetration path lengths. The penetration paths generated by the proposed method and benchmark method 3 under the cooperative relationship setting have higher attack benefit values. Under the other two relationship settings, the penetration paths generated by benchmark methods 1 and 2 have lower reference values. This excellent performance is mainly due to two aspects. First, the cooperative relationship is conducive to information sharing and strategy coordination, enabling agents to discover more valuable attack targets and paths. Second, centralized training uses global information, which helps to comprehensively evaluate and optimize path quality. Distributed execution ensures high efficiency and is suitable for processing large-scale complex network topologies. In contrast, multi-agents under competitive and egoistic relationship settings find it difficult to discover hidden high-value targets due to the lack of cooperation and information sharing. Therefore, the attack benefit values ​​of the generated penetration paths are generally low, and the reference value is relatively limited.

[0149] 4.2 Ablation Experiment

[0150] First, we compare the impact of the number of vulnerabilities on the task completion rate. Fig. 9The task completion rates under different numbers of vulnerabilities are shown. As the number of vulnerabilities increases, the task completion rates of all methods decrease to a certain extent, but the task completion rates of the proposed method under different numbers of vulnerabilities remain at a high level. First, the proposed method uses global information to guide the penetration behavior and ensures efficient execution; second, the unique penetration collaboration promotes mutual understanding and efficient decision-making among agents; finally, the advantage function estimation under the Actor-Critic framework can balance the immediate and long-term rewards of each agent. Even in complex scenarios with a large number of vulnerabilities, the proposed method can avoid risks and complete the expected tasks through coordination and cooperation among agents.

[0151] Next, we compare the impact of the number of vulnerabilities on the number of exploration steps. A low number of exploration steps means that the overall performance of reinforcement learning (including learning efficiency, sample utilization, environmental adaptability, determinism, and computational efficiency) is better. Fig.10 As shown in the figure, the number of exploration steps of each method is positively correlated with the number of vulnerabilities, but the number of exploration steps of the proposed method is significantly lower than that of the baseline method, indicating that it can converge to the optimal strategy faster. Under the CNP task allocation algorithm, the master agent can issue task requirements based on the current state and accept the undertaking of other agents, thereby efficiently allocating tasks and using resources. The sample priority sorting in the experience replay pool ensures that each agent can obtain the most valuable learning samples, and the model's evaluation of the current state and action is relatively certain and stable.

[0152] Figure 11(a) and Figure 11(b) show the failure rate and probability density distribution of each method under the ablation experiment. The joint optimization mechanism helps the agents learn mutual assistance and mutual promotion behavior strategies in a cooperative environment. Collaborative MADDPG can promote information sharing and resource utilization among agents through centralized training, joint optimization, experience replay and other mechanisms, allowing the agents to capture complex network attack and defense dynamics, thereby improving the pertinence and robustness of path planning.

[0153] 5. Summary

[0154] The present invention proposes a collaborative penetration model based on multi-agent reinforcement learning, which utilizes the collaboration of multiple attacking agents to improve the overall attack path generation efficiency and the pertinence of the defense strategy. The algorithm is based on the framework of "centralized training and distributed execution", which promotes the collaboration of multiple agents to jointly formulate and implement accurate penetration path planning strategies. By integrating CNP, the efficiency of task allocation by the central controller in the training phase is improved, ensuring the orderly collaboration between agents. The priority experience replay mechanism improves the guidance of penetration strategy learning. The simulation results show that the proposed method improves the penetration path generation efficiency and strategy quality in large-scale network scenarios, and expands the application potential of the MADRL algorithm in the security field. In the future, we will explore the interaction mode between multiple agents, so that different agents can assume different roles and tasks in the penetration process. We will also try to adopt differentiated strategies for different network environments, or adopt differentiated strategies at different stages to improve penetration efficiency.

[0155] References

[0156] [1]Liu XA network attack path prediction method using attack graph[J]. Journal of Ambient Intelligence and Humanized Computing, 2020:1-8.

[0157] [2]Hu H,Liu Y,Zhang H.Route prediction method for network intrusionusing absorbing Markov chain[J].Journal of Computer Research and Development,2018,55(4):831-845.

[0158] [3] Stellios I, Kotzanikolaou P, Psarakis M, et al. A survey of iot-enabled cyberattacks: Assessing attack paths to critical infrastructures and services [J]. IEEE Communications Surveys&Tutorials, 2018, 20(4): 3453-3495.

[0159] [4]Sun X,Dai J,Liu P,et al.Using Bayesian networks for probabilisticidentification of zero-day attack paths[J].IEEE Transactions on InformationForensics and Security,2018,13(10):2506-2521.

[0160] [5]Jayasuryapal G,Pranay P M,Kaur H.A survey on network penetrationtesting[C] / / 20212nd International Conference on Intelligent Engineering andManagement(ICIEM).IEEE,2021:373-378.

[0161] [6]Kotenko I,Chechulin A.Attack modeling and security evaluation inSIEM systems[J].International Transactions on Systems Science andApplications,2012,8:129-147.

[0162] [7]Fang X,Zhai L,Jia Z,et al.A game model for predicting the attackpath of APT[C] / / 2014IEEE 12th International Conference on Dependable,Autonomic and Secure Computing.IEEE,2014:491-495.

[0163] [8]Liu X,Zhang H,Zhang Y,et al.Optimal network defense strategyselection method based on evolutionary network game[J].Security andCommunication Networks,2020,2020:1-11.

[0164] [9]Khan A A,Wagan A A,Laghari A A,et al.BIoMT:A state-of-the-artconsortium serverless network architecture for healthcare system usingblockchain smart contracts[J].IEEE Access,2022,10:78887-78898.

[0165]

[10] Atzmon D,Zax Y,Kivity E,et al.Generalizing multi-agent pathfinding for heterogeneous agents[C] / / Proceedings of the InternationalSymposium on Combinatorial Search.2020,11(1):101-105.

[0166]

[11] Sharma P K,Fernandez R,Zaroukian E,et al.Survey of recent multi-agent reinforcement learning algorithms utilizing centralized training[C] / / Artificial intelligence and machine learning for multi-domain operationsapplications III.SPIE,2021,11746:665-676.

[0167]

[12] Rawat D B,Njilla L,Kwiat K,et al.iShare:Blockchain-based privacy-aware multi-agent information sharing games for cybersecurity[C] / / 2018International Conference on Computing,Networking and Communications(ICNC).IEEE,2018:425-431.

[0168]

[13] Cui J,Liu Y,Nallanathan A.Multi-agent reinforcement learning-based resource allocation for UAV networks[J].IEEE Transactions on WirelessCommunications,2019,19(2):729-743.

[0169]

[14] Bostock R M.Signal conflicts and synergies in induced resistanceto multiple attackers[J].Physiological and Molecular Plant Pathology,1999,55(2):99-109.

[0170]

[15] Hausken K,Bier V M.Defending against multiple different attackers[J].European Journal of Operational Research,2011,211(2):370-384.

[0171]

[16] Bertenyi B,Burbidge R,Masini G,et al.NG radio access network(NG-RAN)[J].Journal of ICT Standardization,2018,6(1-2):59-76.

[0172]

[17] Khichane A,Fajjari I,Aitsaadi N,et al.5GC-Observer:a Non-intrusive Observability Framework for Cloud Native 5G System[C] / / NOMS 2023-2023 IEEE / IFIP Network Operations and Management Symposium.IEEE,2023:1-10.

[0173]

[18] Lin C S.A lightweight design of 5G private network with edgecomputing[C] / / 2022 25th International Conference on Mechatronics Technology(ICMT).IEEE,2022:1-2.

[0174]

[19] Ikeda T,Shibuya T.Centralized training with decentralizedexecution reinforcement learning for cooperative multi-agent systems withcommunication delay[C] / / 2022 61st Annual Conference of the Society ofInstrument and Control Engineers(SICE).IEEE,2022:135-140.

[0175]

[20] Wan K,Wu D,Li B,et al.ME-MADDPG:An efficient learning-basedmotion planning method for multiple agents in complex environments[J].International Journal of Intelligent Systems,2022,37(3):2393-2427.

[0176]

[21] Peng B,Rashid T,Schroeder de Witt C,et al.Facmac:Factored multi-agent centralised policy gradients[J].Advances in Neural InformationProcessing Systems,2021,34:12208-12221.

[0177]

[22] Chu T,Wang J,CodecàL,et al.Multi-agent deep reinforcementlearning for large-scale traffic signal control[J].IEEE Transactions onIntelligent Transportation Systems,2019,21(3):1086-1095.

Claims

1. A 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning, the steps comprising: Step 1: Build an attack graph for 5G-IoT network scenarios; Step 2: Plan the optimal penetration path in the attack graph; In step 1, the attack graph is obtained based on the network topology and the vulnerability information of the devices in the network; In step 2, each state node in the attack graph represents a device in the network; the directed edges between the state nodes represent the state transfer between the nodes, and the corresponding vulnerability scores are marked on the edges; The optimal penetration path is the path between the starting node and the target node in the attack graph that has the greatest attack benefit; the attack benefit measurement indicators include vulnerability score and attack duration; Its characteristic is that in step one, there are multiple attackers; In step 2, a collaborative penetration model based on the multi-agent deep deterministic policy gradient MADDPG algorithm is used to plan the penetration path; The collaborative penetration model uses the MARL architecture of centralized training and decentralized decision-making as the basic model; the input of the collaborative penetration model includes the current network environment parameters and attack graph information, and the output of the model is the optimal task scheduling strategy π under the current network environment. * , i.e., the optimal permeation path; In the collaborative penetration model, the attacker is taken as the agent, and the scenario where M agents jointly initiate penetration is considered. The strategy parameter of agent i is expressed as θ i ; Agent i corresponds to a policy network π(o i ,θ i ) and a value network Q(s,a); i is the agent number, s represents the state, and a represents the penetration action of the agent; Policy network π(o i ,θ i ) is deterministic, that is, for the observed local network topology state o i , the penetration action a output by the agent i =π(o i ,θ i ) is certain; the input of the centralized value network is the global state of the current penetration testing system S = [o1, o2, ..., o M ] and all agents’ penetration actions a=[a1,…,a M ], the output is the degree of goodness of the penetration action in the current state; The information used by the collaborative penetration model during model training includes: the current i-th record (s t ,a t ,s t+1 ,r i,t ), M strategy networks and the i-th value network; where the current state s t , penetration action a t 、Current reward r i,t , the next state s t+1 All agents share a centralized value network, the Critic network, which provides guidance to each agent's strategy network, the Actor network, during model training. When performing infiltration actions, each agent's Actor network executes independently. In the MARL architecture of centralized training and decentralized decision-making, the central controller is responsible for training and deploying the policy networks and value networks of all attackers. In the centralized training phase, each attacker's policy network will be deployed to the central controller and perform penetration actions according to the instructions of the central controller. The value network evaluates the quality of each penetration action. In the decentralized decision-making phase after training, the trained policy network will be deployed to the corresponding agent. During penetration, the attacker, i.e., agent i, makes independent attack behavior decisions, i.e., penetration actions, based only on its own local observations and using the policy network deployed locally. For M consecutive strategies The action value function of the centralized value network is calculated according to the loss function L(θ i ) to update: Among them, θ i represents the policy network parameters of agent i; r i represents the reward value of agent i; y represents the target value; γ represents the discount factor, It is the set of target strategies used in the updated value function, and the ones in brackets represent the strategies of the M agents.

2. The 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning according to claim 1 is characterized in that In the collaborative penetration model based on the MADDPG algorithm, the central controller broadcasts the penetration tasks to be executed and the task description information to all agents. The agents participate in the bidding, and the winning agents execute the corresponding strategy network to select the penetration behavior according to the bidding results. When assigning penetration tasks, the central controller is regarded as a master agent and each attacker is regarded as an object agent. The value network of the master agent accepts the state observation and behavior of all object agents as input, thereby making a global value assessment of the state of the entire environment. The method by which the central controller assigns penetration tasks to each attacker adopts a task assignment method based on the Contract Net protocol CNP, in which the role of the master agent is the master attacker and the role of the object agent is the collaborative attacker. The task assignment process includes: 1) Task publishing and bidding: The master attacker broadcasts the penetration task to be executed and the task description information to all the collaborative attackers; the collaborative attackers bid according to their own capabilities and conditions, formulate penetration plans, and send bidding requests to the task publisher; 2) Bid winning and contract signing: The master attacker collects all bids and selects the most suitable collaborative attacker to perform the task based on pre-set evaluation criteria; the master attacker and the agent sign an attack contract to clarify the execution details and remuneration of the task; 3) Task execution: The selected collaborative attacker starts to execute the task, plans the local penetration path, and adjusts its own strategy to maximize the task execution efficiency and global benefits; other collaborators continue to bid for the next task to be executed; 4) Task completion and result feedback: After the collaborative attacker completes the task, he returns the result to the task publisher and delivers the results in the manner required by the contract.

3. The 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning according to claim 2 is characterized by: In step 2) of the task assignment process, the evaluation criteria include: vulnerability score, system risk, and time cost.

4. The 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning according to claim 1 is characterized by collaborative During the training process of the penetration model, the test pool priority replay algorithm is used to improve the efficiency of penetration path strategy generation by extracting samples with more learning value; In reinforcement learning, error TD is used to quantify the difference between the agent's predicted future rewards and the actual future rewards. In the experience pool priority replay algorithm, TD is used to assign priority weights to experience samples. The calculation method of TD is: Assume that the agent is in state s t Next, select penetration action a t , get reward r t+1 , and the state is transferred to s t+1 , at this time, TD is expressed as: d t =r t+1 +γmax a Q(s t+1 ,a)-Q(s t ,a t ) Where: γ is the discount factor, which determines the importance of future rewards relative to immediate rewards; Q(s t ,a t ) is the current value function estimate for the attack behavior; max a Q(s t+1 ,a) is the estimate of the maximum reward for the next state transition; The priority weight is calculated by adding a small positive constant to the absolute value of TD to avoid the priority being zero. The larger the weight of a sample, the greater the probability that it will be selected for learning. Priority weight p t It is expressed as: p t =|δ t |+e Among them, ε is a small enough positive integer to ensure that each experience sample has a probability of being drawn.

5. The 5G-IoT network optimal path penetration method based on multi-agent reinforcement learning according to claim 4 is characterized in that In the experience pool priority replay algorithm, the importance sampling weight ISW is used to compensate for the model bias caused by priority sampling. The ISW is inversely proportional to the sampling priority of the experience sample. The sampling probability of each empirical sample is expressed as Where α is a hyperparameter in priority replay, which is used to adjust the impact of priority; when α = 0, sampling becomes a random mode; the ISW of the sampled i-th experience sample is calculated as Where N is the total number of experiences in the experience replay pool; β is another hyperparameter that controls the influence of the weight, which usually gradually increases to 1 over time; In a sampling batch, the importance sampling weights ISW are normalized, that is, each ISW is divided by the maximum value of all weights in the batch; the importance sampling weights ISW are used to adjust the loss function to update the network parameters during the back-propagation process.

Images