IP address and port hopping defense method based on reinforcement learning

Abstract

The application discloses an IP address and port hopping defense method RLAPH based on reinforcement learning. In view of the problems of relatively simple hopping strategy, single hopping dimension and high defense cost in the existing scheme, the application utilizes a reinforcement learning algorithm to perform cooperative hopping of IP addresses and ports, balances defense effect and system performance by designing a reasonable reward function, and reduces unnecessary resource consumption. In addition, considering that network security events have certain continuity and periodicity, the application utilizes a CNN to extract historical data features, can effectively cope with dynamic attacks with time sequence characteristics, and thus improves the accuracy of decision-making. The method comprises the steps of detection log aggregation, strategy generation, hopping execution and flow table updating. The application can be widely applied in cloud computing, edge computing and Internet of Things environments while effectively improving network defense capability and reducing resource consumption.

Description

Technical Field

[0001] This invention relates to the field of network security, and in particular to a reinforcement learning-based method for preventing IP address and port hopping, RLAPH. Background Technology

[0002] Researchers have conducted extensive studies in the field of cybersecurity, establishing various defense systems with different mechanisms, including firewalls, intrusion detection, access control, communication encryption, and digital signatures, to improve network security. However, cybersecurity incidents continue to occur frequently. A significant reason is the inherently static nature of networks, which allows attackers ample time to continuously scan and probe for vulnerabilities in target systems, repeatedly analyzing and penetrating to achieve their ultimate attack objectives. Most of the aforementioned defense technologies are relatively passive, only detecting and implementing defensive measures after an attack has begun, leaving defenders in a consistently passive position in cyber warfare. As cyberattacks become increasingly diverse, covert, and intelligent, the threats facing cyberspace are becoming more severe. New attack methods such as Advanced Persistent Threats (APTs) exhibit not only long latency periods and high concealment but also strong adversarial capabilities, posing a more severe challenge to traditional passive defense technologies. To reverse the disadvantaged position of defenders, the defensive posture is gradually shifting from passive to proactive defense. Moving Target Defense (MTD) is a revolutionary technology designed to address the weak position of defenders in current cybersecurity competition, aiming to break the current imbalance between offense and defense. Instead of attempting to build a system without vulnerabilities, MTD allows the attack elements available to attackers to dynamically change, ensuring the normal and stable operation of the protected network. This makes it difficult for attackers to pinpoint exploitable targets or vulnerabilities. Even if attackers discover exploitable targets, these targets are constantly changing under the protection of the MTD system, preventing attackers from launching long-term, effective attacks.

[0003] Currently, mobile target defense (MTD) technologies have proven to be effective methods against network attacks such as scanning, eavesdropping, and distributed denial-of-service (DDoS) attacks. MTD technologies, by dynamically and continuously shifting the attack surface of a target system, alter its original static, isomorphic, and deterministic network attributes, increasing the difficulty and overhead for attackers to successfully launch attacks, hindering their intrusion into the system, and even forcing them to ultimately abandon their attacks. Therefore, to cope with the ever-evolving network attack methods and the constantly changing real-time network conditions, researching implementation strategies for MTD technologies, and using proactive, randomized, and diversified defense methods to prevent attackers from successfully launching attacks, will be of great significance for protecting cyberspace security in the new era and building a proactive defense system. MTD research mainly focuses on the network layer, which aligns better with MTD's pursuit of cost-effective defense due to its small size, low resource consumption, and ease of operation. However, existing network layer MTD mechanisms have the following shortcomings:

[0004] 1. Limitations of Single-Dimensional Design: Current network-layer MTD defense mechanisms mostly focus on a single transition method, such as address hopping, route hopping, or port hopping. While this design can increase system uncertainty to some extent, its overall defense effectiveness is limited because it only involves one dimension of the attack surface. Furthermore, there is a risk that attackers can discover the patterns of MTD changes through reconnaissance and analysis. Once attackers identify the logic behind these single-dimensional changes, they can bypass the MTD defense mechanism, posing a serious threat to the system.

[0005] 2. The Challenge of Balancing Defense Effectiveness and Resource Costs: In network attack and defense confrontations involving MTD (Multi-Target Defense), how to rationally select defense strategies to balance system security and performance is a major technical challenge. Currently, most dynamic transfer strategies based on the network layer attack surface rely on time-driven or event-driven approaches, but it is difficult to simultaneously consider defense effectiveness and resource costs. In time-driven approaches, if the trigger interval is too long, attackers may complete system penetration and launch attacks within sufficient time; while if the interval is too short, it will lead to frequent triggering of the MTD mechanism, which not only wastes resources and reduces system performance, but may also significantly reduce service quality. In event-driven approaches, existing methods usually perform IP hopping after detecting scanning behavior, but there are problems with the inaccuracy and inefficiency of identifying attack methods. Such aimless hopping may lead to resource waste and is difficult to integrate with the cost-effective defense goals pursued by MTD.

[0006] To address the aforementioned issues, this invention proposes a network layer address hopping defense method, RLAPH, which combines reinforcement learning technology. This method effectively enhances system security through a multi-dimensional dynamic hopping strategy, while balancing defense performance and resource overhead. Summary of the Invention

[0007] To address the problems of relatively simple hopping strategies, single dynamic hopping dimension, and high defense costs in existing solutions, this study aims to propose an adaptive mobile target defense method based on IP and port hopping. Combining the advantages and disadvantages of existing address hopping and port hopping methods, an adaptive network cooperative hopping mechanism is designed to improve the unpredictability of the system and balance the defense effect with the defense cost.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] This invention provides a reinforcement learning-based network layer address hopping method, RLAPH, which decomposes the dynamic address hopping process of the system's network layer into the following steps:

[0010] (1) Network state modeling based on semi-Markov decision process (SMDP): The network state is described as a time-varying model that includes security events and node operating conditions. A reward function is designed by combining reinforcement learning to balance system overhead and security.

[0011] (2) Reinforcement Learning Algorithm Design: The PPO algorithm is adopted to make dynamic decision-making decisions on IP address and port switching by combining historical attack data with the current network status. This algorithm learns attack behavior patterns and determines the optimal switching time through neural networks.

[0012] (3) Software-defined network architecture: Software-defined networking (SDN) is used to centrally control and execute dynamic defense strategies, and to ensure the correct forwarding of network traffic by dynamically modifying flow tables. This architecture makes the defense process transparent to end users and does not affect service continuity.

[0013] (4) Communication protocol design: To ensure communication transparency during the transition process, this invention designs an address and port conversion protocol based on a DNS server, which hides the source IP and port from the outside.

[0014] S1. The scanning events of network nodes are aggregated using the detection logs and sent to the SDN controller. The SDN controller then collects the historical data of the nodes.

[0015] S2, the SDN controller generates the optimal strategy for IP and port switching based on reinforcement learning algorithm (PPO) combined with historical data and real-time status information, and transmits the decision to the network node through the OpenFlow protocol;

[0016] S3. The node performs a transition decision, selecting a new IP address and port from the Virtual Address Range (VAR) and Virtual Port Range (VPR);

[0017] S4, the SDN controller updates the flow tables of all relevant devices in the network to ensure that the transition process is transparent and does not affect service continuity.

[0018] Further, in step S1, the controller selects a hopping action based on the network state using an RL-based endpoint hopping algorithm. Specifically, the state space, action space, and reward settings in the endpoint hopping decision module are as follows:

[0019] State space: The network state is defined as the set of states of all nodes in the cloud-edge collaborative network, represented as a vector. Where n is the total number of hosts in the network. It is network node v i (1≤i≤n) The state at time slot t, For v i The number of events hit by the scan in the t-th time slot S, b i For v i The number of IP / Port mutations that occur in the t-th time slot.

[0020] Action space: The action space is represented as a vector. in It is network node v i (1≤i≤n) Actions at time slot t, a∈O, and O={0,1,2,3}, where a=0 means neither IP nor port changes, a=1 means only IP changes, a=2 means only port changes, and a=3 means both IP and port change.

[0021] Reward Setting: After making an action decision, the endpoint transition decision module receives feedback rewards from the environment. This feedback guides the module's subsequent learning, thereby optimizing its defense strategy. The goal of the endpoint transition decision module is to minimize resource consumption while ensuring defense effectiveness; therefore, the reward function is defined as R. total =R d +R c , where R d Defense Reward, R c This is a resource consumption reward. The reward consists of two parts: a defense reward R. d The purpose is to evaluate defensive performance, calculated based on the number of successful scans. The fewer scans, the higher the reward. If all scans are avoided, a fixed positive reward is given, defined as follows:

[0022]

[0023] Where α is the coefficient, Θ t,i For node v i The number of times a host is successfully scanned in time slot t, where C is a positive constant. That is, if a host is successfully scanned, the defense reward will be a number equal to Θ. t,i Negative values ​​are linearly correlated. Otherwise, the defense bonus is a normal value.

[0024] Resource consumption reward R c This is used to evaluate the resource consumption caused by IP / port changes. It is calculated based on the resource overhead of IP or port hopping. The more hopping times, the lower the reward value.

[0025]

[0026] Where β1 and β2 are coefficients. Represents network node v i Resource consumption for IP address hopping in time slot t Represents network node v i Resource consumption for port switching in time slot t.

[0027] Furthermore, after selecting an action, the controller transmits the decision to the network device via the OpenFlow protocol. The network device then dynamically allocates a new IP address and port within the Virtual Address Range (VAR) and Virtual Port Range (VPR).

[0028] Furthermore, during the transition process, the SDN controller implements transparent updates of IP address and port mappings to ensure the transition is transparent to end users; after the transition is completed, the SDN controller synchronously updates the flow table to ensure normal routing and forwarding of data traffic.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. Enhanced Security: This invention employs a multi-dimensional defense strategy involving dynamically changing IP addresses and ports, significantly increasing the difficulty for attackers to investigate, analyze, and exploit the network environment. Compared to single-dimensional switching mechanisms, this invention achieves coordinated switching of IP addresses and ports, effectively increasing the randomness and uncertainty of the network, making it difficult for attackers to predict or adapt to the switching rules. Furthermore, the reinforcement learning-based defense mechanism can dynamically adjust the switching strategy, making the defense more flexible and further reducing the possibility of attackers detecting defense patterns.

[0031] 2. Reduced System Overhead: This invention optimizes jump operations by designing a precise reward function, avoiding resource waste caused by aimless jumps in traditional methods. Through reinforcement learning model analysis of historical attack behavior and current network state, this invention can intelligently determine jump timing, thereby avoiding the resource overhead and performance degradation caused by frequent triggering of the MTD mechanism. Compared to traditional defense methods based on fixed time intervals or event triggers, this invention better balances defense effectiveness and system performance, reducing unnecessary resource consumption. Adaptable to Complex Scenarios: By combining historical attack data, it effectively addresses dynamic attacks with temporal characteristics.

[0032] 3. High Scalability: This invention leverages the centralized control capabilities of Software-Defined Networking (SDN) to significantly enhance the flexibility of network management and defense strategy execution. By dynamically adjusting flow tables, it can quickly respond to changes in the network environment, achieving efficient multi-dimensional defense. This mechanism is applicable to various network scenarios, including complex distributed environments such as cloud-edge collaboration, the Internet of Things (IoT), smart cities, and the Industrial Internet, demonstrating strong scalability and practical value. Through modular design, the reinforcement learning model and jump strategy of this invention can be quickly deployed in networks of different sizes and are compatible with existing defense systems, further enhancing its practicality.

[0033] In summary, this invention has significant advantages over existing technologies. RLAPH has achieved corresponding optimizations in both improving network security and defense flexibility, as well as reducing system overhead. This invention effectively overcomes the limitations of existing solutions and can demonstrate good stability and applicability in dynamic network scenarios. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0035] Figure 1 This is an architecture diagram of a reinforcement learning-based IP address and port hopping defense method provided in an embodiment of the present invention; and Figure 1 The accompanying diagram illustrates the core technical features of the invention.

[0036] Figure 2 A flowchart of an IP address and port hopping defense method based on reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0037] To address the problems of relatively simple hopping strategies, limited dynamic hopping dimensions, and high defense costs in most existing network defense solutions, this invention proposes a reinforcement learning-based IP address and port hopping method, combining reinforcement learning and mobile target defense technologies.

[0038] Specifically, existing mobile target defense methods typically rely on rule-based or event-based defense strategies, such as IP or port hopping at fixed time intervals. These methods often exhibit insufficient defense effectiveness and resource waste in complex and ever-changing attack scenarios. This invention introduces a reinforcement learning model to achieve intelligent dynamic decision-making based on historical attack data and the current network state, effectively improving defense flexibility and adaptability. Furthermore, this invention combines SDN technology to achieve centralized control and execution of defense strategies, thereby optimizing system resource consumption.

[0039] To better understand this technical solution, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described examples are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0040] The overall architecture of the reinforcement learning-based dynamic IP address and port hopping method RLAPH proposed in this invention is shown in the figure below. Figure 1 As shown, the system comprises three core modules: a data acquisition module, a reinforcement learning decision-making module, and a decision execution module. The data acquisition module collects network status data and related operational statuses from each node in real time, providing input data support for the reinforcement learning decision-making module. The network status records the number of scan events and hopping frequency of nodes in chronological order, forming a state vector to describe the security and dynamic behavior characteristics of each node. The reinforcement learning decision-making module, based on reinforcement learning algorithms, makes intelligent decisions on dynamic hopping strategies and is the core logic of the entire system. It generates the optimal strategy for IP address and port hopping in real time through reinforcement learning algorithms. This module takes network status and historical data as input and outputs the node hopping decision. Through continuous training of reinforcement learning, the model can dynamically adjust the hopping frequency and action type, achieving more efficient defense in complex attack scenarios. The decision execution module achieves rapid execution of network hopping strategies through centralized control. During the hopping process, the SDN controller dynamically updates the flow table, mapping virtual IPs and ports to actual IPs and ports, ensuring that the hopping process is transparent to end users and does not affect service continuity.

[0041] Regarding the RLAPH process, specifically, as follows: Figure 2 As shown, it includes the following steps:

[0042] S1. The scanning events of network nodes are aggregated using the detection logs and sent to the SDN controller. The SDN controller then collects the historical data of the nodes.

[0043] S2 and the SDN controller generate the optimal strategy for IP address and port hopping based on a reinforcement learning algorithm (PPO) combined with historical data and real-time state information. The controller transmits this decision to the network device via the OpenFlow protocol. The state space, action space, and reward settings in the port hopping decision module are as follows:

[0044] State space: The network state is defined as the set of states of all nodes in the cloud-edge collaborative network, represented as a vector. Where n is the total number of hosts in the network. It is network node v i (1≤i≤n) The state at time slot t, For v i The number of events hit by the scan in the t-th time slot S, b i For v i The number of IP / Port mutations that occur in the t-th time slot.

[0045] Action space: The action space is represented as a vector. in It is network node v i (1≤i≤n) Actions at time slot t, a∈O, and O={0,1,2,3}, where a=0 means neither IP nor port changes, a=1 means only IP changes, a=2 means only port changes, and a=3 means both IP and port change.

[0046] Reward Setting: After making an action decision, the endpoint transition decision module receives feedback rewards from the environment. This feedback guides the module's subsequent learning, thereby optimizing its defense strategy. The goal of the endpoint transition decision module is to minimize resource consumption while ensuring defense effectiveness; therefore, the reward function is defined as R. total =R d +R c , where R d Defense Reward, R c This is a resource consumption reward. The reward consists of two parts: a defense reward R. d The purpose is to evaluate defensive performance, calculated based on the number of successful scans. The fewer scans, the higher the reward. If all scans are avoided, a fixed positive reward is given, defined as follows:

[0047]

[0048] Where α is the coefficient, Θ t,i For node v iThe number of times a host is successfully scanned in time slot t, where C is a positive constant. That is, if a host is successfully scanned, the defense reward will be a number equal to Θ. t,i Negative values ​​are linearly correlated. Otherwise, the defense bonus is a normal value.

[0049] Resource consumption reward R c This is used to evaluate the resource consumption caused by IP / port changes. It is calculated based on the resource overhead of IP or port hopping. The more hopping times, the lower the reward value.

[0050]

[0051] Where β1 and β2 are coefficients. Represents network node v i Resource consumption for IP address hopping in time slot t Represents network node v i Resource consumption for port switching in time slot t.

[0052] S3. The node executes the transition decision, and the network device dynamically allocates a new IP address and port in the Virtual Address Range (VAR) and Virtual Port Range (VPR).

[0053] S4, the SDN controller, implements transparent updates to IP address and port mappings, ensuring the transition process is transparent and does not affect service continuity. After the transition is completed, the SDN controller synchronously updates the flow table to ensure normal routing and forwarding of data traffic.

[0054] This invention proposes a reinforcement learning-based dynamic IP address and port hopping method (RLAPH). Compared with existing solutions, RLAPH achieves improvements in the collaborative optimization of multi-dimensional hopping strategies, adaptability to attack patterns, and the balance between resource overhead and defense effectiveness.

[0055] In the collaborative optimization of multi-dimensional hopping strategies, RLAPH achieves coordinated hopping of IP addresses and ports through a reinforcement learning model, overcoming the limitations of existing methods' single-dimensional hopping designs. By introducing state modeling based on a semi-Markov decision process (SMDP), the system can dynamically adjust its strategy according to the number of times a node is scanned and the hopping frequency, increasing network uncertainty and attack difficulty. Compared to traditional fixed-rule hopping strategies, RLAPH can dynamically balance the complexity and resource cost of multi-dimensional hopping, thereby improving defense effectiveness.

[0056] Regarding the adaptability to attack patterns, RLAPH, through joint analysis of historical attack data and current network conditions using a reinforcement learning model, can identify attack behaviors with temporal characteristics and adjust its transition strategy based on predictions, thereby effectively responding to complex and dynamic attack scenarios. Furthermore, by continuously optimizing strategies through the reinforcement learning model, defensive actions can be dynamically adjusted according to real-time changes in attack intensity and network conditions.

[0057] RLAPH offers an advantage over existing methods in balancing resource consumption and defensive effectiveness. Traditional methods, due to periodic or random jump mechanisms, often lead to performance degradation or even a significant decrease in service quality. RLAPH, however, optimizes jump timing and frequency through a reinforcement learning reward mechanism, ensuring defensive effectiveness while avoiding excessive resource consumption. During reinforcement learning training, RLAPH incorporates a reward function design that combines attack success rate and jump cost, enabling the model to autonomously learn how to achieve optimal defensive performance with minimal resource expenditure.

[0058] Overall, RLAPH has significant advantages over existing solutions. It has achieved corresponding optimizations in multi-dimensional jump strategies, attack mode adaptability, and resource consumption reduction, and can provide efficient and reliable security in complex and dynamic attack scenarios.

[0059] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention, based on the ideas provided by the present invention, should be considered within the scope of protection of the present invention.

Claims

1. A reinforcement learning-based IP address and port hopping defense method, RLAPH, comprising the following steps: S1. The scanning events of network nodes are aggregated using the detection logs and sent to the SDN controller. The SDN controller then collects the historical data of the nodes. S2, the SDN controller generates the optimal strategy for IP and port switching based on reinforcement learning algorithm (PPO) combined with historical data and real-time status information, and transmits the decision to the network node through the OpenFlow protocol; S3. The node performs a transition decision, selecting a new IP address and port from the Virtual Address Range (VAR) and Virtual Port Range (VPR); S4, the SDN controller updates the flow tables of all relevant devices in the network to ensure that the transition process is transparent and does not affect service continuity.

2. The dynamic defense method based on reinforcement learning according to claim 1, characterized in that, In step S2, the SDN controller selects a hopping strategy based on the network state using a reinforcement learning algorithm. This invention designs an address hopping decision module based on reinforcement learning (RL). This module receives current network state information and historical attack data, outputs an action selection, i.e., the IP address or port hopping strategy for the next moment, and corrects the strategy through rewards from environmental feedback. Through multiple iterations, a better dynamic hopping strategy is fitted. Specifically, the state space, action space, and reward settings in the address hopping decision module are as follows: State space: The network state is defined as the set of states of all nodes in the cloud-edge collaborative network, represented as a vector. Where n is the total number of hosts in the network. It is network node v i (1≤i≤n) The state at time slot t, For v i The number of events hit by the scan in the t-th time slot S, b i For v i The number of IP / Port mutations that occur in the t-th time slot. Action space: The action space is represented as a vector. in It is network node v i (1≤i≤n) Actions at time slot t, a∈O, and O={0,1,2,3}, where a=0 means neither IP nor port changes, a=1 means only IP changes, a=2 means only port changes, and a=3 means both IP and port change. Reward Setting: After making an action decision, the endpoint transition decision module receives feedback rewards from the environment. This feedback guides the module's subsequent learning, thereby optimizing its defense strategy. The goal of the endpoint transition decision module is to minimize resource consumption while ensuring defense effectiveness; therefore, the reward function is defined as R. total =R d +R c , where R d Defense Reward, R c It is a reward for resource consumption.

3. The reinforcement learning-based IP address and port hopping defense method according to claim 1, characterized in that, In step S2, the reward function is defined as follows: The reward consists of two parts: a defense reward R. d The purpose is to evaluate defensive performance, calculated based on the number of successful scans. The fewer scans, the higher the reward. If all scans are avoided, a fixed positive reward is given, defined as follows: Where α is the coefficient, Θ t,i For node v i The number of times a host is successfully scanned in time slot t, where C is a positive constant. That is, if a host is successfully scanned, the defense reward will be a number equal to Θ. t,i Negative values ​​are linearly correlated. Otherwise, the defense bonus is a normal value. Resource consumption reward R c This is used to evaluate the resource consumption caused by IP / port changes. It is calculated based on the resource overhead of IP or port hopping. The more hopping times, the lower the reward value. Where β1 and β2 are coefficients. Represents network node v i Resource consumption for IP address hopping in time slot t Represents network node v i Resource consumption for port switching in time slot t.

4. The cloud-edge collaborative dynamic active defense method based on reinforcement learning according to claim 1, characterized in that, The jump strategy is optimized using the PPO reinforcement learning algorithm. It improves exploration efficiency by adding an entropy regularization term and uses CNN to extract historical data features, taking into account the continuity and periodicity of cybersecurity events, thereby improving the accuracy of decision-making.

5. The reinforcement learning-based IP address and port hopping defense method according to claim 1, characterized in that, In step S3: After completing the selection of an action, the controller will transmit the decision to the network device through the OpenFlow protocol. The network device will dynamically allocate new IP addresses and ports in the Virtual Address Range (VAR) and Virtual Port Range (VPR).

6. The reinforcement learning-based IP address and port hopping defense method according to claim 1, characterized in that, In step S4: During the transition process, the SDN controller performs a transparent update of the IP address and port mapping to ensure the transition is transparent to end users; after the transition is completed, the SDN controller synchronously updates the flow table to ensure the normal routing and forwarding of data traffic.

Images