Deep reinforcement learning moving target defense system and method based on intrusion detection

Abstract

The invention provides a deep reinforcement learning mobile target defense system and method based on intrusion detection, and mainly solves the problem that the existing method consumes too many resources and cannot perform targeted defense for frequent IP hopping in a complex network environment. The scheme comprises the following steps: 1) constructing an SDN network topology and monitoring network traffic in real time by using an SDN controller; 2) constructing an intrusion detection model based on CNN + LSTM and introducing an attention mechanism, and identifying potential threats and abnormal behaviors; 3) initializing IP resources and maintaining a dynamic mapping table and an IP use record; 4) constructing an IP jump model based on deep reinforcement learning and defining an action and reward function; 5) completing model optimization through strategy updating; and 6) moving target defense is realized by using the optimal IP hopping strategy. The method can improve the recognition rate of complex network attacks, reduces the network security risk, reduces the delay of data processing, and effectively optimizes the utilization rate of network resources.

Description

Technical Field

[0001] The present invention belongs to the field of computer network technology, and further relates to mobile target defense technology, specifically a deep reinforcement learning mobile target defense system and method based on intrusion detection, which can be used to build internal network security systems of various enterprises and institutions. Background Art

[0002] Moving target defense is a new network security strategy that aims to increase the cost of attack for attackers by constantly changing the system or network configuration, thereby effectively preventing attackers from using known system information to carry out attacks. This approach contrasts with traditional static defense strategies, which often rely on fixed security measures. Dynamically changing network configuration can prevent attackers from using patterns observed over a long period of time to attack. Current moving target defense methods rely heavily on correct configuration and continuous management. Frequent configuration changes may lead to increased network latency and slower processing speeds, which in turn affect user experience. Intrusion detection technology can identify abnormal patterns by analyzing network traffic and user behavior, thereby promptly discovering potential security threats. In particular, deep learning technologies, such as convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), can effectively process and analyze large amounts of data and identify complex attack patterns. The introduction of these technologies not only improves the detection accuracy of traditional IDS, but also makes the intrusion detection process more automated and intelligent. By integrating deep learning technology into the moving target defense strategy, a more dynamic and adaptable defense mechanism can be achieved. For example, automatically adjusting the moving target defense configuration based on real-time intrusion detection results can not only confuse attackers, but also block attack paths in real time. In addition, deep learning models can also predict potential attack trends based on historical data and current network status, guide the adjustment of mobile target defense strategies, and make network defense more forward-looking and proactive.

[0003] For example, in the patent document with application publication number CN113114666A and titled "A mobile target defense method for scanning attacks in SDN networks", by performing random data response in the software-defined network scanning phase, using the programmable and logical centralized control characteristics of the software-defined network, different jump defense methods are used for different types of scanning data packets; by performing random operations on the response data packets, the attacker cannot determine the status of the scanned host, thereby achieving the effect of active defense and avoiding attacks to a certain extent. However, once the host address is discovered, this method cannot provide targeted defense against network attack traffic, and frequent mobile target defense will also lead to problems such as increased network latency and slower processing speeds. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a deep reinforcement learning mobile target defense system and method based on intrusion detection, which is used to solve the problem that frequent IP hopping in a complex network environment consumes too many resources and cannot provide targeted defense against network attack traffic in the existing technology. First, by building an experimental network, monitor and obtain the real-time traffic in the extracted network; then use CNN and LSTM to analyze and process network traffic data, so as to effectively identify potential threats and abnormal behaviors; on this basis, integrate deep reinforcement learning methods, and continuously learn and adjust IP hopping strategies through feedback obtained from the environment, and finally achieve the improvement of system adaptability and overall security.

[0005] To achieve the above-mentioned object, the present invention proposes a deep reinforcement learning mobile target defense system based on intrusion detection, comprising an intrusion detection unit, an IP jump unit and a deep reinforcement learning unit; wherein the outputs of the intrusion detection unit and the IP jump unit are connected to the input of the deep reinforcement learning unit;

[0006] The intrusion detection unit is composed of a data preprocessing module, a CNN+LSTM module and an attention mechanism module; wherein the data preprocessing module is used to clean network traffic data, extract network traffic features and standardize them, and then send the data to the CNN+LSTM module, the CNN part of the module extracts spatial features from the data and outputs them to the LSTM part, which captures the time dependency in the data; the attention mechanism evaluates the importance of each time step or feature by weighting the data processed by the LSTM part;

[0007] The IP hopping unit includes an IP mapping module and a sliding window module; wherein the IP mapping module uses the SDN controller to allocate a real IP and a corresponding virtual IP to all servers in the network, and the real IP is recorded as RIP and the virtual IP is recorded as VIP; the RIP is fixed and unchanged, and the VIP is jumped according to the network detection situation; the sliding window module is used to maintain the communication generated when the VIP jumps, and each server has a sliding window table for storing the VIP within N' jumps to maintain the communication quality;

[0008] The deep reinforcement learning unit includes a state definition module, an action selection module and a reward calculation module; wherein the state definition module defines the state space of reinforcement learning according to the current network environment and historical attack data; the action selection module uses the PPO algorithm to implement the optimal IP hopping strategy through strategy update; the reward calculation module calculates the reward according to the network security status and business impact after the action is executed, which is used to guide model learning and optimize future action selection.

[0009] Furthermore, the above network traffic characteristics at least include data packet size, frequency and IP address.

[0010] Furthermore, the above CNN part extracts spatial features from the data, specifically by capturing spatial details and obtaining spatial features through multiple convolutional layers and pooling layers in the CNN network.

[0011] Furthermore, the state space of the above reinforcement learning includes the intrusion detection results output by the intrusion detection unit, the network traffic characteristics and the IP resources of the IP hopping unit; the actions include selecting a new IP address and keeping the current IP address unchanged.

[0012] At the same time, the present invention also proposes a deep reinforcement learning mobile target defense method based on intrusion detection, comprising the following steps:

[0013] (1) An SDN network topology is constructed by an SDN controller, multiple servers, and switches that are no more than the number of servers. The controller communicates with the switches using the OpenFlow protocol.

[0014] (2) Use the SDN controller to monitor and capture network traffic in real time, extract key features from data packets, and remove duplicate or incomplete data packets;

[0015] (3) Building an intrusion detection model based on CNN+LSTM:

[0016] (3a) A CNN model is constructed using three convolutional layers and one maximum pooling layer. The convolution kernel size is set to 3×3 and the stride is 1. The maximum pooling layer size is set to 3×3 and the stride is 2.

[0017] (3b) Build an LSTM model and connect it to the output of the CNN model; set the number of LSTM layers to 2 and configure 128 hidden nodes for each LSTM layer;

[0018] (3c) introducing an attention mechanism after the LSTM model to obtain an intrusion detection model based on CNN+LSTM; the attention mechanism includes calculating a score function for measuring the importance of the hidden state of each LSTM output, and converting the score into a weight through a softmax function, and using the weight to perform a weighted summation on all hidden states to obtain a comprehensive context vector;

[0019] (3d) adjusting and optimizing the learning rate, batch size and number of iterations, and training an optimized intrusion detection model;

[0020] (4) IP resource initialization:

[0021] (4a) Initialize the available virtual IP pool {IP1, IP2, ..., IP3}, denoted as IPS;

[0022] (4b) The SDN controller maintains a mapping table that records the RIP and corresponding VIP of each server in the network. When the servers in the network communicate with each other, the SDN controller converts the IP addresses of outbound and inbound data packets between RIP and VIP according to the mapping table;

[0023] (4c) Establish a sliding window for each server to record the server's most recent M IP address usage history. If the window reaches M records, remove the oldest record and add a new IP address.

[0024] (4d) Set the dynamic set used to record the IP addresses currently used by all servers as U. When a new address needs to be selected for a server, select an IP address from the IPS that is neither in U nor in any server sliding window as the new IP address;

[0025] (5) Constructing an IP hopping model based on deep reinforcement learning:

[0026] (5a) Use the SDN controller to monitor network traffic, count the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each server, the number of attacks on each server, and the VIP address currently used by the server, and combine the prediction results of the CNN+LSTM intrusion detection method to record the statistical content as state S;

[0027] (5b) defining specific executable actions, including: action a0 is to keep the VIP of all current servers unchanged; action a1[i] is to change the VIP address of the i-th server, which is determined according to the new IP address described in step (4d) and assigned to server i;

[0028] (5c) Based on the improvement in network security caused by IP hopping and the network delay caused by IP hopping, the reward function R(s,a) is constructed as follows:

[0029] R(s,a)=γ×ΔV security -δ×C operation ,

[0030] Where, ΔV security Indicates the improvement of network security caused by IP hopping, C operation represents the network delay and interruption caused by IP hopping, γ and δ represent the weight factors of security improvement and operation cost respectively;

[0031] (6) Strategy Update:

[0032] (6a) Given the advantage function A t , measures the benefit of executing an action compared to the average situation, and is used to guide the optimization direction of the strategy;

[0033] (6b) Use the gradient ascent method to update the strategy every M steps, where M is greater than or equal to 500;

[0034] (7) According to the IP hopping model based on deep reinforcement learning after training, the optimal IP hopping strategy is obtained to achieve the best mobile target defense effect.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] First, the intrusion detection algorithm proposed by the present invention combines CNN and LSTM, which can effectively extract and analyze the spatial and temporal features in network data, thereby identifying complex attack patterns. The powerful image processing capability of CNN enables it to accurately extract data features, while LSTM captures potential long-term dependencies through its in-depth understanding of sequence data; combining the advantages of these two technologies can effectively improve the system's recognition rate for complex network attacks, reduce false positives and false negatives, and thus greatly enhance the accuracy and efficiency of network intrusion detection.

[0037] Second, the deep reinforcement learning framework integrated in the present invention allows the system to dynamically adjust the IP hopping strategy based on real-time data. This adaptive capability is not limited to responding to current threats, but also includes the prediction and defense of potential future attacks; as the system continues to learn and adapt, it can respond to new or changing threats more quickly, effectively reducing network security risks.

[0038] Third, because the present invention introduces the attention mechanism, the system can prioritize the most potentially threatening information when processing large amounts of data. By paying more attention to key features in network flows, this mechanism not only improves the accuracy of detection, but also optimizes the processing process, allowing the system to respond quickly, thereby intercepting and handling high-risk events in a timely manner.

[0039] Fourth, since the present invention designs a sliding window mechanism, by recording the IP addresses used during the host IP hopping process, adding new addresses and removing old addresses, the delay of data processing is effectively reduced, the stability of network communication is maintained, and the utilization of network resources is effectively optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall architecture of the system of the present invention;

[0041] Figure 2 Flow chart for realizing the method of the present invention;

[0042] Figure 3 The present invention is a flowchart for implementing iterative training of a deep reinforcement learning IP hopping model based on intrusion detection. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0044] Embodiment 1: Refer to the attached Figure 1 The present invention proposes a deep reinforcement learning mobile target defense system based on intrusion detection, which specifically includes: an intrusion detection unit, an IP jump unit and a deep reinforcement learning unit; wherein the outputs of the intrusion detection unit and the IP jump unit are connected to the input of the deep reinforcement learning unit;

[0045] The intrusion detection unit is composed of a data preprocessing module, a CNN+LSTM module and an attention mechanism module; wherein the data preprocessing module is used to clean network traffic data, extract network traffic features and standardize them. In this embodiment, the extracted network traffic features include at least packet size, frequency and IP address, etc.; then, the data is sent to the CNN+LSTM module, and the CNN part in the module extracts spatial features from the data and outputs them to the LSTM part, which captures the time dependency in the data; in this embodiment, the CNN part extracts spatial features from the data by capturing spatial details and obtaining spatial features through multiple convolutional layers and pooling layers in the CNN network. The attention mechanism evaluates the importance of each time step or feature by weighting the data processed by the LSTM part;

[0046] The IP hopping unit includes an IP mapping module and a sliding window module; wherein the IP mapping module uses the SDN controller to allocate a real IP and a corresponding virtual IP to all servers in the network, and the real IP is recorded as RIP and the virtual IP is recorded as VIP; the RIP is fixed and unchanged, and the VIP is jumped according to the network detection situation; the sliding window module is used to maintain the communication generated when the VIP jumps, and each server has a sliding window table for storing the VIP within N' jumps to maintain the communication quality;

[0047] The deep reinforcement learning unit includes a state definition module, an action selection module and a reward calculation module; wherein the state definition module defines the state space of reinforcement learning according to the current network environment and historical attack data, including the intrusion detection results output by the intrusion detection unit, the network traffic characteristics and the IP resources of the IP hopping unit; the actions include selecting a new IP address, keeping the current IP address unchanged, etc.; the action selection module uses the PPO algorithm to implement the optimal IP hopping strategy through strategy updating; the reward calculation module calculates the reward according to the network security status and business impact after the action is executed, which is used to guide model learning and optimize future action selection.

[0048] Embodiment 2: Refer to the attached Figure 2The present invention proposes a deep reinforcement learning mobile target defense method based on intrusion detection, which includes the following steps:

[0049] Step 1. An SDN controller, multiple servers, and switches no more than the number of servers build an SDN network topology, where the controller uses the OpenFlow protocol to communicate with the switches;

[0050] Step 2. Use the SDN controller to monitor and capture network traffic in real time, extract key features from data packets, and remove duplicate or incomplete data packets. The key features include the source server address IP src 、Target server address IP dst , port number P port 、Protocol type T proto , Data packet length L len and timestamp T stamp wait.

[0051] Step 3. Build an intrusion detection model based on CNN+LSTM:

[0052] (3a) A CNN model is constructed using three convolutional layers and one maximum pooling layer. The convolution kernel size is set to 3×3 and the stride is 1. The maximum pooling layer size is set to 3×3 and the stride is 2.

[0053] (3b) Build an LSTM model and connect it to the output of the CNN model; set the number of LSTM layers to 2 and configure 128 hidden nodes for each LSTM layer;

[0054] (3c) An attention mechanism is introduced after the LSTM model to obtain an intrusion detection model based on CNN+LSTM; the attention mechanism includes calculating a score function to measure the importance of the hidden state of each LSTM output, and converting the score into a weight through a softmax function, and using the weight to perform a weighted summation on all hidden states to obtain a comprehensive context vector. In this embodiment, the score is converted into a weight through a softmax function, and the weight α is calculated by the following softmax function t :

[0055]

[0056] Among them, e t is the score of the tth time step, e j represents the score of the jth time step; T represents the total number of time steps.

[0057] (3d) adjusting and optimizing the learning rate, batch size and number of iterations, and training an optimized intrusion detection model;

[0058] Step 4.IP resource initialization:

[0059] (4a) Initialize the available virtual IP pool {IP1, IP2, ..., IP3}, denoted as IPS;

[0060] (4b) The SDN controller maintains a mapping table that records the RIP and corresponding VIP of each server in the network. When the servers in the network communicate with each other, the SDN controller converts the IP addresses of outbound and inbound data packets between RIP and VIP according to the mapping table;

[0061] (4c) Establish a sliding window for each server to record the server's most recent M IP address usage history. If the window reaches M records, remove the oldest record and add a new IP address.

[0062] (4d) Set the dynamic set used to record the IP addresses currently used by all servers as U. When a new address needs to be selected for a server, select an IP address from the IPS that is neither in U nor in any server sliding window as the new IP address;

[0063] Step 5. Build an IP hopping model based on deep reinforcement learning:

[0064] (5a) Use the SDN controller to monitor network traffic, count the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each server, the number of attacks on each server, and the VIP address currently used by the server, and combine the prediction results of the CNN+LSTM intrusion detection method to record the statistical content as state S;

[0065] (5b) defining specific executable actions, including: action a0 is to keep the VIP of all current servers unchanged; action a1[i] is to change the VIP address of the i-th server, which is determined according to the new IP address described in step (4d) and assigned to server i;

[0066] (5c) Based on the improvement in network security caused by IP hopping and the network delay caused by IP hopping, the reward function R(s,a) is constructed as follows:

[0067] R(s,a)=γ×ΔV security -δ×C operation ,

[0068] Where, ΔV security Indicates the improvement of network security caused by IP hopping, C operation represents the network delay and interruption caused by IP hopping, γ and δ represent the weight factors of security improvement and operation cost respectively;

[0069] Step 6. Policy update:

[0070] (6a) Given the advantage function A t , which measures the benefit of executing an action compared to the average situation and is used to guide the optimization direction of the strategy. The advantage function A in this embodiment t , which is used to measure the benefit of selecting an action at time step t compared to the average situation, and is expressed as follows:

[0071] A t =r t +γV φ (s t+1 )-V φ (s t ),

[0072] Among them, r t represents the reward received at time step t, γ represents the discount rate of future rewards, which affects the current value of long-term rewards, and V φ (s t ) represents the value estimate of the current state.

[0073] (6b) The strategy is updated every M steps using the gradient ascent method, and M is greater than or equal to 500. The strategy is updated using the gradient ascent method. In this embodiment, the following clipping objective function is constructed: CLIP (θ) and optimize it to achieve:

[0074]

[0075] Among them, θ represents the policy parameter, r t (θ) is the probability ratio of the new strategy to the old strategy, and clip is the limit r t (θ) is a function of the range of variation, and ∈ is a hyperparameter used to control the clipping range.

[0076] The optimization is specifically to update the policy parameter θ using the gradient ascent method, which is expressed as follows:

[0077]

[0078] Among them, α represents the learning rate, Indicates L CLIP The gradient of the (θ) function with respect to the policy parameters θ.

[0079] Step 7. According to the IP hopping model based on deep reinforcement learning after training, the optimal IP hopping strategy is obtained to achieve the best mobile target defense effect.

[0080] Example 3: Reference Figure 1-2The overall implementation steps of the mobile target defense method proposed in this embodiment are the same as those in Embodiment 2. Now, a specific example is given to further describe the implementation process of the present invention in detail:

[0081] Step 1) Build SDN network topology:

[0082] Generate 1 controller host, N server hosts, and K switches. The controller uses the OpenFlow protocol to communicate with the switches to implement dynamic scheduling of network traffic and IP hopping strategies, so as to update network configuration in real time and respond to network security events;

[0083] Step 2) Use the SDN controller to monitor and capture network traffic in real time and extract the source host address IP from the data packet src 、Destination host address IP dst , port number P port 、Protocol type T proto , Data packet length L len and timestamp T stamp These key features,remove duplicate or incomplete packets;

[0084] Step 3) Build an intrusion detection model based on CNN+LSTM:

[0085] 3a) Construct a CNN model using three convolutional layers and one max pooling layer, setting the convolution kernel size to 3×3 and the stride to 1; the max pooling layer size to 3×3 and the stride to 2 to reduce the spatial dimension of the feature map;

[0086] 3b) Build an LSTM model and connect it to the output of the CNN model; set the number of LSTM layers to 2 and configure 128 hidden nodes for each LSTM layer; process the time series features provided by CNN;

[0087] 3c) implementing an attention mechanism on the LSTM model output, wherein the attention mechanism includes calculating a score function to measure the importance of the hidden state of each LSTM output, converting the score into weights through a softmax function, and using these weights to perform a weighted summation of all hidden states to obtain a comprehensive context vector;

[0088] The weight α t It is calculated by the following softmax function,

[0089]

[0090] where e t is the score at each time step t, and T is the total number of time steps.

[0091] 3d) Adjust and optimize the learning rate η, batch size Bsize , and the number of iterations n iter To improve the training efficiency and model performance, we use the network traffic training dataset D train Perform model training and test data set D test Verify the intrusion detection capabilities of the model;

[0092] Step 4) IP resource initialization:

[0093] 4a) Initialize a complete available virtual IP pool {IP1, IP2, ..., IP3}, denoted as IPS;

[0094] 4b) The SDN controller maintains a mapping table that records the RIP and corresponding VIP of each host in the network. When the hosts in the network communicate with each other, the SDN controller converts the IP address of the outbound and inbound data packets from RIP to VIP, or vice versa, according to the mapping table to ensure that the data packets are routed correctly;

[0095] 4c) Create a sliding window for each host to record the host's most recent M IP address usage history. If the window reaches M records, remove the oldest record and add a new IP address.

[0096] 4d) Maintain a dynamic set U to record the IP addresses currently in use by all hosts. When a new IP address needs to be selected for a host, select an IP address from the IPS that is neither in U nor in the sliding window of any host to prevent addressing conflicts.

[0097] Step 4) Build an IP hopping model based on deep reinforcement learning and train it to obtain the optimal IP hopping strategy, and achieve the best mobile target defense effect according to the optimal strategy.

[0098] Embodiment 4: Refer to the attached Figure 3 The overall implementation steps of the mobile target defense method proposed in this embodiment are the same as those in Embodiment 2 or 3. The training process of the deep reinforcement learning IP hopping model based on intrusion detection is now further described in detail as follows:

[0099] (a) Use the SDN controller to monitor network traffic, count the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each host, the number of attacks on each host, and the VIP address currently used by the host, and record these contents as status S;

[0100] (b) Define specific executable actions A: Action a0 keeps the VIP of all current hosts unchanged; a1[i] changes the VIP address of the i-th host and selects a new VIP to assign to host i according to (4d);

[0101] (c) The reward function R(s,a) is calculated by the network security improvement ΔV caused by IP hopping. security And the network delay caused by IP hopping C operation The calculation formula is:

[0102] R(s,a)=γ×ΔV security -δ×C operation

[0103] Where, ΔV security Indicates the improvement of network security caused by IP hopping, C operation represents the network delay and interruption caused by IP hopping, γ and δ represent the weight factors of security improvement and operation cost respectively;

[0104] (d) Observe the current network status s t , based on the current strategy π θ (parameterized by a neural network), choose an action a t , action selection is usually based on the probability distribution a t ~π θ (a|s t );

[0105] (e) Execute action a t After that, the model receives a reward r from the environment t New status t+1 ;

[0106] (f) Given an advantage function A t , measures the benefit of performing a certain action compared to the average situation, and is used to guide the optimization direction of the strategy;

[0107] (j) Given probability ratio r t (θ), which is used to measure the relative tendency of the updated strategy and the original strategy to choose the same action in the same state;

[0108] (h) Given the clipping objective function L CLIP (θ), considering the probability ratio r t (θ) and advantage function A t , by optimizing L CLIP (θ), the policy network can learn how to choose the action that brings the maximum expected reward in a given state. The detailed formula is;

[0109]

[0110] Among them, r t (θ) is the probability ratio of the new strategy to the old strategy, and the clip function limits r t The range of variation of (θ) is used to prevent the update step from being too large.

[0111] (i) Update the policy network parameters θ using the gradient ascent method:

[0112]

[0113] Among them, α is the learning rate, which determines the magnitude of each update.

[0114] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A deep reinforcement learning mobile target defense system based on intrusion detection, characterized in that: include: Intrusion detection unit, IP hopping unit and deep reinforcement learning unit; The outputs of the intrusion detection unit and the IP hopping unit are connected to the input of the deep reinforcement learning unit; The intrusion detection unit is composed of a data preprocessing module, a CNN+LSTM module and an attention mechanism module; wherein the data preprocessing module is used to clean network traffic data, extract network traffic features and standardize them, and then send the data to the CNN+LSTM module, in which the CNN part extracts spatial features from the data and outputs them to the LSTM part, which captures the time dependency in the data; The attention mechanism evaluates the importance of each time step or feature by weighting the data processed by the LSTM part; The IP hopping unit includes an IP mapping module and a sliding window module; The IP mapping module uses the SDN controller to allocate a real IP and a corresponding virtual IP to all servers in the network, and the real IP is recorded as RIP and the virtual IP is recorded as VIP; the RIP is fixed and the VIP jumps according to the network detection situation; the sliding window module is used to maintain the communication generated when the VIP jumps, and each server has a sliding window table for storing VIPs within N' jumps to maintain the communication quality; The deep reinforcement learning unit includes a state definition module, an action selection module and a reward calculation module; The state definition module defines the state space of reinforcement learning based on the current network environment and historical attack data; The action selection module uses the PPO algorithm to achieve the optimal IP hopping strategy through strategy update; The reward calculation module calculates rewards based on the network security status and business impact after the action is executed, which is used to guide model learning and optimize future action selection.

2. A system according to claim 1, characterized in that: The network traffic characteristics at least include data packet size, frequency and IP address.

3. A system according to claim 1, characterized in that: The CNN part extracts spatial features from the data, specifically by capturing spatial details and obtaining spatial features through multiple convolutional layers and pooling layers in the CNN network.

4. A system according to claim 1, characterized in that: The state space of the reinforcement learning includes the intrusion detection results output by the intrusion detection unit, the network traffic characteristics and the IP resources of the IP hopping unit; the actions include selecting a new IP address and keeping the current IP address unchanged.

5. A deep reinforcement learning mobile target defense method based on intrusion detection, characterized in that: The following steps are involved: (1) An SDN network topology is constructed by an SDN controller, multiple servers, and switches that are no more than the number of servers. The controller communicates with the switches using the OpenFlow protocol. (2) Use the SDN controller to monitor and capture network traffic in real time, extract key features from data packets, and remove duplicate or incomplete data packets; (3) Building an intrusion detection model based on CNN+LSTM: (3a) A CNN model is constructed using three convolutional layers and one maximum pooling layer. The convolution kernel size is set to 3×3 and the stride is 1. The maximum pooling layer size is set to 3×3 and the stride is 2. (3b) Build an LSTM model and connect it to the output of the CNN model; set the number of LSTM layers to 2 and configure 128 hidden nodes for each LSTM layer; (3c) introducing an attention mechanism after the LSTM model to obtain an intrusion detection model based on CNN+LSTM; the attention mechanism includes calculating a score function for measuring the importance of the hidden state of each LSTM output, and converting the score into a weight through a softmax function, and using the weight to perform a weighted summation on all hidden states to obtain a comprehensive context vector; (3d) adjusting and optimizing the learning rate, batch size and number of iterations, and training an optimized intrusion detection model; (4) IP resource initialization: (4a) Initialize the available virtual IP pool {IP1, IP2, ..., IP3}, denoted as IPS; (4b) The SDN controller maintains a mapping table that records the RIP and corresponding VIP of each server in the network. When the servers in the network communicate with each other, the SDN controller converts the IP addresses of outbound and inbound data packets between RIP and VIP according to the mapping table; (4c) Establish a sliding window for each server to record the server's most recent M IP address usage history. If the window reaches M records, remove the oldest record and add a new IP address. (4d) Set the dynamic set used to record the IP addresses currently used by all servers as U. When a new address needs to be selected for a server, select an IP address from the IPS that is neither in U nor in any server sliding window as the new IP address; (5) Constructing an IP hopping model based on deep reinforcement learning: (5a) Use the SDN controller to monitor network traffic, count the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each server, the number of attacks on each server, and the VIP address currently used by the server, and combine the prediction results of the CNN+LSTM intrusion detection method to record the statistical content as state S; (5b) defining specific executable actions, including: action a0 is to keep the VIP of all current servers unchanged; action a1[i] is to change the VIP address of the i-th server, which is determined according to the new IP address described in step (4d) and assigned to server i; (5c) Based on the improvement in network security caused by IP hopping and the network delay caused by IP hopping, the reward function R(s,a) is constructed as follows: R(s,a)=γ×ΔV security -δ×C operation , Where, ΔV secyroty Indicates the improvement of network security caused by IP hopping, C operation represents the network delay and interruption caused by IP hopping, γ and δ represent the weight factors of security improvement and operation cost respectively; (6) Strategy Update: (6a) Given the advantage function A t , measures the benefit of executing an action compared to the average situation, and is used to guide the optimization direction of the strategy; (6b) Use the gradient ascent method to update the strategy every M steps, where M is greater than or equal to 500; (7) According to the IP hopping model based on deep reinforcement learning after training, the optimal IP hopping strategy is obtained to achieve the best mobile target defense effect.

6. The method according to claim 5, characterized in that: The key features described in step (2), including the source server address IP src 、Destination server IP address dst , port number P port 、Protocol type T proto , Data packet length L len and timestamp T stamp .

7. The method according to claim 5, characterized in that: In step (3c), the score is converted into a weight through the softmax function, and the weight α is calculated by the following softmax function t : Among them, e t is the score of the tth time step, e j represents the score of the jth time step; T represents the total number of time steps.

8. The method according to claim 5, characterized in that: The advantage function A in step (6a) t , which is used to measure the benefit of selecting an action at time step t compared to the average case, and is expressed as follows: From t =r t +γV φ (s t+1 )-V φ (s t ), Among them, r t represents the reward received at time step t, γ represents the discount rate of future rewards, which affects the current value of long-term rewards, and V φ (s t ) represents the value estimate of the current state.

9. The method according to claim 8, characterized in that: The gradient ascent update strategy described in step (6b) is achieved by constructing the following clipping objective function L CLIP (θ) and optimize it to achieve: Among them, θ represents the policy parameter, r t (θ) is the probability ratio of the new strategy to the old strategy, and clip is the limit r t (θ) is a function of the range of variation, and ∈ is a hyperparameter used to control the clipping range.

10. The method according to claim 9, characterized in that: The optimization is specifically to update the policy parameter θ using the gradient ascent method, which is expressed as follows: Among them, α represents the learning rate, Indicates L CLIP The gradient of the (θ) function with respect to the policy parameters θ.

Images