Intrusion detection based deep reinforcement learning mobile target defense system and method

Abstract

The application provides an intrusion detection-based deep reinforcement learning mobile target defense system and method, and mainly solves the problem that the existing method consumes too much resource for frequent IP hopping in a complex network environment and cannot be used for targeted defense. The scheme comprises the following steps: 1) constructing an SDN network topology and using an SDN controller to monitor network traffic in real time; 2) constructing an intrusion detection model based on CNN+LSTM and introducing an attention mechanism to identify potential threats and abnormal behaviors; 3) initializing IP resources and maintaining a dynamic mapping table and IP usage record; 4) constructing an IP hopping model based on deep reinforcement learning and defining an action and reward function; 5) completing model optimization through policy updating; and 6) realizing mobile target defense by using an optimal IP hopping strategy. The application can improve the recognition rate of complex network attacks, reduce network security risks, reduce the delay of data processing, and effectively optimize the utilization rate of network resources.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer networks, further relates to the technical field of mobile target defense, and in particular to a deep reinforcement learning mobile target defense system and method based on intrusion detection, which can be used for internal network security system construction of enterprises and institutions. BACKGROUND

[0002] Mobile target defense is a new type of network security strategy, which aims to increase the attack cost of attackers by constantly changing the system or network configuration, so as to effectively prevent attackers from using known system information to implement attacks. This method is in contrast to traditional static defense strategies, which often rely on fixed security measures. Dynamically changing network configurations can prevent attackers from exploiting regular patterns observed over a long period of time. The current mobile target defense method largely depends on correct configuration and continuous management, and due to the need for frequent configuration changes, it may cause network latency to increase and processing speed to slow down, thereby affecting user experience. Intrusion detection technology can identify abnormal patterns by analyzing network traffic and user behavior, thereby discovering potential security threats in a timely manner. In particular, the use of deep learning technologies such as convolutional neural networks (CNN) and long short-term memory networks (LSTM) can effectively process and analyze large amounts of data to identify complex attack patterns. The introduction of these technologies not only improves the detection accuracy of traditional IDS, but also makes the intrusion detection process more automated and intelligent. By integrating deep learning technology into the mobile target defense strategy, a more dynamic and adaptive defense mechanism can be achieved. For example, automatically adjusting the mobile target defense configuration based on real-time intrusion detection results can not only confuse attackers, but also block attack paths in real time. In addition, the deep learning model can predict potential attack trends based on historical data and current network status, guiding the adjustment of the mobile target defense strategy, making network defense more proactive and active.

[0003] For example, in the patent document with the application publication number CN113114666A and the name "A mobile target defense method for scanning attacks in an SDN network", the method uses different hopping defense methods for different scanning packet types by randomly responding to data during the software-defined network scanning phase, taking advantage of the programmable and logically centralized control features of software-defined networks. By randomizing the response packets, attackers cannot determine the status of the scanned host, thereby achieving the effect of active defense and avoiding attacks to some extent. However, when the host address is discovered, this method cannot provide targeted defense for network attack traffic, and frequent mobile target defense will also cause network latency to increase and processing speed to slow down. SUMMARY

[0004] The present application aims at the deficiencies of the prior art, and provides a deep reinforcement learning mobile target defense system and method based on intrusion detection, which is used to solve the problem that the existing technology consumes too many resources in complex network environment and cannot conduct targeted defense for network attack traffic. First, an experimental network is constructed to monitor and obtain real-time traffic in the network; then CNN and LSTM are used to analyze and process network traffic data, so as to effectively identify potential threats and abnormal behaviors; on this basis, a deep reinforcement learning method is integrated to continuously learn and adjust the IP hopping strategy through feedback obtained from the environment, so as to finally realize the improvement of system adaptive ability and overall security.

[0005] To achieve the above-mentioned purpose, the present application provides a deep reinforcement learning mobile target defense system based on intrusion detection, which comprises an intrusion detection unit, an IP hopping unit and a deep reinforcement learning unit; wherein the outputs of the intrusion detection unit and the IP hopping unit are connected to the input of the deep reinforcement learning unit.

[0006] The intrusion detection unit is composed of a data preprocessing module, a CNN+LSTM module and an attention mechanism module; wherein the data preprocessing module is used to clean network traffic data, extract network traffic features and standardize them, and then send the data to the CNN+LSTM module; the CNN part of the module extracts spatial features from the data and outputs them to the LSTM part, which captures the time dependence in the data; the attention mechanism evaluates the importance of each time step or feature by weighting the data processed by the LSTM part.

[0007] The IP hopping unit comprises an IP mapping module and a sliding window module; wherein the IP mapping module uses an SDN controller to assign a real IP and a virtual IP corresponding to the real IP to all servers in the network, and the real IP is denoted as RIP and the virtual IP is denoted as VIP; the RIP is fixed and the VIP is changed according to the network detection; the sliding window module is used to maintain the communication generated when the VIP is changed, and each server has a sliding window table for storing the VIP within N' times of hopping to maintain the communication quality.

[0008] The deep reinforcement learning unit comprises a state definition module, an action selection module and a reward calculation module; wherein the state definition module defines the state space of reinforcement learning according to the current network environment and historical attack data; the action selection module uses the PPO algorithm to realize the optimal IP hopping strategy through policy update; the reward calculation module calculates the reward according to the network security status and business impact after the action is executed, which is used to guide the model learning and optimize the future action selection.

[0009] Further, the network traffic features at least include packet size, frequency and IP address.

[0010] Further, the above-mentioned CNN part extracts spatial features from data, specifically by capturing spatial details through multiple convolutional layers and pooling layers in the CNN network to obtain spatial features.

[0011] Further, the state space of the above-mentioned reinforcement learning includes the intrusion detection results output by the intrusion detection unit, network traffic features, and IP resource of the IP hopping unit; the action includes selecting a new IP address or keeping the current IP address unchanged.

[0012] Meanwhile, the application also proposes a deep reinforcement learning mobile target defense method based on intrusion detection, comprising the following steps:

[0013] (1) An SDN controller, multiple servers, and no more than the number of switches are used to build an SDN network topology, wherein the controller uses the OpenFlow protocol to communicate with the switches;

[0014] (2) The SDN controller is used to monitor and capture network traffic in real time, extract key features from the data packets, and remove duplicate or incomplete data packets;

[0015] (3) An intrusion detection model based on CNN+LSTM is constructed:

[0016] (3a) A CNN model is constructed using three convolutional layers and one max-pooling layer, with a convolution kernel size of 3x3 and a step size of 1; the max-pooling layer has a size of 3x3 and a step size of 2;

[0017] (3b) An LSTM model is built and connected to the output end of the CNN model; the number of LSTM layers is set to 2, and each LSTM layer is configured with 128 hidden nodes;

[0018] (3c) An attention mechanism is introduced after the LSTM model to obtain an intrusion detection model based on CNN+LSTM; the attention mechanism includes a score function for measuring the importance of each LSTM output hidden state, and a softmax function is used to convert the score into a weight; the weight is used to weight the sum of all hidden states to obtain a comprehensive context vector;

[0019] (3d) The learning rate, batch size, and iteration number are adjusted and optimized to train an optimized intrusion detection model;

[0020] (4) IP resource initialization:

[0021] (4a) The available virtual IP pool {IP1, IP2, …, IP3} is initialized and denoted as IPS;

[0022] (4b) The SDN controller maintains a mapping table recording the RIP and corresponding VIP of each server in the network. When the servers in the network communicate with each other, the SDN controller converts the IP address of the outbound and inbound data packets between RIP and VIP according to the mapping table;

[0023] (4c) A sliding window is established for each server to record the IP address usage history of the server in the last M times. If the window reaches M times of recording, the oldest record is removed and a new IP address is added;

[0024] (4d) Set a dynamic set U to record the IP addresses currently used by all servers. When a new address needs to be selected for a server, an IP address that is neither in U nor in any server's sliding window is selected from the IPS as a new IP address;

[0025] (5) Construct an IP hopping model based on deep reinforcement learning:

[0026] (5a) Use the SDN controller to monitor network traffic, count the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each server, the number of times each server is attacked, and the VIP address currently used by the server. Combine the prediction results of the CNN+LSTM intrusion detection method to record the statistics as state S;

[0027] (5b) Define specific executable actions, including: action a0 is to keep the VIP of all servers unchanged; action a1[i] is to change the VIP address of the i-th server, which is determined according to the new IP address in step (4d) and assigned to server i;

[0028] (5c) According to the improvement of network security caused by IP hopping and the network delay caused by IP hopping, the reward function R(s, a) is constructed as follows:

[0029] R(s, a) = γ × ΔV security - δ × C operation ,

[0030] Where ΔV security represents the improvement of network security caused by IP hopping, C operation represents the network delay and interruption caused by IP hopping, and γ and δ represent the weight factors of security improvement and operation cost, respectively;

[0031] (6) Policy update:

[0032] (6a) Given the advantage function A t , which measures the benefits of performing actions compared to the average case, is used to guide the optimization direction of the policy;

[0033] (6b) using a gradient ascent method to update the strategy, updating the strategy once every M steps, and M is greater than or equal to 500;

[0034] (7) According to the deep reinforcement learning-based IP hopping model after training, the optimal IP hopping strategy is obtained, and the best mobile target defense effect is achieved.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] First, the intrusion detection algorithm proposed in the present application combines CNN and LSTM, which can effectively extract and analyze the spatial and temporal features in network data, so as to identify complex attack patterns. The powerful image processing capability of CNN enables it to accurately extract data features, while LSTM captures potential long-term dependencies through its deep understanding of sequence data; by combining the advantages of these two technologies, the recognition rate of the system for complex network attacks can be effectively improved, reducing false positives and false negatives, thereby greatly enhancing the accuracy and efficiency of network intrusion detection.

[0037] Second, the integrated deep reinforcement learning framework of the present application allows the system to dynamically adjust the IP hopping strategy according to real-time data. This adaptive capability not only responds to current threats, but also predicts and defends against potential future attacks; as the system continuously learns and adapts, it can respond more quickly to new or changing threats, effectively reducing network security risks.

[0038] Third, due to the introduction of the attention mechanism in the present application, the system can prioritize processing information that is most potentially threatening when dealing with large amounts of data. By giving more attention to key features in network flows, this mechanism not only improves detection accuracy, but also optimizes the processing process, enabling the system to quickly respond, thereby timely intercepting and handling high-risk events.

[0039] Fourth, due to the design of the sliding window mechanism in the present application, by recording the IP addresses used during the host IP hopping process, adding new addresses and removing old addresses, the delay of data processing is effectively reduced, maintaining the stability of network communication and effectively optimizing the utilization of network resources. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the overall architecture diagram of the system of the present application;

[0041] Figure 2 is the implementation flowchart of the method of the present application;

[0042] Figure 3 is the implementation flowchart of the iterative training of the deep reinforcement learning IP hopping model based on intrusion detection of the present application. DETAILED DESCRIPTION

[0043] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0044] Embodiment one: refer to the attached Figure 1 The application provides an intrusion detection-based deep reinforcement learning mobile target defense system, which specifically comprises an intrusion detection unit, an IP hopping unit and a deep reinforcement learning unit; wherein the outputs of the intrusion detection unit and the IP hopping unit are connected to the input of the deep reinforcement learning unit.

[0045] The intrusion detection unit is composed of a data preprocessing module, a CNN+LSTM module and an attention mechanism module; wherein the data preprocessing module is used for cleaning network traffic data, extracting network traffic features and performing standardization; in the embodiment, the extracted network traffic features include at least packet size, frequency and IP address; then, the data is sent to the CNN+LSTM module; the CNN part of the module extracts spatial features from the data and outputs them to the LSTM part, which captures the time dependence in the data; in the embodiment, the CNN part extracts spatial features from the data by capturing spatial details through multiple convolutional layers and pooling layers in the CNN network to obtain spatial features; the attention mechanism evaluates the importance of each time step or feature by weighting the data processed by the LSTM part;

[0046] The IP hopping unit comprises an IP mapping module and a sliding window module; wherein the IP mapping module uses an SDN controller to assign a real IP and a virtual IP corresponding to the real IP to all servers in the network; the real IP is denoted as RIP, and the virtual IP is denoted as VIP; the RIP is fixed and unchanged, and the VIP is hopped according to the network detection situation; the sliding window module is used for maintaining communication generated when the VIP is hopped; each server has a sliding window table for storing the VIP within N' times of hopping to maintain the quality of communication.

[0047] The deep reinforcement learning unit comprises a state definition module, an action selection module and a reward calculation module; wherein the state definition module defines the state space of reinforcement learning according to the current network environment and historical attack data, including the intrusion detection results output by the intrusion detection unit, the network traffic features and the IP resources of the IP hopping unit; the actions include selecting a new IP address, keeping the current IP address unchanged, etc.; the action selection module uses the PPO algorithm to realize the optimal IP hopping strategy through policy updating; the reward calculation module calculates the reward according to the network security status and business impact after the action is executed, which is used to guide the model learning and optimize the future action selection.

[0048] Embodiment two: refer to the attached Figure 2The application provides an intrusion detection-based deep reinforcement learning mobile target defense method, which comprises the following steps.

[0049] Step 1: An SDN controller, multiple servers and no more than the number of switches are used to build an SDN network topology, wherein the controller communicates with the switches using an OpenFlow protocol;

[0050] Step 2: The SDN controller is used to monitor and capture network traffic in real time, key features are extracted from the data packets, and repeated or incomplete data packets are removed. The key features include a source server address IP src , a target server address IP dst , a port number P port , a protocol type T proto , a data packet length L len , a timestamp T stamp and the like.

[0051] Step 3: An intrusion detection model based on CNN+LSTM is constructed.

[0052] (3a) A CNN model is constructed by using three convolutional layers and one maximum pooling layer, the convolution kernel size is set to 3x3 and the step is set to 1, and the maximum pooling layer size is set to 3x3 and the step is set to 2;

[0053] (3b) An LSTM model is built and connected to the output end of the CNN model; the number of layers of the LSTM is set to 2, and 128 hidden nodes are configured for each LSTM layer;

[0054] (3c) An attention mechanism is introduced after the LSTM model to obtain the intrusion detection model based on CNN+LSTM; the attention mechanism comprises a score function for measuring the importance of the hidden state of each LSTM output, and a weight is converted from the score by using a softmax function, the weight is used for weighted summation of all hidden states to obtain a comprehensive context vector. In the embodiment, the weight is calculated by the following softmax function: t

[0055]

[0056] wherein e t represents the score of the tth time step, e j represents the score of the jth time step, and T represents the total number of time steps.

[0057] (3d) The learning rate, batch size and iteration number are adjusted and optimized, and the optimized intrusion detection model is trained;

[0058] ​Step 4. IP resource initialization:

[0059] (4a) Initialize the available virtual IP pool {IP1, IP2, …, IP3} as IPS;

[0060] (4b) The SDN controller maintains a mapping table recording the RIP and corresponding VIP of each server in the network. When the servers in the network communicate with each other, the SDN controller converts the IP address of the outgoing and incoming data packets between RIP and VIP according to the mapping table;

[0061] (4c) Establish a sliding window for each server to record the IP address usage history of the last M times. If the window reaches M times of recording, remove the oldest record and add a new IP address;

[0062] (4d) Set the dynamic set for recording the IP addresses currently used by all servers as U. When a new address needs to be selected for a server, select an IP address from IPS that is neither in U nor in any server's sliding window as the new IP address;

[0063] Step 5. Construct IP hopping model based on deep reinforcement learning:

[0064] (5a) Use the SDN controller to monitor network traffic, count the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each server, the number of times each server is attacked, and the current VIP address used by the server. Combine the prediction results of the CNN+LSTM intrusion detection method to record the statistics as state S;

[0065] (5b) Define specific executable actions, including: action a0 is to keep the VIP of all servers unchanged; action a1[i] is to change the VIP address of the i-th server, which is determined according to the new IP address in step (4d) and assigned to server i;

[0066] (5c) According to the improvement of network security caused by IP hopping and the network delay caused by IP hopping, the reward function R(s, a) is constructed as follows:

[0067] R(s, a) = γ × ΔV security - δ × C operation ,

[0068] Where ΔV security represents the improvement of network security caused by IP hopping, C operation represents the network delay and interruption caused by IP hopping, and γ and δ represent the weight factors of security improvement and operation cost, respectively;

[0069] Step 6. Strategy update:

[0070] (6a) Given advantage function A t , which measures the benefit of performing an action compared to the average case, is used to guide the optimization direction of the strategy. The advantage function A t in this embodiment is specifically used to measure the benefit of selecting an action at time step t compared to the average case, and the expression is as follows:

[0071] A t = r t + γV φ (s t+1 )-V φ (s t ),

[0072] wherein r t represents the reward received at time step t, γ represents the discount rate of future rewards, which affects the current value of long-term rewards, and V φ (s t ) represents the value estimate of the current state.

[0073] (6b) Update the strategy using the gradient ascent method, update the strategy every M steps, and M is greater than or equal to 500; the use of the gradient ascent method to update the strategy, this embodiment is specifically realized by constructing the following clipping objective function as L CLIP (θ) and optimizing it:

[0074]

[0075] wherein θ represents the strategy parameter, r t (θ) is the probability ratio of the new strategy and the old strategy, clip is a function that limits the range of variation of r t (θ), and ∈ is a hyperparameter used to control the clipping range.

[0076] The optimization is specifically updating the strategy parameter θ using the gradient ascent method, which is expressed as follows:

[0077]

[0078] wherein α represents the learning rate, represents the gradient of the L CLIP (θ) function with respect to the strategy parameter θ.

[0079] Step 7. According to the deep reinforcement learning-based IP hopping model after training is completed, the optimal IP hopping strategy is obtained, and the best mobile target defense effect is achieved.

[0080] Embodiment three: refer to Figures 1-2The mobile target defense method proposed in the embodiment realizes the steps as a whole, and the implementation process of the application is described in further detail by giving specific examples:

[0081] Step 1) Constructing the SDN network topology:

[0082] Generate 1 controller host, N server hosts, and K switches. The controller uses the OpenFlow protocol to communicate with the switches to realize dynamic scheduling of network traffic and implementation of IP hopping strategies, so as to update the network configuration in real time and respond to network security events;

[0083] Step 2) Real-time monitoring and capturing network traffic using the SDN controller, extracting the source host address IP src , target host address IP dst , port number P port , protocol type T proto , packet length L len and timestamp T stamp These key features from the data packet, remove duplicate or incomplete packets;

[0084] Step 3) Constructing an intrusion detection model based on CNN+LSTM:

[0085] 3a) Construct a CNN model using three convolutional layers and one max-pooling layer, set the convolution kernel size to 3x3 and the step size to 1; the max-pooling layer size is 3x3 and the step size is 2; to reduce the spatial dimension of the feature map;

[0086] 3b) Build an LSTM model connected to the output end of the CNN model; set the number of LSTM layers to 2, and configure 128 hidden nodes for each LSTM layer; process the time series features provided by CNN;

[0087] 3c) Implement an attention mechanism on the output of the LSTM model, which includes calculating a score function to measure the importance of each LSTM output hidden state, and converting the score to weight through a softmax function, and using these weights to weight sum all hidden states to get a comprehensive context vector;

[0088] The weight α t is calculated by the following softmax function,

[0089]

[0090] Where e t is the score at each time step t, and T is the total number of time steps.

[0091] 3d) Adjust and optimize the learning rate η, batch size Bsize , and the number of iterations n iter to improve the efficiency of training and model performance, using network traffic training dataset D train Model training is performed by testing dataset D test Verify the intrusion detection capability of the model;

[0092] Step 4) IP resource initialization:

[0093] 4a) Initialize a complete pool of available virtual IP {IP1, IP2, …, IP3}, denoted as IPS;

[0094] 4b) The SDN controller maintains a mapping table that records the RIP and corresponding VIP of each host in the network. When the hosts in the network communicate with each other, the SDN controller converts the IP addresses of the outbound and inbound packets from RIP to VIP according to the mapping table, or vice versa, to ensure correct routing of the packets;

[0095] 4c) Establish a sliding window for each host to record the IP address usage history of the host in the last M times. If the window reaches M records, remove the oldest record and add a new IP address.

[0096] 4d) Maintain a dynamic set U to record the IP addresses currently used by all hosts. When a new IP address needs to be selected for a host, an IP address that is neither in U nor in any host's sliding window is selected from IPS to prevent addressing conflicts;

[0097] Step 4) Construct an IP hopping model based on deep reinforcement learning and train it to obtain the optimal IP hopping strategy, and implement the best mobile target defense effect according to the optimal strategy.

[0098] Embodiment four: refer to the attached Figure 3 The mobile target defense method proposed in this embodiment implements the same overall steps as in Embodiment Two or Three. Now the training process of the deep reinforcement learning IP hopping model based on intrusion detection will be further described in detail as follows:

[0099] (a) Use the SDN controller to monitor network traffic, count the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each host, the number of attacks each host receives, and the current VIP address used by the host. Record these contents as state S;

[0100] (b) Define specific executable actions A: action a0 keeps the VIP of all hosts unchanged; a1[i] changes the VIP address of the i-th host, and selects a new VIP for host i according to (4d);

[0101] (c) reward function R(s, a) is the improvement of network security ΔV caused by IP hopping security and network latency C caused by IP hopping operation , and the calculation formula is

[0102] R(s, a) = γ × ΔV security - δ × C operation

[0103] where ΔV security represents the improvement of network security caused by IP hopping, C operation represents the network latency and interruption caused by IP hopping, and γ and δ represent the weight factors of security improvement and operation cost, respectively.

[0104] (d) Observe the current state s t of the network, and select an action a θ based on the current policy π t (parametrized by the neural network), which is usually based on the probability distribution a t ~ π θ (a | s t );

[0105] (e) After performing the action a t , the model receives a reward r t from the environment and a new state s t+1 ;

[0106] (f) Given the advantage function A t , which measures the benefit of performing a certain action compared to the average case, it is used to guide the optimization direction of the policy.

[0107] (j) Given the probability ratio r t (θ), which measures the relative tendency of the updated policy and the original policy to select the same action in the same state.

[0108] (h) Given the clipping objective function L CLIP (θ), considering the probability ratio r t (θ) and the advantage function A t , by optimizing L CLIP (θ), the policy network can learn how to select the action that brings the maximum expected return in a given state, and the detailed formula is

[0109]

[0110] where r t (θ) is the probability ratio of the new policy and the old policy, and the clip function limits the range of variation of r t (θ) to prevent the update step from being too large.

[0111] (i) updating the parameters θ of the policy network using the gradient ascent method:

[0112]

[0113] wherein a is the learning rate, determining the magnitude of the update at each step.

[0114] The part of the present application not described in detail belongs to the common knowledge of the person skilled in the art.

[0115] The above description is merely preferred embodiments of the present application but not to confine the present application. Obviously, for the person skilled in the art, after understanding the contents and principles of the present application, various modifications and changes in form and details can be made without departing from the principles and structures of the present application. However, these modifications and changes based on the idea of the present application are still within the protection scope of the claims of the present application.

Claims

1. An intrusion detection based deep reinforcement learning mobile target defense system, characterized in that, The application relates to a network security system based on deep reinforcement learning, comprising an intrusion detection unit, an IP hopping unit and a deep reinforcement learning unit; wherein the outputs of the intrusion detection unit and the IP hopping unit are connected to the input of the deep reinforcement learning unit; the intrusion detection unit is composed of a data preprocessing module, a CNN+LSTM module and an attention mechanism module; wherein the data preprocessing module is used for cleaning network traffic data, extracting network traffic features and standardizing the network traffic features, and then sending the data into the CNN+LSTM module; the CNN part of the module extracts spatial features from the data and outputs the spatial features to the LSTM part, which captures the time dependence in the data; the attention mechanism evaluates the importance of each time step or feature by weighting the data processed by the LSTM part; the IP hopping unit comprises an IP mapping module and a sliding window module; wherein the IP mapping module uses an SDN controller to assign a real IP and a virtual IP corresponding to the real IP to all servers in the network, and the real IP is denoted as RIP and the virtual IP is denoted as VIP; the RIP is fixed and unchanged, and the VIP is hopped according to the network detection situation; the sliding window module is used for maintaining the communication generated when the VIP is hopped, and each server has a sliding window table for storing the VIP in N' times of hopping to maintain the communication quality; the deep reinforcement learning unit comprises a state definition module, an action selection module and a reward calculation module; wherein the state definition module defines the state space of reinforcement learning according to the current network environment and historical attack data; the action selection module uses the PPO algorithm to realize the optimal IP hopping strategy through strategy updating; and the reward calculation module calculates the reward according to the network security situation and business influence after the action is executed, and the reward is used for guiding the model learning and optimizing the future action selection; the network traffic features at least comprise packet size, frequency and IP address; the CNN part extracts spatial features from the data, specifically captures spatial details through multiple convolution layers and pooling layers in the CNN network to obtain spatial features; the state space of reinforcement learning comprises the intrusion detection result output by the intrusion detection unit, the network traffic features and the IP resources of the IP hopping unit; and the action comprises selecting a new IP address or keeping the current IP address unchanged; the network security system based on deep reinforcement learning comprises the following steps: (1) an SDN controller, multiple servers and no more than the number of switches are used to build an SDN network topology, wherein the controller uses the OpenFlow protocol to communicate with the switches; (2) the SDN controller is used for monitoring and capturing network traffic in real time, extracting key features from the data packets and removing repeated or incomplete data packets; (3) an intrusion detection model based on CNN+LSTM is constructed: (3a) a CNN model is constructed by using three convolution layers and one maximum pooling layer, the convolution kernel size is set to 3*3 and the step is set to 1; the maximum pooling layer size is set to 3*3 and the step is set to 2; (3b) an LSTM model is built and connected to the output end of the CNN model; the number of layers of the LSTM is set to 2, and 128 hidden nodes are configured for each LSTM layer. ​ ​ ​ ​ ​ ​ ​ ​ ​ The defined specific executable actions include: action To keep the VIP of all current servers unchanged; action To change the VIP address of the first Server; construct a reward function according to the improvement of network security caused by IP hopping and the network delay caused by IP hopping ; Given the advantage function , measure the benefits of executing the action compared to the average case, and use it to guide the optimization direction of the strategy; update the strategy using gradient ascent method, update the strategy every M steps.

2. The system of claim 1, wherein: ​ 3. The system of claim 1, wherein: ​ 4. The system of claim 1, wherein: ​ 5. An intrusion detection based deep reinforcement learning mobile target defense method, characterized in that, ​ ​ ​ ​ ​ ​ (3c) Introducing an attention mechanism after the LSTM model to obtain a CNN+LSTM-based intrusion detection model; the attention mechanism includes a score function for measuring the importance of each LSTM output hidden state, and the score is converted into a weight through a softmax function, and the weight is used to weight the sum of all hidden states to obtain a comprehensive context vector; (3d) Adjust and optimize the learning rate, batch size and iteration number to train an optimized intrusion detection model; (4) IP resource initialization: (4a) initializing a pool of available virtual IPs , denoted as ; (4b) Record the real IP as RIP and the virtual IP as VIP; the SDN controller maintains a mapping table of each server RIP and the corresponding VIP in the network, and when the servers in the network communicate with each other, the SDN controller converts the IP address of the outgoing and incoming data packets between RIP and VIP according to the mapping table; (4c) for each server, a sliding window of the last occurrences of IP address usage history, if the window reaches the maximum number of records, the oldest record is removed and a new IP address is added; (4d) setting a dynamic set of all IP addresses currently used by the servers to record When a new address needs to be selected for a server, an IP address is chosen from that is neither in nor in any server's sliding window as the new IP address; (5) Constructing an IP hopping model based on deep reinforcement learning: (5a) using the SDN controller to monitor network traffic, counting the total network traffic, the number of attack events, the proportion of abnormal traffic, the traffic of each server, the number of attacks on each server, the current VIP address used by the server, and combining the prediction results of the CNN+LSTM intrusion detection method to record the status ; (5b) defining specific executable actions, including: Action to keep the VIP of all current servers unchanged; Action to change the VIP address of the first server, which is determined according to the new IP address in step (4d), and assign it to the server ;​ (5c) Construct a reward function according to the network security improvement caused by IP hopping and the network delay brought by IP hopping As follows: ， wherein, represents the network security improvement caused by IP hopping, represents the network latency and interruption caused by IP hopping, and represent the weight factors of security improvement and operation cost, respectively; (6) Policy update: (6a) Given a reward function measuring the benefit of performing an action compared to the average case, used to guide the direction of policy optimization; (6b) Update the policy using the gradient ascent method, update the policy every M steps, and M is greater than or equal to 500; (7) According to the IP hopping model based on deep reinforcement learning after training, the optimal IP hopping strategy is obtained, and the best mobile target defense effect is achieved.

6. The method of claim 5, wherein: The key features in step (2) include source server address , target server address , port number , protocol type , packet length and time stamp .

7. The method of claim 5, wherein: The score is converted into weight by the softmax function in step (3c), which is calculated by the following softmax function : ， wherein, is a score at the time step, is a score at the time step; is the total number of time steps.

8. The method of claim 5, wherein: The advantage function in step (6a) , which measures the benefit of choosing action at time step over the average case, is expressed as follows: ， where, represents the reward received at time step t, represents the discount rate for future rewards, influencing the current value of long-term rewards, represents the value estimate for the current state.

9. The method of claim 8, wherein: The strategy of updating the policy in step (6b) is to construct the following clipped objective function and optimize it to achieve: ， wherein, a policy parameter, is a probability ratio of the new policy to the old policy, is a limit as a function of a range of variation, is a hyperparameter for controlling the clipping range.

10. The method of claim 9, wherein: The optimization is specifically on the policy parameters The update is performed using a gradient ascent method, represented as follows: ， wherein, denotes a learning rate, denotes the gradient of the function with respect to the policy parameters .

Images