Network defense method and device, equipment and medium

Abstract

The invention relates to the technical field of network security, in particular to a network defense method, device and equipment and a medium, which are used for effectively coping with continuously changing network threats and improving the network defense effect. The method comprises the following steps: constructing a deep reinforcement learning DRL model, setting a state space and an action space of the DRL model, and constructing a reward function based on a key defense index of a network system; a flexible action-evaluation SAC algorithm is adopted to train the DRL model; and obtaining current network state information of the network system, inputting the current network state information into the trained DRL model to obtain an optimal network defense measure, and executing the optimal network defense measure. Therefore, by using the deep reinforcement learning technology, a protection mechanism can be dynamically adjusted to cope with continuously changing network threats, and in addition, when potential threats are detected, effective network defense measures are automatically executed, so that the system can turn from passive monitoring to active defense.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a network defense method, device, equipment and medium. Background Art

[0002] In the current era of rapid development of digitalization and the Internet, cybersecurity has become a key issue of global concern. With the advancement of technology, cyber attack methods are constantly evolving and upgrading, making network defense face severe challenges. Cyber ​​attack methods include: malware, phishing attacks, Distributed Denial of Service (DDoS) attacks, and zero-day vulnerabilities.

[0003] In the prior art, firewalls are usually used for network defense. Specifically, by setting access control lists (ACLs) and network communication rules to control the flow of data packets, unauthorized access and potential network threats are prevented, and data flows are divided into allowed and prohibited, providing a physical and logical defense line.

[0004] However, the above method is a static defense method. Static rules cannot flexibly respond to ever-changing network threats, resulting in poor network defense effects. In addition, the management and update of rules are relatively complex, making it difficult to achieve dynamic adaptation. Summary of the invention

[0005] The embodiments of the present application provide a network defense method, apparatus, device and medium to dynamically adjust the protection mechanism to cope with ever-changing network threats and improve the network defense effect.

[0006] The specific technical solutions provided by the embodiments of this application are as follows:

[0007] In a first aspect, a network defense method is provided, the method comprising:

[0008] Build a deep reinforcement learning (DRL) model, set the state space and action space of the DRL model, and build a reward function based on the key defense indicators of the network system;

[0009] The DRL model is trained using the flexible action-criteria SAC algorithm;

[0010] The current network status information of the network system is obtained, and the current network status information is input into the trained DRL model to obtain the optimal network defense measures and execute the optimal network defense measures.

[0011] In a possible embodiment, each state in the state space represents comprehensive information of a network situation of the network system, and the comprehensive information includes but is not limited to: network traffic characteristics, system security level, and system alarm;

[0012] Each action in the action space corresponds to a network defense measure.

[0013] In a possible embodiment, the key defense indicators include: an attack detection rate indicator, an attack false alarm rate indicator, a network system performance indicator, and a security recovery time indicator;

[0014] Based on the key defense indicators of the network system, the reward function is constructed including:

[0015] A reward function is constructed based on the attack detection rate indicator, attack false alarm rate indicator, network system performance indicator, security recovery time indicator, and the weights corresponding to each indicator.

[0016] Through the above method, the reward function combines key defense indicators, which are adjusted through weights to ensure that the agent can achieve the best balance between accuracy, efficiency and stability when optimizing the defense strategy.

[0017] In a possible embodiment, the SAC algorithm is used to train the DRL model, including:

[0018] Initialize the parameters of the state value function, the parameters of the target network, the parameters of the action value function, and the parameters of the policy network;

[0019] Establish an experience replay buffer. A tuple in the experience replay buffer includes: current state, current action, current reward, and next moment state.

[0020] Based on the experience replay buffer, each parameter is iteratively updated until the training end condition is met to obtain the trained DRL model.

[0021] Through the above method, the accuracy of the DRL model can be improved, thereby improving the effect of network defense.

[0022] In a possible embodiment, establishing an experience replay buffer includes:

[0023] The policy network samples a current action in the current state;

[0024] Execute the current action to get the current reward and the next state;

[0025] The current state, current action, current reward, and next-moment state are stored as a tuple in the experience replay buffer.

[0026] Through the above method, the intelligent agent can adjust its behavior in real time in a complex network environment and store the observed state transitions.

[0027] In a possible embodiment, each parameter is iteratively updated based on the experience replay buffer, including:

[0028] Randomly extract sample data from the experience replay buffer;

[0029] Calculate the target state value and update the parameters of the state value function by minimizing the loss function of the state value function using the gradient descent method;

[0030] Calculate the target Q value and update the parameters of the action-value function using the gradient descent method by minimizing the loss function of the action-value function;

[0031] By minimizing the loss function of the policy network, the parameters of the policy network are updated using the gradient descent method;

[0032] Update the parameters of the target network based on the parameters of the state value function and the parameter update speed of the target network.

[0033] Through the above method, the parameters are updated to ensure the accuracy of the state value function, action value function, policy network, and target network to cope with the ever-changing network threats.

[0034] In a second aspect, the present application provides a network defense device, the device comprising:

[0035] The construction module is used to build a deep reinforcement learning (DRL) model, set the state space and action space of the DRL model, and build a reward function based on the key defense indicators of the network system;

[0036] A training module, used to train the DRL model using the flexible action-criteria SAC algorithm;

[0037] The determination module is used to obtain the current network status information of the network system, and input the current network status information into the trained DRL model to obtain the optimal network defense measures and execute the optimal network defense measures.

[0038] In a possible embodiment, each state in the state space represents comprehensive information of a network situation of the network system, and the comprehensive information includes but is not limited to: network traffic characteristics, system security level, and system alarm;

[0039] Each action in the action space corresponds to a network defense measure.

[0040] In a possible embodiment, the key defense indicators include: an attack detection rate indicator, an attack false alarm rate indicator, a network system performance indicator, and a security recovery time indicator;

[0041] When building a reward function based on the key defense indicators of the network system, the building blocks are also used to:

[0042] A reward function is constructed based on the attack detection rate indicator, attack false alarm rate indicator, network system performance indicator, security recovery time indicator, and the weights corresponding to each indicator.

[0043] In a possible embodiment, when the SAC algorithm is used to train the DRL model, the training module is further used to:

[0044] Initialize the parameters of the state value function, the parameters of the target network, the parameters of the action value function, and the parameters of the policy network;

[0045] Establish an experience replay buffer. A tuple in the experience replay buffer includes: current state, current action, current reward, and next moment state.

[0046] Based on the experience replay buffer, each parameter is iteratively updated until the training end condition is met to obtain the trained DRL model.

[0047] In a possible embodiment, when establishing the experience playback buffer, the training module is further used to:

[0048] The policy network samples a current action in the current state;

[0049] Execute the current action to get the current reward and the next state;

[0050] The current state, current action, current reward, and next-moment state are stored as a tuple in the experience replay buffer.

[0051] In a possible embodiment, when each parameter is iteratively updated based on the experience playback buffer, the training module is further used to:

[0052] Randomly extract sample data from the experience replay buffer;

[0053] Calculate the target state value and update the parameters of the state value function by minimizing the loss function of the state value function using the gradient descent method;

[0054] Calculate the target Q value and update the parameters of the action-value function using the gradient descent method by minimizing the loss function of the action-value function;

[0055] By minimizing the loss function of the policy network, the parameters of the policy network are updated using the gradient descent method;

[0056] Update the parameters of the target network based on the parameters of the state value function and the parameter update speed of the target network.

[0057] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the program.

[0058] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0059] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect.

[0060] In the embodiment of the present application, a deep reinforcement learning DRL model is constructed, the state space and action space of the DRL model are set, and a reward function is constructed based on the key defense indicators of the network system, and then the flexible action-evaluation SAC algorithm is used to train the DRL model, and finally the current network state information of the network system is obtained, and the current network state information is input into the trained DRL model to obtain the optimal network defense measures, and the optimal network defense measures are executed. In this way, using deep reinforcement learning technology, it is possible to learn and update defense strategies in real time, dynamically adjust protection mechanisms to cope with changing network threats, and through interactive learning, the system can continuously optimize the generation and adjustment of rules, thereby enhancing adaptability and response speed. In addition, automated processing and analysis of security data reduces dependence on manual operations, optimizes the incident response process, significantly reduces the management burden, and reduces operating costs. In addition, when a potential threat is detected, effective network defense measures are automatically executed, not only warning of abnormalities, but also actively responding, so that the system can switch from passive monitoring to active defense and improve defense capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of an application scenario in an embodiment of the present application;

[0062] Figure 2 This is a flowchart of an implementation of a network defense method provided in an embodiment of the present application;

[0063] Figure 3 Schematic diagram of the DRL model in the embodiment of the present application;

[0064] Figure 4This is a schematic diagram of the model application in the embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of a network defense device provided in an embodiment of the present application;

[0066] Figure 6 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0068] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0069] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0070] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0071] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0072] Distributed Denial of Service (DDoS) is an attack launched by multiple malicious devices, which aims to overload a target network or service with a large number of requests or traffic, thereby paralyzing it and denying service requests from normal users.

[0073] Internet of Things (IoT): is a network of numerous devices and objects that can connect and exchange data by embedding sensors, software and other technologies to achieve intelligent interaction and management.

[0074] Access Control List (ACL): A set of rules used in network devices to specify data flows that are allowed or denied to pass through the device. It enforces security control and traffic management policies by matching the source, destination, and other protocol-specific information of data packets.

[0075] Deep Reinforcement Learning (DRL) is a technique that combines deep learning and reinforcement learning. It uses deep neural networks to process high-dimensional perceptual inputs and learn decision-making strategies that maximize cumulative rewards through interactions with the environment.

[0076] Soft Actor-Critic (SAC) algorithm: The SAC algorithm is a reinforcement learning method based on deep learning, which aims to achieve a balance between exploration and exploitation while maintaining high entropy (i.e. randomness) of the policy. The SAC algorithm is suitable for tasks in continuous action space and has been shown to perform well in multiple benchmarks.

[0077] Before introducing the network defense method provided by the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0078] In the current era of rapid development of digitalization and Internet, network security has become a key issue of global concern. With the advancement of technology, network attack methods are constantly evolving and upgrading, making network defense face severe challenges. Network attack methods include: malware, phishing attacks, DDoS attacks, and zero-day vulnerabilities. In addition, with the widespread application of cloud computing and Internet of Things technologies, the complexity of network defense has further increased, and higher requirements have been placed on the real-time response capability and intelligence level of network defense methods.

[0079] In the existing technology, the following network defense methods are usually used:

[0080] Method 1: Firewall-based network defense method.

[0081] By setting ACL and network communication rules to control the flow of data packets, unauthorized access and potential network threats are prevented, and data flows are divided into permitted and prohibited ones, providing a physical and logical defense line. However, static rules may not be able to adapt to rapidly changing attack patterns, resulting in poor network defense effects, and the management and update of rules may be complex, making it difficult to achieve dynamic adaptation.

[0082] Method 2: Network defense method based on intrusion detection system (IDS).

[0083] Identify abnormal behavior patterns by analyzing network traffic. However, IDS systems can only provide warnings and do not have the ability to prevent attacks, which limits their application in dynamic defense.

[0084] Method 3: Network defense approach based on endpoint protection.

[0085] Malware and virus attacks can be prevented by deploying antivirus software, anti-malware software, and other security tools on each network endpoint. However, this requires installing and updating protection software on each endpoint separately, which has high management costs and limited protection against new unknown viruses.

[0086] Method 4: Network defense method based on security information and event management (SIEM).

[0087] By collecting and analyzing log and event information from the entire network, it provides real-time security warnings and incident responses. SIEM helps organizations centrally manage their security alerts and improve the accuracy of event detection by correlating data from different sources. Although SIEM systems can provide comprehensive security insights, their configuration and maintenance are relatively complex, requiring professional security teams to effectively manage and parse large amounts of data. There may also be large performance overheads, all of which may limit their ability to quickly adapt to new threats.

[0088] Method 5: Network defense method based on zero trust architecture.

[0089] Zero Trust Architecture is an emerging network security concept whose core principle is "never trust, always verify". Under this architecture, all access requests, whether internal or external, must be strictly authenticated and authorized. Zero Trust Architecture uses fine-grained access control and authentication mechanisms to ensure that only fully authenticated and authorized users and devices can access resources. Although Zero Trust Architecture can significantly improve the level of network security protection, the implementation of Zero Trust Architecture requires large-scale reconstruction of existing networks and security policies, which may involve complex technical challenges and high costs. In addition, the strict control of Zero Trust Architecture may affect user experience and work efficiency, especially in environments where frequent access to multiple systems and applications is required.

[0090] In the context of the current rapid development of digitalization, the challenges facing network security are increasing. Although existing network defense methods provide multi-layer protection for network security, they are no longer able to effectively deal with increasingly complex and automated network attacks.

[0091] In view of this, in the embodiments of the present application, a network defense method, device, equipment and medium are provided. A deep reinforcement learning DRL model is constructed, the state space and action space of the DRL model are set, and a reward function is constructed based on the key defense indicators of the network system, and then the SAC algorithm is used to train the DRL model, and finally the current network state information of the network system is obtained, and the current network state information is input into the trained DRL model to obtain the optimal network defense measures, and the optimal network defense measures are executed. In this way, using deep reinforcement learning technology, it is possible to learn and update defense strategies in real time, dynamically adjust protection mechanisms to cope with changing network threats, and through interactive learning, the system can continuously optimize the generation and adjustment of rules, thereby enhancing adaptability and response speed. In addition, the automated processing and analysis of security data reduces the reliance on manual operations, optimizes the incident response process, significantly reduces the management burden, and reduces operating costs. In addition, when a potential threat is detected, effective network defense measures are automatically executed, not only warning of abnormalities, but also actively responding, so that the system can switch from passive monitoring to active defense and improve defense capabilities.

[0092] The preferred implementation modes of the present application are described in detail below with reference to the accompanying drawings.

[0093] See also Figure 1 As shown, it is a schematic diagram of an application scenario in an embodiment of the present application. It includes: a network defense system 110 and an external system 120. Among them, the network defense system 110 includes: a monitoring and recording layer 1101, a first intelligent decision support layer 1102, an interface layer 1103, a second intelligent decision support layer 1104, an intelligent defense engine layer 1105, a strategy and management layer 1106, and a defense execution layer 1107, and the external system 120 includes but is not limited to a wide area network, cloud protection, and terminal protection.

[0094] The monitoring and recording layer 1101 is used for log recording and exception handling, and transmits the collected network data such as network logs and exception data to the first intelligent decision support layer.

[0095] The first intelligent decision support layer 1102 includes a DRL model, and the first intelligent decision support layer 1102 is used to analyze network data, identify threats, and formulate strategies.

[0096] The interface layer 1103 includes: data interface, permission interface and linkage interface, etc.

[0097] The second intelligent decision support layer 1104 includes a DRL model, and the second intelligent decision support layer 1104 is used to execute optimal network defense measures.

[0098] The intelligent defense engine layer 1105 includes: a security prediction module, a security monitoring module, a security defense module, etc.

[0099] The strategy and management layer 1106 includes: strategy management, linkage device management, authority management, effect display, etc.

[0100] The defense execution layer 1107 includes: application blocking, network blocking, terminal blocking, etc.

[0101] See also Figure 2 As shown, it is an implementation flow chart of a network defense method provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0102] Step 20: Build a deep reinforcement learning (DRL) model, set the state space and action space of the DRL model, and build a reward function based on the key defense indicators of the network system.

[0103] Each state in the state space represents comprehensive information about a network situation of the network system, including network traffic characteristics, system security level, and system alarms, etc., which are not limited in the embodiments of the present application. Each action in the action space corresponds to a network defense measure.

[0104] Among them, network traffic characteristics include but are not limited to traffic level, traffic type, traffic source and destination, traffic behavior pattern, protocol information and other characteristics.

[0105] Network defense measures include but are not limited to: adjusting security configuration and activating defense mechanisms, etc. Adjusting security configuration includes but is not limited to firewall rule adjustment (such as modifying ACL, blocking or allowing specific IP addresses or port communications) and security policy updates (such as adding defense against SQL injection or cross-site scripting attacks), etc. Activating defense mechanisms includes but is not limited to: IDS activation (such as enabling or adjusting IDS sensitivity and alarm settings) and initiating isolation procedures (such as isolating detected malicious activities or infected systems).

[0106] See also Figure 3 As shown, it is a schematic diagram of the DRL model in the embodiment of the present application. The DRL model includes an agent and an environment. The agent, as the main decision-making entity, is responsible for analyzing the network status information obtained from the environment and taking corresponding actions to maintain network security. The environment represents the network system operated by the agent, including various network devices, services, data flows, and potential security threats. The agent receives the state from the environment, selects and executes actions based on the state, receives rewards and new states from the environment, and through continuous interaction, enables the agent to accurately evaluate the security of various network behaviors and adjust and optimize its response strategy.

[0107] The state provides the agent with comprehensive information about the network status of the network system, including but not limited to network traffic characteristics, system security level, and system alarms. The state helps the agent understand the immediate security status of the environment and is the basis of its decision-making process. The action is the network defense measure taken by the agent based on the state. The reward is the quantitative feedback on the effect of the agent's action.

[0108] In addition, it is worth mentioning that in order to facilitate intelligent agent processing, the information that constitutes the state (such as network traffic characteristics, system security level, system alarm, etc.) is encoded into numerical values ​​or category labels.

[0109] In an embodiment of the present application, when performing environmental modeling, real-time data is collected from network devices (such as routers, switches, servers, etc.) and security systems (such as firewalls and intrusion detection systems, etc.), features (such as network traffic statistics, abnormal behavior indicators, and security alerts, etc.) are identified and extracted, and these features are used to establish an environmental model to ensure that the environmental model can reflect the real-time status of the network system and potential security threats.

[0110] In the embodiment of the present application, the key defense indicators include: attack detection rate indicator, attack false alarm rate indicator, network system performance indicator, and security recovery time indicator.

[0111] Among them, the attack detection rate indicator represents: the proportion of real attacks successfully detected, the attack false alarm rate indicator represents: the proportion of normal behaviors incorrectly marked as attacks, the attack false alarm rate indicator represents: the impact of network defense measures on network performance, and the security recovery time indicator represents: the time required for the network system to recover from an attack to a normal state.

[0112] In the embodiment of the present application, a reward function is constructed based on an attack detection rate indicator, an attack false alarm rate indicator, a network system performance indicator, a security recovery time indicator, and the weights corresponding to each indicator.

[0113] Specifically, the reward function can be expressed as:

[0114] R(s t ,a t )

[0115] =α×TPR(s t ,a t )-β×FPR(s t ,a t )

[0116] -γ×Impact(s t ,a t )-δ×Recovery(s t ,a t )

[0117] Among them, α is the attack detection rate weight, which is used to reward the agent's ability to detect real attacks. Increasing the value of α can encourage the agent to focus more on accurately identifying and preventing real attacks. β is the false alarm rate weight, which is used to punish the agent's behavior of mistakenly identifying normal behavior as an attack. Increasing the value of β helps reduce interference with normal network activities. γ is the network system performance weight, which is used to punish the network performance degradation caused by network defense measures. Adjusting the value of γ can help find a balance between network security and network performance. δ is the security recovery time weight, which is used to encourage the network defense measures taken by the agent to recover to the normal state before the attack as soon as possible. TPR(s t ,a t ) is the attack detection rate indicator, FPR(s t ,a t ) is the attack false alarm rate indicator, Impact(s t ,a t ) is the network system performance indicator, Recovery(s t ,a t ) is the safety recovery time indicator.

[0118] In this way, the learning process of the agent is guided by a reward function that combines key defense indicators, which are adjusted through weights to ensure that the agent can achieve the best balance between accuracy, efficiency, and stability when optimizing defense strategies.

[0119] Step 21: Use the soft action-criteria SAC algorithm to train the DRL model.

[0120] Network defense can be regarded as a continuous control task. The advantages of the SAC algorithm in processing complex tasks with continuous action space meet the requirements of being applied to dynamic network defense scenarios. The SAC algorithm achieves high efficiency, stability and wide applicability in continuous control tasks by integrating the maximum entropy principle and off-policy learning, especially in dynamic and changing network defense scenarios, showing its special value and advantages. Using the SAC algorithm to train the DRL model can improve the network defense effect.

[0121] Optionally, in the embodiment of the present application, a possible embodiment is provided for training the DRL model using the SAC algorithm, and the following operations are specifically performed:

[0122] Step 210: Initialize the parameters of the state value function, the parameters of the target network, the parameters of the action value function, and the parameters of the policy network.

[0123] Among them, the state value function represents: the expected cumulative reward that can be obtained by adopting a specific strategy under a specific state; the action value function represents: the expected value of the long-term cumulative reward that the intelligent agent can obtain after taking a specific action under a specific state. There are two action value functions in the embodiment of the present application. The target network is a copy of the state value function. The parameters of the target network are not updated in real time during the training process, but are intermittently copied from the state value function. This can provide a relatively stable target state value and reduce the variance and instability during the training process.

[0124] In the embodiment of the present application, the parameter ψ of the state value function is initialized, the parameter ψ' of the target network is initialized, the parameters θ1 and θ2 of the two action value functions are initialized, and the parameter φ of the strategy network is initialized.

[0125] Step 211: Establish an experience replay buffer.

[0126] Among them, a tuple in the experience replay buffer includes: current state, current action, current reward, and next moment state.

[0127] Optionally, in the embodiment of the present application, a possible embodiment is provided for establishing an experience playback buffer, and the following operations are specifically performed:

[0128] Step 2110: The policy network samples a current action in the current state.

[0129] In the embodiment of the present application, the policy network π φ (a t |s t ) in the current state s t Downsample a current action a t .

[0130] Step 2111: Execute the current action and obtain the current reward and the state at the next moment.

[0131] In the embodiment of the present application, the current action a is executed. t , observe and get the current reward r(s t , a t ) and the next state s t+1 .

[0132] Step 2112: Store the current state, current action, current reward, and next moment state as a tuple in the experience replay buffer.

[0133] In the embodiment of the present application, the current state s t 、Current action a t 、Current reward r(s t , a t), and the next moment state s t+1 , as a tuple (s t , a t ,r(s t , a t ), s t+1 ) is stored in the experience replay buffer D.

[0134] This enables the agent to adjust its behavior in real time in a complex network environment and store the observed state transitions.

[0135] Step 212: Based on the experience replay buffer, iteratively update each parameter until the training end condition is met to obtain a trained DRL model.

[0136] In the embodiment of the present application, sample data collection and parameter optimization are continuously repeated until the training end condition is met.

[0137] Optionally, in the embodiment of the present application, a possible embodiment is provided for iteratively updating each parameter based on the experience playback buffer, and the following operations are specifically performed:

[0138] Step 2120: Randomly extract sample data from the experience replay buffer.

[0139] Step 2121: Calculate the target state value, and update the parameters of the state value function by minimizing the loss function of the state value function using the gradient descent method.

[0140] Among them, the target state value is the value of the target state value function, that is, the expected value of the target network.

[0141] In an embodiment of the present application, the action value function is subtracted from the logarithmic probability of the strategy to obtain the target state value, and the loss function of the state value function is obtained based on the difference between the estimate of the state value function and the target state value. The parameters of the state value function are updated using the gradient descent method.

[0142] In the embodiment of the present application, the target state value is expressed as:

[0143] Among them, Q θ is the action value function, which estimates the expected return for a given state and action, logπ φ (a t |s t ) is the policy network taking action a t The natural logarithm probability of is used to introduce an entropy reward to encourage the policy network to explore more actions, which can help balance exploration.

[0144] In the embodiment of the present application, the loss function of the state value function with respect to the parameter ψ is defined as:

[0145]

[0146] in, represents the expected operation, sampling to the experience playback buffer D, where s t Representing states randomly sampled from the experience replay buffer, the expectation operation ensures that the loss is evaluated over the entire dataset, making the learning process more stable and generalizable.

[0147] in, is the mean square error term, used to quantify the current state value function V ψ and the target value function V target The square term ensures the non-negativity of the loss and emphasizes the importance of larger errors. ψ (s t ) is an estimate of the current state value function, calculated using the parameter ψ, which represents the state s t The long-term return expectations under V target (s t ) is an estimate of the target state value function, i.e., the target state value, which is usually calculated using a stable target network to help stabilize the training process and reduce the variance in parameter updates. 1 / 2 is a constant factor that simplifies the subsequent gradient calculation, resulting in a more concise expression for the gradient calculation.

[0148] In this way, by optimizing the above loss function, the parameters of the state-value function can be adjusted, which can better predict the true value of each state, thereby suggesting the overall performance of the strategy.

[0149] In the embodiment of the present application, gradient descent is used to update the parameters of the state value function, and the update process is expressed as:

[0150] Among them, ψ is the parameter of the state value function, η is the learning rate, is the loss function J V (ψ) Gradient with respect to ψ.

[0151] In this way, ψ is adjusted according to the gradient of the loss function to improve the accuracy of the state-value function.

[0152] Step 2122: Calculate the target Q value, and update the parameters of the action-value function using the gradient descent method by minimizing the loss function of the action-value function.

[0153] In an embodiment of the present application, the target Q value is obtained based on the target state value function of the next state and the reward function of the current state, and the loss function of the action value function is obtained based on the difference between the predicted Q value and the target Q value. The parameters of the action value function are updated using the gradient descent method.

[0154] In the embodiment of the present application, the target Q value is expressed as:

[0155] Among them, r(s t ,a t ) is in state s t Next, perform action a t The immediate reward obtained. γ is the discount factor used to discount the current value of future rewards. V ψ' (s t+1 ) is in the next state s t+1 The target value function is usually evaluated using the target network parameters ψ' to provide a stable learning objective.

[0156] In the embodiment of the present application, the loss function of the action value function with respect to the parameter θ is defined as:

[0157]

[0158] in, It represents the expected calculation of the samples in the experience replay buffer D. is the mean square error term, which is used to quantify the difference between the predicted Q value and the target Q value. θ (s t ,a t ) is the estimated value in state s using the current parameter θ t Take action a t The predicted Q value of is the target Q value.

[0159] In the embodiment of the present application, gradient descent is used to update the parameters θ1 and θ2 of the action value function, and the update process is expressed as:

[0160] Among them, θ i is the parameter of the i-th action value function, η is the learning rate, and the learning rate is used to control the step size of parameter update. is the loss function J Q (θ) about parameter θ i gradient.

[0161] Step 2123: Update the parameters of the policy network by minimizing the loss function of the policy network using the gradient descent method.

[0162] In the embodiment of the present application, the loss function of the policy network is defined as:

[0163]

[0164] in, Represents the state s used in the experience replay buffer D t Perform expectation calculation, D KL It is the KL divergence calculation, which is an indicator to measure the difference between two probability distributions and is used to quantify the probability distribution of the policy network π φ The difference between the generated probability distribution and the target probability distribution. π φ (·|s t ) is the policy network defined by the parameter φ in a given state s t The probability distribution of the next action. exp(Q θ (s t ,·)-logZ θ (s t )) defines the value function Q based on the current action θ The adjusted target policy is obtained by exponentially scaling the Q value and subtracting a normalization term. θ (s t ,·) is in state s t The Q value of all possible actions. θ (s t ) is the logarithm of the normalization constant, ensuring that the sum of the Q values ​​after the exponential transformation is 1, thus forming a legal probability distribution.

[0165] In the embodiment of the present application, the parameter φ of the strategy network is updated using gradient descent, and the update process is expressed as:

[0166]

[0167] Among them, φ is the parameter of the policy network, η is the learning rate, and the learning rate is used to control the step size of parameter update. is the loss function J π (φ) Gradient with respect to parameter φ.

[0168] In this way, by minimizing the difference between the strategy and the target strategy, the defense decision is optimized, the defense strategy is learned and updated in real time, and the protection mechanism is dynamically adjusted to cope with the ever-changing network threats.

[0169] Step 2124: Update the parameters of the target network based on the parameters of the state value function and the parameter update speed of the target network.

[0170] In the embodiment of the present application, the updating process of the parameter ψ' of the target network is expressed as: ψ'←τψ+(1-τ)ψ'.

[0171] Among them, ψ' is the parameter of the target network, τ is the soft update coefficient, a decimal close to 0 and greater than 0, which controls the speed of updating the target network parameters. A smaller τ value can ensure that the target network parameters are updated more slowly, which helps smooth learning. ψ is the parameter of the state value function.

[0172] In this way, the parameters of the target network are updated regularly to ensure the continuous optimization and stability of the strategy. By slowly updating the target network parameters ψ', relatively stable parameters can be used in the target value function calculation. A stable target value function helps to provide a reliable target state value, which ensures the accuracy of the optimized state value function.

[0173] Step 22: Obtain the current network status information of the network system, and input the current network status information into the trained DRL model to obtain the optimal network defense measures, and execute the optimal network defense measures.

[0174] In an embodiment of the present application, the current network status information of the network system is obtained, and the current network status information and historical reward feedback are used as input parameters and input into the trained DRL model to obtain the optimal network defense measures corresponding to the current network status information, and then the optimal network defense measures are executed.

[0175] Among them, historical reward feedback is the environment’s quantitative feedback on the effectiveness of the agent’s last network defense measures.

[0176] For example, see Figure 4 As shown, it is a schematic diagram of the model application in an embodiment of the present application, in which the current network status information and historical reward feedback are input into the DRL model as input parameters, and the DRL model outputs the optimal network defense measures.

[0177] Based on the same inventive concept, the present application embodiment provides a network defense device, please refer to Figure 5 , the device comprises:

[0178] A construction module 501 is used to construct a deep reinforcement learning (DRL) model, set the state space and action space of the DRL model, and construct a reward function based on key defense indicators of the network system;

[0179] A training module 502, for training the DRL model using a flexible action-evaluation SAC algorithm;

[0180] The determination module 503 is used to obtain the current network status information of the network system, and input the current network status information into the trained DRL model to obtain the optimal network defense measures and execute the optimal network defense measures.

[0181] In a possible embodiment, each state in the state space represents comprehensive information of a network situation of the network system, and the comprehensive information includes but is not limited to: network traffic characteristics, system security level, and system alarm;

[0182] Each action in the action space corresponds to a network defense measure.

[0183] In a possible embodiment, the key defense indicators include: an attack detection rate indicator, an attack false alarm rate indicator, a network system performance indicator, and a security recovery time indicator;

[0184] When constructing a reward function based on the key defense indicators of the network system, the construction module 501 is also used to:

[0185] A reward function is constructed based on the attack detection rate indicator, attack false alarm rate indicator, network system performance indicator, security recovery time indicator, and the weights corresponding to each indicator.

[0186] In a possible embodiment, when the SAC algorithm is used to train the DRL model, the training module 502 is further used to:

[0187] Initialize the parameters of the state value function, the parameters of the target network, the parameters of the action value function, and the parameters of the policy network;

[0188] Establish an experience replay buffer. A tuple in the experience replay buffer includes: current state, current action, current reward, and next moment state.

[0189] Based on the experience replay buffer, each parameter is iteratively updated until the training end condition is met to obtain the trained DRL model.

[0190] In a possible embodiment, when establishing the experience playback buffer, the training module 402 is further used to:

[0191] The policy network samples a current action in the current state;

[0192] Execute the current action to get the current reward and the next state;

[0193] The current state, current action, current reward, and next-moment state are stored as a tuple in the experience replay buffer.

[0194] In a possible embodiment, when iteratively updating each parameter based on the experience playback buffer, the training module 502 is further used to:

[0195] Randomly extract sample data from the experience replay buffer;

[0196] Calculate the target state value and update the parameters of the state value function by minimizing the loss function of the state value function using the gradient descent method;

[0197] Calculate the target Q value and update the parameters of the action-value function using the gradient descent method by minimizing the loss function of the action-value function;

[0198] By minimizing the loss function of the policy network, the parameters of the policy network are updated using the gradient descent method;

[0199] Update the parameters of the target network based on the parameters of the state value function and the parameter update speed of the target network.

[0200] Based on the above embodiments, see Figure 6 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0201] An embodiment of the present application provides an electronic device, which may include a processor 610 (Center Processing Unit, CPU), a memory 620, an input device 630 and an output device 640, etc. The input device 630 may include a keyboard, a mouse, a touch screen, etc., and the output device 640 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0202] The memory 620 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 610 with program instructions and data stored in the memory 620. In the embodiment of the present application, the memory 620 may be used to store the program of any network defense method in the embodiment of the present application.

[0203] The processor 610 calls the program instructions stored in the memory 620, and the processor 610 is used to execute any network defense method in the embodiments of the present application according to the obtained program instructions.

[0204] Based on the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the network defense method in any of the above method embodiments is implemented.

[0205] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0206] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0209] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A network defense method, characterized in that: include: Construct a deep reinforcement learning (DRL) model, set the state space and action space of the DRL model, and construct a reward function based on the key defense indicators of the network system; The DRL model is trained using a flexible action-criteria (SAC) algorithm; The current network status information of the network system is obtained, and the current network status information is input into the trained DRL model to obtain the optimal network defense measure, and the optimal network defense measure is executed.

2. The method according to claim 1, characterized in that Each state in the state space represents comprehensive information of a network situation of the network system, and the comprehensive information includes but is not limited to: network traffic characteristics, system security level and system alarm; Each action in the action space corresponds to a network defense measure.

3. The method according to claim 1, characterized in that The key defense indicators include: attack detection rate indicator, attack false alarm rate indicator, network system performance indicator, and security recovery time indicator; Then the reward function constructed based on the key defense indicators of the network system includes: A reward function is constructed based on the attack detection rate indicator, the attack false alarm rate indicator, the network system performance indicator, the security recovery time indicator, and the weights corresponding to each indicator.

4. The method according to claim 1, characterized in that The adopting the SAC algorithm to train the DRL model includes: Initialize the parameters of the state value function, the parameters of the target network, the parameters of the action value function, and the parameters of the policy network; Establishing an experience replay buffer, wherein a tuple in the experience replay buffer includes: current state, current action, current reward, and next moment state; Based on the experience replay buffer, each parameter is iteratively updated until the training end condition is met, thereby obtaining the trained DRL model.

5. The method according to claim 4, characterized in that The step of establishing the experience playback buffer comprises: The policy network samples a current action in the current state; Execute the current action to obtain the current reward and the next moment state; The current state, the current action, the current reward, and the next-moment state are stored as a tuple in the experience replay buffer.

6. The method according to claim 1, characterized in that The iterative updating of each parameter based on the experience playback buffer includes: Randomly extracting sample data from the experience replay buffer; Calculate the target state value, and update the parameters of the state value function by minimizing the loss function of the state value function using the gradient descent method; Calculate the target Q value, and update the parameters of the action-value function using the gradient descent method by minimizing the loss function of the action-value function; By minimizing the loss function of the policy network, the parameters of the policy network are updated using the gradient descent method; The parameters of the target network are updated based on the parameters of the state-value function and the parameter update speed of the target network.

7. A network defense device, characterized in that: include: A construction module for constructing a deep reinforcement learning (DRL) model, setting the state space and action space of the DRL model, and constructing a reward function based on key defense indicators of the network system; A training module, used for training the DRL model using a flexible action-criteria SAC algorithm; The determination module is used to obtain the current network status information of the network system, and input the current network status information into the trained DRL model to obtain the optimal network defense measures and execute the optimal network defense measures.

8. The device according to claim 7, characterized in that Each state in the state space represents comprehensive information of a network situation of the network system, and the comprehensive information includes but is not limited to: network traffic characteristics, system security level and system alarm; Each action in the action space corresponds to a network defense measure.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images