Dynamic defense method, system and equipment based on reinforcement learning and medium

Abstract

The invention belongs to the technical field of network security protection, and aims to provide a dynamic defense method, system and device based on reinforcement learning and a medium. The method comprises the steps of collecting state feature information and defense actions of a target network at multiple historical moments, obtaining a state set according to the state feature information at the multiple historical moments, and obtaining an action set according to the defense actions at the multiple historical moments; obtaining an initial attack sample based on the state set and the action set; generating an adversarial attack sample based on the initial attack sample by adopting a generative adversarial network; constructing an initial agent, training the initial agent by adopting the initial attack sample and the adversarial attack sample to enable the initial agent to learn an optimal defense strategy, and further obtaining a trained agent; and deploying the trained intelligent agent into the target network so as to realize dynamic defense of the target network. According to the method, complex network attacks can be effectively resisted, and the security of a network system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security protection, and specifically relates to a dynamic defense method, system, equipment and medium based on reinforcement learning. Background Art

[0002] With the rapid development of information technology, network attacks have become increasingly complex and hidden. Traditional static defense mechanisms such as firewalls and access control have been unable to effectively respond to high-risk attacks such as advanced persistent threats and zero-day attacks. In response to these security challenges, dynamic defense has gradually become an important research direction in the field of network security. The core idea of ​​dynamic defense is to dynamically adjust defense strategies to make it difficult for attackers to predict, thereby improving the security and survivability of network systems.

[0003] In recent years, the progress of reinforcement learning technology has provided new solutions for dynamic defense. Dynamic defense methods based on reinforcement learning can continuously optimize defense strategies and improve defense effectiveness in unknown environments through interaction with attackers. At present, defense methods based on Deep Q-Network (DQN) have emerged. They approximate the Q-value function through Deep Neural Network (DNN), which can reduce storage requirements and improve generalization capabilities, making reinforcement learning applicable to large-scale state sets.

[0004] However, in the process of using the prior art, the inventors found that the prior art has at least the following problems:

[0005] The existing technology has problems of slow convergence and sparse training samples in the network security environment, resulting in low learning efficiency of the defense strategy and difficulty in quickly adapting to new attacks; in addition, the existing technology does not fully consider the adaptability of the attacker. The attacker can exploit reinforcement learning vulnerabilities to generate adversarial samples and bypass the defense strategy, making the existing technology vulnerable to adversarial attacks. Summary of the invention

[0006] The present invention aims to solve the above technical problems at least to a certain extent. The present invention provides a dynamic defense method, system, device and medium based on reinforcement learning.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a dynamic defense method based on reinforcement learning, comprising:

[0009] Collect state feature information and defense actions of the target network at multiple historical moments, and obtain a state set based on the state feature information at multiple historical moments, and obtain an action set based on the defense actions at multiple historical moments;

[0010] Based on the state set and the action set, obtaining an initial attack sample;

[0011] Generate an adversarial attack sample based on the initial attack sample using a generative adversarial network;

[0012] Constructing an initial intelligent agent, and using the initial attack sample and the adversarial attack sample to train the initial intelligent agent so that it learns the optimal defense strategy, thereby obtaining a trained intelligent agent;

[0013] The trained agent is deployed to the target network to achieve dynamic defense of the target network.

[0014] In a possible design, the status feature information includes host status information, historical attack types and historical attack information; the multiple defense actions in the action set include adjusting access control policies, limiting network traffic of suspicious IPs, dynamically modifying system ports, inducing attackers to enter the honeypot system and strengthening intrusion detection system rules.

[0015] In a possible design, the adversarial attack sample is:

[0016]

[0017] Where x is the initial attack sample, ∈ is the preset control disturbance amplitude, sign() is the sign function, is the gradient of the initial attack sample, J(θ, x, y) is the preset attack loss function, θ is the preset model parameter of the attack loss function, and y is the sample label of the initial attack sample.

[0018] In one possible design, the initial intelligent agent is constructed based on a deep Q network in reinforcement learning.

[0019] In one possible design, deploying the trained agent into the target network to implement dynamic defense of the target network includes:

[0020] Performing real-time monitoring on the target network to obtain real-time status characteristic information of the target network;

[0021] Inputting the real-time state feature information into the trained agent so as to obtain an optimal defense action according to the trained agent;

[0022] The optimal defense action is executed to achieve dynamic defense of the target network.

[0023] In one possible design, the method further includes:

[0024] Periodically obtaining the latest attack samples of the target network, and using a generative adversarial network to generate the latest adversarial attack samples based on the latest attack samples;

[0025] The trained agent is retrained based on the latest attack sample and the latest adversarial attack sample to obtain a retrained agent.

[0026] In a second aspect, the present invention provides a dynamic defense system based on reinforcement learning, comprising:

[0027] A sample acquisition module, used to collect state feature information and defense actions of the target network at multiple historical moments, and obtain a state set according to the state feature information at multiple historical moments, and obtain an action set according to the defense actions at multiple historical moments; further used to obtain an initial attack sample based on the state set and the action set; further used to generate an adversarial attack sample based on the initial attack sample using a generative adversarial network;

[0028] A model training module, which is in communication with the sample acquisition module and is used to construct an initial intelligent agent, and train the initial intelligent agent using the initial attack sample and the counter-attack sample so that the initial intelligent agent learns the optimal defense strategy, thereby obtaining a trained intelligent agent;

[0029] A model deployment module is communicatively connected to the model training module and is used to deploy the trained intelligent agent to the target network so as to achieve dynamic defense of the target network.

[0030] In a third aspect, the present invention provides an electronic device, comprising:

[0031] a memory for storing computer program instructions; and,

[0032] A processor is used to execute the computer program instructions to complete the operation of a dynamic defense method based on reinforcement learning as described in any one of the above.

[0033] In a fourth aspect, the present invention provides a computer program product, comprising a computer program or instructions, which, when executed by a computer, implements a dynamic defense method based on reinforcement learning as described in any one of the above.

[0034] In a fifth aspect, the present invention provides a computer-readable storage product, characterized in that instructions are stored on the computer-readable storage product, and when the instructions are run on a computer, a dynamic defense method based on reinforcement learning as described in any one of the above is executed.

[0035] The beneficial effects of the present invention are:

[0036] The present invention discloses a dynamic defense method, system, device and medium based on reinforcement learning, which can effectively resist complex network attacks and improve the security of network systems. Specifically, in the implementation process of the present invention, first, the state feature information and defense actions of the target network at multiple historical moments are collected, and a state set is obtained according to the state feature information at multiple historical moments, and an action set is obtained according to the defense actions at multiple historical moments; then, an initial attack sample is obtained based on the state set and the action set; then, a generative adversarial network is used to generate adversarial attack samples based on the initial attack samples; then, an initial intelligent agent is constructed, and the initial intelligent agent is trained using the initial attack sample and the adversarial attack sample to learn the optimal defense strategy, thereby obtaining a trained intelligent agent; finally, the trained intelligent agent is deployed to the target network to achieve dynamic defense of the target network. Based on this, the present invention can improve the robustness and generalization ability of the defense strategy by using a generative adversarial network to generate adversarial attack samples, so that the defense strategy can dynamically adapt to changes in attack methods, so that the present invention can effectively resist complex network attacks and improve the security of the network system.

[0037] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a dynamic defense method based on reinforcement learning in an embodiment;

[0039] Figure 2 is a module block diagram of a dynamic defense system based on reinforcement learning in an embodiment;

[0040] Figure 3 It is a module block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0042] Embodiment 1:

[0043] This embodiment discloses a dynamic defense method based on reinforcement learning, which can be executed by, but is not limited to, a computer device or a virtual machine with certain computing resources, such as a personal computer, a smart phone, a personal digital assistant, or a wearable device, or by a virtual machine.

[0044] like Figure 1 As shown, a dynamic defense method based on reinforcement learning may include but is not limited to the following steps:

[0045] S1. Collect state feature information and defense actions of the target network at multiple historical moments, and obtain a state set S based on the state feature information at multiple historical moments, and obtain an action set A based on the defense actions at multiple historical moments.

[0046] Specifically, in this embodiment, the state feature information includes host state information, historical attack types and historical attack information; the multiple defense actions in the action set include adjusting access control policies, limiting network traffic of suspicious IPs, dynamically modifying system ports, inducing attackers to enter the honeypot system and strengthening intrusion detection system rules. Specifically, the host state information includes CPU occupancy, memory occupancy and bandwidth occupancy, and the historical attack types are port scanning, brute force password cracking or buffer overflow.

[0047] In this embodiment, the state feature information of any historical moment is represented by an n-dimensional vector. For example, in this embodiment, the state feature information s at the historical moment t is t =[cpu t ,mem t ,bw t ,atk t ], where cpu t ,mem t and bw t They represent the CPU usage, memory usage, and bandwidth usage at the historical time t, respectively. t represents the attack type detected at historical time t; the defense action at historical time t is represented as hist t , after deduplication processing, the defense actions of multiple historical moments can be constructed to obtain an action set A. Specifically, in this embodiment, the multiple defense actions in the action set A include adjusting access control policies (such as modifying firewall rules), limiting network traffic of suspicious IPs (such as blocking attacker traffic), dynamically modifying system ports (to increase the scanning difficulty of attackers), inducing attackers to enter the honeypot system (letting attackers attack fake systems) and strengthening intrusion detection system rules and other defense actions.

[0048] S2. Based on the state set and action set, an initial attack sample is obtained. Specifically, the historical attack behavior is analyzed and the attack features are extracted through the state set and action set to obtain the initial attack sample. In this embodiment, each initial attack sample is a four-tuple of state-action-reward-next state. The initial attack sample at the historical moment t is expressed as (s t ,a t ,r t ,s t+1 ), a t represents the defensive action at historical time t, r t Indicates that defensive action a is adopted at historical moment t t The reward value after s t+1 Represents the state feature information at historical time t.

[0049] S3. Generate adversarial attack samples based on the initial attack samples using Generative Adversarial Networks (GAN). In this embodiment, a generative adversarial network is used to generate difficult-to-detect and more deceptive attack samples, which can be used for subsequent intensive training, thereby improving the adaptability of the defense strategy to new attacks and the generalization ability of the defense system.

[0050] Specifically, in step S3, the adversarial attack sample is:

[0051]

[0052] Where x is the initial attack sample, ∈ is the preset control disturbance amplitude, sign() is the sign function, is the gradient of the initial attack sample, J(θ, x, y) is the preset attack loss function, θ is the preset model parameter of the attack loss function, and y is the sample label of the initial attack sample.

[0053] S4. Construct an initial intelligent agent, and use the initial attack sample and the adversarial attack sample to train the initial intelligent agent so that it learns the optimal defense strategy, thereby obtaining a trained intelligent agent.

[0054] In step S4, the initial agent is constructed based on a deep Q network in reinforcement learning. In this embodiment, during the training of the initial agent, reinforcement learning is used to optimize the defense strategy so that the agent can learn the optimal defense decision when facing an attack sample.

[0055] It should be noted that the deep Q network is capable of processing complex attack features, stable training, defending against new attacks, and adapting to dynamic environments. In this embodiment, the initial intelligent agent is constructed through the deep Q network in reinforcement learning, so that the intelligent agent in this embodiment is suitable for active dynamic defense tasks in network security.

[0056] S5. deploy the trained agent to the target network to achieve dynamic defense of the target network. It should be noted that based on the trained agent, intelligent defense decision-making and dynamic adjustment based on reinforcement learning can be achieved, which can adapt to the network security situation in real time and effectively resist various attack threats.

[0057] In step S5, the trained agent is deployed to the target network to achieve dynamic defense of the target network, including:

[0058] S501. Monitor the target network in real time to obtain real-time status characteristic information of the target network. In this embodiment, by sensing the network security status in real time, it can be ensured that the defense system can quickly respond to potential attack threats.

[0059] S502. Inputting the real-time state feature information into the trained agent so as to obtain the optimal defense action according to the trained agent;

[0060] S503. Execute the optimal defense action to achieve dynamic defense of the target network. In this embodiment, by executing the optimal defense action, the attack behavior can be effectively blocked or mitigated, the defense measures can be dynamically adjusted, and the security and adaptability of the target network can be improved.

[0061] In this embodiment, the method further includes:

[0062] S6. Periodically obtain the latest attack samples of the target network, and use a generative adversarial network to generate the latest adversarial attack samples based on the latest attack samples;

[0063] S7. Retrain the trained agent based on the latest attack sample and the latest adversarial attack sample to obtain a retrained agent.

[0064] In this embodiment, retraining is performed at a preset time period. As another triggering method for retraining, retraining can also be performed based on a detected new attack behavior. During the retraining process, the Q value of the trained agent is updated to adapt it to the new attack sample.

[0065] It should be noted that, according to the above steps S6 and S7, this embodiment periodically obtains data such as the state characteristic information of the target network, and constructs the latest attack samples, and generates the latest adversarial attack samples based on the latest attack samples, so as to achieve periodic incremental learning and retraining of the trained agent to continuously optimize the defense strategy. The retrained agent can be deployed to the target network again, so that the agent can adapt to the latest network environment.

[0066] This embodiment can effectively resist complex network attacks and improve the security of the network system. Specifically, during the implementation of this embodiment, first, the state feature information and defense actions of the target network at multiple historical moments are collected, and a state set is obtained based on the state feature information at multiple historical moments, and an action set is obtained based on the defense actions at multiple historical moments; then, based on the state set and the action set, an initial attack sample is obtained; then, a generative adversarial network is used to generate adversarial attack samples based on the initial attack samples; then, an initial intelligent agent is constructed, and the initial intelligent agent is trained using the initial attack sample and the adversarial attack sample so that it learns the optimal defense strategy, thereby obtaining a trained intelligent agent; finally, the trained intelligent agent is deployed to the target network to achieve dynamic defense of the target network. Based on this, this embodiment can improve the robustness and generalization ability of the defense strategy by using a generative adversarial network to generate adversarial attack samples, so that the defense strategy can dynamically adapt to changes in attack methods, so that this embodiment can effectively resist complex network attacks and improve the security of the network system.

[0067] Embodiment 2:

[0068] This embodiment discloses a dynamic defense system based on reinforcement learning for implementing the dynamic defense method based on reinforcement learning in Embodiment 1; Figure 2 As shown, the dynamic defense system based on reinforcement learning includes:

[0069] A sample acquisition module, used to collect state feature information and defense actions of the target network at multiple historical moments, and obtain a state set according to the state feature information at multiple historical moments, and obtain an action set according to the defense actions at multiple historical moments; further used to obtain an initial attack sample based on the state set and the action set; further used to generate an adversarial attack sample based on the initial attack sample using a generative adversarial network;

[0070] A model training module, which is in communication with the sample acquisition module and is used to construct an initial intelligent agent, and train the initial intelligent agent using the initial attack sample and the counter-attack sample so that the initial intelligent agent learns the optimal defense strategy, thereby obtaining a trained intelligent agent;

[0071] A model deployment module is communicatively connected to the model training module and is used to deploy the trained intelligent agent to the target network so as to achieve dynamic defense of the target network.

[0072] It should be noted that the working process, working details and technical effects of the dynamic defense system based on reinforcement learning provided in this embodiment 2 can be found in embodiment 1 and will not be repeated here.

[0073] Embodiment 3:

[0074] Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a laptop computer, or a desktop computer. The electronic device may be referred to as a user terminal, a portable terminal, a desktop terminal, etc. Figure 3 As shown, the electronic equipment includes:

[0075] a memory for storing computer program instructions; and,

[0076] A processor is used to execute the computer program instructions to complete the operation of a dynamic defense method based on reinforcement learning as described in any one of Example 1.

[0077] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen.

[0078] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the dynamic defense method based on reinforcement learning provided in Example 1 of the present application.

[0079] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0080] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0081] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices through electromagnetic signals.

[0082] The display screen 305 is used to display a UI (User Interface). The UI may include any combination of graphics, text, icons, and videos.

[0083] The power supply 306 is used to supply power to various components in the electronic device.

[0084] Embodiment 4:

[0085] Based on any one of Embodiments 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements a dynamic defense method based on reinforcement learning as described in any one of Embodiments 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0086] Embodiment 5:

[0087] Based on any one of Embodiments 1 to 3, this embodiment discloses a computer-readable storage product, on which instructions are stored, and when the instructions are run on a computer, a dynamic defense method based on reinforcement learning as described in any one of Embodiments 1 is executed. The computer-readable storage product refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memories, USB flash drives, and / or memory sticks, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0088] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic defense method based on reinforcement learning, characterized in that: include: Collect state feature information and defense actions of the target network at multiple historical moments, and obtain a state set based on the state feature information at multiple historical moments, and obtain an action set based on the defense actions at multiple historical moments; Based on the state set and the action set, obtaining an initial attack sample; Generate an adversarial attack sample based on the initial attack sample using a generative adversarial network; Constructing an initial intelligent agent, and using the initial attack sample and the adversarial attack sample to train the initial intelligent agent so that it learns the optimal defense strategy, thereby obtaining a trained intelligent agent; The trained agent is deployed to the target network to achieve dynamic defense of the target network.

2. According to claim 1, a dynamic defense method based on reinforcement learning is characterized in that: The status feature information includes host status information, historical attack types and historical attack information; the multiple defense actions in the action set include adjusting access control policies, limiting network traffic of suspicious IPs, dynamically modifying system ports, inducing attackers to enter the honeypot system and strengthening intrusion detection system rules.

3. The dynamic defense method based on reinforcement learning according to claim 1, characterized in that: The adversarial attack samples are: Where x is the initial attack sample, ∈ is the preset control disturbance amplitude, sign() is the sign function, is the gradient of the initial attack sample, J(θ, x, y) is the preset attack loss function, θ is the preset model parameter of the attack loss function, and y is the sample label of the initial attack sample.

4. The dynamic defense method based on reinforcement learning according to claim 1, characterized in that: The initial intelligent agent is constructed based on a deep Q network in reinforcement learning.

5. The dynamic defense method based on reinforcement learning according to claim 1, characterized in that: Deploying the trained agent into the target network to achieve dynamic defense of the target network includes: Performing real-time monitoring on the target network to obtain real-time status characteristic information of the target network; Inputting the real-time state feature information into the trained agent so as to obtain an optimal defense action according to the trained agent; The optimal defense action is executed to achieve dynamic defense of the target network.

6. The dynamic defense method based on reinforcement learning according to claim 1, characterized in that: The method further comprises: Periodically obtaining the latest attack samples of the target network, and using a generative adversarial network to generate the latest adversarial attack samples based on the latest attack samples; The trained agent is retrained based on the latest attack sample and the latest adversarial attack sample to obtain a retrained agent.

7. A dynamic defense system based on reinforcement learning, characterized in that: include: A sample acquisition module, used to collect state feature information and defense actions of the target network at multiple historical moments, and obtain a state set according to the state feature information at multiple historical moments, and obtain an action set according to the defense actions at multiple historical moments; further used to obtain an initial attack sample based on the state set and the action set; further used to generate an adversarial attack sample based on the initial attack sample using a generative adversarial network; A model training module, which is in communication with the sample acquisition module and is used to construct an initial intelligent agent, and train the initial intelligent agent using the initial attack sample and the counter-attack sample so that the initial intelligent agent learns the optimal defense strategy, thereby obtaining a trained intelligent agent; A model deployment module is communicatively connected to the model training module and is used to deploy the trained intelligent agent to the target network so as to achieve dynamic defense of the target network.

8. An electronic device, characterized in that: include: a memory for storing computer program instructions; as well as, A processor, configured to execute the computer program instructions to complete the operation of a dynamic defense method based on reinforcement learning as described in any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the dynamic defense method based on reinforcement learning as described in any one of claims 1 to 6 is implemented.

10. A computer readable storage product, characterized in that: The computer-readable storage product stores instructions, and when the instructions are executed on a computer, a dynamic defense method based on reinforcement learning as described in any one of claims 1 to 6 is executed.

Images