Network virus propagation defense method, device and equipment based on reinforcement learning

Abstract

The application provides a network virus propagation defense method, device and equipment based on reinforcement learning. The method comprises the following steps: abstracting a target network into a two-dimensional space Graph form as training data for storage; constructing a reinforcement learning training model, and defining a training environment of the reinforcement learning training model; the training environment of the reinforcement learning model comprises a training environment rule, a server state, an invader attack rule and a training end condition; training the reinforcement learning training model to obtain a reinforcement learning defense model; and deploying the reinforcement learning defense model to a management server, inputting a server state in the target network into the reinforcement learning defense model, and performing offline on a server in an output result. In this way, the reaction speed of both attack and defense sides can be considered without considering the defense or attack strength on the premise that the virus attack can be detected, the server group is protected from a macro perspective, and the whole can still run when a small part of computers in a complex network is invaded.

Description

Technical Field

[0001] This invention generally relates to the field of network security, and more specifically, to methods, apparatus, and devices for defending against network virus propagation based on reinforcement learning. Background Technology

[0002] Reinforcement learning is a machine learning training method that rewards desired behaviors and / or punishes unwanted behaviors. Generally, reinforcement learning agents can perceive and interpret their environment, and through continuous trial and error, learn from mistakes to ultimately derive excellent strategies that achieve their goals. As a popular field in artificial intelligence, reinforcement learning shows great promise in preventing and responding to cyberattacks.

[0003] Reinforcement learning can be used for hacker attack and defense simulations. Currently, the application of reinforcement learning algorithms in cybersecurity is still in the exploratory stage. One validated simulation experiment is the smallest unit of a simulated network: the "Standard Network." This simulation model depicts a virus attack and defense process as follows: the virus begins its intrusion from the START point, successfully attacks and seizes control of the CPU server, then attacks the server containing critical assets, and finally acquires the target resources. AI adjusts the defense and detection strength on different servers to prevent the virus from intruding. This simulation design is representative and demonstrates the feasibility of reinforcement learning on this network.

[0004] However, in reality, the design of defense and detection strength does not reflect actual conditions. This is because if the same defense AI is installed on all servers, the defense strength of all servers will be identical and unchanging; a hacker's attack on a server will always have only two outcomes: failure or success, and the attack is always completed in an instant. Furthermore, existing designs need to consider several aspects such as attack strength and system defense strength, making the defense process logically complex. Summary of the Invention

[0005] According to an embodiment of the present invention, a network virus propagation defense scheme based on reinforcement learning is provided. This scheme, under the premise of being able to detect virus attacks, only considers the reaction speed of both the attacker and defender, without considering the strength of the defense or attack, providing macro-level protection for the server cluster, so that even if a small number of computers in a complex network are compromised, the overall network can still function.

[0006] In a first aspect of the present invention, a network virus propagation defense method based on reinforcement learning is provided. The method includes:

[0007] The target network is abstracted into a two-dimensional graph and stored as training data;

[0008] Construct a reinforcement learning training model and define the training environment of the reinforcement learning training model; the training environment of the reinforcement learning model includes: training environment rules, server status, intruder attack rules, and training termination conditions.

[0009] In the training environment of the reinforcement learning training model, the reinforcement learning training model is trained to obtain a reinforcement learning defense model.

[0010] The reinforcement learning defense model is deployed to a management server. On the management server, the server status in the target network is input into the reinforcement learning defense model, and the servers in the output result are taken offline.

[0011] Furthermore, the training environment rules include:

[0012] A server is randomly assigned to the hacker as the initial entry point for intrusion; and

[0013] Virtual assets are randomly assigned to one or more servers in the target network.

[0014] Furthermore, the server status includes: compromised state, uncompromised state, and offline state; wherein,

[0015] The term "intrusion status" refers to a situation where the server is compromised by an intruder after executing the intruder's attack rules.

[0016] The offline state refers to the server being disconnected from other servers in the network;

[0017] The "unintruded" state means that the server has not been compromised by an intruder and is not offline.

[0018] Furthermore, the intruder attack rules include:

[0019] Starting from the initial point of intrusion, the system proceeds step by step to compromise adjacent servers, with each step involving a certain number of servers.

[0020] Connected compromised servers form an intrusion zone, and the number of servers compromised in each step is positively correlated with the boundary length of the intrusion zone; the boundary length of the intrusion zone is the number of servers in an uncompromised state that the servers in the intrusion zone can compromise.

[0021] When a server is compromised, a reward is awarded, and the reward is negative.

[0022] Furthermore, the training termination condition is:

[0023] The server hosting the virtual assets has been compromised by an intruder, or the boundary length of the compromised area is 0.

[0024] Furthermore, it also includes:

[0025] Networks are filtered based on their complexity, with those having a complexity greater than a preset threshold being selected as target networks.

[0026] Furthermore, the method also includes: storing the target network in the form of a two-dimensional spatial graph through an adjacency matrix or an adjacency list;

[0027] Calculate the memory space occupied by the adjacency matrix. If it is not greater than the remaining free memory space of the system, then store the target network in the form of a two-dimensional graph through the adjacency matrix; otherwise, store the target network in the form of a two-dimensional graph through the adjacency list.

[0028] In a second aspect of the invention, a network virus propagation defense device based on reinforcement learning is provided. The device includes:

[0029] The abstract storage module is used to abstract the target network into a two-dimensional graph form and store it as training data.

[0030] The model building module is used to build a reinforcement learning training model and define the training environment of the reinforcement learning training model. The training environment of the reinforcement learning model includes: training environment rules, server status, intruder attack rules, and training termination conditions.

[0031] The model training module is used to train the reinforcement learning training model in the training environment of the reinforcement learning training model to obtain a reinforcement learning defense model.

[0032] A defense deployment module is used to deploy the reinforcement learning defense model to a management server. On the management server, the server status in the target network is input into the reinforcement learning defense model, and the servers in the output results are taken offline.

[0033] In a third aspect of the invention, an electronic device is provided. The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect of the invention.

[0034] In a fourth aspect of the invention, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the method of the first aspect of the invention.

[0035] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0036] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0037] Figure 1 A diagram illustrating the relationship between network complexity and cost is shown.

[0038] Figure 2 A flowchart of a network virus propagation defense method based on reinforcement learning according to an embodiment of the present invention is shown;

[0039] Figure 3 A schematic diagram illustrating a network abstracted into a two-dimensional spatial graph form according to an embodiment of the present invention is shown;

[0040] Figure 4 A block diagram of a network virus propagation defense device based on reinforcement learning according to an embodiment of the present invention is shown;

[0041] Figure 5 A block diagram of an exemplary electronic device capable of implementing embodiments of the present invention is shown;

[0042] Among them, 500 is an electronic device, 501 is a CPU, 502 is a ROM, 503 is a RAM, 504 is a bus, 505 is an I / O interface, 506 is an input unit, 507 is an output unit, 508 is a storage unit, and 509 is a communication unit. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0045] This invention, under the premise of being able to detect virus attacks, only considers the reaction speed of both the attacker and the defender, without considering the strength of defense or attack, and provides macro-level protection for server clusters, so that even if a small number of computers in a complex network are compromised, the whole network can still operate.

[0046] Since the main advantage of reinforcement learning lies in its ability to find the optimal solution to a problem in complex environments, and this "standard network" has too few servers and an overly simple network structure, it is difficult to prove whether reinforcement learning is effective in complex network structures. Furthermore, this network lacks nodes that utilize techniques such as "honeypots." Therefore, if it is too simple, the embodiments of this invention are not suitable. Thus, this invention is applicable to networks with a certain degree of complexity.

[0047] For a given network, defense can be implemented by writing a state machine or by using a neural network.

[0048] like Figure 1 As shown, when using a state machine for defense, the cost of programming is directly proportional to the network complexity; that is, the more complex the network structure, the more complex the state machine, and the higher the programming cost. However, when using a neural network for defense, the cost does not change with the complexity of the network structure. It can be seen that the costs in both states coincide at a point, namely point 'a'. When the complexity is greater than 'a', this invention is applicable.

[0049] First, it's necessary to determine if the network itself has a certain level of complexity. This means filtering networks based on their complexity, selecting those with a complexity greater than a preset threshold as target networks. The complexity threshold is represented by the complexity at point 'a'. Of course, networks with less complexity than point 'a' can also use the embodiments of this invention; however, using a state machine is less costly and faster.

[0050] For the target network after complexity filtering, the following method can be performed.

[0051] Figure 2 A flowchart of a network virus propagation defense method based on reinforcement learning according to an embodiment of the present invention is shown.

[0052] The method includes:

[0053] S201. Abstract the target network into a two-dimensional graph and store it as training data.

[0054] In reality, servers are interconnected to transmit information. Servers can be located anywhere, and one server may connect to several other servers. To analyze the relationships between them, we need to abstract them into a two-dimensional graph, where each server is treated as a point, and the connections between servers are treated as lines. Figure 3 As shown in the diagram. A, B, C, D, and E each represent a server, forming points in a two-dimensional space.

[0055] As one embodiment of the present invention, the target network in the form of a two-dimensional spatial graph is stored using an adjacency matrix or an adjacency list. Specifically:

[0056] Calculate the memory space occupied by the adjacency matrix. If it is not greater than the remaining free memory space of the system, then store the target network in two-dimensional spatial graph form using the adjacency matrix; otherwise, store the target network in two-dimensional spatial graph form using the adjacency list. The memory space required for the adjacency matrix can be calculated. If the remaining free memory space of the system is greater than or equal to the required memory, then "sufficient memory space" is considered sufficient.

[0057] The storage space required increases with the complexity of the network. Furthermore, the storage space of an adjacency matrix is ​​proportional to the square of the number of servers, while the storage space of an adjacency list is proportional to the number of connections between servers. Therefore, for sparse matrices, adjacency matrices require more space than adjacency lists, especially for irregular network structures, where the size of an adjacency matrix is ​​much larger. However, theoretically, adjacency matrices are more suitable for inputting into neural networks for computation, while adjacency lists require some processing. Therefore, if memory space is sufficient, adjacency matrices should be used for a more convenient training process; if memory is limited, adjacency lists should be used.

[0058] S202. Construct a reinforcement learning training model and define the training environment of the reinforcement learning training model; the training environment of the reinforcement learning model includes: training environment rules, server status, intruder attack rules, and training termination conditions.

[0059] Reinforcement learning (RL) is a field of machine learning that studies how an agent should act in an environment to maximize cumulative rewards. It is one of the three fundamental modes of machine learning, alongside supervised and unsupervised learning. Unlike supervised learning, reinforcement learning does not require labeled input-output pairs or explicit correction of suboptimal behavior. Instead, it focuses on finding a balance between exploration (unknown domain) and development (existing knowledge). Partially supervised RL algorithms combine the advantages of supervised and RL algorithms. The environment is often represented in the form of a Markov Decision Process (MDP), as many reinforcement learning algorithms used in this context employ dynamic programming techniques. The main difference between classical dynamic programming methods and reinforcement learning algorithms is that the latter does not assume knowledge of an exact mathematical model of the MDP; they are designed for large MDPs where exact methods become infeasible.

[0060] Due to its versatility, reinforcement learning has been studied in many disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, multi-agent systems, swarm intelligence, and statistics. In operations research and control literature, reinforcement learning is referred to as approximate dynamic programming or neural dynamic programming. Problems of interest in reinforcement learning have also been studied in optimal control theory, which primarily focuses on the existence and representation of optimal solutions and their exact computational algorithms, rarely addressing learning or approximation, especially in the absence of a mathematical model of the environment. In economics and game theory, reinforcement learning can be used to explain how equilibria arise under bounded rationality.

[0061] Reinforcement learning processes data in the form of Markov chains: a loop of state-action-reward-state. By providing AI with rewards for different state-action pairs, the AI ​​will automatically optimize its behavior. The goal of reinforcement learning is to enable the agent to learn an optimal or near-optimal policy that maximizes the "reward function" or other reinforcement signals accumulated from immediate rewards provided by the user. This is similar to processes found in animal psychology.

[0062] As one embodiment of the present invention, the DQN reinforcement learning algorithm is used for training.

[0063] DQN algorithm:

[0064] Reinforcement learning algorithms can be divided into three main categories: value-based, policy-based, and actor-critic. Common examples include value-based algorithms, such as DQN, which have only a value function network and no policy network, and actor-critic algorithms, such as DDPG and TRPO, which have both a value function network and a policy network.

[0065] DQN algorithm principle:

[0066] DQN reinforcement learning is an iterative process. Each iteration solves two problems: given a policy, find the value function, and update the policy based on the value function.

[0067] As one embodiment of the present invention, the A3C reinforcement learning algorithm is used for training.

[0068] The A3C algorithm, short for Asynchronous Advantage Actor-Critic Algorithm, is a policy gradient algorithm in the field of reinforcement learning. In A3C, the critic can learn in parallel with multiple actors and frequently synchronize global parameters. The gradient is calculated as part of the stabilization training, similar to parallel stochastic gradient descent (SGD).

[0069] For smaller networks, value-based DQN reinforcement learning algorithms can be used for computation, saving time. However, for larger networks, policy-based A3C reinforcement learning algorithms can be used to avoid the large action space; the more complex the network, the more time-efficient it is.

[0070] As an embodiment of the present invention, the training environment rules include:

[0071] A server is randomly assigned to the hacker as the initial entry point for intrusion; and

[0072] Virtual important assets are randomly allocated to one or more servers in the target network.

[0073] Important assets are typically stored on one or a few servers, so they are pre-configured.

[0074] As one embodiment of the present invention, the server status includes: an intruded state, an unintruded state, and an offline state; wherein,

[0075] The term "intrusion state" refers to a server being compromised by an intruder after executing the intruder's attack rules; and a server in an intrusion state can no longer perform offline defense.

[0076] The offline state refers to the server being disconnected from other servers in the network; and a server in an offline state cannot be compromised by intruders.

[0077] The "unintruded state" refers to a server that has not been compromised by an intruder and is not offline. It is evident that a server in this "unintruded state" can still be compromised by an intruder or subjected to offline defense by AI.

[0078] As an embodiment of the present invention, the intruder attack rules include:

[0079] 1) Starting from the initial intrusion point, intrusions are made step-by-step against adjacent servers, with a certain number of servers being compromised at each step. The initial intrusion point is randomly assigned in the training environment rules to simulate a real-world scenario where hackers are unsure which server contains important assets and thus engage in random intrusions.

[0080] 2) Connected compromised servers form an intrusion zone. The number of servers compromised in each step is positively correlated with the boundary length of the intrusion zone. The boundary length of the intrusion zone is the number of servers in an uncompromised state that the servers in the intrusion zone can compromise.

[0081] Since each intruder attacks servers connected to the previously attacked server, the servers attacked by an intruder form an invasion zone, which consists of several attacked servers. The next server an intruder might attack is determined by the outer servers of the current invasion zone; that is, the intruder selects an attack target from the unattended servers connected to the outer servers of the current invasion zone. Each step of the intruder attacks servers, increasing the number of servers within the invasion zone and thus increasing the boundary length of the invasion zone. A larger number of servers attacked in each step results in a greater increase in the boundary length of the invasion zone; that is, the number of servers attacked in each step is positively correlated with the boundary length of the invasion zone. The boundary length of the invasion zone represents the number of unattended servers that the servers within the invasion zone can attack.

[0082] 3) When a server is compromised, a reward is awarded, and the reward is negative.

[0083] In this embodiment, in reinforcement learning, the AI ​​is trained by receiving rewards, which can be positive, negative, or zero. A negative reward represents a penalty. The training reward is proportional to the number of servers compromised in each step. Specifically, during training in the neural network, the training device is a GPU. GPUs use floating-point calculations, and in floating-point calculations, the absolute value of a parameter is no greater than 1. Therefore, the reward value for an AI's incorrect action can be set to -1.

[0084] As one embodiment of the present invention, the training termination condition is:

[0085] The server hosting the virtual assets has been compromised by an intruder, or the boundary length of the compromised area is 0.

[0086] In this embodiment, two training termination conditions are specified as follows:

[0087] 1) The server containing the virtual assets was compromised by an intruder, meaning that important assets on the server were stolen by the intruder.

[0088] 2) The boundary length of the intrusion area is 0, meaning that all servers in the network have been compromised by the intruder, and the intruder cannot proceed with the next step of the intrusion.

[0089] By specifying two training termination conditions, the AI ​​can terminate training according to the training termination conditions.

[0090] S203. In the training environment of the reinforcement learning training model, the reinforcement learning training model is trained to obtain a reinforcement learning defense model.

[0091] In this embodiment, the target network is abstracted into a two-dimensional spatial graph form as training data. Based on the reinforcement learning algorithm, the reinforcement learning training model is trained in a pre-defined training environment. With each training iteration, the probability of important assets being stolen continuously decreases, the probability of the server being completely compromised continuously decreases, and the time at which the intruder is unable to proceed with the next intrusion continuously advances. Ultimately, the number of compromised servers continuously decreases.

[0092] For example, consider time as a coordinate axis. During the first training iteration, an intruder might be unable to proceed with the next intrusion after 100 steps, thus achieving interception. During the second training iteration, the intruder might be unable to proceed with the next intrusion after 50 steps, also achieving interception. During the third training iteration, the intruder might be unable to proceed with the next intrusion after 20 steps, also achieving interception. "Time" represents the number of "steps" required to complete the interception; the fewer the steps, the earlier the time point. In principle, the fewer steps required to complete the interception, the more servers remain in an unintruded state.

[0093] During training, the number of servers the intruder attacks and the number of servers the AI ​​remains offline are controllable at each step. For example, in each step, the intruder attacks 2 servers and the AI ​​remains offline 3 servers. In this case, the speed ratio of the intruder to the AI ​​is 2:3. The number of servers operated by the intruder and the AI ​​at each step represents their reaction speed.

[0094] In the embodiments of the present invention, only the reaction speed of the attacker and defender is considered, without considering the speed of defense or attack, so as to protect the server group from a macro perspective, which is applicable to the virus attack and defense of modern complex networks.

[0095] As an embodiment of the present invention, after training is completed and the reinforcement learning defense model is obtained, it is necessary to test the parameters of the reinforcement learning defense model in a simulated environment, such as the probability of successful interception and the number of servers remaining in an unintruded state after successful interception. By testing these parameters, it is determined whether the reinforcement learning defense model meets the requirements. Here, "interception" means that all servers connected to the compromised server are offline, preventing the intruder from proceeding with further intrusion.

[0096] S204. Deploy the reinforcement learning defense model to the management server. On the management server, input the server status in the target network into the reinforcement learning defense model, and take the server in the output result offline, that is, start the reinforcement learning defense model to run the defense program.

[0097] The management server does not participate in the attack and defense process, will not be attacked by intruders, and will not be taken offline by AI.

[0098] In addition, the main advantage of reinforcement learning is that it can find the optimal solution to a problem in a complex environment. However, the number of servers in this "standard network" is too small and the network structure is too simple. It is difficult to prove whether reinforcement learning is effective in complex network structures. Furthermore, there are no nodes in this network that use techniques such as "honeypots".

[0099] According to embodiments of the present invention, reinforcement learning can be applied to the calculation of virus attack and defense strategies in complex networks. Under the premise that virus attacks can be detected, only the reaction speed of the attacker and defender is considered, without considering the defense or attack intensity, so as to protect the server group from a macro perspective. This is suitable for virus attack and defense in modern complex networks.

[0100] The embodiments of the present invention take into account the reaction speed of intrusion and defense and the randomness of intrusion during the simulation of virus attack and defense; they consider the situation in which a small number of computers in a large and complex network are intruded, but the overall network can still operate; and they simulate more complex networks to maintain the operation of the overall network as much as possible.

[0101] This invention can be extended to server clusters with non-matrix network structures and to the field of multi-agent systems.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0103] The above is an introduction to the method embodiments. The following describes the solution of the present invention further through device embodiments.

[0104] like Figure 4 As shown, the device 400 includes:

[0105] Abstract storage module 410 is used to abstract the target network into a two-dimensional spatial graph form and store it as training data;

[0106] The model building module 420 is used to build a reinforcement learning training model and define the training environment of the reinforcement learning training model. The training environment of the reinforcement learning model includes: training environment rules, server status, intruder attack rules, and training termination conditions.

[0107] The model training module 430 is used to train the reinforcement learning training model in the training environment of the reinforcement learning training model to obtain a reinforcement learning defense model.

[0108] The defense module 440 is deployed to deploy the reinforcement learning defense model to the management server. On the management server, the server status in the target network is input into the reinforcement learning defense model, and the servers in the output result are taken offline.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0110] The acquisition, storage, and application of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] According to embodiments of the present invention, the present invention also provides an electronic device and a readable storage medium.

[0112] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0113] Device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0114] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as methods S201 to S204. For example, in some embodiments, methods S201 to S204 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of methods S201 to S204 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute methods S201 to S204 by any other suitable means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0120] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0121] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0122] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A network virus propagation defense method based on reinforcement learning, characterized in that, include: The target network is abstracted into a two-dimensional graph and stored as training data; Construct a reinforcement learning training model and define the training environment for the reinforcement learning training model; The training environment of the reinforcement learning training model includes: training environment rules, server status, intruder attack rules, and training termination conditions. In the training environment of the reinforcement learning training model, the reinforcement learning training model is trained to obtain a reinforcement learning defense model. The reinforcement learning defense model is deployed to a management server. On the management server, the server status in the target network is input into the reinforcement learning defense model, and the servers in the output result are taken offline. The training environment rules include: A server is randomly assigned to the hacker as the initial entry point for intrusion; and Randomly allocate virtual assets to one or more servers in the target network; The server status includes: compromised state, uncompromised state, and offline state; wherein... The term "intrusion status" refers to a situation where the server is compromised by an intruder after executing the intruder's attack rules. The offline state refers to the server being disconnected from other servers in the network; The "unintruded state" means that the server has not been compromised by an intruder and is not offline. The intruder attack rules include: Starting from the initial point of intrusion, the system proceeds step by step to compromise adjacent servers, with each step involving a certain number of servers. Connected compromised servers form an intrusion zone, and the number of servers compromised in each step is positively correlated with the boundary length of the intrusion zone; the boundary length of the intrusion zone is the number of servers in an uncompromised state that the servers in the intrusion zone can compromise. When a server is compromised, a reward is awarded, and the reward is negative. The training termination condition is: The server hosting the virtual assets has been compromised by an intruder, or the boundary length of the compromised area is 0.

2. The method according to claim 1, characterized in that, Also includes: Networks are filtered based on their complexity, with those having a complexity greater than a preset threshold being selected as target networks.

3. The method according to claim 1, characterized in that, Also includes: The target network in two-dimensional spatial graph form can be stored using an adjacency matrix or adjacency list; Calculate the memory space occupied by the adjacency matrix. If it is not greater than the remaining free memory space of the system, then store the target network in the form of a two-dimensional graph through the adjacency matrix; otherwise, store the target network in the form of a two-dimensional graph through the adjacency list.

4. A network virus propagation defense device based on reinforcement learning, characterized in that, include: The abstract storage module is used to abstract the target network into a two-dimensional graph form and store it as training data. The model building module is used to build a reinforcement learning training model and define the training environment of the reinforcement learning training model. The training environment of the reinforcement learning training model includes: training environment rules, server status, intruder attack rules, and training termination conditions. The model training module is used to train the reinforcement learning training model in the training environment of the reinforcement learning training model to obtain a reinforcement learning defense model. A defense deployment module is used to deploy the reinforcement learning defense model to a management server. On the management server, the server status in the target network is input into the reinforcement learning defense model, and the servers in the output results are taken offline. The training environment rules include: A server is randomly assigned to the hacker as the initial entry point for intrusion; and Randomly allocate virtual assets to one or more servers in the target network; The server status includes: compromised state, uncompromised state, and offline state; wherein... The term "intrusion status" refers to a situation where the server is compromised by an intruder after executing the intruder's attack rules. The offline state refers to the server being disconnected from other servers in the network; The "unintruded state" means that the server has not been compromised by an intruder and is not offline. The intruder attack rules include: Starting from the initial point of intrusion, the system proceeds step by step to compromise adjacent servers, with each step involving a certain number of servers. Connected compromised servers form an intrusion zone, and the number of servers compromised in each step is positively correlated with the boundary length of the intrusion zone; the boundary length of the intrusion zone is the number of servers in an uncompromised state that the servers in the intrusion zone can compromise. When a server is compromised, a reward is awarded, and the reward is negative. The training termination condition is: The server hosting the virtual assets has been compromised by an intruder, or the boundary length of the compromised area is 0.

5. An electronic device comprising at least one processor; and A memory communicatively connected to the at least one processor; characterized in that, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

6. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

Images