Method and apparatus for configuring a vulnerable host identification model

Abstract

A method for establishing a model for identifying vulnerable hosts in a network is provided. The method for establishing a vulnerable host identification model may include: a step of collecting a dataset containing host event data generated by the operation of a host in an SDN-based network in response to an attack scenario; a step of designing a reinforcement learning MDP for identifying vulnerable hosts based on a plurality of host event data included in the dataset and a DQN algorithm for performing the reinforcement learning based on the MDP; and a step of establishing a model for identifying vulnerable hosts among the hosts in the network that are determined to be vulnerable to an attack based on the MDP and the DQN algorithm.

Description

Technology Field

[0001] The present disclosure relates to a method and apparatus for establishing a model for identifying vulnerable hosts, and more specifically, to a method and apparatus for identifying hosts vulnerable to cyber attacks among hosts in a network through a model by establishing a model that collects host event data from the operation of a host corresponding to a cyber attack scenario and designs MDP and DQN algorithms based thereon to perform reinforcement learning. Background Technology

[0002] A cyber attack is an act of carrying out an attack on computers, servers, networks, or electronic systems through the Internet or network systems, primarily with the intent of stealing information, disrupting services, or damaging systems.

[0003] Recently, as the frequency of cyber attacks increases, their forms and targets are also becoming more diverse. While cyber attacks in the past targeted specific PCs or systems, they are now expanding into cyber warfare within the contexts of international politics, economics, and military affairs.

[0004] From a defender's perspective, while it is realistically difficult to detect and respond to all cyber attacks in advance, identifying the objectives of each stage of a systematically carried out large-scale cyber attack allows for a more effective response.

[0005] To define cyber attack systems, for example, the Cyber ​​Kill-Chain model of the U.S. military company Lockheed Martin and the ATT&CK (Adversarial Tactics, Techniques, Common Knowledge) model of MITRE can be utilized.

[0006] The Cyber ​​Kill Chain model is a framework for analyzing cyber attacks based on traditional attack flows, primarily used to formulate defense strategies against large-scale Advanced Persistent Threat (APT) attacks. This model views cyber attacks as a chain of interconnected attack phases, assuming that the attack objective can only be achieved if each phase is successfully executed. Therefore, from a defender's perspective, it is necessary to break the attack chain by establishing response procedures for each phase to neutralize or delay the attack. In this regard, defense techniques designed to detect and respond to attack chains in conjunction with Cyber ​​Kill Chain models, such as the Cybersecurity Kill Chain (CSKC) proposed by the U.S. Department of Defense, are currently being researched.

[0007] MITRE’s ATT&CK model is a framework that describes the TTPs (Tactic, Technique, Procedure) performed by a cyber attacker at each stage of an attack. Unlike the cyber kill chain, it systematizes and patterns threatening attack tactics and techniques to improve detection performance against sophisticated attacks. This model initially began by documenting TTP information regarding cyber attacks in Windows OS-based enterprise network environments, and has since evolved into a framework used to identify attacker behavior based on the analysis of consistent attack patterns.

[0008] Recently, research on cyber attacks and defenses utilizing reinforcement learning has been actively conducted in Software-Defined Networking (SDN)-based testbed environments. The research team led by I. Akbari established a testbed using SDN and a simulated network and conducted research to distinguish between cyber attacks and normal scenarios through reinforcement learning. In this study, they proposed the ATMoS (Adaptive Threat Mitigation using SDN and Reinforcement Learning) framework, which aims to rapidly apply network security policies learned from reinforcement learning agents to the network via an SDN controller. Although the study aimed for an effective response to APTs by applying a Fitted Q-learning algorithm during reinforcement learning, it had limitations in that it only presented network policies utilizing SDN and failed to identify vulnerable hosts within the network.

[0009] Meanwhile, Li L.'s research team constructed an emulation testbed using OpenAI's OpenAI Gym environment and MITRE's CALDERA attack simulator, and proposed a method for training attack / defense agent models using reinforcement learning. In particular, training was conducted within the CyGIL (Cyber ​​Game with Intelligent Learners) environment, a reinforcement learning training environment based on MITRE's ATT&CK framework, in a manner where each agent optimized sequential action decisions to maximize rewards. Since this study also focused on attack detection through attack / defense agents, it had the limitation of not describing a method for identifying vulnerable hosts.

[0010] The Guo research team proposed DQSP, a secure routing protocol that considers Quality of Service (QoS) based on Deep Reinforcement Learning (DRL). DQSP is characterized by its ability to guarantee QoS while extracting necessary information from previous traffic and dynamically optimizing routing policies through interaction with the network environment. To achieve this, the study added an agent layer utilizing the Deep Deterministic Policy Gradient (DDPG) algorithm on top of the data layer; this agent layer detects the environment through the control layer and derives a more efficient routing method by taking the environment as input. The study confirmed that DQSP demonstrates a performance improvement of at least 10% compared to the traditional Open Shortest Path First (OSPF) routing protocol in QoS-sensitive network situations. However, there were limitations, such as increased load due to the addition of the DQSP layer and the fact that its response was limited to the network layer.

[0011] The research team of Mu and TY proposed a reinforcement learning algorithm for switch table management. In this study, network traffic flows in a cloud environment were classified into long-duration (elephant) or short-duration (mice) patterns, and network resources were efficiently reallocated according to the patterns. The research focused on using reinforcement learning to explore policies (OpenFlow rules) that can more efficiently manage traffic flows in an SDN environment by defining flow characteristics; however, frequent network reconfiguration led to network instability and posed a risk of disconnection at the endpoint level. Additionally, limitations were pointed out, such as the difficulty in establishing fixed criteria that consider the current network situation to classify all flows according to patterns, and the significant overhead caused by network reconfiguration. The problem to be solved

[0012] One objective of the embodiments of the present disclosure is to provide a method and apparatus for establishing a vulnerable host identification model capable of identifying vulnerable hosts within a network by collecting events occurring at a host in response to an attack scenario and designing and applying reinforcement learning MDP and DQN algorithms based on host event data.

[0013] The technical problems to be solved by the present disclosure are not limited to the technical problems described above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below. means of solving the problem

[0014] A method for establishing a model for identifying vulnerable hosts in a network using an electronic device according to one embodiment may be provided, comprising: a step of collecting a dataset containing host event data generated by the operation of a host in a Software-Defined Networking (SDN)-based network in response to an attack scenario; a step of designing a Markov Decision Process (MDP) for reinforcement learning to identify vulnerable hosts based on a plurality of host event data included in the dataset and a Deep Q-Network (DQN) algorithm for performing reinforcement learning based on the MDP; and a step of establishing a model for identifying vulnerable hosts among the hosts in the network that are determined to be vulnerable to an attack based on the MDP and the DQN algorithm.

[0015] A method for establishing a model for identifying vulnerable hosts in a network using an electronic device according to one embodiment may be provided, wherein host event data includes an attack phase identified from an action performed by a host of an SDN-based network in response to an attack scenario, an attack tactic identified from an action performed by a host of an SDN-based network in response to an attack scenario, an attack technique identified from an action performed by a host of an SDN-based network in response to an attack scenario, an identification number of a host of an SDN-based network determined to be an attack target in an attack scenario, and data regarding a risk level calculated from an action performed by a host of an SDN-based network in response to an attack scenario.

[0016] A method for an electronic device according to one embodiment to set up a model for identifying vulnerable hosts in a network is provided, wherein the MDP is designed based on the state of a model agent defined based on host event data for a model agent performing reinforcement learning, the behavior of the model agent selecting one host identified as the most vulnerable among the hosts of an SDN-based network, and the setting of a reward granted to the model agent when the one host selected by the model agent is identical to the host corresponding to the identification number included in the host event data.

[0017] A method for an electronic device according to one embodiment to set up a model for identifying vulnerable hosts in a network may be provided, characterized in that the reward granted to the model agent is set to be granted in proportion to the risk level included in the host event data.

[0018] A method for establishing a model for identifying vulnerable hosts of a network using an electronic device according to one embodiment may be provided, wherein the DQN algorithm is designed based on a Q-function, a plurality of sampling data sets for updating the Q-function are established, and a sampling data set randomly selected from the plurality of sampling data sets is used for updating the Q-function.

[0019] In a method for establishing a model for identifying vulnerable hosts of a network using an electronic device according to one embodiment, a method may be provided in which each sampling data set included in a plurality of sampling data sets is configured to include data regarding a current state, behavior, reward, and next state identified for a model agent.

[0020] A method for establishing a model for identifying vulnerable hosts of a network using an electronic device according to one embodiment may be provided, wherein the next state of a model agent corresponds to the state of a model agent expected to be identified next after the current state of the model agent, and if there are multiple states of a model agent expected to be identified next after the current state of the model agent, one state randomly selected from the multiple states is determined as the next state of the model agent.

[0021] In a method for establishing a model for identifying vulnerable hosts of a network using an electronic device according to one embodiment, a method may be provided in which data included in each sampling data set is configured to be encoded as a binary vector.

[0022] A method for establishing a model for identifying vulnerable hosts of a network using an electronic device according to one embodiment may be provided, wherein the error function for the DQN algorithm is established based on the Mean Square Error (MSE) between the operation results of the Q-function for the target network and the predict network, and the parameters of the predict network are updated based on the error function.

[0023] A method for establishing a model for identifying vulnerable hosts of a network using an electronic device according to one embodiment may be provided, wherein the Q-function is approximated based on an artificial neural network having one or more layers, and the model agent is configured to estimate the expected value of the total reward obtainable by the model agent according to all actions of the model agent corresponding to all states verifiable for the model agent based on a plurality of sampling data sets.

[0024] A non-transient computer-readable storage medium that can be read is provided, wherein the model setting method for identifying vulnerable hosts using reinforcement learning according to one embodiment is stored in a computer and has a program for executing the program on the computer, the model setting method comprises: a step of collecting a dataset containing host event data generated by a host of an SDN-based network operating in response to an attack scenario; a step of designing an MDP for reinforcement learning to identify vulnerable hosts based on a plurality of host event data included in the dataset and a DQN algorithm for performing reinforcement learning based on the MDP; and a step of setting a model to identify vulnerable hosts among the hosts of a network that are determined to be vulnerable to an attack based on the MDP and the DQN algorithm.

[0025] An electronic device for performing a method of setting a model to identify vulnerable hosts using reinforcement learning according to one embodiment may be provided, comprising: a processor; and one or more memories for storing one or more instructions, wherein, when the one or more instructions are executed, the processor controls the processor to set a model to identify vulnerable hosts among the hosts of a network that are determined to be vulnerable to attacks, by: collecting a dataset containing host event data generated by the operation of a host of an SDN-based network in response to an attack scenario; designing a reinforcement learning MDP for identifying vulnerable hosts based on a plurality of host event data included in the dataset and a DQN algorithm for performing reinforcement learning based on the MDP; and, based on the MDP and the DQN algorithm. Effects of the invention

[0026] The present disclosure has a technical effect in that it can identify vulnerable hosts that are targets of cyber attack scenarios by combining and utilizing TTP information from the MITRE ATT&CK model with reinforcement learning based on the DQN algorithm.

[0027] Furthermore, unlike existing static rule-based attack response methods, the present disclosure has technical advantages in that it can dynamically respond to attacks by determining the current status of the host or network and the risk level of the attack, and allows the policy itself, rather than an operator, to determine and select the scope or target of the policy's application.

[0028] The effects obtainable from the present disclosure are not limited to those described above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0029] FIG. 1 is a diagram showing the configuration of an electronic device according to one embodiment. FIG. 2 shows a flowchart of a method for an electronic device according to one embodiment to set up a model for identifying a vulnerable host. FIG. 3 is a diagram illustrating the operation of an electronic device for establishing a vulnerable host identification model according to one embodiment. FIG. 4 is a diagram illustrating the design of an MDP of reinforcement learning according to one embodiment. FIG. 5 is a diagram illustrating an artificial neural network applied to the DQN algorithm of reinforcement learning according to one embodiment. FIG. 6 is a flowchart illustrating the process of an electronic device according to one embodiment updating the Q-function of a vulnerable host identification model. FIG. 7 is a diagram showing pseudocode regarding a method for an electronic device according to one embodiment to set up a model for identifying a vulnerable host. Specific details for implementing the invention

[0030] Hereinafter, various embodiments of the present disclosure are described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice them. The detailed description disclosed below is intended to explain exemplary embodiments of various examples and is not intended to limit the scope to specific embodiments.

[0031] The terms used in the embodiments have been selected to be as widely used as possible, taking into account their functions in the present disclosure; however, these may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section. Therefore, terms used in the present disclosure should be defined not merely by their names, but based on their meanings and the overall content of the present disclosure.

[0032] The following embodiments are combinations of the components and features of various embodiments in a predetermined form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0033] In the description of the drawings, procedures or steps that could obscure the essence of various embodiments were not described, nor were procedures or steps that could be understood by a person with ordinary knowledge in the relevant technical field described.

[0034] Throughout the specification, when a part is described as "comprising" or "including" a component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. In this document, the singular form of a noun corresponding to an item may be used to include both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or any combination thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from other corresponding components and do not limit the components in any other aspect (e.g., importance or order).

[0035] Each component (e.g., module or program) of the components described in this document may include a singular or multiple entities. According to various embodiments, one or more of the components or operations may be omitted, or one or more other components or operations may be added. Additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as they were performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, or repeatedly; one or more of the operations may be executed in a different order; may be omitted; or one or more other operations may be added.

[0036] As used in this document, the terms "module" or "...part" refer to a unit that processes at least one function or operation, and may include a unit implemented by hardware, software, firmware, or a combination thereof.

[0037] Various embodiments of this document may be implemented as software (e.g., a program or application) comprising one or more instructions stored in a storage medium (e.g., memory) readable by a machine. For example, the processor of the machine may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to at least one called instruction. One or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal, and does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0038] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0039] Previous studies have proposed new frameworks, routing protocols, and network resource reallocation methods using reinforcement learning to improve network security, but they have not described methods for identifying vulnerable hosts within a network. Since vulnerable hosts are a primary target for hacking or malicious software, network security can be improved if they are managed in advance to generate response policies.

[0040] To this end, the method for establishing a model for identifying vulnerable hosts proposed in this disclosure aims to establish a model capable of identifying vulnerable hosts among the hosts of a network by collecting events generated when a host of an SDN-based network operates in response to a simulated attack scenario, and by designing and applying reinforcement learning MDP and DQN algorithms based on multiple host event data.

[0041] FIG. 1 is a diagram showing the configuration of an electronic device according to one embodiment.

[0042] Referring to FIG. 1, the electronic device (100) may include a processor (110) and a memory (120) according to one embodiment. The electronic device (100) illustrated in FIG. 1 is illustrated only with components related to the present embodiment, and it will be understood by those skilled in the art related to the present embodiment that other general-purpose components may be included in addition to the components illustrated in FIG. 1.

[0043] For example, an electronic device (100) may include a communication device comprising one or more transceivers, an input unit, and an output unit. The communication device is a device for performing wired / wireless communication and can communicate with an external electronic device. The external electronic device may be a terminal or a server. In addition, communication technologies used by the communication device may include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc. The input unit may be, for example, a traditional type of keypad or keyboard, a mouse, a microphone for receiving voice signals, a camera, and various other types of input means for detecting or receiving various forms of user input. The output unit may be, for example, a display that outputs images, a speaker that outputs sound, a haptic device that generates vibrations, and various other forms of output means.

[0044] Additionally, at least some of the components within the electronic device (100) may be implemented as an integrated unit or as a single or multiple unit. At least some of the components within the electronic device (100) may be connected to each other via a bus, GPIO (general purpose input / output), SPI (serial peripheral interface) or MIPI (mobile industry processor interface), etc., to exchange data and / or signals.

[0045] The electronic device (100) of FIG. 1 can establish a model for identifying vulnerable hosts in a network using reinforcement learning. Specifically, the electronic device (100) can collect a dataset containing host event data generated by the operation of a host in a Software-Defined Networking (SDN)-based network in response to an attack scenario. Subsequently, the electronic device (100) can design a Markov Decision Process (MDP) for reinforcement learning to identify vulnerable hosts based on multiple host event data included in the dataset, and a Deep Q-Network (DQN) algorithm for performing reinforcement learning based on the MDP. Furthermore, the electronic device (100) can establish a model for identifying vulnerable hosts among the hosts in the network that are determined to be vulnerable to attacks, based on the MDP and DQN algorithms.

[0046] A processor (110) is a component capable of performing operations or data processing regarding the control and / or communication of each component of an electronic device (100). For example, the processor (110) can control the electronic device (100) overall by executing programs stored in memory (120) within the electronic device (100). The processor (110) may be implemented as a CPU (central processing unit), GPU (graphics processing unit), AP (application processor), etc., provided within the electronic device (100), but is not limited thereto. Unless there are special circumstances, in this disclosure, the processor (110) may refer to a set of one or more processors (110).

[0047] The processor (110) may be implemented as a computer or a similar device according to hardware, software, or a combination thereof. In terms of hardware, the processor (110) may be implemented in the form of an electronic circuit that processes electrical signals to perform control functions, and in terms of software, it may be implemented in the form of a program that drives the hardware processor (110). Meanwhile, unless otherwise specifically mentioned in the description below, the operation of the electronic device (100) may be interpreted as being performed by the control of the processor (110). That is, when modules implemented in the electronic device (100) for setting up a vulnerable host identification model are executed, the modules may be interpreted as controlling the processor (110) to perform the operations of the electronic device (100) below.

[0048] Memory (120) is hardware that stores various data processed within an electronic device (100), and memory (120) can temporarily or semi-permanently store data processed and data to be processed in the electronic device (100). For example, data regarding an operating system (OS) for operating the electronic device (100) can be stored in the memory (120) of the electronic device (100). Memory (120) may include random access memory (RAM) such as dynamic random access memory (DRAM) and static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disc storage, hard disk drive (HDD), solid state drive (SSD), or flash memory. Such memory (120) may be provided as an embedded type or a removable type. Additionally, memory (120) may store instructions for the operation of the processor (110). Unless there are special circumstances, in the present disclosure, memory (120) may mean a set of one or more memories (120).

[0049] The method for establishing a vulnerable host identification model of the present disclosure, performed by the electronic device (100) of FIG. 1, may also be implemented by a computer-readable, non-transient computer-readable storage medium (or non-transient recording medium) for the operation. The method for establishing a vulnerable host identification model may be implemented as a software module or algorithm and may be stored on a computer-readable recording medium as computer-readable code or program instructions executable on a processor (110). Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD (Digital Versatile Disc)). Computer-readable recording media may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner. The medium is readable by a computer, stored in memory (120), and can be executed on a processor (110).

[0050] An electronic device (100) according to one embodiment may further include a display (not shown). The display may visually provide various information to the outside under the control of a processor (110). The display may include a touch circuit configured to detect a user's touch or a sensor circuit configured to measure the intensity of the force generated by the touch.

[0051] FIG. 2 shows a flowchart of a method for an electronic device according to one embodiment to set up a model for identifying a vulnerable host.

[0052] In step S210, an electronic device (hereinafter referred to as the ‘electronic device’) (100) that sets a vulnerable host identification model according to one embodiment can collect a data set including one or more host event data generated by at least one host operating in response to one or more attack scenarios simulated on an SDN-based network.

[0053] In one embodiment, the SDN-based network may be a simulated network built by an electronic device (100) and may include at least one of a network emulator, an Open vSwitch (OvS), and an SDN controller.

[0054] In one embodiment, one or more attack scenarios may correspond to attack scenarios simulated through an Attack Scenario Emulator on an SDN-based network. The Attack Scenario Emulator can execute a series of attack steps against hosts of an SDN-based network by reproducing a specific pre-configured attack scenario.

[0055] In one embodiment, an SDN-based network may include at least one host. Each host may be configured with a data collection agent (hereinafter referred to as a "beat agent") to collect host event data generated as the host operates in response to an attack scenario. In one embodiment, each beat agent may collect at least one piece of information among the process, memory usage, and network usage of each host in response to an attack scenario. That is, in step S210, the electronic device (100) collects host event data from at least one beat agent configured on each of the at least one host included in the SDN-based network, and can monitor the status of the SDN-based network through the collected host event data.

[0056] In step S220, an electronic device (100) according to one embodiment can identify an attack event among one or more collected host events and map it to a TTP MAP TABLE.

[0057] According to one embodiment, one or more host event data collected by at least one beat agent can be stored in an event storage unit (hereinafter referred to as 'Event DB') of memory (120).

[0058] In one embodiment, the electronic device (100) can identify some of the host event data stored in the Event DB as attack event data through an attack chain configuration module (Attack Chain Identifier). Specifically, the attack chain configuration module can identify the attack event data among the host event data based on at least one piece of information among the process, memory usage, and network usage of each host included in each host event data.

[0059] In one embodiment, the attack chain configuration module can analyze the identified attack event data. Specifically, the attack chain configuration module can analyze at least one of the attack phase, attack tactic, attack technique, target host or the identification number of the target host, and risk information calculated from the action performed by the host in response to the attack scenario from the attack event data, and at this time, previously analyzed cyber attack scenario data may be utilized. The TTP MAP TABLE in which the electronic device (100) maps the attack event in step S220 can be understood as information set in the form of a table, such as the attack phase, attack tactic, attack technique, target host, and risk information analyzed according to each attack event data, and its specific form may be the same as an example of [Table 1] to be described later.

[0060] In one embodiment, one or more attack event data identified and analyzed by the attack chain configuration module may be mapped to a TTP MAP TABLE and stored in memory (120). The TTP MAP TABLE will be described in detail with reference to FIG. 3.

[0061] In step S230, the electronic device (100) can design MDP and DQN algorithms for reinforcement learning of a vulnerable host identification model. Here, MDP (Markov Decision Process) is a mathematical framework used in reinforcement learning for a model agent to learn optimal behavior while interacting with an environment, and can be established based on definitions of the state, reward, and / or behavior of the model agent. The state and reward of the model agent can be defined based on attack event data analyzed in step S220, and the behavior of the model agent can be the act of selecting a host identified as a vulnerable host among the hosts of the network.

[0062] Meanwhile, the DQN algorithm is an algorithm that approximates the Q-function (action value function) in reinforcement learning situations where the state space and action space are large, and the Q-function in the DQN algorithm can be approximated based on an artificial neural network having one or more layers. The electronic device (100) can set up the model so that the model agent acts in accordance with the learning objective of identifying vulnerable hosts by adjusting the parameters of the artificial neural network applied to the DQN algorithm through reinforcement learning. A specific description and embodiment of the MDP design will be discussed with reference to FIG. 4, which will be described later, and the artificial neural network for applying the DQN algorithm will be described in detail with reference to FIG. 5, which will be described later.

[0063] In step S240, the electronic device (100) can preprocess one or more attack event data analyzed in step S220 and mapped to the TTP MAP TABLE, and create a sampling data set based thereon and store it in memory (120). The memory (120) where the sampling data set is stored is hereinafter referred to as the Replay Memory. Specifically, the preprocessing of the attack event data can be performed through a preprocessing module of the electronic device (100) that includes one or more state parsers and one or more environment parsers.

[0064] In one embodiment, the state parser can extract and encode attack event data mapped to a TTP MAP TABLE and set it as the current state of the model agent. Here, encoding means converting specific data into a binary vector. Additionally, the state parser can receive the next state information from the environment parser and encode it.

[0065] In one embodiment, the environment parser can set the next state of the model agent based on sequence correlation information between each attack event data and transmit it to the state parser. At this time, the sequence correlation can be expressed as environment sequence information, and the environment sequence information is described in detail with reference to FIG. 3, which will be described later.

[0066] In step S240, the preprocessing module of the electronic device (100) can encode not only the current state and the next state, but also the behavior of the model agent and the corresponding reward. Afterwards, the electronic device (100) can produce a sampling data set containing the current state, behavior, reward, and next state information encoded by the preprocessing module, and store it in an experience playback memory.

[0067] In step S250, the electronic device (100) can establish a model for identifying vulnerable hosts by performing reinforcement learning based on a sampling data set and adjusting the parameters of a predict network. Here, the sampling data set may be randomly extracted from an experience replay memory and provided as input data for an artificial neural network.

[0068] The trained model can identify the most vulnerable host among the network hosts based on a parameter-adjusted prediction network, transmit the information of the host to an SDN policy builder, and enable network administrators to design intelligent response policies based on this. The reinforcement learning process for setting up the vulnerable host identification model will be explained in detail with reference to Figure 6, which will be described later.

[0069] FIG. 3 is a diagram illustrating the operation of an electronic device for establishing a vulnerable host identification model according to one embodiment.

[0070] As shown in Figure 3, the reason for establishing an SDN-based reinforcement learning design structure in the present disclosure is that it is necessary to be able to dynamically determine and apply response policies to vulnerable hosts identified through reinforcement learning. In order to design response policies, a machine learning process regarding events occurring in each attack scenario is required, and to this end, it is necessary to simulate attack scenarios, organically monitor the state of the network, and configure environmental variables.

[0071] In FIG. 3, the SDN-based network (310) may be a simulated network built by an electronic device (100) to generate dynamic response policies for vulnerable hosts identified through reinforcement learning. According to one embodiment, the SDN-based network (310) may include at least one of a network emulator, an Open vSwitch (OvS), and an SDN controller. Here, the network emulator may perform the role of creating a virtualized network environment, and for example, Mininet may be used as the network emulator. Additionally, the SDN controller may perform the role of applying policies to the network and may be configured, for example, as an Open Network Operating System (ONOS).

[0072] In one embodiment, the SDN-based network (310) may include an Attack Scenario Emulator. The Attack Scenario Emulator can execute a series of attack steps on the hosts of the SDN-based network (310) by reproducing a specific pre-configured attack scenario.

[0073] In one embodiment, the SDN-based network (310) may include at least one host. Additionally, a beat agent may be configured on each host to collect host event data generated as the host operates in response to an attack scenario. In one embodiment, each beat agent may collect at least one piece of information among the process, memory usage, and network usage of each host in response to an attack scenario.

[0074] The attack event identification unit (320) stores one or more host event data collected from at least one beat agent and can identify attack event data among them.

[0075] According to one embodiment, the electronic device (100) can store one or more host event data collected by at least one beat agent in the Event DB of the attack event identification unit (320). Additionally, the electronic device (100) can identify some of the one or more host event data stored in the Event DB as attack event data by using the Attack Chain Identifier of the attack event identification unit (320). Specifically, the attack chain Identifier can identify attack event data based on at least one piece of information among the process, memory usage, and network usage of each host included in each host event data.

[0076] In one embodiment, the attack chain configuration module can analyze identified attack event data. Specifically, the attack chain configuration module can map the attack event data to a specific attack stage of the cyber kill chain model. Furthermore, the attack chain configuration module can label at least one of the attack tactic, attack technique, and target host information from the identified attack event data, or calculate a risk score for the corresponding attack event. Here, the attack tactic and attack technique may be labeled based on the ATT&CK model, and the target host information may be labeled with a pre-configured host identification number. In one embodiment, the analysis by the attack chain configuration module may be performed based on previously analyzed cyber attack scenario data.

[0077] The TTP MAP TABLE (330) is a set of attack event data identified and analyzed by the attack event identification unit (320) and can be expressed in the format shown in [Table 1] below.

[0078] [Table 1]

[0079]

[0080] Below, each component of the TTP MAP TABLE (330) is described in detail. An attack step may be information regarding the order or timing of specific attack events occurring within an attack scenario. In one embodiment, an attack step may correspond to each attack step defined in a cyber kill chain model.

[0081] Attack Tactic, Attack Tactic Name, and Attack Technique are information corresponding to the Attack Tactic and Attack Technique of the ATT&CK model, and one Attack Tactic Name may correspond to one Attack Tactic. In one embodiment, the Attack Tactic may be information indicating which stage of the attack chain the corresponding attack event belongs to within the overall attack scenario. The Attack Technique may include information regarding an attack technique performed by a cyber attacker under a specific Attack Tactic, and may include sub-techniques depending on the attack technique.

[0082] Source Host information is information indicating which host the attack event occurred on, and a pre-configured identification number for each host may be the value corresponding to the Source Host information. Score information is a value calculated by considering the frequency of occurrence of the attack event and the importance at the time of occurrence of the attack event, and in one embodiment, the score information may be a normalized value between 0 and 100.

[0083] The preprocessing module (340) can determine the current state and next state of the model agent based on the TTP MAP TABLE (330) and produce a sampling data set. In one embodiment, the preprocessing module (340) may include at least one state parser and at least one environment parser.

[0084] The state parser can extract and encode one attack event data corresponding to one row in the TTP MAP TABLE (330). In one embodiment of the present disclosure, the binary vector encoded by the state parser can be defined as the current state of a model agent in a reinforcement learning MDP. Additionally, the state parser can receive and encode next state information from the environment parser.

[0085] The environment parser can set the next state of the model agent based on sequence correlation information between each attack event data in the TTP MAP TABLE (330). In one embodiment, the sequence correlation information may be defined as a correlation dictionary data structure that defines the next event to occur when a specific attack event occurs, and there may be one or more next states associated with a current state. In one embodiment, the sequence correlation may be set based on previously analyzed attack scenarios and may be expressed as environment sequence information as shown in [Table 2] below.

[0086] [Table 2]

[0087]

[0088] Below, the components constituting the environment sequence information of [Table 2] are described in detail. In [Table 2], column 1 (Init Step) corresponds to the attack step (Step) information of the TTP MAP TABLE (330) and may correspond to a single state in which the model agent is placed in the MDP. Column 2 (Relation) may be information indicating the number of next states associated with the current state. Steps 1 through 5 may represent attack step information sequentially connected to the Init Step. At this time, the number of Steps connected to the Init Step may be determined based on the Relation information.

[0089] In one embodiment, if there are 2 or more Relations, one of two or more next states associated with the current state may be randomly selected and set as the next state, wherein the next state may be defined as the state of the model agent expected to be identified next after the current state. For example, if the Init Step is 1A1, the next state is one of 3C1 of Step 1, 1A3 of Step 2, 1B1 of Step 3, 2B1 of Step 4, and 3B3 of Step 5. Depending on the embodiment, if there are multiple next states identified for a specific current state, a randomly selected next state may be managed to be set as the next state for the specific current state. Here, if 1B1 of Step 3 is set as the next state, when setting the next state, 1B1 becomes the Init Step, and since there is only one associated next state, 1B2 may be determined as the next state. When the next state is determined in this manner, the environment parser can transmit the next state information to the state parser.

[0090] In one embodiment, the preprocessing module (340) can encode not only the current state and the next state, but also the behavior of the model agent and the corresponding reward. That is, the data included in the sampling data set representing the state, behavior, reward, next state, etc. in this disclosure can be understood as basically corresponding to encoded data. In one embodiment, the preprocessing module (340) of the electronic device (100) may use a one-hot encoding technique to encode the state and behavior, which are unique variables. That is, the encoding of the model agent state performed by the state parser included in the preprocessing module (340) may be one-hot encoding. Here, the one-hot encoding technique refers to an encoding method that sets categories for unique values ​​and converts each unique value into a binary vector.

[0091] In one embodiment, the preprocessing module (340) may produce a sampling data set. In one embodiment, the sampling data set may be configured to include data regarding the current state, action, reward, and next state. In this case, data regarding the current state may be obtained through a state parser, and data regarding the next state may be obtained through an environment parser. The action may be selected based on the result value output when the current state is input into a DQN artificial neural network, and the reward may be determined based on the risk level of the attack event data. Specifically, the value of the reward may be determined to be proportional to the risk level. In one embodiment, the action and reward included in the sampling data set may be configured based on previously analyzed attack scenario information.

[0092] The experience replay memory (350) stores a sampling data set produced by the preprocessing module (340) and can provide the sampling data set during reinforcement learning. Specifically, the experience replay memory (350) is part of the memory (120) and can store a sampling data set up to a preset maximum size. In one embodiment, if the number of sampling data sets stored in the experience replay memory (350) is less than the preset maximum size, the electronic device (100) can store the sampling data set produced by the preprocessing module (340) in the experience replay memory (350).

[0093] The reinforcement learning unit (360) performs reinforcement learning of the model based on a sampling data set extracted from the experience replay memory (350). The reinforcement learning unit (360) includes at least one model agent for performing reinforcement learning, and the reinforcement learning unit (360) can set up a model that identifies vulnerable hosts by giving rewards according to the actions of the model agent based on the designed MDP and adjusting the parameters of the prediction network through reinforcement learning. During reinforcement learning, a sampling data set of a pre-set batch size can be extracted from the experience replay memory (350) and used for learning.

[0094] In one embodiment, a Deep Q-Network (DQN) algorithm may be utilized as the reinforcement learning algorithm of the reinforcement learning unit (360). The DQN algorithm may include a target network and a prediction network, and parameter updates of the prediction network may be performed by utilizing an error function based on the Mean Square Error (MSE) between the operation results of the Q-functions of the two networks. Specific embodiments related to the Q-function update process of the vulnerable host identification model will be described in detail with reference to FIG. 6, which will be described later.

[0095] FIG. 4 is a diagram illustrating the design of an MDP of reinforcement learning according to one embodiment.

[0096] As previously mentioned, an MDP can be understood as a framework that enables a model agent to derive optimal behavior in a given environment. Through MDP design, value can be assigned to the agent's behavior by defining the reward or penalty to be received when the model agent takes action in a given environment, and the model agent can learn the best policy to maximize future rewards on its own. In other words, a reinforcement learning problem can be formally defined from an MDP, and decision rules for the model agent can be established. As shown in FIG. 4, a reinforcement learning MDP can be composed of elements of state, action, and reward, and the definitions for each element may be as shown in Equations 1 to 3 described later (400). Additionally, depending on the case, a reinforcement learning MDP may be configured to include additional elements such as state transition probabilities and discount factors.

[0097] The state is information about the environment in which the model agent is running, that is, it indicates what state the model agent is in within the current environment. In one embodiment of the present disclosure, the state of the model agent may be defined from a set of components that make up one row of the TTP MAP TABLE (330). That is, it may be expressed as [Equation 1] below.

[0098] [Mathematical Formula 1]

[0099]

[0100] In [Mathematical Formula 1], srcHost ... is an element corresponding to the Source Host of the TTP MAP TABLE (330), and the remaining components have the same name as the corresponding components of the TTP MAP TABLE (330). In one embodiment, the state of the model agent may refer to the state information of the model agent encoded as a binary vector in the state parser of the preprocessing module (340). That is, the state of the model agent may be obtained by converting each component of [Equation 1] into a binary vector according to a pre-set encoding method.

[0101] Behavior defines the interaction between the model agent and the environment in a specific state. Given that the purpose of the present disclosure is to configure a vulnerable host identification model, it may be preferable for the behavior of the model agent to be defined as identifying and selecting the host determined to be the most vulnerable among several hosts in the network when each attack event data is given as the state of the model agent. This can be expressed mathematically as [Equation 2].

[0102] [Mathematical Formula 2]

[0103]

[0104] At this time, [Mathematical Formula 2] hosts represents the number of hosts in the network.

[0105] Reward refers to the value received from the environment when a model agent takes a specific action in a specific state. Since reward is associated with the value judgment regarding the model agent's actions, it must be set in a manner consistent with the purpose of reinforcement learning. Considering the purpose of this disclosure, a high reward should be granted if the model agent correctly identifies a vulnerable host, and a low reward should be granted if it selects a non-vulnerable host as a vulnerable host. In this case, whether the model agent has correctly identified a vulnerable host depends on the model agent's current state ( )of srcHost It can be determined by referring to the value. That is, the current state ( Model agent behavior in ) ) is the current state( )of srcHost value( If it is equal to ), it is determined that the model agent has correctly identified the vulnerable host, and a high reward can be granted. In one embodiment, the reward granted when the model agent correctly identifies the vulnerable host is the current state ( It can be determined by the risk (score) value of ).

[0106] Meanwhile, if the identification of a vulnerable host fails, a lower reward or penalty may be imposed compared to when the identification is successful. In one embodiment of the present disclosure, no reward or penalty was imposed when the model agent failed to identify a vulnerable host. That is, if the reinforcement learning agent correctly identified and selected a vulnerable host for a specific attack event, a reward proportional to the risk level of the attack event was imposed on the model agent, and it was defined so that no reward was imposed if the identification of a vulnerable host failed. This can be expressed as shown in [Equation 3] below.

[0107] [Mathematical Formula 3]

[0108]

[0109] The rewards or penalties given when the model agent succeeds or fails in identifying vulnerable hosts are not limited to the manner of [Equation 3], and may be given in various ways depending on the embodiment.

[0110] In summary, the MDP in Fig. 4 can be understood as being designed through settings such as the state of the model agent defined based on the content of the host event data, the action of the model agent selecting one host identified as the most vulnerable among the hosts, and the reward given to the model agent when the one host selected by the model agent is identical to the host corresponding to the identification number of the host included in the host event data.

[0111] A DQN algorithm for performing reinforcement learning based on an MDP designed as shown in one example of FIG. 4 can be designed, and an example of the DQN algorithm and the artificial neural network applied thereto can be as shown in FIG. 5.

[0112] FIG. 5 is a diagram illustrating an artificial neural network applied to the DQN algorithm of reinforcement learning according to one embodiment.

[0113] Deep Q-Network (DQN) is a reinforcement learning algorithm that addresses the problem of requiring significant memory and computational resources to handle large state and action spaces—such as in real-world environments where environmental variables and models are unknown—by utilizing a table-based Q-function to learn the values ​​of combinations of an agent's state and action.

[0114] DQN can be designed to be applicable to large state spaces and large action spaces by incorporating deep learning into reinforcement learning and approximating the Q-function with an artificial neural network. In one embodiment of the present disclosure, the artificial neural network receives the state of a model agent based on a plurality of sampling data sets and outputs a Q-value for each action that the model agent can take. The Q-value may represent the expected value of the total reward corresponding to the model agent's action, and the Q-function may be configured to estimate the expected value of the total reward that the model agent can obtain according to all actions of the model agent that can be verified based on the plurality of sampling data sets.

[0115] The DQN algorithm has two key features: Experience Replay utilizing experience replay memory and the use of a Target Network that provides correct answers for the model agent's actions.

[0116] Experience replay can be performed by creating a sampled data set of experiences performed by an agent and storing it in an experience replay memory, and by randomly extracting the sampled data set from the experience replay memory at the time of learning and utilizing it for learning. Since the sampled data set is randomly extracted during experience replay, it is possible to prevent falling into a local minimum from biased learning that depends on specific situations, and by evenly reflecting various experiences, the variance of learning is reduced and volatility is decreased, thereby improving the stability of learning. In one embodiment of the present disclosure, the electronic device (100) can perform experience replay by randomly extracting a sampled data set of a preset batch size from the experience replay memory (350) and utilizing it for learning a vulnerable host identification model.

[0117] The target network is a network separate from the prediction network where reinforcement learning is performed, and it serves the role of calculating the target Q-value, which is the Q-value assuming the model agent performs the optimal action. The target Q-value serves as a reference when updating the prediction network's Q-function; since the parameters of the target network are updated by referencing the parameters of the prediction network at specific intervals, it can provide a fixed correct answer during that period, thereby resolving the instability of the learning objective, which is a problem of conventional Q-learning. In one embodiment of the present disclosure, the electronic device (100) can establish a vulnerable host identification model by updating parameters of the prediction network using an error function defined based on the mean square error (MSE) of the target Q-value and the Q-value of the prediction network.

[0118] FIG. 5 illustrates the structure of an artificial neural network applied to the DQN algorithm (500), and in one embodiment, the input data of the artificial neural network may be a sampling data set extracted from an experience replay memory (350). Specifically, the sampling data set may be extracted from the experience replay memory (350) in a pre-set batch size and used to update the parameters of the model's prediction network. Here, the parameters may include weight parameters.

[0119] In one embodiment, the input data of the artificial neural network may be the state of a model agent preprocessed by a state parser. In this case, the artificial neural network can determine the model agent's next action by calculating a Q-value for each action of the model agent in that state. That is, the input data of the artificial neural network may vary depending on whether the model is in a training situation or an inference situation.

[0120] In FIG. 5, the artificial neural network may have one or more layers. For example, the artificial neural network illustrated in FIG. 5 includes an input layer (not shown), two hidden layers, and an output layer, and each hidden layer has 24 nodes. In one embodiment, the hidden layer of the artificial neural network may use a ReLU activation function. In one embodiment, the output layer of the artificial neural network may estimate the probability that each host is a vulnerable host and, after passing through a softmax activation function, output the host determined to be the most vulnerable. At this time, the model agent may decide on an action based on the host information output by the artificial neural network.

[0121] In one embodiment, the number of nodes in the output layer may be equal to the number of hosts in the SDN-based network (310), and each node may correspond to one host. For example, node 1 may be a node corresponding to a host assigned identification number 1. After passing through a softmax activation function, data related to the probability that each host is a vulnerable host may be output, and a vector representing the host with the highest probability as 1 and the remaining hosts as 0 may be configured to be output as the final result.

[0122] In one example of Figure 5, a sampling data set according to a fixed batch size sequentially passes through the input layer, hidden layer, and output layer of an artificial neural network, and as a final output result, it can be confirmed that host number 3 among the 5 hosts is identified as the most vulnerable host.

[0123] FIG. 6 is a flowchart illustrating the process of an electronic device according to one embodiment updating the Q-function of a vulnerable host identification model.

[0124] In one embodiment, the Q-function in the vulnerable host identification model may correspond to a function utilized for the prediction network. The setup of the vulnerable host identification model can be performed by running a training episode for a preset number of epochs and continuously updating the Q-function of the prediction network, and FIG. 6 can be understood as illustrating the model training process within a single episode.

[0125] Methods for updating the Q-function of a prediction network include the Q-learning algorithm-based Q-function update method and the loss function-based Q-function update method used in DQN.

[0126] The Q-function update formula for reinforcement learning utilizing the Q-learning algorithm is expressed as follows [Equation 4].

[0127] [Mathematical Formula 4]

[0128]

[0129] In [Mathematical Formula 4] above is the learning rate, represents the discount rate, is the current state ( In ) action( This is the reward when ) is taken. To explain the meaning of [Mathematical Equation 4] in detail, it means that the current Q-value is added to the current Q-value by reflecting the error between the correct answer and the current Q-value by the learning rate. Here, the correct answer is the maximum Q-value in the state after the discount rate is applied and the current state ( In ) action( The reward when ) is taken ( It means the sum of ).

[0130] In the DQN algorithm, the Q-function is approximated using an artificial neural network, and the Q-function can be updated by adjusting the weight parameters of the artificial neural network based on the error function. Below, the process of updating the Q-function of the error function-based prediction network is explained in detail with reference to Fig. 6.

[0131] In step S610, the electronic device (100) extracts sampling data sets of a preset batch size from the experience playback memory (350). At this time, the extraction of sampling data sets may be random.

[0132] In step S620, the electronic device (100) can set the Q-value of the target network based on each extracted sampling data set and set a loss function based on the set target Q-value. Specifically, the loss function can be set based on the Mean Square Error (MSE) of the target Q-value and the Q-value of the prediction network, and can be expressed as shown in [Equation 5] below.

[0133] [Mathematical Formula 5]

[0134]

[0135] In [Mathematical Formula 5] is the parameters of the target network, represents the parameters of the prediction network, and in one embodiment, the network parameters may include weight parameters.

[0136] In step S630, the electronic device (100) can update the parameters of the prediction network based on an error function. For example, the Q-function of the prediction network can be updated using gradient descent, which repeats the process of calculating the gradient of the error function and adjusting the weight parameters until the gradient becomes a sufficiently small value. At this time, the gradient of the error function can be calculated through backpropagation, which propagates the error by moving back through the neural network.

[0137] In one embodiment, S620 and S630 may be repeated as many times as the number of extracted sampling data sets, i.e., the batch size. For each learning process, different sampling data sets may be used for learning according to a pre-set sequence.

[0138] In step S640, the electronic device (100) can check whether the training termination condition of the training episode is satisfied. At this time, the total sum of all rewards of the entire dataset can be defined as a hard cap, and the training episode termination condition can be set based on the hard cap. Specifically, the hard cap itself can be set as a direct termination condition, or a soft cap corresponding to a portion of the hard cap can be set as a termination condition. For example, 80% of the hard cap can be defined as a soft cap, and the training episode can be terminated if the total reward earned by the model agent exceeds the soft cap. If the training termination condition is not satisfied, the electronic device (100) can return to step S610 and perform training again.

[0139] FIG. 7 is a diagram showing pseudocode regarding a method for an electronic device according to one embodiment to set up a model for identifying a vulnerable host.

[0140] First, the method for an electronic device (100) to set up a model for identifying vulnerable hosts may include a process (S710) of initializing various elements related to learning. The elements initialized here include a learning rate ( ), discount rate( ), search rate( ), search reduction rate( ), minimum search rate( Hyper-parameters such as ), batch size (BS), etc., may be included. In addition, the total reward ( ), maximum size of experience playback memory (350) ), hard caps and soft caps for determining learning termination conditions may be initialized.

[0141] In step S710, the attack scenario event data list (D) may refer to the TTP MAP TABLE (330) information of [Table 1] containing one or more attack event data. In this case, one row of the TTP MAP TABLE (330) may correspond to one attack event data. Additionally, the environment correlation data list (E) may correspond to the environment sequence information of [Table 2].

[0142] Also, in step S710, the weight parameters of the prediction network ( ) is initialized randomly, and the target network's weight parameters ( ) can be set to be the same as the weight parameters of the prediction network.

[0143] In one embodiment according to the pseudocode of FIG. 7, for setting up a vulnerable host identification model, training episodes are repeated for an epoch count (C), and in each training episode, steps S720 to S760 below may be performed for the number of attack event data included in the attack scenario event data list (D) unless the training termination condition is satisfied.

[0144] In step S720, the electronic device (100) can randomly select an attack event that has not yet been selected from the attack scenario event data list (D). Subsequently, the selected attack event can be set as the current state of the model agent and encoded through the preprocessing module (340). For example, the state can be encoded in a state parser included in the preprocessing module (340).

[0145] In step S730, the model agent uses a pre-set exploration rate ( Depending on ), exploration or action based on the Q-function of the current prediction network can be performed. Here, the exploration rate ( ) has a value between 0 and 1 and can represent the probability of the model agent performing exploration. That is, the exploration rate ( Depending on ) Perform exploration with a probability equal to , and (1- With a probability equal to ), the action that maximizes the Q-value of the prediction network in the current state can be selected. In this case, exploration may mean that the model agent randomly selects an action regardless of the prediction network's Q-function.

[0146] In step S740, the electronic device (100) can calculate a reward based on the actions of the model agent according to the designed MDP and add the calculated reward to the reward total. Here, the reward total can be used to determine whether the learning termination condition of each learning episode is satisfied. In one embodiment, if the reward total exceeds a preset soft cap, the corresponding learning episode can be terminated and a new learning episode can be started. For example, in the pseudocode of FIG. 7, the soft cap is set to 80% of the hard cap, so if the reward total exceeds the soft cap value initialized in step S710, the corresponding learning episode is terminated and a new learning episode is started.

[0147] If the learning termination condition is not met, the electronic device (100) can obtain the next state from the environment correlation data list (E) and select attack event data corresponding to the next state from the attack scenario event data list (D). Afterward, the selected event can be set as the next state of the model agent and encoded. At this time, encoding can be performed in the preprocessing module (340).

[0148] In step S750, the electronic device (100) can create a sampling data set and store it in the experience playback memory (350). First, the electronic device (100) has the number of sampling data sets stored in the experience playback memory (350) and the maximum size of the experience playback memory (350) initialized in step S710 ( Compare ) if the number of sampling data sets stored in the experience playback memory (350) is the maximum size ( If it is smaller than ), a sampling data set containing the current state information encoded in step S720, the behavior, reward, and next state information of the model agent obtained in step S740 is created and stored in the experience replay memory (350). Then, to update the Q-function of the prediction network, a sampling data set equal to the batch size (BS) is randomly drawn from the experience replay memory (350).

[0149] In step S760, the electronic device (100) according to one embodiment may set a target Q-value based on the Bellman equation for one of the randomly extracted sampling data sets. Then, by performing gradient descent on an error function set based on the Mean Square Error (MSE) between the target Q-value and the Q-value of the prediction network, the weight parameters of the prediction network ( ) can be updated. Afterwards, the electronic device (100) can repeat the above process for the unselected sampling data set among the extracted sampling data sets and perform learning for a batch size (BS).

[0150] After training for a batch size (BS) is completed, the electronic device (100) is the weight parameters of the target network ( ) weight parameters of the prediction network( It can be reset to ). Finally, the exploration rate ( ) is the minimum search rate( If greater than ), search reduction rate( Search rate based on ) The series of learning steps ends as it reduces )

[0151] In the foregoing, although all components constituting the embodiments disclosed in this document have been described as being combined or operating in combination, the embodiments disclosed in this document are not necessarily limited to such embodiments. That is, within the scope of the purposes of the embodiments disclosed in this document, all components may be selectively combined in one or more ways to operate.

[0152] Furthermore, terms such as "include," "compose," or "have" as described above, unless specifically stated otherwise, mean that the relevant component may be inherent; thus, they should be interpreted as allowing for the inclusion of additional components rather than excluding them. All terms, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments disclosed in this document pertain, unless otherwise defined. Commonly used terms, such as those defined in advance, should be interpreted in accordance with their meaning in the context of the relevant technology and, unless explicitly defined in this document, should not be interpreted in an ideal or overly formal sense.

[0153] The foregoing description is merely an illustrative explanation of the technical concept disclosed in this document, and a person skilled in the art to which the embodiments disclosed in this document pertain can make various modifications and variations within the scope of the essential characteristics of the embodiments disclosed in this document. Accordingly, the embodiments disclosed in this document are intended to explain, not limit, the technical concept of the embodiments disclosed in this document, and the scope of the technical concept disclosed in this document is not limited by these embodiments. The scope of protection of the technical concept disclosed in this document shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of this document.

Claims

Claim 1 A method for establishing a model for identifying vulnerable hosts in a network using an electronic device, comprising: a step of collecting a dataset containing host event data generated by a host of a Software-Defined Networking (SDN)-based network operating in response to an attack scenario; a step of designing a Markov Decision Process (MDP) of reinforcement learning for identifying vulnerable hosts based on a plurality of host event data included in the dataset and a Deep Q-Network (DQN) algorithm for performing said reinforcement learning based on said MDP; and a step of establishing a model for identifying vulnerable hosts among the hosts of the network that are determined to be vulnerable to an attack based on said MDP and said DQN algorithm, wherein the host event data includes an attack step, an attack tactic, and an attack technique identified from an operation performed by a host of said SDN-based network in response to said attack scenario, an identification number of a host of said SDN-based network determined to be an attack target in said attack scenario, and data regarding a risk level calculated from an operation performed by a host of said SDN-based network in response to said attack scenario. Claim 2 delete Claim 3 A model configuration method according to claim 1, wherein the MDP is designed based on the state of the model agent defined for the model agent performing the reinforcement learning based on the host event data, the action of the model agent selecting one host identified as the most vulnerable among the hosts of the SDN-based network, and the setting of a reward given to the model agent when the one host selected by the model agent is identical to the host corresponding to the identification number included in the host event data. Claim 4 A model setting method in which, in paragraph 3, the reward granted to the model agent is set to be granted in proportion to the risk level included in the host event data. Claim 5 A model setting method according to claim 3, wherein the DQN algorithm is designed based on a Q-function, a plurality of sampling data sets are set for updating the Q-function, and a sampling data set randomly selected from the plurality of sampling data sets is used for updating the Q-function. Claim 6 A model setting method according to claim 5, wherein each sampling data set included in the plurality of sampling data sets is configured to include data regarding the current state, behavior, reward, and next state identified for the model agent. Claim 7 A model setting method according to claim 6, wherein the next state of the model agent corresponds to the state of the model agent expected to be confirmed next after the current state of the model agent, and if there are multiple states of the model agent expected to be confirmed next after the current state of the model agent, one state randomly selected from the multiple states is determined as the next state of the model agent. Claim 8 A model setting method according to claim 6, wherein the data included in each of the above-mentioned sampling data sets is set to be encoded as a binary vector. Claim 9 In claim 5, the error function for the DQN algorithm is set based on the Mean Square Error (MSE) between the operation results of the Q-function for the target network and the predict network, and the parameters of the predict network are updated based on the error function, a model setting method. Claim 10 A model setting method according to claim 5, wherein the Q-function is approximated based on an artificial neural network having one or more layers and is configured to estimate the expected value of the total reward obtainable by the model agent according to all actions of the model agent corresponding to all states verifiable for the model agent based on the plurality of sampling data sets. Claim 11 In a computer-readable, non-transient computer-readable storage medium storing a program for executing a model configuration method on a computer, the model configuration method comprises: a step of collecting a dataset including host event data generated by the operation of a host of a Software-Defined Networking (SDN)-based network in response to an attack scenario; a step of designing a Markov Decision Process (MDP) for reinforcement learning to identify vulnerable hosts based on a plurality of host event data included in the dataset, and a Deep Q-Network (DQN) algorithm for performing the reinforcement learning based on the MDP. A non-transient computer-readable storage medium comprising the step of establishing a model for identifying vulnerable hosts among the hosts of a network that are determined to be vulnerable to an attack based on the MDP and the DQN algorithm, wherein the host event data includes an attack step, an attack tactic, and an attack technique identified from actions performed by a host of the SDN-based network in response to the attack scenario, an identification number of a host of the SDN-based network determined to be an attack target in the attack scenario, and data regarding a risk level calculated from actions performed by a host of the SDN-based network in response to the attack scenario. Claim 12 An electronic device for performing a model configuration method comprises: a processor; and one or more memories for storing one or more instructions, wherein, upon execution, the one or more instructions include: a step of collecting a dataset containing host event data generated by a host of a Software-Defined Networking (SDN)-based network operating in response to an attack scenario; a step of designing a Markov Decision Process (MDP) for reinforcement learning to identify vulnerable hosts based on a plurality of host event data included in the dataset, and a Deep Q-Network (DQN) algorithm for performing said reinforcement learning based on said MDP. An electronic device comprising: controlling the processor to perform the step of establishing a model for identifying vulnerable hosts among the hosts of a network that are determined to be vulnerable to an attack based on the MDP and the DQN algorithm, wherein the host event data includes an attack step, an attack tactic, and an attack technique identified from an action performed by a host of the SDN-based network in response to the attack scenario, an identification number of a host of the SDN-based network determined to be an attack target in the attack scenario, and data regarding a risk level calculated from an action performed by a host of the SDN-based network in response to the attack scenario.

Images