Method and apparatus for processing real-time accumulated data for integrated analysis based on flow

Abstract

A technical problem to be solved by the present invention is to provide a method for processing real-time accumulated data so as to rapidly search for a packet corresponding to a search condition from a large amount of data called a full packet. The method comprises: a network packet information reception step of acquiring at least one piece of information from flow information including 5-tuple information and package information including meta information, content information, threat detection information, asset information, URL information, and IP information; a step of generating a mapping table in which identification information of the package information and identification information of the flow information are indexed; and a step of generating data specialized for the flow information and storing, in a database, the data including the mapping table.

Description

Real-time cumulative data processing method and device for flow-centric integrated analysis

[0001] The present invention relates to a method and device for processing real-time cumulative data. More specifically, the present invention relates to a method and device for processing cumulative data in real time, enabling high-speed retrieval of data corresponding to input flows in an environment where packet-based network detection and response technology is applied.

[0002]

[0003] Network Detection and Response, a network detection and response technology based on full packet collection and analysis, is a technology that can detect and respond to threats by collecting and analyzing all packets entering and exiting a company's network at high speed.

[0004] Network detection and response technology based on full packet collection and analysis enables analysis of abnormal behavior by internal employees of a company as well as security threats that may arise from services operated by the company.

[0005] Despite these advantages, it still takes a considerable amount of time to search for a desired packet in a large amount of data called a full packet.

[0006]

[0007] The problem to be solved by the present invention is to provide a method and device for high-speed packet search that can search packets corresponding to search conditions in a large amount of data called full packets at high speed.

[0008] More specifically, the present invention provides a real-time cumulative data processing method and device that generates a database capable of flow-centric integrated analysis in real time, thereby enabling high-speed retrieval of package information corresponding to flow information.

[0009] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0010]

[0011] In order to solve the above-described problem, a method for processing accumulated data in real time so as to enable high-speed search centered on flow information of packets collected from a network according to an embodiment of the present invention includes a network packet information receiving step of obtaining at least one piece of information from flow information including 5-tuple information and package information including meta information, content information, threat detection information, asset information, URL information, and IP information, a mapping table generating step of indexing identification information of package information and identification information of flow information, and a step of generating data specialized for flow information including the mapping table and storing the data in a database.

[0012] Specific details of other embodiments are included in the detailed description and drawings.

[0013]

[0014] According to embodiments of the present invention, packets corresponding to a specific flow can be searched for at high speed in a large amount of data and information related thereto can be obtained at high speed, thereby reducing the time required for a corporate security manager to recognize and respond to a cyber attack.

[0015] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0016]

[0017] FIG. 1 is a diagram illustrating the configuration of a real-time cumulative data processing device according to an embodiment of the present invention.

[0018] FIG. 2 is a diagram illustrating a search database according to an embodiment of the present invention.

[0019] FIG. 3 is a diagram illustrating a relationship diagram of a database according to an embodiment of the present invention.

[0020] FIG. 4 is a flowchart of a real-time cumulative data processing method according to an embodiment of the present invention, and FIG. 5 is a drawing for explaining the real-time cumulative data processing method.

[0021] FIG. 6 is a diagram for explaining a mapping table creation step according to an embodiment of the present invention.

[0022] FIG. 7 is a diagram for explaining a flow information retrieval step according to an embodiment of the present invention.

[0023]

[0024] According to one embodiment of the present invention, a method for processing accumulated data in real time to enable high-speed search centered on flow information for packets collected from a network includes a network packet information receiving step of obtaining at least one piece of information from flow information including 5-tuple information and package information including meta information, content information, threat detection information, asset information, URL information, and IP information, a mapping table generating step of indexing identification information of package information and identification information of flow information, and a step of generating data specialized for flow information including the mapping table and storing the data in a database.

[0025]

[0026] The advantages and features of the present invention, as well as the methods for achieving them, will become clearer with reference to the embodiments described in detail below, along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure the completeness of the present invention and to fully inform those skilled in the art of the scope of the invention. The present invention is defined solely by the scope of the claims.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0028] The terminology used herein is for the purpose of describing embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms, unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the components mentioned.

[0029]

[0030] A flow in which data packets are transmitted may be referred to as a "flow." Flows may be created not only at the node or IP level, but also at a more granular level (e.g., an application level). Flows may be created at the TCP session, application, or even at a more granular level. Flow information may include an application ID, a destination IP address, and / or a service port to identify whether a data packet transmitted from a source is an authorized data packet. Flow information may include authorized target information to determine whether to forward the data packet based on the source IP, destination IP, and port information of the data packet, and status information including whether the data flow is valid.

[0031]

[0032] According to an embodiment, the flow information may include authentication information. For example, the authentication information may include a series of information for verifying whether an authorized node transmitted a data packet, a method for inserting authentication information according to a transmission protocol (e.g., in the case of TCP, inserting TCP SYN packets, in the case of UDP, inserting authentication information for each data packet or inserting authentication information at regular intervals (or cycles)), information for encrypting and decrypting authentication information, information on an algorithm for generating and verifying authentication information and a series of information included in the algorithm (e.g., information such as a Secret Key when generating an HMAC OTP), and authentication information including whether the source IP is authenticated. According to an embodiment, the authentication information may be determined (or generated) based on an authentication policy database.

[0033]

[0034] In some embodiments, flow information may include service information. For example, service information may include service IP and port information that an authorized access target can access through a proxy, protocol information (e.g., HTTP, FTP, IoT-specific protocols, etc.), whether service request filtering is required, how service request filtering is processed, and filtering information (e.g., personal information, harmful service request information). In some embodiments, service information may be generated based on a service policy.

[0035]

[0036] In some embodiments, the flow information may include 5-tuple information. A 5-tuple is a set of five different values ​​that constitute a TCP / IP connection, including a source IP address, a source port number, a destination IP address, a destination IP port number, and the protocol in use.

[0037]

[0038] According to an embodiment, package information may refer to various types of information contained in a network packet. More specifically, package information may include metadata, content information, threat detection information, asset information, URL information, and IP information, and may also include any other information contained in a network packet.

[0039]

[0040] Hereinafter, embodiments of the present invention will be described with reference to the attached drawings. In the drawings, like reference numerals represent like components.

[0041] FIG. 1 is a diagram illustrating the configuration of a real-time cumulative data processing device (100) according to an embodiment of the present invention.

[0042] A real-time cumulative data processing device (100) according to an embodiment of the present invention includes a database specialized in a flow search function instead of a relational database management system (RDBMS) having various functions. In particular, the present invention uses a database specialized in flow information that links flow information with various pieces of information such as IP, port, URL, application, and file, and can quickly and accurately acquire other information using the flow information as a search condition. In other words, the present invention is designed to store various pieces of information together with flow information and to create a flow-centric database that can link them, so that when searching for a specific flow, information related thereto can be quickly acquired.

[0043]

[0044] Referring to FIG. 1, a real-time cumulative data processing device (100) includes a package data table generation unit (110), a flow data table generation unit (120), a flow hashing table generation unit (130), a mapping table generation unit (140), a database (150), and a control unit (160).

[0045]

[0046] Among the components illustrated in Fig. 1, the mapping table generation unit (140) can operate as a search engine. When flow information is input as a search condition, the search engine generates data necessary for high-speed searching for package information corresponding to the input flow information and stores the data in a database.

[0047]

[0048] Below, the components illustrated in Figures 1 to 3 will be described in more detail.

[0049]

[0050] FIG. 1 is a diagram illustrating a configuration of a real-time cumulative data processing device according to an embodiment of the present invention, FIG. 2 is a diagram illustrating a search database according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating a relationship diagram of a database according to an embodiment of the present invention.

[0051] Referring to FIGS. 1 to 3, the package data table generation unit (110) can generate a package data table by sorting collected packets according to preset criteria. A package data table refers to a table that stores package information included in acquired network packets in real time or sequentially. The package data table can include package information and identification information for identifying the same.

[0052] According to a specific embodiment of the present invention, the package data table may be a metadata table. The metadata table may be a table that sorts collected network packets or flows in the order in which they were collected. That is, according to one embodiment, the metadata table may be a table that sequentially stores metadata of acquired network packets in real time.

[0053] The metadata table may include identification information that can identify the collected metadata. In one embodiment, the metadata table may store metadata on a flow-by-flow basis, and metadata of flows acquired in real time may be sequentially stored in the metadata table. In another embodiment, the metadata table may store metadata on a network packet-by-network basis, and metadata of network packets acquired in real time may be sequentially stored in the metadata table.

[0054] Each piece of metadata can sequentially acquire identification information based on the order in which it is stored. In one embodiment, the identification information for each piece of metadata can be a primary index, which is a number indicating the location of the metadata within the metadata table, or a meta_id assigned to each piece of metadata. The identification information for each piece of metadata can be assigned in ascending order based on the order in which the metadata is added to the metadata table.

[0055]

[0056] The flow data table generation unit (120) can generate a flow data table (flow_data_table) as illustrated in FIG. 3 by sorting collected packets according to preset criteria. Referring to FIG. 3, the flow data table is assigned flow identification information in the order in which flows are added, and can be sorted in ascending order of the flow identification information. In one embodiment, the flow identification information may be a flow ID and may be a secondary index, which is a number indicating the position of the flow information within the flow data table. The flow data table may include position index information indicating the position of the corresponding flow in the flow information, the flow identification information, and the mapping data.

[0057]

[0058] The flow hashing table generation unit (130) can generate a flow hashing table including flow hashing information. Referring to FIG. 3, the flow hashing table can include hash information, hash index information, a hash ID, a 5-tuple value, and flow identification information (or a secondary index). If necessary, the flow hashing table can be connected to a flow data table. The secondary index can be index information indicating the location of the corresponding flow information in the flow data table.

[0059]

[0060] The mapping table generation unit (140) can generate a mapping table including a primary index and a secondary index. According to a specific embodiment of the present invention, the mapping table generation unit (140) can generate a mapping table including a flow ID and a meta ID.

[0061] The mapping table contains minimal information, allowing for faster connection of flow-related package information, thus improving flow search speed. In other words, the mapping table is structured to capture all information related to a flow at once.

[0062] Additionally, the mapping table is flushed at a pre-determined point in time, at which point it is indexed. Optionally, indexing involves sorting the data in ascending order based on a secondary index.

[0063]

[0064] FIG. 4 is a flowchart of a real-time cumulative data processing method according to an embodiment of the present invention, FIG. 5 is a drawing for explaining a real-time cumulative data processing method, FIG. 6 is a drawing for explaining a mapping table creation step, and FIG. 7 is a drawing for explaining a flow information search step.

[0065] Referring to FIGS. 4 and 5, a method for processing accumulated data in real time to enable high-speed search centered on flow information for packets collected from a network may include a network packet information receiving step, a mapping table creation step, and a database storage step.

[0066] The network packet information receiving step means a step of acquiring at least one piece of information from package information including flow information and meta information, content information, threat detection information, asset information, URL information, and IP information including 5 tuple information.

[0067]

[0068] Referring to Figure 6, the mapping table creation step is a step of creating a mapping table that indexes the identification information of package information and the identification information of flow information.

[0069] The mapping table creation step may include a step of creating a package data table by sorting package information of received packets in real time and assigning a primary index to the package information to identify it, a step of creating a flow data table including the flow information by assigning a secondary index to the flow information to identify the flow information of the received packet, and a step of creating a mapping table that connects the primary index and the secondary index.

[0070] If necessary, the step of generating a flow data table may include a step of assigning a secondary index corresponding to the flow information and updating the location index information of the flow data table if the 5-tuple value of the received packet already exists in the flow hash table.

[0071] Optionally, the step of generating the flow data table may include adding the corresponding flow information to the flow data table and the flow hash table if the 5-tuple value of the received packet does not exist in the flow hash table, and assigning a secondary index to the flow information.

[0072] Optionally, the mapping table generation step may include a step of sorting the mapping table based on identification information of the flow information at a preset cycle.

[0073]

[0074] The step of saving to a database creates data specific to flow information, including a mapping table, and stores it in the database.

[0075]

[0076] Referring to FIG. 7, if necessary, a method for processing accumulated data may further include a step of searching a mapping table for package information corresponding to the input flow information.

[0077] More specifically, the search step may include a step of inputting flow information for search, a step of obtaining location index information, a step of obtaining identification information of package information, and a step of obtaining package information.

[0078] The position index information acquisition step refers to a step of acquiring position index information indicating the position of the flow information in the mapping table of the flow information in the flow data table using the input flow information.

[0079] The step of obtaining identification information of package information means the step of obtaining identification information of package information corresponding to location index information in the mapping table.

[0080] The package information acquisition step refers to the step of acquiring package information corresponding to the identification information of the package information in the package data table.

[0081]

[0082] According to another embodiment of the present invention, a method for processing accumulated data is provided, which connects packets collected from a network with flow information and meta information in real time to enable high-speed searching centered on flow information, and is designed to search meta information more quickly and accurately using flow information as a search condition.

[0083] More specifically, the accumulated data processing method may include a step of receiving network packet information including flow information and meta information including 5-tuple information, a step of creating a mapping table that links identification information of the meta information and identification information of the flow information, a step of creating data specialized for the flow information including the mapping table and storing the data in a database, and a step of searching for meta information corresponding to the input flow information in the mapping table.

[0084] A mapping table generation step according to one embodiment of the present invention may include a step of generating a metadata table by arranging metadata of a received packet in real time and assigning a primary index to the metadata to identify the same, a step of assigning a secondary index to the flow information to identify the flow information of the received packet and generating a flow data table including the same, and a step of generating a mapping table that links the primary index and the secondary index. According to one embodiment, the primary index may be a metadata id and the secondary index may be a flow id.

[0085] The step of generating a flow data table according to one embodiment of the present invention may include, when a 5-tuple value of a received packet already exists in the flow hash table, a step of assigning a secondary index corresponding to the flow information and updating location index information of the flow data table, and, when a 5-tuple value of the received packet does not exist in the flow hash table, a step of adding the flow information to the flow data table and the flow hash table and assigning a secondary index corresponding to the flow information.

[0086]

[0087] A search step according to one embodiment of the present invention may include a step of inputting flow information for search, a step of obtaining identification information corresponding to the flow information from a flow data table, a step of obtaining identification information of meta information corresponding to the identification information of the flow information from a mapping table, and a step of obtaining package information corresponding to the identification information of the package information from the meta data table.

[0088]

[0089] Referring to FIG. 6, the device may (1) receive a network packet containing flow information and metadata including 5-tuple information, and add the received metadata to a metadata table. At this time, the device may store the metadata in the metadata table in the order in which it was acquired. The device may assign identification information to the metadata to indicate the location of the metadata table. If necessary, the identification information may be primary index information or meta_id.

[0090] The device can add identification information of acquired metadata to the flow meta mapping table.

[0091] (2) The device can search for a matching 5-tuple in the hash table using the flow information of the acquired network packet. At this time, if there is a matching 5-tuple value in the hash table, the device can obtain secondary index information (or flow_id, identification information of the flow information) matching the flow information entered in the hash table, and (3) search for secondary index information in the flow data table. The device can obtain location index information corresponding to the secondary index information in the flow data table, and input the secondary index information into the mapping table while updating the location index information. Since the location index information includes location and number information in the mapping table of the corresponding secondary index, when a network packet having the same flow information is added, the number information included in the location index information must be updated.

[0092] If (4) there is no matching 5-tuple value in the hash table, the device adds new flow information to the flow data table and assigns secondary index information (or flow_id, identification information of the flow information). Then, (5) and (6) add the corresponding information to the hash table and mapping table.

[0093] The mapping table can be sorted in ascending order based on the secondary index while being flushed at a preset interval.

[0094]

[0095] Another embodiment of the present invention may include a computer-readable recording medium having recorded thereon a program including commands for performing each step according to the real-time cumulative data processing method of the present invention.

[0096]

[0097] According to embodiments of the present invention, packets corresponding to a specific flow can be searched for at high speed in a large amount of data and information related thereto can be obtained at high speed, thereby reducing the time required for a corporate security manager to recognize and respond to a cyber attack.

[0098]

[0099] Hereinafter, embodiments of the present invention have been described. In the above description, the components constituting the cumulative data processing device illustrated in FIG. 1 may be implemented as modules. A module refers to a software or hardware component such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the module performs certain roles. However, the module is not limited to software or hardware. The module may be configured to reside on an addressable storage medium or may be configured to execute one or more processors.

[0100] Thus, as an example, a module may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided by components and modules may be combined into a smaller number of components and modules or further separated into additional components and modules.

[0101] Meanwhile, the invented embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, generate program modules to perform the operations of the invented embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0102] Computer-readable recording media include all types of recording media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory (200), and optical data storage devices.

[0103] Additionally, a computer-readable recording medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0104] According to one embodiment, the method according to various embodiments of the present invention may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable recording medium (e.g., compact disc read only memory (CD-ROM)), or distributed directly between two user devices (e.g., smartphones) via an application store (e.g., Play Store™), or distributed online (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily generated on a machine-readable recording medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0105] The embodiments of the present invention have been described with reference to the above and the attached drawings. Those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. A method for processing accumulated data in real time to enable high-speed search of packets collected from a network centered on flow information. A network packet information receiving step for obtaining at least one piece of information from package information including flow information, meta information, content information, threat detection information, asset information, URL information, and IP information including 5-tuple information; A step of creating a mapping table that indexes the identification information of the above package information and the identification information of the above flow information; A step of generating data specialized for the flow information, including the mapping table, and storing the data in a database; Real-time cumulative data processing method.

2. In paragraph 1, further comprising a package information retrieval step corresponding to the input flow information; Real-time cumulative data processing method.

3. In paragraph 2, The above search step is, Step for entering flow information for search; A step of obtaining location index information indicating the location of the flow information in the mapping table from the flow data table; A step of obtaining identification information of package information corresponding to the location index information from the above mapping table; A step of obtaining package information corresponding to the identification information of the package information from the package data table; comprising; Real-time cumulative data processing method.

4. In paragraph 1, The above mapping table creation step is; A step of creating a package data table by sorting the package information of the received packet in real time and assigning a primary index to the package information to identify it; A step of assigning a secondary index to the flow information to identify the flow information of the received packet and creating a flow data table including the same; A step of creating a mapping table linking the first index and the second index; including; Real-time cumulative data processing method.

5. In paragraph 4, The steps for creating the above flow data table are: A step of assigning a secondary index corresponding to the flow information and updating the location index information of the flow data table when the 5-tuple value of the received packet already exists in the flow hash table; Including a step of adding the flow information to the flow data table and the flow hash table and assigning a secondary index to the flow information, if the 5-tuple value of the received packet does not exist in the flow hash table; Real-time cumulative data processing method.

6. In paragraph 1, The above mapping table creation step is; A step of sorting the mapping table based on the identification information of the flow information at a preset cycle; including; Real-time cumulative data processing method.

7. A method for processing accumulated data in real time to enable high-speed search of packets collected from a network centered on flow information. A network packet information receiving step for obtaining flow information and meta information including 5-tuple information; A mapping table creation step that links the identification information of the above meta information and the identification information of the above flow information; A step of generating data specialized for the flow information, including the mapping table, and storing the data in a database; A step of searching for meta information corresponding to the input flow information in the mapping table; Real-time cumulative data processing method.

8. In paragraph 7, The above mapping table creation step is; A step of creating a metadata table by sorting the metadata of the received packet in real time and assigning a primary index to the metadata to identify it; A step of assigning a secondary index to the flow information to identify the flow information of the received packet and creating a flow data table including the same; A step of creating a mapping table linking the first index and the second index; including; Real-time cumulative data processing method.

9. In paragraph 8, The steps for creating the above flow data table are: A step of assigning a secondary index corresponding to the flow information and updating the location index information of the flow data table when the 5-tuple value of the received packet already exists in the flow hash table; Including a step of adding the flow information to the flow data table and the flow hash table and assigning a secondary index corresponding to the flow information, if the 5-tuple value of the received packet does not exist in the flow hash table; Real-time cumulative data processing method.

10. A computer-readable recording medium having recorded thereon a program including commands for performing each step according to the real-time cumulative data processing method described in any one of claims 1 to 9.

Images