In-memory-based data processing system and method for collecting, processing, and storing massive edge data in real time

Abstract

Disclosed are an in-memory-based data processing system and method for collecting, processing, and storing massive edge data in real time. To this end, the in-memory-based data processing system according to an embodiment of the present invention comprises: an SNET module configured to enable bidirectional online transaction processing (OLTP) with an edge so as to collect and analyze data from the edge and generate an action command according to a preset action condition to transmit the action command to the edge; an SDB module for storing information collected from an external RDBMS and user-specific and service-specific environment information; and a CAST module for constructing, as an in-memory DB, partially filtered data required for an application service in the SDB module and processing a transaction of the application service on the basis of a data analysis result in the SNET module and the in-memory DB.

Description

In-memory-based data processing system and method for real-time collection, processing, and storage of large-volume edge data.

[0001] The present invention relates to a data processing system for edge data, and more particularly, to an in-memory-based data processing system and method capable of collecting, processing, and storing large amounts of edge data in real time.

[0002]

[0003] With the advancement of Internet and IoT technologies, when the number of simultaneous users or the amount of data that must be processed simultaneously increases in services provided by e-commerce, finance, and public institutions, i.e., when the number of transactions that must be processed increases rapidly, the server providing the service is frequently experiencing service delays or interruptions due to problems such as downtime.

[0004] Although these problems can be partially solved by increasing the server bandwidth on the web server or web application server (WAS) side, the service interruption caused by the rapid increase in transactions has not been resolved because it is essential to entail a large number of SQL queries to store data in the RMDBS, a relational database management system, or to call stored data from the web application server.

[0005] With the recent increase in demand for real-time services, the proliferation of high-performance general-purpose servers, advancements in semiconductor technology, and declining DRAM prices, in-memory computing technology is being utilized to address the above issues. In-memory computing is a method of processing application data in main memory rather than on hard disks. In other words, memory areas, previously considered solely for computation, are now utilized for both computation and storage.

[0006] In the case of these MMDB (Main Memory DMBS) solutions, since the relational DB data stored on the hard disk is moved to the main memory such as RAM, there is a limitation in processing large amounts of data at the edge in real time because it is impossible to move all of the large-capacity RDBMS data to the main memory and there are still limitations in data processing speed due to the characteristics of the relational DB structure.

[0007]

[0008] (Prior art literature)

[0009] (Patent Document)

[0010] Korean Patent No. 10-1905968 (October 1, 2018)

[0011] Korean Patent No. 10-2038529 (October 24, 2019)

[0012] Korean Patent Publication No. 10-2022-0073692 (June 3, 2022)

[0013]

[0014] The present invention is intended to solve the above problems, and more specifically, to provide an in-memory-based data processing system and method capable of real-time collection, high-speed processing, and high-speed storage of large-volume edge data without service interruption.

[0015] The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0016]

[0017] In order to solve the above problem, an in-memory-based data processing system for real-time collection, processing, and storage of large-capacity edge data according to an embodiment of the present invention comprises: a SNET module configured to enable bidirectional online transaction processing (OLTP) with an edge, and to collect and analyze data from the edge, generate action commands according to preset action conditions, and transmit them to the edge; an SDB module for storing information collected from an external RDBMS and user- and service-specific environmental information; and a CAST module for constructing some filtered data necessary for an application service from the SDB module into an in-memory DB, and processing transactions of the application service based on the data analysis results from the SNET module and the in-memory DB.

[0018] In one embodiment, the SNET module may include a packet reader module that collects data from the edge; and a packet analysis module that analyzes data collected by the packet reader module, and the packet analysis module may include a parsing unit that parses packets; a filtering unit that selects storage data based on preset storage conditions and stores the data in the in-memory DB of the CAST module; and an action processing unit that stores and manages the preset action conditions and generates the action command.

[0019] In one embodiment, the packet analysis module may further include an output unit that visualizes and outputs analyzed packet information and packet data-specific environment setting information through a user interface.

[0020] In one embodiment, the in-memory DB has a non-relational data storage structure in the NoSQL manner, and the CAST module can perform transaction processing in a shared memory manner.

[0021] In one embodiment, the in-memory DB may have a hash index-based data storage structure.

[0022] In one embodiment, the CAST module may further include a multi-DB layer that converts data stored in the in-memory DB into a NoSQL format to be compatible with multiple different types of RDBMS.

[0023] In one embodiment, the SDB module may transmit some filtered data required for application services to the in-memory DB in bulk.

[0024] In one embodiment, the SDB module can transmit updated information to an external RDBMS in the form of a batch file.

[0025] In addition, an in-memory-based data processing method according to an embodiment of the present invention includes: (a) a step of storing information collected from an external RDBMS and user-specific and service-specific environmental information in an SDB module; (b) a step of having a CAST module build an in-memory DB with some filtered data required for an application service from the SDB module; (c) a step of having a SNET module collect and analyze data from an edge; (d) a step of having the SNET module transmit an action command to the edge according to a preset action condition based on a data analysis result; and (e) a step of having the CAST module process a transaction of an application service based on the data analysis result from the SNET module and the in-memory DB.

[0026] In one embodiment, the in-memory DB has a non-relational data storage structure in the NoSQL manner, and the CAST module can perform transaction processing in a shared memory manner.

[0027]

[0028] An in-memory-based data processing system and method according to one embodiment of the present invention has the effect of enabling real-time collection, high-speed processing, and high-speed storage of large-volume edge data without service interruption.

[0029] In addition, the in-memory-based data processing system and method according to one embodiment of the present invention has the effect of enabling a quick real-time service response before data is stored in a server (File or RDBMS) by transmitting an immediate action command to the edge through packet analysis of edge data.

[0030] In addition, the in-memory-based data processing system and method according to one embodiment of the present invention has the effect of enabling a user to conveniently set action service execution conditions and data storage conditions and immediately execute them without modifying the program or changing the code by visualizing and outputting packet information and environment setting information for each packet data based on the packet analysis results of edge data.

[0031] In addition, the in-memory-based data processing system and method according to one embodiment of the present invention can maximize transaction processing speed through an in-memory DB having a non-relational data storage structure in the NoSQL method and a CAST module that adopts a data transmission method in the shared memory method.

[0032] The effects according to the present invention are not limited to those exemplified above, and more diverse effects are included in this specification.

[0033]

[0034] Figure 1 is a diagram showing a network configuration to which the in-memory-based data processing system of the present invention is applied.

[0035] Figure 2 is a schematic diagram of the in-memory-based data processing system of the present invention.

[0036] Figure 3 is a schematic diagram of the SNET module of Figure 2.

[0037] Figure 4 is a schematic diagram of the CAST module of Figure 2.

[0038] Figure 5 is a schematic diagram of a hardware device in which the in-memory-based data processing system of the present invention operates.

[0039] Figure 6 is a schematic flowchart illustrating an in-memory-based data processing method of the present invention.

[0040]

[0041] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0042] Hereinafter, an in-memory-based data processing system and method for real-time collection, processing, and storage of large-volume edge data according to embodiments of the present invention will be described with reference to drawings.

[0043]

[0044] FIG. 1 is a diagram showing a network configuration to which the in-memory-based data processing system of the present invention is applied, FIG. 2 is a schematic configuration diagram of the in-memory-based data processing system of the present invention, FIG. 3 is a schematic configuration diagram of the SNET module of FIG. 2, and FIG. 4 is a schematic configuration diagram of the CAST module of FIG. 2.

[0045] Referring to FIG. 1, the in-memory-based data processing system (100) of the present invention is directly or indirectly connected to a web application server (200), an RDBMS (300), and an edge (400) via a network.

[0046] Web Application Server (WAS; Web Application Server, 200) refers to a server that provides transaction processing and management and an application execution environment in a client-server environment.

[0047] The Web Application Server (hereinafter referred to as "WAS", 200) provides each service required by the client. For example, WAS (200) can provide services related to e-commerce, finance, public institutions, hospitals, etc.

[0048] RDBMS (300) is a storage device that stores information related to application services in conjunction with WAS (200). Specifically, RDBMS (300) is a relational database management system that stores and manages data in a form in which data is stored in tables of one or more columns and rows, and relationships between each table are defined. Examples include MySQL, PostgreSQL, MariaDB, MSSQL, and Oracle.

[0049] Edge (400) refers to a terminal that transmits data to a web application server (200). As illustrated in FIG. 1, the edge (400) may include a computer (410), a smartphone (420), an IoT device (430), a sensor (440), etc.

[0050] These edges (400) can be located in homes, offices, factories, public institutions, etc., and can transmit data such as various sensor data and service-related request information to WAS (200) through a web server (not shown) on the Internet.

[0051] The in-memory-based data processing system (100, hereinafter abbreviated as “system”) of the present invention is configured to be connected to a WAS (200) and an RDBMS (300) over a network, and serves to provide data so that the WAS (200) can quickly process transactions based on an in-memory basis when executing an application service.

[0052]

[0053] Referring to FIG. 2, the system (100) is configured to include a SNET module (120), a CAST module (140), and an SDB module (160).

[0054] The SNET module (120) is configured to enable bidirectional online transaction processing (OLTP) with the edge (400), collects and analyzes data from the edge (400), and generates action commands according to preset action conditions and transmits them to the edge (400).

[0055] The SDB module (160) serves to store information collected from the RDBMS (300) and environmental information for each user and service.

[0056] The SDB module (160) can collect and filter data from various types of DBs, such as Oracle, MSSQL, MySQL, Maria, Informix, Hadoop, and Mango, through an RDBMS interface to enable data communication with multiple different types of RDBMS, thereby constructing data.

[0057] Additionally, the SDB module (160) further includes a connection interface that allows access to data in the in-memory DB (148) of the CAST module (140), although not shown.

[0058] The CAST module (140) builds some filtered data required for the application service in the SDB module (160) into an in-memory DB, and processes the transaction of the application service based on the data analysis results from the SNET module (120) and the in-memory DB.

[0059] A user can access the system (100) of the present invention through an ODBC (Open Database Connectivity) or JDBC (Java Database Connectivity) interface.

[0060] The configuration and operation of the above SNET module (120), CAST module (140), and SDB module (160) are described in detail with reference to FIGS. 3 to 5.

[0061]

[0062] Referring to FIG. 3, the SNET module (120) is configured to include a packet reader module (122) and a packet analysis module (124).

[0063] The packet reader module (122) reads (collects) data packets generated at the edge (400) in a direct access (DA) manner.

[0064] The packet analysis module (124) plays a role in analyzing the packets read by the packet reader module (122) for pre-processing.

[0065] The packet analysis module (124) is configured to include a parsing unit (125), a filtering unit (126), an output unit (127), and an action processing unit (128).

[0066] The parsing unit (125) plays a role in parsing packet data collected for analysis, and the filtering unit (126) plays a role in selecting meaningful data according to preset policy conditions based on the parsed data.

[0067] The output section (127) serves to visualize and output analyzed packet information and packet data-specific environment setting information through a user interface.

[0068] The action processing unit (128) stores and manages preset action conditions and generates action commands.

[0069] That is, the SNET module (120) analyzes packets of data collected from the edge (400) and selectively stores only necessary data according to preset policy conditions through the filtering unit (126), thereby minimizing the amount of data stored in the in-memory DB (148), SDB module (160), and RDBMS (300) described later by the WAS (200).

[0070] The output section (127) of the SNET module (120) can maximize the convenience of user- and service-specific environment settings by visualizing packet information and environment setting information for each packet data and outputting it to the user, thereby enabling the user to set the criteria for data to be stored and the criteria for data requiring action without a separate dedicated program.

[0071] The action processing unit (128) of the SNET module (120) can secure immediacy by immediately generating the necessary action command based on the packet information analyzed by the packet analysis module (124) according to preset action conditions and transmitting it to the edge (400).

[0072] For example, in the case where sensor data of overcurrent, overvoltage, or high temperature values ​​are acquired from factory equipment, in a conventional RDBMS-based system, since the data was stored in the RDBMS and then action could be taken based on the information acquired through an SQL query, there was a problem that the occurrence of an accident could not be prevented even if action was taken because a time delay of tens of seconds to minutes occurred in the transaction processing during this process. However, the system (100) of the present invention has the effect of preventing the occurrence of an accident by taking action by immediately executing an action service at a stage before storing the data in the DB through packet analysis of data collected from the edge (400) to secure immediacy.

[0073] Meanwhile, the action processing unit (128) can execute an immediate action service according to a preset action condition (128), but can also be configured to execute the action service by referring to the data of the in-memory DB (148) of the CAST module (140) described later.

[0074] Even in this case, unlike the prior art, since data is referenced from an in-memory DB (148) based on main memory (RAM, etc.) rather than from a hard disk-based RMDBS, the service can be executed at a significantly faster speed than the prior art, thereby ensuring immediacy.

[0075]

[0076] Referring to FIG. 4, the CAST module (140) is configured to include a multi-DB layer (141) and an in-memory DB (148).

[0077] The in-memory DB (148) can be constructed with some filtered data required for application services among the data stored in the SDB module (160) of the system (100).

[0078] Specifically, as illustrated in FIG. 1, necessary data among the data stored in the RDBMS (300) can be loaded in bulk data format and stored in the in-memory DB (148).

[0079] The in-memory DB (180) may have a non-relational data storage structure in the NoSQL style, and in particular, may have a hash index-based data storage structure.

[0080] The multi-DB layer (141) converts data stored in the in-memory DB (180) into a NoSQL format to make it compatible with multiple different types of RDBMS.

[0081] For example, as illustrated in FIG. 4, the multi-DB layer (141) may be configured to include an Oracle DB Layer (142), a PostGreSql DB Layer (143), a MySQL DB Layer (144), an MSSQL (Sql Server) DB Layer (145), a Tibero DB Layer (146), and a Maria DB Layer (147).

[0082] Although not shown in FIG. 4, the CAST module (140) is configured with an API that makes it compatible with multiple different types of RDBMS, and the WAS (200) or other connection terminals (not shown) can query data of the in-memory DB (148) in a NoSQL manner through each API.

[0083] At this time, the data stored in the in-memory DB (148) may have a non-relational data structure as described above, and in particular, may be configured with a key-value based hash index structure to enable very fast data search compared to RMDBS.

[0084] The key-value based data of the in-memory DB (148) is configured to be converted and output to fit the format of multiple different types of RDBMS by passing through the multi-DB layer (141), so that the CAST module (140) substantially includes a NoSQL-based DB corresponding to multiple RDBMS.

[0085]

[0086] In comparison with the RDBMS and MMDB (Main Memory DB) according to the actual prior art, a total of 1,000,000 data were input 10 times each for the SDB module (160) and CAST module (140) of the present invention, and the processing time was measured, and the results as shown in Table 1 below were obtained.

[0087]

[0088]

[0089]

[0090] Referring to Table 1, it can be seen that the SDB module (160) of the system (100) of the present invention shows a speed about 5 times faster than Oracle, which is a type of RDBMS, and the CAST module (140) is in-memory based and shows a speed about 20 to 30 times faster than MMDB.

[0091]

[0092] Figure 5 is a schematic diagram of a hardware device in which the in-memory-based data processing system of the present invention operates.

[0093] The system (100) of FIG. 1 refers to a hardware device in which the in-memory-based data processing engine (software) of the present invention is installed and operates, and the hardware device (500) of FIG. 5 may be the same as the system (100) of FIG. 1, but in FIG. 5, for convenience, the concept of a hardware device in which the in-memory-based data processing engine of the present invention can be installed is described with a separate drawing symbol for convenience.

[0094] Referring to FIG. 5, a hardware device (500) may include at least one processor (510), a memory (520), and a transmission / reception device (530) that is connected to a network and performs communication. In addition, the device (500) may further include an input interface device (540), an output interface device (550), a storage device (560), etc. Each component included in the device (500) may be connected to each other by a bus (570) and communicate with each other.

[0095] Meanwhile, each component included in the device (500) may be connected through an individual interface or individual bus centered around the processor (510), rather than a common bus (570). For example, the processor (510) may be connected to at least one of a memory (520), a transmission / reception device (530), an input interface device (540), an output interface device (550), and a storage device (560) through a dedicated interface.

[0096] The processor (110) can execute a program or instructions stored in at least one of the memory (520) and the storage device (560). The processor (510) may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed.

[0097] Each of the memory (520) and the storage device (560) may be configured with at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (520) may be configured with at least one of a read-only memory (ROM) and a random access memory (RAM).

[0098] Additionally, the operations of the method according to embodiments of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. A computer-readable recording medium includes any type of recording device that stores data readable by a computer system. Furthermore, a computer-readable recording medium can be distributed across network-connected computer systems, allowing the computer-readable program or code to be stored and executed in a distributed manner.

[0099] Additionally, a computer-readable recording medium may include hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. The program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0100] While some aspects of the present invention have been described in the context of a device, they may also represent a description of a corresponding method, wherein a block or block corresponds to a method step or a characteristic of a method step. Similarly, aspects described in the context of a method may also be represented as a corresponding block or item or a characteristic of a corresponding device. Some or all of the method steps may be performed by (or using) a hardware device, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0101] In embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, the field programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by some hardware device.

[0102] The configuration of the device (500) of FIG. 5 is intended to explain the main hardware configuration in which the software engine of the present invention operates, and the configuration and operation of FIGS. 1 to 4 take precedence over FIG. 5.

[0103]

[0104] FIG. 6 is a schematic flowchart explaining an in-memory-based data processing method of the present invention, and the data processing method is explained with reference to the system (100) of FIGS. 1 to 4.

[0105] In order to provide an application service using the system (100) of the present invention, it is necessary to construct data of the SDB module (160), which is the internal DB of the initial system (100).

[0106] The system (100) of the present invention loads some of the data necessary for system operation from among the original data stored in an external RDBMS (300) in bulk data format to build data of the SDB module (160) (S610).

[0107] At this time, the data stored in the SDB module (160) has a data storage format of SQL structure, like an external RDBMS (300). The data stored in the SDB module (160) includes application service-related environment setting information for each user and each service.

[0108] For the next fast data processing, some data required for the SDB module (160) is filtered to build an in-memory DB (148) (S620).

[0109] At this time, the format of the data stored in the in-memory DB (148) is a non-relational data storage structure of the NoSQL structure as described above.

[0110] The system (100) collects and analyzes data from the edge (400) (S630).

[0111] The system (100) determines whether the preset action conditions are satisfied through packet analysis of the collected data (S640).

[0112] If a preset action condition is satisfied, an action command corresponding to the condition is generated and transmitted to the corresponding edge (400) (S650).

[0113] If the preset action condition is not satisfied in step S640 or after step S650, the system (100) performs transaction processing using a shared memory communication method based on data stored in the in-memory DB (148) (S660).

[0114] The system (100) of the present invention can maximize the transaction processing speed by retrieving data of a hash index structure from the main memory in an in-memory manner, and in addition, can maximize the processing speed for multiple transactions without creating a session for each transaction through a shared memory method.

[0115] After step S660, if the data of the in-memory DB (148) is changed, the data of the SDB module (160) and the external RDBMS (300) are updated based on the changed information (S670).

[0116] At this time, the update of the data of the SDB module (160) can be performed by receiving the updated information of the in-memory DB (148) in bulk data format.

[0117] Additionally, updating data in the RDBMS (300) can be performed by receiving a batch file containing update information from the SDB module (160).

[0118]

[0119] By adopting the above configuration and method, the in-memory-based data processing system and method of the present invention have the effect of processing transactions without service interruption even when a large amount of edge data is transmitted in a short period of time.

[0120] Those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering its technical spirit or essential characteristics. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

Claims

1. A SNET module configured to enable edge and bidirectional online transaction processing (OLTP), which collects and analyzes data from the edge, generates action commands according to preset action conditions, and transmits them to the edge; An SDB module that stores information collected from an external RDBMS and environment information for each user and service; and An in-memory-based data processing system including a CAST module that constructs some filtered data required for an application service as an in-memory DB in the above SDB module and processes a transaction of the application service based on the data analysis results in the above SNET module and the above in-memory DB.

2. In paragraph 1, The above SNET module includes a packet reader module that collects data from the edge; and a packet analysis module that analyzes data collected by the packet reader module. The above packet analysis module, Parsing part that parses packets; A filtering unit that selects storage data based on preset storage conditions and stores the data in the in-memory DB of the CAST module; and An in-memory-based data processing system including an action processing unit that stores and manages the preset action conditions and generates the action commands.

3. In paragraph 2, The above packet analysis module is an in-memory-based data processing system that further includes an output section that visualizes and outputs analyzed packet information and environment setting information for each packet data through a user interface.

4. In paragraph 1, The above in-memory DB has a non-relational data storage structure in the NoSQL style. The above CAST module is an in-memory-based data processing system characterized by performing transaction processing in a shared memory manner.

5. In paragraph 4, The above in-memory DB is an in-memory-based data processing system characterized by having a hash index-based data storage structure.

6. In paragraph 4, An in-memory-based data processing system, characterized in that the CAST module further includes a multi-DB layer that converts data stored in the in-memory DB into a NoSQL format so that it is compatible with multiple different types of RDBMS.

7. In paragraph 4, The above SDB module is an in-memory-based data processing system characterized in that it transmits some filtered data required for application service to the in-memory DB in bulk.

8. In paragraph 7, The above SDB module is an in-memory-based data processing system characterized in that it transmits updated information to an external RDBMS in the form of a batch file. 9.(a) A step of storing information collected from an external RDBMS and user-specific and service-specific environmental information in the SDB module; (b) a step in which the CAST module builds some filtered data required for application service in the SDB module into an in-memory DB; (c) a step in which the SNET module collects and analyzes data from the edge; (d) a step of transmitting the SNET module to the edge to generate an action command according to preset action conditions based on the data analysis results; and (e) An in-memory-based data processing method including a step of processing an application service transaction based on the data analysis results from the SNET module and the in-memory DB by the CAST module.

10. In paragraph 9, The above in-memory DB has a non-relational data storage structure in the NoSQL style. The above CAST module is an in-memory-based data processing method characterized in that it performs transaction processing in a shared memory manner.

Images