Real-time data loading system using message queue and execution method thereof

Abstract

A real-time data loading system using a message queue according to the present invention comprises: a message queue; a loading module that determines whether there are a plurality of processing commands of a target to be loaded when the target to be loaded is read from the message queue, and loads the target to be loaded in a database on the basis of table information of the target to be loaded and whether the plurality of processing commands are identical according to the result of the determination; and a distributed database in which the target to be loaded provided through the loading module is loaded.

Description

Real-time data loading system using message queue and method for executing the same

[0001] The present invention relates to a real-time data loading system using a message queue and an execution method thereof, and more specifically, to a real-time data loading system using a message queue and an execution method thereof, which ensures data suitability by loading the load target into a database based on whether the table information of the load target and the plurality of processing commands are identical when there are multiple processing commands for the load target read from the message queue.

[0002] Typically, the message distribution method of a large-scale distributed system uses a method in which a message provider publishes a data list to a database, specifies a recipient and published list information, then sends a message to a message queue, and the designated recipient retrieves the message from the message queue and searches the database for a list of data related to the received message based on the published list information.

[0003] When processing data in units in a distributed database through the loading module, each data record is processed as a single loading operation. That is, each data record is individually read through the message queue and loaded.

[0004] This method facilitates parallel processing and guarantees real-time performance, as each data record is processed independently. Furthermore, since each load task is processed individually upon success or failure, rollback and retry are easy in the event of an error.

[0005] However, processing data in chunks can increase the overhead of the operation. This is because additional work may be required to start and complete the load operation each time.

[0006] Meanwhile, when processing data in a distributed database on a single-item basis through the loading module, multiple data records are bundled together and loaded. In other words, multiple data records are included in a single message, read through the message queue, and loaded.

[0007] This method allows for batch processing, which reduces the overhead of tasks required to load data, and is more efficient than unit work because multiple data records are processed in a single load operation.

[0008] However, when processing multiple instances, the problem arises that if each data record fails individually, the entire load operation may be halted. Therefore, to maintain data consistency, the entire load operation may need to be rolled back and retried.

[0009] Therefore, processing data in chunks facilitates individual processing and ensures real-time performance, but can incur overhead. Conversely, processing multiple chunks allows for batch processing, which is efficient, but requires attention to error handling and consistency.

[0010] Korean Patent Publication No. 10-2013-0107276 relates to message queue management, and includes a step of managing a shared input message queue for processing messages for multiple components of an application, a step of detecting a non-responsive component among the multiple components, and a step of separating the non-responsive component from the shared input message queue. However, no solution for solving the above problem is disclosed.

[0011] The present invention aims to provide a real-time data loading system using a message queue and an execution method thereof, which ensures data suitability by loading the loading target into a database based on whether the table information of the loading target and the plurality of processing commands are identical when there are multiple processing commands for the loading target read from the message queue.

[0012] In addition, the present invention aims to provide a real-time data loading system using a message queue and an execution method thereof, which can improve loading performance while ensuring consistency of the loading target by loading the loading target into a distributed database when the table and loading table key of the loading target are the same and the processing command is the same.

[0013] In addition, the present invention aims to provide a real-time data loading system using a message queue and an execution method thereof, which can ensure data consistency by storing the loading target in a queue linked to a distributed database when the table and loading table key of the loading target are the same and the processing command is different.

[0014] A real-time data loading system using a message queue to achieve this purpose includes a message queue, a loading module that reads a loading target from the message queue, determines whether there are multiple processing commands for the loading target, and loads the loading target into a database based on the result of the determination, depending on whether the table information of the loading target and the multiple processing commands are the same, and a distributed database into which the loading target provided through the loading module is loaded.

[0015] In one embodiment, the loading module can load the loading target into the database based on whether the table, loading table key, and type of processing command of each of the loading targets are the same when there are multiple processing commands for the loading target.

[0016] In one embodiment, the loading module can determine whether the tables of each of the loading targets are identical when there are multiple processing commands for the loading targets, and as a result of the determination, determine whether the loading table keys of each of the tables of the loading targets are identical.

[0017] In one embodiment, the loading module can determine whether the types of the processing commands are the same if the loading table keys of the tables are the same as a result of the determination, and process the processing commands according to the result of the determination.

[0018] In one embodiment, the loading module may process the processing command if the types of the processing commands are the same, and store the loading target in a queue linked to a distributed database if the types of the processing commands are not the same.

[0019] In addition, a real-time data loading method using a message queue, which is executed in a real-time data loading system using a message queue to achieve this purpose, includes a step of determining whether there are multiple processing commands for the loading target when reading a loading target from a message queue, a step of determining whether table information of the loading target and the multiple processing commands are the same based on the determination result, and a step of loading the loading target into a distributed database based on the determination result.

[0020] In one embodiment, the step of determining whether the table information of the load target and the plurality of processing commands are the same based on the judgment result may, if there are a plurality of processing commands for the load target, load the load target into a distributed database based on whether the types of the tables, loading table keys, and processing commands of each of the load targets are the same.

[0021] In one embodiment, the step of determining whether the table information of the load target and the plurality of processing commands are the same based on the determination result may include the step of determining whether the tables of each of the load targets are the same if there are a plurality of processing commands of the load targets, and the step of determining whether the load table keys of the tables of each of the load targets are the same based on the determination result.

[0022] In one embodiment, the step of determining whether the load table keys of each of the tables of the load targets are the same as a result of the determination may include the step of determining whether the types of the processing commands are the same if the load table keys of the tables are the same as a result of the determination, and the step of processing the processing commands according to the result of the determination.

[0023] In one embodiment, the step of processing the processing command according to the judgment result may include a step of processing the processing command if the types of the processing commands are the same, and a step of storing the load target in a queue linked to a distributed database if the types of the processing commands are not the same.

[0024] According to the present invention as described above, when there are multiple processing commands for a load target read from a message queue, there is an advantage in that data suitability can be guaranteed by loading the load target into a database based on whether the table information of the load target and the multiple processing commands are identical.

[0025] In addition, according to the present invention, there is an advantage in that the load performance can be improved while ensuring the consistency of the load target by loading the load target into a distributed database when the table and load table key of the load target are the same and the processing command is the same.

[0026] In addition, according to the present invention, when the table and the load table key of the load target are the same and the processing command is different, there is an advantage in that data consistency can be guaranteed by storing the load target in a queue linked to a distributed database.

[0027] Figures 1 and 2 are drawings for explaining a data loading system using a conventional message queue.

[0028] FIG. 3 is a network configuration diagram for explaining a real-time data loading system using a message queue according to one embodiment of the present invention.

[0029] FIG. 4 and FIG. 5 are flowcharts illustrating one embodiment of a real-time data loading method using a message queue according to the present invention.

[0030]

[0031] <Explanation of symbols>

[0032] 100: Data Provider

[0033] 200: Message Queue

[0034] 300: Loading module

[0035] 400_1~400_N: Distributed database

[0036] The above-described objects, features, and advantages will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can easily practice the technical idea of ​​the present invention. In describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, a detailed description thereof will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

[0037] As used herein, a "message queue" is a system that asynchronously transfers data between a sender and a receiver. It acts as a sort of intermediary, with the sender placing a message in the queue, and the receiver then retrieves it as needed.

[0038] These message queues are used in situations where communication between the sender and receiver does not need to occur simultaneously. The sender can queue a message and then continue with its work, while the receiver can retrieve and process the message as needed.

[0039] Additionally, message queues are used to facilitate data exchange between multiple systems. Each system queues messages, allowing other systems to retrieve them when needed.

[0040] Additionally, message queues are used in distributed systems, which can improve system scalability. Distributing message queues across multiple servers can increase throughput.

[0041]

[0042] Figures 1 and 2 are drawings for explaining a data loading system using a conventional message queue.

[0043] Referring to FIGS. 1 and 2, when data is processed in units of data in a distributed database through a loading module as in FIG. 1, one data record is processed as one loading task. That is, each data record is individually read through a message queue and loaded.

[0044] This method facilitates parallel processing and guarantees real-time performance, as each data record is processed independently. Furthermore, since each load task is processed individually upon success or failure, rollback and retry are easy in the event of an error.

[0045] However, processing data in chunks can increase the overhead of the operation. This is because additional work may be required to start and complete the load operation each time.

[0046] When processing a distributed database in a single unit through the loading module, multiple data records are bundled together and loaded. In other words, multiple data records are included in a single message, read through the message queue, and loaded.

[0047] This method allows for batch processing, which reduces the overhead of tasks required to load data, and is more efficient than unit work because multiple data records are processed in a single load operation.

[0048] However, when processing multiple instances as shown in Figure 2, there is a problem in that if each data record fails individually, the entire load operation may be halted. Therefore, to maintain data consistency, the entire load operation may need to be rolled back and retried.

[0049] Therefore, processing data in chunks facilitates individual processing and ensures real-time performance, but can incur overhead. Conversely, processing multiple chunks allows for batch processing, which is efficient, but requires attention to error handling and consistency.

[0050]

[0051] FIG. 3 is a network configuration diagram for explaining a real-time data loading system using a message queue according to one embodiment of the present invention.

[0052] Referring to FIG. 3, a real-time data loading system using a message queue includes a data provision unit (100), a message queue (200), a loading module (300), and a distributed database (400_1 to 400_N).

[0053] When loading data into the message queue (200), the data provider (100) assigns a data type and a loading table key to the data to be loaded and loads the data into the message queue (200).

[0054] First, the data provider (100) determines the type of data to be loaded when loading data into the message queue (200).

[0055] In one embodiment, the data provider (100) may determine a loading type based on the size, type, and format of the data. In this case, the loading type may include a single record loading method and a batch processing loading method.

[0056] For example, if a system has log data generated periodically, the log data is comprised of multiple small-sized records. In this case, the data provider (100) can select a single-record loading method to immediately load each log event into the database, enabling real-time data processing and analysis.

[0057] In another embodiment, the data provider (100) may determine the loading type based on the processing capability and performance of the target loading system. That is, the data provider (100) may select a single record loading method or a batch loading method, taking into account the processing capacity of the system.

[0058] For example, a financial institution that must process millions of transaction data items daily may choose a batch loading method to efficiently process large amounts of data. All transaction data is loaded at once at a set time each day, stored in a database, and subsequently analyzed and processed.

[0059] In another embodiment, the data provider (100) may determine the loading type based on the real-time requirements of the data. That is, if the data provider (100) needs to load data immediately in real time, a single record loading method may be suitable.

[0060] For example, in a case where order information needs to be loaded into a DB in real time whenever an order is placed in an online store, the data provider (100) selects a single record loading method that loads each order into the DB immediately, so that when an order is placed, the order information is loaded into a distributed database (400_1 to 400_N) immediately via a message queue (200), thereby enabling real-time tracking and processing of the order status.

[0061] In another embodiment, the data provider (100) can determine the load type to maintain data integrity and consistency.

[0062] For example, in a bank, since the consistency of transactions is guaranteed whenever a financial transaction occurs, the data provider (100) selects a single record loading method that loads data in units of transactions, so that when each transaction occurs, the transaction information is loaded into a distributed database (400_1 to 400_N) via a message queue (200) in units of transactions, thereby ensuring the consistency of data.

[0063] In another embodiment, the data provider (100) may determine the load type based on network bandwidth and load. That is, when a large amount of data must be transmitted, the data provider (100) may consider a batch processing method to minimize network load.

[0064] For example, in the case of a large amount of sensor data generated in real time from a sensor device, network bandwidth must be considered in order to process the large amount of data in real time. Therefore, the data provider (100) selects a batch processing loading method to efficiently process the large amount of data, thereby periodically bundling a certain amount of sensor data and loading it into a DB through a message queue, thereby minimizing network load and improving performance.

[0065] Additionally, the data provider (100) assigns a loading table key to the data to be loaded when loading data into the message queue (200). This refers to the primary key or foreign key of the table into which the data will be loaded. By assigning a loading table key to the data to be loaded when loading data into the message queue (200), data consistency can be maintained.

[0066] The message queue (200) transmits and receives data using an asynchronous communication method. The message queue (200) provides a message to the loading module (300) at the request of the loading module (300).

[0067] The loading module (300) manages distributed transactions using a message queue (200) in a distributed database system.

[0068] When the loading module (300) reads a loading target from the message queue (200), it determines whether there are multiple processing commands for the loading target, and based on the determination result, it can load the loading target into a distributed database (400_1 to 400_N) based on whether the table information of the loading target and the multiple processing commands are the same.

[0069] The loading module (300) loads the loading target into a distributed database based on whether the table, loading table key, and type of processing command of each loading target are the same when there are multiple processing commands for the loading target.

[0070] First, the loading module (300) can determine whether the tables of each of the loading targets are the same if there are multiple processing commands for the loading targets, and as a result of the determination, can determine whether the loading table keys of each of the tables of the loading targets are the same.

[0071] In one embodiment, the loading module (300) can determine whether the types of processing commands are the same if the loading table keys of the tables are the same, and as a result of the determination, can determine whether the loading table keys of each table of the loading target are the same.

[0072] That is, the loading module (300) can determine whether the types of the processing commands are the same if the loading table keys of the judgment result table are the same, and process the processing commands according to the judgment result.

[0073] In one embodiment, the loading module (300) can determine whether the tables of each loading target are identical when there are multiple processing commands for the loading target.

[0074] If the tables of each load target are the same, the loading module (300) determines whether the load table keys (i.e., primary keys and unique keys) of each load target table are the same, and if the tables of each load target are not the same, the load target can be stored in a queue linked to a distributed database.

[0075] In the above embodiment, the loading module (300) can determine whether the loading table keys of each table of the loading target are the same, and based on the determination result, determine whether the types of the processing commands are the same.

[0076] If, as a result of the judgment, the loading module (300) determines whether the types of processing commands are the same if the loading table keys of each table of the loading targets are the same, and if the loading table keys of each table of the loading targets are not the same, the loading targets can be stored in a queue linked to a distributed database (400_1 to 400_N).

[0077] In the above embodiment, the loading module (300) processes the processing commands depending on whether the types of the processing commands are the same, and if the types of the processing commands are not the same, the loading target can be stored in a queue linked to a distributed database (400_1 to 400_N).

[0078] Distributed databases (400_1~400_N) manage and process distributed transactions.

[0079] When a load target is loaded from the load module (300), the distributed database (400_1~400_N) executes a processing command for the load target.

[0080]

[0081] FIG. 4 and FIG. 5 are flowcharts illustrating one embodiment of a real-time data loading method using a message queue according to the present invention.

[0082] Referring to FIGS. 4 and 5, when the loading module (300) reads a loading target from the message queue (step S410), the loading module (300) determines whether the loading target meets the loading conditions if there are multiple processing commands for the loading target (step S420).

[0083] Below, the process of determining whether a loading target meets the loading conditions will be described with reference to FIG. 5.

[0084] The loading module (300) loads the loading target into a distributed database based on whether the table, loading table key, and type of processing command of each loading target are the same when there are multiple processing commands for the loading target.

[0085] First, the loading module (300) determines whether the tables of each of the loading targets are the same if there are multiple processing commands for the loading targets (step S510), and as a result of the determination, determines whether the loading table keys of each of the tables of the loading targets are the same (step S520).

[0086] In one embodiment, the loading module (300) can determine whether the tables of each loading target are identical when there are multiple processing commands for the loading target (step S510).

[0087] If the tables of each of the load targets are identical (step S510), the loading module (300) determines whether the load table keys (i.e., primary keys and unique keys) of each of the load targets are identical (step S520), and if the tables of each of the load targets are not identical (step S510), the load targets can be stored in a queue linked to a distributed database.

[0088] In the above embodiment, the loading module (300) can determine whether the loading table keys of each table of the loading target are the same (step S520), and can determine whether the types of the processing commands are the same based on the determination result (step S530).

[0089] If the loading module (300) determines that the loading table keys of each table of the loading targets are the same as a result of the judgment, it determines whether the types of processing commands are the same (step S530), and if the loading table keys of each table of the loading targets are not the same, it can store the loading targets in a queue linked to a distributed database (400_1 to 400_N).

[0090] In the above embodiment, the loading module (300) processes the processing commands depending on whether the types of the processing commands are the same, and if the types of the processing commands are not the same, the loading target can be stored in a queue linked to a distributed database (400_1 to 400_N).

[0091]

[0092] Although the present invention has been described with reference to the above-described embodiments and drawings, it is not limited to the above-described embodiments. Those skilled in the art will appreciate that various modifications and variations are possible based on this disclosure. Therefore, the scope of the present invention should be understood solely by the scope of the claims set forth below, and all equivalent or equivalent modifications thereof are deemed to fall within the scope of the present invention.

[0093] According to the present invention, when there are multiple processing commands for a load target read from a message queue, data suitability can be guaranteed by loading the load target into a database based on whether the table information of the load target and the multiple processing commands are identical. The present invention is a technology that can be widely used in the big data industry to realize its practical and economic value.

Claims

1. Message queue; A loading module that reads a load target from the message queue, determines whether there are multiple processing commands for the load target, and loads the load target into a distributed database based on the determination result, depending on whether the table information of the load target and the multiple processing commands are the same; and It is characterized by including a distributed database into which the load target provided through the above load module is loaded. Real-time data loading system using message queue.

2. In paragraph 1, The above loading module If there are multiple processing commands for the above load target, the load target is loaded into the distributed database depending on whether the table, load table key, and type of processing command for each of the load targets are the same. Real-time data loading system using message queue.

3. In paragraph 2, The above loading module If there are multiple processing commands for the above load targets, it is determined whether the tables of each of the load targets are the same, and as a result of the determination, it is determined whether the load table keys of each of the tables of the load targets are the same. Real-time data loading system using message queue.

4. In paragraph 3, The above loading module If the load table key of the table is the same as the result of the above judgment, it is determined whether the types of the processing commands are the same, and the processing commands are processed according to the result of the judgment. Real-time data loading system using message queue.

5. In paragraph 4, The above loading module If the types of the above processing commands are the same, the processing commands are processed, and if the types of the above processing commands are not the same, the load target is stored in a queue linked to a distributed database. Real-time data loading system using message queue.

6. In a real-time data loading method using a message queue that is executed in a real-time data loading system using a message queue, A step of determining whether there are multiple processing commands for a load target when reading a load target from a message queue; A step of determining whether the table information of the load target and the plurality of processing commands are the same based on the above judgment result; and It is characterized by including a step of loading the load target into a distributed database according to the above judgment result. A method for loading real-time data using a message queue.

7. In paragraph 6, The step of determining whether the table information of the load target and the plurality of processing commands are the same based on the above judgment result is If there are multiple processing commands for the above load target, the load target is loaded into the distributed database depending on whether the table, load table key, and type of processing command for each of the load targets are the same. A method for loading real-time data using a message queue.

8. In paragraph 7, The step of determining whether the table information of the load target and the plurality of processing commands are the same based on the above judgment result is If there are multiple processing commands for the above load target, a step of determining whether the tables of each of the above load targets are the same; and The above judgment result is characterized by including a step of judging whether the loading table keys of each table of the loading target are the same. A method for loading real-time data using a message queue.

9. In paragraph 8, The step of determining whether the load table keys of each table of the load target are the same as a result of the above judgment is A step of determining whether the types of the processing commands are the same if the load table keys of the tables are the same as a result of the above judgment; and characterized in that it includes a step of processing the processing command according to the above judgment result. A method for loading real-time data using a message queue.

10. In paragraph 9, The step of processing the processing command according to the above judgment result is A step of processing the processing command if the types of the processing commands are the same; If the types of the above processing commands are not the same, it is characterized by including a step of storing the load target in a queue linked to a distributed database. A method for loading real-time data using a message queue.

Images