Data Transfer Management Device, Data Transfer Management Method, and Data Transfer Management System

Abstract

To provide a data movement management technique, in a hybrid transaction analysis architecture, that flexibly adjusts a cyclical movement timing of data from an online transaction processing (OLTP) database to an online analysis processing (OLAP) database.SOLUTION: In a data movement management system 200, a data movement management device includes a query log management unit that detects a federated query requesting a first data set from an OLTP database from a first query log set of an OLAP database and identifies a first data freshness requirement of the OLAP database based on the federated query, a data movement management unit that determines a data management operation based on at least the first data freshness requirement, and a data movement timing management unit that determines an updated cyclical movement timing to transfer batch data from the OLTP database to the OLAP database.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a data transfer management device, a data transfer management method, and a data transfer management system.

Background Art

[0002] A database management system is a tool used by companies or other organizations to collect, store, and organize data. Generally, a database management system refers to a system for storing and managing data in a structured format, handles the data in the database, and provides functionality to facilitate access to the data for use in various applications.

[0003] An online transaction processing system (OLTP system) and an online analytical processing system (OLAP system) are two types of database processing systems that can be used to manage data in a database.

[0004] A database that uses an OLTP system (hereinafter, an "OLTP database") captures and holds transaction data in the database. An OLTP database is designed to support high-speed query processing, real-time updates, high concurrency, and strong consistency. Each transaction requires individual database records consisting of a large number of fields or columns. Generally, a query only modifies a few rows of data at a time. Examples of OLTP database applications include banking and credit card operations or retail store checkouts.

[0005] On the other hand, a database that uses an OLAP system (hereinafter referred to as an "OLAP database") applies complex queries to a large amount of historical data containing a small amount of transactions aggregated from an OLTP database and other sources for data mining, analysis, and business intelligence projects. Queries to an OLAP database generally require one or more data columns aggregated from many rows. Examples of OLAP database applications include year-on-year financial performance or prospecting trend. OLAP databases and data warehouses give analysts and decision-makers the ability to derive insights from data using customer reporting tools.

[0006] Generally, due to these differences, OLTP queries and OLAP queries are processed in separate databases. However, in recent years, the boundary between the workloads of transactions and analysis has become blurred, and OLAP systems have become more transactional in nature, and OLTP systems have become more analytical in nature. For example, to facilitate statistical analysis, it may be necessary to execute short queries with high parallelism on the historical data stored in an OLAP database and perform small-range queries generally implemented in an OLTP system. Conversely, when performing a transaction on an OLTP database for a given application, it may be necessary to perform a large-scale analysis on the historical information stored in an OLAP database.

[0007] To facilitate the realization of a hybrid transactional analysis processing system (HTAP) that includes OLTP and OLAP processes, it is necessary to efficiently move data from an OLTP database to an OLAP database using extraction, transformation, and loading (ETL) technologies and other integration tools.

[0008] Conventionally, technologies for facilitating data movement in an HTAP system have been studied.

[0009] For example, Non-Patent Document 1 discloses that "IBM Db2 Analytics Accelerator (IDAA) is a state-of-the-art hybrid database system that seamlessly extends the high transaction performance of Db2 for z / OS, including ultra-fast column store processing in the Db2 data warehouse. This accelerator holds a copy of the data from Db2 for z / OS in a back-end database. Depending on the table granularity and one or more of its partitions, or gradually when a row changes, replication technology can be used to synchronize the data at a single point in time. Since IDAA version 3, IBM Change Data Capture (CDC) has been adopted as the replication technology. In version 7.5.0, an excellent replication technology - unified synchronization - has been introduced as an alternative to the use of CDC by IDDA. This paper presents how unified synchronization has improved performance by orders of magnitude and paved the way for near real-time hybrid transactional analysis (HTAP) processing."

Prior Art Documents

Non-Patent Documents

[0010]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0011] Generally, in an HTAP system, data is periodically moved (e.g., once every 24 hours) from an OLTP database to an OLAP database. However, in some cases, the OLAP database needs to access the latest data that has not yet been moved from the OLTP database to the OLAP database in order to perform analysis tasks and others. In such cases, the OLAP database can generate a federated query, which is a query for requesting the transfer of a specific data set in the form of a temporary table. However, such a federated query has a high computational cost, increases the load on the OLTP database, and potentially leads to a slowdown in query processing time and a decrease in search performance.

[0012] Non-Patent Document 1 discloses a technique for promoting HTAP processing by performing high-speed data replication from a high-performance transaction database system to a column store database system. However, in the technique described in Non-Patent Document 1, the periodic transfer timing is performed at regular intervals, and factors such as the network load and CPU load of the database system are not considered when determining the periodic transfer timing.

[0013] Therefore, in view of the above problems, an object of the present disclosure is to provide a data transfer management technique for flexibly adjusting the periodic transfer timing of data from an OLTP database to an OLAP database based on data freshness requirements, network and computing resource availability, and database status in order to improve the search performance of analysis and transaction queries in an HTAP system.

Means for Solving the Problems

[0014] One representative example of the present disclosure relates to a data movement management device for managing data in a hybrid transaction analysis architecture. The data movement management device detects an association query for requesting a first data set from an online transaction processing database from a first query log set of an online analytical processing database, and a query log management unit for identifying a first data freshness requirement of the online analytical processing database based on the association query, and a data movement management unit for determining a data management operation including at least copying a first data set from the online transaction processing database to the online analytical processing database based on at least the first data freshness requirement, and a data movement timing management unit for determining an updated periodic movement timing for transferring batch data from the online transaction processing database to the online analytical processing database when the frequency of the association query from the online analytical processing database reaches a first frequency threshold.

Advantages of the Invention

[0015] According to the present disclosure, in order to promote the search performance of analysis and transaction queries in an HTAP system, it is possible to provide a data movement management technique for flexibly adjusting the periodic movement timing of data transferred from an OLTP database to an OLAP database based on data freshness requirements, network and computing resource availability, and database status.

[0016] Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments for carrying out the present invention.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

DETAILED DESCRIPTION OF THE INVENTION

[0018] In this specification, embodiments of the present invention are described with reference to the drawings. It should be noted that the embodiments described in this specification are not intended to limit the invention according to the claims, and it should be understood that each of the elements and combinations thereof described in the embodiments is not strictly necessary for implementing aspects of the present invention.

[0019] The following description and the related figures disclose various aspects. Alternative aspects may be devised without departing from the scope of the disclosure. Additionally, well-known elements of the present disclosure are not described in detail or are omitted so as not to obscure the relevant details of the disclosure.

[0020] The words "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Similarly, the phrase "aspects of the disclosure" does not require that all aspects of the disclosure include the recited features, advantages, or modes of operation.

[0021] Furthermore, many aspects are described as sequences of operations performed by elements of a computing device. It will be recognized that the various operations described herein may be implemented by a specific circuit (e.g., an application specific integrated circuit (ASIC)), by program instructions executed by one or more processors, or by a combination of both.

[0022] Additionally, the sequences of operations described herein may be considered to be embodied entirely in a form of a computer-readable storage medium storing a corresponding set of computer instructions that, when executed at runtime, cause the relevant processor to implement the functionality described herein. Thus, the various aspects of the disclosure may be embodied in several different forms, all of which are considered to be within the scope of the subject matter recited in the claims.

[0023] FIG. 1 depicts a high-level block diagram of a computer system 100 according to an embodiment for implementing various embodiments of the present disclosure. The mechanisms and apparatus of the various embodiments disclosed herein are equally applicable to corresponding computing systems. The main components of the computer system 100 include one or more processors 102, a memory 104, a terminal interface 112, a storage device interface 113, an input / output (I / O) device interface 114, and a network interface 115, all of which are directly or indirectly communicatively coupled for component-to-component communication via a memory bus 106, an I / O bus 108, a bus interface unit 109, and an I / O bus interface unit 110.

[0024] The computer system 100 may house one or more general-purpose programmable central processing units (CPUs) 102A and 102B, generically referred to herein as processors 102. In an embodiment, the computer system 100 may house a number of processors, however, in some embodiments, the computer system 100 may alternatively be a single CPU system. Each processor 102 executes instructions stored in the memory 104 and may include one or more levels of on-board cache.

[0025] In an embodiment, the memory 104 may include a random access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile) for storing or encoding data and programs. In certain embodiments, the memory 104 represents the entire virtual memory of the computer system 100 and may also include the virtual memory of other computer systems coupled to the computer system 100 or connected via a network. Conceptually, the memory 104 may be regarded as a single monolithic entity, but in other embodiments, the memory 104 is a more complex mechanism, such as a hierarchy with caches and other memory devices. For example, the memory exists at multiple cache levels, and these caches are further divided by function, so that one cache holds instructions while another cache holds non-instruction data used by one or more processors. As is known in any of various so-called non-uniform memory access (NUMA) computer architectures, the memory may be further distributed and coordinated with different CPUs or sets of CPUs.

[0026] The memory 104 may store all or a portion of various programs, modules, and data structures for processing data transfers as described herein. By way of example, the memory 104 may store the data movement management application 150. In an embodiment, the data movement management application 150 may include instructions or statements executed by the processor 102, or instructions or statements interpreted by instructions or statements executed by the processor 102 and performing functions as further described below.

[0027] In some embodiments, the data movement management application 150 is implemented in hardware through a semiconductor device, chip, logic gate, circuit, circuit card, and / or other physical hardware devices instead of or in addition to a processor-based system. In embodiments, the data movement management application 150 may include data in addition to instructions or statements. In some embodiments, a camera, sensor, or other data input device (not shown) may be provided in direct communication with the bus interface unit 109, the processor 102, or other hardware of the computer system 100. In such a configuration, the need for the processor 102 to access the memory 104 and the latent factor identification application is reduced.

[0028] The computer system 100 may include a bus interface unit 109 that handles communications between the processor 102, the memory 104, the display system 124, and the input / output bus interface unit 110. The input / output bus interface unit 110 may be coupled to an input / output bus 108 for transferring data to and from various input / output units. The input / output bus interface unit 110 communicates with a number of input / output interface units 112, 113, 114, 115, also known as input / output processors (IOPs) or input / output adapters (IOAs), through the input / output bus 108. The display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both types of data to the display device 126. Further, the computer system 100 may include one or more sensors or other devices configured to collect data and provide it to the processor 102.

[0029] As an example, the computer system 100 may include a biometric sensor (e.g., collecting heart rate data, stress level data), an environmental sensor (e.g., collecting humidity data, temperature data, pressure data), a motion sensor (e.g., collecting acceleration data, motion data), and others. Other types of sensors are also possible. The display memory may be dedicated memory for buffering video data. The display system 124 may be coupled to a display device 126 such as a stand-alone display screen, a computer monitor, a television, or a tablet or handheld device display.

[0030] In one embodiment, the display device 126 may include one or more speakers for playing audio. Alternatively, one or more speakers for playing audio may be coupled to the input / output interface unit. In an alternative embodiment, one or more of the functions provided by the display system 124 may be mounted on an integrated circuit that also includes the processor 102. Additionally, one or more of the functions provided by the bus interface unit 109 may be mounted on an integrated circuit that also includes the processor 102.

[0031] The input / output interface unit supports communication with various storage and input / output devices. For example, the terminal interface unit 112 supports the attachment of one or more user input / output devices 116 that may include a user output device (such as a video display device, a speaker, and / or a television) and a user input device (such as a keyboard, a mouse, a keypad, a touchpad, a trackball, a button, a light pen, or other pointing device). The user may operate the user input device using a user interface to provide input data and commands to the user input / output device 116 and the computer system 100, and receive output data via the user output device. For example, the user interface may be presented via the user input / output device 116, such as being displayed on a display device, emitted via a speaker, or printed via a printer.

[0032] The memory device interface 113 supports the attachment of one or more disk drives or direct access storage devices 117 (commonly rotating magnetic disk drive storage devices, but alternatively other storage devices including an array of disk drives configured to be provided as a single large storage device to a host computer, or a solid state drive such as flash memory). In some embodiments, the storage device 117 may be implemented via some type of secondary storage device. The contents of the memory 104 or any portion thereof may be stored in the storage device 117 and retrieved therefrom as needed. The input / output device interface 114 interfaces to various other input / output devices or to any of various other types of devices such as printers or fax. The network interface 115 provides one or more communication channels from the computer system 100 to other digital devices and computer systems. These communication channels may include, for example, one or more networks 130.

[0033] The computer system 100 shown in FIG. 1 shows a particular bus structure providing direct communication channels between the processor 102, the memory 104, the bus interface 109, the display system 124, and the input / output bus interface unit 110, but in alternative embodiments, the computer system 100 may include a variety of buses or communication channels arranged in any of a variety of forms, such as hierarchical star or web configurations, multiple hierarchical buses, parallel and redundant paths, or point-to-point links in any other suitable type of configuration. Further, while the input / output bus interface unit 110 and the input / output bus 108 are shown as a single such unit, the computer system 100 may in fact house a number of input / output bus interface units 110 and / or a number of input / output buses 108. A number of input / output interface units separating the input / output bus 108 from various communication channels leading to various input / output devices are shown, but in other embodiments, some or all of the input / output devices are directly connected to one or more system input / output buses.

[0034] In various embodiments, computer system 100 is a multi-user mainframe computer system, a single-user system, or a server computer or similar device that has little or no direct user interface and accepts requests from other computer systems (clients). In other embodiments, computer system 100 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or other suitable type of electronic device.

[0035] Next, a configuration example of a data movement management system according to an embodiment of the present disclosure will be described with reference to FIG. 2.

[0036] FIG. 2 shows a configuration example of a data movement management system 200 according to an embodiment of the present disclosure. The data movement management system 200 is a system for flexibly adjusting the periodic movement timing at which data is moved from OLTP data to an OLAP database. As shown in FIG. 2, the data movement management system 200 includes an OLTP device 210, an OLAP device 220, and a data movement management device 240 that are communicatively connected by a communication network 230. Here, the communication network 230 may include a local area network (LAN) connection, the Internet, a wide area network (WAN) connection, a metropolitan area network (MAN) connection, and others.

[0037] The OLTP device 210 is a hardware device that supports the operation of the OLTP database 214. As shown in FIG. 2, the OLTP device 210 may include a CPU 211, a memory 212, a storage device 213 for storing the OLTP database 214, and a network interface 215. In an embodiment, the OLTP device 210 may be implemented using a computer system such as the computer system 100 shown in FIG. 1. As described herein, the OLTP database 214 may include a database configured to support high-speed query processing, real-time updates, high parallelism, and strong consistency. Each transaction requires individual database records consisting of a number of fields or columns. Generally, a query modifies only a small number of rows of data at a time. Examples of OLTP database applications include banking and credit card operations or retail store point-of-sale scanning.

[0038] The OLAP device 220 is a hardware device that supports the operation of the OLAP database 224. As shown in FIG. 2, the OLAP device 220 may include a CPU 221, a memory 222, a storage device 223 for storing the OLAP database 224, and a network interface 225. In an embodiment, the OLAP device 220 may be implemented using a computer system such as the computer system 100 shown in FIG. 1. As described herein, the OLAP database 224 may be configured to apply complex queries with a small number of transactions to large amounts of historical data aggregated from an OLTP database (e.g., OLTP database 214) and other sources for data mining, analysis, and business intelligence projects. Queries to the OLAP database 224 may require one or more data columns aggregated from a large number of rows. Examples of OLAP database applications include year-over-year financial performance or prospective customer acquisition trends.

[0039] The data transfer management device 240 is a hardware device for managing the transfer of data from the OLTP database 214 of the OLTP device 210 to the OLAP database 224 of the OLAP device 220. As shown in FIG. 2, the data transfer management device 240 may include a CPU 241, a memory 242, a storage device 243, and a network interface 244. In an embodiment, the OLAP device 220 may be implemented using a computer system such as the computer system 100 shown in FIG. 1.

[0040] It should be noted that the function of the data transfer management device 240 will be described in detail later, so the description thereof is omitted here.

[0041] According to the data transfer management system 200 configured as shown in FIG. 2, it is possible to flexibly adjust the periodic transfer timing at which data is transferred from the OLTP database to the OLAP database.

[0042] FIG. 2 shows a configuration example in which the OLTP device 210 for storing the OLTP database 214, the OLAP device 220 for storing the OLAP database 224, and the data transfer management device 240 are implemented as separate hardware devices connected via a communication network 230. It should be noted that the present disclosure is not limited to this. For example, a configuration in which the OLTP database 214 and the OLAP database 224 are stored in the same hardware device and managed by the data transfer management device 240 via the communication network 230, or a configuration in which the functionality of the OLTP database 214, the OLAP database 224, and the data transfer management device 240 are all implemented in the same hardware device is also possible.

[0043] Next, the functional configuration of the data transfer management system 200 will be described with reference to FIG. 3.

[0044] FIG. 3 is a block diagram showing the functional configuration of the data movement management system 200. As described in this specification, the data movement management system 200 includes an OLTP device 210, an OLAP device 220, and a data movement management device 240 that are communicatively connected by a communication network 230.

[0045] The OLTP device 210 includes an OLTP database load data 312, an OLTP query log file 314, and an OLTP database 214. The OLTP database load data 312 is a collection of data representing CPU resource availability, memory resource availability, average number of data reads per second, average number of data writes per second, and other usage information about the OLTP database 214. The OLTP query log file 314 represents read and write queries of the OLTP database 214.

[0046] The OLAP device 220 includes an OLAP database load data 322, an OLAP query log file 324, and an OLAP database 224. The OLAP database load data 322 is a collection of data representing CPU resource availability, memory resource availability, average number of data reads per second, average number of data writes per second, and other usage information about the OLAP database 224. The OLAP query log file 324 represents read and write queries of the OLAP database 224.

[0047] The data movement management device 240 includes a table status database 341, a table status management unit 342, a query log management unit 343, a resource management unit 344, a data movement management unit 345, and a data movement timing management unit 346.

[0048] The table status database 341 is a database for information representing the data stored in the OLTP database 214 and the OLAP database 224. In an embodiment, the table status database 341 may include status information representing the data freshness requirements of the OLTP database 214 and the OLAP database 224. Generally, data freshness refers to how up-to-date (i.e., current) a particular data set is. Various databases may have various requirements regarding the data freshness of the data they store. These requirements that define the limits of data freshness are referred to herein as data freshness requirements. In an embodiment, these data freshness requirements may be defined by an application that utilizes the data stored in a particular database. As an example, a banking application may have a data freshness requirement that the data be current, whereas a statistical analysis application may have a data freshness requirement that the data be up-to-date for the past six months.

[0049] The table status management unit 342 is a functional unit for managing and updating the table status database 341. When new information is added to the OTLP database 214 or the OLAP database 224, or when the data freshness requirements change, the table status management unit 342 may update the table status database 341. The table status management unit 342 may determine what data is stored in the OLTP database 214 and what data is stored in the OLAP database 224.

[0050] The query log management unit 343 is a functional unit for monitoring and analyzing the query logs of the OLTP database 214 and the OLAP database 224 (i.e., the OLTP query log file 314 and the OLAP query log file 324). In an embodiment, the query log management unit may determine the data freshness requirements of a specific database or a specific data request by analyzing the query log of this database. As an example, the query log management unit 343 may detect an association query from the OLAP database 224 that requests a first data set from the online transaction processing database 214 from the OLAP query log file 324, and based on the detected association query, identify the first data freshness requirement of the OLAP database. As described herein, this first data freshness requirement may be used when determining the appropriate time for implementing the periodic movement timing of data from the OLTP database 214 to the OLAP database 224.

[0051] The resource management unit 344 is a functional unit for monitoring and analyzing the load and availability resources of the OLTP database 214 and the OLAP database 224. In an embodiment, the resource management unit 344 may access and analyze the OLTP database load data 312 and the OLAP database load data 322 to determine CPU resource availability, memory resource availability, average number of data reads per second, average number of data writes per second, and other usage information for each database. Further, the resource management unit may perform a diagnosis on the communication network 230 to obtain information such as download speed, upload speed, and availability of the communication network 230 for data transfer.

[0052] The data transfer management unit 345 is a functional unit for determining and implementing data management operations to facilitate the transfer of data from the OLTP database 214 to the OLAP database 224. Here, the data management operations may include a copy operation in which data from the OLTP database 214 is copied to the OLAP database 224 while remaining held in the OLTP database 214, or a copy-delete operation in which data from the OLTP database 214 is copied to the OLAP database 224 and deleted from the OLTP database 214. In an embodiment, the determination of the data management operation may be based at least on the data freshness requirements of the OLTP database 214 and / or the OLAP database 224.

[0053] The data transfer timing management unit 346 is a functional unit for determining an appropriate periodic transfer timing for the batch transfer of data from the OLTP database 214 to the OLAP database 224. In an embodiment, the data transfer timing management unit 346 may be determined based on the frequency of join queries from the OLAP database 224. As an example, the data transfer timing management unit 346 determines to decrease the interval during periodic data transfer when the frequency of join queries from the OLAP database 224 is high in order to provide the OLAP database 224 with the latest data more frequently, thereby reducing the number of join queries and reducing the processing load on the OLTP database 214. Alternatively, the data transfer timing management unit 346 determines to increase the interval during periodic transfer when the data freshness requirement of the OLAP database 224 is relaxed and the latest data is not required, thereby reducing the processing load on the OLTP database 214.

[0054] According to the data transfer management system 200 configured as shown in FIG. 3, by flexibly adjusting the periodic transfer timing at which data is transferred from the OLTP database to the OLAP database, it becomes possible to accelerate the query processing time and improve the search performance.

[0055] Next, an example of the operation flow in the HTAP system will be described with reference to FIG. 4.

[0056] FIG. 4 is a diagram showing an example of the operation flow in the HTAP system.

[0057] First, the OLTP database 214 receives data in the form of an insertion query 410 from sensor devices such as sensors 401 and 402 via the gateway 405. The OLTP database 214 processes the insertion query 410 and inserts and stores the received data into the OLTP database 214. When sensors 401 and 402 collect measurement values as part of a monitoring application, the insertion query 410 can be continuously streamed to the gateway 405. Upon receiving a read query 420, the OLTP database 214 retrieves the requested data and provides this to the transaction application 430 for display at the interface. Since the OLTP database 214 is designed to support fast access to short-term data, as an example, the transaction application 430 can display the time-series data collected by sensors 401 and 402 in the last 30 minutes in a graph or chart, show the average measurement values at 5-minute intervals, show the measurement values of a specific sensor at a specific time, and so on.

[0058] Data is moved from the OLTP database 214 to the OLAP database 224 at a predetermined data transfer timing (e.g., once every 24 hours). However, as described herein, in some cases, the OLAP database 224 may receive an analysis query 450 from an analysis application 440 that specifies data that meets the data freshness requirement that requires data newer (e.g., more up-to-date) than the data stored in the OLAP database 224 after being transferred. That is, the analysis query 450 requires the use of data that has not yet been transferred from the OLTP database 214 to the OLAP database 224. In such a case, the OLAP database 224 may generate a union query, which is a query when the OLAP database 224 requests the transfer of a specified data set from the OLTP database 214 to perform the analysis query 450. However, such a union query is computationally expensive, increases the load on the OLTP database 214, and potentially leads to a slowdown in query processing time and a decrease in search performance.

[0059] Accordingly, aspects of the present disclosure relate to providing a data transfer management technique for flexibly adjusting the periodic transfer timing of data moved from an OLTP database to an OLAP database based on data freshness requirements, network and computing resource availability, and database status in order to promote the search performance of analysis and transaction queries in an HTAP system. Thus, the OLAP database 224 can be maintained in an up-to-date state while reducing the frequency of union queries and the processing load on the OLTP database 214.

[0060] Next, an example of the flow of a data copy operation in a data transfer management system according to an embodiment of the present disclosure is described with reference to FIG. 5.

[0061] As described herein, the data movement management device 240 is configured to determine and perform data management operations to facilitate the movement of data from the OLTP database 214 to the OLAP database 224. In an embodiment, the data management operations may include a data copy operation that copies a specific data set from the OLTP database 214 to the OLAP database 224. Additionally, in some embodiments, the data management operations may include a data copy and delete operation that copies a specific data set from the OLTP database 214 to the OLAP database 224 and subsequently deletes the copied data from the OLTP database 214. FIG. 5 is a diagram showing an example of the flow of a data copy operation in the data movement management system 200 according to an embodiment of the present disclosure.

[0062] As shown in FIG. 5, the data movement management device 240 may obtain information including a first data freshness requirement of the OLAP database 224, OLAP database load data, network status information of the communication network 230 (not shown in FIG. 5), OLTP database load data, and a second data freshness requirement of the OLTP database 214.

[0063] The first data freshness requirement includes information indicating how recent a specific data set must be for use in response to queries regarding the OLAP database 224. In an embodiment, the first data freshness requirement may be expressed as a date or date range that a specific data set must meet. As an example, the first data freshness requirement may include a date range between "[January 7, 2019 09:30:00]" and "[March 31, 2021 11:00:00]". If the OLAP database 224 does not include dates up to "[March 31, 2021 11:00:00]", it may be necessary for the OLAP database 224 to request data corresponding to a query including this first data freshness requirement from the OLTP database 214. In an embodiment, the data movement management device may determine the first data freshness requirement by extracting the first data freshness requirement from a query (e.g., a union query) in the OLAP query log file 324.

[0064] The OLAP database load data 322 is a collection of data representing CPU resource availability, memory resource availability, average number of data reads per second, average number of data writes per second, and other usage information about the OLAP database 224. Similarly, the OLTP database load data 312 is a collection of data representing CPU resource availability, memory resource availability, average number of data reads per second, average number of data writes per second, and other usage information about the OLTP database 214.

[0065] The network status information 506 is a collection of information including download speed, upload speed, and availability of the communication network 230 for transferring data. In an embodiment, the data movement management device 240 may determine the network status information 506 by performing a series of diagnoses on the communication network 230.

[0066] The second data freshness requirement includes information indicating how current a particular data set must be for use in response to queries regarding the OLTP database 214. In an embodiment, the second data freshness requirement may be expressed as a date or date range that a particular data set must meet. In an embodiment, the data movement management device 240 may determine the second data freshness requirement by extracting the second data freshness requirement from the queries in the OLTP query log file 314.

[0067] Upon receiving the acquisition of the first data freshness requirement of the OLAP database 224, the OLAP database load data, the network status information of the communication network 230 (not shown in FIG. 5), the OLTP database load data, and the second data freshness requirement of the OLTP database 214, the data movement management device can determine, in a timely manner for the data copy operation, a data set to be copied from the OLTP database 214 to the OLAP database as part of the data copy operation. The timely timing for the data copy operation can be determined based on the first data freshness requirement of the OLAP database 224, the OLAP database load data, the network status information of the communication network 230, the OLTP database load data, and the second data freshness requirement of the OLTP database 214. Since a detailed description of the determination of the timing of the data management operation will be given later, the explanation is omitted here.

[0068] After determining, in a timely manner for the data copy operation, a data set to be copied from the OLTP database 214 to the OLAP database as part of the data copy operation, the data movement management device 240 reads a table 510 that stores the requested data in the OLTP database 214, generates a CSV file for storing the data, performs a batch import of the CSV file into a specified table 520 of the OLAP database 224, commits the data, and can update the table status database 341 to reflect the newly added data.

[0069] Next, an example of the flow of the data deletion operation in the data movement management system according to the embodiment of the present disclosure will be described with reference to FIG. 6.

[0070] FIG. 6 is a diagram showing an example of the flow of the data deletion operation in the data movement management system according to the embodiment of the present disclosure. In the embodiment, the data deletion operation shown in FIG. 6 can be shown following the data copy operation shown in FIG. 5 as part of the data copy and deletion operation.

[0071] The data movement management device 240 can first determine, in a timely manner for the data deletion operation, a data set to be deleted from the OLTP database 214. In an embodiment, the data movement management device 240 can determine to delete a data set that was previously copied from the OLTP database 214 to the OLAP database 224. The timeliness for the data deletion operation can be determined based on the first data freshness requirement of the OLAP database 224, the OLAP database load data, the network status information of the communication network 230, the OLTP database load data, and the second data freshness requirement of the OLTP database 214. Since a detailed description of the determination of the timing of the data management operation will be given later, its explanation is omitted here.

[0072] Subsequently, the data movement management device 240 can delete the determined data from the table 610 of the OLTP database 214 in which it is stored, and update the table status database 341 to reflect the deleted data.

[0073] According to the data movement management operations described above with reference to FIGS. 5 and 6, the data movement management device 240 can keep the OLAP database 224 up-to-date while reducing the frequency of the union query and the processing load of the OLTP database 214.

[0074] Next, an example of a data movement management method according to an embodiment of the present disclosure will be described with reference to FIG. 7.

[0075] FIG. 7 is a flowchart showing the flow of a data movement management method 700 according to an embodiment of the present disclosure. The data movement management method 700 is a method for determining and implementing data movement management operations based on the data freshness requirements of the OLTP database and the OLAP database, and for determining the updated periodic data movement timing to promote query processing performance.

[0076] First, in step S705, the query log management unit 343 of the data movement management device 240 monitors the OLTP query log file (OLTP query log file 314) and the OLAP query log file (OLAP query log file 324) for the issuance of federated queries. The query log management unit 343 can detect the issuance of a federated query when the query of the OLAP query log file relates to (e.g., requires) data that is not available in the OLAP database.

[0077] Next, in step S710, the query log management unit 343 can determine whether a federated query from the OLAP database has been detected. If a federated query from the OLAP database has been detected, the process proceeds to step S715. Conversely, if no federated query from the OLAP database has been detected, the process returns to step S705 to continue monitoring the query log file.

[0078] Next, if a federated query has been detected, in step S715, the query log management unit 343 can identify the first data freshness requirement of the OLAP database based on the federated query. As described herein, the first data freshness requirement includes information indicating how recent a particular data set must be for use in response to a query of the OLAP database 224. In an embodiment, the first data freshness requirement can be expressed as a date or date range that a particular data set must meet. As an example, the first data freshness requirement can include a date range between "[January 7, 2019 09:30:00]" and "[March 31, 2021 11:00:00]". The query log management unit 343 can identify the first data freshness requirement by extracting it from the detected federated query.

[0079] Next, in step S720, the data transfer management unit 345 of the data transfer management device 240 determines whether the amount of data requested by the union query satisfies the data amount threshold. The amount of data requested by the union query can be determined based on the first data freshness requirement identified in step S715. For example, the data transfer management unit 345 can use the identified first data freshness requirement to refer to the OLTP database (and / or the table status database), and confirm the amount of data that needs to be transferred from the OLTP database to the OLAP database in response to the union query.

[0080] Here, the data amount threshold is a predetermined value that specifies a specific amount of data. The data amount threshold can be expressed as a specific data size (5 megabytes, 100 megabytes, 1 gigabyte), or as the number of data rows or columns (50 rows, 100 rows, 50 columns). When the amount of data requested by the union query is less than the data amount threshold, the union query is considered to satisfy the data amount threshold. In this case, since the amount of data requested by the union query is substantially small, the union query can be executed without significantly increasing the load on the OLTP database. Conversely, when the amount of data requested by the union query is large (for example, the union query does not satisfy the data amount threshold), it is desirable to perform data management operations to provide the requested data to the OLAP database and correct the periodic data transfer timing to reduce the frequency of future union queries.

[0081] Therefore, when the union query satisfies the data amount threshold, the process proceeds to step S725. Conversely, when the union query does not satisfy the data amount threshold, the process proceeds to step S730.

[0082] In step S725, the data transfer management unit 345 executes a union query to obtain from the OLTP database the data necessary to process the union query in the OLAP database.

[0083] In step S730, the data movement management unit 345 can identify the second data freshness requirement of the OLTP database based on the OLTP query log file (for example, the OLTP query log file 314). The second data freshness requirement includes information indicating how current a specific data set must be for use in response to queries in the OLTP database. In an embodiment, the second data freshness requirement can be expressed as a date or date range that a specific data set must meet. In an embodiment, the data movement management unit 345 can determine the second data freshness requirement by analyzing the queries recorded in the OLTP query log file that represent the most recently accessed data in the OLTP database (for example, the data recently accessed in the OLTP database is also the data required by the OLTP database to process queries).

[0084] Next, in step S735, the data movement management unit 345 can compare the first data freshness requirement identified in step S715 with the second data freshness requirement identified in step S730 to determine whether the first data freshness requirement corresponds to the second data freshness requirement. Here, the data movement management unit 345 can determine that the first data freshness requirement corresponds to the second data freshness requirement if the date range specified by the first data freshness requirement overlaps at least a portion of the date range specified by the second data freshness requirement. Conversely, the data movement management unit 345 can determine that the first data freshness requirement does not correspond to the second data freshness requirement if the date range specified by the first data freshness requirement does not overlap at least a portion of the date range specified by the second data freshness requirement.

[0085] If it is determined that the first and second data freshness requirements correspond, the process proceeds to step S745. Conversely, if the first and second data freshness requirements do not correspond, the process proceeds to step S740.

[0086] In step S740, the data movement management unit 345 may determine to perform a copy-delete operation of copying the first data set requested by the union query from the OLTP database to the OLAP database and deleting the first data set from the OLTP database as a data management operation. That is, since the data freshness requirements of the OLTP database and the OLAP database do not overlap, the first data set requested by the union query is no longer in a state of being used by the OLTP database, and thus can be deleted from the OLTP database. Since the data copy-delete operation will be described later, its detailed description is omitted here.

[0087] In step S745, the data movement management unit 345 may determine to perform a copy operation of copying the first data set requested by the union query from the OLTP database to the OLAP database as a data management operation. That is, since the data freshness requirements of the OLTP database and the OLAP database overlap, the first data set requested by the union query is still in a state of being used by the OLTP database, and thus should be retained in the OLTP database. Since the data copy operation will be described later, its detailed description is omitted here.

[0088] In step S750, the data movement management unit 345 determines whether the frequency of the union query has reached a frequency threshold. Here, the frequency threshold defines a predetermined issuance rate of the union query and can be expressed as the number of union queries issued in the OLAP database per given time (for example, 1 union query per hour, 10 union queries per day). If the frequency of the union query calculated by the data movement management unit 345 for the OLAP database at a specific time is greater than the frequency threshold, the frequency of the union query is considered to have reached the frequency threshold. If the frequency of the union query has reached the frequency threshold, the process proceeds to step S755. Conversely, if the frequency of the union query has not reached the frequency threshold, the process returns to step S705 to continue monitoring the query log files of the OLTP database and the OLAP database.

[0089] In step S755, the data transfer timing management unit 346 determines the updated periodic transfer timing for transferring batch data from the OLTP database to the OLAP database. That is, when the frequency of the union query reaches the frequency threshold, it is desirable to correct the periodic transfer timing to provide the latest data to the OLAP database and reduce the number of union queries issued in the OLAP database. Since the process for determining the updated periodic data transfer timing will be described later, its detailed description is omitted here.

[0090] According to the data transfer management method 700 illustrated in FIG. 7, in order to promote the search performance of analysis and transaction queries in the HTAP system, based on the data freshness requirement, network and computing resource availability, and database status, it is possible to flexibly adjust the periodic transfer timing of data from the OLTP database to the OLAP database. In this way, while reducing the frequency of union queries and the processing load of the OLTP database 214, the OLAP database 224 can be maintained in the latest state.

[0091] Next, a specific example of determining the data transfer management operation according to the embodiment of the present disclosure will be described with reference to FIG. 8.

[0092] As described in this specification, the data transfer management device 240 according to the embodiment of the present disclosure is configured to determine a data transfer management operation based on the data freshness requirements of the OLAP database and the OLTP database. FIG. 8 is a diagram showing a specific example of determining the data transfer management operation according to the embodiment of the present disclosure.

[0093] The query log management unit 343 of the data movement management device 240 can analyze the query logs stored in the OLAP query log file 324 of the OLAP database to identify the first data freshness requirement 810 of the OLAP database. For example, the first data freshness requirement 810 can specify a date range of "January 7, 2019 09:30 to March 31, 2021 11:00". In an embodiment, the query log management unit 343 can analyze the OLAP query log file 324 upon receiving detection of a union query from the OLAP database.

[0094] Next, the query log management unit 343 can analyze the query logs stored in the OLTP query log file 314 of the OLTP database to identify the second data freshness requirement 820 of the OLTP database. In an embodiment, the query log management unit 343 can identify the second data freshness requirement 820 based on a query representing the most recently accessed data in the OLTP database. As an example, the second data freshness requirement 820 can specify a date range of "January 7, 2021 09:30 to January 7, 2021 11:00".

[0095] Subsequently, the data movement management unit 345 of the data movement management device 240 can compare the first data freshness requirement 810 with the second data freshness requirement 820 to determine whether the first data freshness requirement 810 corresponds to the second data freshness requirement 820. Here, the data movement management unit 345 can determine that the first data freshness requirement 810 corresponds to the second data freshness requirement 820 if the date range specified by the first data freshness requirement 810 overlaps at least a part of the date range specified by the second data freshness requirement 820. Conversely, the data movement management unit 345 can determine that the first data freshness requirement 810 does not correspond to the second data freshness requirement 820 if the date range specified by the first data freshness requirement 810 does not overlap at least a part of the date range specified by the second data freshness requirement 820.

[0096] When the first data freshness requirement 810 corresponds to the second data freshness requirement 820, the data movement management unit 345 may determine to perform, as a data management operation, a copy operation of copying the first data set from the OLTP database to the OLAP database. Conversely, when the first data freshness requirement 810 does not correspond to the second data freshness requirement 820, the data movement management unit 345 may determine to perform, as a data management operation, a copy and delete operation of copying the first data set from the OLTP database to the OLAP database and deleting the first data set from the OLTP database.

[0097] In this example, since the first data freshness requirement 810 of "January 7, 2019 09:30~March 31, 2021 11:00" overlaps with the second data freshness requirement 820 of "January 7, 2021 09:30~January 7, 2021 11:00" (for example, both data freshness requirements require data from January 7, 2021 09:30~January 7, 2021 11:00), the data movement management unit 345 may determine to perform, as a data management operation, a copy operation of copying the first data set from the OLTP database to the OLAP database.

[0098] Next, an example of the data copy operation according to the embodiment of the present disclosure will be described with reference to FIG. 9.

[0099] FIG. 9 is a flowchart showing the flow of the data copy operation 900 according to the embodiment of the present disclosure. The data copy operation 900 is a method for copying a data set from the OLTP database to the OLAP database. Performing the data copy operation 900 to provide the requested data to the OLAP database can avoid the need for union query execution that may negatively affect the resource usage of the OLTP database. The data copy operation 900 shown in FIG. 9 substantially corresponds to step S745 in FIG. 7.

[0100] First, in step S905, the resource management unit 344 analyzes the OLTP database load data (e.g., OLTP database load data 312) and the OLAP database load data (e.g., OLAP database load data 322) to determine the availability of computing resources in both the OLTP database and the OLAP database. For example, the resource management unit 344 may determine the availability of CPU resources and memory resources in the OLTP database and the OLAP database based on the OLTP database load data and the OLAP database load data.

[0101] Next, in step S910, the resource management unit 344 determines whether sufficient computing resources are available in both the OLTP database and the OLAP database to perform the data copy operation. In an embodiment, the data movement management unit 345 may determine whether sufficient computing resources are available by comparing the availability computing resources of the OLTP and OLAP databases determined in step S905 with predetermined detailed information specifying the computing resources required for the data copy operation. If sufficient computing resources are available, the process proceeds to step S915. If sufficient resources are not available, the data movement management unit 345 waits for a predetermined time and returns to step S905.

[0102] Next, in step S915, the data movement management unit 345 may generate a CSV file including the first data set requested by the association query detected in step S710 of the data movement management method 700.

[0103] Next, in step S920, the resource management unit 344 analyzes the communication network (e.g., the communication network 230 shown in FIGS. 2 and 3) to determine the availability of network resources and may perform a data copy operation. In an embodiment, the resource management unit 344 may determine the availability of network resources by performing one or more diagnostics on the communication network 230 to check download speed, upload speed, availability, and the like.

[0104] Next, in step S925, the resource management unit 344 may determine whether sufficient network resources are available to perform the data copy operation. In an embodiment, the resource management unit 344 may compare the network resources determined in step S920 with predetermined detailed information specifying the network resources required to perform the data copy operation for a given data size. If sufficient network resources are available, the process proceeds to step S930. If sufficient resources are not available, the data transfer management unit 345 waits for a predetermined time and returns to step S920.

[0105] In step S930, the data transfer management unit 345 may transfer the CSV file generated in step S915 from the OLTP database (e.g., from the OLTP device 210) to the OLAP device (e.g., to the OLAP device 220) via the communication network 230. In an embodiment, the transfer of the CSV file may be performed via a secure or encrypted network channel.

[0106] In step S935, the data transfer management unit 345 may update the table status database 341 to reflect the data of each of the OLTP and OLAP databases after the transfer of the CSV file (e.g., update the status of the OLAP database to include the transferred data).

[0107] According to the data copy operation 900, it is possible to transfer the necessary data from the OLTP database to the OLAP database, and maintain the OLAP database up-to-date while reducing the frequency of join queries and the processing load on the OLTP database.

[0108] Next, an example of the data copy and deletion operation according to the embodiment of the present disclosure will be described with reference to FIG. 10.

[0109] FIG. 10 is a flowchart showing the flow of the data copy and deletion operation 1000 according to the embodiment of the present disclosure. The data copy and deletion operation 1000 is a method for copying a data set from the OLTP database to the OLAP database and then deleting the copied data from the OLTP database. By performing the data copy and deletion operation 1000 to provide the requested data to the OLAP database and then deleting the copied data from the OLTP database, it is possible to avoid the need to execute join queries that may have a negative impact on the resource usage of the OLTP database, and also promote the high-speed query processing of the OLTP database. The data copy and deletion operation 1000 illustrated in FIG. 10 substantially corresponds to step S740 in FIG. 7.

[0110] Since steps S1005, S1010, S1015, S1020, S1025, S1030, and S1035 shown in FIG. 10 substantially correspond to the steps shown in FIG. 9, for the sake of simplicity of explanation, the redundant description of these steps is omitted here.

[0111] In step S1040, the data movement management unit 345 can determine the data to be deleted from the OLTP database. In an embodiment, the data movement management unit 345 can determine to delete the first data set that was requested by a join query of the OLAP database and transferred from the OLTP database to the OLAP database in step S1030.

[0112] In step S1045, the data transfer management unit 345 may perform a data deletion operation to delete the data determined in step S1040 from the OLTP database. Since the data deletion operation will be described later, a detailed description thereof is omitted here.

[0113] Next, an example of the data deletion operation according to the embodiment of the present disclosure will be described with reference to FIG. 11.

[0114] FIG. 11 is a flowchart showing the flow of a data deletion operation 1100 according to an embodiment of the present disclosure. The data deletion operation is a method for deleting a data set copied from an OLTP database to an OLAP database from the OLTP database. The data deletion operation 1100 shown in FIG. 11 substantially corresponds to step S1045 in FIG. 10.

[0115] First, in step S1105, the resource management unit 344 analyzes the OLTP database load data (for example, the OLTP database load data 312) to determine the availability of computing resources in the OLTP database. For example, the resource management unit 344 may determine the availability of CPU resources and memory resources in the OLTP database based on the OLTP database load data.

[0116] Next, in step S1110, the resource management unit 344 determines whether sufficient computing resources for performing the data deletion operation are available in the OLTP database. In an embodiment, the resource management unit 344 compares the available computing resources of the OLTP database determined in step S1105 with predetermined detailed information specifying the computing resources required for the data deletion operation, thereby determining whether sufficient computing resources are available. If sufficient computing resources are available, the process proceeds to step S1115. If sufficient resources are not available, the resource management unit 344 waits for a predetermined time and returns to step S1105.

[0117] Next, in step S1115, the data transfer management unit 345 identifies the data to be deleted. In an embodiment, the data transfer management unit 345 can identify the data determined in step S1040 of the data copy / deletion operation 1000 in the OLTP database (for example, the first data set requested by the federated query of the OLAP database and transferred from the OLTP database to the OLAP database).

[0118] Next, in step S1120, the data transfer management unit 345 deletes the data identified in step S1115 from the OLTP database.

[0119] Next, in step S1125, the data transfer management unit 345 may update the table status database 341 so as to reflect the data in the OLTP database after the data deletion.

[0120] According to the data copy / deletion operation 1000 and the data deletion operation 1100, it is possible to transfer the necessary data from the OLTP database to the OLAP database to keep the OLAP database up-to-date, and reduce the frequency of federated queries while removing unnecessary data from the OLTP database.

[0121] Next, the periodic data transfer timing update process 1200 will be described with reference to FIG. 12.

[0122] FIG. 12 is a flowchart showing the flow of the periodic data transfer timing update process 1200 according to an embodiment of the present disclosure. The periodic data transfer timing update process 1200 is a process for updating the periodic data transfer timing in which data is periodically transferred from the OLTP database to the OLAP database in order to timely provide the latest data to the OLAP database and reduce the frequency of federated queries. The periodic data transfer timing update process 1200 illustrated in FIG. 12 substantially corresponds to step S755 in FIG. 7.

[0123] First, in step S1205, the query log management unit 343 of the data movement management device 240 monitors the OLTP query log file (OLTP query log file 314) and the OLAP query log file (OLAP query log file 324).

[0124] Next, in step S1210, the data movement timing management unit 346 determines the data insertion rate of the OLTP database and the first data freshness requirement of the OLAP database. Here, the data insertion rate of the OLTP database refers to the rate at which new data is inserted into the OLTP database, and can be expressed as the data size per unit time (e.g., 5 mb / second) or the number of rows / columns per unit time (e.g., 50 rows / minute). In an embodiment, the data movement timing management unit 346 can calculate the data insertion rate of the OLTP database based on the data insertion queries included in the OLTP query log file. As described herein, the first data freshness requirement can be determined based on the OLAP query log file 324 (e.g., from the association queries detected in step S710).

[0125] Next, in step S1215, the data movement timing management unit 346 determines whether the current periodic movement is effective in maintaining the OLAP database in the latest state based on the data insertion rate of the OLTP database determined in step S1205 and the first data freshness requirement of the OLAP database.

[0126] More specifically, the data transfer timing management unit 346 can compare the data insertion rate with a predetermined first insertion rate threshold and the first data freshness requirement with a first recency threshold. The first insertion rate threshold can define a specific threshold rate at which data is inserted into the OLTP database. When the data insertion rate calculated in step S1210 is greater than or equal to the first insertion rate threshold, it is considered that the data insertion rate of the OLTP database has reached the first insertion rate threshold. The first recency threshold specifies a date or date range that defines the limit of the first data freshness requirement. When requesting data whose first data freshness requirement is more recent than the first recency threshold, it is considered that the first data freshness requirement has reached the first recency threshold.

[0127] If the data insertion rate of the OLTP database has not reached the first insertion rate threshold or the first data freshness requirement of the OLAP database has not reached the first recency threshold, it is determined that the current periodic transfer time is not effective. This is because when the data insertion rate of the OLTP database is relatively low or the first data freshness requirement of the OLAP database is such that the most recent data is not required, it is not necessary to perform frequent periodic data transfer from the OLTP database to the OLAP database.

[0128] Conversely, if the data insertion rate of the OLTP database has reached the first insertion rate threshold or the first data freshness requirement of the OLAP has reached the first recency threshold, it is determined that the current periodic transfer time is not effective. This is because when the data insertion rate of the OLTP database is relatively high or the first data freshness requirement of the OLAP database is such that the most recent data is required, it is necessary to perform more frequent periodic data transfer from the OLTP database to the OLAP database to provide the latest data.

[0129] When the data insertion rate of the OLTP database has reached the first insertion rate threshold but the first data freshness requirement of the OLAP database has not reached the first timeliness threshold, or when the data insertion rate of the OLTP database has not reached the first insertion rate threshold but the first data freshness requirement of the OLAP database has reached the first timeliness threshold, the periodic data movement timing may be considered effective.

[0130] If it is determined that the current periodic movement is effective, the process may return to step S1205. If it is determined that the current periodic movement is not effective, the process may proceed to step S1220.

[0131] Next, in step S1220, when it is determined that the current periodic data movement is not effective, the data movement timing management unit 346 may update the periodic data movement timing at which bulk data transfer from the OLTP database to the OLAP database is performed. Generally, it should be noted that the periodic data movement timing is set to 24 hours, that is, bulk data transfer from the OLTP database to the OLAP database is performed once a day. However, as described in this specification, according to the embodiments of the present disclosure, it is possible to correct the periodic data movement timing (that is, increase or decrease the length of the interval therebetween) based on the data insertion rate of the OLTP database and the first data freshness requirement of the OLAP database.

[0132] When the data insertion rate of the OLTP database has not reached the first insertion rate threshold or the first data freshness requirement of the OLAP database has not reached the first timeliness threshold, the data movement timing management unit 346 may increase the interval of the periodic data movement timing. As a result, the periodic data movement is not performed as frequently, but this is acceptable because the OLAP database does not require the most timely data (for example, due to the low insertion rate of the OLTP database or the loose data freshness requirement).

[0133] Conversely, when the data insertion rate of the OLTP database reaches the first insertion rate threshold or the first data freshness requirement of the OLAP database reaches the first recency threshold, the data movement timing management unit 346 can reduce the interval of the periodic data movement timing. As a result, since the OLAP database requires more recent data (e.g., due to the high insertion speed or strict data freshness requirements of the OLTP database), the periodic data movement is performed more frequently.

[0134] Here, the degree to which the periodic data movement timing is increased or decreased can be determined by several methods. In an embodiment, the data movement timing management unit 346 can increase or decrease the periodic data movement timing by a predetermined amount (e.g., increase or decrease by 1 hour, 2 hours, 6 hours, 12 hours). In some embodiments, the data movement timing management unit 346 can utilize a statistical analysis or machine learning approach to identify an appropriate periodic data movement timing based on the history of the periodic data movement timing and the resource load of the OLTP and OLAP databases. In an embodiment, based on a prediction of whether the data insertion rate or the first data freshness requirement of the OLTP database has reached or not reached the first insertion rate threshold (e.g., the period when a system configuration change greatly increases or decreases data insertion) or the first recency threshold (e.g., when an application with well-known data freshness requirements is scheduled to be executed), the data movement timing management unit 346 can increase or decrease the periodic data movement timing. Other methods of determining the periodic data movement timing are also possible.

[0135] According to the periodic data movement timing update process 1200 described with reference to FIG. 12, it is possible to update the periodic data movement timing to an appropriate timing to promote resource efficiency in the OLTP database and the OLAP database. For example, when the OLAP database does not require up-to-date data, by increasing the periodic data movement timing (i.e., decreasing the frequency), it is possible to reduce the computing resources and processing load required in the OLTP database. Conversely, when the OLAP database requires up-to-date data, by decreasing the periodic data movement timing (i.e., increasing the frequency), it is possible to reduce the frequency of union queries from the OLAP database to the OLTP database and similarly reduce the computing resources and processing load required in the OLTP database.

[0136] Next, a periodic data movement delay process is described with reference to FIG. 13. The disclosed aspect relates to the recognition that when the data in the OLTP database is frequently updated, it is desirable to delay data management operations until the frequency update decreases so that the latest data can be efficiently transferred to the OLAP database.

[0137] FIG. 13 is a flowchart showing the flow of a data management operation delay process 1300 according to an embodiment of the present disclosure. The data management operation delay process 1300 is a process for delaying data management operations when the data in the OLTP database is frequently updated. The data management operation delay process 1300 shown in FIG. 13 can be implemented following step S735 of FIG. 7 (e.g., prior to the execution of data management operations).

[0138] First, in step S1305, the query log management unit 343 of the data movement management device 240 monitors the OLTP query log file (OLTP query log file 314) and the OLAP query log file (OLAP query log file 324).

[0139] Next, in step S1310, the data movement management unit 345 determines the update frequency of the first data set in the OLTP database and the first data freshness requirement of the OLAP database. Here, the update frequency of the first data set refers to the rate at which the first data set in the OLTP database is updated, and can be expressed as the number of updates per unit time (for example, 10 updates per minute). In an embodiment, the data movement management unit 345 may calculate the update frequency of the first data set in the OLTP database based on the data insertion queries included in the OLTP query log file. As described herein, the first data freshness requirement may be determined based on the OLAP query log file 324.

[0140] Next, in step S1315, the data movement management unit 345 determines whether the execution time of the currently scheduled data management operation is efficient.

[0141] More specifically, when the update frequency of the first data set in the OLTP database reaches the first update frequency threshold and the first data freshness requirement does not reach the first timeliness threshold, the data movement management unit 345 determines that the execution of the current data management operation is not efficient (for example, since the data in the OLTP database is frequently updated and the OLAP database does not require up-to-date data, it is more efficient to perform the data management operation later). In this case, the process proceeds to step S1325. Here, the first update frequency threshold may define a specific threshold rate at which the first data set is updated in the OLTP database. When the update frequency calculated in step S1310 is greater than or equal to the first update frequency, the update frequency of the OLTP database is considered to have reached the first update frequency threshold.

[0142] Conversely, if the update frequency of the first data set in the OLTP database has not reached the first update frequency threshold or the first data freshness requirement has reached the first recency threshold, the data movement management unit 345 determines that the OLAP database is efficient (for example, since the update frequency of the data in the OLTP database is low or the OLAP database requires the latest data, it is more efficient to perform data management operations at that time to avoid union queries). In this case, the process proceeds to step S1320.

[0143] In step S1320, the data movement management unit 345 executes a data management operation. This is because the first data set is updated frequently, so it is desirable to delay copying the first data set to the OLAP database until the update frequency decreases, but the first data freshness requirement of the OLAP database (for example, to respond to queries) requires the first data set earlier.

[0144] In step S1325, the data movement management unit 345 delays the execution of the data management operation. In an embodiment, the data movement management unit 345 may delay the execution of the data management operation until the update frequency of the first data set falls below the first update frequency threshold. In this way, by delaying the performance of the data management operation until the update frequency of the first data set decreases, it is possible to provide more up-to-date data to the OLAP database in an efficient manner.

[0145] The present invention can be a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0146] A computer-readable storage medium can be a tangible device that retains and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination of the foregoing. A comprehensive list of more specific examples of computer-readable storage media includes the following. Portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures within grooves in which instructions are recorded, and suitable combinations of the foregoing.

[0147] As used herein, a computer-readable storage medium is not to be construed as being itself a transitory signal such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating within a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted within an electrical wire.

[0148] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0149] When computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to thereby configure a machine, the instructions executed via the processor of the computer or other programmable data processing apparatus serve as means for implementing the functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium containing the instructions comprises a manufactured article including instructions for implementing the aspects of the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0150] When the computer-readable program instructions are loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to implement a computer-implemented process, the instructions executed on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0151] Embodiments according to the present disclosure may be provided to an end user through a cloud computing infrastructure. Cloud computing generally refers to providing scalable computing resources as services via a network. More formally stated, cloud computing can be defined as a computing function that abstracts between computing resources and their underlying technical architecture (e.g., servers, storage devices, networks) to enable easy on-demand network access to a shared pool of configurable computing resources that can be quickly provisioned and de-provisioned with minimal administrative effort or service provider interaction. Thus, cloud computing enables a user's access to virtual computing resources in the "cloud" (e.g., storage devices, data, applications, and fully virtualized computing systems) without considering the underlying physical systems (or the location of these systems) used to provide the computing resources.

[0152] The flowcharts and block diagrams of the figures illustrate the architecture, functionality, and operation of possible implementation examples of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowchart or block diagram may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing the specified logical function. In some alternative implementation examples, the functions noted in the blocks may be performed in an order different than that noted in the figures. For example, depending on the required functionality, two blocks shown in succession may actually be executed substantially simultaneously, or sometimes the blocks may be executed in reverse order. It should also be noted that each block of the block diagram and / or flowchart diagram, and combinations of blocks in the block diagram and / or flowchart diagram, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0153] The above is directed to exemplary embodiments, but without departing from the basic scope, other and further embodiments of the present invention have been devised, the scope of which is determined by the claims that follow. The description of the various embodiments of the present disclosure is presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to explain the principles of the embodiments, practical applications, or technological improvements found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0154] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be a limitation of the various embodiments. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. Terms such as "set of", "group of", "bunch of", etc. are intended to include one or more. The terms "includes" and / or "including", when used herein, specify the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In the exemplary embodiments already described in detail for the various embodiments, reference is made to the accompanying drawings (where like numerals represent like elements), which form a part of the embodiments and in which are shown, by way of illustration, specific exemplary embodiments in which the various embodiments may be practiced. These embodiments are described in sufficient detail for those skilled in the art to practice the embodiments, but other embodiments may be used and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. In the previous description, numerous specific details were presented to provide a thorough understanding of the various embodiments. However, the various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the embodiments.

Description of Reference Numerals

[0155] 100 Computer system 102 Processor 104 Memory 106 Memory bus 108 Input / output bus 109 Bus IF 110 Input / output bus IF 112 Terminal interface 113 Storage device interface 114 Input / output device interface 115 Network Interface 116 User Input / Output Device 117 Memory Device 124 Display System 126 Display 130 Network 150 Data Movement Management Application 200 Data Movement Management System 210 OLTP Device 214 OLTP Database 220 OLAP Device 224 OLAP Database 230 Communication Network 240 Data Movement Management Device 312 OLTP Database Load Data 314 OLTP Query Log File 322 OLAP Database Load Data 324 OLAP Query Log File 341 Table Status Database 342 Table Status Management Unit 343 Query Log Management Unit 344 Resource Management Unit 345 Data Movement Management Unit 346 Data Movement Timing Management Unit

Claims

1. A data movement management device for managing data in a hybrid transaction analysis processing architecture, a query log management unit that detects an association query for requesting a first data set from an online transaction processing database from a first query log set of an online analytical processing database, and identifies a first data freshness requirement of the online analytical processing database based on the association query; a data movement management unit for determining a data management operation that includes at least copying the first data set from the online transaction processing database to the online analytical processing database based on at least the first data freshness requirement; a data movement timing management unit for determining an updated periodic movement timing for transferring batch data from the online transaction processing database to the online analytical processing database when the frequency of the association query from the online analytical processing database reaches a first frequency threshold; A data movement management device comprising the above.

2. The query log management unit further, identifies a second data freshness requirement of the online transaction processing database based on a second query log set of the online transaction processing database, configured as such, the data movement management unit further, when the first data freshness requirement of the online analytical processing database corresponds to the second data freshness requirement of the online transaction processing database, determines to perform a copy operation of copying the first data set from the online transaction processing database to the online analytical processing database as the data management operation; when the first data freshness requirement of the online analytical processing database does not correspond to the second data freshness requirement of the online transaction processing database, determines to perform a copy / delete operation of copying the first data set from the online transaction processing database to the online analytical processing database and deleting the first data set from the online transaction processing database as the data management operation, configured as such, The data movement management device according to claim 1.

3. For executing the copy operation, the data transfer management unit generates a transfer file including the first data set from the online transaction processing database when the available CPU and memory resources reach a first resource threshold, transfers the transfer file from the online transaction processing database to the online analysis processing database when the available network resources reach a first network resource threshold, configured as the data transfer management device according to claim 2.

4. For executing the copy / delete operation, the data transfer management unit generates a transfer file including the first data set from the online transaction processing database when the available CPU and memory resources reach a first resource threshold, transfers the transfer file from the online transaction processing database to the online analysis processing database when the available network resources reach a first network resource threshold, deletes the first data set from the online transaction processing database, configured as the data transfer management device according to claim 2.

5. The query log management unit further receives the determination of the data management operation, and determines the update frequency of the first data set in the online transaction processing database based on the second query log set in the online transaction processing database, The data transfer timing management unit further determines to delay the execution of the data management operation for a predetermined time when the update frequency of the first data set in the online transaction processing database reaches a first update frequency threshold and the first data freshness requirement does not reach a first timeliness threshold, determines to start the execution of the data management operation when the update frequency of the first data set in the online transaction processing database does not reach the first update frequency threshold or the first data freshness requirement reaches the first timeliness threshold, configured as the data transfer management device according to claim 2.

6. The query log management unit further Based on the second query log set of the online transaction processing database, determine the data insertion rate at which data records are inserted into the online transaction processing database. configured as The data movement timing management unit further When the data insertion rate has not reached the first insertion rate threshold or the first data freshness requirement has not reached the first recency threshold, increase the interval of the periodic movement timing at which batch data is periodically moved from the online transaction processing database to the online analysis processing database. When the data insertion rate has reached the first insertion rate threshold or the first data freshness requirement has reached the first recency threshold, decrease the interval of the periodic movement timing at which batch data is periodically moved from the online transaction processing database to the online analysis processing database. configured as The data movement management device according to claim 2.

7. A data movement management method for a data movement management device comprising a query management unit, a data movement management unit, and a data movement timing management unit, the data movement management device being for managing data in a hybrid transaction analysis architecture, The query management unit detects an association query that requests a first data set from the online transaction processing database from a first query log set of the online analysis processing database. Based on the association query, identify the first data freshness requirement of the online analysis processing database. The data movement management unit determines a data management operation that includes at least copying the first data set from the online transaction processing database to the online analysis processing database based at least on the first data freshness requirement. The data movement timing management unit determines an updated periodic movement timing for transferring batch data from the online transaction processing database to the online analysis processing database when the frequency of association queries from the online analysis processing database has reached a first frequency threshold. A method including.

8. A data movement management system for managing data in a hybrid transaction analysis architecture, An online transaction processing device having an online transaction processing database, An online analytical processing device having an online analytical processing database, A data transfer management device for managing the transfer of data from the online transaction processing database to the online analytical processing database, A data transfer management system comprising: The data transfer management device includes: A query log management unit that detects an association query for requesting a first data set from the online transaction processing database from a first query log set of the online analytical processing database, and based on the association query, identifies a first data freshness requirement of the online analytical processing database; A data transfer management unit for determining a data management operation including at least copying the first data set from the online transaction processing database to the online analytical processing database based on at least the first data freshness requirement; A data transfer timing management unit for determining an updated periodic transfer timing for transferring batch data from the online transaction processing database to the online analytical processing database when the frequency of the association query from the online analytical processing database reaches a first frequency threshold; Including: A data transfer management system.

Images