Asynchronous persistence of replicated data changes in database accelerators

Abstract

A computer-implemented method for crash recovery for linked databases is provided. The linked database includes a source and an associated target database. Selected queries of the source database are transferred to the target database. The method includes synchronizing selected portions of the source database with tables of an in-memory portion of the target database and asynchronously and persistently storing persistently applied changes to the in-memory target database portion upon a database crash of the target database system. The method further includes restoring the in-memory target database portion with a latest available snapshot and applying changes from a source database recovery log file having a later timestamp than the latest snapshot available in the persistent target database storage to the in-memory target database portion.

Description

[Technical Field]

[0001] The present invention relates generally to crash recovery for databases, and more particularly to a computer-implemented method for crash recovery for linked databases. The present invention further relates to a linked database system having crash recovery for linked databases, and a computer program product. [Background technology]

[0002] Managing large amounts of data continues to be a challenge for enterprise IT (information technology) organizations. This is due to (i) the ever-increasing amount of data and (ii) the variety of data perspectives. In addition to traditional structured data, enterprise IT systems also store large amounts of semi-structured and so-called unstructured data. Furthermore, to meet the increasing industrial requirements for data analysis, specialized database systems optimized for analytical processing have been introduced in addition to traditional transaction-oriented databases. On the other hand, attempts have been made to perform all analytical processing in traditional transaction databases. However, analytical processing has proven to have an excessively strong and unpredictable impact on the performance of online transaction processing.

[0003] As a result, successful combinations of transaction-oriented and analytics-optimized database management systems have been introduced. One example of such a combination product is based on the IBM DB2 Analytics Accelerator (IDAA) architecture.

[0004] On one side, a full-fledged row-based database management system (DBMS) may serve as the application endpoint for data manipulation language (DML) operations and query execution. Using heuristic decision criteria and rules, the query optimizer may transparently decide whether a query should be executed in the source DBMS, particularly for online transaction processing, or offloaded to a target DBMS, particularly for online analytical processing.

[0005] The target DBMS may be a full-fledged column-based database management system that maintains a shadow copy of a selected set of source DBMS tables. The strategy for creating a shadow copy of the target DBMS may involve transferring data stored in one or more tables in the source DBMS to the target DBMS at a given point in time. However, query data in the target DBMS may be out of date if the load occurred some time ago and the corresponding source tables were modified in the meantime. Therefore, the contents of the tables are typically adapted incrementally as changes are recorded in the corresponding source database tables.

[0006] Both database systems implement transaction mechanisms to guarantee the ACID (atomic, consistent, isolation, durable) properties of their respective databases: concurrent modifications are properly isolated by locking techniques, consistency checks may ensure that the database state moves from one consistent state to another, and logging techniques, such as write-ahead logging, may be implemented to guarantee atomicity and durability of transactional changes.

[0007] However, guaranteeing ACID properties can incur significant overhead when the source database is modified by the current transaction. In particular, logging slows down the processing of insert / update / delete (IUD) statements because the change record must be written to persistent storage before the transaction can continue. Compared to the frequently used in-memory processing of changes to the contents of database tables, adding this durability layer using a transaction log is relatively slow. On the other hand, crash recovery is required to restore the latest consistent state of each database in case of a failure.

[0008] In the context of a database accelerator such as IDAA, the target accelerator database simply mirrors a snapshot of the source database. That is, the source database may act as the master of the data, or data manipulation via the IUD may be handled by the source database management system, with changes replicated to the target database via update techniques. The accelerator database may therefore act as a cache for storing partitions of the source database that may require fast analytical operations.

[0009] Several disclosures already exist in this context. The document US Pat. No. 6,499,623 describes a method for replicating database data in a clean shutdown state and generating a read-only copy of the replicated data. A related system may include a tracking module that monitors first transactions from a database application to a source storage device and generates a log entry with at least one marker indicating a known good state of the application. The system further includes a computer coupled to a target storage device containing a database and a log file. The computer processes the transactions based on the log entries to replicate the data to the target storage device and performs a first snapshot to recreate the data stored in the log file in the database.

[0010] Additionally, the document in Patent Document 2 describes a distributed database system that provides fast crash recovery. During recovery from a database head node failure, a connection may be established with one or more storage nodes of a distributed storage system that store data for the database implemented by that database head node. Upon establishing a connection with a storage node, the database may be made available for access, for example, for various access requests.

[0011] However, traditional combined databases that are optimized for transactions on one side and analytical processing on the other can all require the well-known overhead of maintaining a persistent recovery log, which can significantly slow down the combined database's processing, in addition to the suboptimal use of changes from the source database in the target database. Therefore, to achieve higher performance values ​​for combined transactional / analytical database management systems and simultaneously elegantly solve the target database recovery challenge, it may be necessary to reduce the required overhead. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] US Patent Application Publication No. 2015 / 0 205 853(A1) [Patent Document 2] US Patent Application Publication No. 2014 / 0 279 930(A1) Summary of the Invention

[0013] According to one aspect of the present invention, a computer-implemented method for crash recovery for linked databases may be provided. The linked databases may include a source database and an associated target database, and selected queries for a database management system including the source database may be transferred to a database management system including the target database for processing. The method may include synchronizing selected portions of the contents of tables in the source database with respective portions of the contents of tables in the target database; applying changes to the source database to an in-memory target database portion of the database management system including the target database using a recovery log file of the source database during synchronization; and asynchronously storing the persistently applied changes to the in-memory target database portion in persistent target database storage.

[0014] The method may include, upon a database crash of the target database system, restoring the in-memory target database portion with the most recent snapshot available in persistent target database storage, and, upon a database crash of the target database system, applying changes from a source database recovery log file having a later timestamp than the most recent snapshot available in persistent target database storage to the in-memory target database portion.

[0015] According to another aspect of the present invention, a linked database system having crash recovery for the linked database may be provided. The linked database may include a source database and an associated target database, and selected portions of the contents of tables in the source database may be synchronized with respective portions of the contents of tables in the target database. The linked database system may include a processor and memory communicatively coupled to the processor, the memory storing program code portions that, when executed, enable the processor to synchronize selected portions of the contents of tables in the source database with respective portions of the contents of tables in the target database, apply changes to the source database to an in-memory target database portion of a database management system including the target database using a recovery log file of the source database during synchronization, and asynchronously store the persistently applied changes to the in-memory target database portion in persistent target database storage.

[0016] Additionally, the memory may store program code portions that, when executed, enable the processor to restore the in-memory target database portion with the most recent snapshot available in persistent target database storage in the event of a database crash of the target database system, and to apply changes from a source database recovery log file having a later timestamp than the most recent snapshot available in persistent target database storage to the in-memory target database portion in the event of a database crash of the target database system.

[0017] The proposed computer-implemented method for crash recovery for linked databases may provide numerous advantages, technical effects, contributions, or improvements, or combinations thereof.

[0018] Updates and modifications to the in-memory portion, as well as queries, may continue without any slowdown due to the changes also being persisted to the target database. Therefore, logging and state storage of physical database data structures may be separated from IUD processing within the database transaction. This means that the persistence phase is skipped when changes to the target database, especially the in-memory portion, are made, and update operations can continue without additional delay. Therefore, records that may be needed for crash recovery of the target database may be written asynchronously by a dedicated asynchronous data persistence service. Changes made by an IUD transaction apply only to the in-memory representation of the data, so they may be immediately processed by the target database, i.e., by database queries offloaded to the accelerator, regardless of whether they have already been persisted to a persistent medium. This may also be interpreted as lifting the durability constraints of the database's ACID properties.

[0019] Because related changes are written asynchronously from in-memory data structures to their persistent counterparts, in the event of a database crash, some changes may be lost in the target database system. Therefore, crash recovery may need to synchronize a potentially incomplete snapshot with the source database system by replaying the lost changes. Unlike other solutions, where offloaded tables are not fully reloaded by a bulk load mechanism, the missing portions may be recovered incrementally. That is, the target database state may be recovered from the latest asynchronously persistent snapshot of the target database, with the help of recovery metadata to determine which additional changes may still be lost. That is, which changes from the source database need to be replayed by the target database from the source database system's transaction or recovery log files to ultimately restore the current consistent state of the target database.

[0020] This allows incoming queries on the target database to be delayed while the recovery process is active until a consistent update state is re-established in the in-memory portion of the target database. While the delay caused by recovery may be significantly lower compared to known solutions, the processing and storage costs of asynchronously maintaining database snapshots in the target database system may be slightly higher. However, this is a small price to pay for higher availability and shorter recovery times of the target database system.

[0021] Asynchronous persistent storage of changes to the target database may also help perform logging more efficiently, I / O operations may be performed more efficiently, for example, by a batch process, or self-cancelling changes may be deleted altogether, i.e., target records that have been modified in a way that can be recovered before the next persistent storage step of the target database changes may be deleted in a buffer between the target database and the persistent storage medium.

[0022] In the following, additional embodiments of the inventive concepts applicable to methods and systems will be described.

[0023] According to one interesting embodiment of the method, synchronizing may involve reading entries from a recovery log file related to the source database and applying the read entries to the target database. This may be performed by an administrative part of the target database management system. However, other techniques for synchronizing may also be used, such as those based on SQL (structured query language) or Q, a query language optimized for column-based databases.

[0024] According to one useful embodiment of this method, the source database may be optimized for transactions, i.e., online transaction processing (OLTP), or the source database may be a row-oriented relational DBMS, or both. This may represent and efficiently support the operational backbone of enterprise IT (information technology) operations. The row-oriented database may be optimized for balanced, high-speed read, write, and modify operations on the data within the database. The row-oriented database may also be effective for generating reports. However, this type of database is often not optimal for analytical-type operations.

[0025] According to one acceptable embodiment of this method, the target database may be optimized for analytical operations, i.e., online analytical processing (OLAP), or the target database may be a column-oriented database, or both. According to a further embodiment, this type of database may be a column-oriented database. This database may support queries with many dependencies and intercorrelations significantly better than OLTP-optimized database systems.

[0026] According to an advantageous embodiment, the method may also include, in the event of a target database crash, delaying queries to the target database until recovery of the target database can be completed. Thus, the recovery process may be transparent to users. Users may ultimately experience a slight delay in their analytical queries. However, because response times for complex analytical queries are relatively long in any event, users may not notice the delay at all.

[0027] Optionally, queries targeting the target database may be executed by the source database during the target database recovery process. This may slow down the source database a little during the transaction, but for overall user satisfaction this may be an acceptable compromise.

[0028] According to a preferred embodiment of this method, the metadata defining the selected tables may be part of the recovery log file. In this way, the general architecture of the in-memory target database may already be defined in the recovery log file of the source database. This may also apply to the data portion of the tables that should be mirrored to the source database. Thus, a single source for data definitions may be used so that conflict situations cannot arise.

[0029] According to an advanced embodiment of this method, storing the persistently applied changes may include waiting until a predetermined number of changes have been completed in the in-memory target database portion. The predetermined number may be configurable during database setup, or may be changed during operation of the database management system, or both. However, an advantage of this approach may be that it does not slow down analytical operations on the target database. Thus, a group of updates to the target database may be extracted and persistently stored at one time. In addition to using a predetermined number of changes to the target database, it may also be considered to monitor the load on the target database and perform persistent storage of changes to the target database during periods of relatively low analytical load on the target database.

[0030] This feature may be implemented in a separate thread or process that can wait until a predetermined number of changes are completed in the in-memory portion. For each change in the set of changes completed since the last execution, those changes are persisted to the persistent database portion of the target database. Additionally, to enable identification of change replay points during the recovery phase, associated metadata describing recently persisted changes (e.g., the last corresponding block record sequence number in the source database system) may also be persistently stored. This may enable the time of the last consistently stored set of changes. This time may be useful for recovering the in-memory portion of the target database from the persistent portion of the target database and incrementally recovering the remaining IUD of the in-memory portion of the target database from the source database recovery log file. This additional data is requested from the source database or the incremental update process when recovery of the target database is required. At the end of the recovery process, the query processor may be notified by a "reactive" signal that queries may now be processed from the target database.

[0031] According to another advantageous embodiment of the method, restoring the tables or partitions thereof of the in-memory target database portion may include prioritizing the recovery or restoration by one selected from the group consisting of data usage, query priority, and data priority.

[0032] These options are detailed in the following paragraphs, but the general concept is to enable different optimization methods during the recovery process to allow queries to run even while the target database is "rebuilding," i.e., in recovery mode. These optimization options may be configurable and may generally be combinable in one implementation.

[0033] According to one optional embodiment of this method, prioritizing recovery by data usage may include maintaining a counter for each table in the target database, or a counter for its partitions. The counter value may indicate how many queries may be waiting for related tables, with the database table with the highest counter value being restored first. Other tables may then be restored subsequently. This option may enable rapid recovery of tables in the target database that are in high demand. Thus, users may be provisioned with fully restored portions of the database, i.e., tables with high demand, as soon as possible. This option may be referred to as demand or data usage optimized recovery.

[0034] Therefore, a list of waiting queries may be considered to control the recovery process and minimize recovery time. When a new database query arrives at the target database while recovery is in progress, its data access may be analyzed. If the database query does not access non-recovered data, it may be processed immediately. Otherwise, the query is registered in the list of waiting queries as part of the recovery state. The recovery phase is evaluated by the recovery schedule to derive a set of recovery actions that prioritizes the tables and / or table partitions that are actually blocking the queries. Upon completion of recovery, the most beneficial recovery action may be scheduled via one of the available recovery strategies, the recovery state may be updated, and waiting queries against the recovered tables and / or partitions may be notified when they no longer need to be blocked.

[0035] In more detail, crash recovery for a target database can be described as follows: First, a list of tables or table partitions that need to be recovered by using known recovery metadata in the target database. Second, the list of tables or table partitions to be recovered is stored in the recovery state. Third, if the list of what to recover is not empty, the following may be done: (i) determining the next table and / or table partition to be recovered from the list; (ii) scheduling recovery actions via a configured recovery strategy for the target database; (iii) waiting for recovery actions to complete; (iv) updating the recovery state, i.e., marking the table / partition to be recovered and removing it from the blocked data list of all queries waiting for that table and / or table partition; and (v) determining the list of queries for which the blocked data list has been emptied and notifying the query processor that the query is now available to be processed;

[0036] Thereby, the list of tables and / or table partitions to be recovered may be implemented by a priority queue, using the algorithm described below to dynamically calculate the next recovery item for the next recovery cycle. To accelerate the recovery process, the step referred to in (iii) may be performed in parallel for multiple tables and / or table partitions.

[0037] The determination of the tables and / or table partitions to recover may be performed as follows: (i) creating empty histograms for the disconnected tables and / or table partitions when the crash recovery process is initiated; (ii) when a blocking query is registered in the recovery state, incrementing the usage counter for each table and / or table partition in the query's blocking area list by one; (iii) when it is necessary to determine the next table / table partition to be recovered, selecting the table / table partition with the highest usage counter; and (iv) When the table / table partition recovery operation is completed, the corresponding histogram data is removed.

[0038] According to another optional embodiment of the method, prioritizing recovery by query priority may include first restoring database tables that receive queries with the highest priority. Such priority values ​​may be assigned to the database system, e.g., production databases versus test databases, or may be assigned based on individual queries. This option may be referred to as query priority or simply data priority optimized recovery.

[0039] According to a further optional embodiment of this method, prioritizing recovery by data priority may include maintaining (at least) two groups of database tables, each group relating to a distinct group of users, and first restoring the database tables of the group with a higher constituent group priority. This situation may arise in a multi-user / multi-group / multi-tenant environment, where one user, group, or tenant may be assigned a higher priority for queries. For example, one tenant may have a higher guarantee of availability of the database system. In such a case, this tenant may be assigned a higher priority. This scenario may work best when the multi-tenant database is operated in a cloud computing data center. This option may be referred to as customer-priority-optimized recovery.

[0040] According to another interesting embodiment, the method may further include determining a data volume to be restored for the next table to be restored and recovering the table(s) using a recovery strategy depending on the volume to be restored. The recovery strategy may be an incremental update strategy or a bulk update strategy. Therefore, it may be determined for each database table (or group of database tables) which update strategy to use in order to minimize the total recovery time of the target database. This option may be referred to as time-optimized recovery.

[0041] Furthermore, embodiments may take the form of a related computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer-readable medium for purposes of this description may be any apparatus that may include means for storing, communicating, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0042] It should be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will infer from the above and following description that, unless otherwise indicated, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matters, in particular any combination of features of a method-type claim with features of an apparatus-type claim, is also considered to be disclosed in this document.

[0043] The above-defined aspects and further aspects of the invention will be apparent from and will be elucidated with reference to the example embodiments described hereinafter, to which the invention is not limited.

[0044] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings: [Brief explanation of the drawings]

[0045] [Figure 1] 1 is a block diagram illustrating an embodiment of the computer-implemented method of the present invention for crash recovery for linked databases. [Figure 2] FIG. 1 is a block diagram illustrating an embodiment of a linked database. [Figure 3] FIG. 1 is a block diagram illustrating an embodiment of how a target database may be synchronized. [Figure 4] FIG. 1 is a block diagram showing an embodiment of the proposed concept in a form closer to implementation. [Figure 5] FIG. 1 is a block diagram illustrating an embodiment of a linked database system including components for a customer priority optimization recovery strategy. [Figure 6] FIG. 1 is a block diagram illustrating an embodiment of a linked database system including components for a volume-optimized recovery strategy. [Figure 7] 1 is a block diagram illustrating an embodiment of a linked database system for crash recovery for linked databases. [Figure 8] FIG. 1 illustrates an embodiment of a computer system including a linked database system. DETAILED DESCRIPTION OF THE INVENTION

[0046] In the context of this description, the following conventions, terms or expressions, or combinations thereof, may be used.

[0047] The term "crash recovery" may refer to the process of reconstructing the state of a database before a crash occurred. When a crash occurs, data may be unavailable or inconsistent.

[0048] The term "linked database" may refer to at least two databases that are closely related to each other. In the context of this document, linked databases may be referred to as databases that may store at least part of the same data. On the other hand, the primary database in such a pair may be optimized for a different task than the secondary database.

[0049] The terms "source database" or "primary database" may refer to a database optimized for, for example, high-speed transactions, i.e., online transaction processing. However, a database optimized in this manner, i.e., for high-speed read, write, and update operations on data, may be slow to execute complex queries involving many tables or a large amount of data, such as in the case of online analytical processing. In addition, online analytical processing may slow down online transaction processing. Thus, highly optimized database management systems for the types of databases just mentioned may function successfully in tandem.

[0050] The terms "target database" or "secondary database" may refer to the second database in such a tandem of databases optimized for different tasks. In the context of the concepts described herein, the target database may be optimized for online analytical processing. The target database may store at least some of the tables and some of the table data of the source database. In addition, the target database may include two parts: an in-memory part for fast execution of complex multidimensional queries, and a persistent part that may store the tables and data of the in-memory part of the target database in more long-term storage, such as a hard disk or flash memory. In this way, the target database may be able to recover most of its contents from persistent storage in the event of a target database crash.

[0051] The term "selected portion of the table contents" may refer to the just-mentioned portion or partition of data of a portion of a table in a primary database that may be synchronously copied and stored in a target database.

[0052] The term "selected query" may refer to a particular type of query directed to a linked database that, based on the nature of the query, is better performed by one of the two databases, particularly the target database. For example, when the query type involves online analytical processing, the query may be forwarded to the target database and not performed by the source database.

[0053] The term "database management system" may refer to a combination of administration / management system, typically implemented in a combination of hardware and software, and at least one associated database that stores data.

[0054] The term "in-memory target database portion" may refer to a portion of a target database that may hold substantially all of its data in a computer system's main memory. A database management system for a target database may include an in-memory portion and a persistent portion of the target database, where the persistent portion may be a durably stored copy of the in-memory portion apart from the last changes made to it in memory.

[0055] The term "persistent target database storage" may describe that portion of a target database management system that enables target database data to be stored persistently, i.e., using a hard disk or flash memory instead of in-memory storage.

[0056] The term "latest snapshot" may refer to the last consistently stored state of the target database.

[0057] The term "later timestamp", and in particular changes having later timestamps, may refer to, for example, records in the source database recovery log file that contain a time indicator that may have been created later in time than the most recent snapshot stored in the persistent portion of the target database.

[0058] The term "recovery log file" may refer to the set of file protocol operations performed on a database, specifically all operations that modify data, i.e., insert, update, and delete operations. A recovery log file may be designed to allow a complete reconstruction of the database. Therefore, the table definitions of a database may also be part of the recovery log file.

[0059] The term "metadata" may refer to data about data, particularly the definitions of data in tables in a database, and potentially the relationships between them.

[0060] A detailed description of the drawings will be provided below. All instructions in the drawings are schematic. First, a block diagram of an embodiment of the computer-implemented method of the present invention for crash recovery for linked databases will be provided. Afterwards, further embodiments and embodiments of a linked database system with crash recovery for linked databases will be described.

[0061] 1 shows a block diagram of a preferred embodiment of a computer-implemented method 100 for crash recovery for linked databases, particularly according to the IDAA architecture, where the linked databases provided (102) include a source or primary database, particularly optimized for transactions, such as a role-based database, and an associated target or secondary database, which may be optimized for online analytical processing operations (OLAP) and advantageously organized on a column basis.

[0062] Selected queries to a database management system including a source database, particularly queries directed to analytical operations, are transferred, or offloaded, to a database management system including a target database for processing.

[0063] Additionally, method 100 includes synchronizing (104) selected portions of the contents of the tables of the source database (in some implementations, only a portion of the data in the portions of the tables, in other implementations, a complete copy) with respective portions of the contents of the tables of the target database by applying (106) changes to the source database during synchronization using the recovery log file of the source database to the in-memory portion of the target database of the database management system that includes the target database.

[0064] Additionally, the method 100 includes asynchronously storing (108) the persistently applied changes to the in-memory target database portion in a persistent target database storage portion; restoring (110) the in-memory target database portion with the most recent snapshot available in the persistent target database storage in the event of a database crash of the target database system; and applying (112) changes from a source database recovery log file having a later timestamp than the most recent snapshot available in the persistent target database storage to the in-memory target database portion in the event of a database crash of the target database system.

[0065] Optionally, the persistent portion of the target database may also be updated in parallel, but this would require that a persistent storage process or storage processor also be active to durably store the changes to the in-memory portion of the target database.

[0066] Additionally, it may be advantageous to bulk load the target database from the source database, especially during the initialization process. This can help avoid the equally slower incremental update or synchronization process because the source database may already have a larger number of entries and therefore a larger number of entries in its recovery log file. This initialization may be effective only for selected tables or their partitions.

[0067] Figure 2 shows a block diagram of a linked database embodiment 200. A primary or source database 202 receives OLTP queries 210 and OLAP queries 214. Queries identified as OLAP queries 214 are migrated or offloaded 222 to a secondary or target database 206. The source database 202 includes multiple tables 204 and associated stored data. The target database 206 also includes tables 208 that represent at least a subset of the database tables 204 of the source database 202 and at least a subset of that data.

[0068] After the OLAP operations are performed by the target database 206, the data is returned 224 and returned to the requesting program or process as OLAP output 220. The OLTP query 210 is executed directly in the source database 202 and returned to the requesting program or process as OLTP output 212. Thus, the OLTP or source database 202 may operate at its peak performance because it is not slowed down by any resource-intensive OLAP queries.

[0069] Because the data organization in the target database 206 may be different, for example, column-oriented instead of row-oriented in the source or OLTP database 202, the target database may return 224 OLAP results significantly faster than the source database 202.

[0070] Figure 3 shows a block diagram of an embodiment 300 of how target databases can be synchronized. A source database management system 302 controls the operation of the source database 202 and its associated tables 204 (compare Figure 2). The same is true for a target database management system 308, which controls the operation of the target database 206 and its associated tables 208.

[0071] The source database management system 302 also maintains a recovery log file 306 for the source database 202. A log reader or log file reader 314 reads the recovery log file 306 and provides this data to an apply unit 316, which applies the changes (i.e., inserts, updates, and deletes) made to the source database to a selected set of selected tables and data in the target database 206. The selected set of selected tables and data may be a subset of related tables and data in the source database 202 as predefined. The apply unit 316 can optimize the application of changes to the target database 206 depending on the OLAP query being executed. A log buffer 318 may be useful for this purpose.

[0072] To initialize the target database 206, a bulk load operation 312 may be performed from the source database 202 to the target database 206 for performance reasons.

[0073] Note that for reasons of conceptual clarity, Figure 3 does not yet show the division of the target database management system into in-memory and persistent portions of the target database 206 and its tables 208. This will be shown in the next figure.

[0074] It should be noted that this synchronization mechanism may represent one of many synchronization techniques, although other synchronization techniques may also be applicable.

[0075] Figure 4 shows a block diagram of an embodiment of the proposed concept in a more practical form 400. Elements of the source database management system (primarily the upper part of Figure 4) that are optionally used to synchronize the source database with the target database will not be described again.

[0076] The target database management system 308 includes an in-memory portion of the target database 402 and a persistent portion of the target database 404. A persistence service 406 writes the state of the in-memory target database portion 402 to the persistent database 404 during operation. During these regular operations, such as after a predetermined number of updates to the in-memory database 402, a query processor 408 directs incoming queries to the in-memory portion of the target database 402.

[0077] However, if the target DBMS is currently undergoing crash recovery, especially if it is performing demand or data usage optimized recovery, the behavior differs as follows: (i) the query processor 408, together with the recovery processor 410, analyzes the query data accesses and determines, using the current query analyzing techniques, a list of target tables and / or a list of table partitions that need to be made available for the query; (ii) determining the blocked data list by looking at all of the tables and / or table partitions in the data access list from the previous step in the recovery state that have been determined not to have been recovered; (iii) if the list from the previous step is not empty, the query and its blocked data list are added to the waiting queries, and the query process waits until recovery is finished (the blocked data list is empty), and then the query can continue; and (iv) After the recovery processor 410 sends a completion notification to the query processor 408, the query of the target database is processed as normal.

[0078] During operations controlled by the recovery processor 410, the latest consistent snapshot available in the persistent database 404 of the relevant tables and / or table partitions is loaded 412 into the in-memory portion 402 of the target database, and entries in the recovery log file 306 of the source database 202 that have a later timestamp than the snapshot in the persistent database 404 are recreated in the in-memory database 402. These are requested by the recovery processor 410 via the query processor 408 and provided by the source DBMS 302, e.g., from the recovery log file 306, via the log reader 314 and apply unit 316.

[0079] 5 shows a block diagram of an embodiment 500 of a linked database system including components for a customer priority optimization recovery strategy. A source DBMS 502 illustratively has a first source database 504 for a first user or tenant (e.g., in a cloud computing environment) and a second source database 506 for a second user or tenant. In addition, separate source database systems for other users or tenants may be available.

[0080] Separate data synchronization subsystems 510, 512 are implemented within the data synchronization system 508 to synchronize the source databases 504, 506 with respective in-memory portions 518, 520 of an in-memory target database 516. A query process 522 receives database queries for execution in the in-memory portion of the target database(s) 516. These queries are typically OLAP queries offloaded from the source database(s) 504, 506.

[0081] The recovery process or processor 524 of the target DBMS 514 also receives data from the query processor regarding queries to be registered and analyzed in the data access analyzer 526. During the target database recovery process, the query analyzer determines which tables should be recovered first based on the priority of a particular user's queries by determining the queued queries 530 and already recovered table partitions 532 in the recovery state management system 528. This is ultimately determined and decided by the recovery schedule 534. To accomplish this task, the recovery schedule 534 constantly exchanges data with the recovery state management system 528 to check the actual recovery status and receives configuration data from the workload management system configuration storage. The recovery schedule 534 also exchanges data with the data synchronization system 508 to trigger the target database management system recovery database load from the source database management system 502.

[0082] In this way, you can ensure that users or customers with higher configured priorities get earlier access to the recovered database tables, and you can also optimize the recovery time depending on the access usage of a particular table.

[0083] In detail, this may be achieved by the following steps: When the crash recovery process for the target database is initialized, an empty histogram of blocked tables / table partitions for each tenant is created. When a new blocked query is registered in the recovery state, the usage counter for each table / partition in the query's blocked data list is incremented by one. Then, when the next table / partition to be recovered needs to be determined, the table recovery priority is determined based on the blocked query usage counter count and the current tenant's WLM (workload management system) configuration (i.e., its priority or importance), and the recovery item with the highest priority is selected. Finally, when the table / partition recovery operation is completed, the corresponding histogram entry is also removed.

[0084] Calculating user or tenant specific recovery priorities can be exemplarily implemented as follows: When strict priorities exist between pairs of tenants, e.g., production systems are more important than test systems (which may be specified by simple configuration text), the list of tables / partitions to be recovered is grouped by user / tenant, the list of groups is ordered by decreasing user / tenant priority, and the next table / partition is selected from the first non-empty group by a query usage counter selection algorithm.

[0085] However, when there is a relative resource share position for each tenant in the WLM configuration (e.g., tenant 1 50%, tenant 2 30%, tenant 3 20%), the priority for each recovery item is calculated as follows: (i) determine the query usage counter q for the recovery item; (ii) then determine the tenant t to which the recovery item belongs; (iii) then determine the resource share position r(t) for tenant t; (iv) based thereon, determine the priority as a*q*b*r(t), where a and b are static configuration parameters, particularly in the range [0...1], and here, to reduce the number of parameters, b can be calculated based on a, e.g., b=1-a. Finally, (v) the list of recovery items can be ordered by priority, e.g., in a priority queue.

[0086] Figure 6 shows a block diagram of an embodiment 600 of a linked database system including components for a volume-optimized recovery strategy. Elements already introduced by Figure 5 are indicated with the same reference numerals. On the upper left side, the source DBMS 502 is shown along with the source database 504 and associated recovery log file 604. Initially, the in-memory database portion 516 (persistent portion not shown) of the target DBMS 514 may be bulk loaded via the bulk loader 602.

[0087] In addition to the in-memory portion 516 of the target DBMS 514, the focus here may be on other metadata 608 and the recovery process or recovery processor 614. The recovery process includes at least three components: a recovery item selection unit 616, a change estimation unit 618, and a recovery schedule 620. The recovery processor 614 exchanges data with the in-memory database 516 to collect state information about tables in the in-memory database portion 516 of the target DBMS 514.

[0088] As already mentioned above, in the event of a database crash at the target DBMS, the target database 516 needs to be restored from the source database 504 either via a bulk load mechanism or incrementally. Additionally, the tables / table partitions to be recovered are dynamically selected by the recovery process 614. The proposed concept is extended by a change estimation component 618, which is responsible for estimating or determining the amount of data that needs to be restored during crash recovery. Therefore, this change estimation component 618 evaluates data change statistics to estimate how many data changes have accumulated in the source database since the recovery baseline timestamp. Based on this information, the recovery scheduler 620 selects the most efficient data synchronization method to restore the data of the tables / table partitions to be recovered. The data change statistics are maintained during regular updates of the target database (i.e., the in-memory database portion 516) via incremental or bulk loads. The change estimation 618 may be triggered by the recovery item selection component 616.

[0089] The recovery scheduler 620 also interacts with the details of the bulk loader 602 and the incremental update process 606 to manage the recovery process of the in-memory database portion 516. For example, when recovery of the in-memory database portion 516 is completed, a recovery completion notification is received from the bulk loader 602. On the other hand, the recovery scheduler 620 requests the reproduction of changes to specific tables in the in-memory database portion 516. Similar to the signal from the bulk loader 602, the recovery scheduler 620 also receives recovery completion notifications from the incremental update process (processor) 606. As already explained in the context of FIG. 3, the incremental update process 606 includes a log reader (not shown) adapted to read recovery log file entries from the recovery log file 604 and a recovery log file application unit (not shown) adapted to incrementally update the in-memory database portion 516 with respective recovery log file entries from the source database 504. Please refer back to Figure 3 for details.

[0090] To successfully manage the processes of the embodiments described herein, data change statistics should be stored in a persistent metadata catalog included in the metadata 608 of the target database system 514 and should be maintained when the target database system 514 is updated, i.e., via incremental updates or bulk load strategies. The data change statistics may store the following information: (i) the timestamp when the update was processed; (ii) schema information of the database table, e.g., column types, column widths, ...; (iii) the amount of data changes per table / table partition affected by the update, i.e., inserted, deleted, and updated records; as well as aggregated metrics, e.g., total number of records changed, total data volume applied, total execution time of the update, etc.

[0091] The statistical data may be linked to a time interval, for example, the last x days, etc. Additionally, the statistical data may be maintained incrementally as part of each update cycle.

[0092] Furthermore, the recovery baseline 610 is also maintained in the persistent metadata 608 (e.g., metadata catalog) of the target database system 514. The recovery baseline 610 is required for estimation to determine the data volume that needs to be restored. Therefore, it is not necessary to determine the exact timestamp; it is sufficient to estimate its value. For example, it may be maintained as a heartbeat timestamp updated by the target database system at regular time intervals, it may be maintained as a starting point for the recovery process after a crash, or it may be maintained as the time of the last successfully persisted target database snapshot. Thereby, a timestamp may be maintained for each table in the target database.

[0093] Crash recovery for the target database can be performed as follows: (i) first, the next table / partition to be recovered is determined; (ii) the data volume that needs to be recovered after the recovery baseline is estimated; (iii) based on the estimated data volume, the best recovery strategy is selected and the recovery time is estimated; (iv) then, table recovery according to the selected strategy is scheduled; and (v) these steps are repeated in a loop until all data is recovered.

[0094] Estimating the data volume to be recovered may be performed as follows: (i) for the table / partition to be recovered, the corresponding data change statistics 612 are looked up; (ii) a recovery baseline for that table / partition is determined; and (iii) the number of changes that need to be replicated via the incremental update process in the interval [recovery baseline, current recovery time] is estimated.

[0095] 7 shows a block diagram of an embodiment of a linked database system 700 for crash recovery for linked databases. The linked database 700 includes a source database 706 and an associated target database 708, where selected portions of the contents of tables in the source database are synchronized with respective portions of the contents of tables in the target database.

[0096] The linked database system 700 includes a processor 702 and a memory 704 communicatively coupled to the processor 702, the memory 704 storing program code portions that, when executed, enable the processor to synchronize selected portions of the contents of tables in a source database 706 with respective portions of the contents of tables in a target database 708, for example, using a synchronization unit 714.

[0097] The stored program code portions, when executed, enable the processor 702 to synchronously apply, using the apply unit 716, changes to the source database 706 to an in-memory target database portion 710 of a database management system that includes the target database, and to asynchronously persist the applied changes to the in-memory target database portion 712 in persistent target database storage, for example by the storage processor 718.

[0098] Further, the stored program code portions, when executed, enable the processor 702 to restore the in-memory target database portion 710 with the latest snapshot available in the persistent target database storage portion 712 in the event of a database crash of the target database system, such as by a restore unit 720, and to apply to the in-memory target database portion 710, such as by a second apply unit 722, changes from a source database recovery log file having a later timestamp than the latest snapshot available in the persistent target database storage portion 712 in the event of a database crash of the target database system.

[0099] Additionally, all modules and units of the linked database system 700 may be electrically interconnected to exchange signals and / or data. This applies to the processor 702, memory 704, source database system 706, target database system 708 including in-memory portion 710 and persistent portion 712, synchronization unit 714, application unit 716, storage processor 718, recovery unit 720, and second application unit 722. Instead of a 1:1 connection between these modules and units, they may be connected to a linked database system internal bus system 724 for signaling and data exchange.

[0100] Embodiments of the present invention may be implemented in conjunction with virtually any type of computer platform, regardless of whether the platform is suitable for storing and / or executing program code. Figure 8 illustrates, by way of example, a computer system 800 suitable for executing program code relating to the proposed method.

[0101] Computer system 800 is merely one example of a suitable computer system, and whether computer system 800 is capable of implementing and / or performing any of the functions identified above is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. Computer system 800 has components that operate in conjunction with numerous other general-purpose or special-purpose computer system environments or configurations. Examples of well-known computer systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 800 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices. Computer system / server 800 may be described in the general context of computer system-executable instructions, such as program modules, executed by computer system 800. Generally, program modules may include routines, programs, objects, components, logic, and data structures that perform particular tasks or implement particular abstract data types. Computer system / server 800 may also be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0102] As shown in the drawings, computer system / server 800 is illustrated in the form of a general-purpose computing device. Components of computer system / server 800 may include, but are not limited to, one or more processors or processing units 802, a system memory 804, and a bus 806 that couples various system components, including the system memory 804, to the processor 802. Bus 806 may represent any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus. Computer system / server 800 typically includes a variety of computer system-readable media. Such media may be any available media that can be accessed by computer system / server 800, including both volatile and nonvolatile media, removable and non-removable media.

[0103] The system memory 804 may include computer-system-readable media in the form of volatile memory, such as random access memory (RAM) 808 and / or cache memory 810. The computer system / server 800 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 812 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such cases, each may be connected to the bus 806 by one or more data media interfaces. As will be further shown and described below, memory 804 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0104] By way of example and not limitation, a program / utility having a set (at least one) of program modules 816, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 804. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may comprise an implementation of a networking environment. The program modules 816 generally perform the functions and / or methods of embodiments of the present invention described herein.

[0105] Computer system / server 800 may also communicate with one or more external devices 818, such as a keyboard, pointing device, display 820, one or more devices that allow a user to interact with computer system / server 800, or any device (e.g., network card, modem, etc.) that allows computer system / server 800 to communicate with one or more other computer devices, or any combination thereof. Such communication may occur via input / output (I / O) interface 814. Additionally, computer system / server 800 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via network adapter 822. As shown, network adapter 822 may communicate with other components of computer system / server 800 via bus 806. Although not shown, it should be understood that other hardware and / or software components may be used with computer system / server 800. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0106] Additionally, a linked database system 700 with crash recovery for the linked databases may be attached to the bus system 806 .

[0107] The descriptions of various embodiments of the present invention have been provided for illustrative purposes and are not intended to be exhaustive or limiting to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements to technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0108] The present invention may be embodied as a system, method, or computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0109] The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system for a propagation medium. Examples of computer-readable media may include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read-only memory (CD-ROM), compact disk-read / write (CD-R / W), DVDs, and Blu-Ray® disks.

[0110] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves with recorded instructions, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted through wires.

[0111] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium into each computing / processing device, or may be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, fiber optic transmission cables, wireless transmission cables, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0112] Computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by using state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0113] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0114] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, cause means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored includes instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0115] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or another device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, where the instructions executed on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0116] The flowcharts and / or block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. In addition, it will be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0117] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the present invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising" as used in this specification specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0118] Structure, material, acts, and equivalents corresponding to all means- or step-plus-function elements in the following claims are intended to include any structure, material, or acts for performing that function in combination with other specifically claimed elements. The description of the present invention has been provided for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The embodiments have been chosen and described to best explain the principles and practical applications of the invention and to enable others skilled in the art to understand the invention in various embodiments with various modifications as may be suitable for the particular uses anticipated.

[0119] Briefly, the concepts of the present invention may be summarized in the following items.

[0120] 1. A computer-implemented method for crash recovery for linked databases, the linked databases including a source database and an associated target database, wherein selected queries for a database management system including the source database are transferred to a database management system including the target database for processing, the method comprising: - synchronizing selected portions of the contents of tables in a source database with respective portions of the contents of tables in a target database; - during synchronization, applying changes to the source database to the in-memory target database portion of the database management system that contains the target database; asynchronously storing the persistently applied changes to the in-memory target database portion in a persistent target database storage; - in the event of a database crash of the target database system, restoring the in-memory target database part by the latest snapshot available in the persistent target database storage; - applying, in the event of a database crash of the target database system, changes from a source database recovery log file having a later timestamp than the most recent snapshot available in persistent target database storage to the in-memory target database portion.

[0121] 2. The method according to item 1, wherein the synchronizing - involves reading recovery log file entries relating to the source database and applying those entries to the target database.

[0122] 3. The method according to item 1 or 2, wherein the source database is optimized for transactions, or the source database is a row-oriented relational database management system, or both.

[0123] 4. A method according to any of the preceding items, wherein the target database is optimized for analytical operations, or the target database is a column-oriented database, or both.

[0124] 5. A method according to any of the preceding items, - further comprising, in the event of a target database crash, delaying queries to the target database until the target database has finished recovering.

[0125] 6. A method according to any of the preceding items, wherein the metadata defining the selected table is part of the recovery log file.

[0126] 7. A method according to any of the preceding items, wherein storing permanently applied changes is - Involves waiting for a predetermined number of changes to be completed in the in-memory target database portion.

[0127] 8. The method according to any of the preceding items, wherein restoring tables of the in-memory target database portion includes prioritizing the recovery by one selected from the group consisting of data usage, query priority, and data priority.

[0128] 9. The method according to item 8, wherein prioritizing recovery by data use: - maintaining a counter for each table in the target database, the counter value indicating how many queries are waiting for the associated table; - This involves restoring the database table with the highest counter value first.

[0129] 10. The method according to item 8 or 9, wherein prioritizing recovery by query priority is - This involves restoring first the database tables that receive the queries with the highest priority.

[0130] 11. The method according to any one of items 8 to 10, wherein prioritizing recovery by data priority is - maintaining two group database tables, each group relating to a distinct group of users; - This involves first restoring the database tables of groups with higher configuration group priority.

[0131] 12. A method according to any of the preceding items, - determining the data volume to be recovered for the next table to be recovered; - Recovering the table using a recovery strategy that depends on the volume to be recovered, the recovery strategy being an incremental update strategy or a bulk update strategy.

[0132] 13. A linked database system having crash recovery for linked databases, the linked databases including a source database and an associated target database, wherein selected portions of the contents of tables in the source database are synchronized with respective portions of the contents of tables in the target database, the linked database system comprising: - a processor; and a memory communicatively coupled to the processor, the memory storing program code portions that, when executed, cause the processor to - synchronizing selected portions of the contents of tables in a source database with respective portions of the contents of tables in a target database; - during synchronization, applying changes to the source database to the in-memory target database portion of the database management system that contains the target database; asynchronously storing the persistently applied changes to the in-memory target database portion in a persistent target database storage; - in the event of a database crash of the target database system, restoring the in-memory target database part by the latest snapshot available in the persistent target database storage; - In the event of a database crash of the target database system, it allows changes from a source database recovery log file with a later timestamp than the most recent snapshot available in the persistent target database storage to be applied to the in-memory target database portion.

[0133] 14. A linked database system according to item 13, wherein the program code portion is - Further allows for reading recovery log file entries relating to the source database and applying them to the target database for synchronization.

[0134] 15. A linked database system according to item 13 or 14, where the source database is optimized for transactions or the source database is a row-oriented relational database management system, or both.

[0135] 16. A linked database system according to any of items 13 to 15, wherein the target database is optimized for analytical operations, or the target database is a column-oriented database, or both.

[0136] 17. A linked database system according to any one of items 13 to 16, wherein the program code portion is - In case of a target database crash, it is further possible to delay queries against the target database until the target database has finished recovering.

[0137] 18. A linked database system according to any of items 13 to 17, wherein the metadata defining the selected tables is part of the recovery log file.

[0138] 19. A linked database system according to any one of items 13 to 18, wherein the program code portion is - Further enabling waiting for a predetermined number of changes to be completed in the in-memory target database portion in order to store the changes to be permanently applied.

[0139] 20. A linked database system according to any of items 13 to 19, wherein restoring tables of the in-memory target database portion includes prioritizing recovery by one selected from the group consisting of data usage, query priority, and data priority.

[0140] 21. A linked database system according to item 20, wherein the program code portion is configured to cause a processor to prioritize recovery by data usage. - maintaining a counter for each table in the target database, the counter value indicating how many queries are waiting for the associated table; - Also allows you to restore the database table with the highest counter value first.

[0141] 22. A linked database system according to item 20 or 21, wherein the program code portion is configured to cause a processor to prioritize recovery by query priority. - What also allows database tables that receive queries with the highest priority to be restored first.

[0142] 23. A linked database system according to any one of items 20 to 22, wherein the program code portion is configured to cause a processor to prioritize recovery by data priority. - maintaining two group database tables, each group relating to a distinct group of users; - Further enables database tables of groups with higher configuration group priority to be restored first.

[0143] 24. A linked database system according to any one of items 13 to 15, wherein the program code portion is - determining the data volume to be recovered for the next table to be recovered; - Recovering the table using a recovery strategy that depends on the volume to be recovered, the recovery strategy being an incremental update strategy or a bulk update strategy.

[0144] 25. A computer program product for a linked database system having crash recovery for linked databases, the linked database including a source database and an associated target database, wherein selected portions of the contents of tables in the source database are synchronized with respective portions of the contents of tables in the target database, the computer program product including a computer-readable storage medium having embodied thereon program instructions executable by one or more computer systems or controllers to cause the one or more computer systems to: - synchronizing selected portions of the contents of tables in a source database with respective portions of the contents of tables in a target database; - during synchronization, applying changes to the source database to the in-memory target database portion of the database management system that contains the target database; asynchronously storing the persistently applied changes to the in-memory target database portion in a persistent target database storage; - in the event of a database crash of the target database system, restoring the in-memory target database part by the latest snapshot available in the persistent target database storage; - In the event of a database crash of the target database system, applying changes from a source database recovery log file with a later timestamp than the most recent snapshot available in the persistent target database storage to the in-memory target database portion.

Claims

1. 1. A computer-implemented method for crash recovery for linked databases, the linked databases including a source database and an associated target database, wherein selected queries for a database management system including the source database are transferred for processing to a database management system including the target database, the method comprising: synchronizing selected portions of the contents of the tables of the source database with respective portions of the contents of the tables of the target database; During said synchronization, applying changes to said source database to an in-memory target database portion of said database management system that includes said target database; asynchronously storing persistently applied changes to the in-memory target database portion to a persistent target database storage; In the event of a crash of the target database, restoring the in-memory target database portion with the latest snapshot available in the persistent target database storage; applying, upon said crash of said target database, changes from recovery log files of said source database having a timestamp later than the most recent snapshot available in said persistent target database storage to said in-memory target database portion; A method comprising:

2. The synchronizing 2. The method of claim 1, further comprising: reading entries from a recovery log file relating to said source database; and applying said read entries to said target database.

3. 2. The method of claim 1, wherein the source database is optimized for transactions or the source database is a row-oriented relational database management system.

4. The method of claim 1 , wherein the target database is optimized for analytical operations or the target database is a column-oriented database.

5. 2. The method of claim 1, further comprising, in the event of the crash of the target database, delaying queries to the target database until recovery of the target database is complete.

6. 2. The method of claim 1, wherein metadata defining selected tables is part of the recovery log file.

7. Storing the permanently applied changes 2. The method of claim 1, further comprising waiting until a predetermined number of modifications are completed in the in-memory target database portion.

8. 2. The method of claim 1, wherein the restoring the tables of the in-memory target database portion includes prioritizing recovery from the crash by one selected from the group consisting of data usage, query priority, and data priority.

9. said prioritizing said recovery by said data usage; maintaining a counter for each table in the target database, the counter value indicating how many queries are waiting on the associated table; The database table with the highest counter value is restored first. The method of claim 8, comprising:

10. said prioritizing said recovery by said query priority; 9. The method of claim 8, further comprising first restoring the database table that receives a query with the highest priority.

11. said prioritizing said recovery by said data priority; maintaining two groups of database tables, each group relating to a distinct group of users; Restoring the database tables of the groups with higher configuration group priority first; The method of claim 8, comprising:

12. determining a data volume to be restored for the next table to be restored; recovering the table using a recovery strategy that depends on the volume to be recovered, the recovery strategy being either an incremental update strategy or a bulk update strategy that recovers via a bulk load mechanism; The method of claim 1 further comprising:

13. A linked database system, the linked database including a source database and an associated target database, wherein selected portions of the contents of tables of the source database are synchronized with respective portions of the contents of the tables of the target database; a processor and a memory communicatively coupled to the processor, synchronizing selected portions of the contents of the table of the source database with respective portions of the contents of the table of the target database; During said synchronization, applying changes to said source database to an in-memory target database portion of a database management system that includes said target database; asynchronously storing persistently applied changes to the in-memory target database portion to a persistent target database storage; In the event of a crash of the target database, restoring the in-memory target database portion with the latest snapshot available in the persistent target database storage; applying, upon said crash of said target database, changes from recovery log files of said source database having a timestamp later than the most recent snapshot available in said persistent target database storage to said in-memory target database portion; A linked database system configured to run

14. The synchronizing 14. The linked database system of claim 13, further comprising reading entries in the recovery log file relating to the source database and applying the read entries to the target database.

15. 14. The linked database system of claim 13, wherein the source databases are optimized for transactions or are row-oriented relational database management systems.

16. 14. The linked database system of claim 13, wherein the target database is optimized for analytical operations or the target database is a column-oriented database.

17. In the event of the crash of the target database, the processor:

14. The linked database system of claim 13, wherein queries to the target database are delayed until recovery of the target database is complete.

18. 14. The linked database system of claim 13, wherein the metadata defining the selected table is part of the recovery log file.

19. Storing the permanently applied changes 14. The linked database system of claim 13, including waiting until a predetermined number of modifications are completed in the in-memory target database portion.

20. 14. The linked database system of claim 13, wherein said restoring tables of said in-memory target database portion includes prioritizing said crash recovery by one selected from the group consisting of data usage, query priority, and data priority.

21. said prioritizing said recovery by said data usage; maintaining a counter for each table in the target database, the counter value indicating how many queries are waiting on the associated table; The database table with the highest counter value is restored first.

21. The linked database system of claim 20, comprising:

22. said prioritizing said recovery by said query priority; 21. The linked database system of claim 20, including first restoring the database table that receives a query with the highest priority.

23. said prioritizing said recovery by said data priority; maintaining two groups of database tables, each group relating to a distinct group of users; Restoring the database tables of the groups with higher configuration group priority first; 21. The linked database system of claim 20 further configured to perform:

24. the processor determining a data volume to be restored for the next table to be restored; recovering the table using a recovery strategy that depends on the volume to be recovered, the recovery strategy being either an incremental update strategy or a bulk update strategy that recovers via a bulk load mechanism; 14. The linked database system of claim 13 further comprising:

25. A computer program for crash recovery for linked databases, the computer program causing a processor to carry out a method according to any one of claims 1 to 12.

Images