Change-aware snapshot replication

Abstract

To provide change-aware snapshot replication.SOLUTION: Typically, an initial replica obtained using snapshot replication is periodically updated. However, the update process can unnecessarily consume computing resources if data in a source data object has not changed with respect to data in a replica data object. Disclosed techniques check to determine whether a snapshot replica is out of date before obtaining a new snapshot. The checks can be performed on manual request or on the occurrence of triggers, such as receiving a query that accesses the replica data object or according to a schedule. Information for current and prior versions of the remote data object can be compared to determine whether a replica is out of date, such as digest values of contents of the remote data object or timestamps associated with the remote data object.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to data replication. Particular implementations relate to determining whether a source data object has changed compared to a target data object before replicating data from the source data object to the target data object. [Background technology]

[0002] It is becoming increasingly common for enterprises to store data in a variety of systems, including one or more local systems and one or more cloud-based systems. The systems can be of different types, storing data in different formats (e.g., a relational database versus a database that stores JSON documents) or using different database management systems (e.g., using software and / or hardware provided by different vendors). Even if the data is stored in the same format using software from the same vendor, there may be differences in what data is stored in a particular location and the schema used to store the data.

[0003] To help address these issues, various techniques have been used, including data replication and database federation. In a federated database environment, a request for a database operation, such as a query, may specify the source of a local database system or a "remote" database accessed using data federation. In some cases, both local and remote data sources may be specified in the same query, such as having a query that retrieves data from a local database table and data from a data source in a remote, federated database system.

[0004] Data replication provides a strategic advantage for the business by allowing the target system to access and utilize data originally present in the source system that supplements the information already natively stored in the target system itself. In this architecture, the source system essentially acts as the more authoritative or primary source of truth for the replicated data. This means that the target system can maintain its own data while also leveraging the additional information replicated from the source system to enhance the target system's data processing or analytical capabilities. However, any discrepancies or conflicts involving the replicated data are typically resolved in favor of the source system given its status as a more trusted source.

[0005] For various reasons, such as query execution, data replication may be a better solution than data federation in some scenarios. However, in many cases, data in a source system is not "static" as data can be added to, removed from, or changed in the source system. Therefore, rather than replicating data from the source system to the target system once, a process is implemented to update the data in the target system based on changes made in the source system.

[0006] One way to perform these updates is to enable real-time replication, where changes in a source system are communicated to a target system as they are made in the source system, often using a log-based approach. While this can help ensure that the data in the target system is the same as the data in the source system, the real-time replication process can be complex and computationally expensive in both the source and target systems. A particular drawback of the log-based approach is that both the source and target systems usually need to be tightly integrated. For example, the target system is usually designed based on the implementation of the source system, so that appropriate logs can be obtained and analyzed. This often makes real-time replication difficult to implement between systems from different vendors.

[0007] Therefore, another type of replication that may be used is snapshot replication, in which all or selected data sources, such as specific tables identified for replication, are caused to periodically transmit their contents to the target system so that the target system may have a more up-to-date version of the source system's data. However, snapshot replication is not real-time replication, so over time the data in the target system may become outdated compared to the data in the source system. Furthermore, performing snapshot updates may be computationally expensive, since typically all of the content identified for replication in the source system is transmitted to the target system during a snapshot update. Thus, there is room for improvement. Summary of the Invention [Means for solving the problem]

[0008] This Summary is provided to introduce a set of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0009] The disclosed techniques and solutions can provide improved snapshot replication. Typically, an initial replica obtained using snapshot replication is updated periodically. However, if the data of the source data object has not changed compared to the data of the replicated data object, the update process may unnecessarily consume computing resources. The disclosed techniques perform a check to determine whether the snapshot replica is out of date before taking a new snapshot. The check may be performed upon manual request, upon the occurrence of a trigger, such as receiving a query that accesses the replicated data object, or according to a schedule. Information about the current and previous versions of the remote data object, such as a digest value of the content of the remote data object or a timestamp associated with the remote data object, may be compared to determine whether the replica is out of date.

[0010] In one aspect, the present disclosure provides a process for performing an update of a snapshot replica after determining that a current snapshot replica is out of date relative to a remote data object. A first instance of a first data object type is created at a local data source. A first set of data is received for the first instance of the first data object type from a remote data source. The first set of data is data extracted from a remote data object of the remote data source. The data is stored in the first instance of the first object type. The first instance of the first object type is determined to be out of date relative to the remote data object. A second set of data is received for the first instance of the first data object type. The second set of data is updated data of the remote data object. At least a portion of the first set of data is replaced in the first instance of the first object type with data of the second set of data.

[0011] The present disclosure also includes computing systems and tangible, non-transitory computer-readable storage media configured to perform or including instructions for performing the above-described methods. As described herein, various other features and advantages may be incorporated into the technology as desired. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an exemplary database system that may be used to implement aspects of the disclosed technology. [Diagram 2] FIG. 1 illustrates a computing environment in which a database system can access data from a remote computing system using data federation, including through virtual tables, or using local tables with data replicated from the remote computing system. [Diagram 3]FIG. 1 illustrates a prior art data replication technique. [Figure 4] 1 illustrates a diagram and flow diagram of a snapshot replication process in accordance with the present disclosure. [Diagram 5] 1 is a flow diagram of the disclosed process of executing a query that includes hints regarding preferred behavior when a snapshot copy is determined to be stale. [Figure 6] FIG. 2 illustrates a remote table and a replicated table, as well as data elements that may be used to determine whether the replicated table is up-to-date compared to the remote table. [Figure 7] 1 is a flow diagram of a process for performing an update of a snapshot replica after determining that a current snapshot replica is out of date compared to a remote data object. [Figure 8] FIG. 1 illustrates an example computing system in which some described embodiments may be implemented. [Figure 9] FIG. 1 illustrates an example cloud computing environment that may be used in conjunction with the technologies described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Example 1 - Overview It is becoming increasingly common for enterprises to store data in a variety of systems, including one or more local systems and one or more cloud-based systems. The systems can be of different types, storing data in different formats (e.g., a relational database versus a database that stores JSON documents) or using different database management systems (e.g., using software and / or hardware provided by different vendors). Even if the data is stored in the same format using software from the same vendor, there may be differences in what data is stored in a particular location and the schema used to store the data.

[0014] To help address these issues, various techniques have been used, including data replication and database federation. In a federated database environment, a request for a database operation, such as a query, may specify the source of a local database system or a "remote" database accessed using data federation. In some cases, both local and remote data sources may be specified in the same query, such as having a query that retrieves data from a local database table and data from a data source in a remote, federated database system.

[0015] Data replication provides a strategic advantage for the business by allowing the target system to access and utilize data originally present in the source system that supplements the information already natively stored in the target system itself. In this architecture, the source system essentially acts as the more authoritative or primary source of truth for the replicated data. This means that the target system can maintain its own data while also leveraging the additional information replicated from the source system to enhance the target system's data processing or analytical capabilities. However, any discrepancies or conflicts involving the replicated data are typically resolved in favor of the source system given its status as a more trusted source.

[0016] For various reasons, such as query execution, data replication may be a better solution than data federation in some scenarios. However, in many cases, data in a source system is not "static" as data can be added to, removed from, or changed in the source system. Therefore, rather than replicating data from the source system to the target system once, a process is implemented to update the data in the target system based on changes made in the source system.

[0017] One way to perform these updates is to enable real-time replication, where changes in a source system are communicated to a target system as they are made in the source system, often using a log-based approach. While this can help ensure that the data in the target system is the same as the data in the source system, the real-time replication process can be complex and computationally expensive in both the source and target systems. A particular drawback of the log-based approach is that both the source and target systems usually need to be tightly integrated. For example, the target system is usually designed based on the implementation of the source system, so that appropriate logs can be obtained and analyzed. This often makes real-time replication difficult to implement between systems from different vendors.

[0018] Therefore, another type of replication that may be used is snapshot replication, in which all or selected data sources, such as specific tables identified for replication, are caused to periodically transmit their contents to the target system so that the target system may have a more up-to-date version of the source system's data. However, snapshot replication is not real-time replication, so over time the data in the target system may become outdated compared to the data in the source system. Furthermore, performing snapshot updates may be computationally expensive, since typically all of the content identified for replication in the source system is transmitted to the target system during a snapshot update. Thus, there is room for improvement.

[0019] As a specific example of how a local database can access data from a remote data source, SAP SE of Walldorf, Germany's Smart Data Access (SDA) data virtualization technology allows data in "virtual tables" in a local database system to access data replicated to the local system from a remote system or to directly access data from a remote system through data federation. In some implementations, a given virtual table may be toggled between accessing data from a local replica and directly from the remote system. The disclosed technology may be used in SAP SDA, but may also be used in other scenarios involving snapshot (or "batch") replication from a remote system to a local system.

[0020] As discussed, snapshot replication may have less overhead than real-time replication, but depending on how often the data changes and how often or if updated snapshots are taken, the process of performing snapshot updates may still be expensive and the data may become stale. Typically, snapshot updates are either requested manually or according to a schedule. The problem with manually requesting snapshot updates is that it requires active action on the part of a user or computing process. Thus, for example, a user may need to consider whether data from a remote data source may be stale compared to the last time a snapshot was taken before running a query. However, the user may forget to perform this action, leading to query results that may have stale data. Or, the user may request an update and incur the resulting computing overhead even if the data in the source system has not changed compared to the current snapshot in the local system.

[0021] In other cases, the snapshot may be refreshed periodically, such as according to a schedule, but this may suffer from the same problems as manually triggering a snapshot update: updates may be performed before the data has changed or a "significant" number of changes have occurred, incurring unnecessary computational costs, or queries may be performed against stale data before the scheduled update has occurred.

[0022] The present disclosure provides techniques for improved snapshot replication. In particular, the disclosed techniques allow a current snapshot of replicated data in a local system to be compared to a current state of the target data in a remote system. Rather than directly comparing data from the remote and replicated systems, a summary of the data in the remote system may be compared to a summary of the data in the local system's replicate. If the comparison indicates that a change has occurred, the local system can obtain an updated replica from the remote system. Thus, snapshot updates are only performed when data changes in the remote system.

[0023] This disclosure provides specific examples of the disclosed techniques implemented in relational database systems, such as for use in processes involving local replicated database tables and remote database tables, but the disclosed techniques may be used with other types of data objects, such as storing data in formats such as JSON, XML, CSV, PARQUEST, BSON (binary JSON), APACHE KAFKA, APACHE CASSANDRA, or SAP BW objects.

[0024] In some cases, the remote object in which the data is stored and the replicated object are of the same type, while in other cases the remote object and the replicated object can be of different types. As described later in this disclosure, characteristics or properties of a first version of a remote data object (as reflected in the replicated object) can be compared to characteristics or properties of a second version of the remote data object (as reflected in the current version of the object on the remote system), and if the values ​​of the properties or properties differ, a change can be indicated and an update process can be triggered. If the value of the property or property of the version of the remote object used to generate or fill the replicated object is available, the property or property does not need to be calculated from the replicated object.

[0025] Various types of summary data may be used to determine whether the data maintained on the remote system has changed compared to the data of the snapshot replica on the local system. In one embodiment, a hash value of the snapshot replica may be calculated and compared to the hash value of the current version of the data on the remote system. In another embodiment, current table statistics for a table may be compared to table statistics associated with the snapshot replica. Other metadata about a particular data object, such as the date the table was last modified or information about the size of the table, may be used for comparison purposes.

[0026] The comparison may be initiated and performed according to a variety of criteria. In some cases, the comparison may occur when a query requesting data from a replica is received at the local system, or upon similar activities such as a request to compile or recompile a query plan. In another example, the comparison may occur according to a schedule.

[0027] The comparison process and triggering of the update process may be performed at the remote system or at the local system, depending on the implementation. When checking whether a replica is up-to-date upon a query request, the comparison process may be initiated by the local system. However, the local system may send a data value to the remote system, which then determines whether the replica is up-to-date, or the local system may request a value from the remote system (or send a value to the local system), which then determines whether the replica is up-to-date. For other types of triggers, such as determining whether a replica is up-to-date according to a schedule, the check may be initiated at the remote system or at the local system.

[0028] This disclosure generally proceeds to discuss determining whether a new snapshot replica should be obtained based on information about an entire data object, such as a relational database table. However, at least certain aspects of the disclosed technology may be applied at a more granular level. For example, rather than determining whether the contents of an entire table differ from the snapshot replica, it may be determined whether one or more particular columns of the snapshot replica have different properties than a corresponding one or more columns of the current version of the remote data source used for the replica. Similarly, rather than (or in addition to) looking at data about the entire table, metadata about the columns, such as modified dates or sizes, may be used in determining whether the snapshot replica should be updated.

[0029] The disclosed technology can thus provide many advantages. By checking whether a snapshot copy is likely to differ from the source data on the remote system, unnecessary data transmission and processing can be avoided compared to scenarios where a new snapshot is taken according to a schedule or trigger event. The comparison to determine changes and take a new snapshot can be automated, which can relieve a user from having to manually determine whether an updated snapshot should be taken and run an update process. Additionally, the disclosed technology provides flexibility in processing queries, such as allowing the querier to indicate what action should be taken if it is determined that a snapshot copy may be out of date.

[0030] Example 2 describes an exemplary database system that may be used to implement the disclosed technology. Example 3 provides an example of a virtual table that includes logical pointers that may be updated to point to different locations, including locations in federated data sources or local data sources (including local tables, or tables maintained in a cache). However, it should be understood that the disclosed technology does not require the use of such virtual tables. That is, the disclosed technology may be applied to any implementation of replicated tables that use snapshot replication to obtain data. Examples 4-8 describe in more detail the disclosed technology for performing change-aware snapshot replication.

[0031] Example 2 - Example Database Architecture 1 illustrates an exemplary database environment 100. The database environment 100 may include a client 104. Although a single client 104 is shown, the client 104 may represent multiple clients. The client or clients 104 may be an OLAP client, an OLTP client, or a combination thereof.

[0032] The clients 104 communicate with the database server 106. Through various sub-components, the database server 106 can process requests for database operations, such as requests to store, read, or manipulate data (i.e., CRUD operations). The session manager component 108 can be responsible for managing connections between the clients 104 and the database server 106, such as clients that communicate with the database server using a database programming interface, such as Java Database Connectivity (JDBC), Open Database Connectivity (ODBC), or Database Shared Library (DBSL). Typically, the session manager 108 can manage connections with multiple clients 104 simultaneously. The session manager 108 can perform functions such as creating new sessions for client requests, allocating client requests to existing sessions, and authenticating access to the database server 106. For each session, the session manager 108 can maintain a context that remembers a set of parameters related to the session, such as settings related to committing database transactions or transaction isolation levels (such as statement-level isolation or transaction-level isolation).

[0033] For other types of clients 104, such as web-based clients (such as those using the HTTP protocol or a similar transport protocol), the clients may interface with an application manager component 110. Although shown as a component of the database server 106, in other implementations the application manager 110 is external to the database server 106 but may communicate with the database server 106. The application manager 110 may initiate new database sessions with the database server 106 and perform other functions in a manner similar to the session manager 108.

[0034] The application manager 110 can determine the type of application making a request for a database operation and mediate the execution of the request at the database server 106, such as by invoking or executing procedure calls, generating query language statements, or translating data between formats usable by the client 104 and the database server 106. In certain examples, the application manager 110 receives requests for database operations from the client 104, but does not store information, such as state information, associated with the requests.

[0035] Once a connection is established between a client 104 and a database server 106, including through an application manager 110, execution of a client request is typically performed using a query language such as Structured Query Language (SQL). In executing a request, the session manager 108 and the application manager 110 may communicate with a query interface 112. The query interface 112 may be responsible for creating a connection with an appropriate execution component of the database server 106. The query interface 112 may also be responsible for determining whether the request is associated with a previously cached statement or stored procedure, and for invoking a stored procedure or for associating a previously cached statement with the request.

[0036] At least certain types of requests for database operations, such as query language statements to write or manipulate data, may be associated with a transaction context. In at least some implementations, each new session may be allocated a transaction. Transactions may be managed by a transaction manager component 114. The transaction manager component 114 may be responsible for operations such as coordinating transactions, managing transaction isolation, tracking ongoing and completed transactions, and managing transaction commit or rollback. In performing these operations, the transaction manager 114 may communicate with other components of the database server 106.

[0037] The query interface 112 may communicate with a query language processor 116, such as a structured query language processor. For example, the query interface 112 may forward query language statements or other database operation requests from the client 104 to the query language processor 116. The query language processor 116 may include a query language executor 120, such as an SQL executor, which may include a thread pool 124. Some requests or components of database operations may be executed directly by the query language processor 116. Other requests or components of database operations may be forwarded by the query language processor 116 to another component of the database server 106. For example, transaction control statements (such as commit or rollback operations) may be forwarded by the query language processor 116 to the transaction manager 114. In at least some cases, the query language processor 116 is responsible for executing operations that retrieve or manipulate data (e.g., SELECT, UPDATE, DELETE). Other types of operations, such as queries, may be sent by the query language processor 116 to other components of the database server 106. The query interface 112 and the session manager 108 can maintain and manage context information associated with requests for database operations. In a particular implementation, the query interface 112 can maintain and manage context information for requests received through the application manager 110.

[0038] When a connection is established between a client 104 and a database server 106 by a session manager 108 or an application manager 110, a client request, such as a query, may be allocated to a thread in a thread pool 124, such as using a query interface 112. In at least one implementation, a thread is associated with a context for performing a processing activity. The thread may be managed by an operating system of the database server 106, or by or in combination with another component of the database server. Generally, at any time, the thread pool 124 includes multiple threads. In at least some cases, the number of threads in the thread pool 124 may be dynamically adjusted, such as in response to a level of activity at the database server 106. Each thread in the thread pool 124 may be allocated to multiple different sessions, in certain aspects.

[0039] When a query is received, the session manager 108 or application manager 110 may determine whether an execution plan for the query already exists, such as in the plan cache 136. If a query execution plan exists, the cached execution plan may be retrieved and forwarded to the query language executor 120, such as using the query interface 112. For example, the query may be sent to an execution thread of the thread pool 124 determined by the session manager 108 or application manager 110. In a particular example, the query plan is implemented as an abstract data type.

[0040] If the query is not associated with an existing execution plan, the query may be parsed using query language parser 128. Query language parser 128 may, for example, check the query's query language statements to make sure they have correct syntax and ensure that the statements are otherwise valid. For example, query language parser 128 may check whether the tables and records mentioned in the query language statements are defined in database server 106.

[0041] The query may also be optimized using a query language optimizer 132. The query language optimizer 132 may manipulate elements of a query language statement to allow the query to be processed more efficiently. For example, the query language optimizer 132 may perform operations such as unnesting the query or determining an optimized execution order of various operations in the query, such as operations in a statement. After optimization, an execution plan may be generated or compiled for the query. In at least some cases, the execution plan may be cached, such as in a plan cache 136, and may be retrieved (such as by the session manager 108 or application manager 110) if the query is received again.

[0042] For purposes of this disclosure, one task that may be performed by the query language optimizer 132 is to determine where a database operation request, or part of it, should be executed. For example, a complex query may be submitted that retrieves data from multiple data sources. At least one of the data sources may be a virtual table, and the request may be executed on a replicated table on the database server 106, or the request may be sent by data federation to a remote system, and the results may be returned to the database server.

[0043] Once a query execution plan has been generated or received, the query language executor 120 may oversee the execution of the query execution plan. For example, the query language executor 120 may invoke the appropriate sub-components of the database server 106.

[0044] When executing a query, the query language executor 120 can invoke a query processor 140, which may include one or more query processing engines. The query processing engines may include, for example, an OLAP engine 142, a join engine 144, an attribute engine 146, or a calculation engine 148. The OLAP engine 142 may apply rules, for example, to create an optimized execution plan for an OLAP query. The join engine 144 may be used to implement relational operators typically for non-OLAP queries, such as join and aggregation operations. In a particular implementation, the attribute engine 146 may implement column data structures and access operations. For example, the attribute engine 146 may implement merge functions and query processing functions, such as scanning columns.

[0045] In certain circumstances, such as when a query includes complex or internally parallelized operations or sub-operations, the query executor 120 can send the operations or sub-operations of the query to a job executor component 154, which may include a thread pool 156. The execution plan of the query may include multiple plan operators. Each job execution thread of the job execution thread pool 156 may be allocated to an individual plan operator in certain implementations. The job executor component 154 may be used to execute at least some of the operators of the query in parallel. In some cases, the plan operators may be further divided and parallelized, such as having operations access different parts of the same table simultaneously. Using the job executor component 154 may increase the load on one or more processing units of the database server 106, but may improve the execution time of the query.

[0046] The query processing engine of the query processor 140 can access data stored in the database server 106. The data may be stored in a row-wise format in the row store 162 or in a column-wise format in the column store 164. In at least some cases, the data may be converted between the row-wise and column-wise formats. Certain operations performed by the query processor 140 may access or manipulate data in the row store 162, the column store 164, or, at least for certain types of operations (such as joins, merges, and subqueries), both the row store 162 and the column store 164. In at least some aspects, the row store 162 and the column store 164 may be maintained in main memory.

[0047] A persistence layer 168 may communicate with row store 162 and column store 164. Persistence layer 168 may be responsible for actions such as committing write transactions, storing redo log entries, rolling back transactions, and periodically writing data to storage to provide persisted data 172.

[0048] In executing a request for a database operation, such as a query or transaction, the database server 106 may need to access information stored elsewhere, such as another database server. The database server 106 may include a communications manager component 180 for managing such communications. The communications manager 180 may also mediate communications between the database server 106 and the client 104 or application manager 110 when the application manager is located external to the database server.

[0049] In some cases, database server 106 may be part of a distributed database system that includes multiple database servers. At least some of the database servers may include some or all of the components of database server 106. A database server of a database system may in some cases store multiple copies of data. For example, a table may be replicated to two or more multiple database servers. Additionally or alternatively, information in a database system may be distributed among multiple servers. For example, a first database server may hold a copy of a first table, and a second database server may hold a copy of a second table. In further implementations, information may be partitioned among database servers. For example, a first database server may hold a first portion of a first table, and a second database server may hold a second portion of the first table.

[0050] In performing a database operation request, database server 106 may need to access other database servers or other sources of information within the database system or in external systems, such as those in which remote data objects reside. Communications manager 180 may be used to mediate such communications. For example, communications manager 180 may receive and route requests for information from components of database server 106 (or from another database server), and receive and route responses.

[0051] The database server 106 may include components for coordinating data processing operations involving remote data sources. In particular, the database server 106 includes a data federation component 190 that at least partially processes requests to access data maintained in remote systems. In performing its functions, the data federation component 190 may include one or more adapters 192, which may include logic, configuration, or connection information that may be used to communicate with a remote system, such as to help generate virtual parameterized data objects or to obtain information for executing a request for data using the virtual parameterized data objects (e.g., to issue a request to a remote system for data to be accessed using a corresponding parameterized data object of the remote system). Examples of adapters include "connectors" implemented in technology available from SAP SE of Walldorf, Germany. Additionally, the disclosed technology may use technology underlying data federation technologies such as SAP SE's Smart Data Access (SDA) and Smart Data Integration (SDI).

[0052] Example 3 - An example virtual table with updatable logical pointers FIG. 2 illustrates a computing environment 200 in which the disclosed embodiments may be implemented. The basic computing environment 200 of FIG. 2 includes several features that may be common to different embodiments of the disclosed technology, including one or more applications 208 that may access a central computing system 210, which may be a cloud computing system. Although the central computing system 210 is illustrated as a monolithic / unitary system, it should be understood that the central computing system may include several computing systems that function together as a single system, particularly in a cloud environment. For example, the central computing system 210 may be implemented as multiple "nodes" including an anchor node and zero or more non-anchor nodes. The central computing system 210 may also be a more typical "distributed" database system including a master node and one or more worker nodes.

[0053] The central computing system 210 may do so by providing access to data stored in one or more remote database systems 212. Further, the remote database systems 212 may be accessed by one or more applications 214. In some cases, the applications 214 may also be applications 208. That is, some applications may only access (directly) data in the central computing system 210, some applications may only access data in the remote database systems 212, and other applications may access data in both the central computing system and the remote database systems.

[0054] The central computing device 210 may include a query processor 220. The query processor 220 may include multiple components, including a query optimizer 222 and a query executor 224. The query optimizer 222 may be responsible for determining a query execution plan 226 for queries executed using the central computing system 210. The query plan 226 generated by the query optimizer 222 may include both a logical plan indicating, for example, the order of operations (e.g., joins, projections) to be performed in the query, and a physical plan for implementing such operations. Once created by the query optimizer 222, the query plan 226 may be executed by the query executor 224. The query plan 226 may be stored in a query plan cache 228 as a cached query plan 230. When a query is submitted again for execution, the query processor 220 may determine whether a cached query plan 230 exists for the query. If so, the cached query plan 230 may be executed by the query executor 224. If not, a query plan 226 is generated by the query optimizer 222. In some cases, the cached query plan 230 may be invalidated, such as when changes are made to the database schema or at least components of the database schema (e.g., tables or views) used by the query.

[0055] A data dictionary 234 (which may also be referred to as a system catalog) may maintain one or more database schemas for the central computing system 210. In some cases, the central computing system 210 may implement a multi-tenant environment, where different tenants may have different database schemas. In at least some cases, at least some database schema elements may be shared by multiple database schemas.

[0056] The data dictionary 234 may contain definitions (or schemas) of different types of database objects, such as the schema of a table or a view. The following discussion refers to tables for ease of explanation, but it should be understood that it is applicable to other types of database objects, particularly database objects that are associated with retrievable data, such as materialized views. The table schema may include information such as the name of the table, the number of attributes (or columns or fields) in the table, the names of the attributes, the data types of the attributes, the order in which the attributes should be displayed, primary key values, foreign keys, associations to other database objects, partition information, or replication information.

[0057] Table schemas maintained by data dictionary 234 may include local table schema 236, which may represent tables maintained primarily on central computing system 210. Data dictionary 234 may include replicated table schema 238, which may represent tables for which at least a portion of the table data is stored on central computing system 210 (or managed primarily by the database management system of the central computing system, even if stored outside the central computing system, such as in a data lake or another cloud service). Tables that have data associated with replicated table schema 238 typically have their data updated periodically from a source table, such as remote table 244 in data store 242 of remote database system 212.

[0058] Replication may be accomplished using one or both of a replication service 246 at the remote database system 212 or a replication service 248 at the central computing system 210. In a particular example, the replication service may be a Smart Data Integration (SDI) service, SAP Landscape Transformation Replication Server, SAP Data Services, SAP Replication Server, SAP Event Stream Processor, or SAP HANA Direct Extractor Connection, all from SAP SE, Walldorf, Germany. As discussed, replication may use real-time replication techniques or may use snapshot / batch replication.

[0059] In some cases, data in remote database systems 212 may be accessed by central computing system 210 without replicating data from the remote database systems, such as using federation techniques. Data dictionary 234 may store virtual table schemas 252 of virtual tables that are mapped to remote tables, such as remote tables 244 in remote database systems 212. Data in remote tables 244 may be accessed using a federation service 256, such as using the Smart Data Access protocol of SAP SE of Walldorf, Germany. Federation service 256 may be responsible for translating query operations into a format that can be processed by the appropriate remote database system 212, sending query operations to the remote database systems, receiving query results, and providing the query results to query executor 224.

[0060] The data dictionary 234 may include an updatable virtual table schema 260 with an updatable logical pointer 262. The updated virtual table schema 260 may optionally be associated with status information 264. The table pointer 262 may be a logical pointer used to identify which table should be accessed for data of the corresponding virtual table schema 260. For example, depending on the state of the table pointer 262, the table pointer may point to a remote table 244 in the remote database system 212 or a replicated table 266 (which may be generated from the remote table 244) in a data store 268 of the central computing system 210. The data store 268 may also store data of a local table 270, which may be defined by the local table schema 236.

[0061] Table pointer 262 may be changed between remote table 244 and replicated table 266. In some cases, a user may manually change the table pointed to by table pointer 262. In other cases, table pointer 262 may be changed automatically, such as in response to detection of a defined condition.

[0062] The status information 264 may include an indicator that identifies the virtual table schema 260 as being associated with a remote table 244 or a replicated table 266. The status information 264 may also include information about the replication status of the replicated table 266. For example, when a request is made to change the table pointer 262 to point to the replicated table 266, it may take some time before the replicated table is ready for use. The status information 264 may include whether the replication process has been initiated, whether it has been completed, or a progress status of the generation of the replicated table 266.

[0063] Changes to the updatable virtual table schema 260 and management of the replicated tables 266 associated with the virtual table schema may be managed by a virtual table service 272. Although shown as a separate component of the central computing system 210, the virtual table service 272 may be incorporated into other components of the central computing system 210, such as the query processor 220 or the data dictionary 234.

[0064] When a query is executed, it is processed by the query processor 220, including executing the query using the query executor 224 to retrieve data from one or both of the data stores 242 of the remote database systems 212 or the data stores 268 of the central computing system 210. The query results may be returned to the application 208. The query results may also be cached, such as in the cache 278 of the central computing system 210. The cached results may be represented as cached views 280 (e.g., materialized query results).

[0065] An application 214 can access data in a remote database system 212, such as through a session manager 286. An application 214 can modify a remote table 244. When a table pointer 262 in an updatable virtual table schema 260 references a remote table 244, changes made by the application 214 are reflected in the remote table. When a table pointer 262 references a replicated table 266, changes made by the application 214 can be reflected in the replicated table using a replication service 246 or a replication service 248.

[0066] Example 4 - Illustrative Replication Techniques 3 illustrates a prior art technique for replicating data from a remote system 304 to a local system 308. The remote system 304 includes a remote table 312. Data from the remote table 312 is replicated to a replicated table 316 on the local system 308.

[0067] Data from the remote table 312 may be replicated to the replicated table 316 in several ways. A real-time replication process 320 sends incremental data changes made to the remote table 312 to the replicated table 316 as those changes occur, such as using techniques that may include change data capture or trigger-based replication. As discussed in Example 1, real-time replication is advantageous in helping ensure that the replicated table 316 is up-to-date with respect to the remote table 312, but can be complex to implement and require significant computing resources.

[0068] 3 also illustrates a snapshot replication process 330. In the snapshot replication process 330, a snapshot 334 of the remote table 312 is taken periodically. Typically, the snapshot 334 includes all of the data in the remote table 312 at the time the snapshot was taken. The snapshot data 334 may be sent to the local system 308. The current replicated table 316 may be replaced with the snapshot data 334.

[0069] As shown, snapshots are typically taken periodically, and thus process 330 may be referred to as periodic replication. Because the replication is periodic, a new snapshot 334 is taken after a delay period 340. The delay period 340 may be a delay depending on a schedule, an event, or the delay between manual update requests. As mentioned in Example 1, snapshot replication may be easier to implement and less resource intensive than real-time replication, but the existence of the delay period 340 may cause queries at the local system 308 to retrieve outdated data compared to the current version of the remote table 312.

[0070] Example 5 - Exemplary snapshot replication with update checking Figure 4 illustrates a snapshot replication process according to the present disclosure. Similar to Figure 3, Figure 4 includes a remote system 404 having a remote table 408 and a local system 412 having a local replicated table 416 that is a snapshot of the remote table 408 at a particular point in time. A snapshot replication process 420 is used to replicate data from the remote table 408 to the replicated table 416.

[0071] 4 shows details of the snapshot replication process 420. The snapshot replication process 420 may begin at 424 where it is determined whether a particular event has occurred that will cause the process to proceed further. Examples of events can be a request, such as from a user or application process, to determine whether the local table 416 should be updated with a newer snapshot of the remote table 408, a particular instance of a particular date / time trigger, or a scheduled event, such as the passage of time according to a schedule (such as to determine every hour whether a snapshot should be updated), the receipt of a query that includes the local table 416, or an indication that the data in the remote table 408 has changed.

[0072] If it is determined at 424 that the event did not occur, the process 420 may return to 424 until it is determined that the event did occur. If it is determined at 424 that the event did occur, the process 420 may proceed to 428. At 428, it is determined whether the data in the replicated table 416 still matches the data in the remote table 408. Performing a comparison of the actual data in both the remote table 408 and the replicated table 416 can be expensive in terms of resources required to copy the data, transmit the data, and compare the data, and therefore, it may be more efficient to simply perform a snapshot update. Thus, the comparison at 428 typically takes into account information about the remote table 408 and the replicated table 416, but does not compare all of the data in both tables. Details of what types of data may be compared at 428 are discussed in Example 7.

[0073] If it is determined at 428 that the replicated table 416 matches the remote table 408 based at least on the comparison criteria, the process 420 may return to 424. If it is determined at 428 that the replicated table 416 does not match the remote table 408, the contents of the tables may be synchronized at 432. Synchronizing the contents at 432 may include replacing the data in the replicated table 416 with data from a new snapshot of the contents of the remote table 408.

[0074] Optionally, at 440, the metadata of the remote table 408, the metadata of the replicated table 416, or the metadata of both tables may be updated. Updating the metadata at 440 may include updating values ​​used for the comparison at 428. For example, a new hash value, last modification date, table size, or set of statistics for the replicated table 416 (corresponding to the metadata of the remote table 408 at the time of the snapshot) may be stored.

[0075] Example 6 - Example query processing with update checks and optional customized processing As described in Example 1, in some cases, a query request that includes access to a local replica table may trigger a check to determine whether the snapshot replica is up-to-date according to certain criteria, such as corresponding to event 424 of process 420 of Figure 4. Typically, if no changes are detected, the query may be performed using the current snapshot replica. However, if the snapshot replica is determined to be out of date, a different process may be implemented.

[0076] If the snapshot copy is determined to be stale, the query may be executed using the current snapshot copy, even though some of the data accessed by the query may be stale. As another option, if the snapshot copy is determined to be stale, the query may be rejected or queued until the snapshot update is complete. A given behavior may be "hard-coded" into the system, or a default behavior may be specified, but the default behavior may be selectively overridden.

[0077] In certain examples, a particular query may directly specify a desired behavior. That is, a query may include "hints," such as keywords, that indicate to a query executor or query optimizer what behavior is desired. A query hint may indicate that a query should be executed against a current version of a snapshot replica without delay, even though the data may be out of date compared to the remote table. A query hint may instead indicate that query processing should be deferred until an updated snapshot is obtained. In a further example, a query hint may be used to indicate that a query should be executed using a remote table instead of a snapshot replica, such as using a data federation technique. In yet another example, a query hint may specify that if the snapshot is not up to date, a failure message should be returned.

[0078] 5 illustrates an example process 520 in which a query executor 514 executes a query 504 based on a default behavior or using a non-default behavior if a query hint is given. Although described as being performed by the query executor, at least some operations of the process 520 may be performed by a query optimizer.

[0079] At 524, a query is received. At 528, it is determined that the query accesses a replicated table. At 532, a check is performed to determine if the replicated table is up-to-date compared to the remote table. If so, the query is executed using the current replicated table at 536 and process 520 can end at 540.

[0080] If the replicated table is determined to be out of date at 532, a check is performed at 544 to determine whether the query 504 contains a hint 506 indicating that the query should be run against the current version of the replicated table, rather than the default behavior of waiting until an updated snapshot is taken. If the query 504 is determined to contain the hint 506 at 544, the query is processed at 548 using the existing replica, and the process 520 can end at 540.

[0081] If it is determined at 544 that the query 504 does not include hint 506, then it is determined at 552 whether the query includes hint 508 indicating that the query should be executed against a remote table if the local replicated table is out of date. If hint 508 is present, the query is executed using the remote table at 556 and process 520 can end at 540.

[0082] If it is determined at 552 that the query 504 does not include the hint 508, the query may be queued at 560 until the local replication table is updated. Once the local replication table is updated, the query 504 may be executed at 564 and the process 520 may end at 540.

[0083] It should be noted that process 520 is only one example of how the disclosed technology may be used. In other implementations, a check may be performed for only hint 506 or hint 508, and the database system may support only hint 506 or hint 508. Similarly, in process 520, the default behavior is to postpone execution of the query until an updated snapshot is taken for the replicated table. However, a process may have a default behavior of executing the query using the current, stale replica, and a hint may be used to determine whether execution should be deferred until an update, or whether federated query execution should be used using a remote table. Another process may have a default behavior of using federated query execution using a remote table if the local replicated table is stale, and a hint may be used to determine whether execution should instead occur on the local system after an update snapshot is received, or whether the query should instead be executed on the local system using the current, but stale, replicated table.

[0084] Example 7 - Example elements for determining if two data objects are different or for triggering an update analysis process Figure 6 illustrates various table properties that may be used in determining whether a replicated table is up to date compared to a remote table whose data is replicated to the replicated table. Figure 6 includes a remote system 604 having a remote table 606 and a local system 608 having a replicated table 610 that corresponds to a snapshot of the remote table.

[0085] The remote table 606 is shown as having content 620. The content 620 may include one or more semantic attributes 624 and one or more synchronization attributes 626. The semantic attributes 624 provide data "about" a particular entity represented as an entry in the remote table 606. For example, an entry in the remote table 606 might correspond to an employee, a customer, or a sales transaction. In the case of an employee, the semantic attributes 624 might include attributes such as name, employee identification number, address, title, or payrate.

[0086] The sync attribute 626 is an attribute that is specifically used or usable to determine whether the data in the remote table 606 differs from the data in the replicated table 610. The sync attribute 626 may be, but is not necessarily, a semantic attribute 624. A particular example of a sync attribute 626 is a timestamp, such as a timestamp indicating when a particular row in the table was last added, deleted, or modified. The contents 640 of the replicated table 610 may include semantic attributes 644 and sync attributes 646 that correspond to the semantic attributes 624 and 626, but the values ​​of the attributes may differ between the tables 606, 610 if the replicated table is older than the remote table. The replicated table 610 may include a last modified date as a sync attribute 644, with a value corresponding to the last modified date of the remote table 606 as of the most recent snapshot. Thus, determining whether the replicated table 610 is out of date may be performed by querying the remote table 606 to determine whether the remote table contains a record having a value for the timestamp synchronization attribute 626 that is later than the most recent value reflected in the timestamp synchronization attribute 646 of the replicated table 610.

[0087] Rather than having a "last modified" date as a synchronization attribute 626, an overall last modified date 632a of the remote table 606 may be associated with the remote table and stored in metadata 630 having metadata elements 632 (shown as 632a-632g). The replicated table 608 may include metadata 650 having metadata elements 652 (shown as metadata elements 652a-652g) and may include the last modified date 652a associated with the remote table 606 in the latest snapshot. When a request is made to check if the remote table 606 has changed compared to the replicated table 610, the metadata 630 may be queried to determine if the last modified date 632a is the same as the last modified date 652a of the metadata 650. If the values ​​are not the same, an updated replica may be obtained.

[0088] As mentioned above, in some scenarios, determining whether a new snapshot should be taken for the replicated table 610 may be performed at a granularity less than the entire remote table 606. For example, changes to some semantic attributes 624 of the remote table 606 may be more noteworthy than other semantic attributes 624. In this case, it may be desirable to update the snapshot when a semantic attribute 624 of particular interest has changed compared to a corresponding semantic attribute 644 of the replicated table 610. Thus, the metadata 630 may include a last modification date 632b of a particular semantic attribute 624 (such as a particular column of the remote table 606), and the metadata 650 of the replicated table 610 may include a last modification date 652b of a corresponding semantic attribute 646. A determination of whether the replicated table 610 is up to date may then be performed by determining whether the last modification date 632b of a given semantic attribute 624 is the same as the last modification date 652b of the corresponding semantic attribute 644.

[0089] Table statistics 632c of the remote table 606 reflected in the metadata 630 can also be used to determine when the remote table may have different content than the content of the replicated table 610. While the metadata 630 stores the current statistics, the metadata 650 of the replicated table 610 includes statistics 652c that were copied from the metadata 630 when the last snapshot update occurred. Although any suitable type of table statistics 632c may be used, examples of table statistics include the following: Table size: The size of the table in memory, which may include memory for the table data and memory used for structural or system management such as indexes. Number of records: The number of records (rows) in the table. The average record length can also be tracked. Column statistics: Information about the data stored in each column (attribute) of a table, such as the number of distinct values, maximum and minimum values, the most common value, the number of nulls, and the average data size. Partition information: Information about how the table is partitioned, which may include the number of partitions, the number of rows or columns in each partition, or the overall size (memory or disk) of each partition. Index statistics: Information such as the index identifier, the index size, and the number of leaf nodes in the index. Histogram: Information about the statistical distribution of column values. Size on disk / clustering information: Information about the size on disk of a particular table, and which rows with similar column values ​​are stored near each other on disk.

[0090] Note that the particular table statistics 632c may depend on the remote system 604. For example, information about memory usage, particularly for indexes or system management, may be specific to the implementation of the remote system 604, and even if the data in the replicated table 610 is identical to the data in the remote table 606, the local system 608 may not use the same memory usage. However, this does not prevent the statistics 632c from being used to determine whether the replicated table 610 may be out of date, since the comparison is between two states of the remote table 606 at the remote system 604, and not between the actual statistics of the replicated table 610 and the statistics of the remote table 606. In other words, such metadata 650 corresponds to the metadata 630 at the time of the snapshot.

[0091] The contents of the remote table 606 may be used in other ways to compare the states of the tables. For example, the contents of the remote table 606 (i.e., the actual values ​​stored in the remote table according to the definition of the remote table) may be provided as input to a hash algorithm to generate a hash value 632d. This hash value 632d may be provided to the local system 608 and stored in the metadata 650 as a hash value 652d. When determining whether the replicated table 610 has the same contents as the remote table 606, the current hash value 632d of the remote table may be compared to the hash value 652d of the most recent snapshot replica used for the replicated table.

[0092] The current hash value 632 of the remote table 606 may be determined (calculated) at various times, depending on the discretion of the particular implementation. For example, the hash value 632d may be determined in response to a request from the local system 608, such as as part of a process to check whether an updated snapshot copy should be taken. In other cases, the hash value 632d may be updated periodically at the remote system 604, including when changes are made to the remote table 606, which may entail significant overhead for tables with frequently changing data. On the other hand, periodic updates of the hash value 632d may be sufficient, although this may create a risk of a "false negative" indication that the copy table 610 does not differ from the remote table 606, when the remote table has in fact changed but an updated hash value has not yet been calculated.

[0093] The discussion uses examples of hash algorithms and hash functions, although similar types of techniques may be used, including less computationally expensive techniques, such as using checksums. As with hash values, in at least some cases, a checksum may be calculated using the entire contents of the remote table 606. Hash values ​​and checksums are examples of a broader category of digests or digest values, and other types of digests may also be used with the disclosed techniques. Additionally, hash values ​​and checksums may be calculated over the contents of a table as represented by a table, but may optionally be calculated in other ways, such as over a binary representation of the contents of the table.

[0094] Either or both of the metadata 630 or the metadata 650 may include a synchronization schedule 632e, 652e that may be used by the synchronization logic 670 of the remote system 604 or the synchronization logic 672 of the local system 608, respectively. More typically, the metadata 650 includes a synchronization schedule 652e, and a check to determine whether an updated snapshot copy should be taken is performed according to the schedule. In another implementation, the remote system 604 may periodically send an updated snapshot copy to the local system 608, such as according to the schedule 632e of the metadata 630. The schedule may be expressed in various ways, such as specifying that a check for whether a new snapshot copy should be taken on a particular date / time should be performed according to a regular schedule (such as hourly or daily), or performing a check within a specified window, such as using a "before" date or a "after" date or both. Synchronization (or more specifically, a check to determine if a synchronization should be performed) can also be performed on demand, including based on the last synchronization date 632f, 652f of the metadata 630, 650. For example, the command can be "synchronize if LAST_SYNC_DATE > 1 day."

[0095] The metadata 630 or metadata 650 may also include triggers 632g, 652g. In this case, "trigger" does not refer to a trigger-based database process such as a trigger-based update or replication. Rather, "trigger" refers to some other event that causes a determination to be made whether the replicated table 610 is consistent with the remote table 606. A synchronization schedule, or performing a check when a query that includes an action on the replicated table 610 is received, may be considered examples of triggers 632g, 652g, but they have been separated, enumerated, and described in this discussion. Other examples of triggers 632g, 652g may be a threshold number of changes being made to the remote table 606, or a threshold number of queries accessing the replicated table 610 being received. Such triggers 632g, 652g can introduce a dynamic aspect to determining whether a replication update should be performed, as compared to a static schedule, because update checks may occur more frequently if there are more updates to the remote table 606 (indicating that the replication table 610 may be significantly more out of date) or if there is more activity on the replication table (indicating that the replication table is of higher interest and therefore may be prioritized for update checks compared to "cold" data).

[0096] Although this specification, including this Example 7, generally describes sending a copy of the entire contents of the remote table 606 when the contents are determined to differ from the replicated table 610, in other cases, a portion of the data may be sent, such as data added since the last replicated snapshot was taken. As a specific example, a table may be designated as append only. For these types of data, only rows added after the last replicated snapshot date need to be sent to the local system 608.

[0097] Example 8 - Example behavior FIG. 7 is a flow diagram of a process 700 for performing an update of a snapshot replica after determining that a current snapshot replica is out of date relative to a remote data object. At 705, a first instance of a first data object type is created at a local data source. At 710, a first set of data is received for the first instance of the first data object type from a remote data source. The first set of data is data extracted from a remote data object at the remote data source. At 715, the data is stored in the first instance of the first object type. At 720, it is determined that the first instance of the first object type is out of date relative to the remote data object. At 725, a second set of data is received for the first instance of the first data object type. The second set of data is updated data of the remote data object. At least a portion of the first set of data is replaced in the first instance of the first object type with data of the second set of data at 730.

[0098] Example 9 - Computing System 8 illustrates a generalized example of a suitable computing system 800 on which the described invention may be implemented. The computing system 800 is not intended to suggest any limitation as to the scope of use or functionality of the present disclosure, as the invention may be implemented in a wide variety of general-purpose or special-purpose computing systems.

[0099] Referring to FIG. 8, a computing system 800 includes one or more processing units 810, 815 and memory 820, 825. In FIG. 8, this basic configuration 830 is included within the dashed line. The processing units 810, 815 execute computer-executable instructions, such as for implementing the database environment and related methods described in Examples 1-8. The processing units can be general-purpose central processing units (CPUs), processors in application specific integrated circuits (ASICs), or any other type of processor. In a multiprocessing system, multiple processing units execute computer-executable instructions to increase processing power. For example, FIG. 8 shows a central processing unit 810 and a graphics processing unit or auxiliary processing unit 815. The tangible memory 820, 825 can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.) accessible by the processing units 810, 815, or some combination of the two. The memories 820, 825 store software 880 in the form of computer-executable instructions suitable for execution by the processing units 810, 815 that implement one or more inventions described herein.

[0100] Computing system 800 may have additional features. For example, computing system 800 includes storage 840, one or more input devices 850, one or more output devices 860, and one or more communication connections 870. An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of computing system 800. Generally, operating system software (not shown) provides an operating environment for other software executing in computing system 800 and coordinates the activities of the components of computing system 800.

[0101] The tangible storage 840 may be removable or non-removable and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within computing system 800. The storage 840 stores instructions for the software 880 that implements one or more inventions described herein.

[0102] The input device 850 may be a keyboard, a mouse, a touch input device such as a pen or trackball, a voice input device, a scanning device, or another device that provides input to the computing system 800. The output device 860 may be a display, a printer, speakers, a CD writer, or another device that provides output from the computing system 800.

[0103] The communications connection(s) 870 enable communication over a communications medium with another computing entity, such as another database server. The communications medium carries information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the communications medium may use an electrical, optical, RF, or other carrier.

[0104] The invention may be described in the broad context of computer-executable instructions, such as those included in program modules, being executed on target real or virtual processors in a computing system. Generally, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided between program modules as desired in various embodiments. The computer-executable instructions of the program modules may be executed within a local or distributed computing system.

[0105] The terms "system" and "device" are used interchangeably herein. Neither term implies any limitation on the type of computing system or computing device unless the context clearly indicates otherwise. In general, a computing system or computing device can be local or distributed and can include any combination of dedicated and / or general-purpose hardware with software that implements the functions described herein.

[0106] For purposes of presentation, the detailed description uses terms such as "determine" or "use" to describe computer operations in a computing system. These terms are high-level abstractions of operations performed by a computer and should not be confused with acts performed by a human. The actual computer operations corresponding to these terms will vary depending on the implementation.

[0107] Example 10 - Cloud Computing Environment 9 illustrates an example cloud computing environment 900 in which the described technologies may be implemented. The cloud computing environment 900 includes a cloud computing service 910. The cloud computing service 910 may include various types of cloud computing resources, such as computer servers, data storage repositories, networking resources, etc. The cloud computing service 910 can be centrally located (e.g., provided by a company or organization's data center) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers, and / or located in different cities or countries). Cloud computing service 910 is utilized by various types of computing devices (e.g., client computing devices), such as computing devices 920, 922, and 924. For example, the computing devices (e.g., 920, 922, and 924) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smartphones), or other types of computing devices. For example, the computing devices (e.g., 920, 922, and 924) can utilize cloud computing service 910 to perform computing operations (e.g., data processing, data storage, etc.).

[0108] Example 11 - Implementation Although some operations of the disclosed methods are described in a particular sequential order for convenience of presentation, it should be understood that such methods of description encompass permutations, unless a particular order is required by specific language set forth herein. For example, operations described in sequence may in some cases be permuted or performed simultaneously. Moreover, for simplicity, the accompanying figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods.

[0109] Any of the disclosed methods may be implemented as computer-executable instructions or computer program products stored on one or more computer-readable storage media, such as tangible non-transitory computer-readable storage media, and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device that includes computing hardware). A tangible computer-readable storage medium is any available tangible medium that can be accessed within a computing environment (e.g., one or more optical media disks, such as DVDs or CDs, a volatile memory component (such as DRAM or SRAM), or a non-volatile memory component (such as flash memory or hard drive). As an example, referring to FIG. 8, a computer-readable storage medium includes memory 820 and 825 and storage device 840. The term computer-readable storage medium does not include signals and carrier waves. Furthermore, the term computer-readable storage medium does not include communication connections (e.g., 870).

[0110] Any of the computer-executable instructions for carrying out the disclosed techniques, and any data created and used in implementing the disclosed embodiments, may be stored on one or more computer-readable storage media. The computer-executable instructions may be, for example, a dedicated software application, or part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment using one or more networked computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network).

[0111] For clarity, only certain selected aspects of the software-based implementation are described. Other details well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C++, Java, Perl, JavaScript, Python, Ruby, ABAP, Structured Query Language, Adobe Flash, or any other suitable programming language, or in some examples, a markup language such as html or XML, or a combination of a suitable programming language and a markup language. Similarly, the disclosed technology is not limited to any particular computer or hardware type. Specific details of suitable computers and hardware are well known and need not be detailed in this disclosure.

[0112] Additionally, any of the software-based embodiments (including, e.g., computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or accessed remotely through suitable communication means including, for example, the Internet, the World Wide Web, an intranet, a software application, a cable (including fiber optic cable), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication means.

[0113] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, both alone and in various combinations and subcombinations with one another. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, and the disclosed embodiments do not require that any one or more particular advantages exist or problems be solved.

[0114] The technology of any example may be combined with the technology described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technology may be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be taken as limitations on the scope of the disclosed technology. Rather, the scope of the disclosed technology includes that encompassed by the scope and spirit of the following claims. [Explanation of symbols]

[0115] 100 Database Environment 104 Client 106 Database Server 108 Session Manager Components 110 Application Manager Component 112 Query Interface 114 Transaction Manager Components 116 Query Language Processor 120 Query Language Executor 124 thread pool 128 Query Language Parser 132 Query Language Optimizer 136 Plan Cache 140 Query Processor 142 OLAP Engine 144 Combination Engine 146 Attribute Engine 148 Calculation Engine 154 Job Executor Component 156 Thread Pool 162 row store 164 Column Store 168 Persistence Layer 172 Persistent Data 180 Communications Manager Component 190 Data Federation Components 192 Adapter 200 Computing Environment 208 Applications 210 Central Computing System 212 Remote Database System 214 Applications 220 Query Processor 222 Query Optimizer 224 Query Executor 226 Query Execution Plan 228 Query Plan Cache 230 Cached Query Plans 234 Data Dictionary 236 Local Table Schema 238 Replication Table Schema 242 Datastore 244 Remote Table 246 Replication Service 248 Replication Service 252 Virtual Table Schema 256 Federation Services 260 Updateable Virtual Table Schema 262 Logical Pointers, Table Pointers 264 Status Information 266 Duplicate Table 268 Datastore 270 Local Table 272 Virtual Table Service 278 Cache 280 Cached Views 286 Session Manager 304 Remote System 308 Local System 312 Remote Table 316 Duplicate Table 320 Real-time Replication Process 330 Snapshot Replication Process 340 Delay Period 334 Snapshot, Snapshot Data 404 Remote System 408 Remote Table 412 Local System 416 Duplicate Table 420 Snapshot Replication Process 504 Query 506 Tips 508 Tips 514 Query Executor 520 Process 604 Remote System 606 Remote Table 608 Local System 610 Duplicate Table 620 Contents 624 Semantic Attributes 626 Synchronization attribute 630 Metadata 632 Metadata Elements 632a Overall Last Modified Date 632b Last modified date 632c Table Statistics 632d hash value 632e Sync Schedule 632f Last Sync Date 632g Trigger 640 Contents 644 Semantic Attributes 646 Synchronization attribute 650 Metadata 652 Metadata Elements 652a Last modified date 652b Last Modified 652c statistics 652d hash value 652e Sync Schedule 652f Last Sync Date 652g Trigger 670 Synchronous Logic 672 Synchronous Logic 700 processes 800 Computing Systems 810 Processing unit, central processing unit 815 Processing units, graphics processing units, auxiliary processing units 820 Memory 825 Memory 830 Basic configuration 840 Storage device 850 Input Devices 860 Output Device 870 Communication Connections 880 Software 900 Cloud Computing Environment 910 Cloud Computing Services 920 Computing Devices 922 Computing Devices 924 Computing Devices

Claims

1. 1. A computing system comprising: At least one memory; one or more hardware processor units coupled to the at least one memory; one or more computer-readable storage media that store computer-executable instructions that, when executed, cause the computing system to perform operations; and wherein the operation comprises: Creating a first instance of a first data object type in a local data source; receiving, from a remote data source, a first set of data for the first instance of the first data object type, the first set of data being extracted from a remote data object of the remote data source; storing the first set of data in the first instance of the first data object type; determining that the first instance of the first data object type is out of date relative to the remote data object; receiving a second set of data for the first instance of the first data object type, the second set of data being updated data for the remote data object; replacing at least a portion of the first set of data with data of the second set of data within the first instance of the first data object type; A computing system comprising:

2. The operation, Receiving a query; determining that the query accesses the first instance of the first data object type; initiating the determining based at least in part on determining that the query accesses the first instance of the first data object type; and The computing system of claim 1 further comprising:

3. The operation, The computing system of claim 2 , further comprising postponing execution of the query until the replacing is completed.

4. The computing system of claim 3 , wherein the deferring is performed in response to a command in the query.

5. The operation, The computing system of claim 2 , further comprising executing the query prior to the replacing.

6. The computing system of claim 5 , wherein executing the query prior to the replacing is performed in response to a command in the query.

7. The operation, The computing system of claim 2 , further comprising: executing at least a portion of the query against the remote data object to access the first instance of the first data object type.

8. 8. The computing system of claim 7, wherein executing at least a portion of the query to access the first instance of the first data object type on the remote data object is performed in response to a command in the query.

9. 2. The computing system of claim 1, wherein determining that the first instance of the first data object type is stale comprises comparing a digest value generated from content of the first set of data to a digest value generated from content of the remote data object and maintained in current metadata of the remote data object.

10. 2. The computing system of claim 1, wherein determining that the first instance of the first data object type is stale comprises comparing statistics maintained for the first instance of the first data object type with statistics maintained in current metadata of the remote data object.

11. 2. The computing system of claim 1, wherein determining that the first instance of the first data object type is stale comprises comparing a last modification date maintained for the first instance of the first data object type with a last modification date maintained in current metadata of the remote data object.

12. 2. The computing system of claim 1, wherein determining that the first instance of the first data object type is stale comprises comparing a size maintained for the first instance of the first data object type to a size maintained in current metadata of the remote data object.

13. 2. The computing system of claim 1 , wherein the first data object type includes an identifier indicating whether a query accessing an instance of the first data object type should be performed using the instance of the first data object type against the local data source or against the remote data object.

14. The computing system of claim 1 , wherein the determining is initiated based on the occurrence of a trigger.

15. The computing system of claim 1 , wherein the determining is initiated based on a synchronization schedule.

16. The computing system of claim 1 , wherein the remote data object and the first data object type share a common set of semantic attributes.

17. The computing system of claim 1 , wherein the remote data object and the first data object type are relational database tables.

18. The computing system of claim 1 , wherein the determining is performed at the remote data source.

19. 1. A method implemented in a computing system including at least one hardware processor and at least one memory coupled to the at least one hardware processor, comprising: creating a first instance of a first data object type in a local data source; receiving a first set of data for the first instance of the first data object type from a remote data source, the first set of data being extracted from a remote data object of the remote data source; storing the first set of data in the first instance of the first data object type; determining that the first instance of the first data object type is out of date relative to the remote data object; receiving a second set of data for the first instance of the first data object type, the second set of data being updated data for the remote data object; replacing at least a portion of the first set of data with data of the second set of data within the first instance of the first data object type; The method includes:

20. computer-executable instructions that, when executed by a computing system including at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to create a first instance of a first data object type in a local data source; computer-executable instructions that, when executed by the computing system, cause the computing system to receive a first set of data for the first instance of the first data object type from a remote data source, the first set of data being extracted from a remote data object of the remote data source; computer-executable instructions that, when executed by the computing system, cause the computing system to store the first set of data in the first instance of the first data object type; computer-executable instructions that, when executed by the computing system, cause the computing system to determine that the first instance of the first data object type is out of date relative to the remote data object; computer-executable instructions that, when executed by the computing system, cause the computing system to receive a second set of data for the first instance of the first data object type, the second set of data being updated data for the remote data object; computer-executable instructions that, when executed by the computing system, cause the computing system to replace at least a portion of the first set of data with data of the second set of data within the first instance of the first data object type; [0023] In one or more non-transitory computer readable storage media,

Images