Change-aware snapshot replication

Abstract

To provide change-aware snapshot replication.SOLUTION: Typically, an initial replica obtained using snapshot replication is periodically updated. However, the update process can unnecessarily consume computing resources if data in a source data object has not changed with respect to data in a replica data object. Disclosed techniques check to determine whether a snapshot replica is out of date before obtaining a new snapshot. The checks can be performed on manual request or on the occurrence of triggers, such as receiving a query that accesses the replica data object or according to a schedule. Information for current and prior versions of the remote data object can be compared to determine whether a replica is out of date, such as digest values of contents of the remote data object or timestamps associated with the remote data object.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] This disclosure generally relates to data replication. Certain implementations relate to determining whether a source data object has changed compared to a target data object before replicating data from the source data object to the target data object.

Background Art

[0002] It is becoming increasingly common for companies to store data in a variety of systems, including one or more local systems and one or more cloud-based systems. The systems can store data in different formats (e.g., a database that stores relational databases vs. JSON documents), or can be of different types, such as using different database management systems (e.g., using software and / or hardware provided by different vendors) to store data. Even if data is stored in the same format using the same vendor's software, there can be differences in which data is stored in a particular location and the schema used to store that data.

[0003] To help address these issues, various techniques have been used, including data replication and database federation. In a federated database environment, requests for database operations, such as queries, can specify the source of a "remote" database that is accessed using a local database system or a data federation. In some cases, both a local data source and a remote data source can be specified in the same query, such as having a query that retrieves data from both data from a local database table and a data source of a remote federated database system.

[0004] Data replication provides strategic business advantages by enabling the target system to access and utilize data originally present in the source system, supplementing information already natively stored in the target system itself. In this architecture, the source system essentially acts as a more trusted, authoritative, or primary source for the replicated data. This also means that the target system can leverage the additional information replicated from the source system to enhance its data processing or analysis capabilities, while maintaining its own unique data. However, any discrepancies or conflicts involving the replicated data are typically resolved with the source system taking precedence, considering its status as a more trusted and superior source.

[0005] For various reasons, such as query execution, data replication can be a better solution than data federation in some scenarios. However, in many cases, data in a source system is not "static" because data can be added to, deleted from, or modified within the source system. Therefore, instead of replicating data from the source system to the target system once, a process is implemented to update the data in the target system based on changes made in the source system.

[0006] One way to perform these updates is to enable real-time replication, which often uses log-based techniques to ensure that changes in the source system are propagated to the target system as soon as they are made in the source system. While this can help ensure that the data in the target system is identical to the data in the source system, the real-time replication process can be complex and computationally expensive on both the source and target systems. A particular drawback of log-based techniques is that both the source and target systems typically need to be tightly integrated. For example, the target system is usually designed based on the implementation of the source system, and therefore, appropriate logs can be obtained and parsed. Often, this makes it difficult to implement real-time replication between systems from different vendors.

[0007] Therefore, another type of replication that can be used is snapshot replication, in which all or selected data sources, such as specific tables identified for replication, are periodically sent to the target system so that the target system can have a more up-to-date version of the data in the source system. However, since snapshot replication is not real-time replication, the data in the target system can become outdated compared to the data in the source system over time. Furthermore, since all the content identified for replication in the source system is usually sent to the target system during a snapshot update, performing a snapshot update can be computationally expensive. Therefore, there is room for improvement. [Overview of the project] [Means for solving the problem]

[0008] This “Outline of the Invention” is provided in a simplified form to introduce the set of concepts that will be further described below in the “Modes for Carrying Out the Invention.” This “Outline of the Invention” is not intended to identify any important or essential features of the claims, nor is it intended to be used to limit the scope of the claims.

[0009] The disclosed technologies and solutions can provide improved snapshot replication. Typically, the initial replica taken using snapshot replication is updated periodically. However, if the data in the source data object has not changed compared to the data in the replicated data object, the update process can unnecessarily consume computing resources. The disclosed technologies perform a check to determine whether the snapshot replica is outdated before taking a new snapshot. The check can be performed on a manual request, when a trigger occurs such as receiving a query to access the replicated data object, or according to a schedule. Information about current and previous versions of the remote data object, such as a digest value of the remote data object's content or a timestamp associated with the remote data object, can be compared to determine whether the replica is outdated.

[0010] In one embodiment, the disclosure provides a process for updating a snapshot replica after determining that the current snapshot replica is outdated compared to a remote data object. A first instance of a first data object type is created in a local data source. A first set of data is received from a remote data source for the first instance of the first data object type. The first set of data is data extracted from a remote data object in the remote data source. The data is stored in the first instance of the first object type. The first instance of the first object type is determined to be outdated compared to a remote data object. A second set of data is received for the first instance of the first data object type. The second set of data is updated data from the remote data object. At least a portion of the first set of data is replaced in the first instance of the first object type with data from the second set of data.

[0011] This disclosure also includes computing systems and tangible, non-temporary, computer-readable storage media configured to perform or containing instructions for performing the methods described above. Various other features and benefits, as described herein, may be incorporated into the technology as desired. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an exemplary database system that may be used to implement the disclosed aspects of technology. [Figure 2] This diagram illustrates a computing environment in which a database system can access data on a remote computing system using data federation, including virtual tables, or using local tables that have replicated data from the remote computing system. [Figure 3]This diagram shows conventional data replication technology. [Figure 4] These are diagrams and flowcharts of the snapshot replication process as disclosed herein. [Figure 5] This is a flowchart of the disclosed process that executes queries containing hints about preferred behavior when a snapshot replica is determined to be outdated. [Figure 6] This diagram shows remote tables and replicated tables, as well as data elements that can be used to determine whether a replicated table is more up-to-date than the remote table. [Figure 7] This flowchart illustrates the process of updating the snapshot replica after determining that the current snapshot replica is outdated compared to the remote data object. [Figure 8] This is a diagram of an exemplary computing system in which several described embodiments may be implemented. [Figure 9] This is a diagram illustrating an exemplary cloud computing environment that may be used in conjunction with the technologies described herein. [Modes for carrying out the invention]

[0013] Example 1 - Overview It is becoming increasingly common for companies to store data across various systems, including one or more local systems and one or more cloud-based systems. These systems can be of different types, such as storing data in different formats (e.g., relational databases versus databases storing JSON documents) or using different database management systems (e.g., using software and / or hardware from different vendors). Even if the data is stored in the same format using software from the same vendor, there can still be differences in where the data is stored and the schemas used to store that data.

[0014] To help address these issues, various techniques have been used, including data replication and database federation. In a federated database environment, requests for database operations, such as queries, can specify a source in the local database system or a “remote” database accessed using data federation. In some cases, both local and remote data sources may be specified in the same query, such as having a query that retrieves data from both the local database table and the data source in the remote federated database system.

[0015] Data replication provides strategic business advantages by enabling the target system to access and utilize data originally present in the source system, supplementing information already natively stored in the target system itself. In this architecture, the source system essentially acts as a more trusted, authoritative, or primary source for the replicated data. This also means that the target system can leverage the additional information replicated from the source system to enhance its data processing or analysis capabilities, while maintaining its own unique data. However, any discrepancies or conflicts involving the replicated data are typically resolved with the source system taking precedence, considering its status as a more trusted and superior source.

[0016] For various reasons, such as query execution, data replication can be a better solution than data federation in some scenarios. However, in many cases, data in a source system is not "static" because data can be added to, deleted from, or modified within the source system. Therefore, instead of replicating data from the source system to the target system once, a process is implemented to update the data in the target system based on changes made in the source system.

[0017] One way to perform these updates is to enable real-time replication, which often uses log-based techniques to ensure that changes in the source system are propagated to the target system as soon as they are made in the source system. While this can help ensure that the data in the target system is identical to the data in the source system, the real-time replication process can be complex and computationally expensive on both the source and target systems. A particular drawback of log-based techniques is that both the source and target systems typically need to be tightly integrated. For example, the target system is usually designed based on the implementation of the source system, and therefore, appropriate logs can be obtained and parsed. Often, this makes it difficult to implement real-time replication between systems from different vendors.

[0018] Therefore, another type of replication that can be used is snapshot replication, in which all or selected data sources, such as specific tables identified for replication, cause their content to be periodically sent to a target system so that the target system can have a more up-to-date version of the data of the source system. However, since snapshot replication is not real-time replication, over time, the data in the target system can become stale compared to the data in the source system. Further, usually all of the content identified for replication in the source system is sent to the target system during a snapshot update, so performing a snapshot update can be computationally expensive. Therefore, there is room for improvement.

[0019] As a specific example of how a local database can access data from a remote data source, the SDA (Smart Data Access) data virtualization technology of SAP SE in Walldorf, Germany, enables data within the "virtual tables" of a local database system to access data replicated from a remote system to the local system or directly access data from a remote system through data federation. In some implementations, a given virtual table can be toggled between access to data from a local replica and direct access to data from a remote system. The disclosed techniques can be used in SAP SDA but can also be used in other scenarios involving snapshot (or "batch") replication from a remote system to a local system.

[0020] As discussed, snapshot replication may have less overhead than real-time replication, but the process of performing snapshot updates can still be expensive, depending on how often the data changes and how often updated snapshots are taken, or whether updated snapshots are taken at all, and the data can become stale. Typically, snapshot updates are requested either manually or on a schedule. The problem with manually requesting snapshot updates is that it requires proactive action on the part of the user or computing process. For example, a user may need to consider whether the data from a remote data source may be stale compared to when the last snapshot was taken, before executing a query. However, the user may forget to perform this action, leading to query results that may contain stale data. Or, a user may request an update even if the data on the source system has not changed compared to the current snapshot of the local system, incurring the resulting computing overhead.

[0021] In other cases, snapshots can be refreshed periodically, such as according to a schedule. However, this can suffer from the same problems as manually triggering snapshot updates. Updates may occur before the data has changed or before a "significant" number of changes have occurred, resulting in unnecessary computational costs, or queries may be executed against the old data before the scheduled update has occurred.

[0022] The present disclosure provides techniques for improved snapshot replication. In particular, the disclosed techniques enable a current snapshot of replicated data in a local system to be compared to the current state of target data in a remote system. Instead of directly comparing data from the remote system and the replicated system, summary data of the data in the remote system can be compared to summary data of the replicated data in the local system. If the comparison indicates that a change has occurred, the local system can obtain an updated replica from the remote system. Thus, snapshot updates are only performed when data has changed in the remote system.

[0023] The present disclosure provides specific examples of the disclosed techniques implemented in a relational database system, such as used in a process that includes local replicated database tables and remote database tables. However, the disclosed techniques can be used with other types of data objects, such as storing data in formats such as JSON, XML, CSV, PARQUEST, BSON (binary JSON), APACHE KAFKA, APACHE CASSANDRA, or SAP BW objects.

[0024] In some cases, the remote object and the replicated object on which the data is stored may be of the same type, but in other cases, the remote object and the replicated object may be of different types. As described later in this disclosure, the characteristics or properties of a first version of a remote data object (such as those reflected in the replicate) may be compared to the characteristics or properties of a second version of the remote data object (such as those reflected in the current version of the object in the remote system), and if the values ​​of the properties or characteristics are different, a change may be indicated, and an update process may be triggered. If the values ​​of the characteristics or properties of the version of the remote object used to generate or populate the replicated object are available, the characteristics or properties do not need to be calculated from the replicated object.

[0025] Various types of summary data may be used to determine whether data maintained on a remote system has changed compared to data in a snapshot replica of the local system. In one embodiment, the hash value of the snapshot replica may be calculated and compared to the hash value of the current version of the data on the remote system. In another embodiment, the current table statistics of a table may be compared to the table statistics associated with the snapshot replica. Other metadata about a particular data object, such as the date the table was last modified or information about the size of the table, may be used for comparison purposes.

[0026] Comparisons can be initiated and performed according to various criteria. In some cases, a comparison may occur when a query requesting data from a replica is received on the local system, or during similar activities such as requests for the compilation or recompilation of a query plan. In another example, comparisons may be performed according to a schedule.

[0027] The comparison process and the triggers for the update process may be executed on the remote or local system, depending on the implementation. When checking whether the replica is up-to-date during a query request, the comparison process may be initiated by the local system. However, the local system may send data values ​​to the remote system, in which case the remote system can determine whether the replica is up-to-date, or the local system may request values ​​from the remote system (or the remote system may send values ​​to the local system), in which case the local system can determine whether the replica is up-to-date. For other types of triggers, such as determining whether the replica is up-to-date according to a schedule, the check may be initiated on the remote or local system.

[0028] This disclosure generally proceeds to consider how to determine whether a new snapshot replica should be taken based on information about the entire data object, such as a relational database table. However, at least certain aspects of the disclosed technology may be applied at a more granular level. For example, instead of determining whether the contents of the entire table differ from the snapshot replica, it may be determined whether one or more specific columns of the snapshot replica have different properties from one or more corresponding columns of the current version of the remote data source used for the replica. Similarly, instead of looking at (or in addition to) data about the entire table, metadata about columns, such as the modified date or size, may be used when determining whether the snapshot replica should be updated.

[0029] The disclosed technology can thus offer many advantages. By checking whether a snapshot replica is likely to differ from the source data on the remote system, unnecessary data transmission and processing can be avoided compared to scenarios where new snapshots are taken according to a schedule or trigger event. The comparison to determine changes and take a new snapshot can be automated, which frees users from the need to manually determine whether an updated snapshot should be taken and execute the update process. Furthermore, the disclosed technology provides flexibility in processing queries, such as allowing the queryer to specify what action should be taken if it is determined that a snapshot replica may be outdated.

[0030] Example 2 illustrates an exemplary database system that could be used to implement the disclosed technology. Example 3 provides an example of a virtual table, which contains a logical pointer that can be updated to point to different locations, including locations within a federated data source or a local data source (including local tables or tables maintained in a cache). However, it should be understood that the disclosed technology does not require the use of such virtual tables; that is, the disclosed technology can be applied to any implementation of replicated tables that retrieve data using snapshot replication. Examples 4-8 describe the disclosed technology in more detail for performing change-aware snapshot replication.

[0031] Example 2 - Exemplary Database Architecture Figure 1 shows an exemplary database environment 100. The database environment 100 may include a client 104. Although a single client 104 is shown, client 104 may represent multiple clients. One or more clients 104 may be OLAP clients, OLTP clients, or a combination thereof.

[0032] Client 104 communicates with database server 106. Through various subcomponents, database server 106 can process requests for database operations, such as requests to store, read, or manipulate data (i.e., CRUD operations). The session manager component 108 can manage connections between client 104 and database server 106, including clients that communicate with the database server using database programming interfaces such as Java Database Connectivity (JDBC), Open Database Connectivity (ODBC), or Database Shared Library (DBSL). Typically, session manager 108 can manage connections with multiple clients 104 simultaneously. Session manager 108 can perform functions such as creating new sessions for client requests, allocating client requests to existing sessions, and authenticating access to database server 106. For each session, session manager 108 can maintain a context that stores a set of session-related parameters, such as database transaction commits or settings related to transaction isolation levels (such as statement-level isolation or transaction-level isolation).

[0033] With regard to other types of clients 104, such as web-based clients (e.g., clients using the HTTP protocol or similar transport protocols), the client can interface with the application manager component 110. Although shown as a component of the database server 106, in other implementations, the application manager 110 may reside outside of the database server 106 but be able to communicate with it. The application manager 110 can initiate new database sessions with the database server 106 and perform other functions in a similar manner to the session manager 108.

[0034] The application manager 110 can determine the type of application making a request for database operation and mediate the execution of the request on the database server 106 by calling or executing a procedure call, generating a query language statement, or converting data between formats usable by the client 104 and the database server 106. In a particular example, the application manager 110 receives a request for database operation from the client 104, but does not store information such as state information related to the request.

[0035] Once a connection is established between client 104 and database server 106, including when established through application manager 110, client requests are typically executed using a query language such as Structured Query Language (SQL). When executing a request, session manager 108 and application manager 110 may communicate with query interface 112. Query interface 112 may be responsible for creating a connection to the appropriate execution component of database server 106. Query interface 112 may also be responsible for determining whether the request is associated with a previously cached statement or stored procedure, and for calling the stored procedure or associating the previously cached statement with the request.

[0036] At least certain types of requests for database operations, such as query language statements for writing or manipulating data, can be associated with a transaction context. In at least some implementations, each new session can be assigned to a transaction. Transactions can be managed by a transaction manager component 114. The transaction manager component 114 can perform actions such as coordinating transactions, managing transaction isolation, tracking running and completed transactions, and managing transaction commits or rollbacks. In performing these actions, the transaction manager 114 can communicate with other components of the database server 106.

[0037] The query interface 112 can communicate with a query language processor 116, such as a structured query language processor. For example, the query interface 112 may forward query language statements or other database operation requests from the client 104 to the query language processor 116. The query language processor 116 may include a query language executor 120, such as an SQL executor, which may include a thread pool 124. Some requests or components of database operations may be executed directly by the query language processor 116. Other requests or components may be forwarded by the query language processor 116 to other components of the database server 106. For example, transaction control statements (such as commit or rollback operations) may be forwarded by the query language processor 116 to the transaction manager 114. In at least some cases, the query language processor 116 is responsible for performing operations that retrieve or manipulate data (e.g., SELECT, UPDATE, DELETE). Other types of operations, such as queries, may be sent by the query language processor 116 to other components of the database server 106. The query interface 112 and session manager 108 can maintain and manage context information related to database operation requests. In certain implementations, the query interface 112 can maintain and manage context information for requests received through the application manager 110.

[0038] When a connection is established between the client 104 and the database server 106 by the session manager 108 or the application manager 110, client requests such as queries may be allocated to threads in the thread pool 124, for example, by using the query interface 112. In at least one implementation, a thread is associated with a context for performing processing activities. Threads may be managed by the operating system of the database server 106, by another component of the database server, or in combination with another component of the database server. Generally, at any given time, the thread pool 124 contains multiple threads. In at least some cases, the number of threads in the thread pool 124 may be dynamically adjusted, for example, depending on the level of activity in the database server 106. Each thread in the thread pool 124 may, in certain embodiments, be allocated to multiple different sessions.

[0039] When a query is received, the session manager 108 or application manager 110 can determine whether an execution plan for the query already exists in the plan cache 136 or elsewhere. If a query execution plan exists, the cached execution plan can be retrieved and forwarded to the query language executor 120, for example, by using the query interface 112. For example, the query may be sent to an execution thread in the thread pool 124 determined by the session manager 108 or application manager 110. In a particular example, the query plan is implemented as an abstract data type.

[0040] If a query is not associated with an existing execution plan, the query may be parsed using the query language parser 128. The query language parser 128 can, for example, check the query language statements of a query to ensure they have the correct syntax and that the statements are otherwise valid. For example, the query language parser 128 can check whether the tables and records described in the query language statements are defined in the database server 106.

[0041] Queries can also be optimized using the query language optimizer 132. The query language optimizer 132 can manipulate elements of the query language statement to enable the query to be processed more efficiently. For example, the query language optimizer 132 may perform actions such as unnesting the query or determining an optimized execution order of various actions within the query, such as actions within a statement. After optimization, an execution plan may be generated or compiled for the query. In at least some cases, the execution plan can be cached in a plan cache 136 or the like, and can be retrieved when the query is received again (for example, by the session manager 108 or the application manager 110).

[0042] For the purposes of this disclosure, one task that may be performed by the query language optimizer 132 is to determine where a request for database operation, or a part thereof, should be performed. For example, a complex query may be sent that reads data from multiple data sources. At least one of the data sources may be a virtual table, and the request may be executed on a replicated table on the database server 106, or the request may be sent to a remote system by data federation, and the results may be returned to the database server.

[0043] Once a query execution plan is generated or received, the query language executor 120 can oversee the execution of the query execution plan. For example, the query language executor 120 can invoke the appropriate subordinate components of the database server 106.

[0044] When executing a query, the query language executor 120 can invoke a query processor 140 which may include one or more query processing engines. These query processing engines may include, for example, an OLAP engine 142, a join engine 144, an attribute engine 146, or a compute engine 148. The OLAP engine 142 may, for example, apply rules to create an optimized execution plan for an OLAP query. The join engine 144 may be used to implement relational operators, typically for non-OLAP queries, such as join and aggregate operations. In a particular implementation, the attribute engine 146 may implement column data structures and access operations. For example, the attribute engine 146 may implement merge and query processing functions, such as column scanning.

[0045] In certain situations, such as when a query involves complex or internally parallelized operations or sub-operations, the query executor 120 can send the query operations or sub-operations to a job executor component 154, which may include a thread pool 156. The query execution plan may include multiple plan operators. Each job execution thread in the job execution thread pool 156 may, in a particular implementation, be assigned to an individual plan operator. The job executor component 154 may be used to execute at least some of the query operators in parallel. In some cases, plan operators may be further divided and parallelized, such as having operations access different parts of the same table simultaneously. Using the job executor component 154 may increase the load on one or more processing units of the database server 106, but it can improve query execution time.

[0046] The query processing engine of the query processor 140 can access data stored in the database server 106. The data may be stored in row store 162 in a row-level format, or in column store 164 in a column-level format. In at least some cases, the data may be converted between row-level and column-level formats. Certain operations performed by the query processor 140 may access or manipulate data in row store 162, column store 164, or, with respect to certain types of operations (such as joins, merges, and subqueries), both row store 162 and column store 164. In at least some embodiments, row store 162 and column store 164 may be maintained in main memory.

[0047] The persistence layer 168 can communicate with the row store 162 and the column store 164. The persistence layer 168 can perform actions such as committing write transactions, storing redo log entries, rolling back transactions, and periodically writing data to storage devices to provide persistent data 172.

[0048] When executing requests for database operations such as queries or transactions, the database server 106 may need to access information stored in another location, such as another database server. The database server 106 may include a communication manager component 180 for managing such communications. The communication manager 180 can also mediate communication between the database server 106 and the client 104 or application manager 110 when the application manager is located outside the database server.

[0049] In some cases, database server 106 can be part of a distributed database system that includes multiple database servers. At least some of the database servers may include some or all of the components of database server 106. Database servers in a database system may, in some cases, store multiple copies of data. For example, a table may be replicated to two or more database servers. In addition or alternatively, information within a database system may be distributed among multiple servers. For example, a first database server may hold a copy of a first table, and a second database server may hold a copy of a second table. In further implementations, information may be partitioned among database servers. For example, a first database server may hold a first portion of a first table, and a second database server may hold a second portion of the first table.

[0050] When performing database operation requests, the database server 106 may need to access other database servers or other information sources within the database system or in external systems, such as external systems where remote data objects reside. The communication manager 180 may be used to mediate such communications. For example, the communication manager 180 can receive and route information requests from components of the database server 106 (or from another database server) and receive and route responses.

[0051] The database server 106 may include components for coordinating data processing operations, including those involving remote data sources. In particular, the database server 106 may include a data federation component 190 that at least partially handles requests to access data maintained in remote systems. In performing this function, the data federation component 190 may include one or more adapters 192, which may include logical, configuration, or connection information usable for communication with remote systems, such as when generating virtual parameterized data objects or obtaining information for executing data requests using virtual parameterized data objects (e.g., issuing requests to remote systems for data accessed using corresponding parameterized data objects in the remote systems). An example of an adapter is the “connector” implemented in the technology available from SAP SE in Walldorf, Germany. Furthermore, the disclosed technology may utilize underlying technologies for data federation technologies such as SAP SE’s Smart Data Access (SDA) and Smart Data Integration (SDI).

[0052] Example 3 - An exemplary virtual table containing an updatable logical pointer. Figure 2 shows a computing environment 200 in which the disclosed embodiments may be implemented. The basic computing environment 200 in Figure 2 includes several features that may be common to different embodiments of the disclosed technology, including one or more applications 208 that can access a central computing system 210, which may be a cloud computing system. Although the central computing system 210 is shown as a monolithic / unitary system, it should be understood that the central computing system may include several computing systems that function together as a single system, particularly in a cloud environment. For example, the central computing system 210 may be implemented as multiple “nodes,” including an anchor node and zero or more non-anchor nodes. The central computing system 210 could also be a more typical “distributed” database system, including a master node and one or more worker nodes.

[0053] The central computing system 210 can do so by providing access to data stored in one or more remote database systems 212. Furthermore, the remote database systems 212 can be accessed by one or more applications 214. In some cases, application 214 may also be application 208. That is, some applications may only access (directly) the data in the central computing system 210, some applications may only access the data in the remote database systems 212, and other applications may access the data in both the central computing system and the remote database systems.

[0054] The central computing device 210 may include a query processor 220. The query processor 220 may include several components, including a query optimizer 222 and a query executor 224. The query optimizer 222 may be responsible for determining the query execution plan 226 for queries executed using the central computing system 210. The query plan 226 generated by the query optimizer 222 may include both a logical plan that shows the order of operations performed in the query (e.g., joins, projections) and a physical plan for performing such operations. Once created by the query optimizer 222, the query plan 226 can be executed by the query executor 224. The query plan 226 may be stored in the query plan cache 228 as a cached query plan 230. When a query is submitted again for execution, the query processor 220 can determine whether a cached query plan 230 exists for the query. If it does, the cached query plan 230 can be executed by the query executor 224. If it does not exist, a query plan 226 is generated by the query optimizer 222. In some cases, the cached query plan 230 may be overridden, for example, if changes are made to the database schema or at least components of the database schema (e.g., tables or views) used by the query.

[0055] The data dictionary 234 (also known as the system catalog) can maintain one or more database schemas for the central computing system 210. In some cases, the central computing system 210 can implement a multi-tenant environment, where different tenants may have different database schemas. In at least some cases, at least some database schema elements may be shared by multiple database schemas.

[0056] Data Dictionary 234 may contain definitions (or schemas) of different types of database objects, such as table or view schemas. The following discussion will refer to tables for simplicity, but it should be understood that it is also applicable to other types of database objects, particularly those associated with retrieveable data, such as materialized views. A table schema may include information such as the table name, the number of attributes (or columns or fields) in the table, the names of the attributes, the data types of the attributes, the order in which the attributes should appear, primary key values, foreign keys, associations with other database objects, partition information, or replication information.

[0057] The table schema maintained by the data dictionary 234 may include a local table schema 236 that can represent tables primarily maintained on the central computing system 210. The data dictionary 234 may also include a replicated table schema 238 that can represent tables in which at least a portion of the table data is stored on the central computing system 210 (or, even if stored elsewhere, such as in a data lake or another cloud service, primarily managed by the database management system of the central computing system). Tables with data associated with the replicated table schema 238 typically have their data periodically updated from source tables, such as remote tables 244 in the data store 242 of the remote database system 212.

[0058] Replication can be achieved using either or both of the replication services 246 of the remote database system 212 or the replication service 248 of the central computing system 210. In a particular example, the replication service could be any of the following: SAP SE's Smart Data Integration (SDI) service, SAP Landscape Transformation Replication Server, SAP Data Services, SAP Replication Server, SAP Event Stream Processor, or SAP HANA Direct Extractor Connection, all of which are based in Walldorf, Germany. As discussed, replication can use real-time replication technology or snapshot / batch replication.

[0059] In some cases, data in the remote database system 212 may be accessed by the central computing system 210 without replicating the data from the remote database system, for example, by using federation technology. The data dictionary 234 can store the virtual table schema 252 of virtual tables that are mapped to remote tables, such as the remote table 244 in the remote database system 212. The data in the remote table 244 may be accessed using the federation service 256, for example, by using the SAP SE Smart Data Access protocol in Walldorf, Germany. The federation service 256 may be responsible for translating query operations into a format that can be processed by the appropriate remote database system 212, sending query operations to the remote database system, receiving query results, and providing query results to the query executor 224.

[0060] The data dictionary 234 may contain an updatable virtual table schema 260 having an updatable logical pointer 262. The updated virtual table schema 260 may optionally be associated with status information 264. The table pointer 262 can be a logical pointer used to identify which tables should be accessed with respect to the data of the corresponding virtual table schema 260. For example, depending on the state of the table pointer 262, the table pointer may point to a remote table 244 in a remote database system 212, or a replicated table 266 (which may be generated from the remote table 244) in a data store 268 of the central computing system 210. The data store 268 may also store data for a local table 270 that may be defined by the local table schema 236.

[0061] The table pointer 262 can be changed between the remote table 244 and the replicated table 266. In some cases, the user can manually change the table pointed to by the table pointer 262. In other cases, the table pointer 262 can be changed automatically, for example, in response to the detection of defined conditions.

[0062] Status information 264 may include indicators that identify the virtual table schema 260 as being associated with a remote table 244 or a replicated table 266. Status information 264 may also include information about the replication status of the replicated table 266. For example, if a request is made to change the table pointer 262 to point to a replicated table 266, it may take some time for the replicated table to be ready for use. Status information 264 may include whether the replication process has started, completed, or the progress status of the generation of the replicated table 266.

[0063] Changes to the updatable virtual table schema 260 and the management of the replicated tables 266 associated with the virtual table schema can be managed by the virtual table service 272. Although shown as a separate component of the central computing system 210, the virtual table service 272 can be integrated into other components of the central computing system 210, such as the query processor 220 or the data dictionary 234.

[0064] When a query is executed, it is processed by the query processor 220, including executing the query using the query executor 224 to retrieve data from either or both of the data store 242 of the remote database system 212 or the data store 268 of the central computing system 210. The query results may be returned to the application 208. The query results may also be cached in the cache 278 of the central computing system 210, for example. The cached results may be represented as a cached view 280 (for example, a materialized query result).

[0065] Application 214 can access data in the remote database system 212, for example, through the session manager 286. Application 214 can modify the remote table 244. When the table pointer 262 of the updatable virtual table schema 260 refers to the remote table 244, changes made by application 214 are reflected in the remote table. When the table pointer 262 refers to the replicated table 266, changes made by application 214 may be reflected in the replicated table using the replication service 246 or replication service 248.

[0066] Example 4 - Exemplary Replication Technology Figure 3 shows a conventional technique for replicating data from a remote system 304 to a local system 308. The remote system 304 includes a remote table 312. Data from the remote table 312 is replicated to a replica table 316 in the local system 308.

[0067] Data from the remote table 312 can be replicated to the replicated table 316 in several ways. The real-time replication process 320 sends incremental data changes made to the remote table 312 to the replicated table 316 as those changes occur, using techniques that may include change data capture or trigger-based replication. As described in Example 1, real-time replication is advantageous in helping to ensure that the replicated table 316 is up-to-date with respect to the remote table 312, but it is complex to implement and may require significant computing resources.

[0068] Figure 3 also shows the snapshot replication process 330. In the snapshot replication process 330, snapshots 334 of the remote table 312 are taken periodically. Typically, snapshot 334 contains all the data of the remote table 312 at the time the snapshot was taken. The snapshot data 334 can be sent to the local system 308. The current replicated table 316 can be replaced with the snapshot data 334.

[0069] As shown, snapshots are typically taken periodically, and therefore process 330 can be called periodic replication. Because replication is periodic, a new snapshot 334 is taken after a delay period 340. The delay period 340 can be a delay depending on the schedule, event, or manual update request. As stated in Example 1, snapshot replication may be easier to implement and may consume fewer resources than real-time replication, but the existence of the delay period 340 may cause queries in the local system 308 to retrieve older data compared to the current version of the remote table 312.

[0070] Example 5 - Exemplary snapshot replication with update checks Figure 4 illustrates the snapshot replication process according to this disclosure. Similar to Figure 3, Figure 4 includes a remote system 404 having a remote table 408 and a local system 412 having a local replicated table 416 which is a snapshot of the remote table 408 at a specific point in time. A snapshot replication process 420 is used to replicate data from the remote table 408 to the replicated table 416.

[0071] Figure 4 shows details of the snapshot replication process 420. The snapshot replication process 420 may be initiated at 424, where it is determined whether a specific event has occurred that would advance the process further. Examples of events may include requests from a user or application process to determine whether a local table 416 should be updated with a newer snapshot of a remote table 408, a specific instance of a particular date / time trigger, or a scheduled event such as the passage of time according to a schedule (e.g., to determine every hour whether the snapshot should be updated), the receipt of a query involving the local table 416, or an indication that the data in the remote table 408 has changed.

[0072] If it is determined in 424 that no event occurred, process 420 can return to 424 until it is determined that an event occurred. If it is determined in 424 that an event occurred, process 420 can proceed to 428. In 428, it is determined whether the data in the replicated table 416 still matches the data in the remote table 408. Performing a comparison of the actual data in both the remote table 408 and the replicated table 416 is expensive in terms of the resources required to copy the data, transmit the data, and compare the data, and therefore it may be more efficient to simply perform a snapshot update. Therefore, the comparison in 428 usually takes into account information about the remote table 408 and the replicated table 416, but does not compare all the data in both tables. Details on what kinds of data may be compared in 428 are discussed in Example 7.

[0073] If, at least based on the comparison criteria, it is determined in 428 that the replicated table 416 matches the remote table 408, then process 420 can return to 424. If, at 428, it is determined that the replicated table 416 does not match the remote table 408, then in 432, the table contents may be synchronized. Synchronizing the contents in 432 may involve replacing the data in the replicated table 416 with data from a new snapshot of the contents of the remote table 408.

[0074] Optionally, in 440, the metadata of the remote table 408, the metadata of the replicated table 416, or the metadata of both tables may be updated. Updating metadata in 440 may include updating values ​​used for comparison in 428. For example, a new hash value, last modified date, table size, or set of statistics for replicated table 416 (corresponding to the metadata of remote table 408 at the time of the snapshot) may be stored.

[0075] Example 6 - Exemplary query processing with update checks and optional customized processing As illustrated in Example 1, in some cases, a query request involving access to a local replica table can trigger a check to determine whether the snapshot replica is up-to-date according to certain criteria, such as event 424 in process 420 in Figure 4. Normally, if no changes are detected, the query may be executed using the current snapshot replica. However, if the snapshot replica is determined to be outdated, a different process may be executed.

[0076] If a snapshot replica is determined to be outdated, the query may still be executed using the current snapshot replica, even if some of the data accessed by the query may be outdated. Alternatively, if a snapshot replica is determined to be outdated, the query may be rejected or queued until the snapshot update is complete. A given behavior can be "hardcoded" into the system, or a default behavior can be specified, although the default behavior can be selectively overridden.

[0077] In certain cases, a particular query can directly specify the desired behavior. That is, a query may include "hints," such as keywords, that indicate to the query executor or query optimizer what behavior is desired. A query hint might indicate that the query should be executed without delay against the current version of a snapshot replica, even if the data may be outdated compared to the remote table. Alternatively, a query hint might indicate that query processing should be deferred until an updated snapshot is taken. In further examples, query hints may be used to indicate that the query should be executed using the remote table instead of the snapshot replica, for example, by using data federation techniques. In yet another example, a query hint might specify that a failure message should be returned if the snapshot is not up-to-date.

[0078] Figure 5 shows an exemplary process 520 in which the query executor 514 executes query 504, based on default behavior or using non-default behavior if query hints are provided. Although described as being performed by the query executor, at least some of the operations of process 520 may be performed by the query optimizer.

[0079] At 524, the query is received. At 528, it is determined that the query accesses a replicated table. At 532, a check is performed to determine if the replicated table is up-to-date compared to the remote table. If it is up-to-date, the query is executed at 536 using the current replicated table, and process 520 can terminate at 540.

[0080] If it is determined at 532 that the replicated table is not up-to-date, instead of the default behavior of waiting until an updated snapshot is taken, a check is performed at 544 to determine whether query 504 contains hint 506, which indicates that the query should be executed against the current version of the replicated table. If it is determined at 544 that query 504 contains hint 506, the query is processed at 548 using the existing replicate, and process 520 can terminate at 540.

[0081] If it is determined at 544 that query 504 does not contain hint 506, then at 552 it is determined whether the query contains hint 508, which indicates that the query should be executed against the remote table if the local replicated table is old. If hint 508 is present, the query is executed using the remote table at 556, and process 520 can be terminated at 540.

[0082] If it is determined at 552 that query 504 does not contain hint 508, the query may be queued at 560 until the local replica table is updated. Once the local replica table is updated, query 504 can be executed at 564, and process 520 can be terminated at 540.

[0083] It should be noted that process 520 is merely one example of how the disclosed technology may be used. Other implementations may perform checks on only hint 506 or hint 508, and the database system may support only hint 506 or hint 508. Similarly, in process 520, the default behavior is to postpone query execution until an updated snapshot is taken for the replicated table. However, the process may have a default behavior of executing the query using the current old replicate, or instead, hints may be used to determine whether execution should be postponed until the update, or whether federated query execution should be used with the remote table. Another process may have a default behavior of using federated query execution with the remote table if the local replicated table is old, or instead, hints may be used to determine whether execution may be performed on the local system after the updated snapshot has been received, or whether the query should be performed on the local system using the current, but old, replicated table.

[0084] Example 7 - Exemplary elements for determining whether two data objects are different or for triggering an update analysis process. Figure 6 shows various table properties that can be used to determine whether a replicated table is up-to-date compared to a remote table whose data is replicated to the replicated table. Figure 6 includes a remote system 604 with a remote table 606 and a local system 608 with a replicated table 610 corresponding to a snapshot of the remote table.

[0085] The remote table 606 is shown as having content 620. The content 620 may include one or more semantic attributes 624 and one or more synchronization attributes 626. The semantic attributes 624 provide data "about" a particular entity represented as an entry in the remote table 606. For example, an entry in the remote table 606 may correspond to an employee, a customer, or a sales transaction. In the case of an employee, the semantic attributes 624 may include attributes such as name, employee identification number, address, title, or payrate.

[0086] Synchronization attribute 626 is an attribute specifically used or available to determine whether the data in remote table 606 differs from the data in replicated table 610. Synchronization attribute 626 may, but may not, be a semantic attribute 624. A specific example of synchronization attribute 626 is a timestamp, such as a timestamp indicating when a particular row in the table was last added, deleted, or modified. The contents 640 of replicated table 610 may contain semantic attributes 644 and synchronization attribute 646, corresponding to semantic attributes 624 and 626, but the attribute values ​​may differ between tables 606 and 610 if the replicated table is older than the remote table. Replica table 610 may include last modified date as synchronization attribute 644, the value of which corresponds to last modified date of remote table 606 at the time of the most recent snapshot. Therefore, determining whether the replica table 610 is outdated can be done by querying the remote table 606 to determine whether the remote table contains records with a timestamp synchronization attribute 626 value that is later than the latest value reflected in the timestamp synchronization attribute 646 of the replica table 610.

[0087] Instead of having a "last modified" date as a synchronization attribute 626, the overall last modified date 632a of the remote table 606 may be stored in metadata 630 associated with the remote table and having metadata elements 632 (indicated as 632a-632g). The replica table 608 includes metadata 650 with metadata element 652 (indicated as metadata elements 652a-652g) and may include the last modified date 652a related to the remote table 606 in the most recent snapshot. When a request is made to check whether the remote table 606 has changed compared to the replica table 610, metadata 630 may be queried to determine if the last modified date 632a is the same as the last modified date 652a in metadata 650. If the values ​​are not the same, an updated replica may be retrieved.

[0088] As described above, in some scenarios, determining whether a new snapshot should be taken for the replicated table 610 may be performed at a finer level than determining the entire remote table 606. For example, changes to some semantic attributes 624 in the remote table 606 may be more noteworthy than changes to other semantic attributes 624. In this case, it may be desirable to update the snapshot when the particularly noteworthy semantic attribute 624 is changed, compared to the corresponding semantic attribute 644 in the replicated table 610. Thus, metadata 630 may include the last modified date 632b of a particular semantic attribute 624 (such as a specific column in the remote table 606), and metadata 650 of the replicated table 610 may include the last modified date 652b of the corresponding semantic attribute 646. Then, determining whether the replicated table 610 is up-to-date may be performed by determining whether the last modified date 632b of a given semantic attribute 624 is the same as the last modified date 652b of the corresponding semantic attribute 644.

[0089] The table statistics 632c of the remote table 606, reflected in metadata 630, can also be used to determine when the remote table may have different content from that of the replicated table 610. While metadata 630 stores the current statistics, metadata 650 of the replicated table 610 contains statistics 652c copied from metadata 630 when the last snapshot update was performed. Any suitable type of table statistics 632c may be used, but examples of table statistics include: Table size: The size of a table in memory, which may include memory for table data and memory used for structure or system administration, such as indexes. Number of records: The number of records (rows) in the table. The average record length can also be tracked. Column statistics: Information about the data stored in each column (attribute) of a table, such as the number of distinct values, maximum and minimum values, mode, number of nulls, and average data size. Partition information: Information about how the table is partitioned, which may include the number of partitions, the number of rows or columns in each partition, or the total size (memory or disk) of each partition. Index statistics: Information such as the index identifier, index size, and the number of leaf nodes in the index. Histogram: Information about the statistical distribution of column values. Disk Size / Clustering Information: Information about the disk size of a particular table, and which rows with similar column values ​​are stored close to each other on disk.

[0090] It should be noted that certain table statistics 632c may depend on the remote system 604. For example, information about memory usage, particularly for indexes or system administration, may be specific to the implementation of the remote system 604, and even if the data in replicated table 610 is identical to the data in remote table 606, the local system 608 may not use memory in the same way. However, this does not prevent statistics 632c from being used to determine whether replicated table 610 may be stale, since the comparison is between two states of remote table 606 in the remote system 604, and not between the actual statistics of replicated table 610 and the statistics of remote table 606. In other words, such metadata 650 corresponds to the metadata 630 at the time of the snapshot.

[0091] The contents of remote table 606 can be used in other ways to compare the state of tables. For example, the contents of remote table 606 (i.e., the actual values ​​stored in the remote table according to the definition of the remote table) can be provided as input to a hash algorithm to generate a hash value 632d. This hash value 632d can be provided to the local system 608 and stored in metadata 650 as a hash value 652d. When it is determined whether a replica table 610 has the same contents as remote table 606, the current hash value 632d of the remote table can be compared to the hash value 652d of the replica of the most recent snapshot used for the replica table.

[0092] The current hash value 632 of the remote table 606 may be determined (calculated) at various points in time, depending on the judgment of the particular implementation. For example, hash value 632d may be determined at the request of the local system 608, such as as part of the process to check whether a replica of an updated snapshot should be taken. Otherwise, hash value 632d may be updated periodically in the remote system 604, including when changes are made to the remote table 606, but this can involve significant overhead for tables with frequently changing data. On the other hand, periodic updates of hash value 632d may be sufficient, although this may create a risk of "false negative" indication, where the replica table 610 appears indistinguishable from the remote table 606 if the remote table has actually changed but the updated hash value has not yet been calculated.

[0093] The discussion uses examples of hash algorithms and hash functions, but similar types of techniques may be used, including less computationally expensive techniques such as using checksums. Similar to hash values, checksums may be calculated using the entire contents of remote table 606 in at least some cases. Hash values ​​and checksums are examples of a broader category of digests or digest values, and other types of digests may also be used in the disclosed techniques. Furthermore, hash values ​​and checksums may be calculated against the contents of a table as represented in a table, but optionally, they may be calculated in other ways, such as against a binary representation of the table contents.

[0094] Metadata 630 or metadata 650, or both, may include synchronization schedules 632e, 652e, respectively, which can be used by synchronization logic 670 of the remote system 604 or synchronization logic 672 of the local system 608. More typically, metadata 650 includes synchronization schedule 652e, and checks to determine whether a copy of an updated snapshot should be taken are performed according to the schedule. In another implementation, the remote system 604 may periodically send copies of updated snapshots to the local system 608, for example, according to schedule 632e of metadata 630. Schedules can be expressed in various ways, such as specifying that checks to determine whether a copy of a new snapshot should be taken on a specific date / time should be performed according to a regular schedule (e.g., hourly or daily), or performing checks within a specified timeframe, for example, by using either or both of "before" or "after" dates. Synchronization (or, more specifically, checks to determine whether synchronization should be performed) can also be performed on demand, including based on metadata 630, 650's last sync date 632f, 652f. For example, the command could be "synchronize if LAST_SYNC_DATE > 1 day".

[0095] Metadata 630 or 650 may also include triggers 632g and 652g. In this case, “trigger” does not refer to a trigger-based database process such as a trigger-based update or replication. Rather, “trigger” refers to any other event that causes a determination to be made as to whether replicated table 610 matches remote table 606. Performing a check when a sync schedule or a query involving an action on replicated table 610 is received could be considered an example of triggers 632g and 652g, but they are separated, enumerated, and described separately in this consideration. Other examples of triggers 632g and 652g could be a change in the number of thresholds performed on remote table 606, or a query being received regarding the number of thresholds that access replicated table 610. Such triggers 632g and 652g can introduce a dynamic aspect to determining whether a replica update should be performed, compared to a static schedule, because update checks may occur more frequently if there are more updates to the remote table 606 (indicating that the replica table 610 may be significantly older) or if there is more activity on the replica table (indicating that the replica table is of higher interest and therefore may take precedence over "cold" data in terms of update checks).

[0096] This specification, including Example 7, generally describes sending a copy of the entire contents of the remote table 606 when it is determined that the contents differ from those of the replicated table 610, but in other cases, only a portion of the data may be sent, such as data added since the last replica snapshot was taken. As a specific example, a table may be designated as append-only. For these types of data, only rows added after the date of the last replica snapshot need to be sent to the local system 608.

[0097] Example 8 - Exemplary Operation Figure 7 is a flowchart of process 700, which updates a snapshot replica after determining that the current snapshot replica is outdated compared to the remote data object. At 705, a first instance of a first data object type is created in the local data source. At 710, a first set of data is received from the remote data source for the first instance of the first data object type. The first set of data is data extracted from the remote data object in the remote data source. At 715, the data is stored in the first instance of the first object type. At 720, the first instance of the first object type is determined to be outdated compared to the remote data object. At 725, a second set of data is received for the first instance of the first data object type. The second set of data is the updated data from the remote data object. At 730, at least a portion of the first set of data is replaced with data from the second set of data within the first instance of the first object type.

[0098] Example 9 - Computing System Figure 8 shows a generalized example of a preferred computing system 800 in which the described invention may be implemented. Since the invention may be implemented in a variety of general-purpose or dedicated computing systems, computing system 800 is not intended to imply any limitation on the scope of use or functionality of the disclosure.

[0099] Referring to Figure 8, the computing system 800 includes one or more processing units 810, 815 and memories 820, 825. In Figure 8, this basic configuration 830 is enclosed within the dashed lines. The processing units 810, 815 execute computer executable instructions, such as to implement the database environment and related methods described in Examples 1-8. The processing units can be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multiprocessing system, multiple processing units execute computer executable instructions to increase processing power. For example, Figure 8 shows a central processing unit 810 and a graphics processing unit or auxiliary processing unit 815. The tangible memories 820, 825 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or any combination of the two, accessible by the processing units 810, 815. The memories 820 and 825 store software 880 that implements one or more inventions described herein in the form of computer executable instructions suitable for execution by the processing units 810 and 815.

[0100] The computing system 800 may have additional features. For example, the computing system 800 may include a storage device 840, one or more input devices 850, one or more output devices 860, and one or more communication connections 870. Interconnection mechanisms (not shown), such as buses, controllers, or networks, connect the components of the computing system 800 to each other. Generally, operating system software (not shown) provides an operating environment for other software running in the computing system 800 and coordinates the activities of the components of the computing system 800.

[0101] The tangible storage device 840 may be removable or non-removable and may include magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other media that can be used to store information in a non-temporary manner and can be accessed within the computing system 800. The storage device 840 stores instructions for software 880 that implement one or more inventions described herein.

[0102] The input device 850 may be a keyboard, mouse, pen, or other touch input device such as a trackball, an audio input device, a scanning device, or another device that provides input to the computing system 800. The output device 860 may be a display, printer, speaker, CD writer, or another device that provides output from the computing system 800.

[0103] The communication connection 870 enables communication with another computing entity, such as another database server, via a communication medium. The communication medium carries information, such as computer executable instructions, audio or video input or output, or other data, within a modulated data signal. A modulated data signal is a signal whose properties are set or modified in such a way that information is encoded within the signal. The communication medium may, but not exclusively, be electrical, optical, RF, or other carriers.

[0104] The invention can be described in a broader context in which computer executable instructions, such as those contained in a program module, are executed on a target real or virtual processor in a computing system. Generally, a program module or component includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The functionality of a program module may be combined or divided among program modules as desired in various embodiments. The computer executable instructions of a program module may be executed within a local or distributed computing system.

[0105] The terms “system” and “device” are used interchangeably herein. Neither term implies any limitation to any type of computing system or computing device unless the context explicitly indicates otherwise. Generally, a computing system or computing device can be local or distributed and may include any combination of dedicated hardware and / or general-purpose hardware with software that implements the functions described herein.

[0106] For the sake of presentation, detailed explanations will use terms such as “determine” or “use” to describe the operation of a computer in a computing system. These terms are high-level abstractions of actions performed by computers and should not be confused with actions performed by humans. The actual computer operations corresponding to these terms will vary depending on the implementation.

[0107] Example 10 - Cloud Computing Environment Figure 9 shows an exemplary cloud computing environment 900 in which the technologies described may be implemented. The cloud computing environment 900 includes a cloud computing service 910. The cloud computing service 910 may include various types of cloud computing resources, such as computer servers, data storage repositories, and networking resources. The cloud computing service 910 can be centralized (for example, provided by a company or organization's data center) or distributed (for example, provided by various computing resources located in different locations, such as different data centers, and / or in different cities or countries). The cloud computing service 910 is utilized by various types of computing devices (e.g., client computing devices), such as computing devices 920, 922, and 924. For example, computing devices (e.g., 920, 922, and 924) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smartphones), or other types of computing devices. For example, computing devices (e.g., 920, 922, and 924) can utilize the cloud computing service 910 to perform computing operations (e.g., data processing, data storage, etc.).

[0108] Example 11 - Implementation While some of the operations of the disclosed methods are described in a specific sequential order for the sake of brevity, it should be understood that such methods of description are reordering unless a specific order is required by certain wording herein. For example, operations described sequentially may be reordered or performed simultaneously. Furthermore, for simplicity, the accompanying diagrams may not illustrate the various ways in which the disclosed methods may be used in conjunction with other methods.

[0109] Any of the disclosed methods may be implemented as computer executable instructions or computer program products stored on one or more computer-readable storage media, such as tangible non-temporary computer-readable storage media, and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device with computing hardware). Tangible computer-readable storage media are any available tangible media that can be accessed within a computing environment (e.g., one or more optical media discs such as DVDs or CDs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or hard drives)). Referring to Figure 8 as an example, the computer-readable storage media includes memories 820 and 825 and storage device 840. The term computer-readable storage media does not include signals and carriers. Furthermore, the term computer-readable storage media does not include communication connections (e.g., 870).

[0110] Any computer-executable instructions for implementing the disclosed technology, and any data created and used in implementing the disclosed embodiments, may be stored on one or more computer-readable storage media. Computer-executable instructions may, for example, be part of a dedicated software application or part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may run, for example, on a single local computer (e.g., any suitable commercial computer) or in a network environment using one or more network computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network).

[0111] For clarity, only specific selected aspects of software-based implementations are described. Other details well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology may be implemented by software written in C++, Java, Perl, JavaScript, Python, Ruby, ABAP, structured query languages, Adobe Flash, or any other suitable programming language, or in some examples, a markup language such as HTML or XML, or a combination of a suitable programming language and a markup language. Similarly, the disclosed technology is not limited to any particular type of computer or hardware. Specific details of suitable computers and hardware are well known and do not need to be detailed in this disclosure.

[0112] Furthermore, any software-based embodiment (including, for example, computer executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed via preferred means of communication. Such preferred means of communication include, for example, the Internet, the World Wide Web, intranets, software applications, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such means of communication.

[0113] The methods, apparatus, and systems disclosed should not be construed as limiting in any way. Rather, this disclosure covers all novel, non-obvious features and aspects of the various disclosed embodiments, both individually and in various combinations and partial combinations thereof. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, and the disclosed embodiments do not require any particular advantage or solution to any particular problem.

[0114] Any example technology may be combined with any one or more of the other examples described. Given the many possible embodiments to which the principles of the disclosed technology may be applied, it should be recognized that the examples shown are merely examples of the disclosed technology and should not be considered limitations on the scope of the disclosed technology. Rather, the scope of the disclosed technology is encompassed by the scope and spirit of the following claims. [Explanation of Symbols]

[0115] 100 Database Environments 104 Clients 106 Database Server 108 Session Manager Components 110 Application Manager Components 112 Query Interface 114 Transaction Manager Components 116 Query Language Processor 120 Query Language Executor 124 thread pool 128 Query Language Parser 132 Query Language Optimizer 136 Planned Cache 140 query processors 142 OLAP engine 144 coupled engines 146 Attribute Engine 148 Computing Engines 154 Job Executor Components 156 thread pool 162-row store 164-row store 168 Persistence Layer 172 Persistent Data 180 Communication Manager Components 190 Data Federation Components 192 Adapter 200 Computing Environments 208 applications 210 Central Computing System 212 Remote Database Systems 214 applications 220 query processors 222 Query Optimizer 224 Query Executor 226 Query execution plan 228 Query plan cache 230 cached query plans 234 Data Dictionary 236 Local Table Schema 238 Replicated Table Schema 242 Datastores 244 Remote Table 246 Replication Service 248 Replication Service 252 Virtual Table Schemas 256 Federation Services 260 Updatable virtual table schemas 262 Logical pointers, table pointers 264 Status Information 266 Replicated Tables 268 datastores 270 Local Tables 272 Virtual Table Service 278 Cache 280 cached views 286 Session Manager 304 Remote System 308 Local System 312 Remote Table 316 Duplicate Table 320 Real-time Replication Processes 330 Snapshot Replication Process 340 Delay period 334 snapshots, snapshot data 404 Remote System 408 Remote Table 412 Local System 416 Duplicate Table 420 Snapshot Replication Processes 504 query 506 Hints 508 Hints 514 Query Executor 520 processes 604 Remote System 606 Remote Table 608 Local System 610 Duplicate Table 620 contents 624 Semantic Attributes 626 Synchronization attribute 630 Metadata 632 metadata elements 632a Overall last modified date 632b Last modified date 632c Table Statistics 632d hash value 632e Synchronization Schedule 632f Final synchronization date 632g trigger 640 contents 644 Gammatical Attributes 646 Synchronization attribute 650 metadata 652 metadata elements 652a Last modified date 652b Last modified date 652c statistics 652d hash value 652e Synchronization Schedule 652f Final synchronization date 652g trigger 670 Synchronous logic 672 Synchronization logic 700 processes 800 Computing Systems 810 Processing Unit, Central Processing Unit 815 Processing unit, graphics processing unit, auxiliary processing unit 820 memory 825 memory 830 Basic configuration 840 Storage device 850 Input Devices 860 Output Devices 870 Communication connection 880 Software 900 Cloud Computing Environments 910 Cloud Computing Services 920 Computing Devices 922 Computing Devices 924 Computing Devices

Claims

1. A computing system, At least one memory, One or more hardware processor units coupled to at least one memory, One or more computer-readable storage media that store computer-executable instructions that cause the computing system to perform an action when executed. The operation includes, In the local data source, create the first instance of the first data object type, Receiving a first set of data for a first instance of the first data object type from a remote data source, wherein the first set of data is extracted from the remote data object of the remote data source, The first instance of the first data object type stores the first set of data, Receiving queries and, Determining that the query accesses the first instance of the first data object type, Determining that the first instance of the first data object type is old if it differs from the remote data object, and that determining that the first instance of the first data object type is old begins at least in part on determining that the query accesses the first instance of the first data object type, If it is determined that the first instance of the first data object type is old, Receiving a second set of data for the first instance of the first data object type, wherein the second set of data is updated data of the remote data object, Replacing at least a portion of the first set of data with data from the second set of data within the first instance of the first data object type A computing system that includes this.

2. The aforementioned operation, The computing system according to claim 1, further comprising postponing the execution of the query until the replacement is completed.

3. The computing system according to claim 2, wherein the postponement is performed in response to a command in the query.

4. The aforementioned operation, The computing system according to claim 1, further comprising executing the query before the replacement.

5. The computing system according to claim 4, wherein the execution of the query before the replacement is performed in response to a command in the query.

6. The aforementioned operation, The computing system according to claim 1, further comprising executing at least a portion of the query that accesses the first instance of the first data object type against the remote data object.

7. The computing system according to claim 6, wherein executing at least a portion of the query that accesses the first instance of the first data object type against the remote data object is performed in response to a command in the query.

8. The computing system according to claim 1, wherein determining that the first instance of the first data object type is old includes comparing a digest value generated from the content of the first set of data with a digest value generated from the content of the remote data object and obtained from the current metadata of the remote data object.

9. The computing system according to claim 1, wherein determining that the first instance of the first data object type is old includes comparing statistics maintained with respect to the first instance of the first data object type with statistics obtained from the current metadata of the remote data object.

10. The computing system according to claim 1, wherein determining that the first instance of the first data object type is old includes comparing the last modified date maintained with respect to the first instance of the first data object type with the last modified date obtained from the current metadata of the remote data object.

11. The computing system according to claim 1, wherein determining that the first instance of the first data object type is old includes comparing the size maintained with respect to the first instance of the first data object type with the size obtained from the current metadata of the remote data object.

12. The computing system according to claim 1, wherein the first data object type includes an identifier indicating whether a query accessing an instance of the first data object type should be executed against the local data source using the instance of the first data object type or against the remote data object.

13. The computing system according to claim 1, wherein the remote data object and the first data object type share a common set of semantic attributes.

14. The computing system according to claim 1, wherein the remote data object and the first data object type are relational database tables.

15. The computing system according to claim 1, wherein the determination is performed in the remote data source.

16. A method for performing an operation in a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, The steps include creating a first instance of a first data object type in a local data source, A step of receiving a first set of data for a first instance of the first data object type from a remote data source, wherein the first set of data is extracted from the remote data object of the remote data source. The steps include storing the first set of data in the first instance of the first data object type, Steps to receive queries, The steps include determining that the query accesses the first instance of the first data object type, A step of determining that the first instance of the first data object type is old if the first instance of the first data object type is different from the remote data object, the step of determining that the first instance of the first data object type is old begins at least in part on the determination that the query accesses the first instance of the first data object type, If it is determined that the first instance of the first data object type is old, A step of receiving a second set of data for a first instance of the first data object type, wherein the second set of data is updated data of the remote data object, The steps of replacing at least a portion of the first set of data with data from the second set of data within the first instance of the first data object type A method that includes this.

17. A computer executable instruction, when executed by a computing system including at least one hardware processor and at least one memory coupled to the at least one hardware processor, causes the computing system to create a first instance of a first data object type in a local data source. A computer executable instruction, when executed by the computing system, causes the computing system to receive a first set of data for a first instance of the first data object type from a remote data source, wherein the first set of data is extracted from a remote data object of the remote data source. A computer executable instruction, when executed by the computing system, causes the computing system to store the first set of data in the first instance of the first data object type, A computer executable instruction that receives a query, A computer executable instruction that causes the query to determine that it accesses the first instance of the first data object type, A computer-executable instruction, when executed by the computing system, causes the computing system to determine that the first instance of the first data object type is old if it differs from the remote data object, wherein the computer-executable instruction causing the first instance of the first data object type to be old is initiated at least in part on the determination that the query accesses the first instance of the first data object type, If it is determined that the first instance of the first data object type is old, A computer executable instruction, when executed by the computing system, causes the computing system to receive a second set of data for the first instance of the first data object type, wherein the second set of data is updated data of the remote data object; A computer executable instruction, when executed by the computing system, causes the computing system to replace at least a portion of the first set of data within the first instance of the first data object type with data from the second set of data. One or more non-temporary computer-readable storage media, including [the specified element].

Images