An Asynchronous Cache Consistency Method and System for Database Systems Based on MVCC

Abstract

A database system includes persistent storage, a log node, and multiple database nodes. The log node includes a memory and a processor. Each database node includes a cache memory for storing database instances and a processor for initiating database transactions by sending a snapshot request to the log node, the snapshot request including a list of pages that have been replaced or newly loaded in the cache memory. The log node processor sends a snapshot response to the database node, the snapshot response including a snapshot of the database and a list of changed pages of the database instance. The database node processor updates the state of the pages in the cache memory based on the snapshot response and executes the database transaction.

Description

Technical Field

[0001] This disclosure relates to a distributed database system in which multiple database instances access shared storage, and more particularly to maintaining the correctness of data across multiple database instances. Background Technology

[0002] A distributed database is a database accessed by multiple entities within a computing system (e.g., a cloud database). In a distributed database system, multiple instances or versions of the database are maintained by separate entities or hosts. This allows multiple database instances to access shared data. Thus, if any one database instance fails, the others can still use the database. When multiple database instances are allowed to access the shared data, these instances need to maintain cache (or buffer pool) consistency; otherwise, they may read outdated data, leading to data inconsistencies between different database versions.

[0003] In traditional distributed database systems, database instances exchange messages to synchronize the database and maintain consistency. When a page is loaded into a database instance's local cache pool, cache-related information is broadcast to all other database instances in global communication. When a page is modified, another global communication is broadcast to invalidate copies of that page in other database instances. Broadcasting consistency information incurs significant network overhead, leading to bottlenecks in database access. This network overhead reduces the bandwidth available for accessing the database. Summary of the Invention

[0004] Various examples will now be described to briefly introduce some concepts, which will be further described in the detailed description. This summary section is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] According to one aspect of this disclosure, a database system is provided, the database system including a persistent storage device, a log node, and a plurality of database nodes, the log node including a memory and a processor. The database node includes a cache memory for storing database instances and a processor for initiating database transactions by sending a snapshot request to the log node, the snapshot request including a list of pages replaced or newly loaded in the cache memory. The log node processor is configured to send a snapshot response to the database node, the snapshot response including a snapshot of the database and a list of modified pages of the database instance. The database node processor is configured to update the state of the pages in the cache memory according to the snapshot response and execute the database transaction.

[0006] Optionally, in the foregoing aspects, another implementation of the aspects provides a database node processor configured to: change the page state to semi-valid and assign a valid log snapshot number (LSN) to the page when the page is loaded from the persistent storage device; include a list of pages having the semi-valid page state and the page's valid LSN in the snapshot request; the implementation may also optionally include a log node processor configured to: identify pages invalid for the database node by comparing the page's valid LSN with the page's latest LSN; and include the identified invalid pages in the snapshot response.

[0007] Alternatively, in the foregoing aspects, another implementation of the foregoing aspects provides a log node processor for sending the valid LSN in the snapshot response.

[0008] Optionally, in any of the foregoing aspects, another implementation of the aspect provides a database node processor configured to: change the page state to half-valid when the page is loaded from the persistent storage device; change the page state from half-valid to invalid if the page is indicated as changed in the list of changed pages; and change the page state from half-valid to valid if the page is a newly loaded page and the page is not indicated as changed in the list of changed pages.

[0009] Optionally, in any of the foregoing aspects, another implementation of the aspect provides a database node processor configured to: change the page state from valid or partially valid to invalid if the page is indicated as changed in the list of changed pages; and change the page state from invalid to partially valid when the page is loaded from the persistent storage device.

[0010] Optionally, in any of the foregoing aspects, another implementation of the aspect provides a log node processor configured to: send records of executed database transactions from the log node to a persistent storage device; and send the snapshot response when the log node sends records of database transactions executed before the initiated database transaction to the persistent storage device.

[0011] Optionally, in the foregoing aspect, another implementation of the aspect provides a persistent storage device for generating the database snapshot using the records of the executed database transaction.

[0012] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides a log node storage for storing a hash table of cached pages of the database node and updating the hash table in response to the snapshot request.

[0013] According to another aspect of this disclosure, a computer-implemented method for achieving database consistency in a system comprising multiple database instances is provided. The method includes: initiating a database transaction by sending a snapshot request from a database node to a log node, the snapshot request including a list of replaced or newly loaded pages in a cache of the database node; receiving a snapshot response from the log node at the database node, the snapshot response including a snapshot of the database and a list of changed pages of the database instance; updating the state of the pages in the cache of the database node according to the snapshot response and executing the database transaction.

[0014] Optionally, in the foregoing aspect, another implementation of the aspect specifies that: when the page is loaded from a persistent storage device, the page state is changed to semi-valid in the database node and a valid log snapshot number (LSN) is assigned to the page; a list of pages having the semi-valid page state and the valid LSN of the page is included in the snapshot request; and invalid pages and valid LSNs identified in the snapshot response are received at the database node.

[0015] Optionally, in the foregoing aspect, another implementation of the aspect specifies that the log node identifies pages that are invalid for the database node by comparing the valid LSN of the page with the latest LSN of the page.

[0016] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that: when the page is loaded from the persistent storage device, the page state is changed to half-valid; if the page is indicated as changed in the list of changed pages, the page state is changed from half-valid to invalid; if the page is a newly loaded page in the database node and the page is not indicated as changed in the list of changed pages, the page state is changed from half-valid to valid.

[0017] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that: if the page is indicated as changed in the list of changed pages, the page state is changed from valid or partially valid to invalid; when the page is loaded from persistent storage, the page state is changed from invalid to partially valid.

[0018] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that: records of database transactions committed before the initiated database transaction are sent from the log node to the persistent storage device; and when the records have been sent, the snapshot response is sent.

[0019] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that the log node responds to the snapshot request by updating the hash table of the cached pages of the database node.

[0020] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that: the database instance among the plurality of database instances is stored in the cache memory of the database node, and the initiation of the database transaction includes the processor of the database node initiating the database transaction.

[0021] According to another aspect of this disclosure, a log node for a database system is provided. The log node includes a memory and one or more processors, the memory including instructions, and the one or more processors communicating with the memory. The one or more processors execute the instructions to: receive a snapshot request from a database node of the database system, the snapshot request including a list of pages replaced or newly loaded in a cache memory of the database node; and send a snapshot response to the database node, the snapshot response including a snapshot of the database and a list of changed pages of the database instance.

[0022] Optionally, in the foregoing aspect, another implementation of the aspect provides one or more processors that execute instructions to: assign a valid log snapshot number (LSN) to the database transaction; identify pages invalid for the database node by comparing the valid LSN of the page with the latest LSN of the page; and include the identified invalid pages in the snapshot response.

[0023] Optionally, in any of the foregoing aspects, another implementation of the aspect provides one or more processors that execute instructions to: send records of executed database transactions from the log node to persistent storage; and send the snapshot response when the log node sends records of database transactions executed before the initiated database transaction to the persistent storage.

[0024] Optionally, in any of the foregoing aspects, another implementation of the aspect provides one or more processors that execute instructions to store a hash table of cached pages of the database node and update the hash table in response to the snapshot request. Attached Figure Description

[0025] Figure 1 A block diagram of a distributed database system provided as an example embodiment.

[0026] Figure 2 A block diagram of another distributed database system provided as an example embodiment.

[0027] Figure 3 A state diagram of an asynchronous cache consistency protocol provided as an example embodiment.

[0028] Figure 4 The example embodiment shows a timeline of database transactions and a list of partially valid pages for a distributed database system.

[0029] Figure 5 This is a state diagram illustrating the state of pages cached by database nodes in a distributed database system, as provided in an example embodiment.

[0030] Figure 6 An example of generating an invalid map for a database node in a distributed database system is shown in an exemplary embodiment.

[0031] Figure 7 A flowchart illustrating the processing of database transactions in a distributed database system, as provided in an example embodiment.

[0032] Figure 8 A block diagram of circuitry for implementing algorithms and execution methods for a client, server, and cloud-based resources, provided as an example embodiment. Detailed Implementation

[0033] The following is a detailed description in conjunction with the accompanying drawings, which are an integral part of the description and illustrate specific embodiments in which the invention can be practiced. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that other embodiments can be utilized, and structural, logical, and electrical modifications can be made without departing from the scope of the invention. Therefore, the exemplary embodiments described below are not intended to be limiting, and the scope of the invention is defined by the appended claims.

[0034] In one embodiment, the functions or algorithms described herein may be implemented in software. The software may include computer-executable instructions stored on a computer-readable medium or computer-readable storage device (e.g., one or more non-transitory memories or other types of hardware-based local or network storage devices). Furthermore, these functions correspond to modules, which may be software, hardware, firmware, or any combination thereof. Multiple functions may be performed in one or more modules as needed; the described embodiments are merely examples. The software may execute on a digital signal processor, ASIC, microprocessor, or other type of processor running on a computer system, such as a personal computer, server, or other computer system, thereby transforming such a computer system into a specially programmed machine.

[0035] In traditional distributed database systems, message passing between multiple database instances is used to synchronize the databases and maintain database consistency. Centralized management of database consistency is achieved by using a global buffer pool to maintain consistency across multiple database nodes. For example, central database management can include global message passing for performing global locking operations and global invalidation operations.

[0036] Figure 1 This is a block diagram of a distributed database system 100. The system includes database instance 1 and database instance 2. Each database instance can reside on database node 103 or database master node. Each database node can access shared data in persistent storage 107, which may include one or more servers. For simplicity, Figure 1 Only two database instances are shown in the diagram. A distributed database system can include dozens or hundreds of database instances. Each database instance includes a buffer pool 105, which can be a cache storage that holds copies of the shared data or the instance itself. When multiple database instances access the shared data, they need to maintain the consistency of the cache (or buffer pool); otherwise, they may read outdated data, leading to data inconsistency.

[0037] Figure 1 Three cross-network operations are illustrated for maintaining cache consistency among database instances. These operations synchronize across all database instances and affect all database instances. Synchronization operation 1 (SYNC1) 109 is a page registration operation. When a page is loaded into the local cache pool, the information is broadcast to all other database instances. For SYNC1, Figure 1This illustrates placing a page registration in a local buffer pool of database instance 1 and broadcasting it to database instance 2. Alternatively, registration can be loaded into a centralized database component (e.g., a global buffer pool). One of the replicas of the buffer pool can be designated as the global buffer pool. When a database node reads a page, the page is registered in the global buffer pool.

[0038] Synchronization operation 2 (SYNC2) 111 is a page invalidation operation. When a database transaction modifies a page, the transaction needs to invalidate the page's copy in the buffer pool of other database nodes before committing. "Commit" refers to the point in a transaction that updates data in persistent storage, where the data becomes visible to other database nodes. For SYNC2, Figure 1 The diagram illustrates a page that has been modified in database instance 2 and a page that has become invalid in database instance 1. In the global pool example, if another node is interested in the same page registered in the global pool, it will acquire a global lock for that page. The page is loaded from persistent storage and cached in a local buffer pool. When another member accesses the page, the two members need to negotiate the state of the global lock. If a member updates the page, that member will invalidate the copies in the other members' local buffer pools before committing the update transaction.

[0039] Synchronization operation 3 (SYNC3) 113 is a page reload operation. When a database node attempts to read an invalid page, a valid version of the page is loaded from another database node or the persistent storage. For SYNC3, Figure 1 The page loaded into database instance 1 is shown.

[0040] All three SYNC operations involve network communication and can introduce significant network overhead when executing database transactions. For example, SYNC operation 2 can be the critical path of a database transaction by directly blocking the commit process. These overhead transactions can become a major obstacle to increasing the throughput of multi-master database systems. The inventors have recognized the need to improve communication in distributed database systems.

[0041] An improved approach to achieving database consistency across database nodes is to perform operations such as page registration and page invalidation asynchronously, rather than synchronously across all database nodes. Consistency information is asynchronously delivered and cached until needed for read consistency and cache consistency. Instead of the SYNC1 page registration operation, cached pages are registered incrementally on demand, avoiding global operations that would hinder higher system throughput. Instead of the SYNC2 invalidation operation, cache invalidation is delayed until needed. As a result, the overhead associated with page registration is removed, and page invalidation is removed from the critical path of database transactions. For the SYNC3 page reload operation, the latest version of the page is retrieved from persistent storage on demand.

[0042] Figure 2 This is a block diagram of another distributed database system 200. The system supports compute-storage separation. The database system 200 includes one or more persistent storage devices 207 (e.g., servers), multiple database nodes 210, and log nodes 212. Database nodes or log nodes may include one or more processors and memory. The memory of the database node 210 includes a cache memory 214 for storing database instances. Database nodes receive queries (e.g., Structured Query Language or SQL queries) from users, or one or more operations among update, insert, delete, and create operations. Data required for the operations is provided through the cache memory of the database node, and if data in the cache memory is lost, it is provided through the persistent storage. Information required for cache consistency is asynchronously transmitted between the database node 210 and the log node 212.

[0043] Figure 3 State diagram 300 for the asynchronous cache consistency protocol. In 325, the processor of the database node 310 initiates a database transaction by sending a snapshot request to the log node 312. The snapshot request includes a list of pages that have been replaced or newly loaded in the cache memory of the database node.

[0044] In step 330, the log node 312 sends log records (Xlog) to persistent storage 307. The log records include database transactions committed by all database nodes prior to the current new database transaction. When a database transaction modifies a page, it creates a log record. When the database transaction commits, the log record is sent to the log node 312. The log node 312 retrieves the log record and sends it to the persistent storage 307, which creates a new version of the data. The persistent storage 307 receives and stores the log record from the log node 312. The persistent storage 307 generates database data by replaying the log records and reading page requests issued from the database nodes to the persistent storage.

[0045] In step 335, the log node 312 sends a snapshot response to the database node 310. The snapshot response includes a snapshot of the database. The snapshot is determined by log records flushed to the persistent storage. For example, if a total of 1,024,000 (1024K) log records are flushed to the persistent storage, the snapshot response includes a 1024K snapshot. The snapshot response also includes a list of modified pages of the database instance. In step 340, the processor of the database node 310 updates the state of the pages in the cache memory of the database node 310 based on the snapshot response and executes a database transaction. The operations of the database transaction are performed only on this returned snapshot. Any concurrent changes from other database nodes are not visible to this transaction.

[0046] In step 345, when the processor of database node 310 needs to access data of a page indicated as invalid in the snapshot response, database node 310 retrieves a version of the page from persistent storage 307. Persistent storage 307 applies the received log records and returns the latest version of the page.

[0047] return Figure 2 The database node may include a cache tracker component 224. The cache tracker component 224 may include software or firmware executed by the database node processor 216. The database node can use the cache tracker component 224 to track or trace the latest pages loaded from the persistent storage server and the replaced pages in the cache memory 214. For each database transaction, the cache tracker prepares two lists: one is a list of newly loaded pages, and the other is a list of pages replaced since the last snapshot was taken.

[0048] The log node 212 may include a global transaction management (GTM) component. The GTM component 222 may include software or firmware executed by the log node processor 220. The GTM component generates an incrementing log snapshot number (LSN). The LSN is used to maintain the snapshots. When a database transaction starts, the database node can obtain snapshots from the log node as a list of active transactions. The log node 212 may also use the LSN to maintain a hash table to track cached pages in each of the database nodes.

[0049] Figure 4 A timeline 405 is shown for a database transaction or reader transaction (T1, T2...Tx). The log node assigns a reader LSN (reader LSN1, reader LSN2...reader LSNx) to each transaction. In some embodiments, the reader LSN includes a timestamp.

[0050] The log node 212 may include an invalid map component 226. The log node can use the invalid map component 226 to generate an invalid map as a list of invalid pages for a specific database node. The invalid pages are pages cached by the database node that have been modified by other database nodes. After committing these changes and sending log records of these changes to the persistent storage 207, the invalid map is returned to the specific database node. The database node may include a page invalidation component 228. The database node can use the page invalidation component 228 to invalidate cached pages based on the invalid map. When the database node 210 wants to access an invalid page, it loads the corresponding page from the persistent storage 207.

[0051] Pages cached in the database node can be valid, partially valid, or invalid. A page is valid if it is loaded by the database from persistent storage and is not listed in the invalid map subsequently received from the log node. If a page is included in the invalid map, it is marked as invalid. A page is marked as partially valid when it is first loaded into the cache.

[0052] Figure 5A state diagram 500 shows the state of pages cached by database nodes in a database system. In 505, when another database node modifies the page and the modification is committed to the log node, a valid page 550 becomes invalid. The invalidity diagram received from the log node includes information about the modified page. In 510, when the database node attempts to access an invalid page, and the database node loads the corresponding page from persistent storage due to its invalid state, the invalid page 555 becomes partially invalid. In 515, a partially invalid page 560 can become valid, and in 520, a partially invalid page can become invalid.

[0053] The database node uses a list of modified pages (invalid map) received from the log node to track whether a partially valid page becomes valid or invalid. When the database node receives a snapshot response from the log node, it includes either an invalid map or a list of modified pages. If the list includes a partially valid page, the page was recently modified, and its state changes to invalid. If the partially valid page is included in the database node's list of newly loaded pages and is not in the returned list of modified pages, its state changes to valid. Pages that have become valid or invalid are removed from the partially valid list.

[0054] The cache tracker component 224 of the database node maintains a list of half-valid pages. Figure 4 An example of a semi-valid page list 410 is shown. This semi-valid page list consists of list pairs (Px: valid LSN) including a page number and a valid LSN. This valid LSN can be the same or different for the list pairs and is the latest reader LSN when the database node begins loading the corresponding page. This means that the loaded page is valid for all snapshots with LSNs less than this valid LSN. This is guaranteed in the asynchronous cache consistency protocol by having the log node check that all committed log records have been sent or flushed to persistent storage when any snapshot (and its reader LSN) returns to the database node.

[0055] When a database node initiates a database transaction, it uses a semi-valid page list to construct a list of newly loaded pages to be included in the snapshot request. This list includes only those newly loaded pages. Any pages selected by previous transactions are excluded from the list of newly loaded pages.

[0056] In the event of a network failure, snapshot request messages or snapshot response messages may be lost and resent, or pages that are about to become valid may be reloaded and marked as partially valid. The database node can compare the valid LSN of the pages in the partially valid page list with the reader LSN of the active database transaction. If the valid LSN of the partially valid page list is greater than or equal to the reader LSN, the page remains partially valid. If the valid LSN is less than the reader LSN, the page in the list is invalid.

[0057] To generate an invalid map, the invalid map component 226 of the log node 212 maintains a hash table of cached pages for each database node. The invalid map component 226 may also maintain a separate list of pages modified for each database node since the last reader LSN of that database node. Pages updated by other database nodes are appended to this separate list of pages. It should be noted that because multiple updates occur at different times, a page may have multiple entries in the list.

[0058] When a database node sends a snapshot request with newly loaded pages, the list of newly loaded pages includes the valid LSNs of these pages. When the log node receives the list, it first inserts these pages into the database node's hash table and then generates a reader LSN for the database transaction. The invalid map component 226 identifies pages with valid LSNs smaller than the reader LSNs and adds these pages to the invalid map. Pages with valid LSNs greater than or equal to the reader LSNs become valid.

[0059] Figure 6 An example of generating an invalid map for a database node is shown. For database (DB) node 1, timeline 605 of database transactions (T1, T2, T3, T4) is shown. For database transaction T3, the reader LSN is LSN3, and the list of newly loaded pages for DB1 in T3 is {(P1, LSN2), (P2, LSN2)}. Figure 6 The list of semi-valid pages 610 for DB node 1 is also shown. The elements of the list of semi-valid pages 610 include page identifiers (page IDs P1, P2, P3...P5) and the minimum valid LSN of the corresponding page.

[0060] Figure 6The document also shows a list of invalid pages maintained by the log node for DB node 1 and DB node 2. The invalid page list 615 for DB node 1 is a list of recently modified pages cached by DB node 1. Elements of the list include the page ID and the commit LSN. The invalid map component 226 compares the list of partially valid pages with the list of invalid pages to identify which partially valid pages have changed between the valid LSN and the reader LSN.

[0061] For page P2, since the valid LSN of the semi-valid page list is greater than or equal to the commit LSN of the invalid page list, the cached version of P2 in DB node 1 is valid. Other versions of P2 in other DB nodes are not visible to DB node 1. For page P1, since the valid LSN of the semi-valid page list 610 is less than the commit LSN of the invalid page list 615, the cached version of P1 in DB node 1 may be older, and the version of P1 in DB node 1 is invalid.

[0062] The log node 212 uses the reader LSN to determine which list elements in the invalid page list should be checked. For example, since the reader LSN of T3 is LSN3, only the first three elements of the invalid page list, P2, P1, and P5, are checked. Elements with valid LSNs greater than LSN3 (such as LSN4 and LSN5) are not checked because the page changes are not visible to transaction T3.

[0063] When a database node sends a list of newly loaded pages to the log node via the snapshot request, the database node can include a minimum valid LSN in the list of newly loaded pages. The log node can use this valid LSN to remove items from the list of invalid pages. If a database node finds its last reader LSN too old (e.g., because no database transactions have been sent in the database node), the database node can send a list of newly loaded pages with an unrealistically large valid LSN. In this way, the database node can proactively truncate the list of modified pages maintained by the log node. This request can be issued periodically to avoid missing some genuine snapshot requests and failing to process the corresponding list of newly loaded pages, thus identifying an invalid page map. If the list of modified pages is not truncated, the loss of the list of newly loaded pages will result in false invalid pages, triggering future accesses that will always result in loading data from persistent storage.

[0064] Figure 7 for Figure 2The flowchart 700 illustrates the process of handling database transactions in a distributed database system. In step 705, a database node that sends a snapshot request to the log node initiates a database transaction. The snapshot request is for obtaining a database snapshot and a reader LSN. The request may include two lists: a list of pages the database node is requesting to be newly added and a list of pages the database node is requesting to be replaced.

[0065] At 710, the log node updates the hash table for each database node's cached pages. At 715, the log node sends records of database transactions that occurred before the database snapshot request to persistent storage. Before returning the snapshot and the list of invalid pages to the requesting database node, the log node checks whether the log records of previously committed transactions have been flushed or sent to the persistent storage server.

[0066] In step 720, the log node generates an invalid map (a list of invalid pages) and returns the invalid map and the snapshot to the database node. In step 725, when the database node reads a page indicated as invalid in the invalid map, the database node retrieves the page from persistent storage. In step 730, the database node commits the database transaction. Upon commit, a log record is sent to the log node.

[0067] In a multi-version database system, a transaction takes a snapshot of the database, and the transaction's operations only apply to that snapshot. Any concurrent changes occurring in the system are invisible to this transaction. The transaction accesses cached data in the database node or data in persistent storage. An asynchronous cache consistency protocol guarantees that the accessed data is in the snapshot taken by the transaction or a more recent snapshot.

[0068] All pages in the cache are the latest version or snapshot version. No committed changes are made to the cached pages before the transaction retrieves the snapshot. Some cached pages become invalid, but other transactions will not load those cached pages until the snapshot is retrieved. The transaction or another transaction with an older snapshot marks those cached pages as invalid. If the transaction accesses these pages, a new version of the cached page is retrieved from persistent storage.

[0069] If other database nodes load pages from the first database node's cache before a transaction occurs on the first database node, these loaded pages are added to the newly loaded list on those other database nodes. A returned invalid map of modified pages can verify whether the page is valid or invalid. If invalid, the first database node retrieves the page from persistent storage. Persistent storage always returns the latest version of the page because the log node refreshes or sends log records of previously committed transactions before returning the snapshot to the database node.

[0070] During a transaction on the first database node, other database nodes may load some pages from the cache of the first database node. If these pages are loaded within the time between the database node sending the snapshot request and the log node returning the snapshot response, these pages are partially valid. If the valid LSN of the page is less than (earlier than) the reader LSN of the snapshot, these pages are retrieved from persistent storage.

[0071] If these pages are loaded after the time the snapshot response (including the snapshot and the reader LSN) is sent, these pages are partially valid, but the valid LSN of the page is greater than or equal to (but not earlier than) the reader LSN of the snapshot, and the page can be used in the transaction. Therefore, the asynchronous cache consistency protocol can guarantee access to correct data in database transactions.

[0072] As can be seen, cache consistency is maintained across multiple master database nodes. The asynchronous cache consistency protocol defers cache invalidation operations until needed, registers cached pages incrementally on demand without global locking, and retrieves the latest version of the page from persistent storage as needed. Compared to traditional synchronous consistency methods, asynchronous cache consistency methods reduce the network interactions required for transaction commits and data access. This reduces network bottlenecks and improves throughput.

[0073] Figure 8 The block diagrams provided for example embodiments illustrate circuitry for one or both of a database node and a log node for performing a method. It is not necessary to use all components in every embodiment. For example, the database node and the log node may each use different sets of components.

[0074] An example computing device in the form of a computer 800 may include a processing unit 802, a memory 803, a removable memory 810, and a non-removable memory 812. Although the example computing device is illustrated and described as the computer 800, the computing device may take different forms in different embodiments. For example, the computing device may be a smartphone, tablet, smartwatch, or other computing device, including... Figure 8 The same or similar elements shown and described. Devices such as smartphones, tablets, and smartwatches are generally referred to as mobile devices or user devices. Furthermore, although various data storage elements are described as part of computer 800, the memory may also include, or optionally include, cloud-based memory accessible via a network (e.g., the Internet or server-based memory).

[0075] The memory 803 may include volatile memory 814 and non-volatile memory 808. The cache memory may be implemented using one or both of the volatile memory 814 and the non-volatile memory 808. The computer 800 may include or have access to a computing environment comprising various computer-readable media, such as a computing environment including the volatile memory 814, the non-volatile memory 808, the removable memory 810, and the non-removable memory 812. Computer storage includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), digital versatile disc (DVD) or other optical disc storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium capable of storing computer-readable instructions.

[0076] The computer 800 may include or have access to a computing environment including input 806, output 804, and communication connection 816. The output 804 may include a display device (e.g., a touchscreen), which may also be used as an input device. The input 806 may include one or more of the following: a touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated into the computer 800 or coupled to the computer 800 via a wired or wireless data connection, and other input devices. The computer can operate in a network environment using the communication connection to connect to one or more remote computers, such as a database server. The remote computer may include a personal computer (PC), server, router, network PC, peer device, or other public network node. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular network, Wi-Fi, Bluetooth, or other networks. According to one embodiment, various components of the computer 800 are connected to a system bus 820.

[0077] Computer-readable instructions (e.g., program 818) stored on a computer-readable medium may be executed by a processing unit 802 of the computer 800. In some embodiments, the program 818 includes software that, when executed by the processing unit 802, performs consistent operation according to any of the embodiments included herein. Hard disk drives, CD-ROMs, and RAM are some examples of articles of manufacture that include non-transitory computer-readable media (e.g., storage devices). The terms computer-readable media and storage devices do not include carrier waves because carrier waves are too transient. Storage may also include networked storage, such as storage area networks (SANs). The computer program 818 may be used to cause the processing unit 802 to execute one or more of the algorithms described herein.

[0078] In an example embodiment, the computer 800 includes: a startup module that initiates a database transaction by sending a snapshot request from a database node to a log node, the snapshot request including a list of pages that have been replaced or newly loaded in the cache of the database node; a receiving module that receives a snapshot response from the log node at the database node, the snapshot response including a snapshot of the database and a list of changed pages of the database instance; and an update module that updates the state of the pages in the cache of the database node according to the snapshot response and executes the database transaction. In some embodiments, the computer 800 may include other modules or additional modules for performing any or a combination of steps described in the embodiments. Furthermore, any additional or alternative embodiments or aspects of the methods as shown in any of the figures or recounted in any of the claims may also be contemplated to include similar modules.

[0079] Although several embodiments have been described in detail above, other modifications are possible. For example, the logical flow depicted in the figures does not require the specific or sequential order shown to achieve the desired result. Additional steps may be provided to the described flow or steps may be removed from the described flow, and additional components may be added to the described system or components may be removed from the described system. Other embodiments are within the scope of the appended claims.

Claims

1. A database system, characterized in that, include: Persistent storage devices; A log node, the log node including a memory and a processor; Multiple database nodes are configured, each including a cache memory for storing database instances and a processor for initiating database transactions by sending a snapshot request to a log node, the snapshot request including a list of pages that have been replaced or newly loaded in the cache memory; the pages cached in the database node are in a state of one or more of valid, partially valid, and invalid; the list of newly loaded pages is constructed based on a list of partially valid pages; the partially valid pages are assigned valid log snapshot numbers (LSNs); the log node processor is configured to send a snapshot response to the database node and identify pages invalid for the database node by comparing valid LSNs with the latest LSN of the page; the identified invalid pages are included in the snapshot response, which includes a snapshot of the database and a list of modified pages of the database instance; wherein information required for cache consistency is asynchronously transmitted between the database node and the log node. The database node processor is used to update the state of the page in the cache storage according to the snapshot response and execute the database transaction; the operation of the database transaction applies to the snapshot of the database.

2. The database system of claim 1, wherein, The log node processor is used to send the valid LSN in the snapshot response.

3. The database system of claim 1 or 2, wherein, The database node processor is used for: When the page is loaded from the persistent storage device, the page state is changed to half-valid. If the page is indicated as changed in the list of changed pages, the page status is changed from half-valid to invalid. If the page is a newly loaded page and the page is not indicated as changed in the list of changed pages, then the page status is changed from half-valid to valid.

4. The database system of claim 3, wherein, The database node processor is used for: If the page is indicated as changed in the list of changed pages, the page status is changed from valid or partially valid to invalid. When the page is loaded from the persistent storage device, the page state is changed from invalid to partially valid.

5. The database system according to any one of claims 1 to 4, characterized in that, The log node processor is used for: The records of executed database transactions are sent from the log node to the persistent storage device; The snapshot response is sent when the log node sends records of database transactions executed before the initiated database transaction to the persistent storage device.

6. The database system of claim 5, wherein, The persistent storage device is used to generate the database snapshot using the records of the executed database transactions.

7. The database system according to any one of claims 1 to 6, characterized in that, The log node storage is used to store a hash table of cached pages of the database node and to update the hash table in response to the snapshot request.

8. A computer-implemented method for achieving database consistency in a system containing multiple database instances, characterized in that, The method includes: A database transaction is initiated by sending a snapshot request from the database node to the log node. The snapshot request includes a list of pages that have been replaced or newly loaded in the cache of the database node. The cached pages in the database node are in a state of one or more of valid, partially valid, and invalid. The list of newly loaded pages is constructed based on the list of partially valid pages. The partially valid pages are assigned a valid log snapshot number (LSN). The log node receives a snapshot response from the database node, the snapshot response including a snapshot of the database and a list of modified pages of the database instance; wherein, information required for cache consistency is asynchronously transmitted between the database node and the log node; the snapshot response also includes: the log node identifying pages that are invalid for the database node by comparing valid LSNs with the latest LSN of the page; The database transaction is executed based on the snapshot response, updating the state of the page in the cache of the database node and performing the database transaction operation on the snapshot of the database.

9. The method of claim 8, wherein, Updating the state of the page in the cache storage includes: When the page is loaded from persistent storage, the page state is changed to half-valid. If the page is indicated as changed in the list of changed pages, the page status is changed from half-valid to invalid. If the page is a newly loaded page in the database node, and the page is not indicated as changed in the list of changed pages, then the page status is changed from half-valid to valid.

10. The method of claim 9, wherein, Updating the state of the page in the cache storage includes: If the page is indicated as changed in the list of changed pages, the page status is changed from valid or partially valid to invalid. When the page is loaded from persistent storage, the page state is changed from invalid to partially valid.

11. The method according to any one of claims 8 to 10, characterized in that, The method includes: The records of database transactions committed before the initiated database transaction are sent from the log node to the persistent storage device; Send the snapshot response when the record has been sent.

12. The method according to any one of claims 8 to 11, characterized in that, The method includes: the log node responding to the snapshot request and updating the hash table of the cached pages of the database node.

13. The method according to any one of claims 8 to 12, characterized in that, The method includes: storing a database instance from the plurality of database instances in the cache storage of the database node, and initiating the database transaction, including the processor of the database node initiating the database transaction.

14. A log node device of a database system, characterized by The log node device includes: Memory, the memory including instructions; One or more processors, which communicate with the memory, execute the instructions to: A snapshot request is received from a database node of the database system. The snapshot request includes a list of pages that have been replaced or newly loaded in the cache storage of the database node. The status of the pages cached in the database node includes one or more of the following: valid, partially valid, and invalid. The list of newly loaded pages is constructed based on the list of partially valid pages. The partially valid pages are assigned a valid log snapshot number (LSN). A snapshot response is sent to the database node, and pages invalid for the database node are identified by comparing valid LSNs with the latest LSN of the page; the identified invalid pages are included in the snapshot response, which includes a snapshot of the database and a list of modified pages of the database instance; wherein, information required for cache consistency is asynchronously transmitted between the database node and the log node.

15. The log node device of claim 14, wherein, The one or more processors execute the instructions to: The records of executed database transactions are sent from the log node to the persistent storage device; The snapshot response is sent when the log node sends records of database transactions executed before the initiated database transaction to the persistent storage device.

16. The log node device of claim 14 or 15, wherein, The one or more processors execute the instructions to store a hash table of cached pages of the database node and update the hash table in response to the snapshot request.

Images