Indexing cloud database logs in distributed database storage

Abstract

This provides a method for indexing cloud database logs in distributed database storage. In the example implementation, a transaction log index is used to index log records. The transaction log index includes an in-memory transaction log index and an on-disk transaction log index. Log records are temporarily indexed in the in-memory transaction log index until the storage server can ensure that the corresponding log record is stored and that there are no missing log records in the sequence, and possibly based on other correctness and performance considerations.

Description

Technical Field

[0001] The examples in this disclosure generally relate to database management systems, and more specifically, but not in a limiting way, to systems for generating indexes for transaction logs in database management systems. Background Technology

[0002] A relational database is a database that organizes data into one or more tables (or "relationships") consisting of columns and rows. The database management system used to maintain relational databases is called a relational database management system (RDBMS). To ensure efficient and accurate operation, an RDBMS adheres to a set of properties, including atomicity, consistency, isolation, and durability (ACID). Atomicity is a database property that ensures that a series of database operations either all occur or none occur. The atomicity property ensures that the effects of a committed (user) transaction are durable, even if the system crashes after the transaction is committed.

[0003] Databases typically use fixed-size pages to actually store database state and user data, with these database pages residing in page storage. Atomicity is achieved by logging every change made during a database transaction as a log entry in the transaction log, enabling full recovery in the event of a database failure. For example, if a series of transactions affects the availability of items in inventory, updating the transaction log each time a transaction is executed means logging the first transaction that increases the item quantity by three, then logging the second transaction that decreases the item quantity by two, and so on. The transaction log can include log entries of variable size. Each log entry can describe the actual transaction to be performed on one or more database pages (although in some cases, log entries may also describe operations not associated with a database page). Each log entry can also be identified by a log entry identifier. The transaction log is stored in log storage. The database ultimately applies the effects of the transactions in the transaction log to the database pages in page storage.

[0004] It should be noted that although some of the example implementations described herein involve RDBMS, the technologies and solutions described herein are not limited to RDBMS and can be applied to other types of database management systems. Summary of the Invention

[0005] In the example implementation, a transaction log index is used to index log records. The transaction log index includes an in-memory transaction log index and an on-disk transaction log index. Log records are temporarily indexed in the in-memory transaction log index until the storage server can ensure that the corresponding log record is stored and that there are no missing log records in the sequence, and possibly based on other correctness and performance considerations. Attached Figure Description

[0006] The accompanying drawings are merely examples of the contents of this disclosure and should not be construed as limiting its scope.

[0007] Figure 1 This is a block diagram illustrating a system utilizing an intelligent distributed storage device according to an example implementation.

[0008] Figure 2 This is a sample transaction log based on the example implementation.

[0009] Figure 3 This is a block diagram illustrating a system utilizing an intelligent distributed storage device and employing a transaction log index according to an example implementation.

[0010] Figure 4 This is a block diagram illustrating an on-disk transaction log index according to an example implementation.

[0011] Figure 5 This is a flowchart illustrating a method for indexing database log identifiers in a database store according to an example implementation.

[0012] Figure 6 It is a block diagram illustrating the software architecture that can be installed on any one or more of the devices described above.

[0013] Figure 7 It is a block diagram of a machine within which instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0014] The following description includes systems, methods, techniques, instruction sequences, and computer program products that implement illustrative examples of the present disclosure. In the following description, numerous specific details are set forth for illustrative purposes to provide an understanding of various examples of the subject matter of the invention. However, it will be apparent to those skilled in the art that examples of the subject matter of the invention can be practiced without these specific details. Generally, well-known examples of instructions, protocols, structures, and techniques are not necessarily shown in detail.

[0015] Database Platform as a Service (dbPaaS) is a cloud-based service model that provides users with access to database systems without requiring them to manage the underlying infrastructure, hardware, or software updates. The database system is entirely managed by the cloud service provider.

[0016] In traditional database systems, whether distributed or not, the database engine controls the management of the database transaction log, the storage of the database transaction log in the log store, and the management of page updates in the database page store. In such systems, the storage device itself does not contain any important functions other than acting as a repository for data.

[0017] However, by offloading certain functions from the database engine to an intelligent storage device, significant functional, scalability, and performance benefits can be achieved, where the intelligent storage device has the capability to perform these functions independently. In some implementations, transaction log writing, log storage management, and page storage management are offloaded from the database engine to such an intelligent storage device. Multiple database engines can use the same intelligent storage device. In some cases, one database engine will be considered the primary database engine and will have the capability to read from and write to the intelligent storage device, while other database engines will be considered secondary database engines and will only have the capability to read from but not write to the intelligent storage device. However, in some systems, there is no distinction between the primary and secondary database engines, and multiple database engines can read from and write to the intelligent storage device.

[0018] Furthermore, data within the intelligent storage device itself can be geographically replicated across multiple geographic locations to enhance the availability, resilience, and performance of distributed database systems.

[0019] However, technical challenges arise when offloading the responsibility of maintaining the underlying storage infrastructure from the database engine. The database engine generates log records identified by log record identifiers and writes them to distributed storage. In some implementations, this numerical identifier is a log sequence number (LSN). Each LSN identifies a corresponding log record and can therefore be used to retrieve the desired log record when needed. This is true even if the log records are stored in a distributed manner. The identifier does not have to be an LSN, but there is some degree of "ordering" for the identifier, allowing it to be sorted sequentially, for example, when the identifier is a sequence number.

[0020] Furthermore, some log records may be missing for various reasons. Various consumers use programs that require retrieving the database transaction logs in sorted order without any missing records.

[0021] Therefore, in the example implementation, the storage server hosting the log storage is responsible for ensuring that there are no missing log records in its transaction log, and that log records can be retrieved in sorted order based on their respective identifiers.

[0022] Figure 1 This is a block diagram illustrating a system 100 utilizing an intelligent distributed storage device according to an example implementation. Client 104 interacts with database engine 106, for example, by issuing create, read, update, and delete requests to database engine 106.

[0023] Database engine 106 processes transactions requested by client 104 and generates transaction logs that record these transactions. Database engine 106 sends these transaction logs to storage servers in the intelligent distributed storage device 102, such as storage server 108A. Storage server 108A is responsible for persisting the transaction logs in log storage 110A and applying the transaction logs to page storage 112A at appropriate times. Additional distributed storage servers 108B and 108C may exist in the intelligent distributed storage device 102, each with its own corresponding log storage 110B and 110C, and page storage 112B and 112C.

[0024] It should be noted that although only a single log store and a single page store are depicted within each storage server, in reality, many different log stores and page stores can exist within a single storage server.

[0025] Technical problems may arise when a storage server misses a sequence of log records from its log store. These problems may occur because database engine 106 generates batch log records for each storage server (e.g., storage server 108A), and these batch log records may be sent, received, or processed out of order at storage server 108A. Furthermore, storage server 108A may also miss a batch, for example, if it is temporarily unavailable or if the batch is rejected due to resource limitations.

[0026] Figure 2This is example transaction log 200. Here, transaction log 200 contains many log records. These log records, however, may be stored out of order. Thus, for example, a log record with a log record identifier in the range of 5000 to 10000 (depicted as reference numeral 202) is stored before a log record with a log record identifier in the range of 1000 to 2000 (depicted as reference numeral 204). Similar problems occur with other ranges of log record identifiers 206, 208, 210, and 212.

[0027] The out-of-order storage of log records and / or the absence of one or more log records for transaction log consumers (e.g.) Figure 1 This presents a problem for transaction log consumer 114. For example, if transaction log consumer 114 needs to read transaction log records sequentially from log record with log record identifier 5000 to log record with log record identifier 5010, it is likely that the following situation may occur: the log records in the sequence have not yet been all stored in their corresponding log stores, or even if they are, they are stored sequentially.

[0028] In the example implementation, transaction log indexes are used to index log records. These indexes include an in-memory transaction log index and an on-disk transaction log index. Once log records sent by the database engine have been persisted to the log store by the storage server, the log record identifiers are indexed in the in-memory transaction log index. The in-memory transaction log index is used until the storage server can ensure that the range of indexed log records is complete and that no log records are missing.

[0029] The process of appropriately writing a portion of the in-memory transaction log index to the on-disk transaction log index (referred to as a "spill") can be performed asynchronously to ensure that the latency of log write operations in Database Engine 106 remains low, which is desirable for performance and scalability. The spill is performed based on correctness and performance considerations. Before the spill occurs, the in-memory transaction log index allows consumers to read a range of log records in sorted order. The on-disk transaction log index has no missing log records.

[0030] Figure 3 This is a block diagram illustrating a system 300 utilizing an intelligent distributed storage device 302 and employing a transaction log index according to an example implementation. System 300 is similar to... Figure 1 In system 100, client 304 interacts with database engine 306, for example, by issuing create, read, update, and delete requests. Database engine 306 processes the transactions requested by client 304 and generates transaction logs that record these transactions. Database engine 306 sends these transaction logs to a storage server, such as storage server 308A, of intelligent distributed storage device 302. Here, however, storage server 308A maintains an in-memory transaction log index 310A and a disk-based transaction log index 312A. These indexes are used to index the contents of disk-based log storage 314A.

[0031] Log records are indexed by the in-memory transaction log index 310A after they have been persisted to the log store. Therefore, for the reasons stated above, this could mean that the in-memory transaction log index 310A indexes a range of incomplete log records. On the other hand, the disk-based transaction log index 312A stores a complete range of log records, with no missing records within that range. In other words, while the in-memory transaction log index 310A may be missing one or more log records in a sequence, the disk-based transaction log index 312A in the disk-based log store 314A does not have any missing log records in any sequence. Therefore, the in-memory transaction log index 310A acts as a temporary index for log records until the range to be overflowed is complete, and facilitates asynchronous updates to the disk-based transaction log index 312A.

[0032] In some example implementations, the in-memory transaction log index 310A may point only to the persistent transaction log stored in the disk-based log storage 314A. In other example implementations, the in-memory transaction log index 310A points to both the persistent transaction log in the disk-based log storage 314A and the transaction log maintained in memory.

[0033] The term "in-memory" here refers to data storage maintained in volatile memory, such as system memory, like random access memory (RAM). "Disk-based" here refers to data storage maintained in non-volatile memory, such as on a hard disk drive or solid-state drive.

[0034] When no missing log record exists in the in-memory transaction log index 310A, the sequence of log records in the in-memory transaction log index 310A is asynchronously overflowed to the on-disk transaction log index 312A. Depending on correctness and / or performance requirements, there may be additional considerations regarding the timing of the overflow. As part of the overflow operation, a range of log records can only be removed from the in-memory transaction log index after the on-disk index has already indexed that range. Note that in this example implementation, database engine 306 is the primary database engine, while database engines 318 and 320 are secondary.

[0035] There are several ways to construct an on-disk transaction log index. In the example implementation, a level 2 transaction log index is used. However, other types of transaction log indexes, such as level 1 and level 3, can be used.

[0036] In addition, with Figure 1 Similarly, the intelligent distributed storage device 302 can maintain multiple storage servers, such as storage servers 308A, 308B, and 308C. Each storage server has its own in-memory transaction log indexes 310A, 310B, and 310C, on-disk transaction log indexes 312A, 312B, and 312C, disk-based log storage 314A, 314B, and 314C, and page storage 316A, 316B, and 316C.

[0037] Figure 4 This is a block diagram illustrating an on-disk transaction log index 400 according to an example implementation. Here, the root node 402 contains entries 404A, 404B, 404C, ..., 404N, which identify leaf nodes 405A, 405B, 405C as entries with a given range. For example, root node entry 404A identifies leaf node 405A, which is responsible for log entry identifiers 1000 to 10000. Each leaf node entry (e.g., 408, 416, 420, 422) is also responsible for a range of log entry identifiers and contains ordered entries. It should be noted that these are merely examples of data structures that can be used, and the same objective can be achieved using data structures of other formats.

[0038] It should also be noted that although the organization of the on-disk transaction log index 400 may look similar to a B-tree or... It uses a tree (B+ tree) organization, but it is different. More specifically, in the on-disk transaction log index 400, it is not necessary to use a tree like in a B-tree or... Nodes can be split or merged as in a tree. Since the maximum size of the log storage is fixed, the size of the on-disk transaction log index can also be easily determined. Due to the simplicity of the data structure, dynamically expanding the root node and adding new leaf nodes is relatively straightforward. Furthermore, root nodes and / or leaf nodes can be reused in a steady state without needing to allocate or release them. They also do not require merging and compressing across multiple levels like in a Log Structured Merge (LSM) tree, and there is no input / output amplification.

[0039] The entries are ordered by the log record identifier in both the root and leaf nodes. This allows for binary search to quickly locate the log record entry of interest, at least when the log record entries are stored in a simple sequence. Other implementations are foreseeable, where log record entries are stored in ways other than a simple sequence, although even in these cases, there are efficient search algorithms available for locating the log record entry of interest.

[0040] There are several ways to store transaction log identifiers in the on-disk transaction log index. In the example implementation, a level 2 transaction log index is used. However, other types of transaction log indexes, such as level 1 and level 3, can be used.

[0041] Figure 5 This is a flowchart illustrating a method 500 for indexing database transaction log identifiers in a database store according to an example implementation. At operation 502, the storage server receives a request to write a set of log records related to multiple database transactions. These log records may be out of order, and even some log records within a given range may be missing.

[0042] At operation 504, the transaction log is asynchronously persisted to a disk-based log store. At operation 506, the log records are indexed in the in-memory transaction log index. At operation 508, it is determined whether the sequence of log records is now complete and without any missing records. It should be noted that additional considerations may exist at this step, such as whether the system is generally prepared to overflow the contents of the in-memory transaction log index to the disk-based transaction log index for correctness and / or performance reasons. If not, method 500 can loop back to operation 502 to process additional log records sent by the database engine.

[0043] If it is determined at operation 508 that the sequence of log records is currently free of any missing log records (and meets any correctness and / or performance considerations), then at operation 510, the on-disk transaction log index is updated to index all log record identifiers in the sequence. Then, at operation 512, the log record identifiers in the sequence are removed from the in-memory transaction log index.

[0044] It should be noted that operations 508, 510, and 512 can be executed in parallel / asynchronously with operations 502, 504, and 506, without needing to be executed afterward. Therefore, method 500 technically loops continuously around operations 502, 504, and 506, and operations 510 and 512 are executed when operation 508 detects that the sequence of log records is now ordered and no log records are missing. This is why operation 506 appears to flow back to operation 502 in addition to flowing to operation 508.

[0045] Figure 6 This is a block diagram 600 illustrating a software architecture 602 that can be installed on any one or more of the devices described above. Figure 6 This is merely a non-limiting example of a software architecture, and it will be understood that many other architectures can be implemented to facilitate the functionality described herein. Software architecture 602 can be derived from, for example... Figure 7 The hardware implementation of a computer system 700 includes one or more processors 702, memory 704, and I / O components 706. In this example, the software architecture 602 can be conceptualized as a stack of layers, each providing a specific function. For example, the software architecture 602 includes layers such as an operating system 604, libraries 606, frameworks 608, and applications 610. Operationally, according to some implementations, application 610 activates application programming interface (API) calls 612 through the software stack and receives messages 614 in response to API calls 612.

[0046] In various implementations, the operating system 604 manages hardware resources and provides public services. The operating system 604 includes, for example, a kernel 620, services 622, and drivers 624. In some implementations, the kernel 620 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 620 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 622 can provide other public services to other software layers. Drivers 624 can be responsible for controlling or interfacing with the underlying hardware. For example, drivers 624 may include display drivers, camera drivers, Bluetooth drivers, etc. ® Drives, flash drives, serial communication drives (e.g., Universal Serial Bus (USB) drives), Wi-Fi® Drivers, audio drivers, power management drivers, etc.

[0047] In some implementations, library 606 provides low-level public infrastructure that can be utilized by application 610. Library 606 may include system library 630 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 606 may include API library 632, such as: media library (e.g., a library supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics library (e.g., an OpenGL framework for rendering in two-dimensional (2D) and three-dimensional (3D) environments on a display), database library (e.g., SQLite providing various relational database functions), web library (e.g., WebKit providing web browsing functionality), etc. Library 606 may also include various other libraries 634 to provide many other APIs to application 610.

[0048] Depending on the implementation, framework 608 provides advanced public infrastructure that can be utilized by application 610. For example, framework 608 provides various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework 608 can provide a wide range of other APIs that can be utilized by application 610, some of which may be specific to a particular operating system or platform.

[0049] According to some examples, application 610 is a program that performs the functions defined in the program. One or more applications of application 610 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language).

[0050] Some examples are described herein as comprising logic or multiple components, modules, or mechanisms. Modules can constitute software modules (e.g., (1) code implemented on a non-transitory machine-readable medium or (2) for transmitting signals) or hardware-implemented modules. Hardware-implemented modules are tangible units capable of performing certain operations and can be configured or arranged in a certain way. In the examples, one or more computer systems (e.g., standalone, client, or server computer systems) or one or more processors can be hardware-implemented modules configured by software (e.g., an application or application portion) to operate to perform certain operations as described herein.

[0051] In various examples, hardware-implemented modules can be implemented mechanically or electronically. For example, a hardware-implemented module may include a dedicated circuit system or logic permanently configured (e.g., as a dedicated processor, such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented module may also include programmable logic or circuit systems temporarily configured by software to perform certain operations (e.g., as included within a general-purpose processor or other programmable processor). It will be understood that cost and time considerations can drive decisions regarding whether to mechanically implement a hardware-implemented module as a dedicated and permanently configured circuit system or as a temporarily configured circuit system (e.g., configured by software).

[0052] Therefore, the term "hardware-implemented module" should be understood to encompass tangible entities, i.e., entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily or provisionally configured (e.g., programmed) to operate in a particular manner and / or perform the specific operations described herein. Consider examples of hardware-implemented modules being provisionally configured (e.g., programmed), where each of the hardware-implemented modules does not need to be configured or instantiated at any given time. For example, in the case where the hardware-implemented modules include a general-purpose processor configured using software, the general-purpose processor can be configured as different hardware-implemented modules at different times. Thus, software can configure the processor to constitute a specific hardware-implemented module at one time and different hardware-implemented modules at different times.

[0053] Hardware-implemented modules can provide information to and receive information from other hardware-implemented modules. Therefore, the described hardware-implemented modules can be considered communicatively coupled. In the presence of multiple such hardware-implemented modules, communication can be achieved through signal transmission (e.g., via suitable circuitry and buses) connecting the modules. In an example where multiple hardware-implemented modules are configured or instantiated at different times, communication between such modules can be achieved, for example, by storing information in a memory structure accessible to the multiple modules and retrieving information from that memory structure. For example, one module can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another module can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware-implemented modules can also initiate communication with input or output devices and operate on resources (e.g., collections of information).

[0054] The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute modules of processor implementations that perform operations to execute one or more operations or functions. In some examples, modules mentioned herein may include processor implementation modules.

[0055] Similarly, the methods described herein can be implemented at least partially by processors. For example, at least some of the operations of the method can be performed by one or more processors or modules implemented by processors. The execution of some operations can be distributed across one or more processors, not only residing on a single machine but also deployed across multiple machines. In some examples, one or more processors may reside in a single location (e.g., a home environment, an office environment, or as a server cluster), while in other examples, processors may be distributed across multiple locations.

[0056] One or more processors may also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some of the operations may be performed by a set of computers (as an example of a machine including processors) that are accessible via a network (e.g., the Internet) and via one or more suitable interfaces (e.g., application programming interfaces (APIs)).

[0057] Examples can be implemented as digital electronic circuit systems or as computer hardware, firmware, software, or a combination thereof. Examples can be implemented using computer program products, such as computer programs tangibly embodied in an information carrier, such as computer programs in machine-readable media, for performing or controlling the operation of a data processing apparatus (e.g., a programmable processor, a computer, or a plurality of computers).

[0058] Computer programs can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, subroutines, or other units suitable for use in a computing environment. Computer programs can be deployed to execute on a single computer or on multiple computers located at a single point or distributed across multiple points and interconnected via a communication network.

[0059] Computing systems can include clients and servers. Clients and servers are typically geographically separated and usually interact via communication networks. The client-server relationship arises from computer programs running on their respective computers and having a client-server relationship with each other. In the example of deploying a programmable computing system, it will be understood that both hardware architecture and software architecture need to be considered. Specifically, it will be understood that the choice of implementing a particular function in permanently configured hardware (e.g., ASIC), temporarily configured hardware (e.g., a combination of software and a programmable processor), or a combination of permanently and temporarily configured hardware can be a design choice. The hardware architectures (e.g., machines) and software architectures that can be deployed in various examples are illustrated below.

[0060] Figure 7 This is a block diagram of a machine within which instructions can be executed to cause the machine to perform any or more of the methods discussed herein. In one example, the machine can be any of the devices described above. In alternative examples, the machine operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can run as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Furthermore, although only a single machine is shown, the term "machine" should also be considered to include any set of machines that, individually or jointly, execute a set (or more) of instructions to perform any or more of the methods discussed herein.

[0061] Example computer system 700 includes processors 702 (e.g., central processing unit (CPU), graphics processing unit (GPU), or both), main memory 704, and static memory 705 that communicate with each other via bus 708. Computer system 700 may also include a video display unit 710 (e.g., liquid crystal display (LCD) or cathode ray tube (CRT)). Computer system 700 also includes an alphanumeric input device 712 (e.g., keyboard), a user interface (UI) navigation device (cursor control device) 714 (e.g., mouse), a hard disk drive unit 716, a signal generation device 718 (e.g., speaker), and a network interface device 720.

[0062] The drive unit 716 includes a machine-readable medium 722 on which one or more sets of instructions and data structures (e.g., software) 724 are stored, which embody or are utilized by any one or more of the methods or functions described herein. The instructions 724 may also reside wholly or at least partially in main memory 704 and / or in processor 702 during execution by computer system 700, which also constitute the machine-readable medium. The instructions 724 may also reside in static memory 705.

[0063] Although the machine-readable medium 722 is shown as a single medium in the example, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more instructions or data instructions 724. The term "machine-readable medium" should also be considered to include: any tangible medium capable of storing, encoding, or carrying instructions 724, or any tangible medium capable of storing, encoding, or carrying data structures utilized by or associated with such instructions 724, said instructions 724 being executed by a machine and causing the machine to perform any one or more of the methods of the present invention. Therefore, the term "machine-readable medium" should be considered to include, but is not limited to, solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, which by way of example includes: semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0064] Instructions 724 can also be sent or received via network 726 using a transmission medium. Instructions 724 can be sent using network interface device 720 and any of many well-known transmission protocols (e.g., HTTP). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet, mobile phone networks, simple old-fashioned telephone (POTS) networks, and wireless data networks (e.g., Wi-Fi and Wi-Max networks). The term “transmission medium” should be considered to include any intangible medium capable of storing, encoding, or carrying instructions 724 executed by a machine and comprising digital or analog communication signals, or other intangible media facilitating communication by such software.

[0065] In various exemplary examples, one or more portions of network 726 may be an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, the Internet, a portion of the Internet, a portion of the PSTN, a Simple Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 726 or a portion of network 726 may include a wireless or cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling can implement any of various types of data transmission technologies, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimized (EVDO), General Packet Radio Service (GPRS), GSM Evolution Enhanced Data Rate (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other standards defined by various standards setting organizations, other telemetry protocols, or other data transmission technologies. Although examples have been described with reference to specific examples, it will be apparent that various modifications and changes can be made to these examples without departing from the broader spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The accompanying drawings, which form part of this document, illustrate specific examples of how the subject matter can be practiced in an illustrative and non-limiting manner. The examples shown have been described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other examples can be utilized and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. Therefore, this specific embodiment is not to be construed as limiting, and the scope of the various examples is defined only by the appended claims together with the full scope of their equivalents.

[0066] For convenience only, such examples of the subject matter of this invention may be referred to individually and / or generally by the term "invention" herein, and if more than one invention or inventive concept is disclosed in fact, it is not intended to voluntarily limit the scope of this application to any single invention or inventive concept. Therefore, while specific examples have been shown and described herein, it should be understood that any arrangement intended to achieve the same purpose may be substituted for the specific examples shown. This disclosure is intended to cover any and all adaptations or variations of the various examples. Combinations of the above examples and other examples not specifically described herein will be apparent to those skilled in the art upon review of the foregoing description.

[0067] An abstract of this disclosure is provided to comply with 37 CFR § 1.72(b), which requires that the reader be able to quickly determine the nature of the disclosure. The abstract is submitted on the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen in the foregoing detailed description, various features are combined in a single example for the purpose of simplification. This approach of the disclosure should not be construed as reflecting an intention that the claimed example requires more features than are expressly listed in each claim. Rather, as reflected in the appended claims, the subject matter of the invention lies in fewer than all features in a single disclosed example. Therefore, the appended claims are thus incorporated into the detailed description, wherein each claim is presented as an independent, separate example.

[0068] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” refer to the same thing and can be used interchangeably. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions 724 and / or data. Therefore, the terms should be considered to include, but are not limited to, solid-state memory, as well as optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term “signal medium” discussed below.

[0069] Instructions can be transmitted or received over a network using a transmission medium via a network interface device (e.g., a network interface component included in a communication component) and utilizing any of the many well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions can be sent or received using a transmission medium via coupling with various devices (e.g., peer-to-peer coupling). The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions for machine execution, and include digital or analog communication signals or other intangible media used to facilitate communication of such software. Therefore, the terms "transmission medium" and "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal.

[0070] The terms “machine-readable medium,” “computer-readable medium,” “device-readable medium,” and “machine storage medium” refer to the same thing and are used interchangeably in this disclosure. These terms are limited to include both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. For example, the embodiments described herein may be implemented using non-transitory media (e.g., non-transitory computer-readable media).

[0071] The implementation of the described subject may include, individually or in combination, one or more features as illustrated by examples below.

[0072] Example 1 is a system comprising: at least one processor; and a memory including instructions that, when executed by the at least one processor, cause the system to perform operations including: receiving a request to write a transaction log relating to a plurality of database transactions, the transaction log having a plurality of log records, each log record having an identifier; asynchronously storing the transaction log in a disk-based log store; indexing the log records in an in-memory transaction log index; updating the disk-based transaction log index in the disk-based log store in response to determining that no log record is missing in a sequence of log records; and deleting log records from the in-memory transaction log index in the sequence.

[0073] In Example 2, the subject of Example 1 includes the following: updates to the on-disk transaction log index are also performed in response to one or more additional correctness and / or performance considerations.

[0074] In Example 3, the subject matter of Examples 1 to 2 includes, wherein the update further includes: persisting all logging identifiers equal to or lower than a specific value to the on-disk transaction log index, while maintaining all logging identifiers higher than the specific value in the in-memory transaction log index, and wherein deleting logging identifiers includes: deleting logging identifiers equal to or lower than the specific value from the in-memory transaction log index.

[0075] In Example 4, the topics of Examples 1 through 3 include an on-disk transaction log index with two levels: a root node and multiple leaf nodes, wherein the root node contains root node entries, and each root node entry points to a leaf node containing log record identifiers within a specified range.

[0076] In Example 5, the subject of Example 4 includes, where each entry in the leaf node identifies the range of log records in the disk-based log storage.

[0077] In Example 6, the topics of Examples 4 and 5 include reusing existing root and leaf nodes without allocating or freeing root or leaf nodes.

[0078] In Example 7, the subject of Example 6 includes the following: no splitting or merging operations are performed on the nodes of the transaction log index on disk.

[0079] In Example 8, the topics of Examples 6 and 7 include situations where merge or compress operations are not performed across the levels of the on-disk transaction log index, and there is no amplification of input or output operations in the on-disk transaction log index.

[0080] Example 9 is a machine storage medium having instructions contained thereon, the instructions being executable by a machine's processor to perform operations including: receiving a request to write a transaction log relating to multiple database transactions, the transaction log having multiple log records, each log record having an identifier; asynchronously storing the transaction log in a disk-based log store; indexing the log records in an in-memory transaction log index; updating the disk-based transaction log index in the disk-based log store in response to determining that no log record is missing in a sequence of log records; and deleting log records from the in-memory transaction log index in the sequence.

[0081] In Example 10, the subject of Example 9 includes an update to the on-disk transaction log index that is also performed in response to one or more additional correctness and / or performance considerations.

[0082] In Example 11, the subject matter of Examples 9 and 10 includes, wherein the update further includes: persisting all logging identifiers equal to or lower than a specific value to the on-disk transaction log index, while maintaining all logging identifiers higher than the specific value in the in-memory transaction log index, and wherein deleting logging identifiers includes: deleting logging identifiers equal to or lower than the specific value from the in-memory transaction log index.

[0083] In Example 12, the topics of Examples 9 through 11 include an on-disk transaction log index with two levels: a root node and multiple leaf nodes, wherein the root node contains root node entries, and each root node entry points to a leaf node containing log record identifiers within a specified range.

[0084] In Example 13, the subject of Example 12 includes, where each entry in the leaf node identifies the range of log records in the disk-based log storage.

[0085] In Example 14, the subject of Examples 12 to 13 includes reusing existing root nodes and leaf nodes without allocating or freeing root nodes or leaf nodes.

[0086] In Example 15, the subject of Example 14 includes the following: no splitting or merging operations are performed on the nodes of the transaction log index on disk.

[0087] In Example 16, the topics of Examples 14 and 15 include situations where merge or compress operations are not performed across the levels of the on-disk transaction log index, and there is no amplification of input or output operations in the on-disk transaction log index.

[0088] Example 17 is a method comprising: receiving a request to write a transaction log relating to a plurality of database transactions, the transaction log having a plurality of log records, each log record having an identifier; asynchronously storing the transaction log in a disk-based log store; indexing the log records in an in-memory transaction log index; updating the disk-based transaction log index in the disk-based log store in response to determining that no log record is missing in a sequence of the plurality of log records; and deleting log records from the sequence in the in-memory transaction log index.

[0089] In Example 18, the subject of Example 17 includes the following: updates to the on-disk transaction log index are also performed in response to one or more additional correctness and / or performance considerations.

[0090] In Example 19, the subject matter of Examples 17 and 18 includes, wherein the update further includes: persisting all logging identifiers equal to or lower than a specific value to the on-disk transaction log index, while maintaining all logging identifiers higher than the specific value in the in-memory transaction log index, and wherein deleting logging identifiers includes: deleting logging identifiers equal to or lower than the specific value from the in-memory transaction log index.

[0091] In Example 20, the topics of Examples 17 through 19 include an on-disk transaction log index with two levels: a root node and multiple leaf nodes, wherein the root node contains root node entries, and each root node entry points to a leaf node containing log record identifiers within a specified range.

[0092] Example 21 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations for implementing any of the examples 1 to 20.

[0093] Example 22 is an apparatus that includes means for implementing any of Examples 1 through 20.

[0094] Example 23 is a system for implementing any of Examples 1 through 20.

[0095] Example 24 is a method for implementing any of Examples 1 through 20.

Claims

1. A system comprising: At least one processor; as well as The memory includes instructions that, when executed by the at least one processor, cause the system to perform operations, including: Receive a request to write a transaction log related to multiple database transactions, the transaction log having multiple log records, each log record having an identifier; The transaction logs are stored asynchronously in a disk-based log store; An index is created on the log records in the in-memory transaction log index; In response to determining that no log record is missing in the sequence of log records, the on-disk transaction log index in the disk-based log store is updated; and Delete the log record in the sequence from the transaction log index in memory.

2. The system according to claim 1, wherein, Updates to the on-disk transaction log index are also performed in response to one or more additional correctness and / or performance considerations.

3. The system according to claim 1, wherein, The update further includes: persisting all log record identifiers equal to or lower than a specific value to the on-disk transaction log index, while maintaining all log record identifiers higher than the specific value in the in-memory transaction log index, wherein deleting the log record identifier includes: deleting log record identifiers equal to or lower than the specific value from the in-memory transaction log index.

4. The system according to claim 1, wherein, The on-disk transaction log index has two levels: a root node and multiple leaf nodes. The root node contains root node entries, and each root node entry points to a leaf node containing log record identifiers within a specified range.

5. The system according to claim 4, wherein, Each entry in the leaf node identifies the range of log records in the disk-based log storage.

6. The system according to claim 4, wherein, Reuse existing root and leaf nodes without allocating or freeing root or leaf nodes.

7. The system according to claim 6, wherein, Do not perform split or merge operations on the nodes of the on-disk transaction log index.

8. The system according to claim 6, wherein, Merging or compressing operations are not performed across the levels of the on-disk transaction log index, and there is no amplification of input or output operations in the on-disk transaction log index.

9. A machine storage medium having instructions contained thereon, the instructions being executable by a machine processor to perform operations, the operations including: Receive a request to write a transaction log related to multiple database transactions, the transaction log having multiple log records, each log record having an identifier; The transaction logs are stored asynchronously in a disk-based log store; An index is created on the log records in the in-memory transaction log index; In response to determining that no log record is missing in the sequence of log records, the on-disk transaction log index in the disk-based log store is updated; as well as Delete the log record in the sequence from the transaction log index in memory.

10. The machine storage medium according to claim 9, wherein, Updates to the on-disk transaction log index are also performed in response to one or more additional correctness and / or performance considerations.

11. The machine storage medium according to claim 9, wherein, The update further includes: persisting all log record identifiers equal to or lower than a specific value to the on-disk transaction log index, while maintaining all log record identifiers higher than the specific value in the in-memory transaction log index, wherein deleting the log record identifier includes: deleting log record identifiers equal to or lower than the specific value from the in-memory transaction log index.

12. The machine storage medium according to claim 9, wherein, The on-disk transaction log index has two levels: a root node and multiple leaf nodes. The root node contains root node entries, and each root node entry points to a leaf node containing log record identifiers within a specified range.

13. The machine storage medium according to claim 12, wherein, Each entry in the leaf node identifies the range of log records in the disk-based log storage.

14. The machine storage medium according to claim 12, wherein, Reuse existing root and leaf nodes without allocating or freeing root or leaf nodes.

15. The machine storage medium according to claim 14, wherein, Do not perform split or merge operations on the nodes of the on-disk transaction log index.

16. The machine storage medium according to claim 14, wherein, Merging or compressing operations are not performed across the levels of the on-disk transaction log index, and there is no amplification of input or output operations in the on-disk transaction log index.

17. A method comprising: Receive a request to write a transaction log related to multiple database transactions, the transaction log having multiple log records, each log record having an identifier; The transaction logs are stored asynchronously in a disk-based log store; An index is created on the log records in the in-memory transaction log index; In response to determining that no log record is missing in the sequence of log records, the on-disk transaction log index in the disk-based log store is updated; as well as Delete the log record in the sequence from the transaction log index in memory.

18. The method according to claim 17, wherein, Updates to the on-disk transaction log index are also performed in response to one or more additional correctness and / or performance considerations.

19. The method of claim 17, wherein, The update further includes: persisting all log record identifiers equal to or lower than a specific value to the on-disk transaction log index, while maintaining all log record identifiers higher than the specific value in the in-memory transaction log index, wherein deleting the log record identifier includes: deleting log record identifiers equal to or lower than the specific value from the in-memory transaction log index.

20. The method of claim 17, wherein, The on-disk transaction log index has two levels: a root node and multiple leaf nodes. The root node contains root node entries, and each root node entry points to a leaf node containing log record identifiers within a specified range.

Images