Method and apparatus for generating redo records for cloud-based databases

Abstract

Methods and apparatus in a cloud-based database management system are described. Data in a database is stored in a plurality of pages within a page storage area of the database. A plurality of redo log records are received to apply to the database. The redo log records within a predefined boundary are parsed to determine, for each given redo log record, a corresponding page to which the given log record is to be applied. The redo log records are reordered by corresponding page. The reordered redo log records are stored for application to the page storage area of the database.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 731,880, filed December 31, 2019, entitled “METHODS AND APPARATUSES FOR GENERATINGREDO RECORDS FOR CLOUD-BASED DATABASE”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to systems and methods for generating records for cloud-based databases. Background Technology

[0004] Cloud computing is a network-based computing (e.g., internet-based computing) that provides access to a shared pool of configurable computing resources and higher-level services, typically offered rapidly over the internet with minimal management effort. Cloud computing represents another shift in behavior following the move from mainframe-based computing to client-server-based computing, and is implemented as a service. Cloud service providers typically offer three main types of services (hereinafter referred to as cloud computing services) by creating virtual machines on demand for customers: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). IaaS provides computing infrastructure that customers can rent and use. Computing infrastructure includes virtualized and shared physical computing resources (e.g., processors, memory, storage, servers, network components, etc.) among customers. PaaS provides a platform that enables customers to develop, run, and manage software applications without building and maintaining computing infrastructure. SaaS provides software applications running on computing infrastructure on demand via a subscription model over the internet.

[0005] Generally, one type of cloud computing service offered by a cloud computing service provider is a database service. A database service is a PaaS that enables a cloud computing service provider to offer a secure, cost-efficient, enterprise-grade database to customers. In a database service, multiple different entities (e.g., different master servers) can modify a database (e.g., using write operations to modify data). To keep track of the changes made to the database, information describing the changes is generally stored as redo log records in a redo log. Thus, the redo log contains information describing a history of all changes made to the contents of the database, which can be used to reconstruct the contents of the database (e.g., in the event of a failure). The redo log records can also be used to ensure that a copy of the database correctly reflects the data contained in the original database (or master database). However, when there are a large number of changes, the memory resources required to maintain the location of the redo log records can be large.

[0006] Accordingly, it would be useful to provide a method of generating redo log records with lower memory requirements. SUMMARY

[0007] The present disclosure describes examples of reordering and grouping redo log records by corresponding pages, which can help reduce the consumption of memory resources and processing resources. The present disclosure describes examples of when such reordering can be performed without negatively affecting the correctness of the database. Furthermore, the present disclosure also describes examples of selecting predefined boundaries for reordering and grouping redo log records.

[0008] In some examples, the predefined boundaries for reordering and grouping redo log records can be based on mini-transaction boundaries. By selecting the predefined boundaries such that redo log records belonging to the same mini-transaction do not cross the predefined boundaries, the correctness of the database can be maintained. Larger boundaries (e.g., group flush buffer boundaries) can be selected as the predefined boundaries, which can reduce resource consumption.

[0009] In some examples, additional memory and / or processing resources can be saved, e.g., by using merged redo log records (which merge two or more redo log records into a single merged redo log record) and / or combined redo log records (which contain two or more redo log records as contents of a single combined redo log record), as described herein.

[0010] In some aspects, the disclosure describes a method in a cloud-based database management system. The method includes receiving a plurality of redo log records to apply to a database, data in the database being stored in a plurality of pages within a page storage area of the database; parsing the plurality of redo log records within a predefined boundary to determine, for each given redo log record, a corresponding page to which the given redo log record is to be applied; reordering the plurality of redo log records by corresponding page; and storing the reordered redo log records for application to the page storage area of the database.

[0011] In some aspects, the disclosure describes a computer-readable medium having stored thereon instructions to be executed by a processor in a cloud-based database. When executed, the instructions cause performance of the following operations in a storage abstraction layer of the cloud-based database: receiving a plurality of redo log records to apply to a database, data in the database being stored in a plurality of pages within a page storage area of the database; parsing the plurality of redo log records within a predefined boundary to determine, for each given redo log record, a corresponding page to which the given redo log record is to be applied; reordering the plurality of redo log records by corresponding page; and storing the reordered redo log records for application to the page storage area of the database.

[0012] In any of the described examples, the plurality of redo log records can be reordered to group two or more redo log records together based on a same page identifier.

[0013] In any of the described examples, the method can include (or the instructions can further cause performance of) generating, for a given group of two or more redo log records corresponding to a same given page, a merged redo log record, the merged redo log record being a single redo log record that is content-wise the two or more redo log records.

[0014] In any of the described examples, the merged redo log record can include a header containing information identifying the given page.

[0015] In any of the described examples, the method can include (or the instructions can further cause the performance of) identifying, in the merged redo log record, at least two redo log records to apply to the given page; determining a combined redo log record that, when applied to the given page, effects a state change of the given page that is equivalent to cumulative state changes of the given page effected by the at least two redo log records; replacing, in content of the merged redo log record, the identified at least two redo log records with the combined redo log record.

[0016] In any of the described examples, the method can include (or the instructions can further cause the performance of) identifying, for a given set of two or more redo log records corresponding to a same given page, at least two redo log records to apply to the given page; determining a combined redo log record that, when applied to the given page, effects a state change of the given page that is equivalent to cumulative state changes of the given page effected by the at least two redo log records; wherein the identified at least two redo log records are replaced with the combined redo log record.

[0017] In any of the described examples, the method (or the computer-readable medium) can be performed in a storage abstraction layer (SAL) within the cloud-based database, and the reordered redo log records are stored in a buffer and transmitted for application to the page storage area.

[0018] In any of the described examples, the method (or the computer-readable medium) can be performed in a storage layer within the cloud-based database, and the page storage area is located in the storage layer within the cloud-based database.

[0019] In any of the described examples, the predefined boundaries can include a predefined start boundary coinciding with a mini-transaction (MTR) boundary, and can include a predefined end boundary coinciding with a same or different MTR boundary.

[0020] In any of the described examples, the predefined start boundary can coincide with a start point of a first MTR boundary, and the predefined end boundary can coincide with an end point of the first MTR boundary.

[0021] In any of the described examples, the predefined start boundary can coincide with a start point of a first MTR boundary, and the predefined end boundary can coincide with an end point of a second MTR boundary.

[0022] In any of the described examples, the predefined boundaries can coincide with group flush buffer (GFB) boundaries. BRIEF DESCRIPTION OF DRAWINGS

[0023] Reference will now be made to the drawings wherein examples of the exemplary embodiments of the present application are shown by way of example, as follows:

[0024] Figure 1 A block diagram illustrating a cloud computing architecture for providing cloud computing services, as provided by the exemplary embodiments described herein, is shown;

[0025] Figure 2 A simplified logical diagram illustrating an exemplary architecture of a cloud-based database, as provided by the exemplary embodiments described herein, is shown;

[0026] Figure 3 A block diagram illustrating a simplified example of a processing unit, as provided by the exemplary embodiments described herein, is shown;

[0027] Figure 4 A flow diagram illustrating an exemplary method performed by a plurality of software entities in a cloud-based database to process redo log records, as provided by the exemplary embodiments described herein, is shown;

[0028] Figure 5 A flow diagram illustrating an exemplary method performed by a single entity in a cloud-based database to process redo log records, as provided by the exemplary embodiments described herein, is shown;

[0029] Figure 6 A schematic diagram illustrating an example of reordering and grouping of redo log records, as provided by the exemplary embodiments described herein, is shown;

[0030] Figure 7 An exemplary format of a merged redo log record, as provided by the exemplary embodiments described herein, is shown;

[0031] Figure 8 An example of Figure 5 A flow diagram illustrating an exemplary detailed implementation of the method shown;

[0032] Figure 9 An example of a buffer content that can be generated by Figure 8 An exemplary buffer content generated by the exemplary method shown;

[0033] Figure 10 A flow diagram illustrating an exemplary method for generating a merged redo log record, as provided by the exemplary embodiments described herein, is shown;

[0034] Figure 11 A flow diagram illustrating an exemplary method for replacing two or more redo log records with a combined record, as provided by the exemplary embodiments described herein, is shown.

[0035] The same reference numerals may be used to denote the same components in different figures. Detailed Implementation

[0036] This invention describes examples in a cloud computing context using a dual-active architecture. Although certain system diagrams and flowcharts are shown and discussed, these are provided by way of example only and are not intended to be limiting.

[0037] Figure 1A logical block diagram of a cloud computing architecture that can provide cloud computing services is shown schematically. The illustrated logical diagram of a cloud computing architecture 100 (hereinafter cloud 100) generally includes an infrastructure platform 102 (e.g., an infrastructure as a service (IaaS) layer), an application platform 104 (e.g., a platform as a service (PaaS) layer), and applications 106 (e.g., a software as a service (SaaS) layer). The infrastructure platform 102 includes physical hardware resources 108 and a virtualization layer 110 that presents an abstraction of the physical hardware resources 108 to the application platform 104. The abstraction presented by the virtualization layer 110 depends on the requirements of applications 112 hosted on the application platform 104. The physical hardware resources 108 include physical machines or servers 114 including physical processing resources 114 (e.g., central processing units (CPUs), graphic processing units (GPUs), accelerators, tensor processing units (TPUs)), physical storage servers 116 including storage resources such as memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), persistent storage devices (e.g., hard disk drives, optical drives, or combinations thereof)), and network resources (not shown) that typically reside within a data center. Those skilled in the art will appreciate that a data center includes a collection of physical hardware resources 108 (typically in the form of servers) that are available as collective computing resources including processing, storage, and network resources. Within a data center, multiple servers can be connected together to provide a pool of computing resources in which virtualized entities can be instantiated. Data centers can be interconnected to form a pool of computing resources that are interconnected through a connecting resource. The connecting resource can take the form of a physical connection such as an Ethernet or optical communication link.

[0038] The virtualization layer 110 provides support for flexible and efficient multi-tenant runtime and hosting environments for applications 112 by providing an IaaS facility. The virtualization layer 110 includes a virtualization manager or hypervisor (not shown) that can provide security and resource “sandboxes” for each application 112 hosted by the application platform 104. Each “sandbox” can be implemented as a virtual machine (VM) 118 that can include an appropriate operating system and provide controlled access to virtualized storage resources 120.

[0039] Virtualization of the physical hardware resources 108 by the virtualization layer 110 is considered a foundational technology for the cloud 100. Virtualization is a technology that enables the creation of a pool of computing resources (e.g., processing, storage, and networking resources) that are connected to each other through a fabric of resources. Virtualization can take the form of instantiating a VM 118 that is indistinguishable from a physical computing device to another entity on the network and to software executing on the VM 118. The VM 118 has its own set of computing resources (e.g., processing, storage, and connectivity resources) on which an operating system can be run. The VM 118 can have a virtual network interface that can be assigned a network address. Between the underlying resources and the VM 118, there is typically a hypervisor (not shown) that manages resource isolation and network interactions. One purpose of the VM 118 is to enable isolation from other processes running on the cloud 100. In its original development, the VM 118 was a mechanism to enable different processes to run without worrying that a single faulty process would bring down the entire system. Instead, the faulty process would be contained in its own VM 118. This isolation enables each VM 118 to have its own set of network interfaces. Typically, a single underlying computing resource can support multiple virtualized entities.

[0040] Those skilled in the art will appreciate that a recent development is the use of containers instead of VMs 118. As noted above, each VM 118 typically includes its own operating system, which often adds redundant computing, storage, and connectivity resources. With the use of containers, a single operating system (OS) kernel is able to support multiple independent applications. Instead of each VM 118 being able to run its own OS, a single OS hosts the containers that are responsible for enforcing the resource isolation provided by the VMs 118 through other means.

[0041] The application platform 104 provides the ability to host applications 112 and includes application platform services 122. The application platform services 122 provide a set of middleware application services and infrastructure services to the applications 112 hosted on the application platform 104. The applications 112 hosted on the application platform 104 can run on VMs or physical machines. In Figure 1 In the example shown, the application platform services 122 include a cache service system 124 for in-memory data storage, a database service 126 for applications, a messaging service 128 for publishing messages to users, and an application program interface (API) gateway service 130 that enables customers to create, publish, and maintain APIs to access other cloud services. Those skilled in the art will appreciate that the application platform services 112 can provide other middleware application services to customers, such as notification services, runtime services, and the like. Applications 112 from customers can be deployed and executed inside respective VMs 118 or physical machines 114.

[0042] As discussed further below, the database service 126 can be implemented using a cloud-based (or cloud-native) database architecture. Some terminology used in the present invention is now discussed.

[0043] In the present invention, a database refers to a collection of organized data that is typically stored and accessed electronically from a computer system.

[0044] A Database Management System (DBMS) is a “software system” that enables users (e.g., database administrators) to define, create, maintain, and control access to databases. A DBMS is the “software” that interacts with end users (e.g., client users), applications 112, and the database itself to capture and analyze data. The DBMS acronym is sometimes expanded to indicate the underlying database model, for example, RDBMS indicates a relational database model, OODBMS or ORDBMS indicates an (object-oriented) database model, ORDBMS indicates an object-relational database model. Other expansions can indicate some other feature, for example, DDBMS indicates a distributed database management system. The core functions provided by a DBMS are data storage, retrieval, and update. A mature general-purpose DBMS typically provides the following functions:

[0045] • data storage, retrieval, and update

[0046] • user-accessible catalog or data dictionary of descriptive metadata

[0047] • support for transactions and concurrency

[0048] • tools for recovering the database in the event of database corruption

[0049] • Support for authorized access and data updates

[0050] • Remote location access support

[0051] • Enforcement of constraints to ensure data in the database adheres to certain rules

[0052] A database engine (or storage engine) is the underlying software component of a DBMS that is used to create, read, update, and delete (CRUD) data from a database. External interactions with the database can be made via an application that interacts with the DBMS.

[0053] A database transaction (or simply transaction) is a unit of work performed within a DBMS.

[0054] A database page (or simply page) is an internal fundamental structure that organizes data in a database file. A database page is a unit of storage, and its size can be configured on a system-wide, database-wide, or enterprise group-specific basis. Pages can be identified by identifiers such as page identifiers and space identifiers.

[0055] A redo log is a file that records a history of all changes made to a database. A redo log (or more generally, a database log) can be stored as a file. Each redo log contains one or more redo log records (or more generally, database log records). A redo log record (also referred to as a redo entry or a log entry) contains a set of change vectors, each of which describes or represents a change made to a single block or page in the database. The term “redo log” can be specific to a particular DBMS model, but a redo log can also be used in a general way to refer to a database log. MySQL is an example database model that uses the term “redo log”; MySQL is an example database model that can be used to implement examples described herein. It should be understood that the present invention can be equally applicable to other database models. A database model is a data model that determines the logical structure of a database and determines the way data is stored, organized, and manipulated. One example of a database model is the relational model, which uses a table-based format to store, organize, and manipulate data.

[0056] A cloud-native database (referred to herein as a cloud-based database) is a database service that can be built, deployed, and offered through the cloud. A cloud-native database is a PaaS that provides a database that enables organizations, end users, and their respective applications to store, manage, and retrieve data in the cloud.

[0057] Figure 2A simplified logical block diagram showing the architecture of the cloud-based database 200 is shown, in which the compute tier 210 and the storage tier 230 are separate. Figure 2 The cloud-based database 200 shown in FIG. 1 introduces an additional tier (i.e., a storage abstraction layer (SAL) 220) that separates the compute tier 210 and the storage tier 230. The SAL 220 provides an abstraction layer in which the physical resources (e.g., physical memory and physical processing resources) provided by the IaaS tier 102 can be pooled in order to service the compute tier 210 and the storage tier 230. The separation of the compute tier 210 and the storage tier 230 and this abstraction of physical resources (and thus the pooling of physical resources) can enable better scalability and load balancing. For example, the compute tier includes a cluster of servers (e.g., a primary server and replica servers), and the storage tier includes a cluster of storage servers.

[0058] The compute tier 210 includes a database primary server 212 (also referred to simply as the primary server 212) and one or more database replica servers 214 (also referred to simply as the replica servers 214). The database primary server 212 and the database replica servers 214 can generally be referred to collectively as database servers (212, 214). In some examples, the database servers (212, 214) can also be referred to as nodes in the cloud-based database 200. For example, there can be a single database primary server 212 and one or more database replica servers 214. Generally, the database primary server 212 handles all modifications to the database (including requests to insert, update, or delete database records), while the database replica servers 214 handle only read requests (i.e., requests to select database records). Database transactions that include multiple statements such as insert, update, delete, select (i.e., read requests), etc. are handled by the database servers (212, 214) in the compute tier 210. It should be noted that the database servers (212, 214) can not be physical hardware servers, but rather software running on the physical processing resources of the cloud (i.e., in the IaaS tier 102). The database servers (212, 214) can be software running on virtual machines 118 or containers provided by the IaaS tier 102 of the cloud (also referred to as instances of the database servers (212, 214)). Generally, the instances of the database servers (212, 214) can be considered to have physicality in that any instance is implemented on a physical machine. For simplicity, the present invention refers to the database servers (212, 214) rather than to the instances of the database servers (212, 214).

[0059] Each database server (212, 214) communicates with a storage abstraction layer (SAL) 220 via a respective SAL module 222. In Figure 1 In the context of the present disclosure, the SAL 220 can be viewed as spanning the database service 126 and the virtualized storage resources 120, and providing an abstraction layer that pools physical resources to serve both the database service 126 and the virtualized storage resources 120. It should be noted that the SAL 220 is not a typical layer of a traditional database service (e.g., a database service provided by a traditional cloud service provider). The cloud-based database 200 described herein includes the SAL 220, and can use the SAL 220 to implement functionality that provides advantages over traditional database services (as discussed further below). The SAL module 222 can be an instance of software implemented in the SAL 220. For simplicity, instances of the SAL module 222 can be referred to herein simply as SAL modules 222. The SAL module 222 provides the functionality of the logical SAL 220. In some examples, one or more functions of the SAL module 222 can also be implemented in the storage layer 230. The SAL 220 serves to isolate the client-facing front end (provided by the compute layer 210) from the organization and management of the database.

[0060] Data (including redo logs and database pages, as described below) is stored in the storage layer 230. In this example, the storage layer 230 is accessible via a network, e.g., a remote direct memory access (RDMA) network. The storage layer 230 can be a distributed storage system provided by the virtualization layer 120 of the laaS 102, or the like, that provides relatively fast, reliable, and scalable storage. The storage layer 230 includes one or more page stores 232 that store pages containing data, and one or more log stores 234 that store one or more redo logs. The page stores 232 service read requests (i.e., requests to read data from one or more pages) received from the database servers (212, 214). The page stores 232 can recreate any version of a page that can be requested by the database servers (212, 214). In the cloud-based database 200, the page stores 232 are run by a cluster of storage resources (not shown). Each page store 232 receives all redo log records that have been generated for a page for which the page store 232 is responsible, and integrates (or applies, with conflict resolution as applicable) the redo log records to the database page.

[0061] While described in the context of a single database above, it should be appreciated that in some examples, two or more databases can be managed using the cloud-based database 200 (e.g., with logical separation separating the individual databases). Each database is divided into a fixed size subset of pages, which can be referred to as a slice. Each page store 232 manages a respective plurality of slices. The slices managed by a single page store 232 can include slices having pages from different databases. A page store server only receives redo log records for pages belonging to the slices that the page store server manages. Typically, a database can have multiple slices, and each slice can be replicated to multiple page stores 232 for durability and availability (e.g., in the event that one particular page store 232 is unavailable, another page store 232 to which the slice has been replicated can be used to continue servicing requests to access data from (i.e., read from) or modify data stored in (i.e., write to) the slice).

[0062] Operations performed by the SAL module 222 include sending updates of redo log records from the database primary server 212 to the database replica server 214 to the physical location of the redo log to enable (the database replica server 214 to) know the access (i.e., read) location of the latest redo log records (arrow 202), accessing (i.e., reading) pages from the page store 232 (dashed arrow 204), writing redo log records to the log store 234 and the page store 232 (arrow 206), receiving updates of redo log records from the log store 234 for the database replica server 214 (arrow 208). The SAL 220 is also responsible for creating, managing, and destroying slices in the page store 232 and assigning pages to slices. In some examples, the SAL module 222 can be linked to the database servers (212, 214); in other examples, the SAL module 222 can be implemented by another component in the cloud-based database 200, e.g., by another server (e.g., a storage server) that is not linked to the database servers (212, 214). Each database server (212, 214) can be served by a respective SAL module 222 (as shown). In some examples, a single instance of the SAL module 222 can serve two or more database servers (212, 214). In some examples, a single instance of the SAL module 222 can serve all of the database servers (212, 214) in the cloud-based database 200. In some examples, the SAL 220 can be implemented using a standalone SAL module 222 that can run on a VM 118, in a container, or on a physical server 114.

[0063] Figure 3A block diagram illustrating a simplified example of a processing unit 300 that can be used to implement a physical machine or server 114 or a physical storage server 116 is shown. Alternatively, the processing unit 300 can be a representation of a virtual machine 118 provided by the virtualization layer 110 of the laaS 102. Other examples suitable for implementing the embodiments described in this disclosure can be used, which can include different components than those discussed below. While Figure 3 A single instance of each component is shown, but there can be multiple instances of each component in the processing unit 300.

[0064] The processing unit 300 can include one or more processing devices 302, such as a processor, microprocessor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), dedicated logic circuitry, a dedicated artificial intelligence processor unit, or a combination thereof. The processing unit 300 can also include one or more optional input / output (I / O) interfaces 304, which can enable connections with one or more optional input devices 306 and / or optional output devices 308.

[0065] In the illustrated example, the input devices 306 (e.g., a keyboard, mouse, microphone, touchscreen, and / or keypad) and the output devices 308 (e.g., a display, speaker, and / or printer) are shown as optional and external to the processing unit 300. In other examples, there can be no input devices 306 and output devices 308, in which case the I / O interfaces 304 can not be needed.

[0066] The processing unit 300 can include one or more network interfaces 310 for wired or wireless communication with entities in a cloud-based database architecture. The network interfaces 310 can include wired links (e.g., an Ethernet cable) and / or wireless links (e.g., one or more antennas) for communicating across different layers (210, 220, 230) of the cloud-based database architecture 200.

[0067] The processing unit 300 can also include one or more storage units 312, which can include mass storage units such as solid state drives, hard disk drives, disk drives, and / or optical disk drives.

[0068] The processing unit 300 can include one or more memories 314, which can include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The one or more non-transitory memories 314 can store instructions executed by the one or more processing devices 302, for example, to perform examples described in the present disclosure. For example, where the processing unit 300 is used to implement the SAL module 222, the memory 314 can store instructions to perform the functions of the SAL module 222 and to manage redo records, as described below.

[0069] The one or more memories 314 can include other software instructions, for example, to implement an operating system and other applications / functions. In some examples, the processing unit 300 can additionally or alternatively execute instructions from external memory (e.g., an external drive in wired or wireless communication with the processing unit 300), or executable instructions can be provided by a transitory or non-transitory computer-readable medium. Examples of non-transitory computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory.

[0070] The processing unit 300 can also include a bus 316 that provides communication among the components of the processing unit 300, including those discussed above. The bus 316 can be any suitable bus architecture, including a memory bus, a peripheral bus, or a video bus, for example.

[0071] Referring again to Figure 2Whenever a cloud service provider creates or extends a database, the SAL module 222 selects the page store 232 associated with the database (or any suitable page store 232 if a new database is to be created) and creates a slice in the selected page store 232. The database master 212 maintains redo log records in a group flush buffer (GFB). Any redo log record that records a database transaction that has been committed by a client of the cloud-based database to the compute tier 210 can be stored in the GFB. The redo log records stored in the GFB can be flushed to the SAL module 222 for updating in the log store 234. The process of transferring redo log records from the buffer and clearing the buffer can be referred to as “flushing” the buffer. The SAL module 222 adds the log records to the currently active log store replica to help ensure the durability of the redo log records. After the SAL module 222 successfully adds all transferred redo log records to all log store replicas 234, the SAL module 222 can acknowledge successful replication of the redo log records to the database master 212. In some embodiments, the redo log records can be parsed before the SAL module 222 adds the log records to the log store replicas 234, simultaneously while the SAL module 222 is adding the log records to the log store replicas 234, or after the SAL module 222 successfully adds the redo log records to the log store replicas 234, and then distributed to a per-slice log buffer (SLB) in the SAL module 222. Each SLB can be flushed when full and / or at predefined time intervals (e.g., when a timeout counter expires).

[0072] Figure 4 An example method 400 performed by a plurality of software entities in the cloud-based database 200 (e.g., by the database master 212, the SAL module 222, and instances of the page store 232 in different ways as described below) to process redo log records is shown. It should be appreciated that the method 400 provides a simplified overview and is merely an example.

[0073] At 402, the database master 212 generates redo log records and adds (i.e., writes) them to the GFB. The database master 212 flushes (or sends) the redo log records stored in the GFB to the SAL 220 (via the SAL module 222). For example, the GFB can be flushed at a certain buffer size (e.g., the GFB is full) and / or at a certain time interval. The GFB contains a set of committed redo log records from the database master 212 (i.e., redo log records of database transactions that have been committed by a client of the cloud-based database 200).

[0074] At 404, the SAL module 222 receives the redo log records from the GFB, parses the redo log records, and copies the redo log records to the SLB corresponding to the appropriate slice. For example, the SAL module 222 parses the redo log records to identify which database page is effected by a given redo log record, and thus determines which database page is the appropriate slice that should have the given redo log record applied. The SAL module 222 then copies the given redo log record to the SLB corresponding to the appropriate slice.

[0075] At 406, the page store 232 receives the redo log records from the appropriate SLB, parses the redo log records, and uses the redo log records to update the data stored in the corresponding database page. For example, the page store of a given page parses the redo log records that are applied to the given page. All data updates indicated by the redo log records applied to the given page are applied so that the given page correctly reflects the modifications indicated by the redo log records.

[0076] The redo logging can take the form of multiple mini-transactions (MTRs). An MTR is a database term that can be used to describe a minimal set of redo log records that should not be split apart in order to maintain database consistency. Database consistency means that the replicated data (including the replicated data in the replica page store 232 and the replicated redo log records in the replica log store 234) correctly reflects the database transactions that have been committed in the compute tier 210. The term MTR can be generally used in the context of MySQL; however, in the present invention, the term MTR is generally used to refer to the minimal, indivisible atomic unit of redo log records, without being limited to any particular database implementation. In the present invention, an “atomic unit” is used to refer to a set of redo log records that should be applied together in order to correctly reflect the data stored in the database (e.g., in order to correctly replicate the database operations of the primary database server 212 in the replica page store 232 and the replica log store 234). For example, a B+ tree includes root page nodes and leaf page nodes. To perform a tree split or a tree merge, the B+ tree data structure can require multiple B+ tree page nodes to be updated at the same time. All of these page updates are included in multiple redo log records that should be combined as a single MTR, as the redo log records need to be applied atomically (i.e., as an indivisible group). The set of redo log records within a single MTR can be defined by MTR boundaries. Thus, the MTR boundaries can be an implicit piece of information that defines which log records should be processed together. The database engine can implement an algorithm to determine the start and end of an MTR. For example, the MySQL InnoDB engine generates MTR BEGIN and MTR END redo records to identify the MTR boundaries. When parsing the redo log records, the SAL module 222 can use this implicit or explicit indicator of the MTR boundaries to determine the start and end of an MTR. In a traditional database service, the redo log records in the GFB can be directly flushed to the storage. However, in a cloud-based database (i.e., a cloud-native database) 200, the redo log records can be sent to different page stores 232 based on their slice partitioning.

[0077] A database typically has tens or even hundreds of slices. Each slice has a corresponding SLB. The contents of the SLB (e.g., redo log records) for a given slice are flushed to the page store 232 corresponding to that slice. The SLB can be flushed, for example, when a certain buffer size is reached (e.g., the SLB is full) and / or at certain time intervals.

[0078] The challenge is that in the page storage area 232, each redo log record occupies an index data structure (also referred to as a log directory version) in memory. The index data structure tracks the location of each redo log record in physical memory (e.g., tracks the logical location of each redo log record in the virtualized storage resource 120) by indexing each redo log record. Tracking relatively small redo log records (e.g., only a few dozen bytes in size, or even just a few bytes) using an index data structure can be an inefficient use of memory. The same page can have multiple small redo log records that can be interspersed with redo log records of different pages. The processing granularity in the storage layer 230 is on a per-page basis, i.e., the storage layer 230 accesses a page when processing redo log records for that page. Thus, multiple redo log records for a given page interspersed among redo log records of different pages can result in longer load times because the given page is repeatedly retrieved. Overall, this arrangement can result in a significant amount of memory and processing resource consumption for the cloud’s IaaS.

[0079] The database servers (the primary server 212 and the replica server 214) generally need to access the page versions within the MTR boundaries to help ensure database consistency during page processing. Multiple MTRs can be contained in a GFB, and redo log records within a single GFB can be defined by the GFB boundaries. Thus, accessing redo log records within a GFB boundary should ensure that the MTR boundaries are also respected. Generally, the range of redo log records tracked in the index data structure can be much larger than a single MTR. For example, all redo log records in a flushed GFB or multiple flushed GFBs can be tracked in a single index data structure. The database primary server 212 always accesses (i.e., reads) the latest version of a page that is within the GFB boundaries. The database replica server 214 lags behind the database primary server 212. The version of a page accessible to the database replica server 214 can be made accessible when the update to the database page is completed based on at least the MTR boundaries. That is, the updated page version (i.e., the version of the updated page) can be made accessible (i.e., readable) to the database replica server 214 only after all redo log records within the same single MTR boundary are processed. Respecting the MTR boundaries in this manner helps ensure database consistency. However, the boundaries for processing the records can be larger, e.g., the GFB boundaries of a single GFB or even a group of GFBs.

[0080] In examples disclosed herein, a method is described in which redo log records can be reordered within a predefined boundary (e.g., by SAL module 222) in SAL 220 before the redo log records are flushed to storage tier 230. The redo log records can be reordered such that redo log records for the same page are grouped together within the predefined boundary (e.g., an MTR boundary or a GFB boundary). Reordering the redo log records in the manner described herein can help improve the usage of memory and processing resources (e.g., physical resources of the cloud, such as resources provided in the IaaS tier and pooled by the SAL) without negatively impacting data consistency in the database. In addition, the reordered redo log records can be merged into a single redo log record, such as a single COMPACT type redo log record. The merged redo log record can be formatted to have a header with individual redo log records as content. The individual redo log records within the merged redo log record can remain substantially unchanged (except for the reordering). In some examples, a combined redo log record can be used in place of two or more redo log records in the merged redo log record. A combined redo log record is a single redo log record that replaces two or more individual redo log records that are to be applied to the same page location and that has the same effect as applying each individual redo log record separately. The combined redo log record can be newly generated, can be a modification of one of the two or more redo log records, or can be one of the two or more redo log records.

[0081] The merged redo log record can be treated as a single redo log record and can be tracked by the index data structure as a single unit, thereby reducing the memory (e.g., physical memory used by physical servers in IaaS tier 102 or virtual memory of virtual machines 118, both of which can be abstracted in the SAL) required to track the merged redo log record. The merged redo log record can be accessed as a single redo log record, which can help reduce the usage of processing resources (e.g., physical processing resources used by physical servers in IaaS tier 102 or virtual processing resources of virtual machines 118, both of which can be abstracted in the SAL). When accessed, the merged redo log record can be parsed to access the individual redo log records stored as its content, and each individual redo log record can be applied according to the storage order.

[0082] Figure 5 A flowchart of an example method 500 that can be performed by SAL module 222 is shown. In examples where SAL module 222 is implemented as an IaaS tier 102 of a cloud (e.g., cloud 100), method 500 can be performed by a physical server in IaaS tier 102 or by a virtual machine 118 in IaaS tier 102. In examples where SAL module 222 is implemented as a virtual machine 118 of a cloud (e.g., cloud 100), method 500 can be performed by virtual machine 118. Figure 3In the illustrated example of a software instance within a physical machine or server in processing unit 300), processing device 302 can execute instructions stored in memory 314 in order to perform method 500. It should also be appreciated that the functionality of SAL module 222 can be performed using abstracted pooling resources, such that instances of SAL module 222 can be run using physical resources distributed across multiple physical machines. Method 500 can be performed as part of the regular operation of SAL module 222, for example during the stage of parsing GFBs and distributing redo log records to appropriate page stores 232. In some examples, method 500 can be performed in SAL 220 or storage layer 230, among others.

[0083] At 502, SAL module 222 receives redo log records, for example, redo log records that have been flushed from the GFB of database primary server 212.

[0084] At 504, each redo log record within the predefined boundary is parsed. The redo log record can include information, for example, a page identification that identifies the page to which the redo log record applies. In some embodiments, the page identification can include a page identifier (ID) and a space ID. The redo log record can also include a sequence number, for example, a log sequence number (LSN) that indicates the position of the redo log record within the redo log. Typically, LSNs are assigned to log records in the order in which the redo log records are entered in the GFB. Redo log records in the GFB are typically ordered in increasing order of LSN. Parsing the log records can enable SAL module 222 to determine which log records are to be applied to which page (e.g., based on the page ID). The log records can be parsed until the end of the predefined boundary (e.g., the GFB boundary) is reached. The predefined boundary can be predefined at SAL module 222. Details of how the predefined boundary is selected will be provided further below.

[0085] At 506, SAL module 222 reorders the redo log records within the predefined boundary. The reordered redo log records are grouped together according to the database page. In some embodiments, SAL module 222 parses the redo log records to determine which redo log records have the same page identification (i.e., redo log records that have the same page ID and space ID), and groups the redo log records having the same page identification (i.e., the same page ID and space ID) together.

[0086] In some examples, steps 504 and 506 can be performed in parallel. For example, parsing previous redo log entries may include identifying the page identifier corresponding to the current redo log entry, and reordering the current redo log entry to combine it with any other (previously parsed) redo log entries corresponding to the same page identifier. For example, reordering may include sorting the redo log entries by page ID and space ID (e.g., using a temporary or intermediate data structure).

[0087] Figure 6 A schematic diagram illustrating an example of reordering and grouping performed by SAL module 222 in step 506 is shown. Figure 6 Multiple redo log records 604 are shown within a predefined boundary 602 (e.g., within a GFB boundary). The redo log records 604 are displayed with different colors and shading; redo log records 604 with the same color and shading correspond to the same database page (e.g., pages with the same page ID and space ID). The number displayed for each redo log record 604 is its LSN.

[0088] like Figure 6 As shown, redo log records in the GFB are arranged in ascending order of LSN. After the reordering in step 506, the redo log records 604 are grouped such that each group 606 of the redo log records 604 corresponds to the same page. It can be noted that a group 606 may contain only one redo log record 604. In this invention, it is not necessary to maintain the LSN order within each group 606 within the predefined boundary 602 (although maintaining the LSN order may be simpler and / or more efficient). This is because the database replica server 214 expects to always look at the database ahead of the GFB boundary and expects never to access pages with LSNs within the GFB. In this way, the GFB can be considered a "visible" boundary, meaning that changes made to the database within the visible boundary are not visible to the client, while the final result of all changes within the visible boundary is visible to the client. In contrast, in traditional databases, the LSN order must be strictly adhered to, especially between database transactions. Otherwise, the data in the database will be incorrect.

[0089] Optionally, at 508, the redo log records in the group can be merged into a single merged redo log record. The present disclosure describes an exemplary format for implementing a merged redo log record, referred to herein as the COMPACT type. The COMPACT type is used as an example below. The merged redo log record can use different formats and types, as long as the page identification (e.g., space ID and page ID) can be indicated appropriately. The merged redo log record can be treated as a single redo log record in the storage area, and can occupy a single index data structure in physical memory (e.g., in the IaaS layer 102) or virtual memory (e.g., of the virtual machine 118) (thereby reducing memory usage for tracking). In addition, the merged redo log record can be accessed as a single redo log record (thereby reducing the number of I / O processes). To apply the merged redo log record, the merged redo log record can be parsed to identify the page identification, and then each individual redo log record contained in the merged redo log record can be applied in turn.

[0090] Figure 7 An exemplary format of a merged redo log record 700 based on the COMPACT type disclosed herein is shown. The merged redo log record 700 includes a header 710 and a body 720. The header 710 in this example follows the COMPACT type. The header 710 includes an MLOG_COMPACT field 712, which identifies the merged redo log record 700 as a merged redo log record of the COMPACT type, a LENGTH field 714, which indicates the total length of the merged redo log record 700, a spaceID field 716, which indicates the common space ID of the individual redo log records contained in the merged redo log record 700, and a pageNumber field 718, which indicates the common page ID of the individual redo log records contained in the merged redo log record 700. The individual log redo records 722 that have been combined together are contained in the body 720.

[0091] Optionally, at 510, the SAL module 222 can replace two or more individual redo log records with a single combined redo log record, instead of leaving the individual redo log records in the merged redo log record unchanged; the single combined redo log record has the same effect as applying each redo log record individually when applying the single combined redo log record. The single combined redo log record can replace two or more individual redo log records in the merged redo log record 700. Details of this step are discussed further below.

[0092] At 512, after performing the reordering (and optional merging and grouping), the redo log records are stored for application to the appropriate page store. For example, if the method 500 is performed in the SAL 220, the log records (e.g., stored in a SLB or like buffer) can be stored for transmission to the page store. Or, if the method 500 is performed in the storage tier 230, the redo log records can be stored in a journal store.

[0093] In the page store, a single index data structure can be used to track a group of redo log records for the same page, rather than needing an index data structure to track each individual redo log record. Thus, physical and / or virtual memory resources can be conserved. It has been found that using the reordering and grouping of redo log records described above can result in a four-fold reduction in physical and / or virtual memory usage. Reordering and grouping redo log records in this way can also conserve physical and / or virtual processing resources. For example, when the page store runs out of physical (or virtual) memory, it can be necessary to swap redo log records in and out of disk (e.g., in the physical memory provided by the IaaS tier 102). By grouping redo log records by page, the number of I / O processes accessed from disk can be reduced.

[0094] In some examples, the method 500 can be performed by a SAL module 222 in the SAL 220. In some examples, the method 500 can be performed in the storage tier 230 rather than in the SAL 220. Generally, the functions of the SAL 220 can be performed using physical (or virtualized) resources, as described above, which can be provided by a single physical server (e.g., a single database server or a single storage server) or distributed across multiple physical servers (e.g., the SAL modules 222 can run on virtual machines or in containers). The method 500 can be performed after flushing the buffers containing the redo log records to the SAL, or the method 500 can be performed after receiving the flushed records in the storage tier.

[0095] In the present invention, redo log records within a predefined boundary are reordered and grouped on a page basis. The predefined boundary includes a predefined start boundary and a predefined end boundary. The predefined boundary defines a set of sequential redo log records starting at the predefined start boundary and ending at the predefined end boundary. Typically, the predefined boundary should be defined such that redo log records that must be applied together (e.g., redo log records belonging to the same MTR) do not cross the boundary. Selecting a predefined boundary for the reordering and grouping of redo log records is not trivial. If the predefined boundary defines redo log record sets that are too small, the advantages of reordering can not be fully realized. Larger predefined boundaries can have the advantage that more redo log record sets can be combined together for more efficient use of physical or virtual memory and processing resources. However, if the predefined boundary defines redo log record sets that are too large, there can be too many redo log record sets combined together for a single page, and the sets can need to receive multiple buffer flushes. Some suitable example predefined boundaries are discussed below.

[0096] The predefined boundary can be based on a buffer boundary. That is, the boundary can be defined as a redo log record set that is flushed from a buffer. This can be a GFB boundary. Other buffer boundaries can also be used. It should be noted that the buffer boundary (or any other boundary) selected as the predefined boundary must coincide with one or more MTR boundaries. That is, the predefined boundary (e.g., a GFB boundary) should not break up MTRs; in other words, MTRs should not cross the predefined boundary. Selecting a GFB boundary as the predefined boundary can be useful because GFB boundaries are typical boundaries for other database operations. For example, database replica servers 214 typically use GFBs as boundaries to advance LSNs, and typically use GFB boundaries to access redo log records. Thus, database replica servers 214 will access the reordered and grouped records from log storage 234 as a whole without any additional splitting or merging operations.

[0097] Another possible predefined boundary can be an MTR boundary. MTR boundaries are expected to be smaller than GFB boundaries. GFBs always end at MTR boundaries (i.e., a GFB boundary will coincide with at least one MTR boundary). Another possible predefined boundary can be multiple GFBs together (referred to herein as a multi-GFB boundary). Selecting larger or smaller boundaries as the predefined boundary can require tradeoffs to be considered, as larger or smaller boundaries can have different advantages and disadvantages, as discussed above. Typically, the predefined boundary should not be smaller than the MTR boundary in order to ensure database consistency, as discussed previously.

[0098] Figure 8A flowchart showing details of an example method 800 for implementing the method 500. The method 800 is based on selecting a GFB boundary as the predefined boundary, using MySQL as the database, and merging the grouped redo log records as COMPACT type. It should be appreciated that the method 800 illustrates only one possible embodiment of the method 500 and is not intended to be limiting. For simplicity, the method 800 will be described in the operational context of the SAL 220; however, it should be appreciated that in some examples, one or more operations can be performed in the storage layer 230.

[0099] At 802, the SAL 220 (e.g., at the SAL module 222) receives a GFB from the compute layer 210 (also referred to as the SQL layer), e.g., from the database master server 212. As previously described, the GFB contains one or more redo log records generated by the compute layer 210. The GFB can contain redo log records from one or more database transactions. Thus, the disclosed method can compress redo log records across database transaction boundaries.

[0100] The SAL module 222 parses the redo log records from the start of the GFB until the GFB boundary (i.e., at the end of the GFB) is reached. At 804, it is determined whether all redo log records in the GFB have been parsed.

[0101] If there are still at least one redo log record in the GFB, then at 806, for the next redo log record, the space ID, page ID, and record length data (i.e., data indicating the length of the redo log record) are extracted from the redo log record. The parsing by the SAL module 222 can be performed based on the specific format of the redo log record generated by MySQL (for that specific database type). It can be noted that the redo log record can be a combined redo log without affecting the operations here.

[0102] At 808, the byte offset value (relative to the start of the GFB) and the record length of the current redo log record are saved. The byte offset value and the record length of the current redo log (which can be referred to as redo log record metadata) will be used in the subsequent compression step.

[0103] At 810, the extracted space ID and page ID (step 806) are paired to form a key for searching an intermediate (e.g., temporary) data structure (e.g., a hash table (HT)) to determine whether an entry already exists in the intermediate data structure. For simplicity, the following discussion will refer to an HT, but it should be appreciated that some other data structure can be used. If an entry already exists in the HT, this means that at least one other record in the GFB corresponds to the same page identification.

[0104] At 812, it is determined whether an entry already exists in the HT.

[0105] If an existing entry is found in the HT, then at 814, the entry in the HT is updated as follows. The total-length field (indicating the total length of all redo log records for the same space ID, page ID) is updated to include the record length of the current redo log record. The record-lsn field (indicating the LSN representing the slice redo log record) is updated. The value in the record-lsn field can depend on whether the slice redo log record is of the single type or the COMPACT type; for the single type, the value in the record-lsn field corresponds to the last byte offset of the redo log record; for the COMPACT type, the value in the record-lsn field corresponds to the last byte offset of the last combined redo log record. This means that the LSNs in the final output buffer can be out-of-order or reordered. The num-records field (indicating the number of combined redo log records for the space ID, page ID pair) is updated to 1. The record meta-info list field (including a list of redo log record metadata (generated in step 808) of all redo log records in the group) is updated in the order of generation using the redo log record metadata. It should be noted that the LSNs of the list entries are not necessarily contiguous and can have gaps.

[0106] Returning to 812, if an entry does not yet exist in the HT, then at 816, a new entry is created in the HT. The new entry in the HT is created with the space ID and page ID pair as the key. The new entry is then populated with the fields described above in step 814.

[0107] After processing the current redo log record, at 818, the parse position is advanced in the GFB to process the next redo log record in the GFB.

[0108] Returning to 804, if all redo log records in the GFB have been processed, then the method 800 proceeds to step 820. For simplicity (and without loss of generality), the following steps will be described based on the assumption that all redo log records in the GFB correspond to the same slice. One of ordinary skill in the art will understand how to modify the following steps to accommodate the case where the redo log records correspond to different slices.

[0109] After processing the GFB, the HT now contains either a combined redo log record or a single redo log record for each unique spatial ID and page ID pair. When populating the SLB for sending to the slice, the entries in the HT can be processed in their natural order and there can be no need to sort the entries in the HT. This is because, as mentioned above, pages with different spatial IDs and page IDs can be arranged in arbitrary order within the GFB boundary.

[0110] At 820, it is determined whether all entries in the HT have been processed. If not, at 822, the next entry in the HT is processed.

[0111] At 824, the values of prev-endLSN, startLSN, endLSN, and total buf len are updated based on the current HT entry. The values of prev-endLSN, startLSN, endLSN, and total buf len will be used for the generation of the subsequent SLB header.

[0112] At 826, the slice redo log record is generated as follows. The record-lsn in the HT entry is copied to the SLB. Then, the redo log record metadata in the record meta-info list field in the HT entry is processed. For each entry in the list, the log record content is copied from the GFB to the SLB based on the byte offset and the record length (obtained in step 808). If the value of num-records is greater than zero, the HT entry corresponds to a group of multiple log records. A new COMPACT type log record is generated to consolidate the grouped records. In addition, other necessary metadata (e.g., checksum, etc.) are also generated for the SLB.

[0113] The method 800 returns to step 820 to determine whether there are any more HT entries to process. If all entries in the HT have been processed, the method 820 proceeds to step 828.

[0114] At 828, the SLB header is generated (containing prev-endLSN, startLSN, endLSN, and total buf len from step 824).

[0115] At 830, it is determined whether the SLB should be sent to the target slice. This determination can be based on the size of the SLB (e.g., whether the buffer is full) and / or based on whether a predefined time interval has been reached (e.g., expiration of a timer). If the SLB has not yet been sent, at 832, the HT is cleared in preparation for processing the next GFB (return to step 802).

[0116] If an SLB is to be sent, the SLB is output to the target slice, and the method 800 ends.

[0117] Figure 9 An example of SLB content that can be generated as described above (e.g., using the method 800) is shown. In this simplified example, there are two slices, slice #1 (with an endLSN value of 50) and slice #2 (with an endLSN value of 20).

[0118] Two GFBs are processed (i.e., GFB 1 and GFB 2), which in this example are two consecutive flush buffers from a GFB with different content. GFB 1 and GFB 2 each include three log records (although in other examples, each GFB can have a different number of log records). Each log record corresponds to a respective page and is assigned a respective LSN. In this example, it is assumed that page 1 is mapped to slice #1 and page 2 is mapped to slice #2. The LSN offset for each redo log record can be determined, and used to track each redo log record as it is processed. The metadata for each GFB includes a preEndLSN value (indicating the previous last LSN in the buffer), a startLSN value (indicating the LSN at which the first record in the buffer begins), and an endLSN value (indicating the LSN at which the last record in the buffer ends).

[0119] As a result of the parsing and compression of the redo log records, as described above, all of the redo log records in GFB 1 that correspond to page 1 are compressed into a consolidated redo log record with LSN = 250, of type COMPACT (where the LSN of the last record in GFB 1 that corresponds to page 1 is used as the LSN of the consolidated redo log record). Similarly, the records in GFB 2 that correspond to page 2 are compressed into a consolidated redo log record with LSN = 400, of type COMPACT (where the LSN of the last redo log record in GFB 2 that corresponds to page 2 is used as the LSN of the consolidated redo log record). In this example, the redo log records in GFB 1 that correspond to page 2 are not compressed; similarly, the redo log records in GFB 2 that correspond to page 1 are also not compressed.

[0120] The slice redo log record in SLB 1 is generated for slice 1. The header of SLB 1 includes a prev-endLSN value of 50, a startLSN value of 250, an endLSN value of 300, and a total buf len value. In SLB 1, the two log records in GFB 1 corresponding to page 1 are compressed into a consolidated redo log record at LSN = 250; and the log record in GFB 2 corresponding to page 1 is stored as a single redo log record at LSN = 300.

[0121] The slice redo log record in SLB 2 is generated for slice 2. The header of SLB 2 includes a prev-endLSN value of 20, a startLSN value of 200, an endLSN value of 400, and a total buf len value. In SLB 2, the redo log record in GFB 1 corresponding to page 2 is stored as a single redo log record at LSN = 200; and the two redo log records in GFB 2 corresponding to page 2 are compressed into a consolidated redo log record at LSN = 400.

[0122] Figure 10 A flowchart illustrating a general method 1000 for consolidating two or more redo log records into a consolidated redo log record is shown. The method 1000 can be performed as part of the method 500 (e.g., performing the optional step 508). The method 1000 can be a generalization of part of the method 800 (e.g., a generalization of steps 820-826).

[0123] At 1002, the log records that have been reordered and grouped by page identification are obtained. For example, step 1002 can follow step 506 as shown. As described above in connection with step 506, the grouping of redo log records can be indicated by an intermediate or temporary data structure (e.g., an HT). Figure 5 Figure 8 As described above in connection with step 506, the grouping of redo log records can be indicated by an intermediate or temporary data structure (e.g., an HT).

[0124] At 1004, for each group of two or more redo log records, a consolidated redo log record is generated. If a group contains only a single redo log record, this operation can be skipped. Steps 1006-1008 can be performed for each group of two or more redo log records to generate a consolidated redo log record.

[0125] At 1006, a new consolidated redo log record is generated (e.g., using the COMPACT type) and a header is created for the consolidated redo log record. The header includes information indicating the page to which the group of redo log records corresponds (e.g., the spaceID field 716 and the pageNumber field 718, as shown in connection with step 506). Other information can also be included in the header, e.g., as shown in connection with step 506. Figure 7 Figure 7 At 1006, a new consolidated redo log record is generated (e.g., using the COMPACT type) and a header is created for the consolidated redo log record. The header includes information indicating the page to which the group of redo log records corresponds (e.g., the spaceID field 716 and the pageNumber field 718, as shown in connection with step 506). Other information can also be included in the header, e.g., as shown in connection with step 506.​​

[0126] At 1008, each individual redo log record in the group is inserted into the body of the merged redo log record. The individual redo log records can be inserted into the merged redo log record in the original LSN order (e.g., in ascending LSN order) or in any LSN order.

[0127] At 1010, the generated merged redo log record is output. Any redo log records that are not included in the merged redo log record (e.g., any redo log records that do not belong to a group of two or more redo log records) can remain unchanged.

[0128] After method 1000 is completed, any groups of two or more redo log records that apply to the same page identifier have been merged into a merged redo log record. Method 500 can continue to be executed (e.g., starting from optional step 510) after method 1000.

[0129] As previously mentioned, a merged redo log record contains the individual redo log records that are unchanged as content in its body. In some examples, a merged redo log record can replace the two or more individual redo log records with a combined redo log record. When the combined redo log record is applied to a database page, it achieves the same effect as the cumulative effect of applying each of the two or more individual redo log records individually. The new combined redo log record can have equivalent “semantics” as the group of two or more redo log records. In this disclosure, semantics can refer to the logical aggregation of database state changes caused by applying a redo log record. Each log record can cause a change to the database state (e.g., increment / decrement a counter). Within a predefined boundary, multiple log records can cause the same type of state change to occur in the database. The type of state change can be defined by the type of data that is changed (e.g., a counter value or a string value) and / or the update operation (e.g., increment / decrement the value or overwrite the value). Since the changes within the predefined boundary are not visible to the client (the client is not aware of the individual changes within the predefined boundary), it is only necessary to ensure that the final cumulative effect of the individual state changes is applied correctly. That is, it is only necessary that the initial state of the database (before applying the redo log records within the boundary) and the changed state of the database (after applying the redo log records within the boundary) are correct. Thus, the individual redo log records in the merged redo log record can be resolved to obtain a single combined redo log record that captures the cumulative effect (or semantics) of the individual redo log records. The combined redo log record can be newly generated, can be a modification of one of the redo log records being combined, or can be one of the redo log records being combined.

[0130] Some examples of how two or more redo log records can be combined into a single combined redo log record are now described. In one example, a first redo log record indicates that a given page at offset 10 should be updated to change that byte to value A, and a second redo log record (which has a larger LSN than the first log record) indicates that the same given page at offset 10 should be updated to change that same byte to value B. Since the LSN of the second redo log record is larger, this means that the second redo log record is the more recent change, and will override the change indicated by the first redo log record. The combined redo log record can then be a redo log record indicating that the given page at offset 10 should be updated to change the byte to value B. In this example, the combined redo log record can be the second redo log record itself.

[0131] In another example, a first redo log record indicates that a given page at offset 10 should be updated to change that byte to value A, and a second redo log record indicates that the same given page at offset 200 should be updated to change another byte to value B. The combined redo log record can then be a log record indicating that the given page at offset 10 should be updated with value A and the same given page at offset 200 should be updated with value B. The combined redo log record can be newly generated, replacing both the first redo log record and the second redo log record; or the combined redo log record can be a modification of one of the first redo log record and the second redo log record, e.g., by inserting the update indicated by the second redo log record into the first redo log record.

[0132] In another example, a first redo log record indicates that a given page at offset 10 should be updated to change that byte to value A, and a second redo log record indicates that the same given page at offset 11 should be updated to change another byte to value B. The combined redo log record can then be a redo log record indicating that the given page at offset 10 should be updated for both bytes, with the first byte updated to value A and the second byte updated to value B. The combined redo log record can be newly generated; or the combined redo log record can be a modification of one of the first redo log record and the second redo log record.

[0133] It will be appreciated that other methods for determining a combined redo log record from two or more redo log records corresponding to the same page can be implemented. For example, a set of combination rules can be defined for determining a combined redo log record in different situations.

[0134] Figure 11A flowchart showing a general method 1100 for combining two or more redo log records into a combined redo log record is shown. The method 1100 can be performed as part of the method 500 (e.g., performing optional step 510).

[0135] At 1102, a merged redo log record containing a plurality of individual redo log records as its content is obtained. For example, step 1102 can follow from step 508 as shown. As previously mentioned, all individual redo log records contained in the merged redo log record are applicable to the same page in the database. Figure 5

[0136] At 1104, the redo log records in the merged redo log record are parsed to determine a combined redo log record from at least two or more individual redo log records that are identified as implementing the same type of state change. If each redo log record in the merged redo log record implements a different type of state change to the database page, then a combined redo log record can not be determined, and this operation can be skipped. Steps 1106-1108 can be performed to determine a combined redo log record. In some examples, determining a combined redo log record can include generating a new combined redo log record; in some examples, determining a combined redo log record can include modifying one of the two or more individual redo log records being combined; in some examples, determining a combined redo log record can include identifying one of the two or more individual redo log records to replace the two or more individual redo log records.

[0137] At 1106, a cumulative state change to the database page implemented by the two or more redo log records is determined. This can include first identifying the type of state change (e.g., based on determining the data type of the redo log record, such as a counter type), and then parsing the redo log records to determine the cumulative state change (e.g., cumulative increase / decrease of a counter).

[0138] At 1108, a combined redo log record is determined, where the combined redo log record is a single redo log record that implements a state change to the database page that is equivalent to the cumulative state change determined in step 1106. The combined redo log record replaces the original two or more individual redo log records.

[0139] In general, there can be various predefined rules that can be used to identify the cumulative effect of redo log records having the same type of state change, and combine the log records into a single combined redo log record.

[0140] ​Some examples of determining a combined redo log record have been discussed above, which can be based on predefined rules. Some other examples of predefined rules are discussed below.

[0141] For example, if two or more redo log records are all updates to values of the same page, a single combined redo log record can be determined that indicates all page identities and corresponding final updated values. The combined redo log record can be newly generated; or, the combined redo log record can be a modification of one of the two or more redo log records (e.g., by selecting a given redo log record among the two or more redo log records, and adding updates of the other redo log records to the given redo log record).

[0142] In another example, if a given redo log record indicates that a given page should be cleaned by emptying all contents (e.g., setting all values to zero), all redo log records with LSNs less than the given redo log record can be discarded, and the given redo log record becomes the combined redo log record that combines the effects of the discarded redo log records and the given redo log record.

[0143] In another example, if two or more redo log records indicate that the same byte values on the same page should be updated to respective values, the redo log record with the largest LSN is kept as the combined redo log record, and all other redo log records with smaller LSNs can be discarded.

[0144] In another example, if two or more redo log records change values of consecutive bytes on a given page, a combined redo log record (e.g., newly generated or modified from one of the two or more redo log records) can be determined that indicates that the page should be updated using the values of the consecutive bytes starting from the earliest offset indicated by the two or more redo log records.

[0145] In another example, for two or more redo log records that apply to the same page identity, the page ID can be removed from all redo log records except one redo log record. Similarly, the LSN (if present) can be removed from all redo log records except the last one (i.e., the last redo log record).

[0146] Other such predefined rules can be implemented as appropriate.

[0147] At 1110, the merged redo log record containing the combined record is output.

[0148] After the method 1100 is completed, the method 500 can continue to be executed (e.g., starting from step 512) after the method 1100.

[0149] The method 1100 has been described above as following the optional step 508 of the method 500. However, in some examples, the combined redo log record can be determined without first generating a merge redo log record. For example, after the redo log records are reordered and grouped by corresponding pages, two or more redo log records within a group can be parsed to determine a combined log redo record (without generating a merge redo log record containing the grouped redo log records).

[0150] The combined redo log record enables two or more page updates (i.e., updates to a page) to be combined in order to process and apply a single update to a page. This can help reduce the use of physical or virtual processing resources.

[0151] The present disclosure describes examples that can help improve the operation of a cloud-based database, for example, by improving the efficiency of the use of memory resources and processing resources of the cloud-based database. The present disclosure enables redo log records that apply to the same page to be reordered, grouped together, and optionally merged and / or combined. This processing of redo log records can be performed in the SAL before being sent to the storage layer for storage in the log store.

[0152] As described above, the log records can be reordered within a predefined boundary without negatively impacting the correctness of the database. For example, the predefined boundary can be an MTR boundary, a GFB boundary, or a multi-GFB boundary.

[0153] The examples described herein can be implemented in the SAL (e.g., using a SAL module). The implementation in the SAL can avoid modifying the underlying database code, which can make it easier to integrate and adopt in a cloud-based database. The process of reordering and grouping (and optionally merging and combining) redo log records can be performed after a GFB is flushed, which is decoupled from the critical path of committing database transactions from the compute layer.

[0154] Although examples have been described that are implemented in the SAL using a SAL module, examples of the present disclosure can be implemented in the storage layer 230.

[0155] The present disclosure provides a way to reorder and group log records that can result in the log records being reordered in the original LSN order. In general, it is not trivial to reorder redo log records across database transactions while ensuring database correctness. The present disclosure describes examples that can reorder redo log records without negatively impacting database correctness. Furthermore, the present disclosure also describes examples of selecting a predefined boundary for reordering and grouping redo log records.

[0156] Although the present application describes certain functions being performed by certain components and physical entities, it should be appreciated that in a distributed system, part or all of the processes can be distributed among multiple components and entities, and that multiple instances of the processes can be executed on a distributed system.

[0157] Although the present application describes methods and processes with steps in a certain order, one or more steps of the methods and processes can be omitted or altered as appropriate. One or more steps can be performed in an order other than the order described.

[0158] Although the present application is described, at least in part, in terms of methods, one of ordinary skill in the art will understand that the application is also directed to various components for performing at least some of the aspects and features described in the methods, whether by hardware components, software, or any combination of the two. Thus, the technical solutions of the present application can be embodied in the form of a software product. The appropriate software product can be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer-readable medium, such as including a DVD, CD-ROM, USB flash disk, mobile hard disk, or other storage medium. The software product includes instructions tangibly stored thereon, which enable a processing device (e.g., a personal computer, a server, or a network device) to perform the method examples disclosed herein.

[0159] The present application can be embodied in other specific forms without departing from the subject matter of the claims. The described exemplary embodiments are to be considered in all respects as illustrative only and not restrictive. Selected features of the above-described embodiments can be combined in alternative embodiments to create alternative embodiments not explicitly described, and all such combinations are understood to fall within the scope of the present application.

[0160] All values and subranges within the disclosed ranges are also disclosed. In addition, while the systems, devices, and processes disclosed and shown herein can include particular amounts of elements / components, these systems, devices, and components can be modified to include more or less of such elements / components. For example, while any disclosed element / component can be referred to as a singular amount, embodiments disclosed herein can be modified to include multiple such elements / components. The subject matter described herein is intended to cover and include all suitable changes.

Claims

1. A method for storing redo log records in a cloud-based database, the method comprising: The method comprises: ​ receiving a plurality of redo log records to apply to a database, data in the database being stored in a plurality of pages within a page storage area; parsing the plurality of redo log records within predefined boundaries to determine, for each given redo log record, a corresponding page to which the given redo log record is to be applied, the predefined boundaries comprising a predefined start boundary coinciding with a mini-transaction, MTR, boundary, and comprising a predefined end boundary coinciding with the same or a different MTR boundary; reordering the plurality of redo log records by corresponding page; storing the reordered redo log records for application to the page storage area of the database.

2. The method of claim 1, wherein, The plurality of redo log records are reordered to group two or more redo log records together based on a same page identifier.

3. The method of claim 1, wherein, The method further comprises: generating, for a given group of two or more redo log records corresponding to a same given page, a merged redo log record, the merged redo log record being a single record containing the two or more redo log records.

4. The method of claim 3, wherein, The merged redo log record comprises a header containing information identifying the given page.

5. The method of claim 3, wherein, The method further comprises: identifying, in the merged redo log record, at least two redo log records to apply to the given page; determining a combined redo log record, the combined redo log record implementing, when applied to the given page, a state change of the given page equivalent to cumulative state changes of the given page implemented by the at least two redo log records; replacing, in the content of the merged redo log record, the identified at least two redo log records by the combined redo log record.

6. The method of claim 1, wherein, The method further comprises: identifying, for a given group of two or more redo log records corresponding to a same given page, at least two redo log records to apply to the given page; determining a combined redo log record, the combined redo log record implementing, when applied to the given page, a state change of the given page equivalent to cumulative state changes of the given page implemented by the at least two redo log records; wherein the identified at least two redo log records are replaced by the combined record.

7. The method of claim 1, wherein, The method is performed in a storage abstraction layer within the cloud-based database, and the reordered redo log records are stored in a buffer for transmission for application to the page storage area.

8. The method of claim 1, wherein, The method is performed in a storage layer within the cloud-based database, and the page storage area is located in the storage layer within the cloud-based database.

9. The method of claim 1, wherein, The predefined start boundary coincides with a start point of a first MTR boundary, and the predefined end boundary coincides with an end point of the first MTR boundary.

10. The method of claim 1, wherein, The predefined start boundary coincides with a start point of a first MTR boundary, and the predefined end boundary coincides with an end point of a second MTR boundary.

11. The method of claim 1, wherein, The predefined boundaries coincide with group flush buffer boundaries. The predefined start boundary coincides with a start point of a first MTR boundary, and the predefined end boundary coincides with an end point of the first MTR boundary. The predefined start boundary coincides with a start point of a first MTR boundary, and the predefined end boundary coincides with an end point of a second MTR boundary. The predefined boundaries coincide with group flush buffer boundaries.

12. A computer readable medium characterized by The computer-readable medium has stored thereon instructions to be executed by a processor in a cloud-based database, the instructions, when executed, cause the following operations to be performed in a storage abstraction layer of the cloud-based database: receiving a plurality of redo log records to be applied to a database, data in the database being stored in a plurality of pages within a page storage area of the database; parsing the plurality of log records within a predefined boundary to determine, for each given redo log record, a corresponding page to which the given redo log record is to be applied, the predefined boundary including a predefined start boundary coinciding with a mini-transaction (MTR) boundary and including a predefined end boundary coinciding with the same or a different MTR boundary; reordering the plurality of redo log records by corresponding page; storing the reordered redo log records to be applied to the page storage area of the database.

13. The computer readable medium of claim 12, wherein, The instructions further cause the following operations to be performed:

14. The computer readable medium of claim 12, wherein, generating, for a given group of two or more redo log records corresponding to a same given page, a merged redo log record, the merged redo log record being a single redo log record containing the two or more redo log records. The instructions further cause the following operations to be performed:

15. The computer readable medium of claim 14, wherein, identifying, in the merged redo log record, at least two redo log records to be applied to the given page; determining a combined redo log record, the combined redo log record, when applied to the given page, implementing a state change of the given page equivalent to cumulative state changes of the given page implemented by the at least two redo log records; replacing, in contents of the merged redo log record, the identified at least two log records with the combined record. The instructions further cause the following operations to be performed:

16. The computer readable medium of claim 12, wherein, identifying, for a given group of two or more redo log records corresponding to a same given page, at least two redo log records to be applied to the given page; determining a combined redo log record, the combined redo log record, when applied to the given page, implementing a state change of the given page equivalent to cumulative state changes of the given page implemented by the at least two redo log records; wherein the identified at least two redo log records are replaced with the combined redo log record. The predefined start boundary coincides with a start of a first MTR boundary and the predefined end boundary coincides with an end of the first MTR boundary.

17. The computer readable medium of claim 12, wherein, The predefined start boundary coincides with a start of a first MTR boundary and the predefined end boundary coincides with an end of a second MTR boundary.

18. The computer readable medium of claim 12, wherein, ​

Images