Time and value ordered data objects for dataset backup

Abstract

Time and value ordering may be applied to items stored in a data backup. A change log that holds changes to a dataset may be updated with changes and used to update a memory table for the dataset that describes changes to items up to a current time. Events that will seal the memory table for subsequent updates may be detected, and persistent data objects, as stored in the change log, may be generated that rank respective items in the memory according to both keys for the items and respective time values for the items, and may store the persistent data object as part of a backup of the data set.

Description

Background Art

[0001] The commoditization of computer hardware and software components has led to the rise of service providers that offer computing and storage capabilities as services. At least some of these services (e.g., management services, such as managed relational database services) can be difficult to scale, including scaling processing capacity. Interruptions to applications or other processes can incur significant costs associated with changing capacity to better match workloads, as client applications may be interrupted (and in some cases, may not even attempt to reconnect) due to a connection loss. Therefore, technologies that can increase the capacity of resources to match workloads are highly desirable. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 A logical block diagram illustrating time- and value-ordered data objects for data set backup according to some embodiments.

[0003] Figure 2 A logical block diagram illustrating a provider network providing a database service that implements time- and value-ordered data objects for data set backup, according to some embodiments.

[0004] Figure 3 is a logical block diagram illustrating the interaction between a storage node and a backup compressor node according to some embodiments.

[0005] Figure 4 A logical block diagram illustrating compression of backup chunks with items sorted by value and time according to some embodiments.

[0006] Figure 5 A high-level flow chart illustrating various methods and techniques for implementing time- and value-ordered data objects for data set backup, according to some embodiments.

[0007] Figure 6 A high-level flow chart illustrating various methods and techniques for compressing time- and value-ordered data objects for data set backups, according to some embodiments.

[0008] Figure 7 is a block diagram illustrating an example computing system in accordance with some embodiments.

[0009] Although embodiments are described herein with reference to several embodiments and schematic diagrams by way of example, it will be appreciated by those skilled in the art that embodiments are not limited to the described embodiments or diagrams. It should be understood that the drawings and their detailed description are not intended to limit the embodiments to the specific forms disclosed, but on the contrary, are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope defined by the appended claims. The titles used herein are for organizational purposes only and are not intended to limit the scope of this specification or the claims. As used throughout this application, the word "may" is used in an allowed sense (i.e., meaning it is possible), rather than a mandatory sense (i.e., meaning it must). Similarly, the word "comprising" means including but not limited to. DETAILED DESCRIPTION

[0010] The techniques described herein can implement time- and value-ordered data objects for backups of data sets. Different types of workloads can utilize different types of techniques to increase the capacity of a database to handle that type of workload. For read workloads, for example, the number of copies of the data that can be read can be increased, thereby supporting parallel access to different copies of the data and therefore increasing the number of read requests that can be executed at any given time. However, write workloads can trigger many different operations (including multiple write operations) in order to perform writes consistently (and replicate changes to other copies). In various embodiments, the time and value data objects of the backup of the data set can provide faster performance for write workloads, as a backup of the database can be efficiently updated using persistent data objects, while minimizing the write operations to be performed so that writes are performed consistently (and across multiple copies).

[0011] Figure 1 A logical block diagram illustrating time- and value-ordered data objects for data set backups according to some embodiments is provided. Data sets, such as databases (e.g., relational, non-relational, time series, and various other types of databases) or other collections of data items, can be stored in a storage area in the form of data objects (e.g., files or other data structures) to provide persistent storage (e.g., backup copies) of data sets that can be accessed for various operations. When changes to a data set are received, the data set can be recorded (and in some embodiments, saved) in a change log, such as change log 110.

[0012] The items in the data set may include various features, attributes, columns, records or values according to the type of the data set. In some embodiments, the data set may store items containing one or more key-value pairs. The key may store an identifier for finding the value paired with the key. The change log 110 may include various updates describing the changes received, such as updates 112a, 112b, 112c and 112d, which include the corresponding key (or other item value identifier), the change of the value paired with the key (modification of the value, insertion of the key and value or removal of the key and value) and the assigned time value (e.g., timestamp). Multiple versions of the same item (e.g., with the same key) may be included in the change log 110 because each update 112 may represent the change of the item over time (e.g., corresponding to different timestamps).

[0013] In addition to the change log 110, an active memory table 120 may also be maintained (e.g., at a node, system, or other component that performs reads and / or writes to a data set), such as described below with respect to Figure 3 The primary storage node in question. The active memory table 120 may be updated based on the change log 110, thereby obtaining new updates and including different updates in the active memory table 120. For example, the "a" item (e.g., key "a") may have a value for the key "a" that changes over time and, therefore, may have different values at different timestamps. Thus, the active memory table 120 may store different versions of the key (e.g., a4, a5, a6, c3, d7, and d8) over time.

[0014] In some embodiments, a triggering event or other condition may cause the active memory table to be sealed (e.g., no longer able to be written to) or otherwise rendered inaccessible to further updates. Simultaneously, new changes to the dataset stored in the change log 110 may be written to the new active memory table 132, such as a modification record for key A as "a7" and a modification record for key "b" as "b2." The sealed memory table 124 may thus store entries, including key-value pairs and corresponding timestamps. Meanwhile, new changes to the dataset stored in the change log 110 may be written to the new active memory table 132.

[0015] To store data from the sealed memory table 134, a new data object 140 (e.g., a file or other data storage unit, collection, or object) may store the items of the sealed memory table 134 in key value and time order (e.g., a1, then a2, then a3, and so on). The data object 140 may implement: a root page 141 that identifies the location of a code page 142 (e.g., indicates the presence and type of encryption and / or compression of the data object 140 for one or more index and / or data pages, and may also allow different encryption and / or compression techniques to be applied to the pages); a metadata page 143 that contains metadata about the data object 140; index pages 144-145 (e.g., b-tree or other index structure pages) that organize the items to be sorted by key value and time order; and data pages 146-147 that may contain items (e.g., record type, storage size, timestamp, key length, key-value pairs, and possibly other information). An example layout of a data page is illustrated, including fields such as record type, storage size, timestamp, key length, key, value, and cyclic redundancy check (CRC) value. In some embodiments, the key and value may be stored as arrays of byte arrays. In some embodiments, multiple data objects, including the new data object 140, may be stored together to form a backup of a data set (e.g., a partition of a database). For example, each of the multiple data objects may together implement a log-structured merge tree (sometimes referred to as an LSM tree).

[0016] Note that the previous descriptions of change logs, memory tables, and data objects are not intended to be limiting, but are provided merely as logical examples.

[0017] This specification begins with a general description of a provider network that can implement a database service that implements time- and value-ordered data objects for dataset backup. Then, in some embodiments, various examples of the database service are discussed, including different components / modules or arrangements of components / modules that can be used as part of implementing the database service. Several different methods and techniques for implementing time- and value-ordered data objects for dataset backup are then discussed, some of which are illustrated in the accompanying flow charts. Finally, a description of an example computing system is provided on which the various components, modules, systems, devices, and / or nodes can be implemented. Various examples are provided throughout this specification.

[0018] Figure 21 is a logical block diagram illustrating a provider network providing a database service that may implement time- and value-ordered data objects for data set backup, according to some embodiments. In some embodiments, provider network 200 may be a private or closed system, or in another embodiment, may be set up by an entity, such as a company or public sector organization, to provide one or more services (e.g., various types of cloud-based storage areas) accessible via the Internet and / or other networks to clients 270. In some embodiments, provider network 200 may be implemented in a single location, or may include numerous data centers hosting various resource pools, such as collections of physical and / or virtualized computer servers, storage devices, networking equipment, etc. (e.g., see below for reference). Figure 7 1000 ), where the data center is required to implement and distribute the infrastructure and storage services provided by the provider network 200. In some embodiments, the provider network 200 may implement various computing resources or services, such as a database service 210 (e.g., a non-relational (NoSQL) database, a relational database service, or other database service that can utilize a collection of items (e.g., a table containing items)), as well as other services (not shown) (e.g., a data stream processing service and / or other large-scale data processing technologies), a data storage service (e.g., an object storage service, a block-based storage service, or a data storage service that can store different types of data for centralized access), a virtual computing service, and / or any other type of network-based service (which may include various other types of storage, processing, analysis, communication, event processing, visualization, and security services).

[0019] In various embodiments, Figure 2 The components shown in can be implemented directly in computer hardware, as instructions executable directly or indirectly by computer hardware (e.g., a microprocessor or computer system), or using a combination of these techniques. Figure 2 The components of can be implemented by a system including several computing nodes (or simply referred to as nodes). In some embodiments, each of the components can be similar to Figure 7 The computer system embodiment shown and described below. In some embodiments, the functionality of a given system or service component (e.g., a component of database service 210) may be implemented by a specific node or may be distributed across several nodes. In some embodiments, a given node may implement the functionality of more than one service system component (e.g., more than one data storage component).

[0020] In some embodiments, database service 210 can implement various types of distributed database services for storing, accessing, and updating data in tables hosted in a database. Such services can be highly scalable and extensible enterprise-level database systems. In some embodiments, access requests (e.g., requests to obtain / get an item, place / insert an item, delete an item, update or modify an item, scan multiple items) can be directed to tables in database service 210 distributed across multiple physical resources, and the database system can be increased or decreased as needed. In some embodiments, clients / subscribers can interactively submit requests to the database system in many ways, such as via a graphical user interface (e.g., a console) or a programming interface. In some embodiments, database service 210 can provide a RESTful programming interface to submit access requests (e.g., to obtain, insert, delete, or scan data). In some embodiments, a query language (e.g., structured query language (SQL)) can be used to specify access requests.

[0021] In some embodiments, client 270 may encompass any type of client that can be configured to submit network-based requests to provider network 200 via network 260, including requests for database service 210 (e.g., to access an item in a table or a sub-index in database service 210). For example, in some embodiments, a given client 270 may comprise a suitable version of a web browser, or may comprise a plug-in module or other type of code module that extends the execution environment provided by the web browser or executes within the execution environment. Alternatively, in various embodiments, client 270 may encompass an application, such as a database client / application (or its user interface), a media application, an office application, or any other application that can utilize a database in database service 210 to store and / or access data to implement various applications. In some embodiments, such an application may include sufficient protocol support (e.g., for a suitable version of the Hypertext Transfer Protocol (HTTP)) for generating and processing network-based service requests, without necessarily implementing comprehensive browser support for all types of network-based data. That is, in some embodiments, client 270 may be an application that interacts directly with provider network 200. In some embodiments, client 270 may generate a network-based service request according to a Representational State Transfer (REST) style network-based service architecture, a document-based or message-based network-based service architecture, or another suitable network-based service architecture. It should be noted that in some embodiments, a client of database service 210 may be implemented within provider network 200 (e.g., an application hosted on a virtual computing service).

[0022] In some embodiments, a client of database service 210 may be implemented on resources within provider network 200 (not shown). For example, the client application may be hosted on a virtual machine or other computing resource implemented as part of another provider network service that can send access requests to database service 210 via an internal network (not shown).

[0023] In some embodiments, client 270 can provide other applications with access to provider network 200 in a manner that is transparent to those applications. For example, client 270 can integrate with a database on database service 210. In such embodiments, applications may not need to be modified to utilize the service model of database service 210. Instead, the details of interfacing with database service 210 can be coordinated by client 270.

[0024] In some embodiments, client 270 may transmit network-based service requests to provider network 200 via network 260 and receive responses from the provider network. In some embodiments, network 260 may include any suitable combination of network hardware and protocols required to establish network-based communications between client 270 and provider network 200. For example, network 260 may include various telecommunications networks and service providers that jointly implement the Internet. In some embodiments, network 260 may also include private networks, such as local area networks (LANs) or wide area networks (WANs), as well as public or private wireless networks. For example, a given client 270 and provider network 200 may be provisioned within an enterprise having its own internal network. In such embodiments, network 260 may include the hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall / security software, etc.) required to establish networking links between a given client 270 and the Internet, and between the Internet and provider network 200. It should be noted that in some embodiments, client 270 may communicate with provider network 200 using a private network rather than the public Internet.

[0025] In some embodiments, the database service 210 may implement a request router 250. In various embodiments, the request router 250 may receive and analyze client access requests to determine various characteristics of the requests and, in some embodiments, further analyze, authenticate, restrict, and / or schedule the access requests. The database service 210 may implement a backup compression node 250 that may perform compression techniques including the generation of new data objects and merged data objects, as described below with respect to Figure 3 and 4 Discussed in detail.

[0026] In some embodiments, the database service 210 may implement a control plane 220 to implement one or more management components, such as an automated administrator instance (which may provide various visibility and / or control functions). In various embodiments, the control plane 220 may, in some embodiments, direct the execution of different types of control plane operations within the nodes, systems, or devices implementing the database service 210. In some embodiments, the control plane 220 may provide visibility and control to system administrators via an administrator console 226. The administrator console 226 may allow system administrators to interact directly with the database service 210 (and / or underlying systems). In some embodiments, the administrator console 226 may be the primary point of visibility and control for the database service 210 (e.g., for configuration or reconfiguration by a system administrator). For example, the administrator console may be implemented as a relatively thin client that provides functional display and control to the system administrator and / or other authorized users, and through which system status indicators, metadata, and / or operating parameters may be observed and / or updated. In some embodiments, control plane 220 may provide an interface or access to stored information at database service 210 regarding one or more detected control plane events (eg, data backup or other management operations for a table).

[0027] In some embodiments, storage node management 224 may provide resource allocation for storing additional data in tables submitted to database service 210. For example, control plane 220 may communicate with processing nodes 230 to initiate execution of various control plane operations, such as moving table partitions, merging table partitions, splitting table partitions, updating tables, deleting tables, creating secondary indexes, etc. In some embodiments, control plane 220 may include node recovery features or components that handle failure events of storage nodes 230, propagation architecture 290, and request routers 250 (e.g., adding new nodes, removing failed or underperforming nodes, deactivating or decommissioning underutilized nodes, etc.).

[0028] Various durability, resilience, control, or other operations may be directed by the control plane 220. For example, the storage node management 224 may detect merge, split, copy, or move events for partitions at a storage node in order to ensure that the storage node maintains a minimum performance level for executing access requests. For example, in various embodiments, there may be situations where a partition (or its replicas) may need to be copied, for example, from one storage node to another. For example, if there are three replicas of a particular partition, each hosted on a different physical or logical machine, and one of the machines fails, then the replica hosted on that machine may need to be replaced by a new replica of the partition on another machine. In another example, if a particular machine hosting multiple partitions of one or more tables experiences heavy traffic, then one of the frequently accessed partitions may be moved (using a copy operation) to a machine experiencing less traffic in an attempt to distribute the system workload more evenly and improve performance. In some embodiments, the storage node management 224 may perform partition moves using a physical copy mechanism (e.g., a physical file system mechanism, such as a file copy mechanism) that copies the entire partition from one machine to another, rather than copying a snapshot of the partition's data rows. As a partition is copied, write operations targeting the partition can be logged. During the copy operation, any logged write operations can be applied to the partition at periodic intervals (e.g., at a series of checkpoints) through a matching process. Once the entire partition has been copied to the destination machine, any remaining logged write operations (i.e., any write operations performed since the last checkpoint) can be performed on the destination partition through a final matching process. Thus, in some embodiments, after the partition move is completed, the data in the destination partition can be consistent. In this way, the storage node management 224 can move partitions among the storage nodes 230 while the partition being moved is still "active" and able to accept access requests.

[0029] In some embodiments, the partition move process described above may be employed by the storage node management 224 in a partition split operation in response to detecting a partition split event. For example, a partition may be split because it is large, such as when it becomes too large to fit on one machine or storage device and / or in order to keep the partition size small enough to quickly rebuild the partition hosted on a single machine (using a large number of parallel processes) in the event of a machine failure. A partition may also be split when it becomes too "hot" (i.e., when it experiences a much larger average traffic volume than other partitions). For example, if the workload for a given partition changes suddenly and / or significantly, the system may be configured to react quickly to the change. In some embodiments, the partition splitting process described herein may be transparent to applications and clients / users, which may allow the data storage service to automatically (i.e., without client / user intervention or initiation) scale.

[0030] In some embodiments, each database partition 234 may be identified by a partition ID, which may be a unique number (e.g., a GUID) assigned when the partition is created. The partition's version number may be incremented each time a partition 234 undergoes reconfiguration (e.g., in response to adding or removing a replica, but not necessarily in response to a master failover). In some embodiments, when a partition is split, two new partitions may be created, each of which may have a corresponding new partition ID, and the original partition ID may no longer be used. In some embodiments, a partition may be split by the system in response to changing conditions using a splitting tool or process.

[0031] Split or move events can be detected in various ways by storage node management 224. For example, partition size and heat can be evaluated based on various performance thresholds, where heat can be tracked by internal metrics (e.g., IOPS), external metrics (e.g., latency), and / or other factors.

[0032] In some embodiments, a system anomaly may also trigger a split or move event (e.g., a network partition that interrupts communication between replicas of a partition in a replica group). In some embodiments, the storage node management 224 may detect storage node failures, or provide other anomaly controls. If the partition replica hosted on the storage node on which the failure or failure is detected is the master for its replica group, a new master may be selected for the replica group (e.g., from among the remaining storage nodes in the replica group). In some embodiments, the storage node management 224 may initiate the creation of a replacement partition replica while the source partition replica is active (i.e., while one or more of the partition's replicas continue to accept and service requests for the partition). In various embodiments, the partition replica on the failed storage node may be used as the source partition replica, or another replica of the same partition (on a working machine) may be used as the source partition replica, for example, depending on the type and / or severity of the detected failure.

[0033] In some embodiments, the control plane 220 may implement table / index creation and management 222 to manage the creation (or deletion) of database tables and / or secondary indexes hosted in the database service 210. For example, a request to create a secondary index may be submitted via an administrator console 226 (or other database service 210 interface), which may initiate execution of a workflow to generate appropriate system metadata (e.g., a table identifier unique to all other tables in the database service 210, secondary index performance or configuration parameters, and / or various other operations for creating secondary indexes, as described below). Backup management (not shown) may process or manage backup requests to make copies of versions or points in time of a database as partition snapshots 242 and partition change logs 244, which together constitute partition backups 241 in the storage service 240, which may be used to perform offline builds of replicated data sets similar to secondary indexes.

[0034] In some embodiments, the database service 210 may also implement multiple storage nodes 230, each of which may manage one or more partitions or sub-indexes of a database table on behalf of a client / user or on behalf of the database service 210, which may be stored in a database storage area 234 (on a storage device attached to the storage node 230 or in a network storage area accessible to the storage node 230).

[0035] In some embodiments, storage node 230 may implement item request processing 232. In some embodiments, item request processing 232 may perform various operations (e.g., read / get, write / update / modify / alter, insert / add, or delete / remove) to access individual items in a table stored in database service 210. In some embodiments, item request processing 232 may support operations performed as part of a transaction, including, for example, locking items within a transaction and / or sorting requests to operate on items as part of a transaction based on timestamps (e.g., timestamp ordering), as well as other techniques for ordering requests so that storage node 230 can accept or reject requests related to the transaction. In some embodiments, item request processing 232 may maintain database partitions 234 according to a database model (e.g., non-relational, i.e., NoSQL or other key-value database models).

[0036] In some embodiments, database service 210 may provide functionality for creating, accessing, and / or managing tables or sub-indexes at nodes within a multi-tenant environment. For example, in some embodiments, database partition 234 may store table entries from multiple tables, indexes, or other data stored on behalf of different clients, applications, users, accounts, or unrelated entities.

[0037] In addition to partitioning or otherwise distributing data (e.g., a database table) across storage nodes 230 in separate partitions, storage nodes 230 may also be used in a number of different arrangements to provide resiliency and / or durability of data as part of a larger collection or resource group. For example, a replica group may consist of multiple storage nodes that maintain replicas of a particular portion (e.g., a partition) of the data of a database service 210. In some embodiments, a replica group may include a primary storage node that may act as, for example, a read and write node for a partition. As described below with respect to Figure 3 As discussed in detail, the primary storage node of a replica group may also participate in management, and further, different replica groups may utilize overlapping nodes, where a storage node 230 may be a member of multiple replica groups, thereby maintaining replicas for each of those groups in which other storage nodes 230 are members different from other replica groups.

[0038] In some embodiments, different models or formats for storing data in database tables within the database service 210 may be implemented. For example, in some embodiments, non-relational, i.e., NoSQL, semi-structured, or other key-value data formats may be implemented. In at least some embodiments, the data model may include tables containing items with one or more attributes. In such embodiments, each table maintained on behalf of a client / user may contain one or more items, and each item may contain a set of one or more attributes. In some embodiments, the attributes of an item may be a set of one or more key-value pairs in any order. In some embodiments, each attribute in an item may have a key, a type, and a value. In some embodiments, items may be managed by assigning a primary key value (which may contain one or more attribute values) to each item, and this primary key value may also be used to uniquely identify the item. In some embodiments, a large number of attributes may be defined across items in a table, but each item may contain a sparse set of these attributes (where a particular attribute specified for an item is unrelated to the attributes of another item in the same table). In some embodiments, all attributes except the primary key attribute and the version attribute may be optional. In some embodiments, tables maintained by the database service 210 (and the underlying storage system) may not have a predefined format beyond their dependence on a primary key.

[0039] In some embodiments, metadata for a table or other system data may also be stored as part of a database partition using a similar partitioning scheme and using similar indexes.

[0040] The database service 210 may provide an application programming interface (API) for requesting various operations targeting tables, indexes, items, and / or attributes maintained on behalf of storage service clients. In some embodiments, the service (and / or the underlying system) may provide both a control plane API and a data plane API. The control plane API provided by the database service 210 (and / or the underlying system) may be used to manipulate table-level entities, such as tables and indexes and / or reconfigure various tables. These APIs may be called relatively infrequently (when compared to the data plane API). In some embodiments, the control plane API provided by the service may be used to create a table or a secondary index for a table at a separate storage node, import a table, export a table, delete a table or a secondary index, explore a table or a secondary index (e.g., to generate various performance reports or deviation reports), modify table configuration or operational parameters of a table or a secondary index and / or describe a table or a secondary index, and create functions and / or associate functions with a table. In some embodiments, the control plane API that performs updates on table-level entries may call an asynchronous workflow to perform the requested operation. Methods requesting "description" information (e.g., via the described table API) may simply return the currently known state of a table or sub-index maintained by the service on behalf of the client / user. The data plane API provided by database service 210 (and / or underlying systems) may be used to perform item-level operations, such as queries, batch operations, and / or scans, on individual items or multiple items in one or more tables.

[0041] In various embodiments, the APIs provided by the services described herein may support request and response parameters encoded in one or more industry-standard or proprietary data exchange formats. For example, in various embodiments, requests and responses may conform to a human-readable (e.g., text-based) data exchange standard (e.g., JavaScript Object Notation or JSON), or may be represented using a binary encoding (which in some cases may be more compact than a text-based representation). In various embodiments, the system may supply default values (e.g., system-wide, user-specific, or account-specific default values) for one or more of the input parameters of the APIs described herein.

[0042] The database service 210 may include support for some or all of the following operations on data maintained in tables (or indexes) by the service on behalf of storage service clients: performing transactions (including one or more operations on one or more items in one or more tables), putting (or storing) items, getting (or retrieving) one or more items with a specified primary key, deleting items, updating attributes in a single item, querying items using an index, and scanning an entire table (e.g., listing items), optionally filtering the items returned or conditional variations of the above automatically performed operations (e.g., conditional put, conditional get, conditional delete, conditional update, etc.). For example, the database service 210 (and / or the underlying system) described herein may provide various data plane APIs for performing item-level operations, such as a TransactItems API, a PutItemAPI, a GetItem (or GetItems) API, a DeleteItem API, and / or an UpdateItem API, as well as one or more index-based search / traversal operations across multiple items in a table, such as a Query API and / or a Scan API.

[0043] Storage service 240 may be a file, object-based, or other type of storage service that can be used to store partition snapshots 242 as backups 241. Storage service 240 may implement splitting, sharding, or other data distribution techniques so that different portions of partition backup 241 are stored in multiple locations (e.g., on separate nodes). In at least some embodiments, update log 244 (e.g., created by item request processing 232 for updates to database partition 234) may be stored as an object in storage service 240.

[0044] Figure 3 310 。 The logical block diagram of the interaction between the storage node and the backup compressor node according to some embodiments is shown. A write request 302 can be received at a request router 310 and forwarded 312 to an appropriate primary storage node for writing the partition targeted. The primary storage node 320a can utilize a replication protocol to perform writes (e.g., by sending writes to secondary storage nodes 320b and 320c, as shown in 321 and 323). Confirmations can be returned, such as confirmations 322 and 324 to indicate to the primary storage node 320a that the write was successfully completed (e.g., to meet a quorum or other durability requirements). The primary storage node 320a can then forward a write response (or failure) as indicated at 314 to the request router 310, which can return a write response 304.

[0045] The primary storage node 320a may store the changes in the change log 330 data store 380, as indicated at 325. For example, the data store 380 may be an append-only data store (e.g., a streaming data store). The primary storage node 320a may maintain an active memory table 371 and (if not determined to be flushed to the data store 370) a sealed memory table 372. These active memory tables 371 and sealed memory tables 372 may be similar to those described above with respect to Figure 1 Each of the secondary storage nodes 320b and 320c and the backup compressor node 350 may maintain corresponding copies of the memory tables discussed.

[0046] As mentioned above about Figure 1 As discussed, backup node compressor 350 may generate new data objects, sometimes referred to as backup objects, by obtaining 351 changes from change log 330. In some embodiments, change 351 may indicate a sealed memory table (as determined by primary storage node 320a based on a detected triggering event as discussed below), such as one of sealed memory tables 372. Backup compressor node 350 may then generate a new backup object corresponding to sealed memory table 372 (as described in the change log) and store 352 the backup object in storage service 240 as part of backup objects 340a, 340b, 340c, 340d, etc.

[0047] A notification that the backup object is being stored may be returned, as indicated at 356, which may allow the primary storage node 320a (and the secondary storage nodes 320b and 320c) to discard the sealed memory table 372, thereby freeing up memory. In addition, the notification 356 may indicate to the primary storage node 320a (and the secondary storage nodes 320b and 320c) that the backup object may be obtained from the storage service 370 (if necessary). Figure 4 As discussed, backup compressor node 350 may also compress backup objects by obtaining backup objects 353 and storing a merged backup object 354 .

[0048] Figure 41 is a logical block diagram illustrating compression of backup blocks that sort items by value and time, according to some embodiments. Partition backup 402 may be a collection of persistent data objects 410a, 410b, and 410c stored as discussed above with respect to data object 140. Each data object 410 may store different versions of an item sorted by time and key (e.g., in persistent data object 410a, key "b" is sorted by time as "b1," then "b2," then "b3"). Furthermore, each persistent data object 410 may correspond to a different time range (e.g., for key "b," no updates were received within the time range of persistent data object 410b). Thus, a persistent data object may store keys across the entire space of a partition, but may have a specific time range that contains the oldest and newest updates described therein.

[0049] Two (or more) data objects 410 may be identified for merging, such as data objects 410a and 410b. Various techniques may be used to identify persistent data objects for merging. For example, one technique may be level compression. In this technique, each persistent data object will be associated with a level. The most recently flushed persistent data object from the in-memory table will have a level = 0. Each time a persistent data object is compressed (and thus merged with another persistent data object), its level will be incremented by 1. Once the number of data objects on a level reaches X (e.g., 2), compression of that level may be triggered. Thus, at Figure 4 In the example shown, persistent data objects 410a and 410b will both be two objects at level = 1, and therefore both will be compressed and merged, with merged data object 412a having level = 2. In this way, compression can continue to merge persistent data objects over time. Consider an example configuration where level 0 is a sealed in-memory table, level 1 is two 64MB files, level 2 is two 128MB files, level 3 is two 256MB files, level 4 is two 512MB files, and so on. In some embodiments, compression can merge all persistent data objects into a single persistent data object.

[0050] Compression techniques can create two data objects to replace the merged data object. The merged data object 412a can store items in two data objects sorted by value and time, while not retaining any items whose time values are not within a retention threshold (e.g., 30 days). In this way, the partition backup 402 can still retain the ability to perform point-in-time or other restores within the retention threshold. The merged data object 412a can contain the same information as described above regarding Figure 1 The structure discussed above is similar to that described above, including a new root page, a code page, a metadata page, an index page, and a data page. The current data object 412b can store the key values of the items in the merged data object that are the latest in terms of time.

[0051] The above description has been given about database services (e.g., relational databases, document databases, non-relational databases, etc.) Figure 2-4 The example of a database implementing a time- and value-ordered data object for data set backup discussed in . However, in other embodiments, various other types of database systems or storage systems may advantageously implement a time- and value-ordered data object for data set backup. Figure 5 A high-level flow chart illustrating various methods and techniques for implementing time- and value-ordered data objects for data set backup according to some embodiments. Figure 6 The techniques discussed above can be used as described in Figure 2-4 The components or systems described herein may be implemented with other types of database or storage systems, and thus the following discussion is not intended to limit other types of systems in which the described techniques may be implemented.

[0052] As indicated at 510, in some embodiments, updates to corresponding items in a data set stored in a change log may be accessed, the corresponding items being sorted by corresponding time values assigned to the updates. Figure 1 As discussed, the change log can store updates corresponding to different values of the key-value pair of the item. Each time value can be, for example, a timestamp or a sequence number that indicates other times with respect to other changes.

[0053] As indicated at 520, according to some embodiments, the memory table may be updated based on multiple updates with corresponding time values to include corresponding entries. For example, the change log may be persistent and may be used to ensure that changes, once committed to the dataset, are persisted even in the event of a failure, while the memory table (e.g., Figure 1 The active memory table 120 in the memory table can be in a temporary or volatile storage area that may lose its contents in the event of, for example, a power or application failure. The memory table can be used to answer access requests (e.g., read an item in a data set) without having to access a persistent copy of the data set.

[0054] As indicated at 530, according to some embodiments, an event may be detected that will seal the memory table for subsequent updates, where the subsequent updates are stored in a change log and used to update a new memory table. In some embodiments, different events may cause the sealing of the memory table. For example, a size threshold (e.g., 64MB) may be applied to the amount of storage (e.g., size) of the memory table such that sealing may occur when the size threshold is exceeded. In some embodiments, the number of items in the memory table may be compared to an item threshold such that sealing may occur when the item threshold is exceeded. In some embodiments, the event threshold or other criteria may be dynamic so as to respond to changing conditions. For example, the memory table may be maintained on a storage node (as described above with respect to Figure 3 If the workload (e.g., the number and / or complexity of requests) results in an increased need for memory to hold, for example, temporary results, connection states, or various other information, then this memory pressure may lower the size or item threshold, causing sealing to occur more quickly. Figure 3 As discussed above, in some embodiments, a remote component (e.g., a backup compression node may perform persistent data object storage) may thus provide an indication or notification of a sealed memory table (e.g., as a message or indication sent directly to the remote component, or as discussed above, as a record, change, or other indication contained in a change log (which may also describe system operation) that may be read or obtained by the remote component). In some embodiments, a single component (e.g., a storage node, a database engine, or other data processing component that provides access to a data set) may perform Figure 5 Each of the features described in .

[0055] As indicated at 540, according to some embodiments, a persistent data object may be generated that orders the respective items according to both their respective keys and their respective time values. For example, a Figure 1 , wherein a root page, metadata pages, code pages, index pages, and data pages may be created. As discussed above, compression techniques may be performed to encode pages in a persistent data object, and in some embodiments, different compression techniques may be applied to different pages as needed (e.g., an optimal compression technique may be identified for each page, and the optimal compression technique may vary from page to page). Similarly, encryption techniques may be performed to encode pages in a persistent data object. As with compression, in some embodiments, different encryption techniques may be applied to different pages, while in other embodiments, a common encryption technique may be applied to pages (e.g., in addition to a root page and / or code page that may hold encryption keys (encryptedly wrapped using a different encryption key obtained elsewhere) to decrypt the page).

[0056] As indicated at 550, according to some embodiments, a persistent data object may be stored as part of a backup of a dataset. For example, a collection of persistent data objects may be maintained as part of a backup to support various recovery operations, such as restoring data in the event of a failure or restoring access to data at a previous point in time. The collection of persistent data objects may collectively provide access to all versions of an item in the dataset (e.g., including before deletion) for access. In at least some embodiments, the collection of persistent data objects may be for a specific partition of the dataset. In at least some embodiments, the collection of persistent data objects may be a log-structured merge (LSM) tree.

[0057] Figure 6A high-level flow chart illustrating various methods and techniques for compressing time- and value-ordered data objects for use in a backup of a data set in accordance with some embodiments. As indicated at 610, in accordance with some embodiments, a compression event may be detected for a backup of a data asset. As discussed in detail above, compression events may be detected in various ways, such as based on the number of persistent data objects (e.g., at a particular size or level). Compression events may be detected based on the availability of resources for performing the compression (e.g., as a background process that does interfere with client application workload (e.g., requests)). In some embodiments, compression events may occur periodically (e.g., hourly). In some embodiments, compression may be requested (e.g., by another component that monitors backup storage utilization in order to cause more efficient read performance or reduce storage utilization).

[0058] As indicated at 620, according to some embodiments, persistent data objects to be merged in the backup of the data set may be identified. For example, the two oldest persistent data objects (e.g., at a level or overall) may be identified. In another example, the two youngest persistent data objects may be identified. In some embodiments, all persistent data objects (e.g., two or more) may be identified for merging. In some embodiments, the persistent data objects identified for merging may have adjacent time ranges such that, when merged, no other persistent data objects also store items with corresponding time values within the time range of the merged objects. For example, if object A is T1 to T10, object B is T11 to T20, and object C is T21 to T30, then A and C will not be identified for merging if B is not identified, but A and B may be identified without C, or B and C may be identified without A.

[0059] As indicated at 630, according to some embodiments, a merged persistent data object containing items from the persistent data objects is generated, the merged persistent data object being sorted according to the value of the item and the corresponding time value of the item, and containing the value of the item if the corresponding time value of the item is within a retention threshold. Figure 1 、 3 The generation of persistent data objects discussed in , 4, and 5 may use an example file format, including a root page, metadata page, encoding page, index page, and data page. In some embodiments, the encoding may be modified, including different compression or encryption (or if not previously applied), but in other embodiments, it may not be modified.

[0060] According to some embodiments, a current item version of a merged persistent data object may be generated, as indicated at 640, containing the values of those items having the most recent time values of the items. In this way, in some scenarios, the current item version persistent data object may be read first to check the most current values of the items without having to read the merged persistent data object.

[0061] In various embodiments, the methods described herein may be implemented by any combination of hardware and software. For example, in some embodiments, the methods may be implemented by a computer system (e.g., Figure 7 The present invention is implemented in a computer system (comprising one or more processors executing program instructions stored on a computer-readable storage medium coupled to the processors). The program instructions may implement the functionality described herein (e.g., the functionality of the various servers and other components of the distributed system described herein). The various methods illustrated in the figures and described herein represent example embodiments of the methods. The order of any method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.

[0062] Embodiments implementing time- and value-ordered data objects for data set backup as described herein may be executed on one or more computer systems, which may interact with various other devices. Figure 7 In various embodiments, computer system 1000 may be any of various types of devices, including, but not limited to, a personal computer system, a desktop computer, a laptop computer, a notebook or netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a camera, a set-top box, a mobile device, a consumer device, a video game console, a handheld video game device, an application server, a storage device, a peripheral device such as a switch, a modem, a router, or generally any type of computing node, computing device, or electronic device.

[0063] In the illustrated embodiment, the computer system 1000 includes one or more processors 1010 coupled to a system memory 1020 via an input / output (I / O) interface 1030. The computer system 1000 further includes a network interface 1040 coupled to the I / O interface 1030, and one or more input / output devices 1050, such as a cursor control device, a keyboard, and a display. In some embodiments, the display may include a standard computer monitor and / or other display systems, technologies, or devices. In some embodiments, it is contemplated that a single instance of the computer system 1000 may be used to implement the embodiments, while in other embodiments, multiple such systems or multiple nodes comprising the computer system 1000 may host different portions or instances of the embodiments. For example, in one embodiment, some elements may be implemented via one or more nodes of the computer system 1000 that are different from those nodes that implement other elements.

[0064] In various embodiments, computer system 1000 may be a uniprocessor system including one processor 1010, or a multiprocessor system including several processors 1010 (e.g., two, four, eight, or another suitable number). In some embodiments, processor 1010 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1010 may be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each of processors 1010 may, but need not, collectively implement the same ISA.

[0065] In some embodiments, at least one processor 1010 may be a graphics processing unit (GPU). In some embodiments, a graphics processing unit (GPU) or a GPU may be considered as a dedicated graphics rendering device for a personal computer, workstation, game console, or other computing or electronic device. Modern GPUs may be very efficient in manipulating and displaying computer graphics, and their highly parallel structure may make them more effective than typical CPUs for a series of complex graphics algorithms. For example, a graphics processor may enable the execution of multiple graphics primitive operations much faster than directly drawing to the screen with a host central processing unit (CPU). In various embodiments, graphics rendering may be implemented at least in part by program instructions for executing on one of such GPUs or executing in parallel on two or more of such GPUs. In some embodiments, a GPU may implement one or more application programming interfaces (APIs) that allow programmers to call the functions of a GPU.

[0066] In some embodiments, system memory 1020 may store program instructions 1025 and / or data accessible by processor 1010 to associate functions with tables in a database system. In various embodiments, system memory 1020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions and data to implement the desired functions (e.g., those described above for performing offline builds for a planned data subset) are shown within system memory 1020 as program instructions 1025 and data storage area 1035, respectively. In other embodiments, program instructions and / or data may be received, sent, or stored on different types of computer-accessible media or on similar media separate from system memory 1020 or computer system 1000. Computer-accessible media may include non-transitory storage media or memory media, such as magnetic or optical media, such as a disk or CD / DVD-ROM coupled to computer system 1000 via I / O interface 1030. In one embodiment, program instructions and data stored via computer-accessible media may be transmitted by a transmission medium or signal (e.g., an electrical, electromagnetic, or digital signal), which may be transmitted via a communication medium (e.g., a network and / or a wireless link), for example, implemented via network interface 1040.

[0067] In some embodiments, I / O interface 1030 may facilitate I / O traffic between processor 1010, system memory 1020, and any peripheral devices in the device, including network interface 1040 or other peripheral interfaces, such as input / output devices 1050. In some embodiments, I / O interface 1030 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 1020) into a format suitable for use by another component (e.g., processor 1010). In some embodiments, I / O interface 1030 may include support for devices attached via various types of peripheral buses, such as variations of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of I / O interface 1030 may be separated into two or more separate components, such as a northbridge and a southbridge. Additionally, in some embodiments, some or all of the functionality of I / O interface 1030, such as the interface to system memory 1020, may be incorporated directly into processor 1010.

[0068] In some embodiments, network interface 1040 may allow data to be exchanged between computer system 1000 and other devices attached to a network (such as other computer systems) or between nodes of computer system 1000. In various embodiments, network interface 1040 may support, for example, communication via a wired or wireless general-purpose data network (e.g., any suitable type of Ethernet network); communication via a telecommunications / telephone network (e.g., an analog voice network or a digital fiber-optic communications network); communication via a storage area network (e.g., a Fibre Channel SAN); or communication via any other suitable type of network and / or protocol.

[0069] In some embodiments, input / output devices 1050 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or retrieving data by one or more computer systems 1000 in some embodiments. In some embodiments, multiple input / output devices 1050 may be present in computer system 1000 or may be distributed across different nodes of computer system 1000. In some embodiments, similar input / output devices may be separate from computer system 1000 and may interact with one or more nodes of computer system 1000 via a wired or wireless connection (e.g., via network interface 1040).

[0070] like Figure 7 As shown in FIG, in some embodiments, memory 1020 may include program instructions 1025 that implement various embodiments of the system as described herein, and data storage 1035 that includes various data accessible by program instructions 1025. In one embodiment, program instructions 1025 may include software elements of embodiments as described herein and shown in the accompanying figures. Data storage 1035 may include data that may be used in the embodiments. In other embodiments, other or different software elements and data may be included.

[0071] Those skilled in the art will understand that computer system 1000 is merely illustrative and is not intended to limit the scope of the embodiments described herein. Specifically, computer systems and devices may include any combination of hardware or software that can perform the indicated functions, including computers, personal computer systems, desktop computers, laptop computers, notebook computers or netbook computers, mainframe computer systems, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, network devices, Internet appliances, PDAs, wireless phones, pagers, consumer devices, video game consoles, handheld video game devices, application servers, storage devices, peripheral devices (such as switches, modems, routers), or generally any type of computing or electronic device. Computer system 1000 may also be connected to other devices not shown, or alternatively may operate as a standalone system. In addition, in some embodiments, the functions provided by the shown components may be combined in fewer components or distributed in other components. Similarly, in some embodiments, the functions of some of the illustrated components may not be provided and / or other additional functions may be used.

[0072] Those skilled in the art will also understand that, although various items are described as being stored in memory or in a repository when in use, for the purposes of memory management and data integrity, these items or portions thereof may be transferred between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components may be executed in memory on another device and communicate with the computer system shown via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable item to be read by an appropriate drive, various examples of which are described above. In some embodiments, instructions stored on a computer-readable medium separate from the computer system 1000 may be transmitted to the computer system 1000 via a transmission medium or signal (e.g., electrical, electromagnetic, or digital signal), which is transmitted via a communication medium (e.g., a network and / or wireless link). This computer-readable storage medium may be non-transitory. Various embodiments may further include receiving, sending, or storing instructions and / or data implemented according to the foregoing description on a computer-accessible medium. Therefore, the present invention may be practiced with other computer system configurations.

[0073] Embodiments of the present disclosure may be described in terms of the following:

[0074] Clause 1. A system comprising:

[0075] at least one processor; and

[0076] a memory storing program instructions that, when executed by at least one processor, cause the at least one processor to implement a database system, the database system being configured to:

[0077] recording a plurality of updates to corresponding items in a data set stored in a change log, the corresponding items being ordered by corresponding time values assigned to the plurality of updates;

[0078] updating the memory table to include corresponding items based on the plurality of updates having corresponding time values;

[0079] Detecting events that will seal the memtable against subsequent updates, where subsequent updates are stored in the change log and used to update the new memtable;

[0080] generating a persistent data object that orders the respective items according to both their respective keys and their respective time values; and

[0081] Stores persistent data objects as part of a dataset's backup.

[0082] Clause 2. The system of clause 1, wherein the database system is further configured to:

[0083] Detect compression events for backups of datasets;

[0084] identifying two or more persistent data objects to be merged in a backup of the data set; and

[0085] A merged persistent data object is generated that contains items from two or more persistent data objects, wherein the merged persistent data object is ordered according to keys of the contained items and corresponding time values of the items, wherein an item is contained in the two or more persistent data objects when the corresponding time value of the item is within a retention threshold.

[0086] Clause 3. The system of clause 2, wherein the database system is further configured to generate a current item version of the merged persistent data object, the current item version containing the keys of those items having the latest time values of the items.

[0087] Clause 4. The system of any of clauses 1 to 3, wherein the database system is a database service implemented as part of a provider network, wherein the data sets are partitions of database tables.

[0088] Clause 5. A method comprising:

[0089] accessing, by one or more computing devices of the distributed data store, a plurality of updates to respective items in the data set stored in the change log, the respective items being ordered by respective time values assigned to the plurality of updates;

[0090] updating, by one or more computing devices, a memory table based on a plurality of updates having corresponding time values so that the memory table includes corresponding entries;

[0091] detecting, by one or more computing devices, an event that will seal the memtable from subsequent updates, wherein the subsequent updates are stored in a change log and used to update a new memtable;

[0092] generating, by one or more computing devices, a persistent data object that orders the respective items according to both their respective keys and their respective time values; and

[0093] The persistent data objects are stored by one or more computing devices as part of a backup of the data set.

[0094] Clause 6. The method of clause 5, further comprising:

[0095] Detect compression events for backups of datasets;

[0096] identifying two or more persistent data objects to be merged in a backup of the data set; and

[0097] A merged persistent data object is generated that contains items from two or more persistent data objects, wherein the merged persistent data object is ordered according to keys of the contained items and corresponding time values of the items, wherein an item is contained in the two or more persistent data objects when the corresponding time value of the item is within a retention threshold.

[0098] Clause 7. The method of Clause 6, further comprising generating a current item version of the merged persistent data object, the current item version containing the values of those items having the most recent time values of the items.

[0099] Clause 8. The method of any one of clauses 5 to 7, wherein generating the persistent data object comprises encrypting the index structure page and the data page of the persistent data object according to an encryption technique, and including an encoding page indicating the encryption technique in the persistent data object.

[0100] Clause 9. A method according to any one of clauses 5 to 8, wherein generating the persistent data object includes compressing index structure pages and data pages of the persistent data object according to one or more compression techniques, and including an encoding page indicating the one or more compression techniques in the persistent data object.

[0101] Clause 10. The method of any one of clauses 5 to 9, wherein the corresponding key of the item is an array of byte arrays.

[0102] Clause 11. The method of any one of clauses 5 to 10, wherein detecting an event that will seal the memtable comprises obtaining log records from a change log indicating which entries in the memtable are sealed.

[0103] Clause 12. The method of any one of clauses 5 to 11, further comprising sending a notification to the primary storage node that the persistent data object corresponding to the sealed memory table is stored as part of the backup of the data set.

[0104] Clause 13. The method of any one of clauses 5 to 12, wherein the data set is a partition of a database.

[0105] Clause 14. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to:

[0106] accessing, by one or more computing devices of the distributed data store, a plurality of updates to respective items in the data set stored in the change log, the respective items being ordered by respective time values assigned to the plurality of updates;

[0107] updating, by one or more computing devices, a memory table based on a plurality of updates having corresponding time values so that the memory table includes corresponding entries;

[0108] detecting, by one or more computing devices, an event that will seal the memtable from subsequent updates, wherein the subsequent updates are stored in a change log and used to update a new memtable;

[0109] generating, by one or more computing devices, a persistent data object that orders the respective items according to both their respective values and their respective time values; and

[0110] The persistent data objects are stored by one or more computing devices as part of a backup of the data set.

[0111] Clause 15. One or more non-transitory computer-readable storage media of Clause 14 storing further program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to further perform the following operations:

[0112] Detect compression events for backups of datasets;

[0113] identifying two or more persistent data objects to be merged in a backup of the data set; and

[0114] A merged persistent data object is generated that contains items from two or more persistent data objects, wherein the merged persistent data object is ordered according to keys of the contained items and corresponding time values of the items, wherein an item is contained in the two or more persistent data objects when the corresponding time value of the item is within a retention threshold.

[0115] Clause 16. One or more non-transitory computer-readable storage media according to clause 15, storing additional program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to further generate a current project version of the merged persistent data object, the current project version including the values of those items having the latest time values of the items.

[0116] Clause 17. One or more non-transitory computer-readable storage media according to any one of clauses 14 to 16, wherein, when generating a persistent data object, program instructions cause one or more computing devices to encrypt index structure pages and data pages of the persistent data object according to an encryption technique, and include an encoding page indicating the encryption technique in the persistent data object.

[0117] Clause 18. One or more non-transitory computer-readable storage media according to any one of clauses 14 to 17, wherein, when generating a persistent data object, program instructions cause one or more computing devices to compress index structure pages and data pages of the persistent data object according to one or more compression techniques, and include an encoding page indicating the one or more compression techniques in the persistent data object.

[0118] Clause 19. One or more non-transitory computer-readable storage media as recited in any one of Clauses 14 to 18, wherein the corresponding key of the item is an array of byte arrays.

[0119] Clause 20. The one or more non-transitory computer-readable storage media of any one of clauses 14 to 19, wherein the database system is a database service implemented as part of a provider network, wherein the data sets are partitions of database tables.

[0120] Various embodiments may further include receiving, sending, or storing instructions and / or data implemented according to the foregoing description on a computer-accessible medium. Generally speaking, a computer-accessible medium may include a storage medium or memory medium (e.g., a magnetic or optical medium, such as a disk or DVD / CD-ROM), a non-volatile medium (e.g., a RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), a ROM, etc.), and a transmission medium or signal (e.g., an electrical, electromagnetic, or digital signal transmitted via a communication medium (e.g., a network and / or a wireless link)).

[0121] The various methods shown in the figures and described herein represent example embodiments of the methods. The methods can be implemented in software, hardware, or a combination thereof. The order of the methods can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0122] Various modifications and variations may be apparent to those skilled in the art having the benefit of this disclosure. It is intended that the invention encompass all such modifications and variations, and, therefore, the foregoing description is to be regarded as illustrative rather than restrictive.

Claims

1. A system comprising: at least one processor; as well as a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a distributed data storage area, the distributed data storage area being configured to: accessing a plurality of updates to corresponding items in a data set stored in a change log, the corresponding items being ordered by corresponding time values assigned to the plurality of updates; updating the memory table to include the corresponding entries according to the plurality of updates having corresponding time values; detecting an event that will seal the memtable from subsequent updates, wherein the subsequent updates are stored in the change log and used to update a new memtable; generating a persistent data object that orders the respective items according to both their respective keys and their respective time values; as well as The persistent data object is stored as part of a backup of the data set.

2. The system according to claim 1, wherein the distributed data storage area is further configured to: detecting a compression event for the backup of the data set; identifying two or more persistent data objects to be merged in the backup of the data set; and Generating a merged persistent data object containing items from the two or more persistent data objects, wherein the merged persistent data object is ordered according to keys of the contained items and corresponding time values of the items, wherein an item is contained in the two or more persistent data objects when the corresponding time value of the item is within a retention threshold. 3 . The system of claim 2 , wherein the database system is further configured to generate a current item version of the merged persistent data object, the current item version containing the keys of those items having the latest time values of the items.

4. The system of any one of claims 1 to 3, wherein the distributed data store is a database service implemented as part of a provider network, wherein the data sets are partitions of a database table.

5. A method comprising: accessing, by one or more computing devices of the distributed data store, a plurality of updates to respective items in the data set stored in the change log, the respective items being ordered by respective time values assigned to the plurality of updates; updating, by the one or more computing devices, a memory table based on the plurality of updates having corresponding time values to include the corresponding entries; detecting, by the one or more computing devices, an event that will seal the memory table from subsequent updates, wherein the subsequent updates are stored in the change log and used to update a new memory table; generating, by the one or more computing devices, a persistent data object that orders the respective items according to both their respective keys and their respective time values; as well as The persistent data object is stored by the one or more computing devices as part of a backup of the data set.

6. The method according to claim 5, further comprising: detecting a compression event for the backup of the data set; identifying two or more persistent data objects to be merged in the backup of the data set; as well as Generating a merged persistent data object containing items from the two or more persistent data objects, wherein the merged persistent data object is ordered according to keys of the contained items and corresponding time values of the items, wherein an item is contained in the two or more persistent data objects when the corresponding time value of the item is within a retention threshold.

7. The method of claim 6, further comprising generating a current item version of the merged persistent data object, the current item version containing the values of those items having the most recent time values of the items.

8. The method according to any one of claims 5 to 7, wherein generating the persistent data object comprises encrypting the index structure page and the data page of the persistent data object according to an encryption technique, and including a coding page indicating the encryption technique in the persistent data object.

9. The method according to any one of claims 5 to 8, wherein generating the persistent data object comprises compressing index structure pages and data pages of the persistent data object according to one or more compression techniques, and including an encoding page indicating the one or more compression techniques in the persistent data object.

10. A method according to any one of claims 5 to 9, wherein the corresponding keys of the items are arrays of byte arrays.

11. The method of any one of claims 5 to 10, wherein detecting the event that will seal the memory table comprises obtaining log records from the change log indicating which entries in the memory table are sealed.

12. The method of any one of claims 5 to 11, further comprising sending a notification to a primary storage node that the persistent data object corresponding to the sealed memory table is stored as part of the backup of the data set.

13. The method according to any one of claims 5 to 12, wherein the data set is a partition of a database.

14. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to: accessing, by one or more computing devices of the distributed data store, a plurality of updates to respective items in the data set stored in the change log, the respective items being ordered by respective time values assigned to the plurality of updates; updating, by the one or more computing devices, a memory table based on the plurality of updates having corresponding time values to include the corresponding entries; detecting, by the one or more computing devices, an event that will seal the memory table from subsequent updates, wherein the subsequent updates are stored in the change log and used to update a new memory table; generating, by the one or more computing devices, a persistent data object that orders the respective items according to both their respective values and their respective time values; as well as The persistent data object is stored by the one or more computing devices as part of a backup of the data set.

15. The one or more non-transitory computer-readable storage media of claim 14, storing additional program instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further: detecting a compression event for the backup of the data set; identifying two or more persistent data objects to be merged in the backup of the data set; as well as Generating a merged persistent data object containing items from the two or more persistent data objects, wherein the merged persistent data object is ordered according to keys of the contained items and corresponding time values of the items, wherein an item is contained in the two or more persistent data objects when the corresponding time value of the item is within a retention threshold.

Images