In-memory key-value store for multi-model databases

Abstract

An in-memory key-value store for multi-model databases is disclosed. The multi-model database, maintained by a database server, includes an in-memory key-value store and a relational database. Key-value records can be generated in volatile memory accessible to the database server by converting data in database tables into a key-value format. The database tables can be stored in a persistent format. The key-value format can differ from and be independent of the persistent format. Database statements referencing the database table can be executed based on whether to access one or more key-value records in volatile memory or to access the data in the database table. In response to determining that one or more key-value records are to be accessed, the database server can access those one or more key-value records in volatile memory.

Description

[0001] This application is a divisional application of invention patent application 201680077564.2, filed on November 15, 2016, entitled "In-memory key-value storage for multi-model database". Technical Field

[0002] The embodiments relate to information retrieval techniques, and more specifically to in-memory key-value storage for multi-model databases. Background Technology

[0003] The methods described in this section are feasible, but not necessarily methods that have been previously conceived or implemented. Therefore, unless otherwise stated, no method described in this section should be assumed to qualify as prior art simply by virtue of its inclusion in this section.

[0004] Different data models often involve trade-offs between advantages in one area and disadvantages in another. For example, relational data models can exhibit internal complexity that supports efficient processing of complex queries at the cost of increased latency, even for simple queries. In contrast, key-value data models, for example, can provide low-latency, high-throughput query access for simple queries but may not be able to handle complex queries. Therefore, it is beneficial and desirable to maintain data across multiple data models to obtain the benefits of each.

[0005] One approach to maintaining data across multiple data models is to implement a dedicated database management system (DBMS) for each distinct data model. For example, there could be a separate DBMS that implements and supports only a key-value data model and another separate DBMS that implements and supports only a relational data model. However, the overhead involved in maintaining separate DBMSs negates the benefits of implementing them. For instance, maintaining transactional consistency between data in separate DBMSs can involve data replication implementations that are more complex than the implementation of the key-value database itself.

[0006] Therefore, a new approach is needed to maintain data across multiple data models, which allows the advantages of each data model to be leveraged while minimizing the disadvantages of each. Attached Figure Description

[0007] In the attached diagram:

[0008] Figure 1 An example computer architecture on which embodiments can be implemented is described.

[0009] Figure 2 A detailed view of the relational database in the example embodiment is depicted.

[0010] Figures 3A-3C An example key-value record is depicted.

[0011] Figure 4 It is a flowchart depicting the methods used to process database statements in a multi-model database.

[0012] Figure 5 A computer system on which embodiments can be implemented is described. Detailed Implementation

[0013] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it is clear that this disclosure can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring this disclosure. Modifiers such as “first” and “second” may be used to distinguish elements, but modifiers do not necessarily indicate any particular order. For example, a second database table may be named as such, although it may actually correspond to a first, second, and / or third database table.

[0014] General Overview

[0015] A computer can maintain data in multiple data models, one of which is a key-value data model for fast access. Data maintained in the key-value data model (hereinafter referred to as "KV data") can be stored in volatile memory, while data maintained in different data models can be persisted to non-volatile memory. In an embodiment, these different data models can be relational data models.

[0016] In a key-value data model, data is stored as a collection of key-value records. Each key-value record consists of key-value pairs. The keys in the collection are referred to as KV keys in this document, and the values ​​associated with the KV keys are referred to as KV values. Each KV key uniquely identifies a key-value record and is used to retrieve the corresponding KV value. KV values ​​can be data that is directly or indirectly referenced by a KV key. For example, a KV value can be a pointer to a specific piece of data.

[0017] Maintaining data across multiple data models can involve maintaining consistency between key-value (KV) data and data in different data models. In an embodiment, changes to data in one data model can be made concurrently to another data model, based on a common language that can be used with any of the data models. This common language can be a data definition language (DDL) and / or a data manipulation language (DML), such as Structured Query Language (SQL), which has been used with different data models and extended for use with KV data.

[0018] A common language enables database statements to be processed in parallel, allowing execution on key-value data and / or data in different data models. Therefore, it is more efficient for a computer to determine whether accessing key-value data or data in different data models to execute all or part of the database statement.

[0019] Overall system architecture

[0020] A single database management system can maintain data across multiple data models, including key-value data models. Figure 1 An example computer architecture on which embodiments can be implemented is described. (Reference) Figure 1 The database server 100 includes volatile memory 102 and persistent storage 106. The volatile memory 102 stores data in key-value format 104. The persistent storage 106 stores data in persistent format 108.

[0021] In this embodiment, the data stored in persistent format 108 may be data stored in a relational data model (hereinafter referred to as "REL data"). REL data may be stored as one or more database tables in a relational database in persistent storage device 106. A copy of the REL data may also be cached in volatile memory 102. Hereinafter, REL data and copies of REL data will be collectively referred to as REL data.

[0022] KV data may include a subset (e.g., none, some, all) of REL data stored in key-value format 104. Key-value format 104 may be different from and independent of persistent format 108. KV data can be generated based on REL data, as will be described in more detail below. For example, volatile memory 102 can be populated with KV data by performing transformations on REL data. Transformations between REL data and KV data can occur at any of several times, such as at startup, on demand, after a failure, and / or whenever data changes.

[0023] It is worth noting that the existence of key-value (KV) data can be transparent to database applications that submit database statements to database server 100. For example, a database application designed to interact with a database management system that only operates on REL data can interact with the database management system that maintains both REL and KV data without modification. Furthermore, being transparent to the database application, the database management system can use the KV data to process some or all of the database statements more efficiently.

[0024] Persistent storage device 106 generally refers to any number of persistent storage devices, such as disks, solid-state drives, flash memory, and / or any other non-volatile memory. Data stored on persistent storage device 106 is generally not lost in the event of a failure (e.g., power loss).

[0025] Volatile memory 102 generally refers to random access memory used by database server 100 and can be implemented by any number of memory devices. Typically, data stored in volatile memory 102 is lost in the event of a failure. Therefore, after a failure, data stored in persistent storage device 106 can be used to reconstruct the lost data in volatile memory 102.

[0026] Database server 100 may be one or more computers that manage a database management system. Database server 100 may manage one or more databases. For example, within volatile memory 102, database server 100 may execute database statements received from one or more database applications (e.g., clients). These database statements may reference one or more databases managed by database server 100.

[0027] Relational format data

[0028] In this embodiment, REL data may be data from one or more database tables in a relational database. Figure 2 A detailed view of the relational database in the example embodiment is depicted. References Figure 2 Relational database 200 includes database table 202. Database table 202 includes fields 204, 206, 208, and 210. Field 208 includes field values ​​212-218.

[0029] Fields 204, 206, 208, and 210 may include field values ​​that uniquely identify database objects, individually or in combination. For example, fields 204, 206, 208, and 210 may include unique indexes and / or alphanumeric strings for specific records (e.g., rows, columns) in database table 202. A subset of fields 204, 206, 208, and 210 may be included in the primary key used for database table 202. Figure 2 In the example, fields 204 and 206 could refer to the department name and user name, respectively, and either or both could be used to identify certain employees of the company. However, if field 204 is used alone to identify certain employees, a single field value (e.g., "Research") might identify more than one employee.

[0030] Fields 204, 206, 208, and 210 can store any number of different data types, such as strings, integers, and binary large objects (BLOBs). For example, field values ​​212-218 could be image or audio records stored in database table 202. As will be described in more detail below, key-value (KV) fields and key-value (KV) fields can each include corresponding subsets of fields 204, 206, 208, and 210 in relational database table 202.

[0031] Relational database 200 may include a database dictionary for managing relational database 200. The database dictionary may include database metadata, which defines database objects physically or logically contained within relational database 200. Database objects may include database tables, columns, indexes, data types, database users, user permissions, storage structures for storing database object data, and logical database objects (such as schemas, applications, and modules). The database dictionary can be modified according to issued DDL commands to add, modify, or delete database objects.

[0032] Key-value options

[0033] Key-value (KV) data can include any subset of REL data. For example, a KV key field can consist of field 206, and a KV value field can consist of field 210. In another example, the KV key field can include fields 204 and 206, and the KV value field can include fields 208 and 210.

[0034] In this embodiment, DDL statements can specify specific REL data that will be available as KV data. Hereinafter, the REL data available as KV data will be referred to as "KV-enabled". KV-enabled data can be specified at any granularity level. For example, KV-enabled data can be specified at at least the following granularities:

[0035] • The entire relational database 200

[0036] • Specified database table 202

[0037] ·Specified columns

[0038] · Specified line

[0039] For example, by issuing the DDL statement "ALTER TABLE 'database table 202' KEYVALUE" to database server 100, database table 202 can be specified as having the KV option enabled. Database metadata can indicate that certain REL data has the KV option enabled. Enabling the KV option for a specific subset of REL data provides database server 100 with the option to access either REL data or KV data in response to database statements.

[0040] In response to the DDL statement "ALTER TABLE 'database table 202' KEYVALUE", database server 100 can also disable the KV option for database table 202 that has the KV option enabled. Additionally or alternatively, database server 100 can manage volatile memory 102 according to an eviction policy. The eviction policy can be based on a predetermined period of time elapsed since the last access to specific KV data, the amount of available resources, and / or any other metric suitable for efficient memory management, to cause the removal of specific KV data. Therefore, KV options can be configured to adapt to changes in performance and resource usage requirements.

[0041] Key-value format data

[0042] Figure 3A A relatively simple example of a key-value record is depicted. Key-value record 300 can be organized into two main parts. One part corresponds to the KV key field 302, which includes the KV key that uniquely identifies key-value record 300. The other part corresponds to the KV value field 304, which includes, for example, the KV values ​​retrieved when a specific key-value record 300 is identified. (Reference) Figure 3A The key field 302 includes field 206 of database table 202, and the value field 304 includes field 210 of database table 202.

[0043] Faster lookup times for REL data can be achieved by transforming a subset of the REL data into KV data. Transforming REL data into KV data can include defining KV data based on REL data and / or propagating changes from REL data to KV data. For example, KV data can be defined based on DDL statements published to database server 100, and KV data can be modified based on DML statements published to database server 100. In response to DDL and / or DML statements, database server 100 can generate KV data based on the specified transformation.

[0044] In this embodiment, database server 100 can automatically define the key-value (KV) key field 302. Therefore, specifying data to be included in and / or excluded from the KV key field 302 may be unnecessary. For example, database server 100 can be configured to automatically copy the primary key of relational database table 202 to the KV key field 302. However, this default configuration can be overridden. Figure 3A In the example, the DDL statement "ALTER TABLE 'database table 202' KEYVALUEALTERNATE KEY('field 206')" may have been issued to define field 206 as KV key field 302.

[0045] Similarly, for KV value field 304, database server 100 can be configured to automatically copy and / or otherwise reference fields 204, 206, 208, and 210 that are not included in the primary key of the relational database table. However, specifying data to be included in and / or excluded from KV value field 304 may be desirable for efficient memory management and efficient data retrieval. Therefore, in Figure 3A In the example, the key-value field 304 can be generated based on the DDL statement "ALTER TABLE 'database table202' KEYVALUE NO KEYVALUE('field 208')".

[0046] Key-value records 300 can be stored as one or more tables (hereinafter referred to as "KV tables"). Relational database tables can have one-to-one and / or many-to-one correspondences with KV tables. For example, each relational database table 202 can be transformed into a separate KV table. Additionally or alternatively, multiple relational database tables can be transformed into a single KV table based on performing operations such as SQL JOIN.

[0047] Composite KV bond

[0048] Figure 3B Example key-value record 300 with a composite key-value key is described. The composite key-value key field includes field values ​​from two or more fields in the REL data. Reference Figure 3B The key field 302 includes fields 204 and 206 of database table 202. The value field 304 includes field 210 of database table 202.

[0049] Therefore, KV key field 302 is a composite KV key field that includes field values ​​from multiple fields from relational database table 202. Each component field of KV key field 302 includes field values ​​referred to herein as partial KV keys.

[0050] According to an embodiment, partial key-value (KV) keys can be used to access key-value (KV) data or Relative Elastic Record (REL) data. This is called partial key-value (KV) key access. Partial key-value (KV) key access enables the retrieval of multiple key-value (KV) values ​​from multiple records in response to a single database statement. This single database statement can be a query with predicates specifying one or more partial KV keys. For example, the SQL query "SELECT 'field 210' FROM 'database table 202' WHERE 'field 204' = 'Research'" specifies an exact match for the partial KV key "Research". Figure 3B In the example, the key-value pair “Research” is found in both the second and fourth key-value records 300, which contain the key-value values ​​“Doctorate” and “Masters”. Therefore, a result set including both “Doctorate” and “Masters” is returned in response to the query.

[0051] exist Figure 3B In the example, KV key field 302 is depicted as encompassing fields 204 and 206 as separate partial KV key fields. However, in embodiments, KV key field 302 can include an aggregation of multiple fields into a single field. For example, KV key field 302 can include the composite KV key “Sales, Alice Smith”, which is an aggregation of two partial KV keys from two different fields. The partial KV key “Sales” corresponds to component field 204, and the partial KV key “Alice Smith” corresponds to component field 206. Any one or both of the partial KV keys in the aggregation can be referenced to access certain key-value records 300.

[0052] Composite KV value

[0053] Figure 3C Example key-value record 300 with a composite KV value is described. A composite KV value field includes field values ​​from two or more fields in the REL data. Reference Figure 3C Key field 302 includes fields 204 and 206 from database table 202. Key value field 304 is a composite of fields 208 and 210.

[0054] Therefore, KV value field 304 is a composite KV value field that includes field values ​​from multiple fields from relational database table 202. Each component field of KV value field 304 includes a field value referred to herein as a partial KV value.

[0055] Partial key-value (KV) values ​​can be used to store rich datasets in KV value field 304. Some or all of the rich dataset can be retrieved in response to a database statement. This database statement can be a query that specifies at least a portion of the composite KV values ​​in the query predicate. (See reference) Figure 3C For example, in key-value record 300, the SQL query "SELECT 'field 210' FROM 'database table 202' WHERE 'field 206' = 'Alice Smith'" will specify the component field 210 of key-value field 304. Therefore, in response to this SQL query, the partial key-value value "Bachelors" can be retrieved. This is a partial key-value value retrieval.

[0056] In this embodiment, the KV value field 304 may encompass multiple fields maintained as separate, partial KV value fields. However, in Figure 3C In the example, KV value field 304 is depicted as an aggregation of multiple fields into a single field. For example, "{field 208: reference field value 212, field 210: 'Bachelors'}" is a composite of two partial KV values ​​from two component fields.

[0057] Indexing key-value format data

[0058] Partial key-value (KV) access and partial KV value retrieval enable efficient memory management. Multiple KV tables can be consolidated into a single KV table, which can handle database statements used for each of the multiple KV tables. For example, any database statement that can be used for... Figure 3A Database statements executed on KV tables can also be targeted at... Figure 3B The key-value table in the code is executed. However, to... Figure 3A Adding partial KV key fields to a KV table in a database involves less memory space than maintaining a separate KV table containing duplicate data.

[0059] Partial key-value access and partial key-value value retrieval can be supported by indexing partial key-value keys and / or partial key-value values. Database server 100 can be configured to automatically index each key-value record 300 (e.g., based on automatically indexing the key-value field 302). However, indexes on the key-value table do not necessarily correspond to indexes on the relational database table.

[0060] For example, to specify that the same indexes defined on REL data should be added to KV data, you can issue the DDL statement "ALTER TABLE 'database table 202' KEYVALUE KEEP INDEXES" to database server 100. Alternatively, to specify that certain indexes defined on REL data should be added to KV data, you can issue the DDL statement "ALTER TABLE 'database table 202' KEYVALUE KEEP INDEXES('index1','index2')" to database server 100. Alternatively, to specify that indexes defined on REL data should not be added to KV data, you can issue the DDL statement "ALTER TABLE 'database table 202' KEYVALUE NO INDEXES" to database server 100.

[0061] Additionally or alternatively, users can specify specific portions of the key-value data to be indexed. For example, a DDL statement can be issued to database server 100: "ALTER TABLE 'database table 202' KEYVALUE ADD INDEX('field210')". In response to this DDL statement, database server 100 generates an index that indexes the value of field 210.

[0062] Aggregated KV data

[0063] Composite key-value pairs and / or composite key-value pairs can be stored as serialized field values. Serialization can be in any format, such as JavaScript Object Notation (JSON), Extensible Markup Language (XML), and Comma Separated Values ​​(CSV). This format can be specified by DDL and / or DML statements.

[0064] For example, the key-value field 304 can be generated by issuing the DDL statement "ALTER TABLE 'database table202' KEYVALUE value AS TRANSFORM JSON('field 208','field 210')" to database server 100. Therefore, the field values ​​from fields 208 and 210 can be stored as JSON objects. Figure 3C This section describes some examples of JSON objects generated by this example DDL statement.

[0065] In this embodiment, key-value record 300 can be generated based on the serialization of key-value keys and values. For example, a DDL statement "ALTER TABLE 'database table 202' KEYVALUE value ASTRANSFORM JSON('field 204','field 206','field 208','field 210')" can be issued to database server 100. All or part of the serialized key-value data can be indexed.

[0066] Aggregating key-value data can also involve other transformations, such as data type transformations. Data type transformations change a field value from one data type (e.g., integer) to another data type (e.g., string). Data type transformations can be used in conjunction with serialization. For example, an integer can be converted to a string to enable string concatenation with another field value.

[0067] Additionally or alternatively, aggregating key-value (KV) data may involve generating dummy fields. Dummy fields can include KV data that does not exactly match the REL data. For example, while the REL data may include "Sales" and "Alice Smith" as separate field values, it does not include the overall field value "Sales_Alice Smith," which can be generated to be included in the KV data. Dummy fields can be generated based on performing one or more operations on the REL data. For example, a dummy field containing "Sales_Alice Smith" can be generated by concatenating the strings 204 and 206 containing "Sales" and "Alice Smith" and using an underscore character as a delimiter.

[0068] Additionally or alternatively, aggregating key-value (KV) data may involve executing database statements to transform and / or generate the KV data. For example, database statements may be executed to retrieve REL data to be aggregated into the KV data. This can be particularly useful if the retrieved REL data is stored in a separate database table 202 from the REL data that will be aggregated together with it. In this case, an SQL JOIN operation may be involved.

[0069] Structure of KV key field and KV value field

[0070] As mentioned earlier, a key characteristic of KV data is that it can be accessed or modified using the same language used with REL data. For example, the SQL query "SELECT 'field210' FROM 'database table 202' WHERE 'field 206' = 'Alice Smith'" can be executed against either relational database 200 or key-value record 300. Because this query avoids excluding REL data or KV data from consideration, database server 100 has the option to access both REL data and / or KV data to execute the query.

[0071] Therefore, different labels for KV and REL data are unnecessary for accessing the data. For example, labeling the KV key field 302 as "key" and the KV value field 304 as "label" would allow queries to reference these labels, but doing so would deprive the database server 100 of the option to execute that query on the REL data (if it would be more efficient). However, such labels can be useful for transforming REL data into KV data. For example, when aggregating REL data into the KV value field 304, DDL statements can reference the label "value". In other words, these labels can be maintained by the database server 100 as internal references for KV data.

[0072] Key-value (KV) data can be stored using any of a variety of representations. KV keys can be stored in at least a B-tree, a hash table, or any other representation that enables fast lookup. Each KV key can reference a representation of a KV value; the KV value itself does not need to be stored in any specific type of representation.

[0073] Processing database statements

[0074] When database server 100 receives a database statement that references REL data, database server 100 analyzes the database statement and determines the most efficient plan for executing the database statement on the REL data. The process of analyzing the database statement and determining the execution plan is referred to herein as "statement optimization." According to an embodiment, in response to a particular database statement, statement optimization determines whether accessing REL data or KV data would be more efficient.

[0075] Efficiency determination can depend on various factors, including the availability of key-value (KV) data. For example, some REL data may be available as KV data, while others may be available only as REL data. For REL data available as KV data, accessing the KV data will be faster than accessing the REL data in persistent storage 106. For REL data that cannot be used as KV data, accessing the REL data even in persistent storage 106 will be faster than first converting the REL data into KV data and then accessing that KV data.

[0076] Even if key-value (KV) data is available for a specific database statement, efficiency determination can become more complex due to other considerations. Generally, executing database statements against KV data rather than REL data is faster when interpreting (e.g., parsing, analyzing) the contents of aggregated KV data is unnecessary. For example, if a query returns the entire JSON object, then accessing KV data will be faster than accessing REL data. However, if a query returns specific name-value pairs of a JSON object, whether accessing KV data is faster than accessing REL data can depend on any of a variety of factors.

[0077] Figure 4 This is a flowchart depicting the method of processing database statements at a multi-model database. At block 400, database server 100 receives a database statement that references REL data. This database statement can be executed using SQL or an application programming interface (API) that enables bypassing SQL (hereinafter referred to as the "Direct Access API"). For example, if the key-value (KV) fields are known, it is possible to directly access the KV data in volatile memory 102 to return the KV value corresponding to the KV key received by the Direct Access API.

[0078] Determine the data for which the KV option is enabled.

[0079] At box 402, database server 100 determines whether it is possible to execute a database statement against KV data. For example, statement optimization may analyze the database statement to determine whether it involves data with KV options enabled. In other words, statement optimization may determine whether the database statement references at least a portion of KV key field 302 and / or at least a portion of KV value field 304. This determination may involve parsing the database statement to obtain the names of fields 204, 206, 208, and 210. This determination may also involve, for example, a directory lookup of these names in a database dictionary.

[0080] If database server 100 cannot execute database statements on KV data, then box 402 proceeds to box 406. Otherwise, box 402 can proceed to optional box 404 for further analysis.

[0081] Further efficiency analysis

[0082] At box 404, database server 100 determines, for each part of the database statement, whether accessing key-value (KV) data is more efficient than accessing recursive (REL) data. This determination can be based on any of several different factors, including the following:

[0083] Accessing key-value (KV) data involves interpreting field values.

[0084] KV data includes known and supported data types and data representations.

[0085] KV data includes indirect references to the corresponding REL data.

[0086] • Location of REL data and / or location of KV data.

[0087] In an embodiment, if a database statement references any partial key-value keys and / or partial key-value values ​​maintained as aggregate data (e.g., serialized data), the database statement may undergo further optimization analysis to determine whether key-value data or REL data should be accessed to execute the database statement. However, if interpreting aggregate data is unnecessary, accessing key-value data may be more efficient, and block 404 may proceed to block 408.

[0088] In an embodiment, if any aggregated data in the aggregated data includes unknown or unsupported data types (e.g., strings, integers) and / or data representations (e.g., JSON, XML, CSV), then box 404 may proceed to box 406. Otherwise, box 404 may proceed to box 408. For example, if an unsupported serialization format is used for the KV value field 304, parsing the composite KV value and locating the relevant partial KV value may be slower than accessing the REL data.

[0089] In an embodiment, if the KV data includes indirect references to REL data (e.g., pointers), then block 404 may proceed to block 406. Otherwise, block 404 may proceed to block 408. For example, if executing database statements against the KV data would involve resolving the memory address corresponding to the REL data, then directly accessing the REL data might be more efficient. Figure 3CIn this context, the KV value field 304 includes KV values ​​containing references to REL data. When such a KV value is retrieved, a second access can occur to resolve that reference into REL data. Depending on when the second access occurs, accessing the REL data may be more efficient. For example, accessing the KV data may be more efficient if the second access occurs automatically when the KV value is constructed in volatile memory 102. However, accessing the REL data may be more efficient if the second access occurs automatically when the KV value is accessed. Efficiency determination can be further complicated by recursive references.

[0090] In an embodiment, the location of the REL data and / or the location of the KV data can be used to determine whether to access KV data or REL data. For example, if the relevant REL data resides in a cache, accessing the REL data can be more efficient than accessing the KV data. As another example, if the relevant data is stored locally in one format and remotely in another format (e.g., stored in a distributed database), accessing the relevant data in the locally stored format can be more efficient.

[0091] At box 406, database server 100 accesses REL data to execute database statements, and at box 408, database server 100 accesses KV data to execute database statements. If any part of the database statement still needs to be parsed, then boxes 406 and / or 408 can return to box 404.

[0092] The complexity of efficiency analysis

[0093] In this embodiment, the efficiency analysis can be comprised of block 402, thus avoiding the computationally more expensive and time-consuming analysis at block 404. Therefore, at block 402, the database server 100 can simply determine whether all REL data referenced in the database statement has the KV option enabled. If so, then block 402 proceeds directly to block 408. This embodiment improves the efficiency of processing dynamically generated (e.g., real-time) database statements.

[0094] In this embodiment, in addition to block 402, a thorough efficiency analysis may also include block 404. While this embodiment ensures optimal efficiency in executing database statements, there is overhead associated with such thorough analysis. Therefore, this embodiment is ideally suited for prepared statements (e.g., pre-compiled database statements).

[0095] compression

[0096] In this embodiment, the key-value (KV) data may be stored in volatile memory 102 in a compressed format. However, different portions of the KV data may be compressed in different ways and / or to different degrees. For example, frequently accessed KV data may be uncompressed or lightly compressed, while rarely accessed KV data may be highly compressed.

[0097] Based on various factors, the database server 100 can automatically determine one or more compression algorithms (e.g., dictionary-based compression, run-length encoding, zip compression) and / or the corresponding compression level used by each compression algorithm. These factors may include access frequency, data size, data priority, and / or available memory. For example, limited available memory and large data size can result in a high compression level.

[0098] Additionally or alternatively, users may specify one or more compression algorithms and / or the corresponding compression level used by each compression algorithm. The DDL can be extended to support user-specified compression hints. For example, the key-value data for database table 202 can be compressed for all queries based on issuing the database statement "ALTER TABLE 'database table 202' KEYVALUE MEMCOMPRESSFOR QUERY" to database server 100. Additional options may include "FOR DML" (i.e., for all DML database statements), "FOR QUERY LOW / HIGH" (i.e., for low / high query access frequency), and / or "FOR CAPACITY LOW / HIGH" (i.e., for low / high available storage). Compression can also be disabled based on issuing the database statement "ALTER TABLE 'database table 202' KEYVALUE NO MEMCOMPRESS" to database server 100.

[0099] Additionally or alternatively, the user may provide runtime hints specifying how quickly certain key-value (KV) data can be loaded into volatile memory 102. For example, a database statement "ALTER TABLE 'database table202' KEYVALUE PRIORITY LOW" may be issued to database server 100. Additional options may include "NONE", "MEDIUM", "HIGH", and / or "CRITICAL". However, database server 100 may ultimately decide when to load certain KV data and how quickly to do so. In embodiments, the user may specify when to load and / or unload KV data. For example, "PRIORITY NONE" may instruct database server 100 to wait for user instructions rather than automatically loading the KV data. Thus, a database application may invoke database server 100 and specify loading and / or unloading KV data.

[0100] Additionally or alternatively, the memory space used for KV data can be allocated based on system parameters specified by the user. For example, based on issuing the database statement "ALTER SYSTEM SET keyvalue_size 125GB" to database server 100, 125GB of volatile memory 102 can be reserved for KV data.

[0101] Compressed key-value (KV) data can be organized into compressed units within volatile memory 102. Each compressed unit can store a different set of KV data. For example, KV data can be organized into compressed units based on different tables, different table partitions, different rows, different columns, etc. The mapping of KV data to compressed units can be stored as metadata in volatile memory 102, indicating which KV data is contained in each compressed unit. In a distributed database system, the mapping of KV data to compressed units can also indicate which database server stores a specific compressed unit. Because accessing local data is generally more efficient than retrieving data from a remote location, the location of the KV data can influence the determination of whether to access KV data or REL data.

[0102] In this embodiment, decompressing the KV data before accessing it may be unnecessary. For example, vector processing operations can be performed directly on the compressed KV data. In another example, the compressed KV data can be decompressed on-chip after it has been transmitted to the CPU.

[0103] Maintaining transaction consistency

[0104] Transactional consistency between REL data and KV data can be maintained based on any of several consistency models. For example, transactional consistency can be maintained between different data formats using any of the techniques described in U.S. Patent Application No. 14 / 337,164 (filed July 21, 2014); U.S. Patent Application No. 14 / 337,142 (filed July 21, 2014); U.S. Patent Application No. 14 / 337,045 (filed July 21, 2014); U.S. Patent Application No. 14 / 337,182 (filed July 21, 2014); U.S. Patent Application No. 14 / 337,179 (filed July 21, 2014); and U.S. Patent Application No. 14 / 819,016 (filed August 5, 2015), the entire contents of each of which are incorporated herein by reference. In embodiments, a synchronous consistency model can be implemented. In embodiments, an asynchronous consistency model can be implemented based on maintaining multiple versions of the data.

[0105] Change propagation can involve transforming REL data into KV data. Change propagation can be available immediately or on demand. For example, data changes can be propagated immediately when they occur, or they can be propagated in batches at predetermined times (e.g., when a predetermined amount of change occurs, periodically, or in preparation for anticipated needs).

[0106] In this embodiment, the database management system's transaction manager (e.g., an online transaction processing server) can be configured to concurrently update REL data and KV data. For example, if a KV value contains a reference to REL data and a dereference occurs when the KV value is constructed in volatile memory 102, changes to the referenced REL data can be tracked to ensure consistency between the REL data and KV data. As mentioned above, DDL and / or DML associated with REL data can be extended to be used with KV data. Therefore, the same database statements can concurrently update both REL data and KV data. For example, updates to REL data can be propagated to KV data based on transforming the update into KV data.

[0107] While updates can be immediately merged into the decompressed and / or uncompressed data, it may be necessary to delay merging updates into the compressed data. However, consistency can be maintained by recording updates in a change log and bitmap stored in volatile memory 102. Recording updates allows for merging updates when it is convenient to decompress the data, thereby avoiding the overhead of decompressing and recompressing the data every time a change occurs.

[0108] A change log can store information about updates not yet reflected in the compressed data within volatile memory 102. The change log can include changed data, change timestamps, and / or transaction identifiers. The change log can be a global change log or a private change log. A global change log can store committed transactions and is accessible to all processes. A private change log can store uncommitted transactions and may only be accessible to specific transactions. When a transaction is committed, the information stored in the private change log for that transaction can be moved to one or more global change logs, where the transaction's commit timestamp can also be stored. Information in the global change log can be merged into the compressed data (e.g., KV data) at any time (such as when it is convenient to decompress the compressed data).

[0109] The bitmap can maintain version control information within volatile memory 102. Instead of merging changes as they occur, the bits corresponding to a data item are flipped in the bitmap whenever a data item changes. When the changes are eventually merged (e.g., merged into KV data), the bitmap can be reset, and a version identifier (e.g., a timestamp for the change merge) can be associated with the bitmap. The bitmap can be dynamically allocated in such a way that wasted storage space is minimized by storing unflipped bits. The bitmap can be organized based on the hierarchy of data units (e.g., rows, columns, blocks, sections, and / or segments) in a way that allows data items to be searched efficiently (e.g., binary search).

[0110] Additionally or alternatively, the cycle of decompressing, modifying, and recompressing KV data can be avoided by transforming the corresponding REL data into KV data that has already been updated and compressed. For example, the corresponding REL data could be uncompressed cached data. This cached data can then be converted into KV data corresponding to the updated REL data.

[0111] Compatibility with other database features

[0112] At least the techniques mentioned above related to maintaining key-value data in a multi-model database should be compatible with any other features of the multi-model database. For example, DDL statements can be issued to database server 100 that store key-value data in columnar format in volatile memory 102. As another example, SQL extensions (e.g., JSON extensions, XML extensions) should be available for both REL data and key-value data. In other words, key-value data should be seamlessly integrated with all the functionalities of the database management system.

[0113] Hardware Overview

[0114] According to one embodiment, the technology described herein is implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute the technology, or may include one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are permanently programmed to execute the technology, or may include one or more general-purpose hardware processors programmed to execute the technology according to program instructions in firmware, memory, other storage devices, or combinations thereof. These dedicated computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement the technology. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement the technology.

[0115] For example, Figure 5 This is a block diagram illustrating a computer system 500 on which embodiments may be implemented. The computer system 500 includes a bus 502 or other communication mechanism for transmitting information, and a hardware processor 504 coupled to the bus 502 for processing information. The hardware processor 504 may be, for example, a general-purpose microprocessor.

[0116] Computer system 500 also includes main memory 506, such as random access memory (RAM) or other dynamic storage devices, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 504. When these instructions are stored in non-transitory storage media accessible to processor 504, they make computer system 500 a dedicated machine customized to perform the operations specified in the instructions.

[0117] Computer system 500 also includes a read-only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. Storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0118] Computer system 500 can be coupled to display 512, such as a cathode ray tube (CRT), via bus 502 for displaying information to the computer user. Input device 514 (which includes alphanumeric and other keys) is coupled to bus 502 for transmitting information and command selections to processor 504. Another type of user input device is cursor control 516, such as a mouse, trackball, or arrow keys, for transmitting directional information and command selections to processor 504 and for controlling cursor movement on display 512. Such input devices typically have two degrees of freedom in two axes (a first axis (e.g., x) and a second axis (e.g., y)), allowing the device to specify a position in a plane.

[0119] Computer system 500 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic. This custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic, combined with the computer system, enable or program the computer system 500 to be a special-purpose machine. According to one embodiment, the techniques described herein are executed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. These instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequence of instructions contained in main memory 506 causes processor 504 to perform the processing steps described herein. In alternative embodiments, hard-wired circuitry may be used instead of or in combination with software instructions.

[0120] As used herein, the term "storage medium" refers to any non-transient medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cassettes.

[0121] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 502. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0122] Various forms of media can involve carrying one or more sequences of instructions to processor 504 for execution. For example, instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them via a telephone line using a modem. A modem local to computer system 500 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 502. Bus 502 carries the data to main memory 506, from which processor 504 retrieves and executes the instructions. Instructions received from main memory 506 may optionally be stored on storage device 510 before or after execution by processor 504.

[0123] Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides bidirectional data communication coupled to network link 520, which is connected to local network 522. For example, communication interface 518 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 518 may be a LAN card providing data communication connectivity to a compatible local area network (LAN). A wireless link may also be implemented. In any such implementation, communication interface 518 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0124] Network link 520 typically provides data communication to other data devices via one or more networks. For example, network link 520 may provide a connection to host computer 524 or to data devices operated by Internet Service Provider (ISP) 526 via local network 522. ISP 526, in turn, provides data communication services through a worldwide packet data communication network now commonly referred to as the "Internet" 528. Both local network 522 and Internet 528 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks, as well as signals on network link 520 and through communication interface 518, are example forms of transmission media in which information carries digital data to or from computer system 500.

[0125] Computer system 500 can send messages and receive data, including program code, through one or more networks, network links 520, and communication interfaces 518. In the Internet example, server 530 can transmit request codes to the application through the Internet 528, ISP 526, local network 522, and communication interface 518.

[0126] The received code may be executed by processor 504 when it is received, and / or stored in storage device 510 or other non-volatile memory for later execution.

[0127] In the foregoing description, embodiments have been described with reference to numerous specific details that may vary from one implementation to another. Therefore, the description and drawings are to be considered illustrative rather than restrictive. The unique and exclusive indication of the scope of this disclosure, and the content that the applicant contemplates as the scope of this disclosure, is the written and equivalent scope of the set of claims arising from this application, in the specific form in which such claims arise, including any subsequent amendments.

Claims

1. A computer-implemented method, comprising: Maintain a relational database managed by a database server on persistent storage. The relational database includes database tables, which are stored in a persistent format on the persistent storage device. By converting the data in the database table into a key-value format, key-value records are generated in volatile memory accessible to the database server. The key-value format is different from and independent of the persistent format; Execute a database statement that includes a FROM clause referencing the database table, wherein executing the database statement includes: Based on determining whether accessing one or more key-value records in the volatile memory involves interpreting one or more field values ​​of aggregated key-value data, it is determined whether to access the one or more key-value records or the data in the database table. When it is determined that one or more key-value records in the volatile memory need to be accessed, the one or more key-value records are accessed.

2. The method according to claim 1, wherein, Generating key-value records by converting data in the database tables into key-value format and storing them in volatile memory accessible by the database server includes: Perform a join operation on the database table and another database table in the relational database; and The one or more key-value records include data from the database table and data from the other database table.

3. The method according to claim 1, wherein, Generating key-value records by converting data in the database tables into key-value format and storing them in volatile memory accessible by the database server includes: When the database server performs a database operation that modifies the data in the database table, the database server maintains the key-value record in the volatile memory to ensure that it is consistent with the data in the database table in terms of transactions.

4. The method according to claim 1, wherein, Generating key-value records by converting data in the database tables into key-value format and storing them in volatile memory accessible by the database server includes: Generate the key field of the key-value record, wherein the key field includes multiple fields of the database table.

5. The method according to claim 1, wherein, Generating key-value records by converting data in the database tables into key-value format and storing them in volatile memory accessible by the database server includes: Generate the value field of the key-value record, wherein the value field includes multiple fields of the database table.

6. The method according to claim 5, wherein, The value fields that generate the key-value record include: For each key-value record in the key-value record, the corresponding field value among the plurality of fields is stored in the value field as a serialization of the field value.

7. The method according to claim 5, wherein, The value fields that generate the key-value record include: Index at least two of the multiple fields.

8. The method according to claim 1, wherein, Determining whether to access one or more key-value records in the volatile memory or to access the data in the database table includes: Determine whether the one or more key-value records include indirect references to corresponding data in the database table.

9. The method according to claim 1, wherein, Generating key-value records by converting data in the database tables into key-value format and storing them in volatile memory accessible by the database server includes: Stores references to one or more field values ​​of the database table.

10. An apparatus comprising: processor; and A memory coupled to the processor and having instructions stored on the memory, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-9.

11. A computer program product comprising instructions that, when executed by one or more processors of a computer, cause the computer to perform the method according to any one of claims 1-9.

Images