Techniques for enabling and integrating in-memory semi-structured data and text document search with in-memory columnar query processing

Abstract

A "hybrid-derived cache" stores semi-structured data or unstructured text data in an in-memory mirror form and stores columns in another form such as a column-major format. The hybrid-derived cache can cache scalar type columns in a column-major format. The structure of the in-memory mirror form of the semi-structured data or unstructured text data enables access and / or enhanced access to perform path-based and / or text-based query operations. The hybrid-derived cache improves cache containment for performing query operations. The in-memory mirror form is used to compute queries in a transaction-consistent manner by using an invalidation vector that is used to determine when to retrieve the transaction-consistent persistent form of the semi-structured data or unstructured text data instead of the in-memory form.

Description

Technical Field

[0001] The present invention relates to database systems, and more particularly, to in-memory caching of semi-structured data stored in columns of a database system. Background Art

[0002] One way to significantly improve the computation of queries in an object-relational database system is to preload and retain database tables in a derived cache. In the derived cache, an in-memory version of at least a portion of the database tables stored in a persistent form can be mirrored in column-major format to lower latency random access memory (RAM) of a database server. In column-major format, a representation of a portion of the column values of a column is stored in a column vector that occupies a contiguous address range in the RAM.

[0003] For several reasons, query operations involving a column can be performed more quickly on the in-memory column vector of the column, such as predicate evaluation and aggregation on the column. First, the column vector is maintained in lower latency memory where it can be accessed more quickly. Second, the run of column values on which the query operation operates is stored continuously in the column vector in memory. In addition, the column vector is compressed to reduce the memory required to store the column. Dictionary compression is often used to compress the column vector.

[0004] Dictionary compression can also be exploited by compressed columnar algorithms that are optimized for performing query operations on compressed column vectors to further increase the speed of performing such query operations on columns. Other forms of compression can also be exploited by compressed columnar algorithms.

[0005] For example, an example of a derived cache is described in U.S. Application No. 14 / 337,179, "Mirroring, In Memory, Data From Disk To Improve Query Performance" ("Mirroring Application") filed on July 21, 2014 by Jesse Kamp et al. and issued as U.S. Patent No. 9,292,564 on March 22, 2016, the entire content of which is incorporated herein by reference.

[0006] For columns containing scalar values such as integers and strings, the benefits of compressed columnar algorithms are realized. However, columns containing semi-structured data such as XML (Extensible Markup Language) and JSON (JavaScript Object Notation) may not be stored in column-major form in a manner that can be exploited by compressed columnar algorithms.

[0007] Semi-structured data is typically stored in a large binary object (LOB) column in a persistent form. Inside the LOB column, the semi-structured data can be stored in various semi-structured data formats, including as the body of tagged text, or in a proprietary format structured for compressibility and fast access. Unfortunately, compression columnar algorithms that work well for scalar columns are ineffective for these semi-structured data formats.

[0008] In addition, there are query operations specific to semi-structured data, such as path-based query operations. Path-based operations are not suitable for being optimized for semi-structured data stored in column vectors.

[0009] Database management systems (DBMSs) that support semi-structured data typically also store database data in scalar columns. Queries processed by such DBMSs can reference both semi-structured and scalar database data. Such queries are referred to herein as hybrid format queries. Although hybrid format queries require access to semi-structured data, hybrid format queries still benefit from a derived column cache because at least part of the work of executing the query can use the derived column cache for scalar columns. For the part of the work that requires access to semi-structured data, these DBMSs use traditional query operations that operate on persistent form data (PF data).

[0010] The ability to store and efficiently access semi-structured data is becoming increasingly important. Described herein are techniques for maintaining semi-structured data in a derived cache to improve the speed of executing queries, including hybrid format queries.

[0011] The methods described in this section are methods that can be adopted, but not necessarily methods that have been previously envisioned or adopted. Therefore, unless otherwise indicated, no method described in this section should be assumed to be prior art solely because it is included in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the drawings:

[0013] Figure 1 is a diagram of a database system according to an embodiment, the database system concurrently maintaining mirror format data in volatile memory and persistent format data on a persistent storage device.

[0014] Figure 2A is a diagram of a table for illustration according to an embodiment.

[0015] Figure 2B is a diagram of the persistent form of stored data for a table according to an embodiment.

[0016] Figure 3 is a diagram showing a hybrid derived cache according to an embodiment.

[0017] Figure 4A is a node tree representing an XML fragment according to an embodiment.

[0018] Figure 4B is a node tree representing a JSON object according to an embodiment.

[0019] Figure 5A is a diagram depicting a JSON object stored in columns in a persistent form according to an embodiment.

[0020] Figure 5B is a diagram illustrating a logo and logo positions according to an embodiment.

[0021] Figure 6A is a publication index that maps a logo to a logo position in a hierarchical data object according to an embodiment.

[0022] Figure 6B depicts a publication index in the form of a serialized hash table according to an embodiment.

[0023] Figure 6C depicts an expanded view of a publication index in the form of a serialized hash table according to an embodiment.

[0024] Figure 7 depicts a process executed when evaluating a predicate against a publication index according to an embodiment.

[0025] Figure 8 depicts an object-level publication list according to an embodiment.

[0026] Figure 9A depicts a process executed when incrementally updating a publication index in the form of a serialized hash table according to an embodiment.

[0027] Figure 9B depicts an incremental publication index according to an embodiment.

[0028] Figure 10 is a diagram of a computer system that can be used to implement the techniques described herein according to an embodiment.

[0029] Figure 11 is a diagram of a software system that can be used to control the operation of a computing system according to an embodiment. DETAILED DESCRIPTION

[0030] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

[0031] General Overview

[0032] This document describes techniques for maintaining semi-structured data in a persistent storage device in a persistent form and in a derived cache in another form called in-memory semi-structured data store (SSDM form). The semi-structured data stored in the persistent format is referred to herein as PF semi-structured data. The semi-structured data stored in SSDM form is referred to herein as SSDM data.

[0033] According to an embodiment, a "hybrid derived cache" stores semi-structured data in SSDM form and stores columns in another form such as column-major format. The data stored in the derived cache as referred to herein has mirror form data (MF data). The hybrid derived cache can cache scalar type columns in column-major format and cache semi-structured data in columns in SSDM form. The structure of the SSDM form enables access and / or enhanced access to perform path-based and / or text-based query operations.

[0034] The hybrid derived cache improves cache containment for performing query operations. Cache containment refers to restricting access to the database data required to perform query operations to access within the hybrid derived cache. Generally, the higher the cache containment level for executing a query, the greater the benefit of the hybrid derived cache, which results in faster execution of queries (including hybrid format queries).

[0035] Hybrid format queries can also include queries that access unstructured text data. The unstructured text data can be stored, for example, in a LOB. The techniques described herein can be used to store unstructured text data in a persistent storage device in a persistent format and in a derived cache in an in-memory form that may be less complex in various aspects than the SSDM form.

[0036] General Architecture

[0037] According to an embodiment, a derived cache is implemented within a DBMS using an in-memory database architecture that keeps the PF data and the MF data transactionally consistent. Such an in-memory database architecture is described in detail in the Mirroring application. The Mirroring application describes, among other things, maintaining multiple copies of the same data item, where one copy is maintained in a persistent form and another copy is maintained in a mirror form in volatile memory.

[0038] Figure 1 is a block diagram of a database system according to an embodiment. Refer to Figure 1, the DBMS 100 includes a RAM 102 and a persistent storage device 110. The RAM 102 generally represents the RAM used by the DBMS and can be implemented by any number of memory devices, including volatile and non-volatile memory devices and combinations thereof.

[0039] The persistent storage device 110 generally represents any number of persistent block-mode storage devices, such as disks, flash memories, solid-state drives, or non-volatile RAM that can be accessed through a block-mode interface to read or write data blocks stored thereon.

[0040] Within the DBMS 100, a database server 120 executes database statements submitted to the database server by one or more database applications (not shown). The data used by those applications is shown as PF data 112. The PF data 112 resides in the persistent storage device 110 in a PF data structure 108. The PF data structure 108 can be, for example, a row-major data block. Although row-major data blocks are used for illustrative purposes, the PF structure can take any form, such as column-major data blocks, hybrid compression units, etc.

[0041] The RAM 102 also includes a buffer cache 106 for the PF data. Within the buffer cache 106, the data is stored in a format based on the format in which the data resides within the PF data structure 108. For example, if the persistent format is a row-major data block, then the buffer cache 106 can contain a cached copy of the row-major data block.

[0042] On the other hand, the MF data 104 has a format different from the persistent format. In an embodiment where the persistent format is a row-major data block, the mirror format can be column-major for scalar columns and SSDM form for columns holding semi-structured data. Since the mirror format is different from the persistent format, the MF data 104 is generated by performing transformations on the PF data. These transformations occur both when initially filling the RAM 102 with the MF data 104 (whether at startup or on demand) and when refilling the RAM 102 with the MF data 104 after a failure.

[0043] Importantly, the presence of the MF data 104 can be transparent to database applications that submit database commands to the database server. Applications that have utilized a DBMS that operates only on PF data can interact with the database server without modification, and the database server maintains the MF data 104 in addition to the PF data 112. Additionally, it is transparent to those applications that the database server can use the MF data 104 to more efficiently process some or all of those database commands.

[0044] Mirror format data

[0045] The MF data 104 can mirror all or a subset of the PF data 112. In one embodiment, the user can specify which part of the PF data 112 is "enabled in memory". The specification can be made at any granularity level, including column and row ranges.

[0046] As will be described below, the data enabled in memory is converted to a mirrored format and stored as MF data 104 in the RAM 102. Thus, when a database statement requires data enabled in memory, the database server can choose to provide the data from either the PF data 112 or the MF data 104. The conversion and loading can be done at database startup or in a lazy or on-demand manner. The MF data 104 does not mirror data that is not enabled in memory. Thus, when a query requests such data, the database server cannot choose to obtain the data from the MF data 104.

[0047] For purposes of explanation, it will be assumed that the PF data structure 108 includes Figure 2A the table 200 shown in. The table 200 includes four columns C1, SSD C2, C3, and C4, and eight rows R1, R2, R3, R4, R5, R6, R7, and R8. The SSD C2 stores semi-structured data.

[0048] Rows in the persistent storage device can be uniquely identified by a row id. In the table 200, the first row is associated with the row id R1, and the last row is associated with the row id R8. The columns of a row can be referenced herein by the concatenation of the row id and the column. For example, the column C1 of row R1 is identified by R1C1, and C3 of R5 is identified by R5C3.

[0049] Figure 2B Illustrates how the data resident in the table 200 can be physically organized on the persistent storage device 110. In this example, the data for the table 200 is stored in four row-major data blocks 202, 204, 206, and 208. The block 202 stores the values of all columns for row R1, and then stores the values of all columns for row R2. The block 204 stores the values of all columns for row R3, and then stores the values of all columns for row R4. The block 206 stores the values of all columns for row R5, and then stores the values of all columns for row R6. Finally, the block 208 stores the values of all columns for row R7, and then stores the values of all columns for row R8.

[0050] According to an embodiment, the column SSD C2 can be a LOB column defined by the DBMS100 to store semi-structured data. For a specific row, the LOB in SSD C2 can be inline or out-of-line. For an inline LOB of a row, the data of the LOB is physically stored in the same data block of that row. For an out-of-line LOB, a reference is stored in that row of the data block; the reference refers to the location where the data for the LOB is stored in another set of data blocks. In effect, the out-of-line LOB is logically but not physically stored in the data block containing the out-of-line LOB. For purposes of illustration, the LOB in a column of a row in a data block may be referred to herein as being stored or contained within that data block, regardless of whether the LOB is inline or out-of-line.

[0051] Copies of data blocks may be temporarily stored in the buffer cache 106. Any of a variety of cache management techniques may be used to manage the buffer cache 106.

[0052] Hybrid Derived Cache and IMCU

[0053] According to an embodiment, the MF data 104 is cached and maintained within a hybrid derived cache. Within the hybrid derived cache, the MF data 104 is stored in units herein referred to as in-memory compression units (IMCUs). Each IMCU stores a different set of MF data.

[0054] Figure 3 Depicted is a hybrid derived cache 300, a hybrid derived cache according to an embodiment of the present invention. As Figure 3 shown, the hybrid derived cache 300 includes IMCUs 302 and 304.

[0055] The IMCUs are organized in a manner corresponding to the organization of the PF data. For example, on the persistent storage device 110, the PF data may be stored in a series of contiguous data blocks (within the address space). In such cases, within the hybrid derived cache 300, the MF data 104 stores data from a series of data blocks. The IMCU 302 holds the MF data from rows R1 - R4, while the IMCU 304 holds the MF data from rows R5 - R8.

[0056] The IMCU 302 holds the column values of C1 for rows R1 - R4 in column vector 320 and holds the column values of C3 for rows R1 - R4 in column vector 322. The SSDM unit 332 holds the semi - structured data for rows R1 - R4 in SSDM form. The IMCU 304 holds the column values of C1 for rows R5 - R8 in column vector 324 and holds the column values of C3 for rows R5 - R8 in column vector 326. The SSDM unit 334 holds the semi - structured data for rows R5 - R8 in SSDM form.

[0057] The column vectors depicted in the hybrid - derived cache 300 are dictionary - compressed. In dictionary - based compression of columns, values are represented by dictionary codes, which are typically much smaller than the values they represent. The dictionary maps the dictionary codes to the values. In the column vectors of columns, the occurrences of values in the column vectors are represented by dictionary codes within the column vectors, and the dictionary maps the dictionary codes to the values.

[0058] According to an embodiment, each IMCU encodes that column vector according to the dictionary for the column vector. Column vector 320 and column vector 322 are encoded according to dictionaries 340 and 342 respectively, and column vector 324 and column vector 326 are encoded according to dictionaries 344 and 346 respectively.

[0059] Each dictionary code in the column vector is stored within the corresponding element of the column vector, and the corresponding element corresponds to an ordinal position or index. For example, in column vector 324, indices 0, 1, 2, and 3 correspond to the first, second, third, and fourth elements respectively.

[0060] When the term "row" is used herein with reference to one or more column vectors, a "row" refers to a set of one or more elements that have the same index in each column vector across a set of column vector elements and correspond to the same row. The row id of the row and the index corresponding to the set of elements can be used to identify the set of elements. For example, row R5 and row 0 refer to the first element in each of column vectors 324 and 326.

[0061] The row id mapping 352 in the IMCU 302 and the row id mapping 354 in the IMCU 304 map the rows in the column vectors to row ids. According to an embodiment, the row id mapping is a column vector containing row ids. The rows in the column vectors are mapped to the row ids at the same index positions of that row in the row id mapping. Element 0 in column vector 324 and column vector 326 is mapped to row R5, and R5 is the value of element 0 in the row id mapping 354.

[0062] Prediction Evaluation and Row ID Resolution

[0063] A conventional database system can respond to a query by first searching for the requested data in buffer cache 106 and thus operate normally. If the data is in buffer cache 106, then the data is accessed from buffer cache 106. Otherwise, the required data is loaded from the PF data structure 108 into buffer cache 106 and then accessed from buffer cache 106. However, since the data in both buffer cache 106 and PF data structure 108 is in a persistent format, performing operations based solely on PF data does not always provide optimal performance. Performing operations on PF data in this way is referred to herein as PF-side processing.

[0064] According to an embodiment, the database server uses a hybrid-derived cache 300 to perform at least some of the database operations required to execute a database query. Such operations include predicate evaluation, projection, and aggregation. The larger the portion of the database access required for the execution of a query that can be satisfied using the hybrid-derived cache, the larger the cache footprint.

[0065] According to an embodiment, predicate evaluation for multiple columns can be performed at least in part by accessing MF data in the IMCU. For each column cached in the IMCU, the result of the predicate evaluation is recorded in an index-aligned result vector. In the indexed-aligned result vector, each bit in the bit vector corresponds to an index of the column vector, bit 0 corresponds to the 0th index, bit 1 corresponds to the first index, and so on. Hereinafter, the term "result vector" is used to refer to the indexed-aligned bit vector. For a join predicate, multiple result vectors can be generated, each representing the result of a predicate connective, and then combined to generate a "return result vector" that represents the result of the join predicate evaluation for the columns cached in the IMCU.

[0066] The return result vector can be used to perform row parsing. Row parsing uses the return result vector and a row ID mapping to generate a return row list that includes the row IDs of the rows identified by the return result vector and that satisfy the evaluation represented by the return result vector. The return row list can be used to perform further PF-side predicate evaluation of the rows in the return row list for any columns not cached in the hybrid-derived cache 300, or to perform other operations (such as projection of non-cached columns, or evaluation of other predicate conditions for that predicate or other predicates in the query).

[0067] For example, the DBMS 100 evaluates a query with predicates C1 = "SAN JOSE", C3 = "CA", and C4 > "1400.00". The DBMS 100 evaluates the predicates for C1 and C3 in the IMCU 304. A bit vector "1110" is generated for the column vector 324 (C1), and a bit vector "0110" is generated for the column vector 326 (C3). Performing an AND operation generates a return vector "0110". Based on the row id mapping 354, rows R6 and R7 are mapped to the set bits in the return vector. The DBMS 100 generates a row return list that includes rows R6 and R7. Based on the row return list, the DBMS 100 evaluates rows R6 and R7 for the predicate condition C4 > "1400.00" using the PF side predicate evaluation.

[0068] Examples of semi-structured data

[0069] The SSDM units within the IMCU are structured to facilitate predicate evaluation on semi-structured data. Before describing the SSDM units, it is useful to describe semi-structured data in more detail. Semi-structured data generally refers to a collection of hierarchical data objects in this document, where the hierarchical structure is not necessarily uniform across all objects in the collection. Hierarchical data objects are often data objects marked up by a hierarchical markup language. XML and JSON are examples of hierarchical markup languages.

[0070] Data structured using a hierarchical markup language consists of nodes. Nodes are delimited by delimiters that mark the nodes and can be tagged with a name (referred to as a tag name in this document). Generally, the syntax of a hierarchical markup language specifies that the tag name is embedded within, juxtaposed with, or otherwise syntactically associated with the delimiter of the delimited node.

[0071] For XML data, nodes are delimited by start and end tags that include the tag name. For example, in the following XML fragment X,

[0072] <zipcode>

[0073] <code>95125< / code>

[0074] <city>SAN JOSE< / city>

[0075] <state>CA< / state>

[0076] < / zipcode>

[0077] The start tag <ZIP CODE> and the end tag < / ZIP CODE> delimit a node with the name "ZIP CODE".

[0078] Figure 4A is a node tree representing the above XML fragment X. Refer to Figure 4A, which describes the hierarchical data object 401. Non-leaf nodes are depicted with double-line boundaries, while leaf nodes are depicted with single-line boundaries. In XML, non-leaf nodes correspond to element nodes, and leaf nodes correspond to data nodes. Element nodes in the node tree are referred to in this text by the name of the node, which is the name of the element represented by the node. For ease of explanation, data nodes are referred to by the value represented by the data node.

[0079] The data between corresponding tags is called the content of the node. For a data node, the content can be a scalar value (e.g., an integer, a text string, a date).

[0080] Non-leaf nodes (such as element nodes) contain one or more other nodes. For an element node, the content can be data nodes and / or one or more element nodes.

[0081] ZIPCODE is an element node that contains child nodes CODE, CITY, and STATE, which are also element nodes. The data nodes 95125, SAN JOSE, and CA are the data nodes of the element nodes CODE, CITY, and STATE, respectively.

[0082] The nodes contained by a specific node are referred to in this text as the descendant nodes of that specific node. CODE, CITY, and STATE are descendant nodes of ZIPCODE. 95125 is a descendant node of CODE and ZIPCODE, SAN JOSE is a descendant node of CITY and ZIPCODE, and CA is a descendant node of STATE and ZIPCODE.

[0083] Thus, non-leaf nodes form a hierarchy of nodes with multiple levels, and the non-leaf nodes are at the highest level. Nodes at each level are linked to one or more nodes at different levels. Any given node at a level below the highest level is a child node of the parent node immediately above that given node. Nodes with the same parent are sibling nodes. A parent node can have multiple child nodes. A node that has no parent node linked to it is a root node. A node that has no child nodes is a leaf node. A node that has one or more descendant nodes is a non-leaf node.

[0084] For example, in the container node ZIP CODE, the node ZIP CODE is the root node at the highest level. The nodes 95125, SAN JOSE, and CA are leaf nodes.

[0085] In this text, the term "hierarchical data object" is used to refer to a sequence of one or more non-leaf nodes, each non-leaf node having child nodes. An XML document is an example of a hierarchical data object. Another example is a JSON object.

[0086] JSON

[0087] JSON is a lightweight hierarchical markup language. A JSON object consists of a collection of fields, each field being a field name / value pair. The field name is actually the tag name of a node in the JSON object. The name of a field is separated from the value of that field by a colon. A JSON value can be:

[0088] An object, which is a list of fields enclosed in curly braces "{}" and separated by commas.

[0089] An array, which is a list of JSON nodes and / or values enclosed in square brackets "[]" and separated by commas.

[0090] An atom, which is a string, a number, true, false, or null.

[0091] The following JSON hierarchical data object J is used to illustrate JSON.

[0092] {

[0093] "FIRSTNAME": "JACK",

[0094] "LASTNAME": "SMITH",

[0095] "BORN": {

[0096] "CITY": "SAN JOSE",

[0097] "STATE": "CA",

[0098] "DATE": "11 / 08 / 82"

[0099] },

[0100] }

[0101] The hierarchical data object J contains the fields FIRSTNAME (given name), LASTNAME (family name), and BORN (birth), CITY (city), STATE (state), and DATE (date). FIRSTNAME and LASTNAME have the atomic string values "JACK" and "SMITH", respectively. BORN is a JSON object containing the member fields CITY, STATE, and DATE, which have the atomic string values "SAN JOSE", "CA", and "11 / 08 / 82", respectively.

[0102] Each field in the JSON object is a non-leaf node and the name of the non-leaf node is the field name. Each non-empty array and non-empty object are non-leaf nodes, and each empty array and empty object are leaf nodes. Data nodes correspond to atomic values.

[0103] Figure 4B The hierarchical data object J is described as a hierarchical data object 410 that includes nodes, as described below. Refer to Figure 4B , there are three root nodes, which are FIRSTNAME, LASTNAME, and BORN. Each of FIRSTNAME, LASTNAME, and BORN is a field node. BORN has descendant object nodes labeled OBJECT NODE.

[0104] OBJECT NODE is referred to as a containing node in this article because it represents a value that can contain one or more other values. In the case of OBJECT NODE, it represents a JSON object. Starting from OBJECT NODE, three descendant field nodes are passed down, which are CITY, STATE, and DATE. Another example of a containing node is an object node that represents a JSON array.

[0105] The nodes FIRSTNAME, LASTNAME, CITY, and STATE have atomic data nodes that represent the atomic string values "JACK", "SMITH", "SAN JOSE", and "CA", respectively. The node DATE has a descendant data node that represents the date type value "11 / 08 / 82".

[0106] Path

[0107] A path expression is an expression that includes a sequence of "path steps" that identify one or more nodes in a hierarchical data object based on each hierarchical position of one or more nodes. "Path steps" can be separated by " / ", ".", or other delimiters. Each path step can be the tag name of a node in the path that identifies a node within the hierarchical data object. XPath is a query language that specifies the path language for path expressions. Another query language is SQL / JSON, which is part of the SQL / JSON standard jointly developed by Oracle and IBM with other RDBMS vendors.

[0108] In SQL / JSON, an example of a path expression for JSON is "$.BORN.DATE". The step "DATE" specifies the node with the node name "DATE". The step "BORN" specifies the node name of the parent node of the node "DATE". "$" specifies the context of the path expression, which, by default, is the hierarchical data object for which the path expression is to be evaluated.

[0109] Path expressions can also specify predicates or criteria for steps that nodes should satisfy. For example, the following query

[0110] $.born.date>TO_DATE('1998-09-09', 'YYYY-MM-DD')

[0111] specifies that the node “born.date” is greater than the date value “1998-09-09”.

[0112] Illustrative SSD columns and posting lists

[0113] According to an embodiment, the SSDM unit includes a posting index that indexes flags to flag positions found in hierarchical data objects. The posting index is illustrated using the hierarchical data object depicted in Figure 5A Illustrative posting indexes are depicted in Figure 6A and a hashed posting index used in an embodiment is depicted in Figure 6B which is a posting index in the form of a serialized hash table.

[0114] Refer to Figure 5A which depicts the hierarchical data object in PF form. Refer to Figure 5A For each row in R5, R6, R7, and R8, a JSON object is stored. The JSON object in persistent form or mirror format is referred to herein by the row id of the row containing the JSON object. Row R5 contains JSON object R5 for illustrative flags.

[0115] The posting index includes flags extracted from the hierarchical data object. The flags can be the tag name of a non-leaf node, a word in a data node, or another value in a data node, a delimiter, or another feature of the hierarchical data object. Parsing and / or extracting the flags from the hierarchical data object is required to form the posting index. Not all flags found in the hierarchical data object need to be indexed by the posting index.

[0116] Generally, the posting index indexes flags that are the tag name or a word in a string value data node. The posting index indexes each flag to one or more hierarchical data objects that contain the flag and to the character sequences in those hierarchical data objects that cover the content of the flag.

[0117] For JSON, the flags can be:

[0118] 1) The start of an object or array.

[0119] 2) A field name.

[0120] 3) Atomic value, in the case of non-string JSON values.

[0121] 4) Words in a string atomic value.

[0122] A publication index can index a collection of hierarchical data objects conforming to a hierarchical markup language such as JSON, XML, or a combination thereof. A hierarchical data object conforming to JSON is used to illustrate the publication index; however, embodiments of the publication index are not limited to JSON.

[0123] Within a hierarchical data object, flags are sorted. The order of the flags is represented by an ordinal number referred to herein as the flag number.

[0124] Flags corresponding to field names or words in a string atomic value are used to form key values for the publication index. Each index entry in the publication index maps a flag to a flag position, which can be a flag number or a flag range defined by, for example, a start flag number and an end flag number.

[0125] Flags corresponding to the tag names of non-leaf nodes are referred to herein as tag name flags. With respect to JSON, the tag name is the field name, and the flag corresponding to the field name is the tag name flag, but may also be referred to herein as a field flag.

[0126] A flag range specifies the region in a JSON object corresponding to a field node and the content of the field node according to the flag numbers of the flags.

[0127] Figure 5B Flags in the JSON object R5 are depicted. Figure 5B Each call out label in references a flag in the JSON object 505 by flag number. Flag #1 is the start of the JSON object. Flag #2 is the field name "name". Flag #3 is the first word in the atomic string value of the field "name". Flag #4 is the second word in the atomic string value of the field "name". Flag #7 corresponds to the field flag "friends". Flag #8 is a delimiter corresponding to the start of an array that is the JSON value of the field "friends". Flags #9 and #15 are each delimiters corresponding to the start of an object within the array.

[0128] Flags that are words in the string value of a data node are referred to herein as word flags. Flag #3 is a word flag. Flags corresponding to non-string values of data nodes are referred to herein as value flags. Value flags are not shown herein.

[0129] Exemplary publication index and hash publication index

[0130] Figure 6ADepicts the posting index 601. The posting index 601 is a table that indexes flags to flag positions within the JSON objects R5, R6, R7, and R8. According to an embodiment, the posting index 601 includes columns Token (flag), Type (type), and column PostingList (posting list). The column Token contains flags corresponding to field flags and word flags found in the JSON object. The column Type contains the flag type, which can be "tagname" to specify a flag as a field flag and can be "keyword" to specify a flag as a word flag.

[0131] The column PostingList contains the posting list. The posting list includes one or more object - flag - position lists. The object - flag - position list includes an object reference to a hierarchical data object such as a JSON object, and one or more flag positions within the hierarchical data object for flag occurrences. Each posting index entry in the posting index 601 maps a flag to the posting list, thereby mapping the flag to each JSON object containing the flag and one or more flag positions within each JSON object.

[0132] Figure 6A The depiction of the posting index 601 in shows the entries generated for the JSON objects in SSD column C2. Refer to Figure 6A , the posting index 601 contains an entry for "name". Since this entry contains "tagname" in the column Type, this entry maps the flag "name" as a field flag to the JSON object and flag positions specified in the posting list (0,(2 - 4)(10 - 12)(16 - 19))(1,(2 - 4))(2,(2 - 4))(3,(2 - 4)). The object - flag - position list for the flag "name" contains the following object - flag - position list:

[0133] (0,(2 - 4),(10 - 12),(16 - 19)) This object - flag - position list includes the object reference 0, which references the index of the row containing the JSON object R5. The object - flag - position list maps the field flag "name" to the regions defined by flag ranges #2 - #4, flag ranges #10 - #12, and flag ranges #16 - #19 in the JSON object R5.

[0134] The object references in the object - flag - position list are called index - aligned because for a row containing a hierarchical data object, the object reference is the index of that row across the column vector in the IMCU. Similarly, the posting index 601 and the posting index entries therein are called index - aligned in this document because the posting index entries map flags to hierarchical data objects based on indexed - aligned object references.

[0135] (1, (2 - 4)) This object - flag - position list refers to the JSON object R6 at index 1 of column vectors 324 and 326. The object list maps the field node "name" to the region in the JSON object R6 defined by flag range #2 - #4.

[0136] The posting index 601 also contains an entry for "Helen". Since the entry contains "keyword" in Type, this entry maps the flag "Helen" as a word in the data node to the JSON objects and regions specified in the posting list (0, (11, 17))(1, (3)). This posting list contains object - level lists as follows:

[0137] (0, (11, 17)) This object - flag - position list refers to the JSON object R5 at index 0. The object - level list further maps the word "Helen" to two flag positions in the JSON object 505 defined by flag positions #11 and #17.

[0138] (1, (3)) This object - flag - position list refers to the JSON object 506 of row R6 at index 0. The object reference maps the word "Helen" to the flag position #3 in the JSON object 506.

[0139] Hashed posting index

[0140] For more efficient storage, the posting index is stored as a serialized hash table within the SSDM unit. Each hash bucket of the hash table stores one or more entries of the posting index entries.

[0141] Figure 6B Shows the posting index 601 in the form of a serialized hash table, as the hashed posting index 651. Refer to Figure 6B , the hashed posting index 651 includes four hash buckets HB0, HB1, HB2, and HB3. Figure 6C Is an exploded view of the hashed posting index 651, which shows the contents of each hash bucket HB0, HB1, HB2, and HB3. Each hash bucket can hold one or more posting list index entries.

[0142] The specific hash bucket in which the posting index entry is held is determined by applying a hash function to the flag of the posting list index entry. For the flags "USA", "citizenship", "YAN", the hash function evaluates to 0; thus, the posting lists for each of the flags "USA", "citizenship", "YAN" are stored in HB0.

[0143] Serialized hash buckets are stored in memory as a byte stream. In an embodiment, each hash bucket and components within the hash bucket, such as posting lists and object - flag - position lists, can be delimited by delimiters and can include a header. For example, hash bucket HB0 can include a header having an offset to the next hash bucket HB1. Each object - level list in HB0 can include an offset to the next object list, if any. In another embodiment, an auxiliary array can contain offsets to each of the serialized hash buckets. Serialized hash buckets can have less memory footprint than the non - serialized form of the hash buckets.

[0144] Predicate Evaluation Using a Distributed Index

[0145] Index alignment of object references in an object list enables and / or facilitates generation of a bit vector for a predicate condition on SSD C2, which can be combined with other bit vectors generated for predicate evaluation on column vectors to generate a return vector. For example, DBMS100 is evaluating a query with join predicates "C1 = "SAN JOSE" AND C3 = "CA" AND C2 contains "UK". DBMS 100 evaluates the predicates for C1 and C3 in IMCU 304. A bit vector "1110" is generated for the predicate connective C1 = "SAN JOSE" on column vector 324 (C1) and a bit vector "0110" is generated for the predicate connective C3 = "CA" on column vector 326 (C3).

[0146] For the predicate condition that C2 contains "UK" on column C2, access the hash table to determine which rows hold JSON objects that contain "UK". Applying the value "UK" to the hash function produces 1. Evaluate hash bucket HB1 to determine which JSON objects contain "UK". The only object list mapped to "UK" includes an object reference with index 2. Based on object reference 2, set the corresponding third bit in the result vector, thereby generating the bit vector 0010.

[0147] Perform an AND operation between the bit vectors generated for the predicates. The AND operation generates a return result vector 0010.

[0148] Predicate Evaluation Requiring Function Evaluation

[0149] When performing an evaluation on a hierarchical data object, it is important to be able to specify the structural characteristics of the data to be returned. The structural characteristics are based on the position of the data within the hierarchical data object. Evaluation based on such characteristics is referred to herein as function evaluation. Examples of structural characteristics include element containment, field containment, string proximity, and path - based position within the hierarchical data object. Flag positions within the posting list index can be used to evaluate the structural characteristics at least in part.

[0150] For example, the predicate condition $.CITIZENSHIP = "USA" specifies that the field "citizenship" within the JSON object "citizenship (nationality)" is equal to the string value "USA". To determine whether the predicate is satisfied, the publication index 651 of the hash can be examined to find the hierarchical data object with the tag name "citizenship", and for each such object, it is determined whether the flag within the range of the flag position of the tag name "citizenship" is a keyword equal to "USA".

[0151] Specifically, the hash function 651 of the publication index for the hash is applied to generate the hash value 0 for "citizenship". The hash bucket HB0 is examined to determine that the tag name "citizenship" is included in the hierarchical data objects referred to by the indices 0, 1, 2, 3, which are the JSON objects R5, R6, R7, and R8. For the JSON object R6 (the JSON object referred to by index 1), the flag position of the tag name is (5 - 6).

[0152] Next, the publication list for "USA" is examined. The hash function is applied to "USA", thereby generating the hash value 0. The hash bucket HB0 is examined to determine that the keyword "USA" is the keyword flag at the flag position 6. Thus, the hierarchical data object at index 1 in row R6 satisfies the predicate.

[0153] Similarly, the other flag positions of the tag name "citizenship" are evaluated. An example of flag - position - based predicate evaluation of structural features, including path evaluation, is described in U.S. Patent No. 9,659,045, titled "Generic Indexing For Efficiently Supporting Ad - Hoc Query Over Hierarchical Marked - Up Data", filed on September 26, 2014 by Zhen Hua Liu et al., the entire content of which is incorporated herein by reference.

[0154] Transaction Consistency

[0155] To make the MF data transactionally consistent with the PF data, the transactions committed to the PF data are reflected in the MF data. According to an embodiment, to reflect the committed changes in the semi-structured data in PF form, the SSDM unit itself need not change, but rather maintains metadata to indicate which hierarchical data objects cached in the IMCU have been updated. A mechanism for maintaining such metadata for the MF data is described in U.S. Patent No. 9,128,972, titled "Multi-Version Concurrency Control On In-Memory Snapshot Store Of Oracle In-Memory Database," filed on July 21, 2014, by Vivekanandhan Raja et al., the content of which is incorporated herein and is referred to herein as the "Concurrency Control application."

[0156] In an embodiment, changes to the rows in the IMCU are indicated by maintaining a "vector of row changes" within the IMCU. For example, when a transaction performs an update to row R6, the vector of row changes of the IMCU 304 is updated by setting the bit corresponding to row R6.

[0157] For a transaction that requests the latest version of a data item, the set bit in the row change bit vector indicates that the MF data of the row is stale, and thus the IMCU 304 cannot be used to evaluate the predicates for that row or return data from that row. Instead, PF-side evaluation is used to evaluate the predicates for that row.

[0158] However, not all transactions request the latest version of a data item. For example, in many DBMSs, a snapshot time is assigned to a transaction, and the data reflecting the state of the database at that snapshot time is returned. Specifically, if a snapshot time T3 is assigned to a transaction, then all versions of the data item that include all changes committed before T3 and do not include changes not committed at T3 (except for the changes made by the transaction itself) must be provided for that transaction. For such transactions, the set bit in the row change bit vector does not necessarily indicate that the IMCU 304 cannot be used for the corresponding row.

[0159] In an embodiment, to account for the snapshot time of a transaction that reads the values mirrored in the IMCU 304, a snapshot invalidation vector is created for each transaction attempting to read data from the IMCU. The snapshot invalidation vector is specific to the snapshot time because the bits in the snapshot invalidation vector are set only for the rows that were modified before the snapshot time associated with the transaction for which the snapshot invalidation vector is constructed. The generation and processing of the snapshot invalidation vector are described in further detail in the Concurrency Control application.

[0160] Predicate evaluation using transaction consistency

[0161] To evaluate the predicate condition for column SSD C2 using the transaction consistency with respect to the snapshot time of the query, the hashed publish index 651 can be used for any row for which the corresponding snapshot invalidation vector specifies valid, i.e., for any row for which the corresponding bit in the snapshot invalidation vector is not set. For any row for which the snapshot invalidation vector specifies invalid, PF side evaluation is performed.

[0162] Figure 7 is a flowchart depicting the process for evaluating predicate conditions for a query and snapshot time. The process is illustrated using IMCU 304, the following predicates C1 = "SAN JOSE" and JSON_EXISTS($.CITIZENSHIP = "USA"), and the snapshot invalidation vector "0100". The snapshot invalidation vector specifies that row R6 is invalid.

[0163] The process selectively performs predicate evaluation on SSD C2 to more efficiently evaluate predicates not only for SSD C2 but also for column vectors used for another column. Specifically, equality predicate conditions based on scalar column vectors can be evaluated more efficiently compared to predicate conditions based on the hashed publish index 651. For a join predicate, once it is determined that a row does not satisfy the predicate connective of the join predicate, that row cannot satisfy the predicate regardless of the result of evaluating another predicate connective of the predicate on that row. Thus, the evaluation of another predicate connective on a row that has been deemed ineligible by a previously evaluated predicate connective can be skipped in order to determine the evaluation result of the predicate, thereby reducing the processing required to evaluate the join predicate.

[0164] Reference Figure 7 , at 705, a snapshot invalidation vector "0100" is generated. The snapshot invalidation vector specifies that row R6 is invalid.

[0165] At 710, the predicate condition C1 = "SAN JOSE" is evaluated for column vector 324. Thus, the predicate connective is evaluated for rows R5, R7, and R8, resulting in the result vector 1110. In an embodiment, for a row that passes the predicate evaluation, the bit in the result vector is set to 1. Thus, when the bit in the result vector or the return vector is set to 0, it specifies that the corresponding row has been deemed ineligible.

[0166] At 715, the predicate condition JSON_EXISTS($.CITIZENSHIP = "USA") is evaluated against the hashed publication index 651. The predicate is evaluated similar to the example described in the "Predicate Evaluation Based on Structural Features" section. However, the object - flag - position list for invalid or non - compliant rows can be ignored. Row R8 is considered non - compliant because row R8 does not satisfy the previously evaluated predicate connective. Thus, the predicate is evaluated against rows R5 and R7 using the hashed distribution index 651. The resulting vector generated is equal to 1100.

[0167] At 720, the resulting vectors generated are combined in an AND operation to generate a return vector. The return vector generated is 1100.

[0168] At 726, PF - side evaluation is performed on the invalid rows that have not yet been considered non - compliant. In the current illustration, the snapshot invalid vector designates row R6 as invalid. Thus, PF - side evaluation is performed on row R6.

[0169] PF - side Evaluation of Hierarchical Data Objects

[0170] Predicate evaluation of hierarchical data objects stored in PF form may or may not require functional evaluation. For example, the hierarchical data object in PF form in SSD C2 is stored in a binary streamable form. To evaluate the operator CONTAINS against the hierarchical data object in a row, the hierarchical data object can simply be streamed to find the specific string required by the operator.

[0171] However, for predicate evaluation that requires functional evaluation, the functional evaluation requires a representation of the hierarchical data object that enables functional evaluation and / or enables more efficient functional evaluation. According to an embodiment of the present invention, an object - level publication index is generated for the hierarchical data object to be evaluated through PF - side processing. The object - level publication index is actually a publication index for a single hierarchical data object. When performing PF - side predicate evaluation, an object - level publication index can be generated for any invalid row in the IMCU.

[0172] Figure 8 The object - level publication index 801 of SSD C2 for row R6 is depicted. The object - level publication index 801 shares many features of the publication index 601, but there is no object reference to the hierarchical data object because the object - level publication index 801 belongs to only a single hierarchical data object.

[0173] Reference Figure 8, in the object-level publication index 801, the column Token contains flags corresponding to the field flags and keyword flags found in the JSON object R6. The column Type contains the flag type for each entry. The column Token LocationList (flag location list) contains one or more flag locations for each entry, for flagging occurrences for that entry.

[0174] Publication index for refreshing the hash

[0175] As mentioned in the Mirroring application, the IMCU can be refreshed so that the MF data in it is synchronized with the PF data at a specific point in time. The publication index for the hash can be refreshed by implementing the publication index for the hash from scratch, which would require processing and parsing the JSON objects in all rows of the IMCU.

[0176] To more efficiently refresh the publication index for the hash, the publication index for the hash can be incrementally refreshed by modifying the existing publication index for the hash. Since the publication index for the hash is a serialized structure, the publication index for the hash may not be modified in segments to reflect any changes to the JSON objects. Instead, the refreshed publication index for the hash is formed by merging the valid part of the existing publication index for the hash with an "incremental publication index" (a publication index formed only for the changed rows). To perform the incremental refresh, for each existing publication list index entry, a sorted merge is performed between any object-level-tag lists that have not changed in it and any object-flag-location lists in the corresponding publication list index in the incremental publication index. It is important to note that within the publication list index entry, the object-flag-location lists are maintained in order based on the object identifier to facilitate the sorted merge.

[0177] Figure 9A Depicts the process of generating a "refreshed publication list entry" for incremental refresh of the publication index for the hash. Figure 9B Depicts an incremental publication index according to an embodiment of the present invention.

[0178] Reference Figure 9B , which depicts the incremental publication index 901, that is, the publication list index for the changed JSON objects in the IMCU 304. The incremental publication index 901 is structured similarly to the publication index 601, except that the incremental publication index 901 only indexes the JSON objects of the changed rows R5 and R6. A bit vector for the row changes in the IMCU 304 specifies that these rows have changed.

[0179] A process for generating a refreshed posting list entry is performed on each posting list entry. Generally, each existing posting list entry in each hash bucket is processed to form a refreshed hash posting index entry. The refreshed posting index entries are appended to the current version of the refreshed hash posting index being formed. As the refreshed posting index entries are appended, delimiters and offsets are added, such as delimiters for the hash bucket and the hash posting index entries. The operation of adding delimiters and offsets is not depicted in Figure 9A which is not shown.

[0180] For each changed JSON object, the incremental posting index 901 indexes the entire JSON object rather than just the changed nodes. Regarding the changes, in the JSON object at row R5, the string "JOHN" has been changed to "JACK". In the JSON object at row R6, the string "USA" has been changed to "UK". The incremental posting index 801 is index-aligned. Thus, the object references therein are column vector indexes.

[0181] The process depicted in Figure 9A is illustrated using the hash posting index 651 and the incremental posting index 901. The refreshed hash posting index is not depicted. The illustration starts with the first distribution index entry in HB0 (the distribution index entry for "USA").

[0182] Refer to Figure 9A , at 905, the next existing object - flag - position list is read from the hash posting index 651. The next existing object - flag - position list is (0,(6)(14)(21)), which is the first existing object - flag - position list for "USA".

[0183] At 910, it is determined whether the next existing object - flag - position list is valid. The row change vector specifies that the JSON object in row R5 indexed to 0 has not changed and is thus valid.

[0184] At 915, any prior incremental object - flag - position lists for the corresponding posting index entry in the incremental posting index 901 are added to the refreshed hash posting index being formed. The object reference of the current next existing object - flag - position list is 0. The corresponding posting index entry in the incremental posting index 901 contains an object - flag - position list which is (2,(6)) and has an object reference 2. Since the object reference 2 is greater than 0, there is no prior object - flag - position list to add.

[0185] At 920, add the next existing object - flag - position list to the publish index entry of the refreshed hash. At 905, read the next existing object - flag - position list (1,(6)).

[0186] At 910, invalidate the object - flag - position list based on the vector of line changes. The object - flag - position list is ignored. The process advances to 905. At 905, read the next existing object - flag - position list (3,(6)).

[0187] At 915, add any prior incremental object - flag - position list in the incremental publish index 901 for the corresponding publish index entry to the publish index entry of the refreshed hash. The corresponding publish index entry contains an object - flag - position list which is (2,(6)) and has object reference 2. Since object reference 2 is less than 3, the object - flag - position list (2,(6)) is appended to the publish index entry of the refreshed hash.

[0188] At 920, append the next existing object - flag - position list to the publish index entry of the formed refreshed hash. At 905, read the next existing object - flag - position list (3,(6)). At 910, determine that the object - flag - position list is valid based on the vector of line changes. At 920, append the next existing object - flag - position list to the publish index entry of the refreshed hash.

[0189] At this point in the execution of the process, the refreshed object - flag - position list is (0,(2 - 4)(10 - 12)(16,19))(2,(6))(3,(6)). There are no more object - flag - position lists in the existing publish index entries, so the execution advances to 930.

[0190] At 930, any remaining object - flag - position lists from the corresponding publish index entries in the incremental publish index 901 that have not been appended are appended to the refreshed object - flag - position list. Since there are none, none are appended.

[0191] At 935, determine whether any object - flag - position lists have been appended to the refreshed publish index entry. If not, then do not append anything to the publish index of the refreshed hash. However, object - flag - position lists have been appended. Therefore, the refreshed object - flag - position list is appended to the publish index of the refreshed hash.

[0192] At the end of reading the publication index entries within a hash bucket, there may be publication index entries that belong to that hash bucket but have not been appended to the flags of that hash bucket. These publication index entries are appended to the publication index of the refreshed hash before proceeding to the next hash bucket (if any).

[0193] Unstructured text data

[0194] The methods for storing and querying SSDM described herein can be used for unstructured text data. Similar to SSDM, except that the posting list does not need to include a type attribute to represent the type of the flag as a keyword or tag name, the IMCU can store unstructured text data from a column in the posting list. Similarly, the object-level posting list and the publication index of the hash do not contain such a type attribute for the flag. Transaction consistency, PF-side evaluation, and refresh of the publication index of the hash are provided in a similar manner.

[0195] Database overview

[0196] Embodiments of the present invention are used in the context of a database management system (DBMS). Accordingly, a description of an example DBMS is provided.

[0197] Generally, a server such as a database server is a combination of integrated software components and the allocation of computing resources such as memory, nodes, and processes on the nodes for executing the integrated software components, where the combination of software and computing resources is dedicated to providing a specific type of function on behalf of the clients of the server. The database server controls and facilitates access to a specific database and processes requests from clients to access the database.

[0198] A database includes data and metadata stored on a persistent storage mechanism (such as a collection of hard disks). Such data and metadata can be logically stored in the database, for example, according to a relational and / or object-relational database structure.

[0199] A user interacts with the database server by submitting commands to the database server of the DBMS that cause the database server to perform operations on the data stored in the database. The user can be one or more applications running on a client computer that is interacting with the database server. Multiple users can also be collectively referred to as users in this document.

[0200] A database command can be in the form of a database statement. In order for a database server to process a database statement, the database statement must conform to a database language supported by the database server. A non-limiting example of a database language supported by many database servers is SQL, including proprietary forms of SQL supported by database servers such as Oracle (e.g., Oracle Database 11g). SQL data definition language ("DDL") instructions are issued to a database server to create or configure database objects such as tables, views, or complex types. Data manipulation language ("DML") instructions are issued to a DBMS to manage data stored within a database structure. For example, SELECT, INSERT, UPDATE, and DELETE are examples of common DML instructions in some SQL implementations. SQL / XML is a common extension of SQL used when manipulating XML data in an object-relational database.

[0201] Generally, data is stored in a database in one or more data containers, each container containing records, and the data within each record being organized as one or more fields. In a relational DBMS, the data containers are typically referred to as tables, the records as rows, and the fields as columns. In an object-oriented database, the data containers are typically referred to as object classes, the records as objects, and the fields as attributes. Other database architectures may use other terms. The systems implementing the present invention are not limited to any particular type of data container or database architecture. However, for purposes of explanation, the examples and terms used herein shall generally be associated with relational databases or object-relational databases. Accordingly, the terms "table", "row", and "column" will be used herein to refer to data containers, records, and fields, respectively.

[0202] A multi-node database management system consists of interconnected nodes that share access to the same database. Typically, the nodes are interconnected via a network and share access to a shared storage device to varying degrees, such as shared access to a set of disk drives and the data blocks stored thereon. The nodes in a multi-node DBMS can be in the form of a group of computers (e.g., workstations, personal computers) interconnected via a network. Alternatively, the nodes can be nodes of a grid consisting of server blades in the form of nodes interconnected with other server blades on a rack.

[0203] Each node in a multi-node DBMS hosts a database server. A server, such as a database server, is a combined allocation of integrated software components and computing resources (such as memory, nodes, and processes on the nodes for executing the integrated software components on a processor), a combination of software and computing resources dedicated to performing a specific function on behalf of one or more clients.

[0204] Resources from multiple nodes in a multi-node DBMS can be allocated to run the software of a specific database server. Each combination of the allocation of software and resources in the nodes is a server that is referred to herein as a "server instance" or "instance". A database server can include multiple database instances, some or all of which run on separate computers (including separate server blades).

[0205] Hardware Overview

[0206] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the techniques, or can include digital electronic devices (such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques), or can include one or more general-purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage devices, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to implement the techniques. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wired and / or program logic to implement the techniques.

[0207] For example, Figure 10 is a block diagram illustrating a computer system 1000 on which embodiments of the present invention can be implemented. The computer system 1000 includes a bus 1002 or other communication mechanism for conveying information, and a hardware processor 1004 coupled to the bus 1002 for processing information. The hardware processor 1004 can be, for example, a general-purpose microprocessor.

[0208] The computer system 1000 also includes a main memory 1006 coupled to the bus 1002, such as a random access memory (RAM) or other dynamic storage device, for storing information and instructions to be executed by the processor 1004. The main memory 1006 can also be used to store temporary variables or other intermediate information during execution of instructions by the processor 1004. When stored in a non-transitory storage medium accessible by the processor 1004, these instructions cause the computer system 1000 to become a special-purpose machine customized to perform the operations specified in the instructions.

[0209] The computer system 1000 also includes a read-only memory (ROM) 1008 or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004. A storage device 1010 (such as a magnetic disk, an optical disk, or a solid-state drive) is provided and coupled to the bus 1002 for storing information and instructions.

[0210] The computer system 1000 can be coupled via a bus 1002 to a display 1012 (such as a cathode ray tube (CRT)) for displaying information to a computer user. An input device 1014 including alphanumeric keys and other keys is coupled to the bus 1002 for transmitting information and command selections to the processor 1004. Another type of user input device is a cursor control 1016 (such as a mouse, trackball, or cursor direction keys) for transmitting direction information and command selections to the processor 1004 and for controlling cursor movement on the display 1012. Such input devices typically have two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0211] The computer system 1000 can implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that in combination with the computer system cause the computer system 1000 to be or program the computer system 1000 to be a special-purpose machine. According to one embodiment, the computer system 1000 performs the described techniques in response to one or more sequences of one or more instructions contained in the main memory 1006 being executed by the processor 1004. These instructions can be read into the main memory 1006 from another storage medium (such as the storage device 1006). Execution of the instruction sequence contained in the main memory 1006 causes the processor 1004 to perform the processing steps described herein. In an alternative embodiment, hardwired circuitry may be used in place of or in combination with software instructions.

[0212] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical discs, magnetic disks, or solid state drives, such as the storage device 1010. Volatile media includes dynamic memory, such as the main memory 1006. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, any physical medium with hole patterns, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0213] Storage media is different from transmission media but can be used in combination with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise the bus 1002. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0214] Various forms of media can participate in carrying one or more sequences of one or more instructions to the processor 1004 for execution. For example, the instructions can initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 1000 can receive the data on the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector can receive the data carried in the infrared signal, and appropriate circuitry can place the data on the bus 1002. The bus 1002 transfers the data to the main memory 1006, and the processor 1004 retrieves and executes the instructions from the main memory 1006. The instructions received by the main memory 1006 can optionally be stored on the storage device 1006 before or after being executed by the processor 1004.

[0215] The computer system 1000 also includes a communication interface 1018 coupled to the bus 1002. The communication interface 1018 provides two-way data communication coupled to a network link 1020, where the network link 1020 is connected to a local network 1022. For example, the communication interface 1018 can be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 1018 can be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 1018 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0216] The network link 1020 typically provides data communication through one or more networks to other data devices. For example, the network link 1020 can provide a connection through the local network 1022 to a main computer 1024 or to a data device operated by an Internet Service Provider (ISP) 1026. The ISP 1026 in turn provides data communication services through the global packet data communication network (now commonly referred to as the “Internet” 1028). Both the local network 1022 and the Internet 1028 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through the various networks and signals on the network link 1020 and through the communication interface 1018, which carry digital data to and from the computer system 1000, are example forms of transmission media.

[0217] The computer system 1000 can send messages and receive data, including program code, via one or more networks, network link 1020, and communication interface 1018. In an Internet example, server 1030 can send request code for an application program via Internet 1028, ISP 1026, local network 1022, and communication interface 1018.

[0218] The received code can be executed by processor 1004 when received, and / or stored in storage device 106 or other non-volatile memory for later execution.

[0219] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details, which may vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the invention, and what the applicant intends to be the scope of the invention, is the literal and equivalent scope of the set of claims issued from this application in the specific form in which such claims are issued, including any subsequent corrections.

[0220] Software Overview

[0221] Figure 11 is a block diagram of a basic software system 1100 that can be used to control the operation of computer system 1000. Software system 1100 and its components, including their connections, relationships, and functions, are merely exemplary and are not meant to limit the implementation of one or more example embodiments. Other software systems suitable for implementing one or more example embodiments may have different components, including components with different connections, relationships, and functions.

[0222] Software system 1100 is provided to direct the operation of computer system 1000. Software system 1100, which can be stored on system memory (RAM) 1006 and fixed storage device (e.g., hard disk or flash memory) 1010, includes a kernel or operating system (OS) 1110.

[0223] OS 1110 manages the low-level aspects of computer operation, including managing the execution of processes, memory allocation, file input and output (I / O), and device I / O. One or more applications, represented as 1102A, 1102B, 1102C... 1102N, can be "loaded" (e.g., transferred from fixed storage device 1010 to memory 1006) for execution by system 1100. Applications or other software intended to be used on computer system 1000 can also be stored as a downloadable set of computer-executable instructions, e.g., for downloading and installation from an Internet location (e.g., a web server, app store, or other online service).

[0224] The software system 1100 includes a graphical user interface (GUI) 1115 for receiving user commands and data in a graphical manner (e.g., “click” or “touch gesture”). In turn, these inputs can be operated on by the system 1100 according to instructions from the operating system 1110 and / or one or more applications 1102. The GUI 1115 is also used to display the operation results from the OS 1110 and one or more applications 1102, and the user can provide additional inputs or terminate the session (e.g., log off).

[0225] The OS 1110 can be executed directly on the bare hardware 1120 of the computer system 1000 (e.g., one or more processors 1004). Alternatively, a hypervisor or virtual machine monitor (VMM) 1130 can be inserted between the bare hardware 1120 and the OS 1110. In this configuration, the VMM 1130 acts as a software “buffer” or virtualization layer between the OS 1110 and the bare hardware 1120 of the computer system 1000.

[0226] The VMM 1130 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine includes a “guest” operating system (such as the OS 1110), and one or more applications (such as one or more applications 1102) designed to execute on the guest operating system. The VMM 1130 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0227] In some instances, the VMM 1130 can allow the guest operating system to run as if it were running directly on the bare hardware 1120 of the computer system 1000. In these instances, the same version of the guest operating system configured to execute directly on the bare hardware 1120 can also execute on the VMM 1130 without modification or reconfiguration. In other words, the VMM 1130 can provide full hardware and CPU virtualization to the guest operating system in some cases.

[0228] In other instances, the guest operating system can be specifically designed or configured to execute on the VMM 1130 for improved efficiency. In these instances, the guest operating system “is aware” that it is executing on the virtual machine monitor. In other words, the VMM 1130 can provide paravirtualization to the guest operating system in certain cases.

[0229] A computer system process includes the allocation of hardware processor time, as well as the allocation of memory (physical and / or virtual), the allocation of memory for storing instructions executed by the hardware processor, the allocation of memory for storing data generated by the execution of instructions by the hardware processor, and / or the allocation of hardware processor state (e.g., the contents of registers) between the allocation of hardware processor time when the computer system process is not running. A computer system process runs under the control of an operating system and can also run under the control of other programs that can be executed on the computer system.

[0230] Cloud computing

[0231] In general, the term "cloud computing" is used herein to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0232] A cloud computing environment (sometimes referred to as a cloud environment or a cloud) can be implemented in a variety of different ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the public. In contrast, a private cloud environment is generally only used by or within a single organization. A community cloud is intended to be shared by several organizations within a community; while a hybrid cloud includes two or more types of clouds (e.g., private, community, or public) that are bound together through data and application portability.

[0233] In general, cloud computing models enable some of those responsibilities that might previously have been provided by an organization's own information technology department to instead be delivered as service layers within a cloud environment for consumption (either within or outside the organization, depending on the public / private nature of the cloud) by consumers. Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure while the SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), where consumers can use software programming languages and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything under the runtime execution environment). Infrastructure as a Service (IaaS), where consumers can deploy and run any software applications, and / or provision processing, storage, networking, and other basic computing resources while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS), where consumers use database servers or database management systems running on cloud infrastructure while the DBaaS provider manages or controls the underlying cloud infrastructure, applications, and servers, including one or more database servers.

Claims

1. A method for accessing non-scalar data objects, comprising: storing one or more tables in a persistent form, the one or more tables including a plurality of columns, the plurality of columns including scalar columns and a column containing semi-structured data or unstructured text data; storing, within a memory compression unit (IMCU) stored in RAM, a subset of rows including the scalar columns and the column, the subset of rows containing a plurality of non-scalar data objects in the column of the subset of rows; wherein the plurality of non-scalar data objects includes one or both of semi-structured data or unstructured text data; wherein, within the IMCU, the scalar columns are stored in column-major format and the column is stored in a representation including a posting index that maps a plurality of tokens in the plurality of non-scalar data objects to token positions within the plurality of non-scalar data objects; maintaining transactional consistency between the IMCU and the scalar columns and the column stored in the one or more tables in the persistent form; receiving a request to execute a database statement that requires predicate evaluation for the column against a predicate; and in response to receiving the request, evaluating the predicate using the IMCU, wherein evaluating the predicate includes evaluating a first predicate condition of the predicate against the posting index; wherein the scalar columns are stored in column vectors within the IMCU; wherein the posting index includes a plurality of posting index entries, each posting index entry mapping a token to one or more token positions within the plurality of non-scalar data objects; and wherein each of the plurality of posting index entries includes a corresponding set of one or more lists, and each corresponding set of one or more lists includes: an index of the column vector that serves as an object reference to a corresponding non-scalar data object among the plurality of non-scalar data objects, and one or more token positions within the corresponding non-scalar data object; generating an incremental posting index that indexes a plurality of changed non-scalar data objects that have changed after loading the IMCU into the RAM; wherein the incremental posting index includes a plurality of incremental posting index entries, each incremental posting index entry mapping a token to one or more token positions within the plurality of changed non-scalar data objects; wherein an incremental posting index entry among the plurality of incremental posting index entries includes a plurality of incremental lists, each of the plurality of incremental lists including an index of the column vector as an object reference to a non-scalar data object among the plurality of non-scalar data objects; generating a refreshed version of the posting index, wherein generating the refreshed version of the posting index includes merging the incremental posting index with the posting index.

2. The method according to claim 1, Evaluating the predicate includes generating a first result vector representing an evaluation of the first predicate condition for the publication index, where generating the first result vector includes setting a bit corresponding to an index of the column vector of the non-scalar data object to which the publication index maps among the plurality of non-scalar data objects.

3. The method of claim 2, wherein evaluating the predicate includes: Before generating the first result vector, generating another result vector representing an evaluation of a second predicate condition for the scalar column, where the other result vector sets another bit corresponding to an index different from the index corresponding to the bit; wherein evaluating the first predicate condition includes foregoing a full evaluation of the first predicate condition for rows indexed to the different index.

4. The method of claim 2, further includes: Generating another result vector representing an evaluation of a second predicate condition for the scalar column; Combining the first result vector and the other result vector by performing an AND operation between the first result vector and the other result vector to generate a third result vector.

5. The method of claim 1, further includes: wherein evaluating the first predicate condition of the predicate for the publication index includes evaluating the first predicate condition for a persistent form of a non-scalar data object among the plurality of non-scalar data objects stored in the column.

6. The method of claim 5, wherein the first predicate condition requires a functional evaluation, and wherein evaluating the first predicate condition for the persistent form of the non-scalar data object includes generating a publication index for the non-scalar data object in response to determining that the first predicate condition requires a functional evaluation.

7. The method of claim 1, wherein the publication index is stored as a serialized hash table.

8. The method of claim 1, further includes: Generating an incremental publication index indexing changed non-scalar data objects having changes in persistent form not reflected in the publication index; wherein the incremental publication index includes a plurality of incremental publication index entries, each incremental publication index entry mapping a flag to one or more flag positions within the changed non-scalar data object in persistent form; wherein one incremental publication index entry among the plurality of incremental publication index entries includes a plurality of incremental lists, and each incremental list among the plurality of incremental lists includes an index of the column vector as an object reference to a non-scalar data object among the plurality of non-scalar data objects; wherein one incremental list among the plurality of incremental lists includes an index as an object reference to the changed non-scalar data object; wherein an existing publication index entry among the plurality of publication index entries includes an existing plurality of lists; wherein an existing list among the existing plurality of lists includes the index as an object reference to the changed non-scalar data object; wherein the publication index is stored as a serialized hash table; Generate another release index as a serialized hash table, where generating another release index includes: Generating new release index entries by performing a sort-merge operation at least between the plurality of delta lists and the existing plurality of lists, the sort-merge operation excluding the existing list that includes the one index as an object reference to the changed non-scalar data object.

9. The method according to claim 1, wherein the release index is stored as a serialized hash table including a plurality of serialized hash buckets, and at least one of the plurality of serialized hash buckets includes a set of release index entries delimited by delimiters among the plurality of release index entries.

10. One or more non-transitory computer-readable media storing one or more sequences of instructions that, when executed by one or more processors, cause the execution of the method according to any one of claims 1-9.

11. A device for accessing non-scalar data objects, comprising: One or more processors; and A memory coupled to the one or more processors and including instructions stored thereon that, when executed by the one or more processors, cause the execution of the method according to any one of claims 1-9.

12. A computer program product including instructions that, when executed by a computer, cause the computer to execute the method according to any one of claims 1-9.

Images