Efficient support JSON document segmentation update

Abstract

The invention relates to efficient support of segmented updating of JSON documents. The invention relates to fine-grained updating of JSON document segments by database statements which can update, delete and insert JSON document parts. In an embodiment, a computer receives a request specifying a modification of a JSON document stored in a persistent storage in a compressed format. The modification adds additional content to the JSON document and / or replaces the old value in the JSON document with a new value that is different in size than the old value. The modification is recorded in a change log. The change log is ultimately applied to the compressed format of the JSON document in the persistent storage device without completely rewriting the compressed format of the JSON document in the persistent storage device.

Description

[0001] This application is a divisional application of the invention patent application with the application number 202080072548.0, the application date of September 29, 2020, and the invention title of "Efficiently Supporting Piecemeal Updates of JSON Documents". Technical Field

[0002] The present disclosure relates to modifying JavaScript Object Notation (JSON) documents stored in a compressed format in a permanent storage device. The techniques herein accelerate the modification of the compressed format of JSON documents in a permanent storage device by avoiding a complete rewrite of the compressed format of the JSON documents in the permanent storage device. Background Art

[0003] As a concise semi-structured data format, JavaScript Object Notation (JSON) is widely popular for supporting schema-free agile style application development of JSON documents stored in a relational database management system (RDBMS) based on Structured Query Language (SQL) and / or a non-SQL style document object store. In such a data application model, JSON objects become the unit for document retrieval, query, and update to support online transaction processing (OLTP) of microservices. However, compared with updates in the relational model where the single update granularity can be at the per-column level, even when only a single field of the JSON document needs to be updated functionally, a key-value non-SQL style repository may have to update the JSON document as a whole.

[0004] When JSON documents are stored as JSON text, there are not many optimizations that can be performed at the storage level other than performing a complete document replacement, unless the required changes are very small such that the update engine can precisely match the JSON text parser character positions with the required changes. Such update optimizations for JSON text storage are not usually attempted because such interleaving of the JSON text parser and the update engine requires a large amount of design work and only provides a small improvement because precise text update size matching does not occur very often.

[0005] An industry-popular binary format is BSON from MongoDB. It can support limited partial updates (basically the same limitations as those applicable to JSON text) when the new content fully fits the original content. For example, since the new and old content fully fit, the scalar string value can be replaced from 'ZHEN' to 'CHEN' by a partial update; however, 'ZHEN' cannot be updated to 'ZHENG' by a partial update. The binary JSON design of MySQL also has similar limitations.

[0006] The file system implementation may be complex enough to support deletion and insertion in the middle of a file storing JSON documents, so binary partial updates may be feasible for JSON encodings such as BSON. However, such a complex file system implementation must internally support an additional bookkeeping layer that maps the logical byte offsets in the file to their physical offsets. This inevitably imposes an additional performance cost burden on the file system design.

[0007] Users have no choice but to use the JSON document object model (DOM) application programming interface (API) to programmatically traverse and update JSON documents. Although such partial updates work functionally, their efficient support at the storage level has not been fully addressed. Implementations by typical vendors end up completely replacing the JSON document even when logically only a small change to the JSON document is required. This becomes a performance issue when the size of the JSON document barely reaches megabytes or tens of kilobytes. From the perspective of the transaction log, a full JSON document update is particularly expensive because it requires a change log size proportional to the size of the updated full JSON document. The change log not only affects the performance of the data manipulation language (DML), but also affects replication performance and other functions that rely on the change log.

[0008] The approaches described in this section are approaches that can be taken, but not necessarily the approaches previously envisioned or taken. Therefore, unless otherwise stated, no assumption should be made that any method described in this section is eligible for prior art solely because it is included in this section. Brief Description of the Drawings

[0009] In the drawings:

[0010] Figure 1 is a block diagram of an example computer depicting accelerating the modification of a persistent JSON document by avoiding a full rewrite of the persistent JSON document;

[0011] Figure 2 is a flowchart of an example computer process depicting accelerating the modification of a persistent JSON document by avoiding a full rewrite of the persistent JSON document;

[0012] Figure 3 is a flowchart of an example computer process depicting applying a change log to a compressed format of a JSON document and avoiding a full rewrite of the compressed format;

[0013] Figure 4 is a flowchart of an example computer process depicting managing the lifecycle of a change log for a compressed format of a JSON document;

[0014] Figure 5 is a flowchart depicting an example computer process for storing a compressed format of a JSON document in a relational table of a database;

[0015] Figure 6 is a block diagram showing a computer system on which embodiments of the present invention can be implemented;

[0016] Figure 7 is a block diagram showing a basic software system that can be used to control the operation of a computing system. Detailed Description

[0017] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiments of the present disclosure. However, it will be apparent that the exemplary embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the exemplary embodiments.

[0018] General Overview

[0019] This document is a document object repository that provides fine-grained updates to multiple JavaScript Object Notation (JSON) document fragments via a declarative language application programming interface (API). Here, a relational database management system (RDBMS) that stores JSON documents also provides segmented updates to JSON documents through Structured Query Language (SQL) capabilities. Such SQL functions can generally update, delete, and insert a portion of a JSON document declaratively through SQL / JSON expressions and path languages. There can be multiple such SQL functions, each performing a JSON update function. For example, JSON_MERGEPATCH() and JSON_TRANSFORM() are SQL callable functions that can perform a series of changes declaratively with a single function call.

[0020] Compared to requiring a user to programmatically traverse and update a JSON document using a JSON document object model (DOM) API, using SQL functions for segmented JSON updates is more user-friendly and ergonomic. The ideal goal achieved here is that the change log size is proportional to the actual segmented changes to the JSON document. The accelerated access to the change log here improves the following performance: a) Data Manipulation Language (DML) operations, b) Replication, and c) Other features that rely on the change log.

[0021] When a JSON document is saved in binary format, by designing a modifiable binary format to balance the need to provide fast queries and enable partial updates in the binary format, there is a good chance to perform segmented updates efficiently. Oracle's OSON binary format performs partial updates in many cases where the new content does not fully fit into the original byte content of the OSON document. The method proposed in this paper is designed for binary formats like OSON to support many segmented updates at the physical level, such that the change log size is proportional to the actual segmented changes required for the JSON document. These methods can detect cases where a large number of changes (cumulative changes from the past and changes for performing the current update operation) are needed on OSON, such that the OSON document can reorganize itself autonomously by generating a new, more compact document when appropriate. The OSON design follows the logical concept that JSON is a tree model and thus provides jump navigation based on native tree offsets, enabling it to naturally support partial updates at the physical OSON byte level, as proposed in the related U.S. Patent 10,262,012. Additionally, in an embodiment, the OSON partial update design here only requires a standard file system, such as the Unix / POSIX file system API, that can seek specific offsets within a file to perform replacement of reserved content of a byte length or append bytes to the end of the file for growth. The underlying file system does not need to have the ability to delete bytes in the middle of the file to create holes or insert extra bytes in the middle of the file to cause the file to grow in the middle.

[0022] A significant technical difference from other solutions is that segmented OSON updates here are performed as much as possible without performing a full document update to reduce the transaction change log size and greatly improve update and replication performance. In a typical case, the number of OSON bytes being updated is proportional to the content being functionally updated, rather than to the size of the entire document. However, if the update is so complex that it distorts the OSON bytes too much, then the OSON document will perform a full document overwrite autonomously when appropriate. The OSON transaction subsystem of the RDBMS makes such a decision for each document at runtime based on the cumulative corresponding changes that have occurred in the past and the changes required for performing the current update operation. Thus, the OSON document lifecycle can include immediate automatic compression or use of intelligent scheduling.

[0023] When the document is small enough to be inlined into a binary large object (BLOB) within a row of a table in a relational database, there may be little benefit in performing a physical partial update. In such a case, the RDBMS may perform a full document replacement.

[0024] When the document is large, the two BLOB APIs used are somewhat similar to the standard UNIX / POSIX file system APIs: length-preserving byte content replacement, or appending bytes to the end. If in-place update is appropriate, the RDBMS performs length-preserving replacement of the BLOB. If in-place update is no longer appropriate, the redirect tombstone overwrites the current location with the new forwarding address of the newly inserted fragment appended to the end of the BLOB.

[0025] Some operations such as JSON array insertions use an exponential growth strategy to dynamically adjust the update behavior, which is used to reserve space to accommodate future in-place update requests. When the accumulated appended new fragments exceed a threshold percentage of the total size of the OSON document, the RDBMS compresses and completely replaces the OSON document.

[0026] Since the partial update semantics required on various JSON constructs (JSON object / member insertion, deletion, member value update, JSON scalar update) are already included in the RDBMS / OSON integration, the OSON bytes can be appropriately designed to accommodate these segmented update operations and control the performance of queries and updates without requesting additional bookkeeping from the underlying storage API.

[0027] In an embodiment, a computer receives a request to modify a JavaScript Object Notation (JSON) document stored in a persistent storage device in a compressed format. The modification adds additional content to the JSON document and / or replaces an old value in the JSON document with a new value of a different size than the old value. The modification is recorded in a change log. The change log is ultimately applied to the compressed format of the JSON document in the persistent storage device without completely rewriting the compressed format of the JSON document in the persistent storage device.

[0028] 1.0 Example Computer

[0029] Figure 1 is a block diagram depicting an example computer 100 in an embodiment. The computer 100 accelerates the modification of a JavaScript Object Notation (JSON) document 130 stored in a persistent storage device 110 in a compressed format 120 by avoiding completely rewriting the compressed format 120 in the persistent storage device 130. The computer 100 can be one or more of a rack server (e.g., blade), a personal computer, a mainframe, a virtual machine, or a smartphone.

[0030] The persistent storage device 110 is a non-volatile storage device such as a disk, flash memory, or network-attached storage device (NAS). The JSON document 130 contains semi-structured data such as scalars and aggregates such as arrays, objects containing name / value pairs, and / or nested arrays or objects such as arrays or objects within arrays or objects. For example, the JSON document 130 contains an old value 140, which can be a scalar or an aggregate value such as an array or an object. Here, an aggregate is a JSON array or a JSON object.

[0031] The compressed format 120 is a binary format different from the original text encoding of the JSON document 130. The compressed format 120 is a self-contained encoding of the JSON document 130 that: a) is not or is not fully encoded as text characters, b) contains metadata based on but not part of the text content of the JSON document 130, and c) contains metadata for navigating and / or decompressing the content of the JSON document 130. In an embodiment, the metadata (b) and (c) are the same metadata. In an embodiment, field and element names and values can be replaced with tokens, each token being mapped to a value by a dictionary such as using dictionary encoding. The dictionary can be stored in the compressed format 120 of the JSON document 130 or elsewhere, such as in a database or a file, so that the same dictionary can be used to encode multiple JSON documents. In an embodiment, the compressed format 120 is the Oracle OSON binary format for JSON. The techniques for implementing the compressed format 120 are set forth in the related U.S. Patent 10,262,012.

[0032] The life cycle of the compressed format 120 of the JSON document 130 can include content mutations. For example, the computer 100 can receive or generate requests such as 151-152 that specify corresponding modifications 161-162 to change the content stored within the JSON document 130. According to an embodiment, the request 151 is represented as a data manipulation language (DML), such as a create read update delete (CRUD) statement or a query by example (QBE). The autonomous life cycle of the compressed format 120 of the JSON document 130 will be presented later herein.

[0033] For example, the request 151 can be a structured query language (SQL) DML statement such as UPDATE. In an embodiment, the request 151 is received via an open database connection (ODBC). For example, the computer 100 can host a database management system (DBMS) that stores the JSON document 130 in a persistent database, as discussed later herein. In an embodiment, the request 151 is a hypertext transfer protocol (HTTP) request, such as: GET, POST, common gateway interface (CGI), and / or representational state transfer (REST).

[0034] Modification 162 specifies that additional content 190 should be added to JSON document 130. The additional content 190 is a JSON value such as a scalar or an aggregate. Modification 162 can implicitly or explicitly specify that the additional content 190 should be appended or inserted into JSON document 130. For example, modification 162 can specify: a) an array within JSON document 130 to which the additional content 190 should be appended, b) an offset for inserting an additional array element within that array, or c) a key (i.e., field name) for adding the additional content 190 as a value within a JSON object.

[0035] Computer 100 does not need to directly and immediately apply modifications 161-162 to the compressed format 120 of JSON document 130 in persistent storage 110. Instead, details of modifications 161-162 such as new value 180 and additional content 190 can be appended to change log 170 to defer application to the compressed format 120 of JSON document 130 in persistent storage 110. According to an embodiment, the change log 170 can be stored in volatile memory and / or persistent storage 110. The format and operation of the change log 170 will be discussed later in this document.

[0036] In any case, the change log 170 is ultimately applied to the compressed format 120 of JSON document 130 in persistent storage 110, including modifications 161-162. In an embodiment, requests 151 and / or 152 can occur within a database transaction such as an atomic consistent isolated durable (ACID) transaction. In an embodiment, the transaction is complete when details of modifications 161 and / or 162 are recorded in the change log 170. In an embodiment, conversely, the transaction is complete when the change log 170 is applied to the compressed format 120 of JSON document 130 in persistent storage 110.

[0037] Thus, according to an embodiment, a request to commit a transaction may or may not cause the change log 170 to be immediately applied to the compressed format 120 of JSON document 130 in persistent storage 110. In the case where request 151 does not occur and the change log 170 only reflects modification 162, then after applying the change log 170, the compressed format 120 of JSON document 130 in persistent storage 110 can contain both the old value 140 and the additional content 190.

[0038] Modification 161 specifies a reassignment that writes a new value 180 to the JSON document 130. The new value 180 overwrites or otherwise replaces the old value 140 within the compressed format 120 of the JSON document 130. For example, modification 161 can specify: a) the offset of an array within the JSON document 130 at which an existing array element is replaced with the new value 180, or b) an existing key (i.e., field name) of a JSON object within the JSON document 130 that is to be reassigned to the new value 180. For example, after fulfilling the request 151 by applying the change log 170: a) the JSON document 130 logically contains the new value 180 but not the old value 140, and b) the compressed format 120 contains the new value 180 and may or may not physically contain the old value 140. In the example, requests 151-152 can be the same request that specifies both the new value 180 and the additional content 190.

[0039] Other methods may have awkward limitations, which can be: a) unable to fulfill request 151 or 152, or b) unable to fulfill request 151 or 152 efficiently. For example, other methods may require that values 140 and 180 have the same size. Other methods can fulfill request 151 or 152 by completely rewriting the compressed format 120 of the JSON document 130 in the permanent storage device 110, which may waste input / output (I / O) bandwidth.

[0040] The nature of non-volatile technologies may degrade other methods as follows. For example, when the compressed format 120 of the JSON document 130 is large enough to require multiple seeks to write, the disk may cause additional latency for track switching and / or rotation. Similarly, writing to flash is slower than reading from flash. Other methods can shorten the expected life of flash, which may permanently fail after too many writes to the same area in flash.

[0041] The techniques here avoid these technical problems by not completely rewriting the compressed format 120 of the JSON document 130 in the permanent storage device 110 to fulfill request 151. For example, requests 151-152 can be implemented by overwriting or appending content to the compressed format 120 of the JSON document 130 in the permanent storage device 110. Thus, based on the change log 170, the computer 100 performs only a partial update to the compressed format 120 of the JSON document 130 in the permanent storage device 110 to fulfill requests 151-152.

[0042] 2.0 Accelerated Modification Without Rewriting the JSON Document

[0043] Figure 2is a flowchart depicting an example process that can be executed by a computer 100 to accelerate modifications to a JSON document 130 stored in a persistent storage device 110 in a compressed format 120 by avoiding a complete rewrite of the compressed format 120 in the persistent storage device 110. Refer to Figure 1 Discussion Figure 2 。

[0044] Step 202 receives a request to modify the JSON document 130, such as 151 or 152. Example formats, semantics, and transmissions of requests 151 - 152 were given earlier in this document. For example, the computer 100 can generate or receive a request 151 on a socket for network or inter - process communication (IPC), such as using a message queue, shared memory, or an operating system (OS) pipe.

[0045] For example, a modification 161 can specify replacing an old value 140 in the JSON document 130 with a new value 180 that has a different size than the old value 140. For example, the new value 180 can be represented by more or fewer bytes than the old value 140. The size of a value can depend on the encoding, especially binary encoding as shown below.

[0046] For example, the text values "123" and "0.3" contain the same number of characters but may have different sizes when encoded in binary form as strongly - typed numbers. For example, one number can be encoded as an integer while the other can be encoded as a floating point. Similarly, different integers can be encoded with different widths (such as different numbers of bytes or machine words). Thus, standard JSON text encoding may or may not indicate the value size in the compressed format 120.

[0047] Other semantic encodings can include data types such as times, dates, and boolean values. For example, "true" can be encoded as a single bit, byte, or short, all of which are smaller than the text. Dictionary encoding or run - length encoding (RLE) are compressions that can affect value size. Character encodings (such as ASCII or Unicode) may affect value size. Strings represented with a length (expresslength) or null terminator may affect value size.

[0048] In another example, a modification 162 specifies adding additional content 190 to the JSON document 130. For example, a modification 162 can result in inserting or appending a new array element or a new JSON object field, as discussed earlier in this document. In any case, step 204 records details of the modification 161 and / or 162, such as the new value 180 and / or the additional content 190, in a change log urchased by the computer 100 can be stored in the persistent storage device 110 in a compressed format 120.

[0049] Finally, step 206 applies the change log 170 to the compressed format 120 of the JSON document 130 in the persistent storage device 110 without completely rewriting the compressed format 120 of the JSON document 130 in the persistent storage device 110. Techniques for writing the compressed format 120 into the persistent storage device 110 based on the change log 170 will be discussed later in this article.

[0050] 3.0 Applying the Change Log

[0051] Figure 3 is a flowchart depicting an example process that can be performed by the computer 100 to apply the change log 170 to the JSON document 130 stored in the compressed format 120 in the persistent storage device 110 so as to avoid completely rewriting the compressed format 120 in the persistent storage device 130. Refer to Figure 1 Discussion Figure 3 。

[0052] As explained earlier in this article, the modifications 161-162 can be specified by the same request instead of separate requests 151-152 as shown previously. For demonstration, Figure 3 a complex request that specifies the modifications 161-162 and other modifications needs to be processed. Although there are many modifications to this request, the various steps of Figure 3 can be executed separately for one, some, or all of the modifications to this request. Therefore, a request that specifies fewer modifications can execute Figure 3 fewer steps of

[0053] Depending on the scenario, there may be various reasons to actually defer writing the modifications to the compressed format 120 of the JSON document 130 in the persistent storage device 110. For example, ACID transactions require isolation, in which the (multiple) modifications of one client should not be visible to other clients until the transaction is committed. In other examples, deferring the write to the persistent storage device 110 increases throughput.

[0054] For example, deferring the write can facilitate combining (i.e., aggregating) multiple modifications into a single modification. Similarly, deferring the write can reduce storage latency by reducing seek or rotational latency, for example, by sorting the modifications into an order that increases the size of sequential writes across many disk blocks. In an embodiment, reducing seek saves energy. When request processing such as OLTP occurs in the foreground, deferring the write can occur in the background such as with lower priority and / or being preemptible.

[0055] These are the reasons why step 301 delays writing some or all of the compressed format 120 of the JSON document 130 to the permanent storage device 110. For example, step 301 can record modifications 161-162 to the change log 170 for later application to the compressed format 120 of the JSON document 130 in the permanent storage device 110. Depending on the embodiment, for example when the JSON document 130 is encapsulated with a Document Object Model (DOM), and regardless of whether the persistent write is delayed, the modifications 161-162 can be immediately: a) applied to a copy of the JSON document 130 in volatile memory in the same format as the compressed format 120 or in a different format such as text, and / or b) recorded in a change log 170 in a volatile or non-volatile storage device.

[0056] Ultimately, the change log 170 should be applied to the compressed format 120 of the JSON document 130 in the permanent storage device 110, at which time the change log 170 may have accumulated many unapplied modifications to the JSON document 130 from the same or different requests in the same transaction. Steps 302-306 illustrate activities that can occur separately to apply various modifications to the compressed format 120 of the JSON document 130 in the permanent storage device 110. For example, step 302 can be repeated for each of the many modifications, and each of these modifications may or may not be processed by Figure 2 other steps in.

[0057] Depending on the scenario or embodiment, such as when the new value 180 is not larger in size than the old value 140, fulfilling the modification 161 may require physically overwriting the old value 140 with the new value 180 in the compressed format 120 of the JSON document 130 in the permanent storage device 110 during step 302. For example, string truncation may require shortening the representation length or setting characters to null.

[0058] Similarly, array truncation may require reducing the representation length or setting array elements to null or storing a tombstone value indicating deletion. Likewise, JSON object truncation may require setting fields to null or tombstone values. Tombstones will be discussed later in this document.

[0059] Step 303 demonstrates appending various objects to the compressed format 120 of the JSON document 130 in the permanent storage device 110, and which object to append can depend on the scenario as follows. Any one of additional content 190, new value 180, or unused space for future use can be appended.

[0060] In an embodiment, and whether the additional content 190 adds an element to an existing aggregation or the new value 180 replaces an existing aggregation element such as the old value 140, the modification may require a copy-on-write of the entire aggregation. For example, step 303 may apply the modification of the aggregation to the compressed format 120 of the JSON document 130 in the persistent storage 110 by appending a copy of the aggregation reflecting the modification to the compressed format 120 of the JSON document 130 in the persistent storage 110. This repositioning of the previous content will be discussed further later. In an embodiment, the persistent storage 110 is append-only (i.e., write-once).

[0061] After various triggers, step 303 may append unused space to the JSON document 130, which is reserved for growth and may or may not be reserved for a specific aggregation. In an embodiment, the aggregation has a linked list of segments such that: a) each segment has a pointer or offset to the next segment, and b) the aggregation may reactively and / or proactively grow new empty segments, for example, according to the autonomous lifecycle of the compressed format 120 of the JSON document 130 in the persistent storage 110 as discussed later herein. In an embodiment, segments of the same aggregation have the same size, for example, the number of bytes or capacity to store the same number of elements.

[0062] There is a risk that the unused space within the compressed format 120 of the JSON document 130 in the persistent storage 110 may never be needed, in which case the unused space is wasted. Space waste can be avoided by appending small unused space. However, sometimes the growth of the JSON document 130 actually requires a lot of space.

[0063] If a lot of unused space is needed, but the unused space is provided in small segments, many segments are required, which may cause many allocation delays, such as due to repeated I / O and / or system calls, both of which have high latency. Therefore, there may be a performance tension between: a) the delay of adding many small segments, and b) the space wasted by adding large segments instead. Step 304 balances these design forces by appending segments with increasing sizes for the same aggregation. In an embodiment, the size grows exponentially.

[0064] A tombstone is a marker that overwrites the stale value in place. Therefore, a tombstone should not be larger than the stale value. In an embodiment, there are two types of tombstones: deletion tombstones and redirect tombstones.

[0065] If the value is stale due to deletion, the deletion tombstone only needs to be recognizable. Otherwise, the value becomes stale due to re - assignment, for example when the stale value is the old value 140 replaced by the new value 180, especially when the old value 140 and the new value 180 have different sizes, making it impossible for the new value 180 to completely overwrite the old value 140 in - place. In such a case, during step 305A, the old value 140 can be rewritten in - place with a redirect tombstone that contains a pointer or offset to the new value 180.

[0066] Therefore, during step 305B, what address or offset of the new value 180 appended within the compressed format 120 of the JSON document 130 in the persistent storage device 110 should be available for step 305A which writes the pointer. The value of the pointer as an address or offset can be calculated before steps 305A - B. Thus, in one embodiment, steps 305A - B occur in parallel as shown. In one embodiment, steps 305A - B occur sequentially, for example when steps 305A - B write different sectors to the same sector cluster in the same disk track to reduce track seeking. In other words, at least steps 305A - B can be combined into the same single I / O write.

[0067] As described above, step 305A rewrites the old value 140 in - place with a redirect tombstone. In an embodiment, the change log 170 is completely truncated after being fully applied, and subsequent modifications to the JSON document 130 can then be appended to the change log 170. Thus, the change log 170 may experience repeated flushes that require applying and then truncating the change log 170.

[0068] Any such flush provides a barrier across which the merging of pre - write modifications does not occur. For example, before the flush, modifications to adjacent elements of an aggregate can be merged. Modifications resulting from after the flush are not merged with the modifications that have already been flushed.

[0069] In an embodiment, the redirect tombstone can provide a pointer for a linked list that chains progressive revisions / re - assignments of the value of the same data item (such as a scalar or an aggregate) in a daisy - chain form, such that reading the linked list to find the current value of the same data item may require linearly reading the redirect tombstones to traverse the entire linked list. In an embodiment, instead, constant - time retrieval of the current value is provided by neither daisy - chaining nor retaining multiple redirect tombstones for the same data item, because step 305C overwrites the previous redirect tombstone in - place with a new redirect tombstone.

[0070] In an embodiment, the size of the redirect tombstone is not greater than the size of the delete tombstone, such that the delete tombstone can overwrite the redirect tombstone in-place. In an embodiment, the size of the delete tombstone is not greater than the size of the redirect tombstone, such that the redirect tombstone can overwrite the delete tombstone in-place. In an embodiment, the two types of tombstones have the same size, such that either type of tombstone can overwrite the other tombstone in-place.

[0071] In an embodiment, a single bit in each of the two types of tombstones indicates that the item is a tombstone. In an embodiment, another single bit indicates whether the tombstone is a delete tombstone or a redirect tombstone.

[0072] The compressed format 120 of the JSON document 130 in the persistent storage device 110 can be somewhat sparse, having holes (plural) of unused bytes at various locations. For example, an aggregate (e.g., an array) can store some elements but have unused internal space for more elements. Instead of increasing the size of the JSON document 130, step 306 can fill the unused space of an already stored collection with additional content 190 or a new value 180.

[0073] For example, steps 303-304 as described above can append unused space to the compressed format 120 of the JSON document 130 reserved for a particular aggregate in the persistent storage device 110. When unrelated values are later appended to the compressed format 120 of the JSON document 130 in the persistent storage device 110, the previously appended unused space can become a hole between two unrelated values that may be located within the compressed format 120 of the JSON document 13 in the persistent storage device 110.

[0074] Step 306 can fill the hole with elements of the aggregate, e.g., when the hole has unused space reserved for the aggregate. Thus, growth of the aggregate does not need to result in growth of the compressed format 120 of the JSON document 130 in the persistent storage device 110. For example, writing to an existing hole may be faster than allocating and writing to new space.

[0075] 4.0 Autonomous Lifecycle of Components

[0076] Figure 4 is a flowchart of an example process depicting the lifecycle of a change log 170 that can be executed by the computer 100 to manage changes to the compressed format 120 of the JSON document 130. Refer to Figure 1 Discussion Figure 4 。

[0077] Modifications 161-162 can be limited in scope such that most of the JSON document 130 is not affected by modifications 161-162. In an embodiment, according to the command software design pattern, modification 161 or 162 is appended as a command to the change log 170. For example, the change log 170 can store change commands, each change command specifying a portion of the JSON document 130, such as fields and / or new scalar or aggregate values to be added, reassigned, or deleted for that portion.

[0078] For example, the change log 170 can reside in a volatile buffer and / or a volatile or non-volatile queue (such as a file) such that the latest change is appended to one end of the queue, and when the change reaches the other end of the queue, the change is ultimately physically applied to the compressed format 120 of the JSON document 130 in the permanent storage device 110 and removed from the queue in a first-in, first-out (FIFO) order. Instead of appending the entire revised JSON document 130 or the compressed format 120 of the JSON document 130 to the change log 170, step 401 appends only the change command or other change representation to the change log 170. Thus, the change log 170 is a compact specification of cumulative pending changes, the cumulative pending changes being represented as change commands that are applied to the compressed format 120 of the JSON document 130 in the permanent storage device 110 when dequeued.

[0079] The change log 170 can have its own life cycle that includes transitions such as accumulating modifications and flushing by applying the modifications to the compressed format 120 of the JSON document 130 in the permanent storage device 110. Accumulation and flushing can be decoupled. For example, flushing can occur in the background when autonomously triggered and requires an activity that is autonomously determined. In other words, the change log 170 can have its own autonomous life cycle that can sometimes be more or less independent of the client requests to modify the JSON document 130.

[0080] For example, two values can be consecutive, such as aggregate elements of the compressed format 120 of the JSON document 130 in the permanent storage device 110. As explained earlier herein, modifications to consecutively stored values can be merged into a single modification, such as during step 402. For example, when a modification is appended to the change log 170, the modification can be immediately merged with another modification that the change log 170 already contains.

[0081] Because the change log 170 has an autonomous life cycle, such an eager merge may not be required. For example, the change log 170 can have its own computing thread that merges the changes in the change log 170 at any time in the background or at any time before the change log 170 is applied to the compressed format 120 of the JSON document 130 in the permanent storage device 110.

[0082] In an embodiment, the change log 170 is replicated for use in a standby database for high availability (HA), such as for failover from a primary database to a standby database. For example, dequeuing changes from the change log may require applying the changes to the primary database and sending the same changes to another machine for eventual application to the standby database.

[0083] Background merging of modifications to sequentially stored elements is an example of the autonomous activity of the change log 170. Even when the elements are not sequential, modifications to those elements can be written to different sectors within the same cluster of sectors on the same disk track to reduce track seeking, which may require steps 403A - 403C for sequentially or non-sequentially stored elements.

[0084] Step 403A associates one or two offsets with each modification, for example when each modification is appended to the change log 170, which can facilitate merging and clustering as follows. The offset of the modification can be included in the modification in the change log 170, such as the offset of the old value 140 and / or the offset of the new value 180.

[0085] Steps 403B - C can occur automatically at a later time without the need for step 403A, such as in the background. Step 403B sorts the modifications based on the storage offset of their new values. After the modifications are sorted, clustering and / or identifying the modifications for merging becomes easier.

[0086] Such sorting can: a) occur incrementally according to sorted insertions when each modification is logged, such that the change log 170 is always sorted, such as when the change log 170 is a priority queue, b) occur in the background, such as during non-peak demand periods, such that the change log 170 can automatically change from unsorted to sorted, or c) occur just when the change log 170 is finally applied to the compressed format 120 of the JSON document 130 in the permanent storage device 110. Regardless of how and when the change log 170 is sorted, step 403C applies the change log 170 to the compressed format 120 of the JSON document 130 in the permanent storage device 110 based on that sorting.

[0087] As explained earlier in this document, the DOM can encapsulate the JSON document 130 to provide a convenient application programming interface (API) to the client. According to an embodiment, the DOM can be integrated with the following: a change log 170, a compressed format 120 of the JSON document 130 in the persistent storage 110, and / or a representation of the JSON document 130 in the volatile memory (which may or may not be the same as the compressed format 120). For example, the DOM can detect whether the JSON document 130 is cached in the volatile memory.

[0088] According to an embodiment, the DOM itself can be stateful and / or have its own autonomous lifecycle, such as having a background thread and / or an asynchronous trigger. Steps 404 - 405 show activities that the DOM can autonomously decide to perform or not perform.

[0089] The DOM can perform steps 404A - B as shown simultaneously, for example, to implement the same modification. Step 404A records the modification in the change log 170. Step 404B applies the same modification to the representation of the JSON document 130 in the volatile memory, which may or may not be the same as the compressed format 120.

[0090] Steps 405A - B are mutually exclusive. For acceleration during step 405a, the DOM can record the modification in the change log 170 without applying the modification to the representation of the JSON document 130 in the volatile memory. For example, the DOM can invalidate the representation of the JSON document 130 in the volatile memory.

[0091] In cases such as when the database connection is configured to automatically and separately commit each data manipulation language (DML) statement such as SQL in its own transaction, the DOM can bypass the change log 170 or otherwise operate without the change log 170. Step 405B applies the modification to the compressed format 120 of the JSON document 130 in the persistent storage 110 without recording the modification in the change log 170.

[0092] The compressed format 120 of the JSON document 130 in the permanent storage device 110 can also have its own autonomous life cycle, which can sometimes be more or less independent of the change log 170, the DOM, and client requests. For example, when the free disk space is below a threshold, the computer 100 can autonomously compress the compressed format 120 of the JSON document 130 in the permanent storage device 110 to reduce the size of the compressed format 120 of the JSON document 130 in the permanent storage device 110. For example, holes with unused space can be removed, and / or tombstones can be replaced with actual values. Such compression may require completely rewriting the compressed format 120 of the JSON document 130 in the permanent storage device 110.

[0093] 5.0 Database Configuration

[0094] Prior to rendering Figure 5 Before, the following database discussion shows the database configurability that database integration may or may not require. As discussed earlier in this document, the permanent storage device 110 can contain a relational database that contains the compressed format 120 of the JSON document 130. Table 1 below lists some or all of the example database configurations 801-804 that the computer 100 can implement.

[0095] Database Configuration Meaning 801 The request to modify the persistent JSON document is an SQL statement 802 The persistent JSON document is stored in a relational table 803 The persistent JSON document is stored in the LOB column of a relational table 804 The persistent JSON document is stored out of line in a relational table

[0096] The compressed format 120 of the JSON document 130 can be stored in a database in the permanent storage device 110 and / or volatile memory. For example, the computer 100 can host a database management system (DBMS) that contains a database, and the database contains the compressed format 120 of the JSON document 130. In database configuration 801, requests 151-152 are SQL statements such as UPDATE. In predicates and / or assignments, the SQL statements can include expression languages such as JSON, a subset of JavaScript, extensible markup language (XML) paths (XPath), or regular expressions. The expression language can navigate, select, filter, and / or assign the content within one or more permanent JSON documents, such as the compressed format 120 of the JSON document 130.

[0097] The DBMS here can understand a JSON expression language embedded in a native DML language such as SQL. A Turing-complete example JSON expression language for finding, navigating, reading, and writing collections and parts of JSON documents stored in relational tables embedded in SQL was proposed in related U.S. Patent Application 16 / 703,499. The SQL:2016 standard and the related non-patent literature (NPL) "Technical Report ISO / IEC TR 19075-6" provide sufficient integration of JSON into SQL, both of which define and standardize built-in functions for converting JSON to relational data or otherwise accessing JSON data from SQL, including functions such as JSON_TABLE and JSON_EXISTS explained therein.

[0098] In database configuration 802, the compressed format 120 of the JSON document 130 is stored in a relational table in the database. In an embodiment, such as in corresponding columns, the relational table may contain zero or more persistent JSON documents in each table row.

[0099] In database configuration 803, the compressed format 120 of the JSON document 130 is stored as a large object (LOB) in a column of the relational table. The encoding of the LOB may or may not depend on the data type of the column. The LOB can be a binary LOB (BLOB) or a character LOB (CLOB). The LOB will be discussed later.

[0100] In database configuration 804, for the rows of the relational table, the JSON document 130 is stored in an out-of-line manner, outside of the row itself. For example, the relational table may contain columns stored separately from other columns of the same relational table. In an embodiment, regardless of whether the relational table has row-major or column-major storage: a) all columns are stored in database blocks, b) for the same or different table rows, many columns of the same relational table are stored in the same database block, except c) the JSON column of the JSON document 130 and / or the compressed format 120 are stored in separate (multiple) database blocks. In an embodiment, the JSON column of the JSON document 130 and / or the compressed format 120 are stored in separate files.

[0101] 6.0 Database Integration

[0102] Figure 5 is a flowchart depicting an example process that can be executed by a computer 100 to store the compressed format 120 of the JSON document 130 in a relational table of a database. Refer to Figure 1 Discussion Figure 5 。

[0103] As discussed previously herein, the compressed format 120 of the JSON document 130 can be stored inline within a row of a relational table along with other columns, such as in a row-major format in a volatile and / or non-volatile storage device. The compressed format 120 of the JSON document 130 can alternatively be stored out-of-line, outside of the table row. Inline memory can accelerate access and improve efficiency by increasing data locality.

[0104] However, inline storage may be of limited capacity and is considered as follows. Step 502 detects whether the compressed format 120 of the JSON document 130 has been stored inline within a row of the relational table. For example, the relational schema can indicate that certain JSON columns are always stored inline or not stored inline. In an embodiment, some rows store the same JSON column inline while other rows of the same JSON column do not.

[0105] In an embodiment, each row contains its own metadata that indicates whether the JSON column is stored inline. In embodiments, the indicating metadata is stored in the header metadata of the row or adjacent to the inline persistent JSON document. In an embodiment, the indicating metadata is stored inline or as a single bit in a bitmap for multiple rows.

[0106] Step 504 calculates the size that the compressed format 120 of the JSON document 130 will have if the current request to modify the JSON document 130 is honored. In other words, during the execution of the modification request, the newly increased or decreased size of the compressed format 120 of the JSON document 130 is calculated.

[0107] According to step 506, steps 508A-B are mutually exclusive. When step 506 detects that the new size of the compressed format 120 of the JSON document 130 is small enough to store the compressed format 120 of the JSON document 130 inline, the table row itself provides storage for the write request. If the compressed format 120 of the JSON document 130 is already inline, in-place partial updates may occur in some scenarios. In other scenarios, step 508A completely inlines and rewrites the compressed format 120 of the JSON document 130 regardless of whether the compressed format 120 of the JSON document 130 is already inline.

[0108] Whether in-place rewrite or append, inlining does not always result in a complete rewrite, for example for compaction. For example, redirect tombstones pointing to other locations within the table row space may occur. However, due to overflow, inlining is more likely to result in compression.

[0109] Overflow occurs when the compressed format 120 of the JSON document 130 has been inlined but the table row capacity is insufficient to accommodate the current write request. In some cases, inlining compression can solve the overflow problem. In some cases, the overflow is solved by moving only a part of the compressed format 120 of the JSON document 130 outside the table row. For example, the redirected tombstone of the compressed format 120 of the JSON document 130 can reside within the row but point outside the table row in an out-of-line manner after the overflow is solved. In an embodiment, when the JSON document 130 is too large to be inlined, even the initially inserted table row contains a redirected tombstone. Otherwise, step 508B completely rewrites the JSON document 130 outside the table row.

[0110] As described above, the compressed format 120 of the JSON document 130 and its DOM can have an autonomous life cycle. For example, whether autonomously or not, the computer 100 can detect that compaction of the compressed format 120 of the JSON document 130 stored outside the table row will save enough space to enable the compressed format 120 of the JSON document 130 to fit within the table row in an inlined manner. In this case, step 508A occurs, and the compressed format 120 of the JSON document 130 or its DOM autonomously deallocates the out-of-line storage that is no longer needed due to inlining.

[0111] In an embodiment, the DOM is integrated with the LOB API, and the LOB API manages the storage of the LOB outside the table row. In an embodiment, the change log 170 contains change commands that are corresponding calls to the LOB API such as described later herein.

[0112] 7.0 Rewrite Threshold

[0113] As discussed earlier herein, the JSON document 130 can be wrapped by a DOM, and the DOM can switch between alternative storages of the compressed format 120 of the JSON document 130, such as using hierarchical storage such as a memory cache. In an embodiment, the DOM fulfills read requests by accessing storage layers in the following order: 1) the change log 170, 2) the cache, and 3) the permanent storage device 110. For example, a request to read two fields can be fulfilled by reading one field from the change log 170 and reading the other field by: a) detecting that the JSON document 130 is not cached, such as in the case of a cache miss, and b) reading some or all of the compressed format 120 of the JSON document 130 in the permanent storage device 110.

[0114] Partial updates to the compressed format 120 of the JSON document 130 may invalidate previous values and / or over-allocate unnecessary space within the compressed format 120 of the JSON document 130. In any case, the compressed format 120 of the JSON document 130 may accumulate holes of wasted space, which may reduce efficiency.

[0115] Regardless of whether space is wasted, multiple partial updates to the same or different JSON fields add (a) level(s) of indirection to field access via redirected tombstones and / or the change log 170, which reduces efficiency because traversing pointers increases computation and latency, especially when reading the change log 170 on the critical path of an operation. Such complications as indirection or reading the change log are examples of data storage fragmentation, which may reduce efficiency in some cases.

[0116] In embodiments, fragmentation is reduced or eliminated by applying the change log 170 and / or compacting the JSON document 130 in volatile and / or non-volatile storage. Compacting requires partially or fully rewriting the compressed format 120 of the JSON document 130 as if writing the original version of a new JSON document in the compressed format 120. Through overwriting, the partial or full rewrite can be done in-place. Compression may or may not be done according to the autonomous life cycle of components such as the compressed format 120 of the JSON document 130, its DOM, or the change log 170.

[0117] Conversely, the entire rewrite can be done by logically replicating the compressed format 120 of the JSON document 130, and for some parts of the compressed format 120, an exact physical replication is not required. For example, multiple values that are not stored contiguously within the compressed format 120 of the JSON document 130 can become contiguously stored through logical replication. In embodiments, fields within a JSON object can be reordered during replication.

[0118] In some embodiments, compacting occurs unconditionally at certain life cycle transitions of the compressed format 120 of the JSON document 130, whether autonomous or non-autonomous. For example, compacting should precede certain archival or online analytical processing (OLAP) scenarios. Online transaction processing (OLTP) may compact when committing a transaction. For example, during cache write-back, copying or moving the compressed format 120 of the JSON document 130 from a faster storage tier to a slower storage tier may result in compacting.

[0119] When any of various rewrite thresholds is exceeded, compaction of the compressed format 120 of the JSON document 130 may occur. For example, depending on the embodiment or scenario, sometimes reading, writing, or both of the compressed format 120 of the JSON document 130 may cause immediate or deferred compaction in the background or foreground. In an embodiment, the numeric threshold is manually adjustable, automatically optimizable, and / or trainable by machine learning (ML). In any case, the threshold setting should avoid the extremes of compaction being too frequent or too infrequent. In an embodiment, compaction is a cost-based decision. The performance cost of compression can be amortized over many future accesses to the compressed format 120 of the JSON document 130 and may depend more or less on the prediction of future accesses.

[0120] The following Table 2 columns enumerate example rewrite thresholds 811 - 815 that trigger compression, and some or all of which may be implemented by the computer 100.

[0121]

[0122] In an embodiment, the compressed format 120 of the JSON document 130 has a last compressed size, which is the amount of bytes or data chunks that the compressed format 120 of the JSON document 130 had at the last compression or initial filling. Similarly, the cumulative additional size is the amount of content appended to the compressed format 120 of the JSON document 130 since the last occurrence of the compact size. The rewrite threshold 811 can be the ratio or difference between the last compact size and the cumulative additional size.

[0123] In an embodiment, regardless of the last compact size or ratio or difference, the cumulative additional size itself is compared to the rewrite threshold 812. Embodiments can have multiple rewrite thresholds, such as some or all of 811 - 815. In an embodiment, compaction occurs when any one of the rewrite thresholds is exceeded. In an embodiment, an additional threshold specifies how many different rewrite thresholds must be exceeded simultaneously to cause compaction.

[0124] The rewrite threshold 813 is compared to the amount of write requests to the compressed format 120 of the JSON document 130 since the last compact size occurred, regardless of how many bytes each write request has and regardless of whether the write occurs in-place or by appending. The rewrite threshold 814 only counts write requests that: a) increase the size of the JSON document 130, or b) require appending. In some embodiments or scenarios, (a) and (b) are equivalent conditions that occur simultaneously.

[0125] Rewrite threshold 815 checks the duration elapsed since the size of the compressed format 120 of the JSON document 130 first increased or last increased after the last compaction size occurred. According to an embodiment, the time duration can be checked when accessing the compressed format 120 of the JSON document 130 or via a background timer.

[0126] When the compressed format 120 of the JSON document 130 is compacted, the bookkeeping for many, most, or all rewrite thresholds is reset. Each persistent JSON document is used separately for each rewrite threshold, so each persistent JSON document is triggered separately for compaction.

[0127] 8.0 Exemplary Embodiments

[0128] The following exemplary embodiments can be based on the embodiments presented earlier herein. This exemplary embodiment shows example implementation choices. The requirements of this exemplary embodiment are not necessarily the requirements of the embodiments presented earlier herein.

[0129] In this exemplary embodiment, OSON is included, which is an efficient and optimized binary format for JSON documents presented in the related U.S. Patent 10,262,012. OSON can appear in volatile and / or non-volatile storage and / or transmission, such as in a communication network. OSON can be wrapped by the DOM API discussed earlier herein.

[0130] In this exemplary embodiment, a relational DBMS (RDBMS) such as Oracle is included. The RDBMS can provide: a) row-major and / or column-major storage of relational tables in volatile and / or non-volatile storage devices, b) volatile cache and / or buffer management such as in a Memory Compression Unit (IMCU), c) Atomic Consistent Isolation Durable (ACID) transactions, d) SQL including JSON or JavaScript expressions, e) in-line or out-of-line LOB storage of table rows, and / or f) indexing of tables and / or JSON content. For example, a relational join may associate a scalar column in one table with a scalar field in a JSON document in a JSON column of another table. The integration of JSON, SQL, and IMCU in an RDBMS is introduced in the related U.S. Patent Application XX / XXX,XXX.

[0131] 8.1 SQL / JSON Query Compiler Support

[0132] The included RDBMS includes a query compiler that supports the JSON_TRANSFORM() and LOB ASSIGNMENT operators, as follows. The following is an example DML update query for the table JSON_COLLECTION with a JSON type column JCOL. The update query updates the JSON_COLLECTION table to assign a value to the JCOL column.

[0133] UPDATE JSON_COLLECTION SET JCOL = JSON_TRANSFORM(JCOL, SET '$.person.friends[3]' = 'John Smith')

[0134] The UPDATE statement generally changes the data values in one or more columns based on specific conditions. The following are example predicates for JSON columns such as JCOL.

[0135] '$.person.friends[3]' = 'John Smith'

[0136] At the SQL compilation stage, a LOB ASSIGMENT operator is created to copy the shortcut ($) to the 'person' object with the 'friend' attribute, and the third element of the array attribute sets its recognized frient attribute to 'John Smith'.

[0137] At the SQL compilation stage, a LOB ASSIGMENT operator is created to copy the right hand side (RHS) of the assignment (which calculates the result of JSON_TRANSFORM() in this case) to the left hand side (LHS) of the assignment, which is a permanent JSON column for which a full JSON document replacement is to be made. The SQL compiler detects this situation and may cause a partial OSON update at runtime because the JSON type input of JSON_TRANSFORM() is the same as the JSON column to be updated on the LHS side. Therefore, the compiler adds additional flags and offsets to the metadata structure of the LOB ASSIGNMENT operator to indicate that the RHS result should be specially handled at runtime for the partial OSON update situation. The metadata structure of the JSON_TRANSFORM() operator is annotated with the corresponding flags and the same offsets so that it can communicate its update status to its parent LOB ASSIGNMENT operator on a per-document basis at runtime.

[0138] 8.2 SQL / JSON JSON_TRANSFORM() Runtime Evaluation Using the LOB ASSIGNMENT Operator

[0139] At runtime, when computing JSON_TRANSFORM(), if the partial update flag is set, JSON_TRANSFORM() checks the input of the JSON type column. If the input JSON column OSON document size is small enough, such as in the case of BLOB inlining, the function evaluation loads the mutable DOM in memory from the OSON bytes for the JSON transformation engine that uses DOM queries and updates the API directly on the DOM in memory. Then, the mutated DOM in memory is encoded as OSON bytes as the return value of JSON_TRANSFORM().

[0140] After that, LOB ASSIGNMENT directly copies the OSON bytes for a full replacement on the left JSON column. Loading the OSON bytes into the mutable DOM in memory is fast because it directly maps the OSON field name dictionary to the DOM field name hash table in memory without deep copying the OSON leaf scalar values. Thus, the DOM leaf scalar content in the JSON memory directly points to the OSON bytes. The DOM update of the leaf scalar value in memory can be performed in-place in the OSON or by copy-on-write.

[0141] When the OSON bytes overflow the inlining capacity of the JSON column, external linkage is used for the JSON document. In this case, the JSON transformation engine still calls the same DOM API for querying and updating. However, this time the DOM API delegates to the underlying OSON DOM instead of the JSON DOM in memory. The DOM API that performs the update operation on the OSON can perform partial updates on the OSON bytes. After the transformation is complete, no new OSON bytes are returned from JSON_TRANSFORM(). Instead, JSON_TRANFROM() uses an offset shared with the LOB ASSIGNMENT operator to point to the memory location that records the completion status of the partial update and a pointer to the data structure that captures the partial OSON update entries in the update log.

[0142] When evaluating the LOB ASSIGNMENT operator, it dereferences based on the offset (shared with JSON_TRANSFORM()) to obtain the content to detect whether a partial OSON update has occurred instead of a full rewrite. If a partial update has occurred, the BLOB length-preserving update or append API is called based on the partial OSON update log structure. Otherwise, the result of JSON_TRANSFORM() is used for a full replacement.

[0143] 8.3 Partial OSON Update Log

[0144] The OSON DOM partial update API performs the actual update on the input OSON bytes copied into a modifiable buffer, while generating an OSON partial update log that is a sequence of partial update log entries. There are two types of update log entries:

[0145] A. Length-preserving replacement: Specifies the starting offset for a length-preserving update and the length of the new replacement bytes. The actual new OSON bytes have already been written into the modifiable buffer, and

[0146] B. Appending bytes: Copies all the new bytes that need to be appended to the end of the current OSON document into an append buffer.

[0147] When JSON_TRANSFORM() completes, it calls the OSON DOM to complete the OSON update log. The OSON length-preserving replacement update log entries are merged by sorting them by their offsets and coalescing all contiguous segments. This reduces the actual number of calls used to replace bytes in the OSON. Ideally, with this kind of merging, only one combined append bytes call is needed.

[0148] 8.4 If the cumulative update for appending is too large, automatically switch to a full OSON update

[0149] If, during an OSON DOM partial update operation, the total number of appended bytes will exceed 25% of the size of the original OSON document, stop tracking the OSON update log entries. Instead, when the request execution completes, new OSON bytes reflecting the final updated document are generated and returned as the return value of JSON_TRANSFORM(), such that the calling LOB ASSIGNMENT operator can perform a full document replacement.

[0150] 8.5 OSON bytecode changes to support partial OSON updates

[0151] The OSON is extended with new opcodes to indicate that the JSON node content is not in-place but at a forwarding address to a new offset (where, for example, a redirected tombstone stores the updated content).

[0152] For object and array new element insertions, the OSON partial update can pre-allocate sub-items in the object or array in amounts that are powers of two. This requires adding another new opcode, UPD_XSZ_RES, which indicates that the node has extra space reserved for growth. If the JSON tree node header is UPD_XSZ_RES, the next byte encodes the size of the extra space reserved for that node. The size is encoded as a power of two, so one byte is sufficient. After the space size byte can be the normal opcode.

[0153] 8.6 Client-side OSON Decoder

[0154] The client-side OSON decoder is enhanced to handle OSON that has undergone partial updates. The client interface of the binary image exposes the logical document (current state), and the end user is not directly aware of any partial updates that may have occurred. Other client technologies for OSON are presented in related U.S. Patent Application XX / XXX,XXX.

[0155] These technologies may be forensically detected because the explain plan of JSON_TRANSFORM() will show partial OSON update optimizations, and there are runtime statistics tracking how many partial updates have been completed during runtime.

[0156] Hardware Overview

[0157] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing device may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) permanently programmed to perform the techniques, or may include one or more general-purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage devices, or a combination thereof. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to implement these techniques. The special-purpose computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wired and / or program logic to implement the techniques.

[0158] For example, Figure 6 is a block diagram showing a computer system 600 on which embodiments of the present invention may be implemented. The computer system 600 includes a bus 602 or other communication mechanism for conveying information, and a hardware processor 604 coupled to the bus 602 for processing information. The hardware processor 604 may be, for example, a general-purpose microprocessor.

[0159] The computer system 600 also includes a main memory 606 coupled to the bus 602, such as a random access memory (RAM) or other dynamic storage device, for storing information and instructions to be executed by the processor 604. The main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 604. When stored in a non-transitory storage medium accessible by the processor 604, these instructions cause the computer system 600 to present as a special-purpose machine customized to perform the operations specified in the instructions.

[0160] The computer system 600 also includes a read-only memory (ROM) 608 or other static storage device coupled to the bus 602 for storing static information and instructions for the processor 604. A storage device 610, such as a magnetic disk, optical disk, or solid state drive, is provided and coupled to the bus 602 for storing information and instructions.

[0161] The computer system 600 can be coupled via the bus 602 to a display 612, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 614, including alphanumeric keys and other keys, is coupled to the bus 602 for communicating information and command selections to the processor 604. Another type of user input device is a cursor control 616, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 604 and for controlling cursor movement on the display 612. This input device typically has two degrees of freedom in two axes, e.g., a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0162] The computer system 600 can implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which in combination with the computer system causes or programs the computer system 600 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by the computer system 600 in response to execution of one or more sequences of one or more instructions contained in the main memory 606 by the processor 604. Such instructions can be read into the main memory 606 from another storage medium, such as the storage device 610. Execution of the instruction sequence contained in the main memory 606 causes the processor 604 to perform the processing steps described herein. In an alternative embodiment, hardwired circuitry may be used in place of or in combination with software instructions.

[0163] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular fashion. Such storage medium may include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, or solid state drives such as storage device 610. Volatile media includes dynamic memory, such as main memory 606. Common forms of storage medium include, for example, floppy disk, flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, RAM, PROM, and EPROM, flash-EPROM, NVRAM, any other memory chip or cartridge.

[0164] A storage medium is different from a transmission medium, but can be used in combination with a transmission medium. The transmission medium participates in the transfer of information between storage media. For example, the transmission medium includes coaxial cables, copper wire, and fiber optics, including the wires that make up bus 602. The transmission medium can also take the form of acoustic or light waves, such as those generated during radio-wave and infrared data communications.

[0165] When one or more sequences of one or more instructions are transferred to processor 604 for execution, various forms of media may be involved. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 600 can receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector can receive the data carried in the infrared signal, and appropriate circuitry can place the data on bus 602. Bus 602 transfers the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received by main memory 606 may optionally be stored on storage device 610 before or after execution by processor 604.

[0166] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides a two-way data communication coupling to a network link 620 connected to a local network 622. For example, communication interface 618 can be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 618 can be a local area network (LAN) card to provide a data communication connection to a compatible local area network. A wireless link can also be implemented. In any such implementation, communication interface 618 sends and receives electrical, electromagnetic, or optical signals that carry a digital data stream representing various types of information.

[0167] Network link 620 typically provides data communication to other data devices through one or more networks. For example, network link 620 can provide a connection through local network 622 to a main computer 624 or to a data device operated by an Internet service provider (ISP) 626. ISP 626 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 628. Both local network 622 and the Internet 628 use electrical, electromagnetic, or optical signals that carry a digital data stream. Signals through various networks and signals on network link 620 and through communication interface 618 are examples of forms of transmission media that carry digital data to and from computer system 600.

[0168] The computer system 600 can send messages and receive data, including program code, via one or more networks, network links 620, and a communication interface 618. In an Internet example, a server 630 can send request code for an application through the Internet 628, an ISP 626, a local network 622, and a communication interface 618.

[0169] The received code can be executed by the processor 604 when it is received and / or stored in a storage device 610 or other non-volatile memory for later execution.

[0170] Software Overview

[0171] Figure 7 is a block diagram of a basic software system 700 that can be used to control the operation of the computing system 600. The software system 700 and its components, including their connections, relationships, and functions, are intended to be merely exemplary and not limiting of the implementation of the example embodiments. Other software systems suitable for implementing the example embodiments can have different components, including components with different connections, relationships, and functions.

[0172] The software system 700 is provided to direct the operation of the computing system 600. The software system 700, which can be stored on the system memory (RAM) 606 and fixed storage (e.g., hard disk or flash memory) 610, includes a kernel or operating system (OS) 710.

[0173] The OS 710 manages the low-level aspects of computer operation, including managing the execution of processes, memory allocation, file input and output (I / O), and device I / O. One or more applications, represented as 702A, 702B, 702C…702N, can be “loaded” (e.g., transferred from the fixed storage device 610 to the memory 606) for execution by the system 700. Applications or other software intended to be used on the computer system 600 can also be stored as a set of downloadable computer-executable instructions, e.g., for downloading and installation from an Internet location (e.g., a web server, an app store, or other online service).

[0174] The software system 700 includes a graphical user interface (GUI) 715 for receiving user commands and data in a graphical (e.g., “click” or “touch gesture”) manner. These inputs can in turn be operated on by the system 700 according to instructions from the operating system 710 and / or the one or more applications 702. The GUI 715 is also used to display the results of operations from the OS 710 and the one or more applications 702, whereupon the user can provide additional input or terminate the session (e.g., log off).

[0175] OS 710 can execute directly on the bare hardware 720 (e.g., the processor 604) of the computer system 600. Alternatively, a hypervisor or virtual machine monitor (VMM) 730 can be inserted between the bare hardware 720 and the OS 710. In this configuration, the VMM 730 acts as a software “buffer” or virtualization layer between the OS 710 and the bare hardware 720 of the computer system 600.

[0176] The VMM 730 instantiates and runs one or more virtual machine instances (guest machines). Each guest machine includes a “guest” operating system such as the OS 710, and one or more applications designed to execute on the guest operating system, such as the (multiple) applications 702. The VMM 730 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0177] In some cases, the VMM 730 can allow the guest operating system to run as if it were running directly on the bare hardware 720 of the computer system 700. In these cases, the same version of the guest operating system configured to execute directly on the bare hardware 720 can also execute on the VMM 730 without modification or reconfiguration. In other words, in certain cases, the VMM 730 can provide full hardware and CPU virtualization to the guest operating system.

[0178] In other cases, the guest operating system can be specifically designed or configured to execute on the VMM 730 for increased efficiency. In these cases, the guest operating system “knows” that it is executing on the virtual machine monitor. In other words, in certain cases, the VMM 730 can provide para-virtualization to the guest operating system.

[0179] A computer system process includes the allocation of hardware processor time and a quota of (physical and / or virtual) memory for storing instructions executed by the hardware processor, for storing data generated by the hardware processor executing the instructions, and / or for storing the hardware processor state (e.g., the contents of registers) between quotas of hardware processor time when the computer system process is not running. Computer system processes run under the control of an operating system and can also run under the control of other programs executing on the computer system.

[0180] Cloud computing

[0181] The term “cloud computing” is commonly used here to describe a computing model that enables on-demand access to a shared pool of computing resources such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0182] Cloud computing environments (sometimes referred to as cloud environments or the cloud) can be implemented in a variety of different ways to best meet different needs. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that provides cloud services to other organizations or the public. In contrast, a private cloud environment is typically used only by or within a single organization. A community cloud is intended to be shared by multiple organizations within a community; while a hybrid cloud includes two or more types of clouds (e.g., private cloud, community cloud, or public cloud) bound together through data and application portability.

[0183] Generally, cloud computing models enable some of the responsibilities that might previously have been provided by an organization's own information technology department to be delivered as service layers in a cloud environment for use by consumers (either within or outside the organization, depending on the public / private nature of the cloud). Depending on the specific implementation, the exact definition of the components or functions provided by each cloud service layer may vary, but common examples include: Software as a Service (SaaS), where the consumer uses software applications running on the cloud infrastructure, and the SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), where the consumer can use software programming languages and development tools supported by the PaaS provider to develop, deploy, or control their own applications, and the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything under the runtime execution environment). Infrastructure as a Service (IaaS), where the consumer can deploy and run arbitrary software applications, and / or provision processing, storage, network, and other basic computing resources, and the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS), where the consumer uses a database server or database management system running on the cloud infrastructure, and the DBaaS provider manages or controls the underlying cloud infrastructure and applications.

[0184] The basic computer hardware and software and cloud computing environments described above are presented to illustrate the basic underlying computer components that can be used to implement the example embodiment(s). However, the example embodiment(s) are not necessarily limited to any particular computing environment or computing device configuration. Instead, the example embodiment(s) can be implemented in any type of system architecture or processing environment that those skilled in the art will understand, based on this disclosure, to be capable of supporting the features and functions presented in the example embodiment(s).

[0185] Database Overview

[0186] Embodiments of the present invention are used in the context of a database management system (DBMS). Accordingly, a description of an example DBMS is provided.

[0187] Typically, a server such as a database server is a combination that integrates software components and the allocation of computing resources, such as memory, nodes, and processes on nodes for executing the integrated software components, where the combination of software and computing resources is dedicated to providing a specific type of functionality on behalf of the clients of the server. The database server manages and facilitates access to a specific database and processes requests from clients to access the database.

[0188] Users interact with the database server of the DBMS by submitting commands that cause the database server to perform operations on data stored in the database. The users can be one or more applications running on a client computer that interact with the database server. Multiple users can also be collectively referred to as users here.

[0189] Database dictionary

[0190] The database includes data and a database dictionary stored on a permanent storage device mechanism such as a set of hard disks. The database is defined by its own separate database dictionary. The database dictionary can include multiple data structures that store database metadata. For example, the database dictionary can include multiple files and tables. Parts of the data structures can be cached in the main memory of the database server.

[0191] The database dictionary includes metadata that defines database objects contained in the database. In fact, the database dictionary defines most of the content of the database. When a database object is considered to be defined by the database dictionary, the database dictionary contains metadata that defines the attributes of the database object. For example, the metadata in the database dictionary that defines a database table can specify the column names and data types of the columns, as well as one or more files or parts thereof that store the data of the table. The metadata in the database dictionary that defines a procedure can specify the name of the procedure, the parameters and return data type of the procedure, and the data types of the parameters, and can include the source code and its compiled version.

[0192] Database objects include tables, table columns, and table spaces. A table space is a set of one or more files used to store data for various types of database objects (such as tables). If the data of a database object is stored in a table space, the database dictionary maps the database object to one or more table spaces that hold the data of the database object.

[0193] Database objects can be defined by a database dictionary, but the metadata in the database dictionary itself may only partially specify the properties of the database object. Other properties can be defined by data structures that may not be considered part of the database dictionary. For example, a user-defined function implemented in a Java class can be partially defined by the database dictionary by specifying the name of the user-defined function and by specifying references to the file containing the source code of the Java class (i.e., the Java file) and the compiled version of the class (i.e., the class file).

[0194] Database operations

[0195] The DBMS refers to the database dictionary to determine how to execute database commands submitted to the DBMS. Database commands can access database objects defined by the dictionary. Database commands can be in the form of database statements. For a database server to process a database statement, the database statement must conform to the database language supported by the database server. A non-limiting example of a database language supported by many database servers is SQL, including proprietary forms of SQL supported by database servers such as Oracle (e.g., Oracle Database 11g). SQL data definition language ("DDL") instructions are issued to the database server to create or configure database objects such as tables, views, or complex types. Data manipulation language ("DML") instructions are issued to the DBMS to manage the data stored within the database structure. For example, SELECT, INSERT, UPDATE, and DELETE are common examples of DML instructions in some SQL implementations. SQL / XML is a common extension of SQL used when operating on XML data in an object-relational database.

[0196] A multi-node database management system consists of interconnected nodes that share access to the same database. Typically, the nodes are interconnected via a network and share access to shared storage to varying degrees, such as shared access to a set of disk drives and data blocks stored thereon. The nodes in a multi-node database system can be in the form of a set of computers (e.g., workstations, personal computers) interconnected via a network. Alternatively, the nodes can be nodes of a grid consisting of server blades in the form of server blades interconnected with other server blades on a rack.

[0197] Each node in a multi-node database system hosts a database server. A server such as a database server is a combination of an integrated software component and the allocation of computing resources such as memory, nodes, and processes on the node for executing the integrated software component on a processor, the combination of the software and computing resources being dedicated to performing a specific function on behalf of one or more clients.

[0198] Resources from multiple nodes in a multi-node database system can be allocated to software running a particular database server. Each combination of the software and the resource allocation from the nodes is a server that is referred to herein as a "server instance" or "instance". A database server can include multiple database instances, some or all of which run on separate computers, including separate server blades.

[0199] Query Processing

[0200] A query is an expression, command, or set of commands that, when executed, causes the server to perform one or more operations on a set of data. A query can specify the (multiple) source data objects from which to determine the (multiple) result sets, such as (multiple) tables, (multiple) columns, (multiple) views, or (multiple) snapshots. For example, the (multiple) source data objects can appear in the FROM clause of a Structured Query Language ("SQL") query. SQL is a well-known example language for querying database objects. As used herein, the term "query" is used to refer to any form representing a query, including queries in the form of database statements and any data structure for internal query representation. The term "table" refers to any source object that is referenced or defined by a query and represents a set of rows, such as a database table, a view, or an inline query block (such as an inline view or a subquery).

[0201] A query can perform operations on the data in the (multiple) source data objects row by row when loading the (multiple) objects, or on the (multiple) entire source data objects after the (multiple) objects have been loaded. The result sets generated by (multiple) some operations can be available to (multiple) other operations, and in this way, the result sets can be filtered or narrowed based on certain criteria, and / or joined or combined with (multiple) other result sets and / or (multiple) other source data objects.

[0202] A subquery is a part or component of a query that is different from the (multiple) other parts or (multiple) components of the query, and can be evaluated separately (i.e., as a separate query) from the (multiple) other parts or (multiple) components of the query. The (multiple) other parts or (multiple) components of the query can form an outer query, which may or may not include other subqueries. When computing the result for the outer query, the subqueries nested within the outer query can be evaluated separately one or more times.

[0203] Typically, a query parser receives a query statement and generates an internal query representation of the query statement. Typically, the internal query representation is a set of interconnected data structures that represent the various components and structures of the query statement.

[0204] The internal query representation can be in the form of a node graph, with each mutually-linked data structure corresponding to a node and a component of the query statement being represented. The internal representation is typically generated in memory for evaluation, manipulation, and transformation.

[0205] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details, which may vary according to the implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the invention and what the applicant desires to be the scope of the invention is the literal and equivalent scope of the set of claims issued from this application, in the specific form in which such claims are issued, including any subsequent corrections.

Claims

1. A computer - implemented method, comprising: Storing a JavaScript Object Notation (JSON) document in compressed format in a permanent storage device; After storing the compressed - format JSON document in the permanent storage device, automatically determining to compact the compressed - format JSON document in the permanent storage device; And In response to automatically determining to compact the compressed - format JSON document in the permanent storage device, compacting the compressed - format JSON document in the permanent storage device.

2. The method according to claim 1, wherein Storing the compressed - format JSON document in the permanent storage device includes inlining the compressed - format JSON document in a table row in the permanent storage device; The method further includes receiving a write request that will cause the compressed - format JSON document to exceed the capacity of the table row; The determination to compact the JSON document is in response to receiving the write request that will cause the compressed - format JSON document to exceed the capacity of the table row.

3. The method according to claim 2, further comprising, in response to compacting the compressed - format JSON document, maintaining the compressed - format JSON document in the table row.

4. The method according to claim 1, wherein: ​ ​ ​ ​ ​ 6. The method according to claim 1, wherein, ​ ​ ​ ​ ​ ​ ​ 7. The method according to claim 6, wherein, ​ ​ ​ ​ ​ ​ The duration elapsed since the last increase in the size of the compressed JSON document in the persistent storage device without completely rewriting the compressed JSON document in the persistent storage device.

8. The method according to claim 6, wherein the automatic determination to compact the JSON document is based on at least one selected from the group consisting of: A timer, A plurality of numerical compaction thresholds, A threshold for counting the specified numerical compaction threshold, and A prediction of future access to the compressed JSON document.

9. The method according to claim 1, wherein: An activity occurs in one selected from the group consisting of foreground and background; The activity is one selected from the group consisting of: The determination to compact the JSON document, and The compaction of the compressed JSON document.

10. The method according to claim 1, wherein the automatic determination to compact the JSON document is not in response to a client request.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the steps of the method according to any one of claims 1-10 to be performed.

Images