Immutable data storage for low-latency reading and writing of large data sets

Abstract

A process is provided, comprising: receiving a write command requesting that a document associated with the write command be stored in an immutable data structure, the immutable data structure preventing an attacker attempting to modify the document from concealing that the document has been modified after the document is stored in the data structure; forming a tamper-evident immutable directed acyclic graph, with the documents located in one or more nodes of the graph; and storing the directed acyclic graph.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent claims the benefit of U.S. Patent Application No. 15 / 675,460, entitled “IMMUTABLE DATASTO RE FOR LOW-LATENCY READING AND WRITING OF LARGE DATA SETS,” filed on August 11, 2017. The entire contents of each of these earlier filed applications and the contents incorporated herein by reference are hereby incorporated by reference for all purposes. Technical Field

[0003] The present disclosure relates generally to network security and, more particularly, to immutable data storage configured for low-latency access when storing relatively large data sets. Background Art

[0004] Data stores such as document repositories, file systems, relational databases, non-relational databases, memory images, key-value stores, etc. are used in a variety of different types of computing systems. Typically, data to be stored is received by the data store and then retrieved during a read operation. In many cases, the data store arranges the data in a manner that facilitates access based on the address of the data in the data store (e.g., a file name) or the content of the data (e.g., a select statement in a structured query language query).

[0005] In many cases, the security and integrity of data in data stores is not trusted. Often, attackers who break into a computer network modify or leak records in the data store that are intended to be confidential. Furthermore, in many cases, the attacker may be a qualified entity within the network, such as a rogue employee, which in some cases renders many traditional data store security methods insufficient. In many cases, such attackers may obscure their activities in the network by deleting access logs stored in the data store, which exacerbates the risk. Summary of the invention

[0006] The following is a non-exhaustive list of some aspects of the present technology. These and other aspects are described in the following disclosure.

[0007] Some aspects include a process comprising: receiving a write command by using one or more processors, the write command requesting that a document associated with the write command be stored in an immutable data structure, the immutable data structure preventing an attacker attempting to modify the document from concealing that the document was modified after the document was stored in the data structure; forming at least a portion of a directed acyclic graph by utilizing one or more processors, the directed acyclic graph having a plurality of nodes and edges linking corresponding pairs of nodes of the nodes, wherein: the nodes of the directed acyclic graph have corresponding node identifiers, the node identifiers distinguishing different nodes of the directed acyclic graph; the nodes of the directed acyclic graph have node content; each of the at least some nodes of the directed acyclic graph has a node content. nodes having node contents including a plurality of encrypted hash values; the encrypted hash value of each node is (i) an encrypted hash function output based on the node contents of another corresponding node of the directed acyclic graph, and (ii) associated with a node identifier of another corresponding node, thereby specifying an edge of the directed acyclic graph; and forming at least a portion of the directed acyclic graph comprises: writing data encoding at least some of the documents into the node contents of a first node set of the directed acyclic graph; and starting from the first node set, traversing the edges of the directed acyclic graph and forming the node contents of the visited nodes by determining the encrypted hash values ​​of the nodes visited via the traversal; and storing the directed acyclic graph in a memory using one or more processors.

[0008] Some aspects include a tangible, non-transitory, machine-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations including the processes described above.

[0009] Some aspects include a system comprising: one or more processors; and a memory storing instructions that, when executed by the processors, cause the processors to implement the operations of the above-described processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other aspects of the present technology will be better understood when this application is read with reference to the following drawings, in which like reference numerals represent similar or identical elements:

[0011] Figure 1 is a logical and physical architecture block diagram illustrating an example of a computing environment in which the present technology may be implemented according to some embodiments;

[0012] Figure 2 is a flow chart illustrating an example of a process for reading and writing documents to a tamper-evident immutable data repository according to some embodiments;

[0013] Figure 3 is a data model block diagram showing that according to some embodiments, Figure 2 Examples of tamper-evident immutable data repositories that are operated on by processes;

[0014] Figure 4 is a flow chart illustrating an example of a process according to some embodiments by which a stored value may be fragmented and stored in multiple different tamper-evidently immutable directed acyclic graphs, such as Figure 3 As shown;

[0015] Figure 5 is a data model block diagram illustrating a tamper-evident immutable data repository according to some embodiments, which stores data fragments distributed among multiple directed acyclic graphs and can be accessed through Figure 4 and Figure 6 The process operates on the data repository;

[0016] Figure 6 is a diagram showing that according to some embodiments, Figure 5 A flowchart of an example of a process of reading data from a data structure;

[0017] Figure 7 is a flow chart illustrating an example of a process by which a cryptographic hash pointer in a tamper-evident data store may be formed to modularize the data store in accordance with some embodiments;

[0018] Figure 8 is a flow chart illustrating an example of a process according to some embodiments by which existing workload applications and user interfaces may be retrofitted to securely store and access relatively high security data in a transparent manner;

[0019] Fig. 9 is a flow chart illustrating an example of a process by which access requests to decentralized data may be logged to a tamper-evident immutable data repository according to some embodiments;

[0020] Fig.10 is a flow chart illustrating an example of a process by which data in a tamper-evident immutable data repository may be compressed by storing differences between versions of documents or other units of content, according to some embodiments;

[0021] Fig.11 is a flow chart illustrating an example of a process by which a version graph storing a sequence of changes between versions of a document may be traversed to read a current version of a document according to some embodiments;

[0022] Fig.12is a data model block diagram illustrating an example of a data structure by which a version graph may be stored as a sequence of content graphs on a collection of verification graphs or other tamper-evident immutable data repositories according to some embodiments; and

[0023] Fig.13 is a physical architecture block diagram illustrating an example of a computing device by which the above-described techniques may be implemented.

[0024] Although the present technology is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will be described in detail herein. The drawings may not be drawn to scale. However, it should be understood that the drawings and their detailed description are not intended to limit the present technology to the particular form disclosed, but on the contrary, it is intended to cover all modifications, equivalents and alternative forms that fall within the spirit and scope of the present technology as defined by the appended claims. DETAILED DESCRIPTION

[0025] In order to mitigate the problems described herein, the inventors must both invent solutions and, in some cases, equally importantly, recognize problems in the field of network security that others have overlooked (or have not yet foreseen). Indeed, the inventors wish to emphasize that it is difficult to identify those emerging problems, and if industry trends are as the inventors expect, these problems will become more apparent in the future. In addition, because multiple problems are solved, it should be understood that some embodiments are problem-specific, and not all embodiments solve every problem of the traditional systems described herein or provide every benefit described herein. That is, improvements that solve various arrangements of these problems are described below.

[0026] Various issues related to the security of data storage and networks of computers used by organizations are addressed by various versions of the technology described below. In some cases, these different technologies can be used in conjunction, so their descriptions are grouped into a single description that will be submitted in multiple patent applications with different sets of claims for different technologies and their combinations. In view of this approach, it should be emphasized that these technologies are also independently useful and can be deployed separately from each other or in any arrangement that combines different subsets of the technology, and none of these implies that any other description herein is restrictive. These technologies are preceded by the following titles before conceptually related groups. These titles should not be understood to mean that the subject matter under different titles cannot be combined, that each embodiment described under a title has all the features of that title, or that each feature under a given title should be present in an embodiment consistent with a corresponding conceptually related set of technologies, nor does it imply that other descriptions are restrictive.

[0027] Given that Figure 1The techniques may be best understood with reference to the example computing environment 10 shown. The computing environment 10 is one example of many computing architectures in which the techniques may be implemented. In some embodiments, the techniques are implemented as a multi-tenant distributed application in which some computing hardware is shared by multiple tenants who access resources on computing hardware in computing devices controlled by those tenants (e.g., on various local area networks operated by the tenants). Or in some cases, a single tenant may execute each of the illustrated computing entities on privately controlled hardware, with multiple instances of the computing environment 10 existing for different organizations. Or some embodiments may implement a hybrid approach in which multi-tenant computing resources (e.g., computers, virtual machines, containers, microkernels, etc.) are combined with pre-built computing resources or private cloud resources. In some embodiments, the computing environment 10 may include and expand upon the security features of the computing environment described in U.S. patent application 15 / 171,347, filed on June 2, 2016, entitled “COMPUTER SECURITY AND USAGE-ANALYSIS SYSTEM” (Case Nos. 043788-0447379), the contents of which are incorporated herein by reference.

[0028] In some embodiments, the computing environment 10 includes multiple client computing devices 12, a low-trust database 14, a secure distributed storage 16, a domain name service 18, and a converter server 20 (or an elastically scalable collection of instances of the converter server set behind a load balancer). In some embodiments, each of these components can communicate with each other via the Internet 22 and various local area networks in some cases. In some embodiments, communication can be carried out via a virtual private network overlaid on the public Internet. In some embodiments, the components shown can be geographically distributed, for example, more than 1 kilometer apart, more than 100 kilometers apart, more than 1000 kilometers apart, or distributed, for example, across content events in North America or the world. Or in some cases, the components can be co-located and hosted in a private network with or without gaps. In some embodiments, the following references can be used Fig.13 One or more computing devices are depicted implementing each illustrated block connected to the Internet 22 .

[0029] In some embodiments, each client computing device 12 may be one of a plurality of computing devices operated by a user or application of an entity wishing to securely store data. For example, a given enterprise or government organization may have more than 10, more than 100, more than 1,000, or more than 10,000 users and applications, each of which has an associated computing device that accesses data stored in a low-trust database 14 (or a collection of such databases or other types of data storage) and a secure distributed storage 16. In some embodiments, multiple entities may access a system in a competitive environment 10, for example, more than five, more than 50, more than 500, or more than 5,000 different entities may access shared resources with corresponding client computing devices or may have their own instances of the computing environment 10. In some embodiments, some client computing devices 12 are end-user devices that, for example, execute client-side components of a distributed application that stores data in a low-trust database 14 and secure distributed storage 16 or reads such data. The client computing device may be a laptop, desktop computer, tablet computer, smart phone, or rack-mounted computing device (e.g., server). In some embodiments, the client computing device is an Internet of Things device, such as a smart TV, a set-top media player, a security camera, a smart lock, an autonomous car, a drone, an industrial sensor, an industrial actuator (e.g., an electric motor), or an in-store kiosk. In some embodiments, some client computing devices 12 may be headless computing entities, such as containers, microkernels, virtual machines, or rack-mounted servers, that execute one or more services in a monolithic application or a service-oriented application such as a microservices architecture, which stores or otherwise stores data in a low-trust database 14 or a secure distributed storage 16.

[0030] In some embodiments, both the low-trust database 14 and the secure distributed storage 16 may store a portion of the data accessed by the client computing device 12, and in some cases, a pointer therebetween is stored in one or both of these data stores. In some embodiments, as described below, the data may be stored in a manner that extracts the secure distributed storage 16 from a workload application, through which the data is accessed (e.g., read or written). In some embodiments, data access operations may store or access data in the low-trust database 14 and the secure distributed storage 16 using a workload application that is not specifically configured to access data in the secure distributed storage 16, for example, configured to operate without regard to whether the secure distributed storage 16 exists, and the data storage in the secure distributed storage 16 is transparent to the workload application that stores content in the low-trust database 14 and the secure distributed storage 16. In some embodiments, such a workload application may be configured and otherwise designed to interface only with the low-trust database 14 when storing such data, and, as described below, some embodiments may wrap the interface of the low-trust database 14 with additional logic that routes some data to and retrieves some data from the secure distributed storage 16 in a manner that is transparent to the workload application accessing the content (i.e., data written or read by the workload application).

[0031] A variety of different types of applications (e.g., monolithic applications or multi-service distributed applications (e.g., implementing a microservices architecture in which each service is hosted by one of the client computing devices 12) may be used to create or access content stored in the low-trust database 14 and secure distributed storage 16. Examples include email, word processing systems, spreadsheet applications, version control systems, customer relationship management systems, human resources computer systems, accounting systems, enterprise resource management systems, inventory management systems, logistics systems, secure chat computer systems, industrial process control and monitoring, trading platforms, banking systems, etc. Such applications that generate or access content in the database 14 in order to serve the functions of the application are referred to herein as "workload applications" to distinguish those applications from the infrastructure code that implements the present technology, which is not meant to imply that in some embodiments, these code bodies cannot be integrated into a single workload application with infrastructure functions. In some cases, several workload applications (e.g., more than 2, more than 10, or more than 50) such as those selected from the previous list can share resources provided by the infrastructure code and functions described herein.

[0032] In some embodiments, the low-trust database 14 is one of the various types of data storage described above. In some cases, the low-trust database 14 is a relational database having multiple tables, each table having a set of columns corresponding to different fields or types of values ​​stored in a row or record (i.e., a row in some embodiments) in the table, and in some cases, each record corresponding to a row may be a tuple having a unique primary key in the corresponding table, one or more foreign keys that are primary keys in other tables, and one or more other values ​​corresponding to different columns specifying different fields in the tuple. Or in some cases, the database may be a column-oriented database in which records are stored in columns, in which different rows correspond to different fields. In some embodiments, the low-trust database 14 may be a relational database configured to be accessed using a structured query language (SQL) command, such as a command for selecting records that satisfy a condition specified in the command, a command for joining records from multiple tables, or a command for writing values ​​to records in these tables.

[0033] Or in some cases, the low-trust database 14 can be another type of database, such as a non-SQL database, such as various types of non-relational databases. In some embodiments, the low-trust database 14 is a document-oriented database, such as a database that stores multiple serialized hierarchical data format documents (e.g., JavaScriptTM object tag (JSON) documents or extensible markup language (XML) documents). In some cases, the access request can take the form of an x-path or JSON-path command. In some embodiments, the low-trust database 14 is a key value data store, which has a collection of key value pairs in which data is stored. Or in some cases, the low-trust database 14 is any one of a variety of other types of data stores, such as documents in a version control system, memory images, distributed or non-distributed file systems, etc. Examples. A single low-trust database 14 is shown, but the embodiment is consistent with substantially more different databases (e.g., more than two, more than five, or more than ten different databases) that are different types in the above examples in some cases, and the database may be included in a commercial instance. In some embodiments, some low-trust databases may be databases of software-as-a-service applications hosted by a third party, and are accessed via a third-party application program interface by exchanging with, for example, a user's web browser or another application. In some cases, low-confidence database 14 is a mutable data store or an immutable data store.

[0034] In some cases, access to data in low-trust database 14 and corresponding access to corresponding records in secure distributed storage 16 may be specified in part using roles and permissions stored in association with various user accounts of applications used to access the data. In some embodiments, these permissions may be modified (e.g., revoked or otherwise adjusted) using the techniques described in U.S. patent application Ser. No. 15 / 171,347, filed Jun. 2, 2016, entitled “COMPUTER SECURITY AND USAGE-ANALYSIS SYSTEM” (Case Nos. 043788-0447379), the contents of which are incorporated herein by reference.

[0035] Database 14 is described as “low trust.” The term “low trust” does not require an absolute measure relative to the trustworthiness of any party or any particular psychological state, but is used to distinguish database 14 from secure distributed storage 16, which in some embodiments described below has certain security features and may be referred to as a “high trust” database in some cases.

[0036] In some cases, some data written or written to the low-trust database 14 by the application can be intercepted or moved to the secure distributed storage 16 by the techniques described below. In addition, access requests from workload applications to the low-trust database 14 can be intercepted, or responses from such access requests can be intercepted, and data from the low-trust database 14 can be merged with data from the secure distributed storage 16 before being presented to the application in response to the request, as described in more detail below. In addition, as described below, read requests can be intercepted, modified, and iteratively executed in a manner that limits the amount of information in the secure distributed storage revealed to the client computing device at any one time.

[0037] In some embodiments, the secure distributed storage 16 may include a collection of data centers 24, which may be geographically distributed and have a heterogeneous architecture. In some embodiments, the data centers 24 may be various public or private clouds or local data centers for one or more organizational users (e.g., tenants) of the computing environment 10. In some embodiments, the data centers 24 may be geographically distributed across the United States, North America, or the world, with different data centers in some cases being more than 100 or 1000 kilometers apart, and in some cases, different data centers 24 being in different jurisdictions. In some embodiments, each data center 24 may include a different private subnet through which computing devices (e.g., rack-mounted computing devices in the subnet) communicate behind a firewall relative to the Internet 22, for example, through a top-of-rack switch within the data center. In some embodiments, each data center 24 or different subsets of data centers 24 may be operated by different entities that implement different security architectures and have different application program interfaces to access computing resources, examples of which include Amazon Web Services TM 、Azure from Microsoft TM and Rack Space TM Three different data centers 24 are shown, but embodiments are consistent with more data centers, and a commercial implementation may include more data centers, such as greater than five, greater than 15, or greater than 50 data centers. In some cases, the data centers may be from the same provider but located in different regions.

[0038] In some embodiments, each data center 24 includes a plurality of different hosts exposed by different computing entities, such as microkernels, containers, virtual machines, or computing devices executing non-virtual operating systems. Each host may have an Internet Protocol address on a subnet of the corresponding data center 24, and may listen and transmit via a port assigned to an instance of an application described below, through which data is stored in a distributed ledger. In some embodiments, each storage computing node 26 may correspond to a different network host, each network host having a server monitoring a port, and configured to implement an instance of one of the directed acyclic graphs described below using a hash pointer that implements an immutable tamper-evident distributed ledger (examples include blockchains and related data structures). In some cases, these storage computing nodes 26 may be replicated between data centers 24, where, for example, three or more instances are used as replication instances, and some embodiments may implement the techniques described below to determine a consensus between these replication instances about the state of the stored data. In addition, some embodiments may elastically scale the number of such instances based on the amount of stored data, the amount of access requests, and the like.

[0039] Some embodiments may further include a domain name service (DNS) 18, such as a private DNS, which maps uniform resource identifiers (e.g., uniform resource locators) to Internet Protocol address / port number pairs, such as storage computing nodes 26, transformers 20, and in some cases other client computing devices 12 or other resources in the computing environment 10. In some embodiments, a client computing device 12, storage computing node 16, database 14, or transformer 20 may encounter a uniform resource identifier, such as a uniform resource locator, and the computing entity may be configured to access the DNS 18 at the IP address and port number pair of the DNS 18. The entity may send a request to the DNS 18 with the uniform resource identifier, and the DNS 18 may respond with a network and processing address, such as an Internet Protocol address and port number pair corresponding to the uniform resource identifier. As a result, the underlying computing device may be replaced, copied, moved, or otherwise adjusted without compromising cross-references between information stored on different computing devices. Or some embodiments may achieve this flexibility without using domain name service 18, for example, by implementing a distributed hash table or load balancing that consistently maps data to appropriate computing devices or hosts based on the content of the data (e.g., based on a hash-based prefix or suffix, based on the data or an identifier of the data). For example, some embodiments may implement a load balancer that routes requests to storage computing nodes 26 based on a prefix of a node identifier (e.g., a threshold number of characters before or after).

[0040] Some embodiments may further include a virtual machine or container manager configured to orchestrate or otherwise elastically scale instances of compute nodes and instances of converter 20, for example, automatically applying corresponding images to provisioned resources within one or more data centers 24 in response to demand and slowly reducing instances as demand decreases.

[0041] In some embodiments, the converter 20 may be configured to perform a routine described in more detail below that converts between the address space of the low-trust database 14 and the address space of the secure distributed storage 16. In some embodiments, the converter 20 may receive one or more records to be written to the low-trust database 14 from the client computing device 12, or may receive such records from the low-trust database 14, and those records may be mapped to the fragment identifiers (or other pointers, such as other node identifiers) described below in the secure distributed storage 16. The converter 20 may then cause those records to be stored in the secure distributed storage 16, and cause the fragment identifiers to be stored in the low-trust database 14 in place of those records, such as in place of individual values ​​in the records. In some embodiments, the conversion may occur at the level of individual values ​​corresponding to individual fields in individual records, such as rows of a table in the database 14, or some embodiments may convert larger data sets, for example, accepting entire records, such as entire rows or multiple columns, such as a master key and other values ​​in addition to the master key in a given row. Some embodiments may accept files or other binary large objects (binary large objects S). The converter 20 may then replace those values ​​in the low-trust database 14 with pointers (e.g., fragment identifiers in the secure distributed storage) in the manner described below, and then cause those data to be stored in the secure distributed storage 16 in the manner described below. In some examples, documents may be stored, which may be relatively small, independent values ​​relative to binary large objects that encode file system objects such as word processing files, audio files, video files, chat logs, compressed directories, and the like. In some cases, a document may correspond to a single value within a database, or a document may correspond to a file or other binary large object. In some cases, a document may be larger than one byte, 100 bytes, 1 kB, 100 kB, 1 MB, or 1 GB. In some embodiments, a document may correspond to a message in a messaging system or a document in a printable document format, such as a Microsoft Word document. TM Documents, audio files, video files, etc.

[0042] In some embodiments, transformer 20 may include code that receives requests from a driver and facilitates data transformation. In some cases, transformer 20 may be one transformer in a collection of elastically scaled transformers 20 remotely hosted in a public or private cloud. In some cases, transformer 20 may implement the following functions:

[0043] 1. Verify the request

[0044] a. Some embodiments use a database to verify a combination of user-supplied parameters, such as a predefined software ID, client ID, and a machine-specific identifier registered at installation time. This is compared to a known list and then further verified using an IP address and / or other network-specific parameters.

[0045] 2. Data Verification

[0046] a. Parse the HTTP body and then decode some embodiments to determine a unique list of reference values ​​to replace with plain text. Some embodiments use a database to first check whether the requesting machine has permission to access the data. Some embodiments next use the database to find the network name of the first hop for the piece of data and place it in an array.

[0047] 3. Threshold Check

[0048] a. Using the location of each unique requested fragment (or node, document, or content) identifier, some embodiments check against a series of thresholds or rate objects. Some embodiments look for access rate, time window, or location-based rules and apply the requested data to the map of rules. If any particular data exceeds the threshold, in some embodiments, an exception is generated in the system, resulting in notification and logging.

[0049] 4. Work

[0050] a. The converter 20 can split the data request into multiple jobs and then place the jobs into a work queue. The splitting can be done with a static per-message work size and can use a transactional letter exchange to retry and eventually fail the message

[0051] 5. Response function

[0052] a. Data can be returned from the queue, and plain text values ​​can be matched and replaced with corresponding pointers (e.g., fragment, document, node, or content unit identifiers, which does not imply that these or any other category lists describe disjoint sets). In some embodiments, once all jobs have returned responses, responses can be returned.

[0053] In some embodiments, the client computing devices 12 may each execute an operating system in which one or more applications 28 are executed. These applications may include the client-side portion of the above-mentioned example of a workload application, which may include business logic and other program code, and the services in the microservice architecture are implemented by the business logic and other program code. In some embodiments, the application 28 may be different in different client computing devices, and a single client computing device may execute multiple different applications. In some embodiments, the application 28 may be configured to interface with the low-trust database 14 via a database driver 32 executed within the operating system. The database driver 32 may be any of a variety of different types of drivers, such as an ODBC driver, a JDBC driver, etc. In some embodiments, the database driver 32 may be configured to access the low-trust database 14 via a network interface 34 of the client computing device 12 (e.g., a network interface card connected to the physical medium of a local area network, through which the Internet 22 is accessed).

[0054] Some embodiments may further include a security driver 30 that interfaces between the application 28 and the database driver 32. In some embodiments, the security driver 30 may be transparent to the application 28, such that an application program interface of the database driver 32 is presented to the application 28 by the security driver 30, and in some cases, the application program interface may not be modified from the perspective of the application 28 relative to the application program interface provided by the database driver 32. In some embodiments, the security driver 30 may wrap the application program interface of the database driver 32, such that the security driver 30 receives application program interface requests from the application 28 to the driver 32, acts on those requests, and in some cases modifies those requests, and then in some cases provides the requests with the modifications to the database driver 32. Similarly, responses to the application 28 may be provided by the security driver 30 and in a manner consistent with that provided by the driver 32, as described in more detail below.

[0055] In some embodiments, the security driver 30 is configured to enable the converter 20 after the data written (or attempted to be written) to the low-trust database 14 by the application 28 is split (or the split is to be performed) into higher security data and lower security data. Similarly, the terms "lower security" and "higher security" are used to distinguish data that are classified differently for security purposes, and do not need to be measured for absolute security metrics or psychological states. Then, the lower security data can be written to the low-trust database 14 by the database driver 32 in a manner provided by the application 28, regardless of whether the security driver 30 is present.

[0056] On the other hand, the higher security data can be stored by the transformer 20 in a manner described below, which makes the data relatively robust to attacks by malicious actors. When the data is returned to the application 28, for example in response to receiving a read request, these operations can be reversed in some cases. Again, these operations will be described in more detail below. Generally, in some embodiments, in some cases, the data from the low-trust database 14 and the data from the secure distributed storage 16 can be merged before being presented to the application 28. By acting on the higher security data within the client computing device 12 before it leaves the client computing device 12, some embodiments can reduce the attack service of the computing environment 10. That is, not all embodiments provide this benefit, and some embodiments can implement the functionality of the security driver 30 outside the client computing device 12, such as in a database gateway, in a database management system implemented at the low-trust database 14, or on another independent application executed in a computing device between the low-trust database 14 and the network and the client computing device 12 in the path of the low-trust database 14.

[0057] In some embodiments, the security driver 30 includes an outbound path and an inbound path. In some embodiments, the outbound path includes an output parser 36, a validator 38, and a data multiplexer 40. The output parser can apply one or more rules in the data policy described below to classify the value as a higher security value or a lower security value. The validator can perform the statement verification function described below. The multiplexer can route the data to the low-trust database 14 or the transformer 20 based on the security classification. In some embodiments, the inbound path includes a parser 42 and a data demultiplexer 44. The inbound path may include a parser 42, which is configured to detect a pointer to data in a query response from the low-trust database 14, which points to data in the secure distributed storage 16. The parser 42 can call the transformer 20 to request to replace the pointer with more securely stored data. In some cases, the demultiplexer 44 can merge the data from the transformer 20 with the lower security data in the same query response. In some cases, the security driver can implement the following reference Figure 8 The process described above performs the following functions:

[0058] 1. Statement parsing

[0059] a. For a SELECT statement, there may be a WHERE clause that looks for matches to data in protected columns. During this phase, some embodiments parse the SELECT statement and check if any plain text values ​​in the WHERE clause need to be flipped into the reference space. The statement may be marked as processed and proceed to the next one.

[0060] b. For INSERT or UPDATE statements, data may exist in either the statement body or the WHERE clause (INSERT). During this phase, some embodiments parse the statement and check if any plain text values ​​in the WHERE clause or body need to be flipped into the reference space. The statement may be marked as processed and proceed to the next one.

[0061] c. The security driver may use a locally retained copy of the current protection settings for a given client. In some embodiments, the database, table, and column names in the statement are compared to this locally saved and updated (e.g., periodically or continuously) table. The time between obtaining new status tables is determined by various factors.

[0062] 2. Statement Verification

[0063] a. During the operation of some database commands, some embodiments check whether there is a potential injection or other malicious SQL statement in the statement, and block the query or log of the event. In some cases, this is a locally supported operation that each driver can complete.

[0064] 3. Statement processing

[0065] a. Depending on the result of the parsing, the driver may make an HTTP request to a preset URL and request that the plain text data be switched to a reference space (eg, switched by the transformer 20).

[0066] b. The statement can be updated with reference spatial data as needed and can be passed to the low-trust database 14 server.

[0067] 4. Result set processing

[0068] a. For a SELECT statement, the result set is processed and if a column in the returned data matches any entry in the locally held table, the security driver 20 may perform an HTTP request to switch the reference space data to the plain text space.

[0069] b. The driver 30 can iterate through the data and select different values ​​to put into the HTTP body and the request made by, for example, enabling the transformer 20 to use a preset URL and the system DNS 18.

[0070] c. In some cases, data may be returned and replaced for each event in the result set and in some cases returned to the application 28 .

[0071] Various aspects of the above system or other architectures may implement various techniques expanded under different headings.

[0072] Immutable data storage for low-latency reading and writing of large data sets

[0073] Traditional databases are often inadequate to prevent data tampering by threat actors or internal resources (employees, IT personnel, etc.). At best, such systems often provide the ability to audit access and modifications to stored data, but audit logs are often mutable and can therefore be altered just as easily as the data itself.

[0074] Recent examples of immutable databases include blockchain-based databases such as Bitcoin and MultiChain. Blockchain systems are built on ideas first described in an October 2008 article by the pseudonymous Satoshi Nakamoto titled “Bitcoin: A Peer-to-Peer Electronic Cash System”. These systems are typically based on some combination of cryptography, consensus algorithms, and proof-of-X to implement a peer-to-peer system, where X is some aspect that is difficult to integrate across the entire network, such as proof-of-work, proof-of-stake, proof-of-storage, etc. Typically, actors on the network with proof-of-X typically use a variety of consensus algorithms such as Paxos, Raft, or hash tables to reach consensus on peer-to-peer transaction verification. Alternatively, some private blockchains do not implement proof-of-X consensus, for example, where the computing hardware implementing the blockchain is controlled by a trusted party. Chained cryptographic operations bind a sequence of such transactions into a chain, which, once verified, is typically computationally prohibitively expensive to tamper with.

[0075] However, many existing blockchain-based databases are not suitable for certain use cases, particularly those involving latency-sensitive access (e.g., reading or writing) to large files (e.g., documents or other binary data sets processed as a single entity, often referred to as "binary large objects"), such as in a file system hosted by the blockchain. In fact, many blockchain databases are not easily configured to store large objects and object files (e.g., on the order of 500KB or larger, depending on the use case and acceptable latency) because such systems are often highly specialized for small-load "transactional" applications. In such systems, when storing larger sets of binary data (e.g., files or binary large objects), the chain may become significantly slower as it becomes larger, particularly for write operations.

[0076] As mentioned above, blockchains generally allow small amounts of information to be stored in an immutable data structure, but once the data in the chain is changed, the chain is broken and can no longer function in a way that represents the data as valid. The most common blockchain implementation is the publicly accessible Bitcoin ledger (for which the blockchain was designed).

[0077] As described above, there are certain problems with such systems for some use cases. These and other problems are alleviated by some embodiments of the system referred to below as "Document Chain". Document Chain is a blockchain software suite specifically designed for low-latency placement and retrieval operations of binary large objects (Blobs). In some embodiments, Document Chain uses an improved version of the underlying data structure known as a "Merkle tree" to provide the immutable properties of blockchain. In some embodiments, the document chain is operable to respond to commands in less than 50 milliseconds (ms), for example, and it is expected that many commercial implementations provide operations of less than 10ms as perceived by applications interacting with the chain, and operations of less than 5ms at the level of the chain. In addition, it is expected that such response times will scale with the size of the chain. Some embodiments may scale the response time with nlog(n), where n is the number of entries in the chain.

[0078] Merkle trees usually work by cryptographic hashing trees, where each element of the tree contains a hash of the information contained in its children, and the leaf elements are the hashes of the subject data stored in the Merkle tree. In many traditional Merkle tree implementations, for the purpose of storing data, as in many early blockchain databases, the data is stored in a separate logical data store, hashed, and only the hash is brought into the Merkle tree. That is, the data stored by the database is not part of the Merkle tree, only the hash summary of the data is part of the Merkle tree.

[0079] To mitigate some of the issues with traditional blockchain databases, some embodiments of the document chain store data directly in a Merkle tree, although embodiments are not limited to data being stored in a Merkle tree, which does not mean that other descriptions are restrictive. That is, when data is written to or read from the database, the data will be written to or read from a specific field of an element of the Merkle tree (e.g., an attribute of the node content of a node) (rather than just the hash summary of the data residing in the Merkle tree, the entire data resides in an external data store). With this tight coupling, if the data is completely changed, the entire hash of the tree (different from the hash of the file stored in the database) will be immediately discarded, and the final chain will be destroyed and detected as destroyed during the verification operation. In addition, some embodiments prevent binary large objects from being stored in external (potentially mutable and less secure) data stores. Ultimately, the tight coupling of data to the tree is expected to reduce the number of read / write operations required to conduct transactions with the chain, thereby further reducing potential latency.

[0080] In some embodiments, the document chain includes two components: LibDocuchain and DocuSocket.

[0081] In some embodiments, the library document chain is a software library that includes the following data structures:

[0082] MerkleTree<H,T>

[0083] *DataBlock<H,T>

[0084] *BlockManager<H,T>

[0085] Where "H" represents a cryptographic hash function and "T" represents the type of data stored in the chain. In general, a hash function maps data of arbitrary size to data of fixed size. Examples of cryptographic hash functions include SHA256, SHA-2, MD5, etc. (e.g., in some embodiments, document chains are applied more than once, multiple different types of hash functions may be used, and it is expected that hash algorithms developed in the future (e.g., new post-quantum cryptographic hashes) may be included).

[0086] In some embodiments, the BlockManager is responsible for taking the data, putting it into the appropriate block and inserting it into the appropriate Merkle tree, writing it to disk at the appropriate time, increasing (e.g., ensuring) the durability of the chain (i.e., tamper-evident). In addition, in some embodiments, there is a JobManager that manages concurrent operations on the data structure to pass operation time.

[0087] In some embodiments, DocuSocket is a software library that maintains a connection with a client (e.g., a remote computing device attempting to access data stored in a database). The connection from the client may be relatively stateless (in some cases, after the session is confirmed, there is no other state). In some embodiments, the connection can accept any number of requests in any order, and will return responses to the requests in any order. For example, the connection may receive write (A), write (B), read (C) operations, and in some embodiments, DocuSocket may respond to request C before A or B. However, once the transaction response for A or B arrives, in some embodiments, those operations are considered final and are committed to the chain and can be queried immediately. This "always online" and highly available connection is a differentiating factor that enables DocuSocket to outperform standard blockchains in comparable benchmarks. (That said, since various independently useful inventions are described, not all embodiments provide these benefits, and some of those inventions may be used for other purposes.)

[0088] In some embodiments, write requests are accepted by Docusocket and passed to the BlockManager for writing to a MerkleTree object. In some cases, the result of this operation is a hash consisting of the position in the MerkleTree, the hash from the previous block, and the contents of the data written; this is called a transaction ID (TXID), which is an example of a node identifier or pointer (this does not mean that these or other categories are disjoint). In some embodiments, these TXIDs are stored in a backup data store for later reference and retrieval, for example, in a low-trust database 14 instead of the data they point to, or in a low-trust file system instead of a document referenced by a TXID. In some cases, a TXID is a fragment pointer to a fragment pointer and a linked list of fragments encoding the stored data.

[0089] As mentioned earlier, early blockchain databases were able to store binary large objects, but the way data was stored and accessed often resulted in severe performance penalties when accessing larger data sets (e.g., large binary large objects). Such systems often rely on additional functionality originally designed to handle the transactional nature of Bitcoin and other cryptocurrencies. This additional functionality is often provided by an "in-chain" scripting language that defines transactions that go into the chain. For many use cases, this is a secure and efficient way to maintain the ledger, but for larger chains, these features come at a significant time complexity cost, especially in write operations. As these chains grow, they slow down significantly. Blockchain document databases that store documents in-chain are expected to grow very quickly, thus imposing significant limitations on what is actually possible with existing technology.

[0090] In contrast, because some embodiments of document chains store data directly in a Merkle tree in its exact representation (which should not be interpreted as excluding various transformations and may include storing a compressed encrypted copy of the data), these embodiments are expected to circumvent the need for an in-chain scripting language and be able to provide O(log(n)) get / put operations on the tree once it is loaded into memory. In addition, storing the data submitted to the database in a Merkle tree, rather than just a hash digest, is expected to prevent (and in some cases defeat) hash collision attacks on the stored data. In such an attack, malicious content is selected and designed to produce the same hash as the stored data, and the malicious data replaces the reliable data in the database. In a traditional system, the Merkle tree would produce a hash value that verifies the malicious content as reliable. In contrast, some embodiments circumvent this attack vector by storing the data submitted to the database in a Merkle tree. In addition, unlike many systems that only store hash digests of binary large objects in the chain, some embodiments can detect changes without continuously verifying each binary large object with its corresponding hash digest. That said, not all embodiments offer these benefits, for example, some embodiments may avoid the use of traditional on-chain scripts to access hash digests in the chain without having to store the entire document, thereby speeding up operations.

[0091] These libraries enable Figure 2 An example of the process shown in Figure 2 An example of a process 50 that may be implemented to store and access a document in an immutable, tamper-evident data structure is shown. Figure 3 An example of this data structure is shown in . In some embodiments, process 50 may be implemented in computing environment 10 as described above, but it should be emphasized that the embodiments are not limited to this implementation and this does not mean that any other description is limiting. In fact, some embodiments may implement a version of the process shown on a single computing device as part of a monolithic application, such as a file manager that stores documents locally in the manner described.

[0092] In some embodiments, the operations of process 50 and other processes described herein may be implemented in an order different from that described, may include repeated steps, may include steps performed simultaneously, may include steps that are omitted, or may be otherwise arranged differently from the exemplary arrangements described, without implying that the other descriptions are limiting. In addition, the operations of process 50 and other processes and functions described herein may be implemented by executing instructions stored on a tangible, non-transitory machine-readable medium using one or more processors, such that when the instructions are executed, the described functions are implemented. In some cases, although a single medium is used, the operations may be stored in a distributed manner on multiple media, for example, storing different subsets of the instructions in the memory of different computing devices that execute those different corresponding instructions, which are arranged consistent with the term single "medium" used herein. In addition, the storage of data need not be persistent in the absence of power, and may be implemented by persistent data storage or dynamic data storage, such as dynamic random access memory that may dissipate the stored data when power is removed.

[0093] In some embodiments, process 50 includes receiving a write command that requests that a document associated with the write command be stored in an immutable data structure, such as indicated by block 52, such as a tamper-evident immutable data structure. In some cases, the write command may be an SQL command received from a secure driver 30 for an application 28 to write to the low-trust data store 14 described above, or the command may be a command to store a file or other document in a file system. For example, in some cases, the write command may be a request received by a file manager of an operating system to store a file in a repository represented in the form of network drivers within a user interface of the operating system, where some or all of the network drivers are hosted in an immutable data structure, such as described below with reference to FIG. Figure 3 and Figure 5 In some embodiments, the write command may be generated programmatically or, in some cases, in response to a user input requesting, for example, to store a document. As noted above, the term "document" is used relatively broadly herein and may include individual bits of stored information or larger binary large objects, such as files ranging up to multiple gigabytes or larger. A file may include stored data associated with metadata, such as the author, file size, creation date, modification date, file name, extension, and the like. Such files may be stored with Windows TM 、Android TM 、iOS TM , Linux TM or Mac TM The file definition in the operating system is consistent.

[0094] The term "immutable" in the phrase "immutable data structure" refers to a data arrangement that the computing system and write correct commands are configured to remain in place even after the information represented by the data has changed. For example, the data may represent a user's phone number, and an embodiment using an immutable data structure may write a new record indicating the new phone number instead of overwriting the existing record. Therefore, even after the value has changed, both the older and newer versions are retained (and may be marked as older or newer). Such an embodiment may then reference the latest representation of the value of the field to implement the change instead of deleting the old instance. In some embodiments, an immutable data structure may be a tamper-evident data structure that is computationally unmodifiable in the case where the modification is undetectable by the party auditing the data structure. Some embodiments may implement the cryptographic hash pointers described below in a directed acyclic graph, which makes it computationally impossible for a party to modify the stored data without making the modifications apparent due to inconsistent hash values ​​elsewhere in the data structure. The feasibility of the computation may depend on the use case, for example, whether the economy and time are available to provide a certain amount of competitive resources for implementation. In many cases, operations that require an average of more than 10^128 hash values ​​to create conflicts with the changed data are computationally infeasible.

[0095] In some embodiments, process 50 may include forming a node of a directed acyclic graph having node content including a document, as shown in block 54. Forming a node may include accessing existing nodes and inserting node content into those nodes, or forming a completely new data structure by which the node is encoded. In some embodiments, each node may include node content having a set of attributes, and in some cases, an attribute may include a pointer to another node. In some embodiments, those node attributes may include a pointer as an identifier of an adjacent node in a directed acyclic graph and a cryptographic hash value based on one or more attributes of the identified adjacent node. In some embodiments, these last two pairs of attributes (an identifier of another node and a cryptographic hash value of at least some of the content of the node) may correspond to a cryptographic hash pointer from one node to another. In some embodiments, a cryptographic hash pointer may define an edge of a graph. In some embodiments, a single node may contain zero cryptographic hash pointers (e.g., a leaf node in a binary tree), a single cryptographic hash pointer (e.g., a link in a blockchain or a linked list), a cryptographic hash pointer pair (e.g., in a non-leaf node of a binary tree directed acyclic graph), or other various numbers of cryptographic hash pointers, such as in a splay tree or a skip list.

[0096] In some embodiments, the node attribute includes an identifier of the corresponding node. In some embodiments, the identifier can be a fragment identifier of the type described below, or in some cases, the identifier can be an identifier within the namespace of a directed acyclic graph, for example, a count that increases with each node. In some embodiments, the identifier can be arranged hierarchically, for example, indicating blocks and blockchains, and then specifying a sequence of binary values ​​that pass through a binary tree as used in prefix trees in some cases (for example, the value 00101 can define a path from the root node to left, left, right, left, and then right, where, in a binary tree, left is represented as 0 and right is represented as 1). In some embodiments, the identifier of the node is content-based, for example, based on a cryptographic hash value or a non-cryptographic hash value of one or more or all attributes of the node, which attribute may include a hash pointer for the node.

[0097] In some embodiments, the document may be stored in a single node of a directed acyclic graph. Or in some cases, for example, according to the following reference Figures 4 to 6 The technology described above, documents can be fragmented and stored in multiple nodes. In some embodiments, some nodes of the graph can store other documents or parts of other documents. In some embodiments, nodes can be added to the graph over time, and in some cases, groups of nodes are added sequentially. In some embodiments, the attributes of a node or a block of nodes (which can themselves be characterized as nodes in some cases) can further include a timestamp when the node is formed, an identifier of a tenant account or a data repository, a date when the node was created, etc., in which the data in the annotation is stored in a lower trust data store 14.

[0098] In some embodiments, groups of nodes may be added as "blocks," e.g., each block corresponding to a binary tree storing documents in leaf nodes. Or, for example, a block may be a linked list, a skip list, a splay tree, or a combination thereof. In some embodiments, a single node may store multiple documents as attributes of the node. In some embodiments, a block has an integer index, a block capacity, a cryptographic hash based on all nodes in the block (e.g., a Merkle root), nodes within the block, and a cryptographic hash block based on the contents of previous blocks (e.g., based on all values ​​in the block, based on the Merkle root of the block, etc.).

[0099] In some embodiments, forming nodes of the directed acyclic graph includes forming (e.g., updating or creating) a sequence of nodes along one or more paths through the directed acyclic graph. In some embodiments, this may include calculating one or more cryptographic hash values ​​to form one or more cryptographic hash pointers along the path, which in some cases may include or terminate at a node storing a document or a portion of a document, such that each cryptographic hash pointer along the path is based on the document.

[0100] A cryptographic hash pointer can be based on a cryptographic hash function that can take multiple inputs, such as one or more node attributes, and produce a fixed-size output. These functions can have pre-imaging resistance, second pre-imaging resistance, and collision resistance. Examples include SHA-256, BLAKE, BLAKE2, SHA-1, SHA-2, and SHA-3 hash functions. In some embodiments, a cryptographic hash function can be a one-way function in which a given input string deterministically produces an output string that is relatively difficult or impossible to reversely determine the input from the output, while relatively easy to confirm that the input corresponds to the output. For example, it is computationally infeasible to identify a hash collision in which different instances of the input produce a given output. In some embodiments, a cryptographic hash function can implement structure.

[0101] In some embodiments, the cryptographic hash function may be based on a compression function that accepts a fixed-size input and produces a fixed-size output using a one-way compression function. In some embodiments, because the input to the cryptographic hash function may be of various sizes, the transformation may be performed in various iterations and multiple iterations. Some embodiments may determine the length of the input accepted by the one-way compression function (e.g., the number of bytes), the length of the output of the one-way compression function, and determine the difference between the two lengths. Some embodiments may then parse the input to the cryptographic hash function into a sequence of size equal to the difference, and iteratively input the parsed sequence into the one-way compression function, and then combine the output of the iteration with the next decomposition portion from the input (e.g., increasing from the beginning of the input to the end), and perform an iterative cascade, such as prepending or appending or otherwise mixing the output of the previous iteration of the one-way compression function with the next parsed portion of the input of the cryptographic hash function. Some embodiments may repeat this process until the end of the input to the cryptographic hash function is reached (e.g., reaching a certain threshold number of bytes or a point within the last byte), as indicated by block 56. In some embodiments, for example where multiple inputs are applied, some embodiments may combine these inputs in various ways, such as prepending or appending or otherwise mixing these inputs to form a string, on which these operations may be performed to produce a fixed-size output based on the entirety of the inputs. The resulting directed acyclic graph may be stored in a memory.

[0102] In completing the Figure 2 Before discussing Figure 2 An example of a directed acyclic graph used in the process would be helpful. Figure 3An example of a directed acyclic graph 70 according to the present technology is shown. In some embodiments, the directed acyclic graph 70 may include multiple blocks in a sequence, such as blocks 72, 74, and 76, which may be arranged in a linked list, wherein the links formed by the cryptographic hash pointers correspond to the arrows shown between the blocks shown. In some embodiments, blocks 72, 74, and 76 may be added sequentially over time, for example as each block is completed. In the example shown, block 72 may be the latest block, and block 76 may be the earliest block. In some embodiments, the cryptographic hash pointers between each block may be based on the node content in the previous block, and the node content may include a cryptographic hash pointer based on the node content in the previous block to the block. Therefore, the cryptographic hash value of the encrypted block 72 may be based on the cryptographic hash value of each previous block and all content that is presented as tamper-evident through the data structure.

[0103] In some embodiments, each block may include a binary tree with a root node 80. In some embodiments, each arrow between blocks in the binary tree of each of blocks 72, 74, and 76 may also be, for example, a cryptographic hash pointer based on the identifier of the node pointed to by the cryptographic hash pointer and a cryptographic hash value based on the node content of the node, which may include the attributes of the node, which itself is the cryptographic hash value of another hash pointer. Therefore, in some cases, the cryptographic hash value of the root node 80 may be based on the node content of each node of the binary tree in each block 72, 74, or 76. In some embodiments, the node 80 may include three such hash pointers corresponding to six node attributes, which include three pairs of node identifiers and cryptographic hash values ​​based on the node content of those nodes. In some embodiments, the node content may further include a cryptographic hash value based on each of these values, or the cryptographic hash value may be stored in another node pointing to the node. The illustrated graph is acyclic. As used herein, the term may refer to a non-cyclic subset of a larger cyclic graph. Therefore, the requirement for acyclic directed graphs may not be avoided simply by adding unused cycles.

[0104] In some embodiments, once a given block 72, 74 or 76 is identified, the binary tree structure can facilitate relatively fast access to records within a given binary tree. In some embodiments, in order to speed up access to a particular block, some embodiments may include a skip list data structure, which, for example, has a value within another node or node 80, which includes a cryptographic hash pointer pointing to a certain number of earlier blocks in the sequence, pointing to a block four positions earlier, another pointing to a block eight positions earlier, another pointing to a block 16 positions earlier, another pointing to a block 32 positions earlier, etc. Therefore, some embodiments can skip some parts of the block sequence to access a specified block. Three blocks 72, 74 and 76 are shown, but it is expected that commercial embodiments will substantially include more blocks in a given directed acyclic graph 70, such as more than 50, more than 100, more than 500 or more than 10,000 blocks. In some embodiments, the earliest block 76 may include a cryptographic hash pointer to a seed node 78, which may include as the node content a random value (e.g., a pseudo-random value) longer than 64 bytes as a property that provides a minimum level of entropy for each subsequent cryptographic hash value that directly or indirectly points to node 78.

[0105] In the example shown, blocks 72, 74, and 76 include a binary tree with three levels of a hierarchical structure, but it is contemplated that embodiments include more levels of a hierarchical structure, such as four levels, five levels, six levels, seven levels, eight levels, or more, each level containing twice as many blocks as the previous level in the hierarchy. In some embodiments, the binary tree can be a balanced binary tree, or in some cases, the binary tree can be unbalanced. In some embodiments, nodes in the binary tree below the lowest level (e.g., nodes 80, 84, and 86 that are non-leaf nodes) can include node content based on cryptographic hash pointers but not based on node attributes in those nodes that store document content, or in some cases, documents may also be stored in these nodes. In some embodiments, the content of a document, such as the document itself or a portion of a document, can be stored in attributes of the node content of leaf nodes 88, 90, 92, and 94 (e.g., in some cases, a single leaf node among these leaf nodes). Thus, for example, a given document stored in leaf node 90 as an attribute in the node content of leaf node 90 may cause the cryptographic hash values ​​in the cryptographic hash pointers of nodes 84 and 80 to be based on the document. As a result, a modification of the document stored in node 90 may cause the cryptographic hash pointers in nodes 80 and 84 to be inconsistent with the document, because it is expected that the cryptographic hashing of the modified document will not produce the same cryptographic hash values ​​in the cryptographic hash pointers of these nodes 80 and 94. In some embodiments, the verification operation can be repeated along the sequence in the path of such nodes connected by the cryptographic hash pointer, in some cases, the cryptographic hash pointer from or to any leaf node until it is added to the latest root node of the directed acyclic graph 70, thereby relatively finely identifying nodes with data that has been modified from the workload application, and since it is computationally infeasible to elaborately create hash collisions consistent with modifications to the data along the path, the data is relatively reliably made tamper-evident. In some cases, the leaf nodes can be arranged based on content, for example, each leaf node stores a range of values ​​from left to right, or each leaf node stores a threshold amount of data arranged at most alphabetically, chronologically, or in some other arrangement.

[0106] In the directed acyclic graph 70 shown, a sequence of binary trees is shown, but embodiments are according to directed acyclic graphs of other types. In some embodiments, a directed acyclic graph is a graph without unconnected subgraphs. In some embodiments, a directed acyclic graph is a single binary tree, or in some cases, a directed acyclic graph is an octree with a binary tree or a binary tree with an octree. In some embodiments, a directed acyclic graph includes a skip list of an octree or a binary tree. In some embodiments, a directed acyclic graph includes an octree of a skip list or a binary tree or a linked list. (Some embodiments can promote the rearrangement of trees and other structures to facilitate faster access using the summary cryptographic hash function described below, or hash pointers can be recalculated when reconstructing).

[0107] Thus, the directed acyclic graph 70 may include a plurality of nodes, some of which have node content including cryptographic hash pointers corresponding to edges in the directed acyclic graph 70, and some of which include node content including workload application data protected by the directed acyclic graph 70 as attributes.

[0108] In some embodiments, directed acyclic graph 70 may take the form of a distributed ledger, such as a blockchain having a linked list of blocks, where each block comprises a Merkle tree having a Merkle root that serves as a node attribute for the corresponding block, and subsequent blocks have node attributes that include cryptographic hash values ​​based on the Merkle root of each prior block.

[0109] In some embodiments, in some cases, adding a node or a set of nodes (e.g., a block) to a directed acyclic graph 70 can be implemented by an untrusted computing device in an untrusted computing environment based on a consensus protocol (e.g., Paxos, Raft, etc.), where different computing devices perform or demonstrate some difficult-to-fix proofs, such as proof of work or proof of storage, for example, by calculating hash collisions as a threshold number of prefix or suffix digits of a cryptographic hash value. For example, in some embodiments, a collection of untrusted computing devices can execute a consensus algorithm to determine whether a block should be added or whether a block is valid. Or to speed up operations, some embodiments can avoid the overhead associated with proof of work or proof of storage techniques by executing in an environment with more control over the hardware performing the operations described herein.

[0110] It should be emphasized that as long as the data structure includes links between data elements present in the graph, it is not necessary to mark the data structure as a graph or have nodes or edges in the program code to constitute the graph. In some cases, the graph can be encoded as a list of links between data elements, for example in the form of key value pairs or in a hierarchical data structure (for example, in a hierarchical data serialization format such as JOSN or XML). The graph can also be encoded as a relationship in a relational database. In addition, multiple data structures such as different graphs can overlap each other, while still retaining the existence of different graphs, for example in the manner described below.

[0111] In some embodiments, each of the above-mentioned storage computing nodes 26 can store Figure 3 One or more instances of the directed acyclic graph 70 shown. In some embodiments, a given directed acyclic graph may be replicated in multiple instances on multiple storage computing nodes 26, such as in multiple data centers 24. In some cases, an odd number of replicated instances may be stored. When reading back data, some embodiments may implement a consensus protocol to determine an authoritative copy, which identifies, for example, a version of a document returned by multiple or a majority of replicated instances after a read operation as described below.

[0112] In some embodiments, a given directed acyclic graph 70, which may be replicated in multiple instances, may store data from a single workload application from a single tenant, and in some cases, may store a subset of the data from a single workload application, or the data may be intermixed. In some embodiments, a given directed acyclic graph may store data from multiple applications of a given tenant. In some embodiments, depending on the implementation, a given directed acyclic graph may store data from multiple tenants.

[0113] In some embodiments, as data accumulates, when the directed acyclic graph 70 exceeds a threshold size, a new directed acyclic graph may be instantiated, for example, using a cryptographic hash value of the root node of the last block added to the previous directed acyclic graph as a seed value 78 in the seed node.

[0114] therefore, Figure 3 shows an example of a directed acyclic graph where the Figure 2The process of storing documents is stored in an immutable data structure in a tamper-evident manner. When a document is revised, a new record can be added to the data structure 70 along with the new version of the document, and pointers to the latest version of the document can be updated, for example, in the low-trust database 14 to reference those newer versions. In some embodiments, these pointers can implement the techniques described below, by which pointers can replace records in the data store. In some embodiments, those pointers can point to a single node in a single directed acyclic graph. In some embodiments, the pointers are node identifiers or fragment identifiers described below.

[0115] In some embodiments, the transformer 20 described above may maintain an index in memory that maps pointers to uniform resource identifiers of storage computing nodes 26 and directed acyclic graphs maintained by the storage computing nodes, and in some cases, the DNS 18 may map those uniform resource identifiers to Internet Protocol addresses and port numbers of corresponding storage computing nodes 26. Thus, after a document is written to the directed acyclic graph 70, a pointer (e.g., a node identifier that distinguishes a node from all other nodes accessible to a given application, tenant, or all tenants) may be stored in the low-trust database 14 or other data store in place of the document.

[0116] In some embodiments, the internal consensus of the directed acyclic graph 70 may be periodically verified to detect instances of tampered documents. Some embodiments may recalculate encrypted pointers along various paths, for example, each time a new document or other record is added to the directed acyclic graph 70, such as each time a new block is added. Upon detecting that an encrypted hash pointer does not correspond to a path of a document or other node content in the path, some embodiments may designate the document or other value as having been modified. Some embodiments may further issue an alarm in response to detection of tampering, such as sending an email, sending a text message, and suspending various user accounts, such as users who have accessed the document. In order to detect tampering, some embodiments may recalculate the encrypted hash value of each encrypted hash pointer along each path to each document, and determine whether the recalculated encrypted hash values ​​match those stored as node attributes of the node storing the corresponding encrypted hash pointer. Thus, if a given document in, for example, node 90 of block 72 is tampered with after being stored, the cryptographic hash pointer from block 84 to block 90 will not match the cryptographic hash value stored in the node attributes of node 84, nor will the cryptographic hash pointer from block 80 to block 84 match the stored value in node 80. Furthermore, if additional blocks are added after block 72, the cryptographic hash pointers between blocks after block 72 will also not match, thereby providing a traceable path from the latest node to the node in which the attacker tampered with the node content. In some embodiments, these verification operations may be performed each time a node is written, each time a block is written, periodically (e.g., hourly, daily, or weekly), or each time a read operation is performed.

[0117] Figure 2 Further included is an operation by which the document can be read back from the secure distributed storage 16. In some embodiments, the operation of process 50 can include receiving a read command requesting retrieval of a document from the immutable data structure, as shown in block 60. In some cases, the read command can be received after a write command (e.g., substantially later, such as more than an hour, a day, or a week later). In some cases, the read command can be received after multiple write commands for the same document, in which different versions are written to different nodes in different blocks, and in some cases are written as described above with reference to FIG. Figure 3Different directed acyclic graphs described. In some embodiments, the read command can reference an identifier of a document that indicates the latest version of the document to be retrieved, or in some cases, the read command can reference a specific version of the document. In some cases, receiving a read command can cause the security driver 30 to access the low-trust database 14 or other low-trust data store and retrieve a pointer to a node or sequence of nodes in which the specified document is stored. As described above, in some cases, the pointer can be a value mapped to a uniform resource identifier, and the DNS 18 maps the uniform resource identifier to an Internet Protocol address and port number, where the corresponding node is stored in the corresponding directed acyclic graph.

[0118] Next, some embodiments may access one or more nodes of the directed acyclic graph to retrieve the document, as shown in block 62. In some cases, this may include: sending a pointer to a node storing at least a portion of the document to the converter 20; accessing the uniform resource identifier of the directed acyclic graph in which the node resides with the index stored by the converter 20; converting the uniform resource identifier to an Internet Protocol address and port number pair using the DNS 18; requesting the corresponding node from the storage computing node 26 that hosts the specified directed acyclic graph at the Internet Protocol address and port number; and returning the requested document to the requesting application 28.

[0119] Additionally, in some cases, the accessed document may be verified. For example, some embodiments may recalculate cryptographic hash values ​​based on the path of the document along the most recent node of the directed acyclic graph back to the document (or a subset thereof), and determine whether any of the calculated cryptographic hash values ​​do not match those hash values ​​stored in the corresponding nodes of the directed acyclic graph, as shown in block 64.

[0120] For clarity of terminology, it should be noted that storage computing nodes 26 refer to computing entities (eg, service topology), while the nodes of directed acyclic graph 70 refer to elements of a data structure.

[0121] Embodiments may determine whether these calculated values ​​are internally consistent, as shown in box 66. This may include determining whether the calculated value matches the value stored in the node pointer of the directed acyclic graph 70. When it is determined that the values ​​match, some embodiments may return the requested document, as shown in box 69. When it is determined that the values ​​do not match, some embodiments may indicate that the document has been modified, as shown in box 68. Therefore, the entity retrieving the document can have a certain degree of confidence that the document has not been tampered with. In addition, because some embodiments store documents in nodes, documents can be retrieved relatively quickly (e.g., within 200ms, 100ms, or 50ms) even when storing relatively large amounts of data (e.g., more than 100GB or more than 10TB). Although the embodiments are not limited to systems that provide these benefits, independently useful techniques are described, which does not mean that any other description is limiting.

[0122] In some embodiments, the documents or other values ​​of the secure distributed storage 16 written into the nodes of the directed acyclic graph 70 can be subjected to various types of encoding before the write operation. In some embodiments, the data can be encoded with redundant information so that if some data is modified before or after writing, the encoding reveals errors, and in some cases provides enough redundant information so that the data can be recovered. For example, some embodiments can encode the data with Hamming codes or Turbo codes so that if a subset of the data is lost, the document can be reconstructed based on the redundant information in another part of the document. In another example, data can be compressed, for example, using entropy coding, to facilitate reduced use of bandwidth and faster retrieval. In some embodiments, the sequence of values ​​in the document can be analyzed to identify the frequency of occurrence of the sequence, and those sequences can be mapped to a dictionary, in which relatively frequent characters are represented by relatively short values ​​in the dictionary, and longer less frequent sequences are mapped to longer values. The mapping and dictionary can then be stored in a secure distributed storage in a compressed format. In some embodiments, after compression, some embodiments may also encrypt the data, for example using a symmetric encryption algorithm in which a key is maintained by the client computing device 12, for example by XORing the document with a random string that serves as a symmetric key. In some cases, the symmetric key may be stored in the distributed storage 16 in the manner described above.

[0123] Fragment data for persistent storage purposes across multiple immutable data structures

[0124] As mentioned above, traditional databases are not sufficient to prevent threat actors or internal resources (employees, information technology personnel, etc.) from tampering with data. Data residing in a single logical space provides an easy target for threat actors. Once the computing device storing the data is compromised, the entire data is potentially at risk. Similar problems arise when a single computing device controls access to all other computing devices in a distributed master-slave architecture, because once the master device is compromised, the data on other devices may be exposed.

[0125] There are techniques for storing data in a distributed form in a peer-to-peer network, but such systems often have other problems. Examples include various systems built around the BitTorrent protocol. These systems typically operate without a central authority being compromised. But in these systems, when or if data is altered (perhaps for the purpose of hiding or obfuscating the changes), there are no resources or methods to detect these changes. In addition, many distributed peer-to-peer storage networks (such as many networks that implement the BitTorrent protocol) maintain a complete copy of a file (or other binary large object) at each peer, potentially leaving the complete file open to inspection by untrusted parties.

[0126] To mitigate these and other issues, in some embodiments, when writing data, a component called an "arbitrator" (which may be a piece of middleware and may be implemented in transformer 20) is able to take as input fully formed material starting from the last byte of the data (e.g., a unit of content, such as a value in a document or database), the fully formed data (e.g., a unit of content, such as a value in a document or database), divide the data into N fragments (where N is an integer greater than 1, greater than 5 or 10 in many business-related use cases, and in some cases may be related to the size of the initial data), and place each fragment on a physically (or virtually) independent blockchain-enabled storage data structure, each fragment containing a pointer to the next storage location of the fragmented data. In some embodiments, the arbitrator (or other middleware) then returns the TXID (or other node identifier) ​​of the last written fragment (the first byte) to the application requesting the fragmentation of the data.

[0127] In some embodiments, when an application or resource requests to reassemble fragmented data, the arbitrator (or other middleware) is provided with the TXID (or other node identifier) ​​of the first byte of the data. In some embodiments, after reading the first byte, the arbitrator or middleware then reads subsequent pointers until a null character or end-of-sequence character is read. Once all fragments have been read into memory, the arbitrator or middleware responds to the application with the synthesized non-fragmented data.

[0128] In some embodiments, other data structures, such as files or databases, may be used to store the fragments. In addition, some embodiments pre-process the data and count the number of required fragments from the beginning before fragmenting the data.

[0129] In some embodiments, another enhancement is to segment the data into a tree-like structure rather than a list-like structure, which is expected to accommodate concurrency during read operations and reduce operation time.

[0130] In some embodiments, when discrete data is read or written, the execution operation may be written to an "auxiliary" chain along with other metadata such as user credentials in order to audit access to the discrete data.

[0131] Advantageously, the blockchain data structure may deter or prevent modification of stored data fragments. In addition, some embodiments may store data across fragments with a certain degree of redundancy, for example utilizing Hamming codes or Turbo codes, so that even if one blockchain is compromised, fragments from other changes will be inconsistent with the modified data and signal the compromise. In some embodiments, it is expected that fragmentation of data will facilitate concurrent data verification on different chains, thereby speeding up read operations. In addition, it is expected that fragmentation of data will make the data more robust to attacks on a single peer machine because even if the peer machine is compromised, only a relatively small fragment is at risk. That said, not all embodiments provide these benefits, as multiple independently useful inventions are described, and various engineering and cost trade-offs may result in performance advantages in one aspect being deployed to improve another, for example, using a faster algorithm to accommodate slower and cheaper hardware.

[0132] To implement these and other techniques, some embodiments may perform Figure 4 The process 100 shown in , to fragment a document and distribute the fragments across multiple storage computing nodes 26 having multiple different directed acyclic graphs, in some cases, each of these directed acyclic graphs is replicated on multiple storage computing nodes. In some embodiments, the stored data can be a single value and a single field having portions of a record of a database, such as a value at a given row and column position in a given table. In some cases, the data stored as a unit can include metadata, such as a master key, a table identifier, a database identifier, and a tenant identifier. In some embodiments, the stored data can be a document such as the above-mentioned document that is larger than a single value.

[0133] In some embodiments, process 100 may include receiving a write request to write a value to a database, as shown in block 102. In some embodiments, this operation may be performed by the above-described secure driver 30. In some embodiments, the write request may be a request to write a value having a structured query language statement to a low-trust database 14, or a request to write a file to a file system. Alternatively, in some cases, the value subject to subsequent operation may be a value already stored in a low-trust database 14 or a file system, for example, a value processed during an installation process, by which the low-trust database 14 is protected by moving data into the above-described secure distributed storage 16.

[0134] Next, some embodiments may classify values ​​as higher or lower security, as shown in box 104. For example, a write request may specify that a given record is added to a new row in a table, and the record may specify a tuple in which a value is assigned to each of several fields, each corresponding to a different column in the table. In some cases, some of these fields may be of lower security, while other fields may be of higher security. In some embodiments, the security driver 30 may include a table that maps table / field combinations to security specifications, such as a binary value indicating whether a value is a lower security value or a higher security value. Or some embodiments may use other types of data security policies described below. For example, a "social security number" field may be a higher security value, while a username in a public-facing account may be a lower security value. In another example, credit card information (e.g., a credit card number) may be a higher security value, while a zip code in a mailing address may be a lower security value.

[0135] Next, some embodiments may cause the database to store the lower security value, as shown in box 106. In some cases, this operation may include modifying the SQL statement or other database command received by the security driver 30 from the application 28, and providing the modified command to the database driver 32. For example, some embodiments may modify a portion of the received command that would otherwise specify that a higher security value is written to the low-trust database 14. In some cases, the modification takes the form of modifying the value itself rather than the reserved item in the command, so that the modified value is written to the location of the low-trust database 14 where the corresponding higher security value would otherwise be written by the application 28 (thereby maintaining the relationship stored in the database). Or a pointer may be written to a directory in the file system in place of a file or other document. For example, some embodiments may cause a node identifier or other pointer to a node to be written, such as an identifier of a fragment (e.g., a first fragment) of the data to be written formed in the operation described below.

[0136] Next, as shown in block 108, some embodiments may determine whether there are more higher security values ​​identified to be processed. In some cases, some embodiments may iterate through each higher security value, or process some or all of the higher security values ​​simultaneously, and operation 108 may determine whether the entire set has been processed. After determining that all higher security values ​​that have been classified have been processed, some embodiments may return to block 102 and wait for the next write request. Alternatively, some embodiments may select the next value of higher security, as shown in block 110, for subsequent operations.

[0137] Next, some embodiments may fragment the current value into multiple fragments, as shown in block 112. As described above, the current value may in some cases be a single value for a single field of a single row of a database, or in some cases, the value may include additional information (e.g., the entire row) or a value associating the value in the database with the table and the primary key of the row. Or in some cases, the value may be a document similar to the above-described document, which may include additional data.

[0138] Next, some embodiments may fragment the current value into a plurality of fragments, as shown in block 112. In some cases, the value may be fragmented by character position in a sequence or byte position in a byte sequence, for example, each sequential byte or each four sequential bytes in a serialized representation of the value is fragmented. In some embodiments, the fragmentation may include fragmentation using the entropy coding techniques described above. For example, some embodiments may perform fragmentation according to a dictionary encoding in an entropy encoding of the value, for example, each dictionary entry corresponds to a different fragment. In another example, the fragmentation may include an operation of dividing the value into a plurality of fragments, each fragment containing a non-contiguous portion of the value, for example, placing every fifth byte into a first fragment starting from a first byte, placing every fifth byte into a second fragment starting from a second byte, placing every fifth byte into a third fragment starting from a third byte, and so on. In some embodiments, the value may be fragmented into a specified number of fragments, for example, dividing the value into quarters so that one quarter of the bytes used to encode the value go into a first fragment, one quarter go into a second fragment, one quarter go into a third fragment, and one quarter go into a fourth fragment.

[0139] In some embodiments, the value can be fragmented using techniques from network coding (e.g., linear network coding) so that the resulting fragments are combined in a bit-by-bit manner to reconstruct the value. For example, a given value of 64 bytes in length can be fragmented into two 64-byte fragments, which, when combined together and XORed by bit, produce the original value. In some embodiments, some fragments may contain redundant information relative to other fragments. For example, some embodiments may encode data using Hamming codes or Turbo codes before or during fragmentation so that if two of three consecutive fragments are available, the information and another fragment can be recreated, or the encoding may be applied using less redundant information so that an error in one fragment makes the fragment inconsistent with the redundant information in the adjacent consecutive fragments. In some embodiments, the fragments may be represented as sequences, or in some cases, the fragments may be represented in other data structures. For example, in some cases, the fragments may be represented as a set of paths by a prefix tree.

[0140] Next, as shown in block 114, some embodiments may cause a database such as the low-trust database 14 to store a pointer to the first fragment. In some cases, the operation may also include causing a record to be added to an index maintained by the converter 20, the index being associated with the pointer, the index being a unique fragment identifier, which in some cases may be an identifier of a single node in a data structure such as the directed acyclic graph 70 described above, or a single attribute of a node in the data structure having an address of the directed acyclic graph. In some cases, the association may be with multiple addresses, each corresponding to a different replica instance of the directed acyclic graph, such as multiple different uniform resource identifiers mapped by the DNS 18 to the Internet Protocol address and port number of the corresponding storage computing node 26, the storage computing node 26 storing those different instances of the directed acyclic graph, and in some cases performing the above operations, by which the node is formed and the data is verified. In some embodiments, causing the database to store a pointer to the first fragment may include performing the above operations, by which the write command received from the application 28 is modified to replace the higher security value with the pointer before providing the command to the database driver 32.

[0141] Some embodiments may iterate through each fragment so that each fragment is stored and determine in each iteration whether there are more fragments to be processed, as shown in block 116. When it is determined that more fragments of a given value are to be processed, some embodiments may select the next fragment, as shown in block 118. Then, some embodiments may select a computing device, such as a computing device that executes one of the above-mentioned storage computing nodes 26 to store the current fragment, as shown in block 120. In some cases, a list of storage computing nodes 26 may be maintained in a circular linked list, and some embodiments may maintain a pointer that advances with each storage operation, loop the pointer in the list, and use the pointed storage node to store the current fragment. Or some embodiments may select, for example, a storage computing node 26 with a random number or with a pseudo-random value generator (such as a linear shift register). In some embodiments, a storage computing node for storing a given fragment may be selected based on a unique identifier assigned to the fragment. In some embodiments, the unique identifier may be a hash value based on the content of the fragment, and a prefix or suffix of the hash may uniquely identify one of the storage computing nodes 26. In some embodiments, such a prefix or suffix may uniquely identify a directed acyclic graph 70 that may be replicated on multiple storage computing nodes. In some embodiments, the content of the fragment itself can be hashed, and the prefix or suffix can specify one of the above-mentioned storage computing nodes 26 or an instance of a directed acyclic graph that can be replicated on multiple storage computing nodes 26. For example, 64 different directed acyclic graphs can be maintained, each of which is repeated three times, and the last six digits of the binary representation of the hash of the fragment content can specify a value between 0 and 63, and this value can be used as an identifier of one of the directed acyclic graphs in which the fragment is to be stored. In some embodiments, selecting a directed acyclic graph can cause one or more (e.g., three) computing devices to be selected, and those are the computing devices that store and maintain the corresponding directed acyclic graph.

[0142] Next, some embodiments may instruct the selected computing device to store the fragment in memory in association with a pointer to a previous fragment, as shown in block 122 (except for the first fragment, which may not have a pointer to a previous fragment). In some embodiments, this may include the above-described transformer 20 sending an instruction to each of the odd set of storage computing nodes 26, the instruction copying one of the directed acyclic graphs 70 to form a node in which the fragment content and the pointer are used as one or more attributes of the node, such as one of the leaf nodes 88 to 94 described above. In some embodiments, forming the node may further include forming a cryptographic hash pointer to the node via a data structure of the type described above, via which a different directed acyclic graph may be formed, such as via a higher level of a hierarchy in a binary tree, and in some cases, adding an additional block to a linked list of blocks, each block having a different binary tree and linked to a previous block according to a cryptographic hash pointer as described above.

[0143] Next, some embodiments may return to determine if there are fragments associated with the current value. After determining that there are no more fragments, some embodiments may return to block 108 and determine if there are more high security values ​​to process. In some embodiments, multiple values ​​may be processed simultaneously. In some embodiments, values ​​may be fragmented from front to back or vice versa.

[0144] Figure 5 An example of a resulting data structure 130 is shown, where multiple fragments are stored in different directed acyclic graphs 132, 134, and 136, each of which has a reference to Figure 3 In some embodiments, a fragment of a given value may be stored in a content graph overlaid on each of these directed acyclic graphs 132, 134, and 136, which may be characterized as a verification graph to distinguish the overlaid content graph (and in some embodiments serve as an immutable, tamper-evident record of the fragment's value).

[0145] In this example, the fragments in the content graph 138 form a linked list, where the first node of the fragment content graph 138 is stored in the leaf node 139 of the verification graph 132, and the first fragment is designated as a node 140 in the content graph. The node 140 in the content graph can be stored as an attribute in the node content in the node 138 of the verification graph, and the content node 140 can include a pointer 142 pointing to the next node in the content graph 138, which can be stored on a different verification graph 134. In some cases, the pointer 142 can be characterized as an edge in the content graph 138, and can be represented as an identifier of a node 143 in the verification graph 134, or in some cases, can be represented as an identifier of an attribute in the node, where multiple fragments are stored in different attributes of a given node of the verification graph. In some embodiments, in the content graph 138, the node 140 points to another fragment 144, which can then point to another fragment using a pointer 146 in the verification node 147 of the verification graph 136. Verification node 147 may include one or more attributes as node content that specify the final fragment in content graph 138 specified by element number 148. In some cases, node 148 may specify that there are no additional nodes in the value.

[0146] As a result, even if a given malicious actor somehow compromises one of the verification graphs 132, 134, or 136, that attacker will only have access to a set of fragments of the value, and not to the other fragments needed to make the full value stored in the distributed storage 16 complete. Furthermore, because the fragments are stored in a tamper-evident directed acyclic graph with the aforementioned hash-point cryptographic hash pointers, evidence of tampering is computationally difficult to conceal.

[0147] therefore, Figure 5 A plurality of verification directed acyclic graphs 132, 134, and 136 are shown, each of which may be replicated and each having nodes that may store as node contents data encoding a content graph 138, which in this case is a linked list of fragments, with each fragment in turn pointing to the next fragment and its corresponding address, and in some cases also to attribute identifiers in a base verification graph.

[0148] In this example, the fragments are arranged in a one-dimensional linked list, but embodiments are based on other arrangements of the content graph. For example, some fragments may include pointers to multiple subsequent fragments, such as in a skip list, to facilitate concurrent retrieval, and in some cases, the fragments may be stored in association with fragment position identifiers (e.g., the order in which the fragments are arranged in sequence), and the fragmented values ​​are reconstructed by the converter 20 during a read operation. In another example, the fragments in the content graph that encode multiple fragments of a single value may be stored in a binary tree content graph, a skip list content graph, or a combination of a binary tree, a linked list, a skip list, etc.

[0149] Three fragments of a given value are shown, but in some cases, it is contemplated that embodiments include substantially more fragments. In some cases, binary data encoding a single text character can be fragmented, such as a given Unicode character is fragmented into two or more fragments, a given value generates 10 or more or 20 or more fragments, in some cases, each fragment can be stored in a different distributed acyclic graph, in some cases, each acyclic graph can be replicated multiple times, such as 3 times or more. Therefore, a given value can be partially stored on 30 different storage computing nodes 26. In some cases, different instances can be compared to determine an authoritative copy, for example, the value stored and returned is selected in the replicated instance according to a majority rule method. In some cases, for example, in the case where the replicated instance of the graph is on a licensed computing device, an embodiment can vote for a given value without performing proof of work or proof of storage operation, or in the case where the device storing the graph is untrusted, some embodiments can, for example, determine consensus with proof of work, storage or shares according to a consensus protocol (e.g., Paxos, Raft, etc.). In some embodiments, for example, in an untrusted system, instances can be resolved according to the Interplanetary File System (IPFS) or by utilizing various distributed hash table methods.

[0150] exist Figure 5 In the example of , each of directed acyclic graphs 132, 134, and 136 is a directed acyclic graph of the same type, in this case a linked list of binary trees, where the edges are formed by cryptographic hash pointers. In other embodiments, a heterogeneous collection of directed acyclic graphs can be combined with different fragments stored in different types of graphs, for example. For example, the initial fragment can be stored in a binary tree, while the other fragments are stored in Figure 5 The directed acyclic graph shown, for example, is stored in a linked list of a binary tree.

[0151] Figure 6 An example of a process 150 is shown by which Figure 5 Data structure 130 or using Figure 4In some embodiments, process 150 includes detecting an application that queries a database, as shown in block 152. In some cases, this includes executing Figure 1 The security driver 30 shown in FIG. Figure 1 154. In some embodiments, receiving the query response may occur at Figure 1 After the database driver 32 in the low-trust database 14 sends a query to the low-trust database 14, the low-trust database 14 can return a query response, for example, a response to an SQL statement that selects certain records that meet certain criteria. Or the response can be a document accessed in the file system. In some embodiments, the query response can be received by the database driver 32, and then the database driver 32 can send the query response to the application 28. The response can be intercepted and modified by the security driver 30 before being provided to the application 28. In some embodiments, the security driver 30 can, for example, Figure 5 In the data structure of Figure 4 , and sends those pointers to the converter 22 to convert them back into values ​​that are sent to the data structure in the appropriate secure distributed storage 16 (e.g., in multiple fragments). Thus, a given query response may include a less secure portion of the query response that corresponds to the value in the field that is in Figure 4 The process is not classified as a higher safety value, and the higher safety value is classified as Figure 4 , or at least pointers to those values ​​in the secure distributed storage 16. In some embodiments, these pointers may be detected using a regular expression configured to detect such prefixes or suffixes of marked pointers, for example using a reserved character sequence.

[0152] Thus, some embodiments may include determining whether there is a received query response such as Figure 4 After determining that at least some pointers exist, some embodiments may determine whether there are more pointers to be processed in the query response, as shown in block 158. Some embodiments may iteratively or concurrently process each pointer present in the query response from the low confidence database 14 to obtain the pointers. Figure 4 The process is stored in the secure distributed storage 16 (for example, in Figure 5 The pointers are replaced with corresponding values ​​in the data structure.

[0153] Upon determining that there are more pointers to be processed, some embodiments may select the next pointer, as shown in block 160, and if an associated pointer is stored in association with the fragment, retrieve the fragment and the associated pointer, as shown in block 162. The fragments may be retrieved in reverse order relative to the order in the fragmented values, or vice versa, depending on how the content graph is constructed, for example, based on the order in which the fragments are written. In some embodiments, retrieving the fragments and the associated pointers may include causing the transformer 20 to access an index that associates pointers (e.g., fragment identifiers) with one or more URLs of one or more computing devices that store a replicated instance of one of the directed acyclic graphs 132, 134, or 136 that stores the fragments pointed to by the pointers. In some embodiments, the index may further identify a node identifier associated with the pointer, such as a block and a path through a binary tree to a specific node that stores the fragment in question and the current associated pointer as node content in the attributes of the node, for example, if the fragment is not the last fragment to be retrieved. In some embodiments, the converter 20 may access the DNS 18 to identify the Internet Protocol address and port number of the secure computing node 26 that maintains the identified directed acyclic graph, or each secure storage computing node 26 stores one of the replicated instances of the directed acyclic graph. Some embodiments of the converter 20 may send a request to those storage computing nodes to return the identified fragments, and the receiving storage computing node 26 may traverse the edges specified by the cryptographic hash pointer in the corresponding directed acyclic graph along the designated node specified by the request to store the fragment. Or some embodiments may navigate directly to the fragment, for example based on a node identifier that identifies the address of the fragment in memory without the need for traversal. In some embodiments, in order to speed up access, the location of the node in memory may be stored in a binary tree corresponding to the node identifier, or in an ordered list, so as to facilitate relatively fast access to the contents of the node.

[0154] In some embodiments, a storage computing node 26 or a collection of storage computing nodes 26 that replicates the directed acyclic graph can return the specified fragment to the transformer 20. In some embodiments, the transformer 20 can receive multiple instances of the requested fragment, and some embodiments can determine whether each fragment matches each other or determine the content of the fragment based on the content returned by multiple or majority responses, thereby preventing modifications to one of the instances from affecting the authoritative content of the fragment determined by the system. That is, not all embodiments provide this benefit, which does not mean that any other features described herein are limiting.

[0155] Some embodiments may maintain in memory a partially or fully formed encoding of the value to which the fragment belongs, and some embodiments may concatenate the retrieved fragments to a sequence of fragments accumulated in the value, as shown in box 164. For example, some embodiments may prepend or append each consecutive retrieved fragment to the sequence of fragments. Or in some cases, the fragments may be combined with other techniques. For example, some embodiments may combine the fragments by XORing the fragments to recreate the value. The term "cascade" is used broadly herein to refer to each of these different operations by which information in different fragments is combined to produce the value from which the fragments are taken. In some embodiments, for example, the fragments may be retrieved out of order, with the fragments being marked with a value indicating a position in the sequence, and some embodiments may sort the fragments according to that value to recreate the value from which the fragments are taken.

[0156] Next, some embodiments may return to block 158 to determine whether there are more pointers to process. Upon determining that there are no more pointers to process, some embodiments may continue to form a revised version of the query response, as shown in block 166, wherein the query response is modified by the query response. Figure 4 The process of replacing the pointer with a pointer that is fragmented and stored in the secure distributed storage 16 (e.g., Figure 5 In some embodiments, the replacement operation may be performed by transformer 20 described above or within security driver 30. In some embodiments, the revised version of the query response may include the replacement value in the same format and at the same location as the pointer within the query response, thereby maintaining associations between different values ​​and metadata in the query response, for example, associations where records are distinguished and associations where values ​​are indicated as applying to specific fields in those records. In some cases where no pointer is present, when it is determined in block 156 that no pointer is present, the revised version of the query response may be an unchanged version of the query response.

[0157] Next, some embodiments may provide a revised version of the query response to the requesting application, as shown in block 168. In some cases, a revised version of the query response may be provided by the security driver 30 to the database driver 32 according to the application program interface. Figure 1 The application 28 in the secure distributed storage 16 will receive query data transparently without being redesigned to interface with the database driver 32 but not designed with the security driver 30 in mind.

[0158] In some embodiments, the Figure 1 The system (or other system) is implemented in a different part than the part described. Figure 4 A write operation or Figure 6For example, in some cases, these operations can be performed by a database gateway or database management system through which the client device communicates with the database. In some embodiments, these operations can be performed by a dedicated device on the network through which the client computing device accesses the database. In some embodiments, these operations can be performed entirely within a single computing device, for example, as part of a monolithic application in which features of the secure distributed storage operate within a single computing device, or in which different components are executed in different containers or virtualized operating systems accessed via loopback Internet Protocol addresses.

[0159] Generation of hash values ​​within the blockchain

[0160] As mentioned above, blockchains generally allow small amounts of information to be stored in immutable data structures; once the data in the chain is changed, the chain is broken and generally no longer functioning. The most common blockchain implementation is the publicly accessible Bitcoin ledger (for which the blockchain was designed).

[0161] However, many existing blockchain systems are not well suited for certain use cases. Blockchains are typically immutable, so once data is committed to the chain, it is considered permanent in most cases. This can lead to scalability challenges as the chain can become very large without being able to reduce the size of the chain (these are not meant to abandon systems with immutable blockchains for all embodiments or are inconsistent with some implementations).

[0162] These and other problems are alleviated by some embodiments of the system described above (hereinafter referred to as "Document Chain"). In some embodiments, Document Chain is a blockchain software suite for low-latency put and get operations for binary large objects. In some embodiments, Document Chain uses an improved version of the hash function for blockchain.

[0163] In some embodiments, the document chain utilizes a digest function (a term that also often refers to methods and other routines) to define which properties of each block are used to create the block hash (sometimes called a Merkle root). By varying the properties used to construct the hash, some embodiments increase or decrease the ambiguity of the hash output. In some embodiments, a lambda function can be passed into the BlockManager constructor, which receives a block entry as a parameter and returns a value to be passed into the hash function, such as the expression BlockManager<H,T> As defined by the “H” in .

[0164] For example, if the data of a block is only used when generating its Merkle root, then the head of the chain can be removed without destroying the chain itself. Alternatively, if the system uses data, location, and potentially even more specific properties (e.g., read / write times), then the chain becomes completely immutable and must (in some implementations) remain intact in its entirety to function.

[0165] Some embodiments increase this control by removing the scripting language component within the chain and adding the ability to control the hash value as part of the configuration rather than as part of the source code. In some embodiments, the hash function accepts data that does not depend on the location of the block within the chain. In some embodiments, the hash function only accepts values ​​as input that can be recreated in another chain.

[0166] In other embodiments, a similar output as a result of this function may be produced by performing some kind of reconstruction of the chain. This operation may include writing all data to be preserved to the new chain, and then updating all reference TXIDs (or other pointers similar to those described above) in any external data stores that exist.

[0167] To implement these methods and other methods, some embodiments may perform process 170 by which a cryptographic hash function is configured to make it feasible to reorganize portions of directed acyclic graphs having edges specified by cryptographic hash pointers formed by the cryptographic hash function, for example to accommodate directed acyclic graphs that have grown larger than a threshold size and divide those graphs into smaller portions, or to generate new directed acyclic graphs that link back to those directed acyclic graphs.

[0168] In some embodiments, process 170 may be performed by Figure 1 The storage computing node 26 of the embodiment is executed, but is not limited to this embodiment, which does not mean that any other description in this article is limiting. In fact, the operation of process 170 can be applied to other types of directed acyclic graphs with cryptographic hash pointers as edges, for example, in blockchains for cryptocurrency and in distributed tamper-evident ledgers for other purposes.

[0169] In some embodiments, process 170 includes obtaining a tamper-evident log, as shown in block 172. In some embodiments, the tamper-evident log is referenced Figure 3 and Figure 5 In some embodiments, the tamper-evident log includes a series of nodes or blocks as a set of nodes, for example, each block is a binary tree of nodes. In some embodiments, the tamper-evident log can indicate the sequence of adding entries in the log by the topology of the directed acyclic graph. In some embodiments, each entry can be characterized as a node or block in the tamper-evident log.

[0170] Next, some embodiments may obtain a plurality of records to be protected, as shown in block 174. In some cases, the records may be segments or documents as discussed above (this does not mean that documents cannot be segments). In some cases, for example, when received from the transformer 20, the records to be protected may be obtained by the storage computing node 26, as described with reference to FIG. Figure 1 Just as described.

[0171] Some embodiments may iteratively or simultaneously process the records and determine whether there are more records to be processed, as shown in block 176. When it is determined that there are no records to be processed, some embodiments may return to block 174 and wait for additional records to be processed. When it is determined that there are more records to be processed, some embodiments may proceed to select the next record among the unprocessed records, as shown in block 178.

[0172] Some embodiments may then access the content of the latest entry of the tamper evident log, as shown in block 180. In some cases, the latest entry may be the last block written to the log, or the last node within a block component, such as a leaf node or a higher level node in a binary tree within the block. In some cases, the accessed content may be one or more attributes of the latest entry, such as one or more attributes of the node content. In some embodiments, those attributes may include attributes similar to those described above with reference to the following nodes: Figure 3 and Figure 5 A node in the directed acyclic graph described.

[0173] Next, some embodiments may access a digest cryptographic hash function, as shown in block 182. The term "digest" refers to a programming construct by which a particular implementation of a function or method is defined by other code other than a placeholder having a function designated as a digest (e.g., an object, class, subroutine, or function), and in some cases, certain aspects of the digest function are defined with other code, such as in a class definition in an object-oriented programming language, where a non-digest implementation of the digest function is defined when an object is implemented in the class. In another example, a digest function may be designated as a digest by code that is configured to receive a lambda function from a calling code body or from a calling code body called by a code body that receives the lambda function. In some cases, the implementation may access a scope having a code body that references the digest function, such as an instance of an object in a digest class or a method that receives a lambda function as a parameter.

[0174] Some embodiments may determine that a digest cryptographic hash function has been accessed, for example, upon detecting the presence of a reservation indicating that a function or method of this type has been accessed.

[0175] Some embodiments may then determine an implementation of the digest cryptographic hash function, as shown in block 184. In some cases, the implementation may be determined as part of instantiating an object within a digest class, and in some cases, the implementation may be determined by receiving a lambda function defined by the calling code or the called code. In some embodiments, the implementation may specify a cryptographic hash algorithm, such as SHA-256 or other examples described above, or in some cases, the type of cryptographic hash function may be specified in the specification of the digest cryptographic hash function, with other aspects of the algorithm configured in the implementation. In some embodiments, the implementation may specify which inputs the cryptographic hash function processed to determine the output, while the digest representation of the cryptographic hash function may not specify which inputs were used in the implementation. Thus, a single digest cryptographic hash function may be implemented in a variety of different ways within a given body of code, with the body of code, for example, calling those different ways at different times in different scenarios based on different conditions.

[0176] In some embodiments, the implementation of the summary cryptographic hash function determined in box 184 can specify that certain types of inputs will be used when calculating the cryptographic hash value output. In some embodiments, those types of inputs can be selected from a subset of the attributes in the node content accessed in box 180, such as the attributes and node contents of the neighboring nodes of the node to be added to the tamper-evident record. In some embodiments, these types of inputs can be position-agnostic inputs, such as node attributes, which do not indicate the position of the node accessed in box 180 within the tamper-evident record obtained in box 172. For example, the selected type of input can exclude timestamps, dates, counter values, which indicate positions in sequences specific to the graph, such as array indexes, etc. In some embodiments, the selected position-agnostic input can include items such as attributes, which specify fragment content or document content stored in the latest entry. In another example, the attributes of the access node content used as the position-agnostic input of the implemented cryptographic hash function can include the cryptographic hash value of the cryptographic hash pointer pointing to another node (the node accessed in box 180 points to the other node), thereby retaining the benefits of linking cryptographic hash pointers.

[0177] In some embodiments, the same implementation of the cryptographic hash function can be applied to calculate each cryptographic hash value in each cryptographic hash pointer of the directed acyclic graph that encodes the tamper-evident record obtained in box 172. Or in some cases, different implementations can be accessed for different directed hash pointers at different times. For example, each directed edge encoded by the cryptographic hash pointer can also include a value identifying the implementation, for example, indicating whether the edge is a location-agnostic implementation of the summary cryptographic hash function or a value specified by a location-dependent implementation, and the location-dependent implementation includes non-location-agnostic values ​​of the above types such as timestamp date, location index, etc. as input. In some embodiments, after determining that the tamper-evident log in a given directed acyclic graph or a portion thereof has reached a threshold size and determining to create a new directed acyclic graph that stores subsequent entries in the tamper-evident log, a location-agnostic implementation is selected, while other cryptographic hash function implementations that are not location-dependent can be used at other times. In order to simplify the code, the same location-agnostic implementation can be used for each cryptographic hash pointer.

[0178] Next, some embodiments may calculate a cryptographic hash value based on the accessed content from block 180 using the implementation determined in block 184, as shown in block 186. In some embodiments, process 170 also includes forming the content of a new entry of the tamper-evident log, which includes the hash value calculated in block 186 and the current record selected in block 178, as shown in block 188. In some cases, forming the new content may include modifying an existing entry or creating a new entry. In some embodiments, forming the content may include adding a value to an attribute in the content of the new entry, such as Figure 3 and Figure 5 In some embodiments, the hash values ​​in the current record may be separately labeled for different attributes in the formed content, or in some cases, these values ​​may be combined with a single encrypted hash value, for example based on both the accessed content and the current record from block 180. In some embodiments, the current record may be stored remotely, and a hash digest, such as based on an encrypted hash value of the content, may be stored in the tamper-evident record obtained in block 172. Alternatively, some embodiments may store the current record in the log itself to be viewed based on, for example, reference to Figure 2 and Figure 3 Some of the above technologies can speed up access.

[0179] Next, some embodiments may append the new entry before the tamper-evident log, as shown in block 190. In some embodiments, this may include adding the new entry to a sequence of entries, such as Figure 3The term "append before" in certain programming constructs does not require any particular array index or position as long as the new entry is implicitly or explicitly designated as newer than an existing entry or other older entry in the tamper-evident log.

[0180] Some embodiments may then determine whether the size of the log is greater than a threshold, as shown in block 182. In some embodiments, this may include determining whether the size of the log is greater than or equal to a threshold, based on a specified configuration, or greater than a configuration typically used herein. Upon determining that the size of the log is not greater than the threshold, some embodiments may return to determine in block 176 whether there are more records to process. In some embodiments, the size may be measured by, for example, the number of blocks, the number of nodes, or the amount of data stored in a directed acyclic graph measured in megabytes.

[0181] After determining that the size of the log does exceed the threshold, some embodiments may proceed to split the tamper-evident log into an older tamper-evident log and a newer tamper-evident log, as shown in block 194. In some embodiments, the split tamper-evident logs may each have a directed acyclic graph of the same type, and in some cases, the older tamper-evident log is larger when split. In some embodiments, the split may be performed by determining not to append a new entry before or after a new entry to the tamper-evident log, but to instantiate a new directed acyclic graph to receive subsequent entries. In some embodiments, the older and newer tamper-evident logs may both have the same type (e.g., topological category) and have a reference to Figure 3 and Figure 5 In some embodiments, the system may stop adding new entries to an older tamper-evident log and continue adding new entries to a newer tamper-evident log until the log exceeds a threshold, in which case some embodiments may then split the newer tamper-evident log into yet another newer tamper-evident log and then continue indefinitely to increase the number of tamper-evident logs.

[0182] In some embodiments, the initial nodes or blocks in those tamper-evident logs may include a cryptographic hash pointer to an older tamper-evident log, for example, based on the cryptographic hash value of the root node of the latest block of the successive older tamper-evident logs. In some embodiments, this reference content may be used to generate a seed value, such as Figure 3 and may be associated with an identifier of an older tamper-evident log. In some embodiments, Figure 1The newer tamper-evident log may be instantiated on a different storage compute node 26 than the node of the older tamper-evident log, or the tamper-evident log may be instantiated on the same storage compute node 26.

[0183] Thus, some embodiments may reconfigure a directed acyclic graph while maintaining provable tamper-evidentness to accommodate arbitrarily large data sets.

[0184] Transparent client for quorum data storage between mutable and immutable data repositories

[0185] As mentioned above, in many cases, traditional databases are not secure enough. Many mainstream databases provide some level of permissions, but these permissions are often broad and difficult to manage, which leads to opportunities for attacks and abuse. In addition, the most widely adopted databases implement transactional SQL (T-SQL) or similar variants. In most implementations, poor design practices and traditional code may have vulnerabilities such as SQL injection attacks, which can trick the database into returning potentially sensitive and unintended data. In addition, there is little auditability and permissions for individual cells in tables of other types of individual table values. Moreover, most database management systems rely on drivers that reside on the machine of the database client. Typically, these clients are open source and easily available, making development easier. (This does not mean that some embodiments will not also suffer from subsets of these problems, because several inventions are described, and those inventions can be used independently without solving every problem described herein.)

[0186] To address these issues, various approaches have been implemented to protect stored data. Examples include isolating the database in mid-air or carefully managing permissions to access the database. However, many approaches are difficult to manage (e.g., using baroque permission schemes), slow (e.g., isolating in mid-air), or break backward compatibility with expensive enterprise applications that expect specific database driver interfaces, which is not meant to reject any of these approaches in all embodiments.

[0187] Some embodiments utilize dedicated client applications that can identify sensitive data (by column type, predefined column name, or other predefined method) to alleviate some of the above problems. Some embodiments can capture data as it is transferred to the database and route sensitive data to more secure storage mechanisms, such as those mentioned above. Thus, more secure storage technologies (although perhaps slightly slower) can be retained for the most sensitive data, while faster and cheaper but less secure storage solutions (such as traditional relational databases) can be used to handle less sensitive data. In addition, in some embodiments, the client application can abstract this different routing and query format changes between systems from other applications.

[0188] In some embodiments, as described above, a subset of more sensitive data may be stored by spreading the data across multiple blockchains. Similarly, when a client application detects that sensitive data is being queried (using the same approach), in some embodiments, the client application may obtain the TXID (or other pointer) from the database and send it to the arbitrator instance for reorganization, confirming that the request has valid permissions, and if so, placing the reorganized data in place of the TXID.

[0189] Because some embodiments intercept data in the data path, some embodiments can generate additional audit logs that show all attempts to access data, as described in more detail below. In some cases, these access logs can be marked with request-specific information, such as: username, geographic location, client computer IP address, etc.

[0190] With this approach, it is expected that other applications implementing traditional database drivers will require little to no dissemination or blockchain-specific configuration. In some cases, the process is completely transparent to other traditional applications. In addition, since access and modification to data can be easily detected, the complexity of permissioning can be relaxed by routing secure data to different immutable secure data structures.

[0191] It is expected that certain types of data are particularly suitable for use with the present technology. Often, system access credentials (e.g., usernames and passwords) are particularly sensitive because if such information is subject to unauthorized access, the entire account may be compromised. Storing passwords on a local machine or in a database where the entire password is accessible in one location provides an easy target for threat actors who wish to manipulate, steal, or otherwise abuse authentication credentials. Other examples include credit card numbers, social security numbers, or health-related data.

[0192] Some embodiments interface with a blockchain as a storage data structure, where an arbitrator or other middleware is able to take as input a full text representation of a user's credential, fragment the credential into N fragments starting from the last byte of the credential, and place each on a physically (or virtually) independent blockchain-backed storage data structure, each of which contains a pointer to the next storage location of the fragmented credential. In some embodiments, when an application or resource requests to reassemble a fragmented credential, the location of the first byte of the credential is provided to the arbitrator or middleware. In some embodiments, after reading the first byte, the arbitrator or middleware then reads subsequent pointers until a null character or end-of-sequence character is read. Once all fragments have been read into memory, the arbitrator or other middleware can respond to the application with the synthesized non-fragmented credential. Some embodiments may pre-process the credential and count the number of fragments required from the beginning before fragmenting the credential. Some embodiments may require a credential to generate a threshold number of fragments. Some embodiments may sort the fragments or credentials before fragmentation to defeat or impair rainbow table attacks.

[0193] These and other techniques can be used Figure 8 The process in Figure 8 An example of a process 200 is shown that can be performed by a client computing device to transparently transform an existing workload application to interface with a heterogeneous mix of databases, particularly a combination of databases that include a database of higher security than the database to which the application as originally written is configured to interface, such as the databases described above. However, it should be emphasized that the present technology is not limited to embodiments utilizing the above types of more secure databases, and in some cases, can be used in conjunction with other relational databases such as another relational database or another type of database, for example, a database that is considered to have higher security or lower latency than the database to which the application accessing the data is configured to interface. In some embodiments, process 200 can be performed by the above-described security driver 30, but it should be noted that in some cases, some or all of the functionality can be performed in a transformer 20 in a database gateway, in a database management system, or in some other computing device.

[0194] In some cases, such as where a workload application executing on a client computing device that is configured to access has been compromised, it is expected that performing process 200 in the client computing device before the data leaves the client computing device or when the data arrives at the client computing device will produce certain security benefits. In this case, the attacker may have compromised other computing devices on the network, and it is expected that merging or splitting the data on the client computing device immediately before or after writing or reading, respectively, will reduce the attack surface of the network. That is, embodiments are not limited to systems that provide these benefits, and in some cases, these operations can be performed in another computing device on the network that is considered relatively secure, which does not mean that any other features described herein are limiting.

[0195] In some embodiments, the process 200 can be made transparent to a workload application executed on a client computing device, such as a service on one of multiple hosts that execute different services in a microservices architecture, or an application executed on a single computing device as a monolithic application. In some embodiments, the process 200 can be made transparent to the application by registering the process in the operating system of the client computing device to make it look like a database driver that the workload application is configured to access, and then wrapping the application program interface of the original database driver with the operation described below. Therefore, some embodiments can respond to the same set of application program interface requests that the database driver responds to while providing additional functionality. In addition, some embodiments can then pass modified or unmodified application program interface exchanges between the workload application and the database driver. In many cases, the source code of the workload application is not available or the modification cost is very high. Therefore, it is expected that it is particularly desirable to transform the existing workload application in a way that does not require changing the code of the application. That is, the present technology is also applicable to the use case where the source code can be used for the workload application and is modified to implement the present technology, which does not mean that any other description is restrictive.

[0196] In some embodiments, process 200 includes registering a secure driver that wraps a database driver, as shown in block 202. In some embodiments, the secure driver may be registered in an operating system in which a workload application (e.g., application 28 above) that makes a database access request described in subsequent operations of process 200 is executed, and the operating system may also be an environment in which a database driver (e.g., driver 34 above) described below operates. In some embodiments, as a result of the registration operation, when an application sends an application program interface request to the database driver, the request may instead be received by the secure driver, and the secure driver may be configured to then communicate with the database driver as an intermediary between the database driver and the application within a single operating system on the client computing device.

[0197] In some embodiments, other types of access may be handled in a similar manner. For example, some embodiments may wrap a file system driver to obtain exchanges between the file system driver and, for example, a workload application that operates on documents. In some cases, a file system filter driver may be instantiated that issues events indicating an application program interface exchanged with the file system driver, and some embodiments may classify these events as being related to (or not related to) higher security documents, and in some cases, modify driver behavior in response, for example, replacing a document stored in secure distributed memory 16 for an access request related to, for example, a locally stored text file with a pointer to the document.

[0198] Then, some embodiments may include receiving a first write request, as shown in block 204. In some cases, this operation may be performed at some time after the secure driver is registered, and may occur with relative frequency. In some embodiments, the write request is similar to the above-described request. In some embodiments, the write request is an application program interface request to a database driver from a workload application executing on a computing device. In some embodiments, the request may be a request to a first remote database, such as the low-trust database 14 described above or some other data store. In some embodiments, the first write request may specify that a set of values ​​will be written to a set of fields in a set of records in a database, which may include adding new values ​​to new fields of new records or modifying existing data. The first write request may be formatted in a scheme specified by the application program interface of the database driver, and the first write request may be operable to cause the database driver to respond, regardless of whether the registration step of block 202 has occurred, although in some cases, due to the omission of the operation in the secure driver, the application may then continue to operate in a less secure manner.

[0199] Next, some embodiments may obtain a data policy, as shown at block 206. In some cases, the security driver 30 may maintain a set of one or more policies in memory, each policy including a set of rules, such as a policy for each application or each low-trust database 14.

[0200] Some embodiments may include classifying the value to be written as a higher security value or a lower security value, as shown in block 208. In some cases, this operation may include selecting one or more rules from one or more data policies, for example based on the application writing the data, the low-trust database receiving the data or intending to receive the data, or other criteria. Some embodiments may include applying one or more of the above rules to each value in the write request to classify the value as higher security or lower security.

[0201] In some embodiments, the rule may include one or more conditions by which the data written into the database may be classified as lower security data or higher security data. In some cases, these rules may include rules that specify certain types of fields as lower security or higher security, such as text fields and integer fields or fields within a specified pattern. In some embodiments, the condition may explicitly list higher security fields and lower security fields, and due to attempts to place data in these fields in a write request, the data may be classified as higher security or lower security. In some embodiments, the condition may apply a regular expression to a field identifier, such as a field name, to determine whether the value in the field is higher security or lower security. In some embodiments, the rule may be applied to the content of the value being written, so that some values ​​in a given field may be higher security, while other values ​​in the given field may be lower security. For example, a certain condition may include a regular expression, and the pattern matches the content of the value to determine whether these values ​​are higher security or lower security, for example, a specified value (regardless of which field they belong to) in a format consistent with a telephone number, a credit card number, or a social security number may belong to a higher security.

[0202] Next, some embodiments may store the lower security value in a first remote database, as shown in block 210. In some cases, the first remote database may be a database that the workload application is configured to interface with when initially installed or off-the-shelf. Thus, the workload application may be configured to interface with the first remote data plane without modification. In some cases, this may include, for example, writing the value to the low trust database 14, as described above with reference to Figure 4 The operations described in block 106 are consistent with those described in block 106 of FIG.

[0203] Some embodiments may then store the higher security value in a second remote database, as shown in block 212. In some cases, the second remote database may be a database that implements Figure 3 and Figure 5 One of the above-mentioned higher security databases, such as hosted at Figure 1 Alternatively, in some cases, the secure second remote database may be another relational database or other type of database, for example, a database that implements additional security features relative to the first remote database or is isolated only from the first remote database. In some cases, storing a higher security value may include the above reference Figure 2 and Figure 4 The operation described.

[0204] Next, as shown in block 214, some embodiments of the security driver may cause a pointer to be stored, for example, a pointer to a higher security value to be stored in the first remote database. For example, this may include modifying an application program interface request from the workload application to the database driver to replace the higher security value with a node identifier, a fragment identifier, a document identifier, or other type of identifier (such as the above) before pushing the modified application program interface request to the database driver. In some cases, the database driver may then convert the application program interface request into a command and data stream suitable for causing the first remote database to store a pointer identifying a location where a corresponding value is stored in the second remote database.

[0205] Next, embodiments may read the data back at a later time. In some cases, this may include receiving a query and then receiving a query response, including the pointer in the query response, as shown in block 216. In some embodiments, the query itself may be modified, for example, where the conditions in the query depend on the content of the higher security value. For example, some embodiments may use a modified query to select all values; then, in the security driver or converter described above, replace the pointer with the corresponding value retrieved from the second remote database; and then apply the conditions of the original query to these values ​​to determine which values ​​are responsive to the query issued by the workload application.

[0206] In some cases, a single query from a workload application may generate a cascade of iterative subsequent queries, for example, where a join operation is performed and where data from a first remote database and a second remote database are merged to determine an intermediate query response. For example, a workload application may request the email addresses of all users with credit card numbers that begin with a specified sequence, and the email addresses and credit card numbers may be maintained in different tables, where the credit card numbers are assigned a higher security value and the email addresses are assigned a lower security value. In some cases, the two different tables may be linked by a master key in one table that is referenced by a foreign key in another table, and the query from the workload application may specify a join. Some embodiments may, for example, retrieve each record in the first database that has a pointer to a value in the second database that reflects a credit card number, merge these values, determine which foreign keys in the table with the email address are responsive to the query condition, and then issue a subsequent query to the first remote database for those records. In some cases, to speed up these operations, an index may be maintained in which pointers are associated with values ​​that indicate whether a value is responsive to certain conditions (e.g., a threshold number of prefix characters or suffix characters), and embodiments may access this index to identify a subset of pointers whose values ​​were retrieved from the secure data store.

[0207] Some embodiments may interface with a database using techniques described in U.S. Provisional Patent Application 62 / 527,330, filed on June 30, 2017, entitled “REPLACING DISTINCT DATA IN A RELATIONAL DATABASE WITH A DISTINCT REFERE NCE TO THAT DATA AND DISTINCT DE-REFERENCING OF DATABASE DATA,” the contents of which are incorporated herein by reference.

[0208] In some embodiments, these operations can be accelerated by assigning pointers or other types of unique identifiers based on the content of the value to which the pointer points, such as based on a cryptographic hash value based solely on the content of the value to which the pointer points. As a result, for example, different instances of the same value in different rows or other tuples of a database can correspond to the same pointer. These pointers can be said to be unique identifiers in the sense that they uniquely identify the content, in some cases without revealing semantic information in the content, such as using a cryptographic hash identifier, while still, for example, replicating the same unique identifier for multiple instances of a value appearing at multiple rows and databases.

[0209] In some cases, the value may be cryptographically hashed (e.g., by concatenating the value with a delimiter prior to inputting the hash function) in conjunction with a tenant identifier, such as a random string of a certain amount of entropy (e.g., longer than 64 bytes), so that the same value for a given tenant always hashes to the same pointer, while the same value for different tenants hashes to different pointers. Or in some cases, the unique identifier may also be unique between instances. Thus, the same value appearing in two different rows may have a different unique identifier in each case, although some embodiments may operate more slowly as a result. In some embodiments, the security driver may detect duplicate instances of a pointer when reading back data, and cause a single request for a single value from a second remote database to populate each of those locations held by different instances of the same pointer. Thus, fields with relatively low cardinality may facilitate relatively fast joins, and a relatively small number of values ​​may populate a relatively large number of rows, even if those fields are designated as relatively high security.

[0210] When writing, similar operations are performed, for example, by grouping data classified as high security according to value, such as by sorting the data, and then detecting a group of instances where the value is the same, or storing the data in a hash table and detecting duplications through hash collisions, where the same value is written to the hash table at the same index. In these examples, some embodiments can then assign the same unique identifier to each instance in the group with the same value, and cause the unique identifier (which can be used as a pointer) to be stored in place of those higher security values ​​in the first remote database.

[0211] After replacing the pointer with the value from the second remote database as shown in block 218, some embodiments may provide a modified query response to the query application as shown in block 220. In some embodiments, the query application may be a different workload application than the application that wrote the data, and in some cases, may be on a different computing device. In some cases, for a given write, the data is read multiple times, or the data may be written multiple times before the latest version of the value is read.

[0212] Immutable record of access requests to a distributed file system

[0213] Use a third-party off-site data storage platform (e.g., Dropbox TM or Google Drive TM ) and on-site platforms (e.g., SharePoint TM 、Confluence TM 、OwnCloud TMetc.) are becoming more common, but in these systems, auditing changes to these files becomes a difficult and often impractical task. In some cases, files are stored in large, contiguous blocks, making them easily accessible to threat actors, and in many cases, changes to the data do not produce reliable records that would allow such changes to be detected. For example, in breaking into such a system and making changes, a threat actor could also tamper with log records or other records that would allow the changes to be detected by other means. (Again, none of this is meant to disregard this approach.)

[0214] As mentioned above, traditional databases are not adequate to prevent threat actors or internal resources (employees, IT personnel, etc.) from tampering with data. To the extent such systems have audit logs, these logs are typically only as secure as the data that was accessed, meaning that if the data is compromised, the audit logs are typically also compromised. Some products provide "secure storage" through the use of password-protected folders or data stores, however, such products typically do not provide immutable distributed properties and their audit logs can often be modified, especially by employees with high access credentials.

[0215] As described above, by spreading (e.g., breaking into fragments and distributing) files into one or more blockchains, some embodiments are able to provide immutable distributed properties to files. In addition, because the files are distributed, in some embodiments, there are applications for reassembling the files. By making the reassembly process necessary, some embodiments form a control point through which embodiments are able to generate a blockchain-backed audit trail for each access to a file (or database value or log entry) stored using this method.

[0216] As described above, in some embodiments, data access is implemented in an application that is able to listen to events generated by the operating system's file system, read the changes that cause the notifications, and report them to an application called an "arbitrator" instance for scattering and storage in the blockchain. In some embodiments, the resulting TXID of the scattering operation is stored on the client file system instead of the actual data, or a similar method similar to the above can be implemented through the above-mentioned secure driver.

[0217] When a read operation is requested from the file system, in some embodiments, the stored TXID (or other pointer) is sent to the arbitrator instance for reassembly and loading at the appropriate location, which is then opened using the default application handler for the file (e.g., Adobe Reader is typically used in some cases). TMIn some embodiments, if the user does not wish to place a file system reader on the entire file system or a subset (directory) of the file system, the user can scatter the file by accessing the context menu and directing the file into ScatterFS.

[0218] Finally, in some embodiments, an application programming interface (API) may be exposed so that any third-party application can pass file handles, buffers, or other data streams for distribution and storage.

[0219] Thus, some embodiments may distribute data among multiple computing devices, in some cases in a peer-to-peer network hosting an immutable distributed data structure (e.g., a blockchain), and those embodiments may record and store access records in another (or the same) blockchain, thereby monitoring the necessary control points for accessing the data and creating an immutable record of such access attempts. It should be understood that the present technology is not limited to blockchain-based distributed databases, and similar technology can be envisioned for other distributed file systems, such as network file systems, self-certifying file systems, server message blocks, MapR FS, Amazon S3, and Starfish distributed file systems.

[0220] In some embodiments, Fig. 9 The process 230 shown implements logging and log analysis. In some embodiments, the process 230 may be performed at least in part by a system that implements one or more of the above-described techniques. In some embodiments, logging may be implemented on a computing device that acts as a choke point in the aggregation or disaggregation of information stored in or read from a distributed data store as described above or other types of distributed data stores. In some embodiments, information may be distributed in the following manner: each unit of content requires that information from a different computing device will be accessed (e.g., by means of fragmentation or separation of an encryption key from a ciphertext), and information from different computing devices may be aggregated by a computing device that enables logging in a tamper-evident log. Similarly, write operations may be logged by a computing device, which causes information to be decomposed and distributed among different computing devices (e.g., by fragmenting or separating ciphertext from an encryption key, or a combination thereof). Therefore, in some cases, logging occurs during each access operation because logging by a computing device is necessary for reading or writing information that would otherwise be inaccessible unless the information passes through the computing device that causes the logging. In some embodiments, this information may be recorded in a tamper-evident immutable log as described above, in some cases in the same data structure that stores the accessed information (e.g., workload content such as database entries and various types of documents in a file system). In some cases, the log records may be recorded by Figure 1 The above-described converter 20 is implemented (eg, caused by), but the embodiments are not limited to this implementation, and this does not mean that any other description is limiting.

[0221] In some embodiments, process 230 includes receiving a request to access a distributed data store, as shown in box 230. In some embodiments, the request can be a read request or a write request. In some embodiments, a correct request is accompanied by a content unit, such as a value shifted by a pointer in a low-trust database 14 according to the above-mentioned technology, or a document replaced by a pointer in a low-trust file system according to the above-mentioned technology, which does not mean that the document cannot be stored in the database because the value in the database or any other description in this article is restrictive. Similarly, in some cases, a read request can be accompanied by a pointer stored in a low-trust data store in place of the content unit and read according to the above-mentioned technology. In some cases, the content unit to be accessed is referenced by a pointer that points to a fragment used as the first fragment in the content graft as described above, such as a linked list of fragments, where different fragments of the content unit are distributed in different tamper-evident directed acyclic graphs.

[0222] In some embodiments, process 230 may include aggregating or disaggregating content units associated with the access request, as shown in block 234. In some embodiments, this may include the fragmentation operation described above with reference to block 112 or the union of the fragments described above with reference to block 164. Figure 4 and Figure 4 Alternatively or additionally, this may include encrypting and storing the ciphertext separately from the encryption key, or putting the separately stored encryption key and ciphertext together for efficient decryption.

[0223] Before or after aggregation or disaggregation (this does not mean that any other steps herein are limited to the order described in this example), some embodiments may cause the access request to be recorded in an entry in a tamper-evident log, with the logging caused by the computing device involved in aggregating or disaggregating the content unit, as shown in block 236. In some embodiments, the logging may be caused by the computing device necessary to enable access to the content unit through aggregation or disaggregation. In some embodiments, the logging may be caused by the transformer 20 described above, which does not mean that the transformer 20 is required in all embodiments according to all present techniques, or that any other description herein is limiting.

[0224] In some cases, causing logging includes selecting one or more of the directed acyclic graphs described above, such as referring to Figure 5 Discuss and store in Figure 1In some cases, causing the log record includes sending an instruction to one or more of those directed acyclic graphs (e.g., to a service that monitors a network socket and manages the graph in response to the instruction) to store a record describing the logged event in, for example, a network socket according to the above reference. Figure 2 and Figure 3 The described techniques store records as a directed acyclic graph of documents and, in some cases, in accordance with the above reference Figure 4 and Figure 5 The described techniques fragment the record into multiple fragments. Alternatively, in some cases, the record may be stored outside of the tamper-evident log, and the encrypted hash of the record and the timestamp of the record may be stored as the node contents of one of the above-mentioned tamper-evident directed acyclic graphs, which have encrypted hash pointers as edges. In some embodiments, due to these graphs, modifications to the records describing the log entries may be computationally difficult to hide, because the chain sequence of encrypted hash values ​​in the directed acyclic graph based on those records may produce insurmountable computational challenges to calculate hash collisions along the entire sequence of paths through the directed acyclic graph, which conflict with the values ​​produced by the original unchanged record. (It should be noted that the attributes of the nodes can be stored in the edges, and vice versa.)

[0225] In some embodiments, process 230 includes recording entries that describe various aspects of the access request. In some embodiments, the entry is recorded with a record that includes an identifier of the user account that made the access request, an identifier of the application (e.g., the workload application through which the access request was submitted), and the content of the access request, such as a command including the content to be written or a command identifying the content unit to be read. Such a record may further include a timestamp, such as a date or date and time indicating when the access request was received. In some cases, the record may include the geographic location of the computing device that submitted the request. In some embodiments, such a record that records the entry in the tamper-evident log may further include an identifier of the computing device through which the access request was submitted to the above-mentioned system, such as a MAC address, an Internet Protocol address, a browser fingerprint, a hardware fingerprint, etc.

[0226] As shown in block 238, some embodiments may store a tamper-evident log in memory, which in some cases may include adding new entries or modifying entries in a tamper-evident log already stored in memory, which may be replicated in multiple instances according to the techniques described above. This may include forming part of a block to be later added to such a log, e.g., a blockchain.

[0227] Some embodiments of process 230 may include verifying a tamper-evident log, as shown in 240. Verification may include determining whether the tamper-evident log indicates that the entry was modified after being recorded. In some embodiments, the operation may include accessing the current log version of the entry and then calculating a cryptographic hash value based on the current log entry. Some embodiments may then compare the current cryptographic hash value with one or more other cryptographic hash values ​​in the tamper-evident log, for example, some embodiments may compare the cryptographic hash value with the cryptographic hash values ​​stored in one of the adjacent nodes in the directed acyclic graph to determine whether the cryptographic hash values ​​match, or if they do not match, the indication record has been modified. In some embodiments, a sequence of cryptographic hash values ​​based on each other may be calculated to determine which matches and identify the path through the directed acyclic graph to the node where the cause of the difference is located. In some embodiments, verification may occur at each write operation, periodically, at each read operation, or according to some other schedule or event.

[0228] Since it is difficult to modify log entries in a concealable manner, the detectability of tampering can deter threat actors from modifying log entries or maliciously accessing data. In addition, some embodiments may fragment log entries according to some of the above techniques, thereby further increasing the difficulty of modification because even if a threat actor is somehow able to compromise the mechanism that indicates tampering, it may be necessary to compromise a heterogeneous collection of different computing devices on different networks to modify each fragment.

[0229] In some embodiments, process 230 includes determining whether the verification operation has been tampered with, as shown in box 241, such as indicating a mismatch between cryptographic hash values ​​in a directed acyclic graph. When tampering is detected, some embodiments may sound an alarm, as shown in 242. In some embodiments, the alarm may be sounded by sending an email, a text message, or a chat message. Some embodiments may further take certain actions to lock down portions of the system, such as disabling credentials, reducing the level of access associated with the credentials, adjusting some of the following thresholds to reduce the amount of data that can be accessed before subsequent access is slowed or blocked, or limiting access to less secure data.

[0230] In some embodiments, when it is determined that there is no evidence of tampering, or after an alarm is issued, some embodiments may continue to determine a risk metric based on the access requests recorded in the tamper-evident log, as shown in box 244. The risk metric may take a variety of different forms, and in some cases, the risk metric may be calculated for a variety of different entities. For example, different risk metrics may be calculated for different users, workload applications, computing devices, or a combination thereof. In some embodiments, the risk metric may be the number of access requests received within the tail duration, the total amount of access requests, or a combination thereof. In some embodiments, the risk metric is based on deviations from previous behavior patterns. For example, some embodiments may train machine learning algorithms such as hidden Markov models, recursive neural networks, etc. based on historical log events to predict the likelihood of various types of access requests (or sequences of such requests), such as access requests for: a given user, workload application, computing device, a part of a network, or a combination thereof, a specific content unit, a type of content unit, the number of access requests, the frequency of access requests, etc. Then, some embodiments may compare these predictions with access requests received later based on the current tail sequence of recorded events, and determine the risk metric based on the likelihood, such as the probability of a given access request given previous behavior. Failure of such a predictive model may be evidence of anomalous, malicious behavior. Some embodiments may use this probability as a risk metric, or determine the sum of these probabilities over multiple risk metrics, such as a measure of central tendency, such as the mean, median, or mode of these probabilities over tail duration or number of access requests.

[0231] In another example, the risk metric may be based on the content of the content unit being written. For example, some embodiments may calculate the entropy of the content unit to be written and compare that entropy to a metric of entropy associated with other content units that have been historically written to the distributed data store (e.g., a previous database entry or document). In some embodiments, the difference may indicate a ransomware attack in which a relatively high entropy encrypted version of the data is written as the current version. (Although it should be noted that some implementations may use an immutable data store in which the earlier value would remain in place, and in some embodiments, in the event of such an attack, the system may be rolled back to the earlier value by replacing a pointer to the current version in a low-trust store with a pointer to the last good version.)

[0232] Next, some embodiments may determine whether the risk metric exceeds or otherwise satisfies a first threshold, as shown in block 246. Comparisons to thresholds described herein should not be limited to comparisons that rely on the sign of the values ​​applied, and any other description is not intended to be limiting. The term "satisfies" generally refers to scenarios where a threshold is satisfied by exceeding (or, in some cases, being equal to) the threshold, and an equivalent threshold of multiplying a value by -1 is satisfied by being less than (or, in some cases, being equal to) the threshold, and vice versa. Upon determining that the risk metric satisfies the first threshold, some embodiments may block subsequent responses to subsequent access requests, as shown in block 248. In some cases, blocking may be achieved by a computing device participating in the aggregation or disaggregation of content units discussed above with reference to block 236.

[0233] If access is not blocked, some embodiments may compare the risk metric to a second threshold, as shown in block 250. In some embodiments, the second threshold may be less stringent than the first threshold, for example, corresponding to a lower risk level. In some embodiments, upon determining that the risk metric meets the second threshold, some embodiments may delay a subsequent response to a subsequent access request, as shown in block 252. In some embodiments, this may include starting a countdown timer and determining whether a specified duration has elapsed before aggregating or returning a content unit or disaggregating and writing a content unit, in some cases, for each content unit in an access request associated with a plurality of content units. Thus, some embodiments may implement a soft blocking mechanism by which functionality may be provided at a certain level while delaying human response time in the event of a script attack. In some embodiments, when the first threshold or the second threshold is met in blocks 246 and 250, some embodiments may use techniques similar to those described above to raise an alarm to facilitate investigation and human intervention when necessary. The embodiment may then return to wait for the next request to access the distributed data store in block 232.

[0234] Storing file differences in a distributed blockchain

[0235] In general, blockchain-based databases are not well suited for storing large, frequently modified collections of data, such as files or other binary data blocks. Due to the immutable nature of the blockchain, previous entries in the chain (which might represent files or values ​​in the database) cannot usually be deleted or overwritten. Therefore, each version of a file increases the size of the blockchain ledger, and as files are stored and frequently modified in the ledger, the size of the blockchain ledger may become too large to be accessed efficiently.

[0236] To overcome the above-mentioned negative effects of the immutable property of blockchain, some embodiments store only the changes (differences) of a file over time, rather than storing an entire copy of the file every time it is changed.

[0237] Some embodiments may receive a write request for a modified file (or other binary large object of data) and determine whether the data has changed from a previous version. In some embodiments, when the data is initially read, a hash digest (e.g., an MD5 hash) of the data may be calculated and saved in memory. When writing, a new hash may be calculated based on the data to be rewritten to memory, and the hash may be compared to the earlier hash to determine whether the file has changed.

[0238] When a change is detected, some embodiments may determine the delta between an earlier version (e.g., the latest version) and the current version. In some cases, the database may store deltas of previous changes, and some embodiments may traverse a sequence of deltas to recreate prior versions of a document or other file. When the latest version is obtained, some embodiments may determine the delta to the new version, e.g., by determining the set of longest common subsequences between versions, and store the resulting differences, e.g., in a unified format.

[0239] In most cases, it is expected that the resulting difference will be much smaller than the new version of the file (or other binary large object). As such, storing the difference in the blockchain is expected to be cheaper than storing the entire new version. In addition, because the file itself is ultimately stored in the blockchain (rather than just the hash digest), the system is expected to be more robust to various attacks (e.g., hash collision attacks). In this attack, malicious content is selected and designed to produce the same hash as the stored data, and the malicious data replaces the reliable data in the database. In traditional systems, the blockchain will produce a hash value that verifies the malicious content as reliable. In contrast, some embodiments circumvent this attack vector by storing the data submitted to the database in the blockchain. In addition, some embodiments provide enhanced control over files (or other binary large objects) compared to systems that only store hash digests of documents in the chain. In these older systems that only store hash digests, the system has no control over the file. Such a system can delete the file from the external data store, and all that remains in the chain is the hash digest. That is, such a system can authenticate the file, but they cannot reproduce the file. That said, not all embodiments provide these benefits, as various engineering and cost trade-offs are foreseeable, and multiple independently useful inventions are described, but this is not meant to be limiting of any other description.

[0240] In some embodiments, the following references may be used Fig.10 and Fig.11 The described processes are used to implement the above techniques, which are referred to below Fig.12 Describes the data structure to operate on.

[0241] like Fig.10As shown, some embodiments may include a process 260 of writing the difference between a new version of a document and a previously stored version of the document to a tamper-evident immutable data repository (e.g., a blockchain or one of the above examples of a directed acyclic graph with cryptographic hash pointers). In some embodiments, process 260 may be performed by the above-described security driver 30 in conjunction with the above-described transformer 20 or by other components, such as a gateway residing on a network between a network storage or database and a client computing device. In some embodiments, process 260 may be performed by a client-side application that wraps or otherwise interfaces between a client-side workload application and a database driver or a file system driver (e.g., a file system driver executing in the same operating system as the workload application), and the process performs Fig.10 and Fig.11 operation.

[0242] In some embodiments, the file system of the local computing device includes a hierarchical structure of directories in which files are arranged, for example, binary large objects with various examples of metadata, such as file names, creation dates, modification dates, authors, access permissions, etc. In some embodiments, these directories can be local directories stored on the computing device executing the workload application or stored on a remote network-attached storage device. In some embodiments, some files in the directory can be replaced by text files with computer-readable pointers, which, for example, point to documents stored using the above-mentioned techniques, and some embodiments can intercept (e.g., directly receive, pass, observe when filtering, etc.) read or write access requests from file system browser applications to file system drivers and detect pointers placed in these directories to replace documents. Then, some embodiments can, in response, perform corresponding operations on documents stored in tamper-evident immutable data repositories, such as those described above or other types of remote storage. In some cases, this can include creating new versions, updating pointers stored in directories, and various other operations. In addition, these techniques can be similarly applied to database read and write operations, for example, storing the differences between previously stored values ​​and relatively large databases and new versions of these values ​​in the database.

[0243] In some embodiments, process 260 includes receiving a request to write a new version of a document to a tamper-evident immutable data repository, as shown in block 262. In some embodiments, this operation may be performed, for example, by Figure 8The process is performed by the above-mentioned security driver or other client site arbitrator. In some embodiments, the request may include designating the document as higher security and a pointer to a previous version of the document stored in a directory or database accessible by the client-side computer. For example, the user may have previously read the previous version into the local memory of the client computing device workload application, converted the previous version, and then was requested to store the new version, so that the request is received in the operation of block 162.

[0244] Next, some embodiments may determine whether the new version of the document is different from the previous version of the document, as shown in block 264. In some embodiments, this may be accomplished by, for example, utilizing a method similar to that described below with reference to Fig.11 The described reading process is performed by retrieving the previous version from the tamper-evident immutable data repository. Alternatively, some embodiments can accelerate the determination by storing an index outside the tamper-evident immutable data repository, which associates a pointer to the document with the hash digest of those documents. In some embodiments, the hash digest is an encrypted hash value based on the content of the document, or a non-encrypted hash value based on the content of the document. It should be noted that not all hash functions are encrypted hash functions with the above-mentioned properties exhibited by such functions. In some cases, the calculation speed of the non-encrypted hash function can be faster than the encrypted hash function, or the embodiment can use the encrypted hash function to enhance security. Therefore, some embodiments can make the determination of box 264 without retrieving the previous version of the document from the tamper-evident immutable data repository, for example, by calculating a new hash digest based on the new version, and comparing the new hash digest with the previously calculated hash digest of the previous version stored outside the tamper-evident immutable data repository, for example, in association with the pointer to the previous version in the index. When determining that the hash value matches, some embodiments can terminate process 260 because it can be determined that the previous version is the same as the new version.

[0245] Alternatively, upon determining that the new version of the document is different from a previous version of the document, some embodiments may proceed to obtain the previous version of the document from a tamper-evident immutable data repository, as shown in block 266. In some embodiments, this may include retrieving the previous version via a read operation, e.g., by referring to Fig.11 The operations described above may, in some cases, include those described above with reference to Figure 2 , Figure 5 , Figure 6 and Figure 7 Describe the process.

[0246] Next, some embodiments may determine the set of changes required to convert the prior version to the new version of the document, as shown in box 268. In some embodiments, this operation may be relatively expensive in computation and includes determining the longest matching substring between the two documents in the process of determining the minimum set of changes required to convert the prior version to the new version. In some cases, these changes may be limited to deletion, additional text (e.g., prepending or postpending) and changing the value of existing text. In some embodiments, a differential operation may be used to determine, in which the prior version and the new version are input into a differential function, which may return a set of changes. In some embodiments, the changes may be implemented using the Hunt-McIlroy algorithm, as described in the paper entitled "AN ALGORITHM FOR DIFFERENTIAL FILECOMPARISON" (1976, Bell Labs Computing Science Technical Report) by Hunt and McIlroy, the contents of which are incorporated herein by reference.

[0247] In some cases, to speed up the comparison, each line of the two documents can be converted into a hash digest of the line, and the hash digest operation is relatively fast, such as converting each line with a hash function into a 64-bit or shorter, such as a 32-bit or shorter or 16-bit or shorter hash digest of the contents of the corresponding line. Therefore, each of the two documents can be converted into an ordered list of hash digests, and some embodiments can then compare the two ordered lists of hash digests to identify text sequences that may not have changed between the two versions. For example, some embodiments can identify line numbers at the beginning and end of the list that have not changed. When the line where the change begins is detected, some embodiments can then search forward in a list until a match is found, thereby potentially detecting an insertion.

[0248] Some embodiments may then store the changeset in a tamper-evident immutable data repository, as shown in block 270. In some cases, this may include Figure 3 Data structure execution Figure 2 to store the document, or according to the above reference Figure 5 and Figure 6 The described technique fragments the document and stores it in this manner.

[0249] Some embodiments may further include storing a pointer to a previous version of the document in association with the change set, as shown in block 272. In some embodiments, the pointer to the previous version of the document may be stored as metadata for the change set in a tamper-evident immutable data repository, thereby forming nodes and edges of a version graph, as described below with reference to Fig.12In some embodiments, for example, in the above data structure, a document may be encoded as a linked list of change sets, where pointers between the change sets can be traced back to the initial version of the document, and the document may be retrieved by retrieving each change in the change set and then iteratively applying them, as described below with reference to Fig.11 Described in detail.

[0250] In some embodiments, the updated pointer from block 270 to the newly stored change set may be stored in the file system (or database unit) to replace the previous pointer for which the request to write the new version was submitted in block 262. Thus, subsequent read requests for the file may identify the new pointer and retrieve the new change set, and then trace back to the initial version through the previous version, as described below with reference to Fig.11 Similar operations as described above may be performed on values ​​stored in a database to replace pointers in database cells to reference new versions.

[0251] In some embodiments, the version graph can branch. For example, a user can request to write a previous version of a document or write a new version to a previous version of a document, thereby forming parallel changes to version graphs that share a subset of their version history with each other, but then later branch a portion of the version graph. This is expected to further compress data and increase the number of documents that can be stored in a given amount of storage in a tamper-evident immutable data repository.

[0252] Fig.11 An example of a process 280 is shown that reads a document or other content unit from a tamper-evident immutable data repository that stores a version graph of the document as a sequence of differences between versions of the document, which traces back to an initial version. In some embodiments, Fig.10 Process 280 is executed in the process of obtaining a previous version of the document in box 266, or when a request from a workload application to access a stored document is received, for example, after intercepting a request to the file system driver to access a document stored in one of the above-mentioned data structures (the document is replaced by a pointer to the document).

[0253] In some embodiments, process 280 includes receiving a read request that identifies a pointer to the latest version of a document in a tamper-evident immutable data repository, as shown in block 282. In some cases, for example, when a user requests to revert to a prior version, the pointer may point to the prior version. In some embodiments, the read request may be intercepted using the techniques described above for intercepting write requests.

[0254] Next, some embodiments may read the version located by the current pointer, for example by performing the above-mentioned read operation, as indicated by block 284, by which the read operation is performed. Figure 3 or Figure 5 Retrieve documents from a data structure or receive other database entries.

[0255] Some embodiments may then determine whether the red version is the initial version of the document or other content unit, as shown in block 286. In some cases, the initial version may be explicitly marked when stored, or may be implicitly designated as such due to the lack of a pointer to an earlier version. As described above, the pointer may be stored in association with the subsequent version, as discussed with reference to block 272.

[0256] Upon determining that the retrieved version is not the initial version, some embodiments may designate a pointer associated with the retrieved version (e.g., stored as metadata for the retrieved version in a tamper-evident immutable data repository) as the current pointer, as shown by box 288.

[0257] Next, some embodiments may add a set of changes associated with the retrieved version (e.g., stored in a certain location or sequence of locations in the tamper-evident immutable data repository identified by the current pointer) to a last-in-first-out buffer, as shown in block 290. In some cases, the buffer may be characterized as a stack. Some embodiments may then return to block 284 to read the next earlier version located by the updated current pointer in block 284.

[0258] When the initial version is encountered in block 286, some embodiments may proceed to block 292 and initialize the copy of the document to the initial version, e.g., setting the working copy of the document equal to the retrieved initial version. Some embodiments may then determine whether there are other changes in the buffer to process, as shown in block 294. In some cases, some embodiments may determine whether the buffer is empty or whether there are other values ​​(i.e., a set of changes) stacked in the buffer.

[0259] Some embodiments may retrieve the next change set from the buffer after determining that there are more changes, as shown in block 296, which in some cases may include deleting the next change set from the buffer to update the buffer to reflect that the next change set is to be applied. In some cases, this may be characterized as popping the change set from the stack.

[0260] Next, some embodiments may apply the next set of changes to the current working copy of the document, as shown in block 298. In some cases, this may include accessing a set of delete operations, change operations, and append operations. In some embodiments, each of these changes may be associated with a line number to which the changes are to be applied, an order in which the changes are to be applied, and in some cases, content to be applied, such as indicating replacement text or appended text (or bits).

[0261] Some embodiments may then return to box 294 to determine if there are more changes in the buffer. When the end of the buffer is reached, such as the bottom of the stack, some embodiments may proceed to box 300 and return a copy of the document. In some cases, for example, a copy of the document may be returned to a workload application configured to access the document based on a file system extension in the operating system that is mapped to a particular workload application. In some embodiments, the read operation may be transparent to the user and may appear to the user as if the user is operating on a locally stored or network stored copy of the document, where the user experience is the same or substantially the same as that experienced by a user not connected to the secure distributed storage 16 interface described above, thereby providing greater security without imposing it on the user or, in some cases, requiring the modification of the workload application, which may be expensive. (That is, the embodiments are not limited to systems that provide these benefits, and this does not mean that the other descriptions are limiting.)

[0262] In some embodiments, Fig.10 and Fig.11 The process can produce Fig.12 The data structure 310 shown. In some embodiments, the data structure 310 may include the above reference Figure 5 The verification graph described in data structure 130 and the above reference Figure 5 In this example, the content graph 138 may be the latest stored version of the document, which may be a collection of changes relative to an earlier version. In some embodiments, the earlier version may be, for example, an initial version of the document, which is stored in Fig.12 314. Similar to content graph 138, content graph 314 may include a sequence of content nodes connected by pointers between fragments, such as fragments, through which an initial version of a document or a set of changes associated with a given version may be assembled during a read operation. Thus, in some embodiments, data structure 310 may store a third overlay graph, which may be version graph 312. In this example, version graph 312 includes edge 316, which may be a pointer from a latest version stored in content graph 238 to an earlier initial version stored in content graph 314. In some embodiments, the pointer may point to the initial fragment of the content graph storing the earlier version.

[0263] Fig.13 1000 is a diagram illustrating an exemplary computing system 1000 according to an embodiment of the present technology. Portions of the systems and methods described herein may include or be executed on one or more computer systems similar to computing system 1000. In addition, the processes and modules described herein may be executed by one or more processing systems similar to computing system 1000.

[0264] The computing system 1000 may include one or more processors (e.g., processors 1010a-1010n) coupled to a system memory 1020, an input / output I / O device interface 1030, and a network interface 1040 via an input / output (I / O) interface 1050. The processor may include a single processor or multiple processors (e.g., a distributed processor). The processor may be any suitable processor capable of executing or otherwise executing instructions. The processor may include a central processing unit (CPU) that executes program instructions to perform arithmetic, logic, and input / output operations of the computing system 1000. The processor may execute code (e.g., processor firmware, protocol stack, database management system, operating system, or a combination thereof) that creates an execution environment for program instructions. The processor may include a programmable processor. The processor may include a general-purpose or special-purpose microprocessor. The processor may receive instructions and data from a memory (e.g., system memory 1020). The computing system 1000 may be a single processor system including one processor (e.g., processor 1010a), or a multiprocessor system including any number of suitable processors (e.g., 1010a-1010n). Multiple processors may be employed to provide parallel or sequential execution of one or more portions of the techniques described herein. The processes (e.g., logic flows) described herein may be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating input data and generating corresponding outputs. The processing described herein may also be performed by a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)), and the apparatus may also be implemented as the dedicated logic circuit. The computing system 1000 may include multiple computing devices (e.g., a distributed computer system) to implement various processing functions.

[0265] The I / O device interface 1030 may provide an interface for connecting one or more I / O devices 1060 to the computer system 1000. An I / O device may include a device that receives input (e.g., from a user) or outputs information (e.g., to a user). The I / O device 1060 may include, for example, a graphical user interface presented on a display (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor), a pointing device (e.g., a computer mouse or trackball), a keyboard, a keypad, a touchpad, a scanning device, a voice recognition device, a gesture recognition device, a printer, an audio speaker, a microphone, a camera, etc. The I / O device 1060 may be connected to the computer system 1000 via a wired or wireless connection. The I / O device 1060 may be connected to the computer system 1000 from a remote location. For example, an I / O device 1060 located on a remote computer system may be connected to the computer system 1000 via a network and a network interface 1040.

[0266] The network interface 1040 may include a network adapter that provides connection of the computer system 1000 to a network. The network interface 1040 may facilitate data exchange between the computer system 1000 and other devices connected to the network. The network interface 1040 may support wired or wireless communications. The network may include an electronic communications network, such as the Internet, a local area network (LAN), a wide area network (WAN), a cellular communications network, etc.

[0267] System memory 1020 may be configured to store program instructions 1100 or data 1110. Program instructions 1100 may be executed by a processor (e.g., one or more of processors 1010a-1010n) to implement one or more embodiments of the present technology. Instructions 1100 may include a module of computer program instructions that is used to implement one or more of the techniques described herein with respect to various processing modules. Program instructions may include a computer program (referred to as a program, software, software application, script, or code in some form). Computer programs may be written in a programming language, including a compiled or interpreted language or an explanatory or procedural language. Computer programs may include units suitable for use in a computing environment, including as independent programs, modules, components, or subroutines. Computer programs may or may not correspond to files in a file system. Programs may be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordination files (e.g., a file storing a portion of one or more modules, subroutines, or codes). A computer program can be deployed to be executed on one or more computer processors located locally at one site or distributed across multiple remote sites and interconnected by a communication network.

[0268] The system memory 1020 may include a tangible program carrier having program instructions stored thereon. The tangible program carrier may include a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include a machine-readable storage device, a machine-readable storage substrate, a storage device, or any combination thereof. The non-transitory computer-readable storage medium may include a non-volatile memory (e.g., flash memory, ROM, PROM, EPROM, EEPROM memory), a volatile memory (e.g., a random access memory (RAM), a static random access memory (S RAM), a synchronous dynamic RAM (SDRAM)), a mass storage memory (e.g., a CD-ROM and / or DVD-ROM, a hard disk drive), etc. The system memory 1020 may include a non-transitory computer-readable storage medium on which program instructions executable by a computer processor (e.g., one or more of the processors 1010a-1010n) may be stored to cause the subject matter and functional operations described herein. The memory (e.g., the system memory 1020) may include a single memory device and / or multiple memory devices (e.g., a distributed memory device). Instructions or other program codes that provide the functions described herein may be stored on a tangible, non-transitory computer-readable medium. In some cases, the entire instruction set may be stored on the medium at the same time, or in some cases, different portions of the instructions may be stored on the same medium at different times.

[0269] The I / O interface 1050 may be configured to coordinate I / O traffic between the processors 1010a-1010n, the system memory 1020, the network interface 1040, the I / O devices 1060, and / or other peripheral devices. The I / O interface 1050 may perform protocol, timing, or other data conversions to convert data signals from one component (e.g., the system memory 1020) into a format suitable for use by another component (e.g., the processors 1010a-1010n). The I / O interface 1050 may include support for devices attached via various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard.

[0270] Embodiments of the techniques described herein may be implemented using a single instance of computer system 1000 or multiple computer systems 1000 configured to host different portions or instances of the embodiments. Multiple computer systems 1000 may provide for parallel or sequential processing / execution of one or more portions of the techniques described herein.

[0271] Those skilled in the art will appreciate that computer system 1000 is only illustrative and is not intended to limit the scope of the technology described herein. Computer system 1000 may include any combination of devices or software that can perform or otherwise provide the performance of the technology described herein. For example, computer system 1000 may include cloud computing systems, data centers, server racks, servers, virtual servers, desktop computers, laptop computers, tablet computers, server devices, client devices, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, on-board computers or global positioning systems (GPS), etc., or a combination thereof. Computer system 1000 may also be connected to other devices not shown, or may be operated as an independent system. In addition, in some embodiments, the functions provided by the components shown may be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functions of some of the components shown may not be provided, or other additional functions may be available.

[0272] Those skilled in the art will also understand that, although various items are shown as being stored in memory or stored in storage when in use, for the purpose of memory management and data integrity, these items or parts thereof may be transferred between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components may be executed in a memory on another device and communicate with the computer system shown via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer accessible medium or portable article to be read by an appropriate driver, various examples of which are described above. In some embodiments, instructions stored on a computer accessible medium separated from the computer system 1000 may be transmitted to the computer system 1000 via a transmission medium or a signal such as an electrical, electromagnetic or digital signal via a communication medium such as a network or a wireless link. Various embodiments may further include receiving, sending or storing instructions or data according to the foregoing description of the computer accessible medium. Therefore, the present technology may be practiced with other computer system configurations.

[0273] In the block diagram, the components shown are depicted as discrete functional blocks, but the embodiments are not limited to the system in which the functions described herein are organized as shown. The functions provided by each component may be provided by software or hardware modules different from the currently depicted organization, for example, such software or hardware may be mixed, combined, copied, separated, distributed (for example, in a data center or according to a geographical location), or otherwise organized differently. The functions described herein may be provided by one or more processors of one or more computers executing codes stored on a tangible non-temporary machine-readable medium. In some cases, although the singular term "medium" is used, instructions may be distributed on different storage devices associated with different computing devices, for example, each computing device has a different subset of instructions, which is consistent with the use of the singular term "medium" herein. In some cases, a third-party content delivery network may host some or all of the information transmitted over the network, in which case, to the extent that information (for example, content) is supplied or otherwise provided, the information may be provided by sending instructions to retrieve information from a content distribution network.

[0274] The reader should understand that this application describes several independently useful technologies. Instead of separating these technologies into multiple separate patent applications, the applicant grouped these technologies into a single document because their related subject matter can make the application process more economical. However, the unique advantages and aspects of these technologies should not be confused. In some cases, the embodiments solve all the defects pointed out herein, but it should be understood that these technologies are independently useful, and some embodiments only solve a subset of these problems, or provide other unmentioned benefits, which are obvious to those skilled in the art who consult this disclosure. Due to cost constraints, some of the technologies disclosed herein may not be currently claimed for protection, and may be claimed in later applications, such as continued applications or by amending the claims. Similarly, due to space limitations, neither the abstract nor the content of the invention section of this document should be regarded as a comprehensive list containing all such technologies or all aspects of such technologies.

[0275] It should be understood that the description and the drawings are not intended to limit the present technology to the specific form disclosed, but on the contrary, it is intended to cover all modifications, equivalents and alternative forms that fall within the spirit and scope of the present technology as defined in the appended claims. In view of this description, further modifications and alternative embodiments of various aspects of the present technology will be obvious to those skilled in the art. Therefore, the description and the drawings should be interpreted as illustrative only, and the purpose is to teach those skilled in the art the general way of implementing the present technology. It should be understood that the forms of the present technology shown and described herein will be regarded as examples of embodiments. Elements and materials can replace those illustrated and described herein, parts and processes can be reversed or omitted, and certain features of the present technology can be utilized independently, all of which will be obvious to those skilled in the art after benefiting from the description of the present technology. Without departing from the spirit and scope of the present technology as described in the appended claims, the elements described herein may be changed. The titles used herein are only for organizational purposes and are not meant to limit the scope of this specification.

[0276] As used throughout this application, the word "may" is used in a permissible sense (i.e., meaning that it is possible) rather than a mandatory sense (i.e., must). The words "include", "comprise" and "contain" etc. represent, but are not limited to, including. As used throughout this application, the singular "one", "one" and "the" include plural indicators unless the content explicitly indicates otherwise. Therefore, for example, referring to "element" or "an element" includes a combination of two or more elements, although other terms and phrases, such as "one or more" are used for one or more elements. Unless otherwise specified, the term "or" is non-exclusive, i.e., including "and" and "or". Terms describing conditional relationships, such as "in response to X, Y", "at X, Y", "if X, Y", "at X, Y", etc., include causal relationships, wherein the premise is a necessary causal condition, the premise is a sufficient causal condition, or the premise is a common causal condition of the result, for example, "state X occurs when condition Y is obtained" is common to "X occurs only at Y" and "X occurs at Y and Z". Such conditional relationships are not limited to the occurrence of results immediately after the antecedents are obtained, as some results may be delayed, and in the conditional statements, the antecedents are associated with their results, for example, the antecedents are related to the results that may occur later. A statement in which multiple attributes or functions are mapped to multiple objects (e.g., one or more processors that perform steps A, B, C, and D) includes both the cases where all of these attributes or functions are mapped to all of these objects or objects and the cases where a subset of the attributes or functions is mapped to a subset of the attributes or functions (e.g., all processors perform steps AD, and processor 1 performs step A, processor 2 performs step B and part of step C, and processor 3 performs step C and part of step D), unless otherwise stated. In addition, unless otherwise stated, a statement that a value or action is "based on" another condition or value includes both the case where the condition or value is the only factor and the case where the condition or value is one of multiple factors. Unless otherwise stated, a statement that "each" instance of a set has a certain attribute should not be understood to exclude the case where some other identical or similar members of the larger set do not have the attribute, that is, each does not necessarily represent every one. Unless explicitly stated (e.g., using explicit language such as "after performing X, performing Y"), limitations regarding the order of recited steps should not be read into the claims, unlike statements that may inappropriately imply a sequential order (e.g., "perform X on the item, perform Y on the item after X"), in order to make the claim more readable rather than to specify an order. Statements that refer to "at least Z of A, B, and C" and the like (e.g., "at least Z of A, B, or C") refer to at least Z of the listed categories (A, B, and C), and do not require at least Z in every category.Unless expressly stated otherwise, it will be apparent from the discussion that throughout the specification, discussions utilizing terms such as "processing," "computing," "calculating," "determining," etc. refer to the actions or processes of a specific apparatus (e.g., a special purpose computer or similar special purpose electronic processing / computing device).

[0277] In this patent, certain U.S. patents, U.S. patent applications, or other materials (e.g., articles) have been incorporated by reference. However, the text of such U.S. patents, U.S. patent applications, and other materials is incorporated herein by reference only to the extent that there is no conflict between such materials and the statements and drawings set forth herein. In the event of such a conflict, the text of this document controls.

[0278] The present technology will be better understood with reference to the following set of enumerated embodiments referred to as Group A:

[0279] 1. A tangible, non-transitory machine-readable medium storing instructions, which, when executed by one or more processors, implements operations including: receiving a write command by using one or more processors, the write command requesting that a document associated with the write command be stored in an immutable data structure, the immutable data structure preventing an attacker attempting to modify the document from concealing that the document was modified after the document was stored in the data structure; forming at least a portion of a directed acyclic graph by using one or more processors, the directed acyclic graph having a plurality of nodes and edges linking corresponding pairs of nodes of the nodes, wherein: the nodes of the directed acyclic graph have corresponding node identifiers, the node identifiers distinguishing different nodes of the directed acyclic graph; the nodes of the directed acyclic graph have node contents; the directed acyclic graph has at least one node content. Each of the at least some nodes of the directed acyclic graph has node content including multiple encrypted hash values; the encrypted hash value of each node is (i) an encrypted hash function output based on the node content of another corresponding node of the directed acyclic graph, and (ii) is associated with the node identifier of the other corresponding node, thereby specifying an edge of the directed acyclic graph; and forming at least a portion of the directed acyclic graph includes: writing data encoding at least some of the documents into the node content of a first node set of the directed acyclic graph; and starting from the first node set, traversing the edges of the directed acyclic graph and forming the node content of the visited nodes by determining the encrypted hash values ​​of the nodes visited via the traversal; and using one or more processors to store the directed acyclic graph in a memory.

[0280] 2. A medium according to embodiment 1, wherein: the document is a file larger than 1KB; the directed acyclic graph includes a binary tree; the document is stored in one or more leaf nodes of the binary tree; the directed acyclic graph includes a second node set that is disjoint from the first node set, the second node set is part of the binary tree, and is visited nodes during traversal of the edges of the directed acyclic graph; traversal includes forming visited nodes where traversal occurs, so that at least a portion of the binary tree is formed during the traversal; traversal includes traversing from the leaf nodes of the binary tree to the root node of the binary tree; forming at least a portion of the directed acyclic graph includes forming the entire binary tree; and storing the directed acyclic graph in a memory includes: storing three or more different instances of the directed acyclic graph in three or more different computing devices on three or more different subnetworks.

[0281] 3. A medium according to any one of embodiments 1-2, wherein the directed acyclic graph is a binary tree; at least a portion of the document is stored in one or more leaf nodes of the binary tree; and a root node of the binary tree has node content including an encrypted hash value based on the node content of each node of the binary tree.

[0282] 4. A medium according to embodiment 3, wherein: the entire document is stored in the node content of a single leaf node; the document is encoded in an encrypted form in the node content of the single leaf node; and before the document is encrypted into an encrypted form, the document is compressed using entropy coding.

[0283] 5. A medium according to embodiment 3, wherein: the document is stored in multiple leaf nodes of the binary tree; different parts of the document are stored in node contents of different leaf nodes; the node contents of nodes adjacent to the leaf nodes in the binary tree all include: a first identifier of a first adjacent leaf node adjacent to the corresponding node in the binary tree; a first encrypted hash value based on an encrypted hash of the node contents of the first adjacent leaf node; a second identifier of a second adjacent leaf node adjacent to the corresponding node in the binary tree; and a second encrypted hash value based on an encrypted hash of the node contents of the second adjacent leaf node.

[0284] 6. The medium according to embodiment 3, wherein storing the directed acyclic graph in a memory comprises: appending a node to a linked list of nodes, wherein the appended node has a node content comprising: (i) an encrypted hash value based on the node content of each node in the linked list; and (ii) an encrypted hash value based on the node content of each node in the binary tree.

[0285] 7. A medium according to any one of embodiments 1-6, comprising: receiving a read command requesting to retrieve the document from the immutable data structure; accessing the first node set of the directed acyclic graph to retrieve at least a portion of the document; determining whether at least a portion of the document has been modified at least in part by: calculating one or more encrypted hash values ​​based on node contents of the first node set; comparing the calculated one or more encrypted hash values ​​with one or more corresponding encrypted hash values ​​of node contents of nodes adjacent to the first node set in the directed acyclic graph; and determining, based on the comparison, that at least a portion of the document has not been modified.

[0286] 8. A medium according to embodiment 7, wherein: calculating the one or more cryptographic hash values ​​includes inputting node contents of at least some nodes in the first node set into a SHA-256 hash function; and comparing includes determining that the output of the function matches corresponding values ​​stored in the node contents of adjacent nodes of the directed acyclic graph.

[0287] 9. A medium according to embodiment 7, wherein determining whether at least a portion of the document has been modified comprises: based on the node content of each node in the directed acyclic graph, accessing the nodes of the directed acyclic graph having node content with a given encrypted hash value; and based on the node content of the first node set, recalculating the given encrypted hash value; and determining that the given encrypted hash value is the same as the recalculated encrypted hash value.

[0288] 10. A medium according to embodiment 9, wherein determining whether at least a portion of the document has been modified includes: traversing a chain of encrypted hash values ​​extending from a most recently added node in the directed acyclic graph to nodes adjacent to the first set of nodes; and verifying that the current node content produces a matching encrypted hash value for each encrypted hash value in the chain.

[0289] 11. The medium of embodiment 7, wherein: the data structure stores more than 100,000 documents, and the documents occupy more than 100 Gb of storage.

[0290] 12. The medium of embodiment 11, wherein: the document is retrieved and confirmed to have not been modified within 250 milliseconds of receiving the read command.

[0291] 13. A medium according to any one of embodiments 1-12, wherein: the read latency of the data structure is proportional to nlog(n), where n is the number of entries in the data structure.

[0292] 14. A medium according to any one of embodiments 1-13, wherein writing data encoding at least some of the documents into node contents of a first node set of the directed acyclic graph includes the step of storing the documents in the directed acyclic graph.

[0293] 15. The medium of any one of embodiments 1-14, wherein forming at least a portion of the directed acyclic graph comprises the step of forming the directed acyclic graph having edges corresponding to cryptographic hash pointers.

[0294] 16. A medium according to any one of embodiments 1-15, comprising: fragmenting the document into multiple fragments; and storing a first subset of the fragments in the directed acyclic graph; storing a second subset of the fragments in another directed acyclic graph, the second subset being a different set from the first subset.

[0295] 17. A medium according to embodiment 16, wherein: fragmentation includes: determining the frequency of occurrence of symbol sequences in the document or corpus; and fragmenting the document at boundaries of symbol sequences selected based on at least some of the frequencies.

[0296] 18. A medium according to any one of embodiments 1-17, comprising: storing a first subset of information from the document in the directed acyclic graph; storing a second subset of information from the document in a second directed acyclic graph; storing a third subset of information from the document in a third directed acyclic graph, wherein: each pair of combinations of the first subset of information, the second subset of information, and the third subset of information contains sufficient redundant information so that if a given subset of information that is not part of a corresponding pairwise combination is unavailable, the given subset of information can be retrieved from the corresponding pairwise combination, but cannot be retrieved from individual members of the pairwise combination.

[0297] 19. The medium according to any one of embodiments 1-18, comprising: storing a first portion of an indication of an immutable relational database in the data structure; and providing an enterprise application based on the database.

[0298] 21. A method comprising: performing an operation according to any one of embodiments 1-19.

[0299] 22. A system comprising: one or more processors; and a memory storing instructions, wherein the instructions, when executed by the processors, cause the processors to perform operations, the operations comprising: operations according to any one of embodiments 1-19.

[0300] The present technology will be better understood with reference to the following set of enumerated embodiments referred to as Group B:

[0301] 1. A tangible, non-transitory, machine-readable medium storing instructions, which, when executed by one or more processors, implement operations comprising: receiving, with one or more processors, one or more write requests to write a plurality of values ​​to a plurality of fields in one or more tuples of a relational database, different values ​​in the values ​​corresponding to different fields in the fields; selecting, with one or more processors, a first subset of the values ​​based on a first subset of the values ​​corresponding to fields in a first subset of the fields, the first subset of fields being designated as fields with a higher security than a second subset of the fields in the plurality of fields; fragmenting, with one or more processors, a first value in the first subset of values ​​into a plurality of fragments, each fragment having a subset of information in the first value, and different fragments having different subsets of information in the first value, the write request having a first value to be written to a plurality of fields in the plurality of tuples of a relational database, wherein the first value is to be written to a plurality of fields in the plurality of tuples; an indication of a given field of a given tuple being written; instructing a first computing device via a network, by one or more processors, to store a first subset of the plurality of fragments in a memory; instructing a second computing device via a network, by one or more processors, to store a second subset of the plurality of fragments in a memory, wherein: the first computing device cannot access the second subset of fragments; and the second computing device cannot access the first subset of fragments; and using one or more processors, causing a relational database to store a second subset of values ​​and one or more pointers to at least some of the fragments, wherein: in response to one or more write requests, the first subset of values ​​is not written to the relational database; and the one or more pointers are stored in the relational database in association with the given field of the given tuple, and the one or more write requests request that the first value be written to the given field of the given tuple.

[0302] 2. A medium according to embodiment 1, wherein: the first fragment is stored at the first computing device together with a pointer to the second fragment; and the pointer or a value associated with the pointer indicates that the second fragment is stored at the second computing device, wherein the second computing device is one of multiple computing devices that store instances of the second fragment.

[0303] 3. A medium according to any one of embodiments 1-2, wherein: the one or more pointers are a single pointer to the first fragment and indicate that the first fragment is stored at the first computing device, or are associated with a value indicating that the first fragment is stored at the first computing device.

[0304] 4. A medium according to any one of embodiments 1-3, wherein the one or more pointers include: a first pointer pointing to the first fragment, the first pointer indicating that the first fragment is stored at the first computing device or is associated with a value indicating that the first fragment is stored at the first computing device; and a second pointer pointing to the second fragment, the second pointer indicating that the second fragment is stored at the second computing device or is associated with a value indicating that the second fragment is stored at the second computing device.

[0305] 5. A medium according to any one of embodiments 1-4, wherein: five or more of the multiple fragments are stored on five or more different computing devices; the five or more different computing devices include a first computing device and a second computing device; and the five or more computing devices are unable to store or otherwise access at least three of the five or more fragments.

[0306] 6. A medium according to any one of embodiments 1-5, wherein: the fragments are stored on three or more computing devices as content nodes of a content directed acyclic graph, and the content directed acyclic graph includes edges defined by pointers from fragments to other fragments.

[0307] 7. The medium of embodiment 6, wherein: the directed acyclic graph comprises a linked list of three or more nodes.

[0308] 8. A medium according to any of embodiments 6-7, wherein: the directed acyclic graph includes a skip list having 1,000 or more nodes.

[0309] 9. A medium according to any one of embodiments 6-8, wherein: the directed acyclic graph includes a binary tree.

[0310] 10. A medium according to any one of claims 6-9, wherein: the content directed acyclic graph is distributed among multiple verification directed acyclic graphs; the verification directed acyclic graphs each include multiple verification nodes linked by verification edges; the verification directed acyclic graphs each store different subsets of content nodes as verification node attributes of different nodes in the verification nodes; the verification edges of the verification directed acyclic graphs are at least partially defined by encrypted hash pointers linked between the verification nodes of the verification directed acyclic graphs; and the encrypted hash pointers indicate whether one or more verification node attributes of the corresponding verification node pointed to by the corresponding encrypted hash pointer have been modified by encoding the encrypted hash value based on one or more verification node attributes of the verification node pointed to by the corresponding encrypted hash pointer; and the verification node attributes of at least some of the verification nodes include encrypted hash values.

[0311] 11. A medium according to embodiment 10, wherein: verifying a given one of the directed acyclic graphs includes a linked list of verification nodes, the linked list having pointers to a binary tree of corresponding verification nodes; and a given fragment in the fragments is stored as a verification node attribute of a leaf verification node of the given one of the binary trees.

[0312] 12. A medium according to embodiment 10, wherein: each verification directed acyclic graph includes a corresponding linked list of verification nodes, which has a pointer to the corresponding binary tree of the verification nodes; and the content directed acyclic graph is a linked list of pointers between leaf verification nodes of the binary tree in the verification directed acyclic graph.

[0313] 13. A medium according to embodiment 10, wherein: the cryptographic hash is a SHA-256 hash of one or more attributes of adjacent verification nodes in a corresponding verification directed acyclic graph.

[0314] 14. A medium according to any one of embodiments 1-13, comprising: after an application queries a relational database, receiving a query response from the relational database, the relational database including at least one or more pointers; in response to receiving the query response, determining that at least some of the query responses are stored outside the relational database; in response to the determination, requesting multiple fragments from multiple computing devices including a first computing device and a second computing device based on at least some of the one or more pointers; combining the multiple fragments to form a first value; forming a revised version of the query response, wherein at least some of the one or more pointers are replaced by the first value; and providing the revised version of the query response to the application.

[0315] 15. A medium according to embodiment 14, wherein requesting multiple fragments includes: requesting a first fragment identified by at least some of the one or more pointers; and receiving the first fragment and a first associated pointer pointing to the second fragment; requesting the second fragment at a computing device indicated by the first associated pointer or a value associated therewith; receiving the second fragment and a second associated pointer pointing to the third fragment; requesting the third fragment at a computing device indicated by the third associated fragment; and receiving the third fragment.

[0316] 16. The medium of embodiment 14, comprising: determining a rate at which data is requested from a relational database; and determining that the rate exceeds a threshold; and in response to the determination, delaying providing at least a portion of a revised version of a query response to an application.

[0317] 17. A medium according to embodiment 14, wherein: in response to the write request, each fragment is stored on three or more different computing devices; after receiving the query response, the value of the given fragment retrieved from the three or more different computing devices is determined based on the value of the given fragment determined to be returned by a majority of the three or more different computing devices.

[0318] 18. The medium of embodiment 14, comprising: writing a record describing at least a portion of a query or a query response to a blockchain ledger, wherein the record is associated with an identifier of a user that caused the query.

[0319] 19. A method comprising: performing the operations according to any one of embodiments 1-18.

[0320] 20. A system comprising: one or more processors; and a memory storing instructions, wherein the instructions, when executed by the processors, cause the processors to perform operations, the operations comprising: operations according to any one of embodiments 1-18.

[0321] The present technology will be better understood with reference to the following set of enumerated embodiments referred to as Group C:

[0322] 1. A tangible, non-transitory, machine-readable medium storing instructions, which, when executed by one or more processors, implement operations comprising: obtaining, with one or more processors, a plurality of records to be protected; forming, with one or more processors, a tamper-evident log, the tamper-evident log being configured to prevent an attacker from undetectably modifying any of the plurality of records stored in the tamper-evident log, wherein: the tamper-evident log comprises a plurality of entries sequentially added to the tamper-evident log at different times according to a sequence of entries specified by the tamper-evident log; each entry comprises: an identifier distinguishing the entry, a corresponding record in the plurality of records, a tamper-evident date The invention relates to a method for storing a tamper-evident log in a memory comprising: generating a cryptographic hash value for a given entry in the tamper-evident log, the cryptographic hash value for a given entry being sequence-agnostic to the sequence of entries due to being based on a value that does not specify a position in the sequence of entries; forming the cryptographic hash value for the given entry by selecting which types of parameters are input to a cryptographic hash function used to calculate at least some of the cryptographic hash values ​​so as to exclude parameters that specify positions in the sequence; and storing the tamper-evident log in a memory using one or more processors.

[0323] 2. A medium according to embodiment 1, wherein: the selected parameter type does not include the time, date, and order of the sequence of entries specified by the tamper-evident log; and the selected parameter type includes records recorded by the tamper-evident log, and multiple records of the tamper-evident log are corresponding instances; and the cryptographic hash function is configured by providing a lambda function.

[0324] 3. The medium of embodiment 2, wherein: the selected parameter type consists essentially of records acquired to be recorded by a tamper-evident system, the plurality of records of the tamper-evident system being corresponding instances.

[0325] 4. The medium of embodiment 3, wherein: the selected parameter type consists of records acquired to be recorded by a tamper-evident system, the plurality of records of the tamper-evident system being corresponding instances.

[0326] 5. A medium according to any one of embodiments 1-4, wherein selecting which types of parameters to input into the cryptographic hash function includes: determining that the cryptographic hash function or method is designated as a digest function or method in a programming language that supports delegation or inheritance; and determining a non-digest implementation of the digest class or method, the implementation including a selection of parameter types.

[0327] 6. A medium according to any one of embodiments 1-5, wherein: each record is based on a cryptographic hash value of an attribute of each node in a corresponding directed acyclic graph having 16 or more nodes.

[0328] 7. A medium according to embodiment 6, wherein: each directed acyclic graph is a binary tree; the leaf nodes of the binary tree store the values ​​referenced by the placeholders in a relational database; and each record is an encrypted hash value based on the value referenced by the placeholder in the database.

[0329] 8. A medium according to any one of embodiments 1-7, comprising: splitting a tamper-evident log at or near a given entry into a newer tamper-evident log and an older tamper-evident log, wherein the encrypted hash values ​​in both the newer tamper-evident log and the older tamper-evident log are valid relative to the protected record after the split.

[0330] 9. The medium of embodiment 8, wherein the splitting is performed without changing each encrypted hash value in the newer tamper-evident log or the older tamper-evident log.

[0331] 10. The medium of embodiment 9, wherein: the splitting is performed without changing any cryptographic hash values ​​in the newer tamper-evident log or the older tamper-evident log.

[0332] 11. The medium of claim 8, wherein the splitting comprises changing pointers between entries in the newer tamper-evident log without changing cryptographic hash values ​​in the newer tamper-evident log.

[0333] 12. The medium of embodiment 8, wherein: the operation includes determining that the size of the tamper-evident log has or will exceed a threshold; and splitting in response to determining that the size of the tamper-evident log has or will exceed the threshold.

[0334] 13. A medium according to any one of embodiments 1 to 12, wherein: the cryptographic hash value of each entry in the tamper-evident log is sequence-agnostic to the sequence of entries due to being based on values ​​at positions in an unspecified sequence of entries.

[0335] 14. The medium of any one of embodiments 1-13, wherein different cryptographic hash functions implement different cryptographic hash values ​​for different entries.

[0336] 15. A medium according to any one of embodiments 1-14, wherein: the cryptographic hash function repeatedly includes: for an input longer than the one-way compression function configured to accept the cryptographic hash function, performing the following operations until the beginning or end of the input input to the cryptographic hash function is reached: obtaining a parsed portion by parsing a remaining portion of the input, the length of which is determined based on a difference between the input length of the one-way compression function configured to accept the cryptographic hash function and the length of the output of the one-way compression function configured to emit; appending or prepending the output of a previous iteration of the one-way compression function to the parsed portion to form an input of a current iteration of the one-way compression function; converting the input of the current iteration using the one-way compression function to form the output of the one-way function.

[0337] 16. The medium of any one of embodiments 1-15, wherein: the operation comprises recording a database access request to the tamper-evident log, wherein the recording is based on the recorded database access request.

[0338] 17. A medium according to any one of embodiments 1-16, wherein: the tamper-evident log is part of a tamper-evident log collection, the tamper-evident log collection stores more than one million database values, each database value is distributed in multiple tamper-evident logs in the tamper-evident log collection; and the collection of tamper-evident log collections is configured so that database values ​​can be retrieved within 100 milliseconds of a query; the collection of tamper-evident log collections is configured so that the retrieval latency of the database value scales according to nlog(n), where n is the number of database values ​​in the tamper-evident log collection; and the operation includes limiting the response to the retrieval based on a risk assessment associated with the retrieval.

[0339] 18. A medium according to any one of embodiments 1-17, comprising: executing an application, wherein the application generates data on which the entry is based; and intercepting a write request from the application to a database, wherein the data on which the entry is based is stored in the database; and replacing a pointer in the write request, and causing the pointer to be stored in the database in place of the data on which the entry is based.

[0340] 19. A method comprising: performing the operations according to any one of embodiments 1-18.

[0341] 20. A system comprising: one or more processors; and a memory storing instructions, wherein the instructions, when executed by the processors, cause the processors to perform operations, the operations comprising: operations according to any one of embodiments 1-18.

[0342] The present technology will be better understood with reference to the following set of enumerated embodiments referred to as Group D:

[0343] 1. A method for modifying an application, wherein the application is configured to access a given data repository, instead of accessing a heterogeneous collection of data repositories, rather than the given data repository, to obtain at least some data, the method comprising: receiving a first write request from the application via a network using a computing device, the application requesting to write data to a first remote database, wherein: the first write request specifies a first value to be written to a first field in the first remote database; the first write request specifies a second value to be written to a second field in the first remote database; the first write request complies with an application program interface of a database driver configured to interface with the first remote database; and the database driver and at least a portion of the application issuing the first write request are executed in the same instance as an operating system executed on the computing device; using the computing device to obtain a data policy having one or more rules, by which a value is classified as a higher security value or a lower security value; using the computing device and at least one of the one or more rules to classify the first value as a lower security value classifying the second value as having a higher security by utilizing the computing device and at least one of the one or more rules; in response to classifying the second value as having a higher security, redirecting the second value from a destination specified by the application in a received first write request by: obtaining a unique identifier for the second value that does not reveal the second value; causing the second value to be stored in a second remote database different from the first remote database; and updating an index using an entry that maps the unique identifier to the second value in the second remote database; and forming one or more application program interface requests to a database driver by utilizing the computing device, the requests: (i) instructing the database driver to cause the first remote database to store the first value in association with the first field; (ii) instructing the database driver to cause the first remote database to store the unique identifier in association with the second field, wherein the second value classified as having a higher security is inaccessible to an attacker having full access to the first database; and wherein the code of the application is not modified to interface with two databases instead of the first remote database.

[0344] 2. A method according to embodiment 1, wherein: receiving the first write request is performed using a security driver registered in the operating system as a database driver that the application is configured to call; and the security driver implements an application program interface wrapper for the application program interface of the database driver.

[0345] 3. A method according to any one of embodiments 1-2, wherein: the first value and the second value correspond to the same record in a given tuple of a relational database; the given tuple includes a primary key value and one or more foreign key values; and a database driver causes the relational database to store a unique identifier in the given tuple.

[0346] 4. The method according to any one of embodiments 1-3, wherein: the index maps the unique identifier to multiple uniform resource identifiers of multiple different replication instances of the second remote database.

[0347] 5. The method according to any one of embodiments 1-4, wherein: the index is stored in a location remote from the computing device and updated by another computing device; and the unique identifier is obtained from the other computing device.

[0348] 6. The method of any one of embodiments 1-5, wherein the unique identifier comprises a cryptographic hash value based on the second value.

[0349] 7. The method according to embodiment 6 includes: calculating an encrypted hash value using an encrypted hash function with a second value as input, wherein: the second value is longer than a one-way compression function configured to accept the encrypted hash function; and the encrypted hash function includes an operation, which includes: repeating until a beginning or an end of the second value is reached: obtaining a parsed portion by parsing a remaining portion of the second value, the length of which is determined based on a difference between an input length of the one-way compression function configured to accept the encrypted hash function and a length of an output of the one-way compression function configured to emit; appending or prepending an output of a previous iteration of the one-way compression function to the parsed portion to form an input of a current iteration of the one-way compression function; converting the input of the current iteration with the one-way compression function to form an output of the one-way function.

[0350] 8. The method according to any one of embodiments 1-7 comprises: receiving a second write request from the application using the computing device, the second write request requesting to write a third value associated with the second field to the first remote database, wherein the third value is equal to the second value; classifying the third value as having a higher security; after classifying the third value as having a higher security, instructing a database driver to cause the first remote database to store a unique identifier associated with the second field in two different records, one of the two different records having data from the first write request, and the other of the two different records having data from the second write request.

[0351] 9. The method of embodiment 8, wherein the operation comprises: after receiving the second write request: determining that the unique identifier already exists in the index; and in response to the determination, determining not to add a new entry corresponding to the third value that is different from the existing entry to the index.

[0352] 10. A method according to any one of embodiments 1-9, wherein: the second database includes multiple blockchains; and the index maps the unique identifier to a given blockchain in the blockchain; the second value is fragmented into multiple fragments; different fragments of the multiple fragments are stored in different blockchains in the multiple blockchains.

[0353] 11. The method according to any one of embodiments 1-10 includes: using the computing device to receive a query response via the database driver based on a query from the application to the first database rather than the second database; detecting a unique identifier in the query response; obtaining a second value from a second remote database using the unique identifier; forming a revised query response in which the unique identifier is replaced with the first value; and providing the revised query response to the application as a response to the query.

[0354] 12. A method according to claim 11, wherein: the unique identifier is not provided to the application; the method comprising determining that the second value is responsive to the criteria of the query before replacing the unique identifier with the second value in the revised query response.

[0355] 13. The method according to embodiment 11 includes: determining that the query response includes two instances of the unique identifier associated with two different records in the first remote database, the two records respectively including the second value and the third value, the third value being the same as the second value; and retrieving a single value mapped to the second value and the third value from the second remote database.

[0356] 14. The method according to embodiment 11 includes: determining a total amount of data requested in a read request associated with the application or the credentials used by the application to request access; and determining that the total amount of data requested does not exceed a threshold.

[0357] 15. The method according to embodiment 14 includes: after receiving another read request, determining that the total amount of updated data requested in the read request associated with the application or the credentials used by the application to request access exceeds a threshold; and in response to the determination, delaying or blocking a query response associated with the other read request.

[0358] 16. A method according to any one of embodiments 1-15, wherein: a data policy having one or more rules includes conditions based on fields designated as having higher security.

[0359] 17. A method according to any one of embodiments 1-16, wherein: a data policy having one or more rules includes a condition based on a regular expression indicating a higher security value.

[0360] 18. A method comprising: performing the operations according to any one of embodiments 1-17.

[0361] 19. A system comprising: one or more processors; and a memory storing instructions, wherein the instructions, when executed by the processors, cause the processors to perform operations, the operations comprising: operations according to any one of embodiments 1-17.

[0362] The present technology will be better understood with reference to the following set of enumerated embodiments referred to as Group E:

[0363] 1. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by one or more processors, implement operations including: receiving, with one or more processors, a request to access a distributed data store, wherein: the request is a request to read or write a content unit associated with a workload application; the distributed data store stores a plurality of content units, each content unit being distributed among a plurality of computing entities that have a different subset of data residing in the distributed data store; each of the plurality of computing entities corresponds to a different host at a different user space instance of one or more different computing devices; each content unit requires information from the plurality of computing entities to read the corresponding content unit; when writing each content unit, different portions of the information required to access the corresponding content unit are written to different computing entities in the plurality of computing entities, such that no one of the plurality of computing entities stores all of the information required to access the corresponding content unit; using one or more processors of a computing device configured to participate in combining information from the plurality of computing entities to access the content unit, recording the request in an entry in a tamper-evident log, wherein the tamper-evident log defines one or more sequences of cryptographic hash values ​​based on previously recorded entries; and storing the tamper-evident log in a memory using the one or more processors.

[0364] 2. According to the medium of embodiment 1, the operation includes: determining whether the tamper-evident log shows tampering with log entries based on the consistency of at least one of the encrypted hash value sequences with at least some entries in the tamper-evident log.

[0365] 3. The medium of embodiment 2, wherein the operations include: determining a risk metric based on requests recorded in a tamper-evident log and other access requests; and storing the risk metric in a memory.

[0366] 4. The medium of embodiment 3, wherein the operation comprises: determining that the risk metric satisfies a threshold; and in response to the determination, delaying a subsequent response to a subsequent access request.

[0367] 5. The medium of embodiment 3, wherein the operation comprises: determining that the risk metric satisfies a threshold; and in response to the determination, blocking a subsequent response to the access request.

[0368] 6. The medium of embodiment 3, wherein the operation comprises: determining that the risk metric satisfies a threshold; and in response to the determination, disabling a user account associated with the request.

[0369] 7. The medium of embodiment 6, wherein disabling the user account associated with the request comprises disabling the user account during an active session in which the user account is authenticated in real time based on the determination.

[0370] 8. According to the medium described in any one of embodiments 1-7, the operation includes: selecting a computing entity storing a given content unit related to an access request from multiple entities; requesting different parts of information of the given content unit from the selected computing entity; and combining different parts of the information to re-form the given content unit.

[0371] 9. A medium according to embodiment 8, wherein: the multiple computing entities are different computing devices on multiple different subnetworks, each subnetwork is separated from the Internet by different firewalls; different parts of the information are related to different fragments of the content unit, or to the encrypted form of the ciphertext of the content unit and the encryption key that can decrypt the ciphertext.

[0372] 10. The medium of any one of embodiments 1-9, wherein the tamper evident log stores both information used to access content units and logged access requests.

[0373] 11. A medium according to any one of embodiments 1-10, wherein: the tamper-evident log includes a blockchain having encrypted hash pointers between sequential blocks of the blockchain and a Merkle tree within the blocks of the blockchain, wherein the encrypted hash pointers are based on an implementation Hash functions are cryptographic hash functions.

[0374] 12. A medium according to embodiment 11, wherein: determining an authoritative entry in a distributed set of instances of the blockchain does not require proof of work from a computing device that implements each of the set of instances.

[0375] 13. A medium according to any one of embodiments 1-12, wherein a given entry in the tamper-evident log includes: an identifier of a user account with credentials through which authorization is requested to perform a corresponding database access operation; and a timestamp of the corresponding database access operation.

[0376] 14. The medium according to embodiment 13, wherein the given entry further comprises: a database application program interface command, and the corresponding access operation is expressed by the database application program interface command.

[0377] 15. A medium according to any one of embodiments 1-14, wherein: recording the request in the entry includes: recording a cryptographic hash value based on a record, wherein the record describes the time of the access request, a user account associated with the access request, and an access request command.

[0378] 16. The medium of embodiment 15, wherein the record is stored in the tamper-evident log in association with the cryptographic hash value.

[0379] 17. The medium of embodiment 15, wherein the record is not stored in the tamper-evident log and is stored in a data repository different from the tamper-evident log in association with a pointer to the cryptographic hash value based on the record.

[0380] 18. The medium according to any one of embodiments 1-17 comprises: using historical entries recorded in the tamper-evident log to train a machine learning algorithm; and determining whether a subsequent access request is abnormal based on the trained machine learning algorithm.

[0381] 19. A method comprising: performing the operations according to any one of embodiments 1-18.

[0382] 20. A system comprising: one or more processors; and a memory storing instructions, wherein the instructions, when executed by the processors, cause the processors to perform operations, the operations comprising: operations according to any one of embodiments 1-18.

[0383] The present technology will be better understood with reference to the following set of enumerated embodiments referred to as Group F:

[0384] 1. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by one or more processors, implement operations, the operations comprising: receiving, by the one or more processors, a request to write a new version of a document to a tamper-evident immutable data repository, wherein: the tamper-evident immutable data repository stores a prior version of the document; the tamper-evident immutable data repository stores multiple versions of multiple other documents; and the tamper-evident immutable data repository stores versions of the document and other documents in a data structure that makes it computationally infeasible to modify previously stored versions of the document without causing the data structure to be internally inconsistent; upon receiving the request, determining, by the one or more processors, that the new version of the document is different from the prior version of the document; and in response to determining that the new version of the document is different from the prior version of the document, The invention relates to a method for storing a difference between a previous version of a document and a new version of a document by using one or more processors to: obtain a previous version of the document from an immutable data repository that is tamper-evident; determine a set of changes by comparing the new version of the document with the previous version of the document, and when the set of changes is made to the previous version of the document, the previous version of the document is converted to the new version of the document; store the set of changes in the immutable data repository that is tamper-evident, rather than storing a complete copy of the new version of the document in the immutable data repository that is tamper-evident; and store a pointer to a previous version of the document in association with the set of changes, the pointer directly or indirectly indicating where in the immutable data repository that is tamper-evident, a previous version of the document is obtained, the previous version being an initial version or a set of differences.

[0385] 2. A medium according to embodiment 1, wherein: receiving a request to write a new version of the document includes: receiving a request to a file system driver executing on the computing device from a workload application executing in an operating system on the computing device; the tamper-evident immutable data repository is not stored on the computing device; the file system of the computing device includes pointers to documents stored in the tamper-evident immutable data repository; the pointers are arranged in a hierarchical directory data structure; the pointers appear to be files in a file system resource manager workload application executing on the computing device; the pointers are marked with the file system file name of the document; one of the pointers references a previous version of the document in the tamper-evident immutable data repository and is marked in the file system with the file system file name of the document.

[0386] 3. A medium according to any one of embodiments 1-2, wherein: the tamper-evident immutable data repository includes a blockchain.

[0387] 4. A medium according to any one of embodiments 1-3, wherein: the tamper-evident immutable data repository includes multiple directed acyclic graphs having edges defined by cryptographic hash pointers based on versions of documents stored in the tamper-evident immutable data repository.

[0388] 5. A medium according to any one of embodiments 1-4, wherein determining that the new version of the document is different from the previous version of the document comprises: calculating a hash digest of the new version of the document; accessing a hash digest of a previous version of the document stored outside of a tamper-evident immutable data repository; and determining that the hash digest of the new version of the document is different from the hash digest of the previous version of the document, wherein determining that the new version of the document is different from the previous version of the document is performed before retrieving the previous version of the document from the tamper-evident immutable data repository in response to a write request.

[0389] 6. A medium according to any one of embodiments 1-5, wherein obtaining a prior version of the document comprises: retrieving a prior difference between the prior version of the document and another version of the document before the prior version, the prior difference comprising another change set, the change set converting the other version of the document before the prior version to the prior version of the document; accessing a pointer associated with the prior difference, the pointer identifying a location of a copy of the other version of the document before the prior version in the tamper-evident immutable data repository; retrieving the other version of the document before the prior version using the pointer; and applying another change set to the retrieved other version of the document to form the prior version of the document.

[0390] 7. A medium according to any one of embodiments 1-6, wherein obtaining the previous version of the document comprises: in three or more iterations, through three or more previous versions of the document, iteratively and sequentially applying the previous change set, retrieving versions starting from the latest version until the initial version is reached, and then combining the versions starting from the initial version of the document until the previous version is reached to form the previous version.

[0391] 8. The medium of embodiment 7, wherein: the three or more prior versions form a version graph for the document, the version graph comprising a set of changes to the document between writes to the tamper-evident immutable data repository.

[0392] 9. A medium according to embodiment 8, wherein: one of the three or more previous versions of the document is branched into a different document based on one of the three or more previous versions of the document; and after writing, current versions of two documents in the tamper-evident immutable data repository share an overlapping subset of the previous versions.

[0393] 10. A medium according to any one of embodiments 1-9, wherein: a version graph of a document is stored in a verification graph of a tamper-evident immutable data repository, wherein version nodes of the version graph correspond to sets of changes between consecutive writes to the tamper-evident immutable data repository and edges of the version graph indicate an order of the sets of changes.

[0394] 11. A medium according to embodiment 10, wherein: a node of a version graph of a document has multiple changes stored in multiple fragments, and the multiple fragments are stored in nodes of a content graph distributed in multiple verification graphs of an immutable data repository that is tamper-evident.

[0395] 12. A medium as described in any one of embodiments 1-11, wherein determining the set of changes includes: determining a minimum set of changes required to convert a previous version of the document to the new version of the document.

[0396] 13. The medium according to any one of embodiments 1-12, wherein the change set mainly includes deletion, modification and addition operations.

[0397] 14. The medium of embodiment 12, wherein determining the set of changes comprises hashing each line of each of the new version of the document and the previous version of the document into a corresponding hash digest of 4 bytes or less, and comparing the hash digests to identify that the line has been changed.

[0398] 15. The medium of any one of embodiments 1-14, wherein determining the set of changes comprises finding a longest common sequence between the new version of the document and a previous version of the document.

[0399] 16. A medium according to any one of embodiments 1-15, wherein the operation includes reading a new version of a document from a tamper-evident immutable data repository by: retrieving an initial version of the document by retrieving each in a sequence of change sets from a tamper-evident log along a linked list of change sets to trace back from the latest version; and applying these changes in sequence, wherein: the data repository stores more than 100,000 documents and more than 100GB of data; the operation retrieves the document in less than 200 milliseconds, and the latency of the operation is proportional to or faster than nlog(n), where n is the number of versions of the document in the tamper-evident immutable data repository; and the read request or write request is recorded in the tamper-evident immutable data repository or another tamper-evident immutable data repository.

[0400] 17. A medium according to any one of embodiments 1-16, wherein the operation includes: executing the workload application that issues the write request within a computing device on an enterprise network configured to access multiple enterprise workload applications.

[0401] 18. A method comprising: performing the operations according to any one of embodiments 1-17.

[0402] 19. A system comprising: one or more processors; and a memory storing instructions, wherein the instructions, when executed by the processors, cause the processors to perform operations, including the operations according to any one of embodiments 1-17.

Claims

1. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by one or more processors, perform operations comprising: Retrieving data to be stored by receiving, using one or more processors, a write command requesting that a document associated with the write command be stored in an immutable data structure, the immutable data structure preventing an attacker from concealing that the document was modified after the document was stored in the immutable data structure; storing at least some of the data in an immutable data structure, the immutable data structure comprising a directed acyclic graph having a plurality of nodes and edges linking respective pairs of the nodes, wherein: The nodes of the directed acyclic graph have corresponding node identifiers, and the node identifiers distinguish different nodes of the directed acyclic graph; The nodes of the directed acyclic graph have node content; Each of at least some of the nodes of the directed acyclic graph has node content including a plurality of encrypted hash values; The cryptographic hash value of each node is (i) an output of a cryptographic hash function based on the node contents of another corresponding node of the directed acyclic graph and (ii) associated with a node identifier of the other corresponding node, thereby specifying an edge of the directed acyclic graph; and Storage includes: writing data encoding at least some of the documents into node contents of a first set of nodes of the directed acyclic graph; and Starting from the first set of nodes, traversing the edges of the directed acyclic graph, and forming node contents of the visited nodes by determining cryptographic hash values ​​of the nodes visited via the traversal; accessing at least some of the data stored in the immutable data structure; and The immutable data structure is used to verify that the accessed data has not been modified.

2. The medium according to claim 1, wherein The operations include: At least some of the data is stored in an immutable data structure by storing the directed acyclic graph in a memory using one or more processors.

3. The medium of claim 1, wherein: The document is a file larger than 1 KB; The directed acyclic graph comprises a binary tree; The document is stored in one or more leaf nodes of the binary tree; The directed acyclic graph includes a second set of nodes that is disjoint from the first set of nodes, the second set of nodes being part of the binary tree and being visited nodes during traversal of the edges of the directed acyclic graph; The traversal includes forming visited nodes at which the traversal occurs such that at least a portion of the binary tree is formed during the traversal; The traversal includes traversing from a leaf node of the binary tree to a root node of the binary tree; Forming at least a portion of the directed acyclic graph includes forming an entire binary tree; as well as Storing the directed acyclic graph in a memory includes storing three or more different instances of the directed acyclic graph in three or more different computing devices on three or more different subnetworks.

4. The medium of claim 1, wherein: The directed acyclic graph is a binary tree; At least a portion of the document is stored in one or more leaf nodes of the binary tree; and A root node of the binary tree has node content including a cryptographic hash value based on the node content of each node of the binary tree.

5. The medium of claim 4, wherein: The entire document is stored in the node content of a single leaf node; encoding the document in encrypted form in the node content of the single leaf node; as well as The document is compressed using entropy coding before being encrypted into an encrypted form.

6. The medium of claim 4, wherein: The documents are stored in a plurality of leaf nodes of the binary tree; Different parts of the document are stored in node contents of different leaf nodes; The node contents of the nodes adjacent to the leaf node in the binary tree all include: a first identifier of a first adjacent leaf node adjacent to the corresponding node in the binary tree; a first encrypted hash value based on an encrypted hash of the node content of the first adjacent leaf node; a second identifier of a second adjacent leaf node adjacent to the corresponding node in the binary tree; and A second encrypted hash value based on an encrypted hash of the node content of the second adjacent leaf node.

7. The medium according to claim 4, wherein Storing the directed acyclic graph in a memory includes appending a node to a linked list of nodes, the appended node having node content including: (i) a cryptographic hash value based on the node content of each node in the linked list; and (ii) a cryptographic hash value based on the node content of each node in the binary tree.

8. The medium according to claim 1, comprising: receiving a read command requesting retrieval of the document from the immutable data structure; accessing the first set of nodes of the directed acyclic graph to retrieve at least a portion of the document; Determining whether at least a portion of the document has been modified is performed at least in part by: Calculate one or more encrypted hash values ​​based on the node contents of the first set of nodes; comparing the calculated one or more cryptographic hash values ​​to one or more corresponding cryptographic hash values ​​of node contents of nodes in the directed acyclic graph that are adjacent to the first set of nodes; as well as Based on the comparison, it is determined that at least a portion of the document has not been modified.

9. The medium of claim 8, wherein: Calculating the one or more cryptographic hash values ​​includes: inputting node contents of at least some of the nodes in the first set of nodes into a SHA-256 hash function; and The comparing includes determining that an output of the SHA-256 hash function matches a corresponding value stored in node content of nodes adjacent to the first set of nodes in the directed acyclic graph.

10. The medium according to claim 8, wherein Determining whether at least a portion of the document has been modified includes: Based on the node content of each node in the directed acyclic graph, accessing a node of the directed acyclic graph having a node content with a given encrypted hash value; and recalculating a given encrypted hash value based on the node contents of the first set of nodes; and Determines whether a given cryptographic hash value is identical to a recomputed cryptographic hash value.

11. The medium according to claim 10, wherein Determining whether at least a portion of the document has been modified includes: traversing a chain of encrypted hash values ​​extending from a most recently added node in the directed acyclic graph to nodes adjacent to the first set of nodes; and Verifying that the current node content produces a matching cryptographic hash value for each cryptographic hash value in the cryptographic hash value chain.

12. The medium of claim 8, wherein: The immutable data structure stores over 100,000 documents, which occupy over 100 Gb of storage; and Retrieve the document within 250 milliseconds of receiving the read command and verify that the document has not been modified: The read latency of the immutable data structure is proportional to nlog(n), where n is the number of entries in the immutable data structure.

13. The medium according to claim 1, comprising: fragmenting the document into a plurality of fragments; as well as storing a first subset of the segments in the directed acyclic graph; A second subset of the segments is stored in another directed acyclic graph, the second subset being a different set than the first subset.

14. The medium according to claim 1, comprising: storing a first subset of information from the document in the directed acyclic graph; storing a second subset of information from the document in a second directed acyclic graph; A third subset of information from the documents is stored in a third directed acyclic graph, wherein: Each pairwise combination of said first subset of information, said second subset of information, and said third subset of information contains sufficient redundant information such that a given subset of information can be retrieved from the pairwise combination.

15. A method comprising: Operation according to claim 1 and any one of claims 3 to 14.

Images