Method and system for data storage

Abstract

A method of data storage is disclosed comprising: adding a hash to a Merkle mountain range (MMR) stored in a local memory, wherein the MMR is persisted to a second MMR stored in a remote memory, the remote memory also comprising cryptographic information; copying the second MMR and the cryptographic information to the local memory and obtaining a version identifier identifying the copied data; determining a total storage size of the data added to the MMR; determining whether a portion of the remote memory has capacity for the data; and if the portion has capacity: writing the second MMR with the hash back to the remote memory and providing the version identifier; or if the portion has insufficient capacity: writing the hash, the cryptographic information, and further cryptographic information to a new portion of the remote memory; and transmitting the version identifier. A corresponding device is also disclosed.

Description

[0001] Method and system for data storage

[0002] FIELD OF THE INVENTION

[0003] The invention relates to preserving and verifying data integrity, especially in distributed systems.

[0004] BACKGROUND TO THE INVENTION

[0005] Present means of ensuring the authenticity of data include cryptographic enveloping, operating system controls, digital signatures, or metadata added to files. The extension of these methods to distributed systems often involves the use of consensus mechanisms to ensure that data is modified only in ways that may be tracked and agreed upon by multiple parties, in the hope of preventing malicious actors from falsifying or back-dating particular data.

[0006] SUMMARY OF THE INVENTION

[0007] The inventors have recognised that existing methods of ensuring data integrity may be vulnerable with regard to data that moves between computing systems and devices. Signatures, envelopes and metadata are all easily stripped while operating system controls are useless once the file moves to another computer. Multiple copies of what should be the same document may have different metadata and security state, raising questions over whether they are the same file or otherwise which is the ‘correct’ one.

[0008] Embodiments of the present disclosure provide more reliable data integrity verification that is more robust against attempts to tamper, and that may be maintained with less use of memory and reduced consumption of processor and network resources.

[0009] According to one aspect of the present disclosure there is provided a method of data storage, the method performed by a processor of a computing device and comprising: receiving data for storage; processing the data to obtain a hash; adding the hash as a further leaf to a first Merkle mountain range (MMR) stored in a local memory accessible to the processor, wherein the first MMR is persisted to a part of a second MMR stored in a first remote allocated portion of a remote memory on a remote computing device, the first remote allocated portion comprising cryptographic information, wherein the cryptographic information comprises a copy of a node of the second MMR; copying the part of the second MMR and the cryptographic information to a first local allocated portion of the local memory to obtain a copy of the part of the second MMR, a copy of the cryptographic information, and a version identifier identifying the copy of the part of the second MMR and the copy of the cryptographic information; determining from the first MMR a total storage size of the further leaf and any nodes required to be added with the further leaf to maintain the first MMR as a valid MMR; determining whether the first remote allocated portion has sufficient storage capacity for the total storage size; and if the first remote allocated portion has sufficient storage capacity: appending the further leaf and any said nodes to the copy of the part of the second MMR to generate a modified second MMR; writing the modified second MMR and the copy of the cryptographic information back to the first remote allocated portion; and transmitting the version identifier to the remote computing device; or if the first remote allocated portion has insufficient storage capacity: allocating a second local allocated portion of the local memory, wherein the second local allocated portion comprises a copy of the cryptographic information; storing further cryptographic information to the second local allocated portion, the further cryptographic information comprising a copy of a node of the first MMR; storing the further leaf to the second local allocated portion; writing the further leaf, the cryptographic information, and the further cryptographic information to a second remote allocated portion of the remote memory; and transmitting the version identifier to the remote computing device.

[0010] This method has the benefit that data integrity may be preserved without the need for a resource-intensive consensus mechanism (as described herein, the need for this may be avoided due to the version identifier). This saves processor and network resources during data maintenance. Additionally, after the further leaf is stored to the second remote allocated portion, the first remote allocated portion is no longer necessary for preserving data integrity (as described below) and so may be deleted, allowing storage space to be conserved long term. Forks are prevented through the risk of version control. Additionally, the use of the MMR data structure, the storage on a remote memory, and the use of version identifiers together ensure that data integrity may be preserved even in a distributed system where multiple unknown parties may be copying and modifying the stored data in unknown ways. This method has the further benefit that data integrity may be preserved without the need for a resource-intensive consensus, lock coordination, or leadership election mechanisms (as described herein, the need for these may be avoided due to the version identifier).

[0011] Adding the hash as a further leaf to the first MMR may comprise adding further nodes to the first MMR in order to maintain the first MMR as a valid MMR.

[0012] Determining whether the first remote allocated portion has sufficient storage capacity for the further leaf and any said nodes may comprise: comparing a remaining storage capacity of the first remote allocated portion to the total storage size; and if the remaining storage capacity is smaller than the total storage size, concluding that the first remote allocated portion has insufficient storage capacity; or otherwise concluding that the first remote allocated portion has sufficient storage capacity.

[0013] Determining whether the first remote allocated portion has sufficient storage capacity for the further leaf and any said nodes may comprise: determining a new leaf count of the first MMR once the hash has been added as a further leaf to the first MMR; comparing the new leaf count to a predetermined maximum leaf count; if the new leaf count exceeds the predetermined maximum leaf count, concluding that the first remote allocated portion of memory has insufficient storage capacity; or otherwise concluding that the first remote allocated portion of memory has sufficient storage capacity.

[0014] The second remote allocated portion may comprise a portion of the remote memory with sufficient capacity to store an MMR of a predetermined maximum leaf count.

[0015] The predetermined maximum leaf count may be a power of 2, and may preferably be 8192.

[0016] The second remote allocated portion may comprise a portion of the remote memory with sufficient capacity to store an MMR of a different maximum leaf count to a previous portion.

[0017] A storage capacity of the second remote allocated portion may be equal to a storage capacity of the first remote allocated portion. The data may comprise any or all of raw bytes, a data file, a JSON payload, a CBOR file, information encoded according to ASN.1 , a reference to a file, or a string.

[0018] The remote memory may comprise any or all of file storage, blob storage, a database, object storage, cloud vendor block storage, or a container.

[0019] The remote memory may be publically accessible.

[0020] This ensures that a verifiable record of the integrity of all the stored documents may be kept publically accessible and generally usable.

[0021] Processing the data to obtain a hash may comprise: generating a unique identifier; processing the data and the unique identifier to obtain modified data; and hashing the modified data.

[0022] The data may comprise a file, and processing the data and the unique identifier to obtain modified data may comprise: hashing the file; processing the hashed file to obtain a JSON payload; encrypting the JSON payload; hashing the encrypted JSON payload; and concatenating the hashed JSON payload with the unique identifier to obtain the modified data.

[0023] The data comprises a JSON payload, and wherein processing the data and the unique identifier to obtain modified data comprises: encrypting the JSON payload; hashing the encrypted JSON payload; and concatenating the hashed JSON payload with the unique identifier to obtain the modified data.

[0024] The unique identifier may be a snowflake ID.

[0025] Writing the modified second MMR and the copy of the cryptographic information back to the first remote allocated portion may comprise overwriting the part of the second MMR and the cryptographic information as previously stored in the first remote allocated portion.

[0026] The cryptographic information may comprise a cryptographic asynchronous accumulator. A cryptographic asynchronous accumulator is a highly compact means of preserving the verifiability of the integrity of previously-stored data. This reduces the memory that is required to preserve a data trail for authentication.

[0027] According to a further aspect of the present disclosure there is provided a computer program product comprising instructions that, when executed by a processor of a computing device, cause the processor to execute the method described above.

[0028] According to a further aspect of the present disclosure there is provided a computing device comprising a processor and a local memory accessible to the processor, the processor configured to: receive data for storage; process the data to obtain a hash; add the hash as a further leaf to a first Merkle mountain range (MMR) stored in the local memory, wherein the first MMR is persisted to a part of a second MMR stored in a first remote allocated portion of a remote memory on a remote computing device, the first remote allocated portion comprising cryptographic information, wherein the cryptographic information comprises a copy of a node of the second MMR; copy the part of the second MMR and the cryptographic information to a first local allocated portion of the local memory to obtain a copy of the part of the second MMR, a copy of the cryptographic information, and a version identifier identifying the copy of the part of the second MMR and the copy of the cryptographic information; determine from the first MMR a total storage size of the further leaf and any nodes required to be added with the further leaf to maintain the first MMR as a valid MMR; determine whether the first remote allocated portion has sufficient storage capacity for the total storage size; and if the first remote allocated portion has sufficient storage capacity: append the further leaf and any said nodes to the copy of the part of the second MMR to generate a modified second MMR; write the modified second MMR and the copy of the cryptographic information back to the first remote allocated portion; and transmit the version identifier to the remote computing device; or if the first remote allocated portion has insufficient storage capacity: allocate a second local allocated portion of the local memory, wherein the second local allocated portion comprises a copy of the cryptographic information; store further cryptographic information to the second local allocated portion, the further cryptographic information comprising a copy of a node of the first MMR; store the further leaf to the second local allocated portion; write the further leaf, the cryptographic information, and the further cryptographic information to a second remote allocated portion of the remote memory; and transmit the version identifier to the remote computing device.

[0029] This device has the same advantages as the method described above.

[0030] It will be appreciated that the functionality of the devices we describe may be divided across several modules. Alternatively, the functionality may be provided in a single module or a processor. The or each processor may be implemented in any known suitable hardware such as a microprocessor, a Digital Signal Processing (DSP) chip, an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Arrays (FPGAs), etc. The, or each processor may include one or more processing cores with each core configured to perform independently. The, or each processor may have connectivity to a bus to execute instructions and process information stored in, for example, a memory.

[0031] The invention further provides processor control code to implement the above-described systems and methods, for example on a general purpose computer system or on a digital signal processor (DSP). The invention also provides a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non- transitory data carrier - such as a disk, microprocessor, CD- or DVD-ROM, programmed memory such as read-only memory (Firmware), or on a data carrier such as an optical or electrical signal carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (Firmware). Code (and / or data) to implement embodiments of the invention may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog™ or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate such code and / or data may be distributed between a plurality of coupled components in communication with one another. The invention may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system. These and other aspects will be apparent from the embodiments described in the following. The scope of the present disclosure is not intended to be limited by this summary nor to implementations that necessarily solve any or all of the disadvantages noted.

[0032] BRIEF DESCRIPTION OF THE DRAWINGS

[0033] For a better understanding of the present disclosure and to show how embodiments may be put into effect, reference is made to the accompanying drawings in which:

[0034] • Fig. 1 illustrates the construction of a Merkle mountain range (MMR);

[0035] • Fig. 2 shows a communication network according to an embodiment;

[0036] • Fig. 3 illustrates a cryptographic accumulator according to an embodiment;

[0037] • Figs. 4A, 4B, and 4C illustrate a method of data storage according to an embodiment; and

[0038] • Figs. 5A, 5B, 5C, 5D, 5E, and 5F illustrate the effects of various steps of the method shown in figs. 4A, 4B, and 4C.

[0039] DETAILED DESCRIPTION

[0040] Embodiments will now be described by way of example only.

[0041] Methods disclosed herein involve the use of a data structure called a Merkle mountain range (MMR), which is a type of Merkle tree. Fig. 1 illustrates the construction of an MMR.

[0042] An MMR is constructed by adding leaves 102, 104, 108, 110, 116, 118, 122. Each leaf may, for example, be a cryptographic hash. In the context of data integrity, each leaf may be a cryptographic hash of data that has been provided for secure storage in the MMR.

[0043] The smallest possible MMR is a single leaf 102 (for example, a hash of a single data file). This may be considered to be an MMR of height zero.

[0044] When a second leaf 104 is added, the two leaves are processed to produce a node 106. For example, the node 106 may be produced by concatenating the two leaves 102, 104 and hashing the concatenated result. The result is an MMR of height one comprising the two leaves 102, 104 and the node 106. Since the node 106 is at the “top” of the MMR as illustrated in Fig. 1 (that is, the node 106 has not yet been used to produce any further nodes), the node 106 may be referred to as a peak node. Since the node 106 is produced from the combination of the leaves 102, 104, the node 106 comprises all cryptographic information present in the leaves 102, 104, and can be used e.g. to verify the integrity of data represented by the leaves 102, 104.

[0045] When a further leaf 108 is added, no nodes are required. The MMR now comprises two distinct sections (which may be referred to as peaks), with one section comprising the leaves 102, 104 and the node 106, and the other section comprising only the leaf 108. These two sections may be viewed as distinct Merkle trees, with the first Merkle tree having height one and the second having height zero. Alternatively, the two sections together may be considered to be a single MMR of height one. The name “mountain range” reflects this view, with the idea of each section being a distinct “mountain” in the range.

[0046] When a further node 110 is added, two new nodes 112 and 114 are required to maintain the MMR as a valid MMR. The first node 112 is produced by combining the leaves 104 and 108 as described above, and the second node 114 is then produced from the two existing nodes 106, 112. The second node 114 becomes the new peak of the MMR, which is now a single Merkle tree of height 2. The second node 114 therefore comprises all the cryptographic information of the complete tree through the process of combining nodes. Nodes that are not peak nodes, such as nodes 106 and 112, may be referred to as interior nodes.

[0047] Further leaves 116, 118, 122 may be added in the same manner, requiring one new node 120 produced by combining leaves 116 and 118 as shown. The resulting MMR has three sections or peaks, with the first having height two, the second having height one, and the third having height 0 (comprising only the leaf 122). This MMR may also be referred to as an MMR of height two.

[0048] It will be appreciated from the above description that a section of an MMR of a given height n may contain a maximum number of leaves 2". A number 2n, with n an integer, is referred to herein as a power of 2. It is therefore generally possible to calculate the height of an MMR from the total number of leaves and vice-versa.

[0049] Fig. 2 illustrates a possible means of summarising the cryptographic information contained in an MMR. The figure shows an MMR 200 of height 2, comprising three sections 202, 204, 206 similar to the sections shown in Fig. 1 . The first section 202 has a peak node 208, the second section 204 has a peak node 210, and the third section 206 comprises only a single leaf 212.

[0050] As noted above, the process of combining leaves or nodes to obtain new nodes retains cryptographic information (for example, by concatenating and hashing), such that the peak node of a section 202, 204 retains the cryptographic information represented by the leaves and nodes within that section. As a result, in order to verify the integrity of data represented by the leaves of the tree 200, it is sufficient only to have peak nodes 208, 210 and leaf 212.

[0051] In cases where storage is a concern, this allows the integrity of the tree 200 to be ensured by storing only peak nodes 208, 210 and leaf 212. The other leaves and nodes of the tree 200 may be discarded while still allowing the integrity of all the represented data to be ensured. In this context, the peak nodes 208, 210 and leaf 212 may be referred to as an accumulator or summary of the MMR 200. More specifically, the particular method of defining an accumulator shown in Fig. 2 may be referred to as a cryptographic asynchronous accumulator. In general, a cryptographic asynchronous accumulator refers collectively to all the peak nodes and single leaves of an MMR, which may be stored separately to (or instead of) the MMR to allow verification of data integrity while saving storage space.

[0052] Fig. 3 shows a communication network comprising a first device 304 (otherwise referred to herein as a client device 304), a second device 306 (otherwise referred to herein as a local device 306), and a third device 308 (otherwise referred to herein as a remote device 308). The three devices are able to communicate via a network 310.

[0053] The client device 304 may be associated with a user 302. The local device 306 comprises a processor 312 that is configured to access a memory 316 (otherwise referred to herein as a local memory 316), and a network interface 314 that allows the local device 306 to access the network 310. The local memory 316 stored an MMR 318 referred to herein as the first MMR 318. The first MMR 318 is persisted to (i.e. a copy of) the part of the second MMR 326 described below.

[0054] While Fig. 3 shows the user 302 operating the client device 304 and the system 300 comprising three devices, in some embodiments the client device 304 may not be present and the user 302 may operate the local device 306.

[0055] The remote device 308 comprises a processor 320 that is configured to access a memory 324 (otherwise referred to herein as a remote memory 324), and a network interface 322 that allows the remote device 308 to access the network 310.

[0056] The local memory 316 and the remote memory 324 may use any known memory technology, such as file storage, blob storage, a database, object storage, cloud vendor block storage, or a container.

[0057] The remote memory 324 may be publically accessible.

[0058] The remote memory 324 may be at least partially divided into portions 330, 332. These portions may merely be a means of labelling memory addresses within the remote memory 324. While two portions 330, 332 are shown in Fig. 3, there may be only one portion or there may be more than two portions. Elsewhere herein, portion 330 may be referred to as the first remote allocated portion, and portion 332 may be referred to as the second remote allocated portion.

[0059] Throughout this disclosure, the term “portion” or “allocated portion” is used as a label for referring to parts of the memory of a computing device. This term is not necessarily intended to indicate any rigid predetermined physical or administrative allocation of memory (though it does not exclude such schemes). Rather, in any storage system where particular data is stored in a particular part of a memory (e.g. at particular memory addresses), that particular part is referred to as a “portion” herein, merely to establish a clear and concise means of reference. Each portion 330, 332 may be of an appropriate size to store a Merkle mountain range of a predetermined maximum number of leaves, otherwise referred to herein as leaf count. In the context of this disclosure, this means that the portion 330, 332 has sufficient storage space to store an MMR of the predetermined leaf count X. Further MMRs, or further parts of the same MMR, may then be stored in further portions. For example, if the predetermined maximum leaf count were set to 8192, each portion 330, 332 may then have sufficient storage capacity to store an MMR (or a part of an MMR) comprising 8192 leaves, together with the accompanying nodes necessary to construct a valid MMR.

[0060] As an illustrative example, this may either mean that each portion 330, 332 is an allocated section of the memory having storage capacity sufficient to store an MMR of the predetermined leaf count, or it may simply mean that, once an MMR stored in the respective portion 330, 332 exceeds the predetermined leaf count, any further leaves will be stored elsewhere regardless of the actual capacity of any particular portion of the memory, those further leaves forming a continuation of the MMR that is referred to herein as a further part of the MMR.

[0061] As a further illustrative example, it is noted that Fig. 2 shows an MMR comprising 7 leaves. If the predetermined maximum leaf count were set to 7, each portion 330, 332 may have sufficient storage capacity to store the arrangement shown in Fig. 2, and would then allow no further leaves to be added.

[0062] It is noted that, due to the relationship between leaf count and MMR height described above, defining the maximum size of an MMR in terms of leaf count can be converted to a maximum size of the MMR in terms of a maximum height of the MMR. Defining a maximum leaf count and defining a maximum MMR height are therefore broadly interchangeable.

[0063] In embodiments, it may be that not all portions 330, 332 are configured to store an MMR of the same predetermined leaf count. Some portions 330, 332 may be configured for a different leaf count from some other portions.

[0064] A first portion 330 of the remote memory 324 (otherwise referred to herein as a first remote allocated portion 330) stores a most recent part 326 of a larger MMR referred to herein as the second MMR. This part 326 is referred to herein as the part of the second MMR 326. As noted above, the part of the second MMR 326 stored in the first remote allocated portion 330 is initially identical to the first MMR 318 (that is, the two MMRs are identical until the below-described steps are carried out to modify them).

[0065] The reason why the part of the second MMR 326 is only the most recent part of the second MMR is as follows. It is envisioned that, if leaves continue to be stored to the remote memory 324 over time, then rather than storing all these leaves in a single MMR, they can be divided into fixed- or variable-size parts, each part structured as an MMR of a particular leaf count. For example, each part may be an MMR of leaf count 7 as shown in Fig. 2. In that case, once a part of leaf count 7 of the kind shown in Fig. 2 is complete, the next leaf to be added may become the first leaf of a new part rather than increasing the leaf count of the existing part. Each part may then be stored in a respective memory portion of appropriate storage capacity. This approach is described in more detail throughout the rest of the disclosure.

[0066] While a maximum leaf count of seven is used here for ease of illustration, in embodiments other values may be used. For example, the predetermined maximum height may be between a power of 2. The inventors have found that a convenient leaf count for ease of use in many storage systems is 8192.

[0067] The first portion 330 further stores cryptographic information 328 relating to earlier parts of the second MMR (i.e. to parts of the second MMR that precede the part of the second MMR 326). These earlier parts may, for example, be stored in other portions of the remote memory 324 not shown in Fig. 3. For example, the cryptographic information 328 may be a cryptographic asynchronous accumulator of the kind described above with reference to Fig. 3, comprising copies of each peak node and solitary leaf of each earlier part of the second MMR. Alternatively, the cryptographic information 328 may be any other kind of accumulator comprising a copy of at least one node from an earlier section of the second MMR.

[0068] Storing the most recent part 326 of the second MMR together with cryptographic information 328 of this kind allows the integrity of every leaf that has ever been added to the second MMR, including leaves in earlier portions, to be verified without needing to store the entire second MMR (i.e. without every leaf and the accompanying nodes needing to be retained). That is, the leaves in the part 326 stored in the first remote allocated portion 330 can be verified directly, and leaves in earlier portions can be verified from the cryptographic information 328 without the need to consult or retain those earlier portions.

[0069] In embodiments, the part of the second MMR 326 may be the entirety of the second MMR (that is, there may be no previous parts). In that case, the cryptographic information 328 may be blank.

[0070] Figs. 4A, 4B, and 4C show a method 400 of data storage that may be implemented in a communications network such as the network 300 described above with reference to Fig. 3. In particular, the method may be performed by the processor 312 of the local device 306.

[0071] It is noted that, for further clarity, the states of the local memory 316 and remote memory 324 at various stages of the method 400 are illustrated in Fig. 5 and described below.

[0072] Fig. 4A shows the initial steps of the method 400, which are then continued in either Fig. 4B or Fig. 4C.

[0073] At step S402, the method comprises receiving data for storage.

[0074] For example, data may be transmitted by a user 302 from the client device 304 via the network 310 to the processor 312 of the local device 306. Alternatively, the data may already be present on the local device 306, for example in the local memory 316, and the processor 312 may be instructed by the user 302 to retrieve the data from the local memory 316.

[0075] In embodiments, the data may comprise raw bytes, a data file, a JSON payload, a CBOR file, information encoded according to ASN.1 , a reference to a file, a string, or any combination of the above.

[0076] At step S404, the method comprises processing the data to obtain a hash. For example, the processor 312 may directly hash the received data to obtain the hash. Alternatively the processor 312 may use any other method that provides enhanced security and / or verifiability to obtain a hash from the received data.

[0077] For example, the processor 312 may process the data together with a unique identifier such as a snowflake ID, which may be generated by the processor 312, before hashing the result of this processing.

[0078] More specifically, in embodiments where the data is a file, the processor 312 may hash the file and produce a JSON payload from the hash. This payload may optionally be encrypted and hashed again. The resulting hash may then be concatenated with a unique identifier as described above, and the result hashed again.

[0079] In other embodiments, the received data may be a JSON payload, in which case the initial steps taken to generate a JSON payload from the received data may not be necessary.

[0080] At step S406, the method comprises adding the hash as a further leaf to a first Merkle mountain range (MMR) stored in a local memory accessible to the processor, wherein the first MMR is persisted to a part of a second MMR stored in a first remote allocated portion of a remote memory on a remote computing device, the first remote allocated portion comprising cryptographic information, wherein the cryptographic information comprises a copy of a node of the second MMR.

[0081] For example, the processor 312 of the local device 306 may add the hash to the first MMR 318 stored in the local memory 316.

[0082] Adding the hash to the first MMR 318 may be accomplished according to the procedures described above with reference to Fig. 1. For example, the hash may be added as a new peak of the MMR 318, and / or may be added to an existing peak requiring the addition of one or more nodes to maintain the first MMR 318 as a valid MMR.

[0083] At step S408, the method comprises copying the part of the second MMR and the cryptographic information to a first local allocated portion of the local memory to obtain a copy of the part of the second MMR, a copy of the cryptographic information, and a version identifier identifying the copy of the part of the second MMR and the copy of the cryptographic information.

[0084] For example, the processor 312 may copy the part of the second MMR 326 and the cryptographic information 328 from the remote memory 324 to the local memory 316 via the network 310 to obtain verifiable copies of the part of the second MMR 326 and the cryptographic information 328.

[0085] It is generally envisioned that the version identifier will be provided to the processor 312 of the local device 306 by the processor 320 of the remote device 308 via the network 310. The version identifier may indicate a version of the part of the second MMR 326 and the cryptographic information 328 that was copied.

[0086] For example, the processor 320 of the remote device 308 may maintain a version identifier relating to some or all of the contents of the remote memory 324 that may be updated each time said contents are changed. For example, the processor 320 may initially record that the part of the second MMR 326 and cryptographic information 328 as initially stored in the remote memory 324 are version 1. When either of these elements is modified, the processor 320 may update this to version 2, then version 3 and so on. When step S408 is performed and the processor 312 copies the part of the second MMR 326 and the cryptographic information 328, this version identifier is provided to the processor 312 as a record of which version was copied (e.g. version 3). Any known system of version control may be used to implement this feature.

[0087] At step S410, the method comprises determining from the first MMR a total storage size of the further leaf and any nodes required to be added with the further leaf to maintain the first MMR as a valid MMR.

[0088] For example, the processor 312 may determine a storage size of the new hash together with any further nodes added to the first MMR 318. As described above with reference to Fig. 1 , adding a leaf to an MMR may or may not require additional nodes to be added depending on the position of the leaf within the MMR.

[0089] At step S412, the method comprises determining whether the first remote allocated portion has sufficient storage capacity for the total storage size. For example, the processor 312 may determine whether the portion 330 of the remote memory 324 has sufficient storage capacity to store the hash and any nodes that were added to the first MMR 318 at step S408, having storage size determined at step S410.

[0090] For example, the processor 312 may have a record of a predetermined leaf count allocated for the portion 330 (that is, the maximum leaf count of an MMR or part of an MMR to be stored in the portion 330). In this case, the processor 312 may compare the leaf count of the first MMR 318 after modification to this predetermined leaf count, and determine whether the predetermined leaf count has been exceeded by the first MMR, in which case the portion 330 will be determined not to have sufficient capacity.

[0091] There are two possible continuations of the method 400 depending on whether the portion 330 is determined at step S412 to have sufficient storage capacity or not.

[0092] Fig. 4B shows the continuation of the method 400 in the case where the first remote allocated portion has sufficient storage capacity for the total storage size of the further leaf and any new nodes.

[0093] At step S414, the method comprises appending the further leaf and any said nodes to the copy of the part of the second MMR to generate a modified second MMR.

[0094] For example, the processor 312 of the local device 306 may add the hash generated from the received data as a new leaf to the copy of the part of the second MMR 326 stored in the local memory 316. The hash may be added according to the procedures described above with regard to Fig. 1. The resulting modified form of the part of the second MMR 326 is referred to herein as the modified second MMR for brevity, even though it may be a modification of only a most recent part of the second MMR.

[0095] At step S416, the method comprises writing the modified second MMR and the copy of the cryptographic information back to the first remote allocated portion.

[0096] For example, the processor 312 may write the modified second MMR 326, together with the unmodified cryptographic information 328, back to the portion 330. This may be done by requesting permission from the processor 320 of the remote device 308 to modify the remote memory 324, via the network 310.

[0097] In embodiments, this may overwrite the existing part of the second MMR 326 and cryptographic information 328. This may result in the contents of the portion 330 being the same as before the write operation, except that the new leaf and any necessary nodes are added to the part of the second MMR 326.

[0098] It is noted that this overwriting does not remove any information from the remote memory 324, since nothing of the part of the second MMR 326 or the cryptographic information 328 is deleted in the handling of this information by the processor 312 of the local device 306. Rather, the only modification is that a leaf (and possibly one or more nodes) have been added to the part of the second MMR 326. Therefore, even if the write operation comprises overwriting, the net effect is as though the new leaf has been appended to the part of the second MMR 326 stored in the remote memory 324.

[0099] Additionally, this overwriting does not require any restructuring or reconfiguration of the existing nodes of the part of the second MMR 326. Rather, the leaf and any additional nodes are merely appended without changing any of the existing structure. This avoids the computational load of reconfiguring a large data structure when adding new information.

[0100] At step S418, the method comprises transmitting the version identifier to the remote computing device.

[0101] For example, the processor 312 of the local device 316 may transmit the version identifier that was received at step S408 back to the processor 320 of the remote device 308.

[0102] The purpose of transmitting back the version identifier is to enable the processor 320 of the remote device 308 to determine whether to grant the request of the processor 312 of the local device 306 to modify the remote memory 324. In particular, it is generally envisioned that the processor 320 of the remote device 308 may only permit the processor 312 of the local device 306 to modify the remote memory 324 by performing the write operation S416 if the version identifier provided at step S418 matches a current version of the part of the second MMR 326 and the cryptographic information 328 stored in the remote memory 324.

[0103] That is to say, the processor 320 of the remote device 308 may only permit the write operation if the contents of the remote memory 324 has not been otherwise modified in between copying the part of the second MMR 326 and the cryptographic information 328 at step S408, and attempting to write back the modified second MMR and the unmodified cryptographic information 328 at step S416.

[0104] This ensures that if, for example, another computing device is also attempting to execute the method 400, or to perform any other operation that involves modifying the contents of the remote memory 324, there is no risk of clashing modifications. For example, if another device had modified the part of the second MMR 326 in the time between the processor 312 executed step S408 and step S416, there would be a risk of making an incompatible modification of the part of the second MMR 326 that may result in corrupted or non-verifiable data when the write operation S416 is carried out. Providing the version identifier for comparison avoids this risk.

[0105] It is noted that, in embodiments where several devices are attempting to modify the remote memory 324 simultaneously, the result may be thought of as a “race” between the several devices. Once the remote memory 324 has been modified by one of the devices, any subsequent attempts at modification by another device will be unsuccessful if still based on the same version identifier. In that case, the other devices will need to copy the contents of the remote memory 324 again in order to obtain an updated version identifier, and then try again to modify the contents of the remote memory 324. This paradigm may be referred to as optimistic concurrency control.

[0106] It is further noted that this procedure avoids the need for any consensus mechanism. That is, if multiple devices are attempting to modify the remote memory 324, the use of version identifiers as described above allows them to do so in ignorance of each other and without needing to reach a consensus as to how the data should be modified. This allows for far greater efficiency and far less usage of processor and network resources when modifying the contents of the remote memory 324. Depending on the operation of the remote memory 324, step S418 may be performed before step S416.

[0107] Fig. 4C shows the continuation of the method 400 in the case where the first remote allocated portion does not have sufficient storage capacity (i.e. has insufficient storage capacity) for the total storage size of the further leaf and any new nodes.

[0108] At step S420, the method comprises allocating a second local allocated portion of the local memory, wherein the second local allocated portion comprises a copy of the cryptographic information.

[0109] For example, the processor 312 of the local device 306 may allocate a new portion of the local memory 316 and move (or copy) the copy of the cryptographic information 328 that was obtained at step S408 into this new portion.

[0110] Allocating the new portion of the local memory 316 may simply comprise assigning one or more memory addresses in the local memory 316 for the storage of particular data.

[0111] At step S422, the method comprises storing further cryptographic information to the second local allocated portion, the further cryptographic information comprising a copy of a node of the first MMR.

[0112] For example, the processor 312 may store further cryptographic information to the new portion of the local memory 316. In cases where the cryptographic information 328 is an asynchronous cryptographic accumulator as described above with reference to Fig. 2, the further cryptographic information may be a further asynchronous cryptographic accumulator collecting peak nodes and solitary leaves of any parts of the copy of the part of the second MMR 326.

[0113] For clarity, it is noted that, as described above, the part of the second MMR 326 may only be the most recent part of the second MMR, with the cryptographic information 328 representing an accumulator of the earlier parts of the second MMR. In this case, the further cryptographic information may be an extension of this accumulator to the most recent part represented by the part of the second MMR 326. That is to say, the further cryptographic information may comprise copies of the peak nodes and any solitary leaves of the part of the second MMR 326. This may be considered to effectively bring the cryptographic asynchronous accumulator (or other type of accumulator) up to date with the new part represented by the part of the second MMR 326.

[0114] It is noted that, since the first MMR 318 is persisted to the part of the second MMR 326, it may equivalently be said that the further cryptographic information comprises a copy of a node of the first MMR 318.

[0115] At step S424, the method comprises storing the further leaf to the second local allocated portion.

[0116] For example, the processor 312 may store the further leaf to the new portion of the local memory 316.

[0117] In this way, the further leaf becomes the first leaf of a new part of the MMR which is to be stored in the newly allocated portion of the local memory 306 along with the cryptographic information and the further cryptographic information.

[0118] At step S426, the method comprises writing the further leaf, the cryptographic information, and the further cryptographic information to a second remote allocated portion of the remote memory.

[0119] For example, the processor 312 may write the further leaf, the copy of the cryptographic information 328, and the further cryptographic information to the empty portion 332 of the remote memory 324.

[0120] Additionally or alternatively, the processor 312 of the local device 306 may contact the processor 320 of the remote device 308 to request permission to allocate a new portion of the remote memory 324 (for example, by assigning particular memory addresses within the remote memory 324 for the storage of particular data). The processor 320 of the remote device 308 may grant permission to allocate the new portion, and the processor 312 of the local device 306 may write the further leaf, the copy of the cryptographic information 328, and the further cryptographic information to the newly allocated portion of the remote memory 324. In embodiments, if two processors (such as the processor 312 of the local device 306 and another processor on a further device) attempt to allocate new portions of the remote memory 324 at the same time, the processor 320 of the remote device 308 may decide which of the two processors is granted permission to allocate the new portion. For example, this decision may be implemented using “fail if exists” semantics. The processor that is not granted permission will have its attempt to allocate a portion fail, and may retry at a later time (e.g. after reading the modified contents of the remote memory 324).

[0121] It is noted that, once this write operation is complete, the contents of the portion 330 no longer need be retained if the only desired function of the remote memory 324 is data integrity verification. This is because the cryptographic information 328 is sufficient to verify the integrity of all previous parts of the MMR, the further cryptographic information may be used to verify the integrity of the part of the second MMR 326, and the new leaf is stored in the portion 332. Therefore, if desirable in order to save memory, the portion 330 may at this point be deleted without comprising the audit trail of data verification, and only the portion 332 need be retained.

[0122] At step S428, the method comprises transmitting the version identifier to the remote computing device.

[0123] For example, the processor 312 of the local device 316 may transmit the version identifier that was received at step S408 back to the processor 320 of the remote device 308.

[0124] This step is performed for the same reason as step S418 described above. That is to say, the processor 320 of the remote device 308 may only permit the writing step S426 if the version identifier received at step S408 matches a current version of the contents of the remote memory 324 (that is, if the remote memory 324 has not been modified in the time between the processor 312 executing step S408 and step S426). The comments made above with regard to step S418 apply equally to step S428.

[0125] Figs. 5A, 5B, 5C, 5D, 5E, and 5F show what is stored in each of the local memory 316 and the remote memory 324 at various stages of the method 400 described above. Fig. 5A shows the local memory 316 and the remote memory 324 immediately after step S406 of the method 400.

[0126] At this stage, the new hash has been added as a leaf to the first MMR 318. The first MMR 318 with this new leaf included is represented as a modified first MMR 502.

[0127] Fig. 5B shows the local memory 316 and the remote memory 324 immediately after step S408 of the method 400.

[0128] At this stage, the part of the second MMR 326 and the cryptographic information 328 have been copied to the local memory 316, resulting in a copy 506 of the part of the second MMR 326 and a copy 508 of the cryptographic information 328. The copies 506, 508 are stored in a portion 504 of the local memory 316. A version identifier 510 relating to the part of the second MMR 326 and the cryptographic information 328 is also provided and may be stored in the local memory 316 as shown.

[0129] Portion 504 is otherwise referred to herein as the first local allocated portion.

[0130] Fig. 5C shows the local memory 316 and the remote memory 324 immediately after step S414 of the method 400.

[0131] At this stage, the copy 506 of the part of the second MMR 326 has been modified with the addition of the new leaf to obtain a modified second MMR 512 stored in the local memory 316.

[0132] Fig. 5D shows the local memory 316 and the remote memory 324 immediately after steps S416 and S418 of the method 400.

[0133] At this stage, the modified second MMR 512 and the copy 508 of the cryptographic information 328 have been written back to the portion 330 of the remote memory 324. In addition, the version identifier 510 has been provided back to the processor 320 of the remote device 308, and may be stored in the remote memory 324.

[0134] Fig. 5E shows the local memory 316 and the remote memory 324 immediately after step S424 of the method 400. At this stage, a new portion 524 of the local memory 316 has been allocated. The new portion contains the new leaf 518, a further copy 520 of the cryptographic information 328, and further cryptographic information 522 comprising a copy of at least one node of the copy 506 of the part of the second MMR 326 (or equivalently, a copy of a node of the first MMR 318). Portion 524 is otherwise referred to herein as the second local allocated portion.

[0135] Fig. 5F shows the local memory 316 and the remote memory 324 immediately after steps S426 and S428 of the method 400.

[0136] At this stage, the new leaf 518, the further copy 520 of the cryptographic information 328, and the further cryptographic information 522 have been written to the portion 322 of the remote memory 324. In addition, the version identifier 510 has been provided back to the processor 320 of the remote device 308, and may be stored in the remote memory 324.

[0137] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

24CLAIMS1 . A method of data storage, the method performed by a processor of a computing device and comprising: receiving data for storage; processing the data to obtain a hash; adding the hash as a further leaf to a first Merkle mountain range (MMR) stored in a local memory accessible to the processor, wherein the first MMR is persisted to a part of a second MMR stored in a first remote allocated portion of a remote memory on a remote computing device, the first remote allocated portion comprising cryptographic information, wherein the cryptographic information comprises a copy of a node of the second MMR; copying the part of the second MMR and the cryptographic information to a first local allocated portion of the local memory to obtain a copy of the part of the second MMR, a copy of the cryptographic information, and a version identifier identifying the copy of the part of the second MMR and the copy of the cryptographic information; determining from the first MMR a total storage size of the further leaf and any nodes required to be added with the further leaf to maintain the first MMR as a valid MMR; determining whether the first remote allocated portion has sufficient storage capacity for the total storage size; and if the first remote allocated portion has sufficient storage capacity: appending the further leaf and any said nodes to the copy of the part of the second MMR to generate a modified second MMR; writing the modified second MMR and the copy of the cryptographic information back to the first remote allocated portion; and transmitting the version identifier to the remote computing device; or if the first remote allocated portion has insufficient storage capacity: allocating a second local allocated portion of the local memory, wherein the second local allocated portion comprises a copy of the cryptographic information; storing further cryptographic information to the second local allocated portion, the further cryptographic information comprising a copy of a node of the first MMR; storing the further leaf to the second local allocated portion;writing the further leaf, the cryptographic information, and the further cryptographic information to a second remote allocated portion of the remote memory; and transmitting the version identifier to the remote computing device.

2. The method of claim 1 , wherein adding the hash as a further leaf to the first MMR comprises adding further nodes to the first MMR in order to maintain the first MMR as a valid MMR.

3. The method of claim 1 or 2, wherein determining whether the first remote allocated portion has sufficient storage capacity for the further leaf and any said nodes comprises: comparing a remaining storage capacity of the first remote allocated portion to the total storage size; and if the remaining storage capacity is smaller than the total storage size, concluding that the first remote allocated portion has insufficient storage capacity; or otherwise concluding that the first remote allocated portion has sufficient storage capacity.

4. The method of claim 1 or 2, wherein determining whether the first remote allocated portion has sufficient storage capacity for the further leaf and any said nodes comprises: determining a new leaf count of the first MMR once the hash has been added as a further leaf to the first MMR; comparing the new leaf count to a predetermined maximum leaf count; if the new leaf count exceeds the predetermined maximum leaf count, concluding that the first remote allocated portion of memory has insufficient storage capacity; or otherwise concluding that the first remote allocated portion of memory has sufficient storage capacity.

5. The method of any preceding claim, wherein the second remote allocated portion comprises a portion of the remote memory with sufficient capacity to store an MMR of a predetermined maximum leaf count.

6. The method of claim 4 or 5, wherein the predetermined maximum leaf count is a power of 2, and is preferably 8192.

7. The method of any preceding claim, wherein the second remote allocated portion comprises a portion of the remote memory with sufficient capacity to store an MMR of a different maximum leaf count to a previous portion.

8. The method of any of claims 1 to 6, wherein a storage capacity of the second remote allocated portion is equal to a storage capacity of the first remote allocated portion.

9. The method of any preceding claim, wherein the data comprises any or all of raw bytes, a data file, a JSON payload, a CBOR file, information encoded according to ASN.1 , a reference to a file, or a string.

10. The method of any preceding claim, wherein the remote memory comprises any or all of file storage, blob storage, a database, object storage, cloud vendor block storage, or a container.

11. The method of any preceding claim, wherein the remote memory is publically accessible.

12. The method of any preceding claim, wherein processing the data to obtain a hash comprises: generating a unique identifier; processing the data and the unique identifier to obtain modified data; and hashing the modified data.

13. The method of claim 12, wherein the data comprises a file, and wherein processing the data and the unique identifier to obtain modified data comprises: hashing the file; processing the hashed file to obtain a JSON payload; encrypting the JSON payload; hashing the encrypted JSON payload; and concatenating the hashed JSON payload with the unique identifier to obtain the modified data.

14. The method of claim 12, wherein the data comprises a JSON payload, and wherein processing the data and the unique identifier to obtain modified data comprises:27 encrypting the JSON payload; hashing the encrypted JSON payload; and concatenating the hashed JSON payload with the unique identifier to obtain the modified data.

15. The method of any of claims 12-14, wherein the unique identifier is a snowflake ID.

16. The method of any preceding claim, wherein writing the modified second MMR and the copy of the cryptographic information back to the first remote allocated portion comprises overwriting the part of the second MMR and the cryptographic information as previously stored in the first remote allocated portion.

17. The method of any preceding claim, wherein the cryptographic information comprises a cryptographic asynchronous accumulator.

18. A computer program product comprising instructions that, when executed by a processor of a computing device, cause the processor to execute the method of any preceding claim.

19. A computing device comprising a processor and a local memory accessible to the processor, the processor configured to: receive data for storage; process the data to obtain a hash; add the hash as a further leaf to a first Merkle mountain range (MMR) stored in the local memory, wherein the first MMR is persisted to a part of a second MMR stored in a first remote allocated portion of a remote memory on a remote computing device, the first remote allocated portion comprising cryptographic information, wherein the cryptographic information comprises a copy of a node of the second MMR; copy the part of the second MMR and the cryptographic information to a first local allocated portion of the local memory to obtain a copy of the part of the second MMR, a copy of the cryptographic information, and a version identifier identifying the copy of the part of the second MMR and the copy of the cryptographic information; determine from the first MMR a total storage size of the further leaf and any nodes required to be added with the further leaf to maintain the first MMR as a valid MMR;28 determine whether the first remote allocated portion has sufficient storage capacity for the total storage size; and if the first remote allocated portion has sufficient storage capacity: append the further leaf and any said nodes to the copy of the part of the second MMR to generate a modified second MMR; write the modified second MMR and the copy of the cryptographic information back to the first remote allocated portion; and transmit the version identifier to the remote computing device; or if the first remote allocated portion has insufficient storage capacity: allocate a second local allocated portion of the local memory, wherein the second local allocated portion comprises a copy of the cryptographic information; store further cryptographic information to the second local allocated portion, the further cryptographic information comprising a copy of a node of the first MMR; store the further leaf to the second local allocated portion; write the further leaf, the cryptographic information, and the further cryptographic information to a second remote allocated portion of the remote memory; and transmit the version identifier to the remote computing device.

Images