A log storage method based on a hierarchical authentication data structure and an electronic device

Abstract

This application relates to the field of data management technology, proposing a log storage method and electronic device based on a hierarchical authentication data structure. The method includes: acquiring the current log entry to be stored; writing the current log entry into the memory component currently receiving log data, where the memory component is a storage component located in the memory layer and possessing an authentication data structure; if the number of log entries stored in the memory component reaches a capacity threshold, then converting the memory component into a corresponding disk component, where the disk component is a storage component located in the disk layer and possessing an authentication data structure. This method avoids the double-layer I / O access amplification problem by directly embedding the authentication data structure into the hierarchical architecture of the log storage engine and utilizing native storage components created separately in the memory and disk layers to store log data.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a log storage method and electronic device based on a hierarchical authentication data structure. Background Technology

[0002] Modern operating systems, databases, network devices, application services, and IoT devices continuously generate a large number of log entries that monotonically increase in timestamp. If these logs are modified, deleted, forged, or rearranged by malicious administrators or intruders, it will directly affect fault diagnosis, security auditing, attack tracing, and compliance forensics. Merkle trees and their variants can verify that log data has not been tampered with through root hash digests and Merkle proofs. However, existing technologies typically store Authenticated Data Structures (ADS) such as Merkle Trees, Merkle Patricia Trees, and Merkle B+ Trees on top of each other in general-purpose key-value stores or ledger databases. This means that a single Merkle Tree traversal becomes multiple key-value lookups, and each key-value lookup may access a multi-level Sorted String Table (SST) file or trigger compaction operations, resulting in a double I / O amplification problem: ADS traversal amplification and storage access amplification. Summary of the Invention

[0003] In view of this, embodiments of this application provide a log storage method and electronic device based on a hierarchical authentication data structure. By directly sinking the authentication data structure to the hierarchical architecture of the log storage engine, and using native storage components created in the memory layer and disk layer respectively to store log data, the problem of double-layer I / O access amplification can be avoided.

[0004] The first aspect of this application provides a log storage method based on a hierarchical authentication data structure, including: Retrieve the current log entry to be stored; Write the current log entry to the memory component that is currently receiving log data. The memory component is a storage component that is in the memory layer and has an authentication data structure. If the number of log entries stored in the memory component reaches the capacity threshold, the memory component will be converted into the corresponding disk component. The disk component is a storage component that is located at the disk layer and has an authentication data structure.

[0005] The technical solution of this application adopts a layered authentication data structure storage architecture, that is, storage components with authentication data structures are deployed in both the memory layer and the disk layer, referred to as memory components and disk components, respectively. After obtaining the current log entry to be stored, the current log entry is first written to the memory component that is currently receiving log data. If the number of log entries stored in the memory component reaches the capacity threshold, the memory component is converted into the corresponding disk component, and the log data stored in the memory component is migrated to the disk component for storage. The above operation abandons the traditional general-purpose key-value store and ledger database. By directly sinking the authentication data structure to the layered architecture of the log storage engine and using native storage components created in the memory layer and disk layer respectively to store log data, the problem of double-layer I / O access amplification can be avoided.

[0006] In one implementation of this application, the memory component maintains a main tree structure and cache nodes; the main tree structure is a structure that maintains ordered leaf nodes and Merkle root digests, used to store log indexes ordered by timestamps and complete Merkle hash paths; the cache nodes are ordered buffer nodes with capacity f, used to cache the latest f written log entries, f > 1; the component root hash of the memory component is determined based on the root hash of the main tree structure and the node hash of the cache nodes; writing the current log entry to the memory component currently receiving log data includes: Write the current log entry to the cache node; If the cache node is not full, update the node hash, and based on the updated node hash, update the component root hash of the memory component; If the cache node is full, the f log entries in the cache node are packaged into at least one new leaf node in chronological order, the new leaf node is added to the rightmost side of the main tree structure, and the internal node hash of the main tree structure is updated from bottom to top to obtain the updated root hash; the cache node is cleared and the node hash is reset, and the component root hash of the memory component is updated based on the updated root hash and the reset node hash.

[0007] In one implementation of this application, converting a memory component into a corresponding disk component includes: Freeze the memory component as an immutable component and create a new memory component to receive subsequent log data; An immutable component is converted into a disk component using a background thread.

[0008] In one implementation of this application, the disk component maintains a value file, a timestamp file, an index file, and a Merkle file; the value file is used to store log content sequentially; the timestamp file is used to store the timestamp and the position offset of the corresponding log content in the value file; the index file is used to build a lookup index for the timestamp file; the Merkle file is used to store a complete Merkle tree laid out hierarchically; converting the memory component into the corresponding disk component includes: A value file is generated based on the contents of each log entry stored in the memory component; A timestamp file is generated based on the storage location of each log entry's content in the value file and the timestamp of each log entry; An index file is generated based on a preset time range positioning logic. Based on the content and timestamps of each log entry, a complete Merkle tree is constructed to obtain the Merkle file. The root hash of the complete Merkle tree is used as the component root hash of the disk component.

[0009] In one implementation of this application, the method further includes: Write the storage log time range and root hash of each memory component, as well as the storage log time range and root hash of each disk component, into the component metadata table; A trusted global digest is generated based on the root hashes of each component stored in the component metadata table.

[0010] In one implementation of this application, a trusted global digest is generated based on the root hashes of each component stored in the component metadata table, including: A global Merkle tree is constructed using the root hashes of each component stored in the component metadata table as leaf nodes, and the root hash of the global Merkle tree is used as a trusted global digest. Alternatively, the root hashes of each component stored in the component metadata table can be concatenated in order to obtain the overall root hash, and the hash value of the overall root hash can be calculated as a trusted global digest.

[0011] In one implementation of this application, the method further includes: Upon receiving a log query request from a client carrying a target time range, the relevant and irrelevant components are determined based on the component metadata table and the target time range. Relevant components refer to all memory and disk components whose log storage time range intersects with the target time range, while irrelevant components refer to all memory and disk components whose log storage time range does not intersect with the target time range. Obtain the component root hash of each unrelated component through the component metadata table; Obtain target log entries within the target time range through each relevant component, and generate local Merkel proofs for each relevant component; The target log entry, the component root hash of each unrelated component, and the local Merkle proof of each related component are returned to the client to instruct the client to reconstruct the component root hash of each related component based on the target log entry and the local Merkle proof of each related component. Based on the reconstructed component root hash of each related component and the component root hash of each unrelated component, the target global digest is calculated, and the data verification result is determined by comparing the target global digest with the trusted global digest.

[0012] In one implementation of this application, generating local Merkel proofs for each relevant component includes: For each related component belonging to the memory component, find the minimum and maximum timestamp log entries within the target time range from that related component; starting from the leaf node storing the minimum timestamp log entry, scan the ordered chain of leaf nodes to the leaf node storing the maximum timestamp log entry to obtain all log entries within the target time range, and collect the hashes of the left sibling nodes of the lower bound path, the hashes of the right sibling nodes of the upper bound path, and the leaf node records of both boundaries to form the local Merkle proof of that related component.

[0013] In another implementation of the embodiments of this application, generating local Merkel proofs for each relevant component includes: For each related component belonging to the disk component, the logical position range corresponding to the target time range is determined from the timestamp file of the related component according to the time range positioning logic set in the index file of the related component; all log entries within the target time range are obtained from the value file of the related component according to the logical position range; and a local Merkle proof of the related component is generated based on the hash value of the leaf node corresponding to the logical position range in the Merkle file of the related component.

[0014] A second aspect of this application provides a log storage device based on a hierarchical authentication data structure, comprising: The log entry retrieval module is used to retrieve the current log entry to be stored. The memory component writing module is used to write the current log entry to the memory component that is currently receiving log data. The memory component is a storage component that is in the memory layer and has an authentication data structure. The disk component conversion module is used to convert the memory component into the corresponding disk component if the number of log entries stored in the memory component reaches the capacity threshold. The disk component is a storage component that is located at the disk layer and has an authentication data structure.

[0015] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the log storage method based on a hierarchical authentication data structure as provided in the first aspect of this application.

[0016] A fourth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute a log storage method based on a hierarchical authentication data structure as provided in the first aspect of this application.

[0017] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the log storage method based on a hierarchical authentication data structure provided in the first aspect of this application.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a verifiable log storage architecture based on a hierarchical Merkle Tree provided in an embodiment of this application; Figure 2 This is an operation flowchart of a log storage method based on a hierarchical authentication data structure provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the operating principle of a BMB+Tree memory component provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the operating principle of an MLT disk component provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the operational principle of a local Merkel proof for generating memory components provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the operational principle of generating a local Merkel proof for a disk component, as provided in an embodiment of this application. Figure 7 This is a schematic diagram of a log storage device based on a hierarchical authentication data structure provided in an embodiment of this application; Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0021] Existing log systems typically assume the storage server is trusted, preventing auditing clients from independently verifying the authenticity, completeness, and order of returned logs without trusting the server. Storage servers often employ centralized ledger databases or verifiable databases, such as QLDB, SQL Ledger, and LedgerDB. These systems typically use ADS (Advanced Data Storage) methods like Merkle Tree, Merkle Patricia Tree, and Merkle B+ Tree to generate root hashes and proofs. The basic operation involves using log or transaction data as leaf nodes, internal nodes storing child node hashes, and the root hash as a trusted digest. During a query, the system returns a Merkle Proof composed of the result and the hashes of adjacent sibling nodes. The client then recalculates the root hash and compares it with the trusted digest. However, existing ledger databases often store Merkle Tree nodes as key-value items in general-purpose key-value stores or ledger databases like LevelDB and RocksDB. This results in a single Merkle Tree traversal becoming multiple key-value lookups, each of which may access multiple layers of SST files or trigger compaction operations. This demonstrates a double I / O amplification problem: ADS traversal amplification and storage access amplification.

[0022] To address the aforementioned technical issues, this application provides a log storage method and electronic device based on a hierarchical authentication data structure. By directly embedding the authentication data structure into the hierarchical architecture of the log storage engine and utilizing native storage components created in the memory and disk layers respectively to store log data, the problem of double-layer I / O access amplification can be avoided.

[0023] Please see Figure 1 This is a schematic diagram of a verifiable log storage architecture based on a hierarchical Merkle Tree, provided in an embodiment of this application. Figure 1The log storage architecture shown integrates the Merkle Tree directly into the log storage engine, rather than using it as a separate index overlaid on a general-purpose key-value store or ledger database. The overall system consists of three parts: a log producer, a storage server, and a query client. The log producer continuously generates timestamped log entries (t, v) and writes them to the storage server. Instead of storing Merkle Tree nodes in a general-purpose key-value store / ledger database, the storage server natively maintains the Merkle authentication structure within the storage engine, persisting logs and generating query proofs. The query client holds or obtains a trusted global digest and independently verifies the log data after receiving the query results and Merkle proofs. Specifically, the storage server deploys a memory component Buffered Merkle B+Tree (BMB+Tree) with an authenticated data structure at the memory layer. Each BMB+Tree consists of a Main Merkle B+Tree and a BufferNode, which can reduce the cost of Merkle hash updates through buffering and batch commits. The storage server also deploys a disk component Merkle Learned Table (MLT) with an authenticated data structure at the disk layer. Each MLT can be regarded as an independent and immutable Merkle subtree, which contains a Value File, a Timestamp File, an Index File, and a Merkle File. The storage server also maintains a component metadata table ComponentTable, which is used for query routing and global integrity proof. It is responsible for recording the storage log time range and component root hash of each memory component BMB+Tree and each disk component MLT, and can be combined in the order of components to form a trusted global digest. After the latest log entry output by the log producer is sent to the storage server, it is first written to the buffer node of the memory component BMB+Tree. The buffer node and the root hash of the main Merkle B+Tree together form the component root hash of the current memory component BMB+Tree. The component root hash, also known as the root digest, is essentially the hash of the root node of the Merkle Tree and can serve as a cryptographic commitment for all data in the tree. The memory component BMB+Tree has a capacity threshold. After writing a number of log entries that reach the capacity threshold, the memory component BMB+Tree will be full. At this time, the memory component BMB+Tree is frozen and asynchronously flushed to disk, generating the corresponding disk component MLT.After the query client initiates a range query to the storage server, the storage server returns the log query results, the relevant component local Merkle Proofs, and the root hashes of unrelated components. The client then reconstructs the root hashes of each component in parallel based on these, and combines them to form a target global digest. This target global digest is then compared with the trusted global digest to complete the log data verification and determine whether the log results have been tampered with, forged, or omitted. Figure 1 For more specific technical implementation details of the architecture shown, please refer to the various method implementations described below.

[0024] It should be understood that the implementing entity of the various method embodiments proposed in this application may be... Figure 1 The storage server in the architecture shown can essentially be various types of electronic devices, such as mobile phones, tablets, desktop computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0025] Please see Figure 2 This application illustrates a log storage method based on a hierarchical authentication data structure, comprising: 201. Retrieve the current log entry to be stored; The log producer continuously sends the latest log entries to the storage server. Each time the storage server receives a new log entry, it treats it as the current log entry to be stored. Each log entry consists of a log identifier, a timestamp t, and log content v. The log identifier is a unique identifier used to distinguish different log entries, such as a log number or log name.

[0026] 202. Write the current log entry to the memory component that is currently receiving log data. The memory component is a storage component that is in the memory layer and has an authentication data structure. After receiving the current log entry, the storage server writes it to the memory component that receives the log data for storage. The memory component is a storage component that is located in the memory layer and has an authentication data structure.

[0027] In one implementation of this application, the memory component maintains a main tree structure and cache nodes; the main tree structure is a structure that maintains ordered leaf nodes and Merkle root digests, used to store log indexes ordered by timestamps and complete Merkle hash paths; the cache nodes are ordered buffer nodes with capacity f, used to cache the latest f written log entries, f > 1; the component root hash of the memory component is determined based on the root hash of the main tree structure and the node hash of the cache nodes; writing the current log entry to the memory component currently receiving log data includes: (1) Write the current log entry to the cache node; (2) If the cache node is not full, update the node hash and update the component root hash of the memory component based on the updated node hash; (3) If the cache node is full, pack the f log entries in the cache node into at least one new leaf node in chronological order, add the new leaf node to the rightmost side of the main tree structure, and update the internal node hash of the main tree structure from bottom to top to obtain the updated root hash; clear the cache node and reset the node hash, and update the component root hash of the memory component based on the updated root hash and the reset node hash.

[0028] This application proposes a memory component called BMB+Tree, a Merkle B+Tree variant optimized for log append-only writes. It consists of a Main Tree and Buffer Nodes. The Merkle B+Tree combines the ordered indexing capabilities of a B+Tree with the hash authentication capabilities of a Merkle Tree. The Main Tree maintains ordered leaf nodes and Merkle root digests, storing timestamp-ordered log indexes and complete Merkle hash paths. Besides using a Merkle B+Tree, the Main Tree can also employ a Merkle B Tree, a Merkle skip list, or a Merkle LSM memory layer adapted for append-only writes, but must retain the ability to buffer batch commits and generate Merkle Proofs. A Buffer Node is an ordered buffer node with a capacity of f, used to cache the latest f written log entries. f can typically be adjusted based on CPU cache, memory budget, and write throughput requirements; for example, f=4. The root hash of the memory component BMB+Tree is determined jointly by the root hash of the Main Tree and the node hash of the Buffer Node, and serves as the root digest of the BMB+Tree. Each BMB+Tree has a total capacity of B and a fan-out of f. This means the maximum number of log entries that can be stored jointly by the Main Tree and Buffer Nodes is B, and the maximum number of log entries that can be stored individually by a Buffer Node is f. When a Buffer Node is full, its stored f log entries are batch-committed to the rightmost path of the Main Tree. The Main Tree structure is a tree structure similar to a Merkle B+Tree, constructed by incrementing timestamps. Its leaf nodes store log entries or references to log entries, intermediate nodes store separator keys and child node hashes, and the root node hash represents the cryptographic commitment to the currently stored content of the Main Tree, i.e., the root hash. Because log entries are strictly appended according to their timestamps, the Main Tree does not need to handle arbitrary insertions and complex rebalancing like a general B+Tree; batch commits only occur on the rightmost path.

[0029] Upon arrival of the current log entry (t, v), the leaf-level hash h = Hash(t||v) is calculated first, and the current log entry is written to the buffer node. "||" represents the concatenation operator. After writing the current log entry, if the number of log entries stored in the buffer node is less than f, it indicates that the buffer node is not full. In this case, the node hash hb is calculated and updated according to the data currently stored in the buffer node. Based on the updated node hash hb, the root hash of the memory component BMB+Tree is updated to Hash(hm||hb), where hm represents the root hash of the main tree structure. It can be seen that this operation does not require updating the root hash hm of the main tree structure, therefore it does not trigger a large-scale hash update of the path from the leaf node to the root node in the main tree structure.

[0030] After writing the current log entry, if the number of log entries stored in the Buffer Node reaches f, it indicates that the Buffer Node is full. The system triggers a delayed batch commit, packaging the f log entries in the Buffer Node into at least one new leaf node in chronological order. These new leaf nodes are added to the rightmost side of the Main Tree structure. Subsequently, the internal node hashes on the rightmost path of the Main Tree structure are updated from bottom to top, resulting in the updated root hash hm. Furthermore, all log entries stored in the Buffer Node are cleared and the node hash hb is reset. Finally, based on the updated root hash hm and the reset node hash hb, the component root hash of the memory component BMB+Tree is updated to Hash(hm||hb). Since the reset node hash hb is null, the updated component root hash is equal to Hash(hm). Although this operation requires updating the root hash hm of the Main Tree structure, an update is triggered only once every f log entries are written. This means that a path node hash update is amortized across f log entries, thus reducing the amortized hash update cost per log entry from O(log n) in the traditional Merkle Tree to O(log n). ), which is approximately a constant when B and f are fixed.

[0031] As an example, Figure 3 This is a schematic diagram illustrating the operating principle of a BMB+Tree memory component provided in an embodiment of this application. Figure 3In the process, the capacity of the buffer node is f=4. It originally stored 3 log entries 81, 82 and 83. After writing the current log entry 86, the buffer node is full. The system then performs a commit buffer node operation, which commits all log entries 81, 82, 83 and 86 stored in the buffer node as a batch to the rightmost path of the main tree structure, forming a new leaf node hn9, and creating or updating the parent node hn10. Finally, from nodes hn9 and hn10, node split and node hash update operations are performed upwards along the rightmost path until the root hash hm of the main tree structure is updated. The hash hm is then bound together with the reset node hash hb of the buffer node to the component root hash Root Node of the memory component BMB+Tree.

[0032] In traditional Merkle Tree, Merkle Patricia Tree, or Merkle B+Tree structures, each log entry written requires updating the entire hash along the path from the leaf node to the root node. The complexity increases with tree height, clearly exhibiting a cascading amplification problem of write hashes. To address this issue, this application proposes the aforementioned BMB+Tree structure. By introducing buffer nodes and delayed batch commits, only the node hash (hb) and component root hash need to be updated when the buffer node is not full. When the buffer node is full, batch commits are made to the rightmost path of the main tree structure, thus amortizing Merkle hash updates across a batch of logs, significantly reducing the cost of Merkle path hash updates. Furthermore, when batch committing all log entries stored in the buffer node to the main tree structure, a rightmost path commit method based on log appending characteristics is used. This avoids complex splits and rebalancing caused by arbitrary insertions and maintains a verifiable Merkle root digest.

[0033] The key similarity between the BMB+Tree structure and the Merkle Tree lies in the fact that the BMB+Tree does not eliminate the root hash commitment and sibling path proof mechanism of the Merkle Tree. Instead, it changes the update granularity and storage hierarchy of the Merkle Tree in log appending scenarios. The Buffer Node itself participates in the root hash calculation, so that the latest log that has not yet been committed to the MainTree is also included in the current trusted digest. The MainTree continues to maintain a tree-like hash structure that can generate MerkleProof.

[0034] 203. If the number of log entries stored in the memory component reaches the capacity threshold, the memory component will be converted into the corresponding disk component. The disk component is a storage component that is located at the disk layer and has an authentication data structure.

[0035] As log volumes grow, the depth and proof path length of traditional single global Merkle Trees continuously increase, making it difficult to meet the requirements of continuous writing, low-latency querying, and rapid verification in log scenarios. To address this issue, the memory component BMB+Tree designed in this embodiment has a capacity threshold B. It is foreseeable that as the storage server continuously receives new log entries, the number of log entries stored in the memory component BMB+Tree will reach the capacity threshold B. At this point, an asynchronous disk flushing operation is triggered, converting the memory component BMB+Tree into a corresponding disk component. The disk component is a storage component at the disk layer with an authentication data structure, thus preventing the depth and proof path length of the BMB+Tree from increasing indefinitely.

[0036] In one implementation of this application, converting a memory component into a corresponding disk component includes: (1) Freeze the memory component as an immutable component and create a new memory component to receive subsequent log data; (2) Convert immutable components into disk components through a background thread.

[0037] When converting the memory component BMB+Tree into its corresponding disk component, a freeze and asynchronous disk flushing operation is employed. First, the memory component BMB+Tree is frozen as an immutable component (immutable BMB+Tree), and a new memory component is immediately created to receive subsequent log data, ensuring normal reception of log data. Then, the immutable component is handed over to a background thread, which converts it into the corresponding disk component. This way, foreground writes do not need to wait for disk I / O. After the corresponding disk component is created, the immutable component can be destroyed to save memory resources. It should be noted that the memory component currently receiving log data can be called the active BMB+Tree. When the active BMB+Tree is full, it is frozen as an immutable BMB+Tree awaiting background conversion, and a new BMB+Tree is immediately created as the active BMB+Tree. Therefore, the system may have one active BMB+Tree currently receiving writes, and several immutable BMB+Trees that are frozen and awaiting background conversion. After a BMB+Tree is converted into a corresponding disk component, the log data stored in the BMB+Tree will be migrated to the disk component for storage. Therefore, the BMB+Tree and the corresponding disk component will cover the same time range and log batches.

[0038] In one implementation of this application, the disk component maintains a value file, a timestamp file, an index file, and a Merkle file; the value file is used to store log content sequentially; the timestamp file is used to store the timestamp and the position offset of the corresponding log content in the value file; the index file is used to build a lookup index for the timestamp file; the Merkle file is used to store a complete Merkle tree laid out hierarchically; converting the memory component into the corresponding disk component includes: (1) Generate a value file based on the content of each log entry stored in the memory component; (2) Generate a timestamp file based on the storage location of the content of each log entry in the value file and the timestamp of each log entry; (3) Generate an index file based on the preset time range positioning logic; (4) Based on the content and timestamp of each log entry, construct a complete Merkle tree to obtain the Merkle file. The root hash of the complete Merkle tree is used as the component root hash of the disk component.

[0039] This application proposes a disk component called MLT. Whenever a memory component BMB+Tree reaches its capacity threshold, the system freezes it as an immutable object and asynchronously flushes it to disk, generating an independent disk component MLT. Through the limited-capacity disk component MLT, the logically growing global Merkle structure can be split into multiple independent, immutable, and manageable substructures. Therefore, each disk component MLT is equivalent to an independent subtree of the global log Merkle Tree. Each disk component MLT has a Value File, a Timestamp File, an Index File, and a Merkle File. The Value File stores log content sequentially. The Timestamp File stores the timestamp and the position offset of the corresponding log content within the Value File. The Index File builds a lookup index for the Timestamp File to quickly locate the time range. The Merkle File stores a complete Merkle tree with a hierarchical layout. This complete Merkle tree is built based on all log records contained in the corresponding BMB+Tree and can adopt a structure such as a complete binary tree, a k-way Merkle Tree, or a hierarchical hash tree with page aggregation. Its root hash can be used as the component root hash of the disk component MLT, i.e., the root digest. In addition, the Merkle File can also store the hashes of each node at each level of the complete Merkle tree.

[0040] When converting a memory component BMB+Tree to its corresponding disk component MLT, a Value File can be generated based on the content of each log entry stored in the BMB+Tree. This Value File stores the log content `v` sequentially, suitable for appending and sequential reading. A Timestamp File can be generated based on the storage location of each log entry's content within the Value File and the timestamp of each log entry. This Timestamp File stores fixed-length records in timestamp order, with each record including a timestamp `t` and a pointer `offset` indicating the corresponding log content's position within the Value File. The Value File and Timestamp File can be merged or have compression, encryption, or cold / hot tiered storage mechanisms added according to application requirements, but the logical order and content binding relationships required for leaf node hash calculations must remain unchanged. An Index File can be generated based on a preset time range positioning logic. Preferred indexes such as PGM-Index, binary index, sparse index, and B+Tree index are preferred. This allows for predicting logical positions based on timestamp distribution and then correcting scans within a small range, thus quickly locating the query boundary. A complete Merkle tree can be constructed based on the content and timestamps of each log entry, generating a corresponding Merkle file. The logical order of the timestamp file is used as the leaf node order to construct a complete Merkle tree. The hash of the i-th leaf node is Hash(t_i||v_i), where t_i is the timestamp of the log entry stored in the i-th leaf node, and v_i is the content of the log entry stored in the i-th leaf node. The hash of the intermediate node of this complete Merkle tree is Hash(left_child_hash||right_child_hash), which is the hash of the digest of its left and right subtrees stored in the intermediate node. The root hash of this complete Merkle tree is used as the component root hash of the disk component MLT. Merkle files are written hierarchically from leaf to root. Therefore, the offsets of the parent and sibling nodes of any node can be calculated arithmetically from the node's layer number and position within the layer. There is no need to search for tree nodes through key-value queries. When verifying, the client only needs to obtain the query results and a small number of sibling node hashes to reconstruct the root digest to determine whether the query results have been tampered with.

[0041] As an example, Figure 4 This is a schematic diagram illustrating the operating principle of an MLT disk component provided in an embodiment of this application. For example... Figure 4As shown, a frozen immutable BMB+Tree is converted into a corresponding independent disk component MLT in the background. The physical structure of the disk component MLT consists of four types of files: a Value File stores the log content, a Timestamp File stores the timestamp and content offset, an Index File is used to quickly locate the time range, and a Merkle File stores the complete Merkle Tree laid out in layers. When querying on the client side, the system first uses the Index File and Timestamp File to find the time range boundaries and logical positions, then reads the Value File to return the log content, and simultaneously calculates the sibling node hash addresses directly in the Merkle File based on the layer number and position within the layer, generating a range Merkle Proof. Figure 4 As can be seen, the disk component MLT is not a regular data table, but a verifiable log segment with an independent Merkle subtree and a computable proof path.

[0042] In summary, each disk component MLT can be viewed as an independent, immutable, and self-contained Merkle subtree. Multiple disk component MLTs form a global logical authentication structure based on the time range of the stored logs. This decomposes what would otherwise be an infinitely growing global Merkle Tree into multiple Merkle subtrees of bounded depth. The length of a single proof path is constrained by the capacity B of a single MLT, and does not grow infinitely with the total system log volume n, thus meeting the requirements of continuous writing, low-latency querying, and fast verification in log-based scenarios. Furthermore, leveraging the immutable and timestamped nature of log entries, the disk component MLT employs a combination of sequentially written value files, fixed-timestamp files, and learned index positioning with Merkle file arithmetic addressing, thereby improving the efficiency of log data writing, reading, and proof generation.

[0043] In one implementation of this application, the method further includes: (1) Write the storage log time range and component root hash of each memory component, and the storage log time range and component root hash of each disk component into the component metadata table; (2) Generate a trusted global digest based on the root hash of each component stored in the component metadata table.

[0044] The system can create a Component Table, which records the storage log time range and component root hash of all currently existing BMB+Tree memory components, and the storage log time range and component root hash of all currently existing MLT disk components. This data can be used for query routing and forming global integrity proofs. In addition, the Component Table can also record other metadata such as component number, file location, and component status for all BMB+Tree memory components and all MLT disk components. Based on the component root hashes stored in the Component Table, a trusted global digest can be generated. This trusted global digest can be generated either by concatenating the component root hashes in order, or by building a higher-level global Merkle tree, to support component-level appending and more fine-grained global proofs. The two specific methods for generating this trusted global digest are described below.

[0045] In one implementation of this application, a trusted global digest is generated based on the root hashes of each component stored in the component metadata table, including: A global Merkle tree is constructed using the root hashes of each component stored in the component metadata table as leaf nodes, and the root hash of the global Merkle tree is used as a trusted global digest.

[0046] For the method of generating a global Merkle tree by building an upper layer, the root hashes of each component stored in the component metadata table (Component Table) in time sorting are used as the leaf nodes of the upper layer global Merkle tree. The parent nodes of each leaf node are generated by hashing each other in sequence. If the number of nodes at a certain level is odd, the last node is copied or the node is directly promoted according to the system convention. Finally, a complete global Merkle tree is constructed, and the root hash of the global Merkle tree can be used as the generated trusted global digest.

[0047] In another implementation of this application's embodiments, a trusted global digest is generated based on the root hashes of each component stored in the component metadata table, including: The root hashes of each component stored in the component metadata table are concatenated in order to obtain the overall root hash, and the hash value of the overall root hash is calculated as a trusted global digest.

[0048] For the method of generating a trusted global digest by sequentially concatenating the root hashes of each component, there is no need to construct an additional upper-level global Merkle tree. Instead, the root hashes of each component stored in the component metadata table are first concatenated in order to obtain the overall root hash, and then the hash value of this overall root hash is calculated as the generated trusted global digest. For example, assuming that the root hashes of each component stored in the component metadata table are componentDigest_1, componentDigest_2, ..., componentDigest_k, these k component root hashes are first concatenated in order to obtain the overall root hash, and then the hash value of this overall root hash is calculated to obtain the system-level trusted global digest, that is, calculating Hash(componentDigest_1|| componentDigest_2||...|| componentDigest_k) as the trusted global digest.

[0049] In summary, the overall process of writing log entries can be summarized as follows: (1) The log producer generates a log entry e=(t, v) and sends the log entry e to the storage server; (2) The storage server inserts log entry e into the buffer node of the currently active memory component BMB+Tree. If the buffer node is not full, the node hash of the buffer node and the component root hash of the memory component BMB+Tree are updated. (3) If the buffer node is full, pack all log entries in the buffer node into at least one new leaf node in chronological order, submit them in batch to the rightmost path of the main tree structure, and update the relevant Merkle hash. (4) If the number of log entries stored in the currently active memory component BMB+Tree reaches the capacity threshold B, the memory component BMB+Tree is frozen, a new active BMB+Tree is created, and the background converts the frozen memory component BMB+Tree into the corresponding disk component MLT, and generates four files and the component root hash of the disk component MLT in parallel. (5) After the disk component MLT is generated, the component metadata table is atomically updated to register the storage log time range, component root hash and file metadata of the disk component MLT. The file metadata mainly includes MLT identifier, time range, number of log entries, four types of file paths, file size, index parameters, Merkle File hierarchy layout, hash algorithm and hash length, etc.

[0050] Existing log management systems typically assume the storage server is trusted, preventing clients from independently verifying the authenticity, completeness, and correct order of returned logs when the server is not trusted. In contrast, the technical solution of this application allows clients to independently verify the authenticity, completeness, and correct order of logs even if the storage server is compromised or untrusted, through Merkle root digests and Merkle proofs. This improves the reliability of log forensics and compliance auditing. The following details the client's verifiable reading and Merkle proof generation process.

[0051] In one implementation of this application, the method further includes: (1) Upon receiving a log query request with a target time range submitted by the client, determine each relevant component and each irrelevant component based on the component metadata table and the target time range; relevant components refer to all memory components and disk components whose log storage time range intersects with the target time range, and irrelevant components refer to all memory components and disk components whose log storage time range does not intersect with the target time range. (2) Obtain the component root hash of each unrelated component through the component metadata table; (3) Obtain the target log entries within the target time range through each relevant component, and generate the local Merkel proofs for each relevant component; (4) Return the target log entry, the component root hash of each unrelated component, and the local Merkle proof of each related component to the client, so as to instruct the client to reconstruct the component root hash of each related component based on the target log entry and the local Merkle proof of each related component. Based on the reconstructed component root hash of each related component and the component root hash of each unrelated component, calculate the target global digest and determine the data verification result by comparing the target global digest with the trusted global digest.

[0052] First, the client submits a log query request Q=[lb, ub] to the storage server, carrying a target time range, where lb and ub represent the start and end timestamps of the log entry to be queried, respectively. Upon receiving the log query request Q, the storage server first determines the relevant and irrelevant components based on the component metadata table (Component Table) and the target time range. Relevant components refer to all memory and disk components whose log time range intersects with the target time range, while irrelevant components refer to all memory and disk components whose log time range does not intersect with the target time range. On one hand, the storage server collects the root hashes of each irrelevant component through the component metadata table (Component Table), obtaining an irrelevant component root hash set (irrelevantHashSet), which serves as part of the data integrity proof and prevents the storage server from intentionally omitting certain time segments. On the other hand, the storage server obtains all log entries within the target time range from each relevant component, denoted as the target log entry, and generates local Merkle proofs for each relevant component. The specific generation method of the local Merkle proof is described below.

[0053] In one implementation of this application, generating local Merkel proofs for each relevant component includes: For each related component belonging to the memory component, find the minimum and maximum timestamp log entries within the target time range from that related component; starting from the leaf node storing the minimum timestamp log entry, scan the ordered chain of leaf nodes to the leaf node storing the maximum timestamp log entry to obtain all log entries within the target time range, and collect the hashes of the left sibling nodes of the lower bound path, the hashes of the right sibling nodes of the upper bound path, and the leaf node records of both boundaries to form the local Merkle proof of that related component.

[0054] The relevant components could be either the memory component BMB+Tree or the disk component MLT. For the memory component BMB+Tree, the process begins by finding the smallest and largest timestamp log entries within the target time range [lb, ub]. Specifically, this can be done by locating the smallest timestamp log entry satisfying t ≥ lb and the largest timestamp log entry satisfying t ≤ ub in the main tree structure and buffer nodes of the memory component BMB+Tree. Then, starting from the leaf node storing the smallest timestamp log entry in the main tree structure, the process scans along the ordered chain of B+Tree leaf nodes to the leaf node storing the largest timestamp log entry, obtaining all log entries within the target time range [lb, ub]. The hashes of the left sibling nodes of the lower bound path, the hashes of the right sibling nodes of the upper bound path, and the necessary records of the leaf nodes on both sides are collected, thus forming a local Merkle proof for the memory component BMB+Tree.

[0055] As an example, Figure 5 This is a schematic diagram illustrating the operational principle of a local Merkel proof for generating memory components provided in an embodiment of this application. Figure 5 In the log query request Q=[6, 25], meaning the target time range is [6, 25], the process first uses a B+Tree to find the first log entry with the smallest timestamp (timestamp ≥ 6) of the minimum timestamp (9), while simultaneously collecting the boundary leaf node records (5) that do not belong to the result. Then, using a B+Tree, the process finds the last log entry with the largest timestamp (timestamp ≤ 25) of the maximum timestamp (25), while simultaneously collecting the hashes of the right sibling nodes (hn8, hn10, and hb) on the upper bound path, and collecting the boundary leaf node records (34, 38, and 40) that do not belong to the result. Next, the process continues from the date... Starting from the leaf node of log entry 9, scan to the right along the ordered chain of leaf nodes of B+Tree until 25, to obtain all log entries 9, 11, 13, ..., 25 within the target time range [6, 25]; finally, generate the local Merkle proof of the memory component BMB+Tree, which specifically includes the following: (1) Query result: log entries 9, 11, 13, ..., 25; (2) Lower bound proof: the boundary leaf node record 5 to the left of the lower bound; (3) Upper bound proof: the right sibling node hashes hn8, hn10 and hb of the upper bound path, and the boundary leaf node records 34, 38 and 40 to the right of the upper bound.

[0056] In another implementation of the embodiments of this application, generating local Merkel proofs for each relevant component includes: For each related component belonging to the disk component, the logical position range corresponding to the target time range is determined from the timestamp file of the related component according to the time range positioning logic set in the index file of the related component; all log entries within the target time range are obtained from the value file of the related component according to the logical position range; and a local Merkle proof of the related component is generated based on the hash value of the leaf node corresponding to the logical position range in the Merkle file of the related component.

[0057] For components of type MLT (Disk Component Level), the logical location range [pos_lb, pos_ub] corresponding to the target time range [lb, ub] can be determined from the MLT's timestamp file based on the time range positioning logic set in the MLT's index file. Specifically, the system can first load the index model in the index file, predict the positions of lb and ub in the timestamp file, and then sequentially correct to obtain the logical positions pos_lb and pos_ub. Then, based on the logical position range [pos_lb, pos_ub], all log entries within the target time range [lb, ub] are obtained from the MLT's value file. Here, the corresponding position offset pointer can be read from the timestamp file based on the two logical positions, and then the log content at the corresponding storage location can be read from the value file using this position offset pointer, thus obtaining all log entries within the target time range [lb, ub]. Subsequently, based on the hash values ​​of the leaf nodes corresponding to the logical location range [pos_lb, pos_ub] in the Merkle File of the disk component MLT, a local Merkle proof for the disk component MLT is generated. Since the Merkle File is laid out in layer order, the hash position of the sibling node can be directly calculated as follows: if the node position p in a certain layer is odd, then its sibling node position is p-1; if the node position p in a certain layer is even, then its sibling node position is p+1; the global offset is the starting offset of the layer plus the sibling node position in the layer, multiplied by the hash length to obtain the physical address. The system collects the necessary sibling node hashes outside the coverage interval from bottom to top until the component root hash of the disk component MLT. For example, assuming that the hash values ​​of the eight leaf nodes of Layer 0 in a Merkle file are h0, h1, h2, h3, h4, h5, h6, and h7, then the hash values ​​of the intermediate nodes of Layer 1 are H01, H23, H45, and H67, where H23 = Hash(h2||h3), and so on; the hash values ​​of the intermediate nodes of Layer 2 are H0123 and H4567, where H0123 = Hash(H01||H23), and so on; the hash value of the root node of Layer 3 is ROOT = Hash(H0123||H4567).Assuming the query hits the interval [h3, h4, h5], meaning the two boundary positions L=3 and R=5, for Layer0: since L=3 is odd, it means h3 is the right child node, and the left sibling node h2 needs to be hashed. Since R=5 is odd, it means h5 is the right child node, and the right sibling node hash is not needed. Therefore, Layer0 adds proof{h2}. For Layer1, the two boundary positions L=... 3 / 2 =1, R= 5 / 2 =2, the interval becomes [1, 2] = H23, H45. Since the left boundary L=1 is odd, it means H23 is the right child node, and the left sibling node H01 needs to be hashed. Since the right boundary R=2 is even, it means H45 is the left child node, and the right sibling node H67 needs to be hashed. Therefore, proof{H01, H67} is added to Layer1. For Layer2, the position boundaries L on both sides = 1 / 2 =0, R= 2 / 2 =1, the interval is [0, 1] = H0123, H4567, which has already covered the two child nodes of the root node, and no additional sibling node hash is needed. Continuing upwards, the root hash can be obtained. Therefore, the final generated local Merkle proof is {h2, H01, H67}.

[0058] As an example, Figure 6 This is a schematic diagram illustrating the operational principle of a local Merkel proof for generating disk components provided in an embodiment of this application. Figure 6 In this example, assuming the query result covers the leaf layer hashes h1 and h2 of the Merkle file, they are first merged to obtain the parent node hash h29. To continue calculating upwards, the system needs to read the right sibling node hash h30 of the parent node hash h29. Since the starting offset of Layer 1 is 28, and h30's position within this layer is 1, its global position is 28 + 1 = 29, and its physical address is 29 × hashLength. After reading h30, the system can sequentially calculate the upper-level node hash h54, and then continue reading the sibling node hash h55 of h54, finally calculating the root node hash h56. Therefore, the final generated local Merkle proof is {h30, h55}.

[0059] After generating the local Merkle proofs for each relevant component in the manner described above, the storage server will return the query result R (i.e., the target log entries within the target time range), the component root hashes of each unrelated component (i.e., the unrelated component root hash set irrelevantHashSet), the local Merkle proofs of each relevant component, the order of each component, and the necessary boundary information (mainly including the logical position of each relevant component, the actual lower bound timestamp, the actual upper bound timestamp, and the predecessor and successor records or timestamps immediately outside the query range, used to prove that the query result has not been omitted at the left and right boundaries) to the client.

[0060] After receiving the aforementioned data from the storage server, the client can choose not to trust the storage server and independently execute the data verification process. The main steps of this process include: (1) Based on the target log entries and the local Merkle proofs of each related component, reconstruct the component root hash of each related component; for example, assuming that the log results returned by a certain related component are 1 and 2, and the local Merkle proof is {h30, h55}, then the component root hash of the related component can be reconstructed as Hash(Hash(Hash(h1||h2)||h30)||h55), and so on; in addition, considering that the number of related components may be large, the client can start multiple threads to reconstruct the component root hash of each related component in parallel to improve processing efficiency; (2) Based on the component root hash of each related component and the component root hash of each unrelated component obtained by reconstruction, calculate the target global digest. Here, the component root hash of each related component and the component root hash of each unrelated component can be arranged into a complete component digest sequence according to the determined order of the component metadata table Component Table, and the global digest is recalculated based on the sequence to obtain the target global digest. (3) The data verification result is determined by comparing the target global digest with the trusted global digest. The trusted global digest here can be the trusted global digest generated by the root hash of each component stored in the component metadata table as described above, or it can be the global digest that the client holds in advance or obtains from trusted channels (including but not limited to client local secure storage, log producer signature, third-party audit service, on-chain digest anchoring, trusted timestamp service or trusted execution environment signature, etc., which serve to provide the client with a global digest trust anchor that does not depend on the storage server). The client will compare the recalculated target global digest with the trusted global digest. If the two are consistent, it means that the returned log has passed the Merkle Tree authentication in terms of content, order and scope integrity, and the data verification result can be determined to be successful. If the two are inconsistent, it means that the returned log result or proof has been tampered with, forged or omitted, and the data verification result can be determined to be unsuccessful.

[0061] As an example, assuming the log query range is [t100, t200], which involves related components MLT_3, MLT_4, and the current BMB+Tree, the storage server generates corresponding local Merkle proofs in the Merkle files of MLT_3 and MLT_4 according to the leaf node intervals, and generates corresponding local Merkle proofs in the current BMB+Tree. It then returns the component root hashes of unrelated components such as MLT_1 and MLT_2 as an unrelated component hash set, `irrelevantHashSet`, to the client. The client can reconstruct the component root hashes of MLT_3, MLT_4, and the current BMB+Tree in parallel, combine them with the unrelated component hash set `irrelevantHashSet` to obtain the target global digest, and finally confirm whether the storage server has deleted, modified, or omitted log content within [t100, t200] by comparing the target global digest with the trusted global digest. It can be seen that the client can verify whether the log results returned by the storage server are true, complete, and untampered, relying only on the trusted root digest.

[0062] In summary, the technical solution of this application, by adopting a storage architecture with a hierarchical authentication data structure, can avoid the problem of double-layer I / O access amplification; by designing an original memory component BMB+Tree, Merkle hash updates can be distributed across a batch of log entries, with write costs approaching constant; by decomposing the global authentication structure into multiple independent and immutable MLT subtrees, a single proof path can be constrained by segmented capacity, resulting in shorter range query proofs; even if the storage server is compromised or untrusted, the client can still independently verify the authenticity, integrity, and correctness of the logs through Merkle root digests and Merkle Proofs, thereby improving the reliability of log forensics and compliance auditing.

[0063] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0064] The above mainly describes a log storage method based on a hierarchical authentication data structure. The following will describe a log storage device based on a hierarchical authentication data structure.

[0065] Please see Figure 7 This application illustrates a log storage device based on a hierarchical authentication data structure, comprising: Log entry acquisition module 701 is used to acquire the current log entry to be stored; The memory component writing module 702 is used to write the current log entry to the memory component that is currently receiving log data. The memory component is a storage component that is located in the memory layer and has an authentication data structure. The disk component conversion module 703 is used to convert the memory component into the corresponding disk component if the number of log entries stored in the memory component reaches the capacity threshold. The disk component is a storage component that is located at the disk layer and has an authentication data structure.

[0066] In one implementation of this application, the memory component maintains a main tree structure and cache nodes; the main tree structure is a structure that maintains ordered leaf nodes and Merkle root digests, used to store log indexes ordered by timestamps and complete Merkle hash paths; the cache nodes are ordered buffer nodes with a capacity of f, used to cache the latest f written log entries, f > 1; the component root hash of the memory component is determined based on the root hash of the main tree structure and the node hash of the cache nodes; the memory component writing module includes: The log entry writing unit is used to write the current log entry to the cache node; The first hash update unit is used to update the node hash if the cache node is not full, and update the component root hash of the memory component based on the updated node hash. The second hash update unit is used to, if the cache node is full, package the f log entries in the cache node into at least one new leaf node in chronological order, add the new leaf node to the rightmost side of the main tree structure, and update the internal node hash of the main tree structure from bottom to top to obtain the updated root hash; clear the cache node and reset the node hash, and update the component root hash of the memory component based on the updated root hash and the reset node hash.

[0067] In one implementation of this application, the disk component conversion module includes: The memory component freeze unit is used to freeze a memory component as an immutable component and create a new memory component to receive subsequent log data. The disk component conversion unit is used to convert immutable components into disk components via a background thread.

[0068] In one implementation of this application, the disk component maintains a value file, a timestamp file, an index file, and a Merkle file; the value file is used to store log content sequentially; the timestamp file is used to store the timestamp and the position offset of the corresponding log content in the value file; the index file is used to build a lookup index for the timestamp file; the Merkle file is used to store a complete Merkle tree with a hierarchical layout; the disk component conversion module includes: The value file generation unit is used to generate value files based on the content of each log entry stored in the memory component; The timestamp file generation unit is used to generate timestamp files based on the storage location of the content of each log entry in the value file and the timestamp of each log entry; The index file generation unit is used to generate an index file based on a preset time range positioning logic; The Merkle file generation unit is used to construct a complete Merkle tree based on the content and timestamp of each log entry to obtain the Merkle file. The root hash of the complete Merkle tree is used as the component root hash of the disk component.

[0069] In one implementation of this application, the log storage device based on a hierarchical authentication data structure further includes: The component metadata table writing module is used to write the storage log time range and component root hash of each memory component, as well as the storage log time range and component root hash of each disk component, into the component metadata table. The trusted global digest generation module is used to generate a trusted global digest based on the root hashes of each component stored in the component metadata table.

[0070] In one implementation of this application, the trusted global digest generation module includes: The first global digest calculation unit is used to construct a global Merkle tree using the root hashes of each component stored in the component metadata table as leaf nodes, and to use the root hashes of the global Merkle tree as trusted global digests. The second global digest calculation unit is used to concatenate the root hashes of each component stored in the component metadata table in order to obtain the overall root hash, and calculate the hash value of the overall root hash as a trusted global digest.

[0071] In one implementation of this application, the log storage device based on a hierarchical authentication data structure further includes: The relevant component determination module is used to determine each relevant component and each irrelevant component based on the component metadata table and the target time range after receiving a log query request submitted by the client carrying a target time range. Relevant components refer to all memory components and disk components whose storage log time range intersects with the target time range, while irrelevant components refer to all memory components and disk components whose storage log time range does not intersect with the target time range. The component root hash retrieval module is used to retrieve the component root hash of each unrelated component from the component metadata table; The local Merkel proof generation module is used to obtain target log entries within the target time range through various related components and generate local Merkel proofs for each related component. The result return module is used to return the target log entry, the component root hash of each unrelated component, and the local Merkle proof of each related component to the client. This instructs the client to reconstruct the component root hash of each related component based on the target log entry and the local Merkle proof of each related component. Based on the reconstructed component root hash of each related component and the component root hash of each unrelated component, the client calculates the target global digest and determines the data verification result by comparing the target global digest with the trusted global digest.

[0072] In one implementation of this application, the local Merkel proof generation module includes: The memory component proof generation unit is used to find the minimum and maximum timestamp log entries within the target time range for each related component belonging to the memory component. Starting from the leaf node storing the minimum timestamp log entry, it scans the ordered chain of leaf nodes to the leaf node storing the maximum timestamp log entry to obtain all log entries within the target time range. It also collects the hashes of the left sibling nodes of the lower bound path, the hashes of the right sibling nodes of the upper bound path, and the leaf node records of both boundaries to form the local Merkle proof of the related component.

[0073] In another implementation of this application's embodiments, the local Merkel proof generation module includes: The disk component proof generation unit is used to determine the logical position range corresponding to the target time range from the timestamp file of each related component belonging to the disk component, based on the time range positioning logic set in the index file of the related component; obtain all log entries within the target time range from the value file of the related component based on the logical position range; and generate the local Merkle proof of the related component based on the hash value of the leaf node corresponding to the logical position range in the Merkle file of the related component.

[0074] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the log storage method based on a hierarchical authentication data structure described in any of the above embodiments.

[0075] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute a log storage method based on a hierarchical authentication data structure as described in any of the above embodiments.

[0076] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, it implements the steps in the embodiments of the log storage methods based on hierarchical authentication data structures described above, for example... Figure 2 Steps 201-203 are shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the above-described device embodiments, for example, implementing... Figure 7 The functions of modules 701-703 of the device shown.

[0077] The computer program 82 can be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 82 in the electronic device 8.

[0078] The processor 80 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0079] The memory 81 can be an internal storage unit of the electronic device 8, such as a hard disk or memory. The memory 81 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 81 can include both internal and external storage units of the electronic device 8. The memory 81 is used to store the computer program and other programs and data required by the electronic device. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0084] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0088] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A log storage method based on a hierarchical authentication data structure, characterized in that, include: Retrieve the current log entry to be stored; The current log entry is written to the memory component that is currently receiving log data. The memory component is a storage component that is in the memory layer and has an authentication data structure. If the number of log entries stored in the memory component reaches the capacity threshold, the memory component is converted into a corresponding disk component, which is a storage component at the disk layer and has an authentication data structure.

2. The method as described in claim 1, characterized in that, The memory component maintains a main tree structure and cache nodes; the main tree structure is a structure that maintains ordered leaf nodes and Merkle root digests, used to store log indexes arranged in order of timestamps and complete Merkle hash paths; the cache node is an ordered buffer node with a capacity of f, used to cache the latest f written log entries, f > 1; the component root hash of the memory component is determined based on the root hash of the main tree structure and the node hash of the cache node; The step of writing the current log entry into the memory component that is currently receiving log data includes: Write the current log entry to the cache node; If the cache node is not full, update the node hash, and based on the updated node hash, update the component root hash of the memory component; If the cache node is full, the f log entries in the cache node are packaged into at least one new leaf node in chronological order, the new leaf node is added to the rightmost side of the main tree structure, and the internal node hash of the main tree structure is updated from bottom to top to obtain the updated root hash; the cache node is cleared and the node hash is reset, and the component root hash of the memory component is updated based on the updated root hash and the reset node hash.

3. The method as described in claim 1, characterized in that, The step of converting the memory component into a corresponding disk component includes: The memory component is frozen as an immutable component, and a new memory component is created to receive subsequent log data; The immutable component is converted into the disk component via a background thread.

4. The method as described in claim 1, characterized in that, The disk component maintains a value file, a timestamp file, an index file, and a Merkle file; the value file is used to store log content sequentially; the timestamp file is used to store timestamps and the position offsets of corresponding log content within the value file; the index file is used to build a lookup index for the timestamp file; The Merkle file is used to store a complete Merkle tree laid out in a hierarchical manner; The step of converting the memory component into a corresponding disk component includes: The value file is generated based on the contents of each log entry stored in the memory component; The timestamp file is generated based on the storage location of the contents of each log entry in the value file and the timestamp of each log entry. The index file is generated based on a preset time range positioning logic; Based on the content and timestamp of each log entry, the complete Merkle tree is constructed to obtain the Merkle file, and the root hash of the complete Merkle tree is used as the component root hash of the disk component.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Write the storage log time range and root hash of each memory component, as well as the storage log time range and root hash of each disk component, into the component metadata table; A trusted global digest is generated based on the root hashes of each component stored in the component metadata table.

6. The method as described in claim 5, characterized in that, The process of generating a trusted global digest based on the root hashes of each component stored in the component metadata table includes: Using the root hashes of each component stored in the component metadata table as leaf nodes, a global Merkle tree is constructed, and the root hash of the global Merkle tree is used as the trusted global digest. Alternatively, the root hashes of each component stored in the component metadata table can be concatenated in order to obtain the overall root hash, and the hash value of the overall root hash can be calculated as the trusted global digest.

7. The method as described in claim 5, characterized in that, The method further includes: Upon receiving a log query request from a client carrying a target time range, the relevant components and irrelevant components are determined based on the component metadata table and the target time range. The relevant components refer to all memory components and disk components whose log storage time range intersects with the target time range, and the irrelevant components refer to all memory components and disk components whose log storage time range does not intersect with the target time range. The component root hash of each unrelated component is obtained through the component metadata table; The target log entries within the target time range are obtained through the respective related components, and the local Merkel proofs of the respective related components are generated. The target log entry, the component root hash of each unrelated component, and the local Merkle proof of each related component are returned to the client to instruct the client to reconstruct the component root hash of each related component based on the target log entry and the local Merkle proof of each related component. Based on the reconstructed component root hash of each related component and the component root hash of each unrelated component, a target global digest is calculated. The data verification result is determined by comparing the target global digest with the trusted global digest.

8. The method as described in claim 7, characterized in that, The generation of the local Merkel proofs for each of the relevant components includes: For each of the related components belonging to the memory component, the minimum timestamp log entry and the maximum timestamp log entry within the target time range are found from the related component; starting from the leaf node storing the minimum timestamp log entry, the sorted chain of leaf nodes is scanned to the leaf node storing the maximum timestamp log entry to obtain all log entries within the target time range, and the hashes of the left sibling nodes of the lower bound path, the hashes of the right sibling nodes of the upper bound path, and the leaf node records of both boundaries are collected to form the local Merkle proof of the related component.

9. The method as described in claim 7, characterized in that, The generation of the local Merkel proofs for each of the relevant components includes: For each of the related components belonging to the disk component, based on the time range positioning logic set in the index file of the related component, the logical position range corresponding to the target time range is determined from the timestamp file of the related component; based on the logical position range, all log entries within the target time range are obtained from the value file of the related component, and based on the hash value of the leaf node corresponding to the logical position range in the Merkle file of the related component, a local Merkle proof of the related component is generated.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the log storage method based on a hierarchical authentication data structure as described in any one of claims 1 to 9.

Images