Log-structured storage system

Abstract

Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing and storage. One of the methods includes a storage system receiving a request for processing of data for a centralized ledger system. A plurality of data types of the centralized ledger system includes block data, transaction data, and index data. The storage system determines that the data belongs to one of the plurality of data types of the centralized ledger system and applies a processing engine type designated for processing the one data type.

Description

TECHNICAL FIELD

[0001] This document relates to log-structured storage systems. BACKGROUND

[0002] A distributed ledger system (DLS), which can also be referred to as a consensus network and / or a blockchain network, enables participating entities to store data securely and immutably. Without reference to any particular use case, a DLS is often referred to as a blockchain network. Examples of blockchain network types can include public blockchain networks, private blockchain networks, and consortium blockchain networks. A consortium blockchain network is provided for a selected group of entities that control the consensus process and that includes a layer of access control.

[0003] Generally, each node (e.g., a blockchain network node) of a DLS stores or has a complete backup of the blockchain network data, such that each node can be independent and the local data at each node can be trusted to provide service. However, this storage scheme presents demanding storage requirements and increases the storage cost for each node, especially as the DLS reaches scale. Accordingly, solutions are desired to improve efficiency and reduce the cost of storage systems. SUMMARY

[0004] Described herein are techniques for log-structured storage systems for storing data in, for example, a distributed ledger system (e.g., a blockchain network) and / or a blockchain-based centralized ledger system (e.g., a general-purpose auditable ledger service system) that employs the data structure of a blockchain to leverage the immutability, reliability, and trustworthiness of data stored on the blockchain.

[0005] Also provided herein are one or more non-transitory computer-readable storage media coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, will cause the one or more processors to perform operations in accordance with embodiments of the methods provided herein.

[0006] Also provided herein are systems for implementing the methods provided herein. A log-structured storage system includes one or more processors and a computer-readable storage medium coupled to the one or more processors and having instructions stored thereon that, when executed by the one or more processors, will cause the one or more processors to perform operations in accordance with embodiments of the methods provided herein.

[0007] It should be appreciated that methods in accordance with this document can include any combination of aspects and features described herein. That is, methods in accordance with this document are not limited to the combinations of aspects and features specifically described herein, but also include any combination of the aspects and features provided.

[0008] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 FIG. 1 is a diagram illustrating an example of an environment that can be used to perform embodiments herein.

[0010] Figure 2 FIG. 2 is a diagram illustrating an example of an architecture in accordance with embodiments herein.

[0011] Figure 3 FIG. 3 is a diagram illustrating an example of a blockchain-based log-structured storage system in accordance with embodiments herein.

[0012] Figure 4 FIG. 4 is a diagram illustrating an example of a tiered storage system in accordance with embodiments herein.

[0013] Figure 5 FIG. 5 is a flow diagram illustrating an example of a process for performing a write operation for a log-structured storage system in accordance with embodiments herein.

[0014] Figure 6 FIG. 6 is a flow diagram illustrating an example of a process for generating an index related to a write operation for a log-structured storage system in accordance with embodiments herein.

[0015] Figure 7 FIG. 7 is a flow diagram illustrating an example of a process for performing a read operation for a log-structured storage system in accordance with embodiments herein.

[0016] Figure 8 FIG. 8 is a flow diagram illustrating an example of a process for improving a read operation for a log-structured storage system in accordance with embodiments herein.

[0017] Figure 9 FIG. 9 is a flow diagram illustrating an example of a process for managing data log files stored in a log-structured storage system in accordance with embodiments herein.

[0018] Figure 10 FIG. 10 is a flow diagram illustrating an example of a process for performing data migration in a log-structured storage system in accordance with embodiments herein.

[0019] Figure 11 FIG. 11 is a flow diagram illustrating an example of a process for performing data flow control in a log-structured storage system in accordance with embodiments herein.

[0020] Figure 12 FIG. 1 is a flow diagram illustrating an example of a process that can be performed in accordance with embodiments herein.

[0021] Figure 13 FIG. 2 is a flow diagram illustrating an example of a process that can be performed in accordance with embodiments herein.

[0022] Figure 14 FIG. 3 is a flow diagram illustrating an example of a process that can be performed in accordance with embodiments herein.

[0023] Figure 15 FIG. 4 is a diagram illustrating an example of modules of an apparatus in accordance with embodiments herein.

[0024] In the various figures, like reference numerals and designations refer to like elements throughout. DETAILED DESCRIPTION

[0025] Techniques are described herein for a log-structured storage system for storing data in, for example, a distributed ledger system (e.g., a blockchain network) and / or a blockchain-based centralized ledger system (e.g., a general-purpose auditable ledger service system) that employs the data structure of a blockchain to leverage the immutability, reliability, and trustworthiness of data stored on the blockchain. In some embodiments, the distributed ledger system and the blockchain-based centralized ledger system can be collectively referred to as a blockchain-based ledger system.

[0026] In some embodiments, the blockchain-based centralized ledger system can be a centralized ledger system that can provide a cryptographically verifiable, stateless data ledger storage with time-critical audit (with non-repudiation and tamper resistance). In some embodiments, the blockchain-based centralized ledger system can provide ledger services based on a cloud platform that is characterized by a centralized endorsement with trustworthiness and neutrality. The blockchain-based centralized ledger system can provide a high-reliability and high-performance auditable stream ledger service that combines the high trustworthiness of a blockchain system with the high performance and low latency of a centralized system to handle various data and logs with audit requirements, traceability, and tracking.

[0027] The techniques described herein yield several technical effects. In some embodiments, the described techniques can be applied in various applications and scenarios to provide efficient, trustworthy, scalable, cost-effective, and high-performance data storage and management. The described techniques can provide a simple and well-defined set of application programming interfaces (APIs) for blockchain data including, for example, transaction data, block data, state data, and index data.

[0028] The described technology provides a log-structured storage system that not only provides I / O services, but also considers cost and individualized needs to provide functionalities such as tiering, data compression, shared storage, error correction coding, and state snapshots, especially after the amount of data stored in a blockchain system reaches a scale. The log-structured storage system can provide features such as log-structured data storage, as well as asynchronous and / or concurrent processing, to enable performance optimization, efficient processing, a trusted environment, versatility (e.g., both distributed ledger systems and blockchain-based centralized ledger systems are available), and improved storage schemes. The described technology can provide an overall framework or architecture for providing such functionalities and features.

[0029] Generally, data generated and / or stored in a distributed ledger system (e.g., a blockchain network) can be referred to as blockchain data. Blockchain data can include or be categorized as transaction data, block data, state data, and index data. In some embodiments, data generated and / or stored in a blockchain-based centralized ledger system (e.g., a general-purpose auditable ledger service system) can include or be categorized as transaction data, block data, and index data.

[0030] In some embodiments, various blockchain data can be received in the form of a key-value pair (KVP) represented as <hash(value), value>. The value can be actual data representing one or more of a block, a transaction, or a state of the blockchain network. The key can be a hash value of the value.

[0031] In some embodiments, for block data, each block can include a block header and a block body. The block header can include identity information of the particular block, and the block body can include transactions confirmed with the block. In some embodiments, block data is a data structure in a blockchain system, and generally has one or more of the following properties. For example, (1) the content of the block data stored in each node in the blockchain network is theoretically consistent after consensus is reached in the blockchain network. (2) Block numbers increase densely. (3) There is hash entanglement between consecutive blocks. (4) Block data is append-only. That is, once consensus is reached, historical block data will not be modified. (5) The access frequency of block data is generally low. The storage space occupied by block data is generally large.

[0032] In some embodiments, the state data can be assembled as a globally shared state (also referred to as a world state). The world state can include a mapping between account addresses and account states. The world state can be stored in a data structure such as a Merkle Patricia tree (MPT). In some embodiments, for example, in a smart contract scenario, the state data can be designed based on the content of a Merkle tree. It is an incremental content-addressed dataset. The storage space occupied by the state data is typically large.

[0033] In some embodiments, the state data can be further classified into a current state and a historical state. In some embodiments, the current state is the state data corresponding to the latest block and is the source of data when executing the latest transaction on the blockchain network. In some embodiments, the historical state is a content-addressed dataset storing all state data from the genesis block to the latest block. In some embodiments, the historical state data is stored in a historical state tree. The historical state tree can store state information as key-value pairs (KVPs) represented as <hash(node value), node value> that are content-addressed. The value or node value can be the account state of an account associated with a blockchain node, while the key can be the hash value of the corresponding account state. In some embodiments, the current state data is stored in a current state tree. In some embodiments, the current state tree can be location-addressed based on one or more location-related identifiers (IDs). For example, the current state tree can store state information as KVPs represented as <node ID, node value> that are addressable according to the corresponding node ID.

[0034] In some embodiments, the transaction data can include data related to the input and output of a series of operations. In some embodiments, the transaction data can include data related to the exchange of a valuable (e.g., asset, product, service, currency).

[0035] In some embodiments, the index data can indicate a mapping correspondence between data (e.g., transaction data, block data, and state data) and a physical location in a storage system where the data is stored for addressing or retrieving the data. In some embodiments, the index data can indicate where the corresponding data is stored in the storage system. In some embodiments, the index data can include one or more of: an index indicating a correspondence from a block hash value to a block number, an index indicating a correspondence from a block hash value to a storage location, an index indicating a correspondence from a transaction hash value to a transaction, or an index indicating a correspondence from a receipt hash value to a receipt. In some embodiments, the index data does not include the content of the blockchain data.

[0036] As more and more transactions are input into the blockchain, the size of the blockchain data (e.g., state data and block data) can grow larger and larger. In DLS, each node of the DLS stores an entire copy of the blockchain, which can take up a large amount of storage space even though some old block data or state data is not frequently accessed.

[0037] In some embodiments, the blockchain data is stored in data files by the log-structured system, and the data files are continuously appended and partitioned based on time. In some embodiments, the data can not be rearranged according to the categorization of keys (e.g., the data is not arranged by key values or other metrics such that hot data and cold data are not mixed in multiple data log files), thereby greatly reducing the technical challenges of the tiered implementation.

[0038] In some embodiments, the log-structured storage system uses two append-only data files to store the blockchain data to provide data persistence: data log files and index log files. For example, block data, transaction data, state data, and additional self-describing data can be stored in data log files, while index data indicating the storage locations of the transaction data, block data, and state data (e.g., the identification and offset of the data log files) can be stored in index log files.

[0039] In the blockchain data, the transaction data and block data can be log-structure friendly, can include append-only data, such that they can be written into the data log files by directly adding or appending these data into the corresponding data log files. In some embodiments, the writing of the transaction data and block data does not require a large amount of compaction. For example, it can require a relatively small amount of transaction reproduction, and can not require block rollback. In some embodiments, the state data can be log-structure friendly data, such that the historical state data can be increased without the need for compaction.

[0040] In some embodiments, the log-structured storage system can support multiple levels of data tiering, and support multiple storage devices, such as cloud disks, network attached systems (NAS), and object storage services (OSS) (low frequency, archive). For example, the log files can be stored in cloud-based storage systems, NAS or OSS devices, or self-built distributed storage systems.

[0041] In some embodiments, different types of log files can have different storage strategies. For example, data log files that have not been accessed for a relatively long time can be stored in inexpensive and relatively low-speed storage devices, such as NAS / OSS, and can be processed using compression and error correction coding for storage. As another example, index log files can be stored on high-speed storage devices such as cloud disks.

[0042] In some embodiments, the log-structured storage system can perform data tiering by using a least recently used (LRU) memory cache and disk cache to optimize read performance of low-speed storage devices.

[0043] In some embodiments, the log-structured storage system can provide a tiered pool manager that manages multiple tiers of storage device pools. In some embodiments, each pool supports multiple disks or storage devices in a cluster. The tiered pool manager can manage the space, stress, and health of the pools. In some embodiments, the log-structured storage system can provide a migration task manager that manages bi-directional migration tasks of data between different tiers of storage devices; manages result callbacks, statistics, life cycle of migration tasks, etc. In some embodiments, the log-structured storage system can provide a migration scheduler that supports pluggable policies, manages data migration policies, and provides data create / query / update / delete interfaces.

[0044] The disclosed log-structured storage system adopts the idea of the merge tree (LSM-Tree) architecture. In some embodiments, the log-structured storage system can include multiple log-structured storage instances (or streams), where each log-structured storage instance is responsible for storing and managing data of a distributed ledger system (e.g., a blockchain system) or a blockchain-based centralized ledger system. In some embodiments, the log-structured storage system can convert random write operations into ordered append operations to mitigate the write amplification problem caused by frequent “dirty” page flushes due to a large number of random write operations. In some embodiments, the log-structured storage system can delay write flush operations in high-performance scenarios and reduce the number of synchronization operations to improve the efficiency and performance of the entire system.

[0045] To provide further context for the embodiments herein, and as noted above, a distributed ledger system (DLS), which can also be referred to as a consensus network (e.g., composed of peer-to-peer nodes) and a blockchain network, enables participating entities to securely and immutably transact and store data. While the term blockchain is often associated with a particular network and / or use case, a blockchain, as used herein, generally refers to a DLS without reference to any particular use case.

[0046] A blockchain is a data structure that stores transactions in a manner that is unalterable. As a result, transactions recorded on a blockchain are reliable and trustworthy. A blockchain includes one or more blocks. Each block in the chain is linked to the immediately preceding block in the chain by including a cryptographic hash of the preceding block. Each block also includes a timestamp, its own cryptographic hash, and one or more transactions. Transactions that have been validated by nodes in the blockchain network are hashed and incorporated into a Merkle tree. A Merkle tree is a data structure in which data at the leaf nodes of the tree is hashed, and all hashes in each branch of the tree are concatenated at the root of the branch. This process continues down the tree until the root of the entire tree, which stores a hash representing all of the data in the tree. The hash can be quickly verified as being the hash of the transactions stored in the tree by determining whether the hash is consistent with the structure of the tree.

[0047] In some embodiments, e.g., in a blockchain network that is a network of computing nodes, a blockchain can be implemented in a distributed or decentralized or at least partially decentralized manner for storing transactions. Each computing node (also referred to as a blockchain network node) can manage, update, and maintain one or more blockchains by broadcasting transactions, validating transactions, and confirming transaction validity, among other things. As noted above, a blockchain network can be provided as a public blockchain network, a private blockchain network, or a consortium blockchain network. Embodiments herein are described in further detail herein with reference to a consortium blockchain network. However, it is contemplated that embodiments herein can be implemented in any appropriate type of blockchain network.

[0048] Generally, a consortium blockchain network is private among the participating entities. In a consortium blockchain network, consensus processing is controlled by a set of authorized nodes, which can be referred to as consensus nodes, one or more of which are operated by a respective entity (e.g., a financial institution, an insurance company). For example, a consortium of 10 entities (e.g., financial institutions, insurance companies) can operate a consortium blockchain network, each entity operating at least one node in the consortium blockchain network.

[0049] In some examples, within a consortium blockchain network, a global blockchain is provided as a blockchain that is replicated across all nodes. That is, for the global blockchain, all consensus nodes are in a state of full consensus. To reach consensus (e.g., agree to add a block to the blockchain), a consensus protocol is implemented within the consortium blockchain network. For example, a consortium blockchain network can implement a Practical Byzantine Fault Tolerance (PBFT) consensus, which is described in further detail below.

[0050] In some embodiments, the centralized ledger system can also employ the data structure of a blockchain to leverage the immutability, reliability, and trustworthiness of the data stored on the blockchain. In some embodiments, for example, the centralized ledger system can be referred to as a blockchain-based centralized ledger system or a general purpose auditable ledger service system. In some embodiments, the blockchain-based centralized ledger system can include a central trusted authority that provides transparent, immutable, and cryptographically verifiable data stored in blocks of the blockchain data structure. The stored data can be in a log format, for example, including not only transaction logs, but also other transaction data and block data. Due to the presence of the central trusted authority, the blockchain-based centralized ledger system can establish trust without the need to perform consensus processing. In some embodiments, the blockchain-based centralized ledger system can be more efficient compared to typical blockchain-based distributed or decentralized ledger systems. In some embodiments, the blockchain-based centralized ledger system can provide a cloud-based storage service with enhanced trust, efficiency, and storage performance.

[0051] Figure 1 FIG. 1 is a diagram illustrating an example of an environment 100 that can be used to implement embodiments herein. In some examples, the environment 100 enables entities to participate in a consortium blockchain network 102. The environment 100 includes computing devices 106, 108 and a network 110. In some examples, the network 110 includes a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and connects web sites, user devices (e.g., computing devices), and backend systems. In some examples, the network 110 can be accessed through wired and / or wireless communication links. In some examples, the network 110 enables communication with or within the consortium blockchain network 102. Generally, the network 110 represents one or more communication networks. In some cases, the computing devices 106, 108 can be nodes of a cloud computing system (not shown), or each computing device 106, 108 can be a separate cloud computing system that includes multiple computers interconnected by a network and working as a distributed processing system.

[0052] In the depicted example, the computing devices 106, 108 can each comprise any suitable computing system capable of participating as a node in the consortium blockchain network 102. Examples of computing devices include, but are not limited to, servers, desktop computers, laptops, tablets, and smartphones. In some examples, the computing devices 106, 108 host one or more computer-implemented services for interacting with the consortium blockchain network 102. For example, the computing device 106 can host a computer-implemented service, e.g., a transaction management system, of a first entity, e.g., user A, that the first entity uses to manage its transactions with one or more other entities, e.g., other users. The computing device 108 can host a computer-implemented service, e.g., a transaction management system, of a second entity, e.g., user B, that the second entity uses to manage its transactions with one or more other entities, e.g., other users. In Figure 1 In the depicted example, the consortium blockchain network 102 is represented as a Peer-to-Peer network of nodes, and the computing devices 106, 108 provide nodes of a first entity and a second entity, respectively, that participate in the consortium blockchain network 102.

[0053] Figure 2 is a diagram illustrating an architecture 200 in accordance with embodiments herein. The example conceptual architecture 200 includes participant systems 202, 204, 206 corresponding to participants A, B, and C, respectively. Each participant (e.g., user, business) participates in a blockchain network 212 provided as a Peer-to-Peer network that includes a plurality of nodes 214, at least some of which record information in a blockchain 216 in a non-tamperable manner. As further detailed in the figure, although a single blockchain 216 is depicted schematically in the blockchain network 212, multiple copies of the blockchain 216 are provided and maintained on the blockchain network 212.

[0054] In the depicted example, each participant system 202, 204, 206 is provided by or on behalf of participants A, B, and C, respectively, and functions as a respective node 214 in the blockchain network. As used herein, a node generally refers to an individual system (e.g., computer, server) that connects to the blockchain network 212 and enables the respective participant to participate in the blockchain network. In Figure 2In the example, a participant corresponds to each node 214. However, it is expected that a participant can operate multiple nodes 214 within the blockchain network 212, and / or multiple participants can share a single node 214. In some examples, participant systems 202, 204, 206 communicate with or through the blockchain network 212 using protocols (e.g., Hypertext Transfer Protocol Security (HTTPS)) and / or using Remote Procedure Calls (RPC).

[0055] Node 214 can have different levels of participation within the blockchain network 212. For example, some nodes 214 may participate in consensus processing (e.g., as monitoring nodes adding blocks to blockchain 216), while other nodes 214 may not participate in this consensus processing. As another example, some nodes 214 may store a complete copy of blockchain 216, while other nodes 214 may store only a partial copy of blockchain 216. For example, data access privileges can restrict the amount of blockchain data that a given participant can store within their respective system. Figure 2 In the example, participant systems 202, 204 and 206 store corresponding complete copies 216', 216” and 216”' of blockchain 216.

[0056] Blockchain (e.g., Figure 2 A blockchain (216) consists of a series of blocks, each storing data. Examples of data include transaction data representing a transaction between two or more participants. Although the term "transaction" is used herein by way of non-limiting example, it is expected that any suitable data (e.g., documents, images, videos, audio) can be stored in the blockchain. Examples of transactions can include (but are not limited to) the exchange of valuables (e.g., assets, products, services, currencies). Transaction data is stored immutably in the blockchain. That is, transaction data cannot be changed.

[0057] Before transaction data is stored in a block, it is hashed. Hash processing is the process of converting transaction data (provided as string data) into a fixed-length hash value (also provided as string data). It is impossible to dehash the hash value to obtain the transaction data. Hash processing ensures that even slight changes to the transaction data will result in a completely different hash value. Furthermore, as mentioned above, the hash value has a fixed length. That is, the length of the hash value is fixed regardless of the size of the transaction data. Hash processing involves processing the transaction data using a hash function to generate a hash value. Examples of hash functions include (but are not limited to) the Secure Hash Algorithm (SHA)-256, which outputs a 256-bit hash value.

[0058] Transaction data for multiple transactions is hashed and stored in a block. For example, the hash values of two transactions are provided and hashed themselves to provide another hash value. This process is repeated until a single hash value is provided for all transactions to be stored in the block. This hash value is referred to as the Merkle root hash value and is stored in the header of the block. Any change in any transaction causes its hash value to change and ultimately the Merkle root hash value to change.

[0059] Blocks are added to the blockchain through a consensus protocol. Multiple nodes in the blockchain network participate in the consensus protocol and compete to add blocks to the blockchain. Such nodes are referred to as consensus nodes. PBFT introduced above is used as a non-limiting example of a consensus protocol. Consensus nodes execute the consensus protocol to add transactions to the blockchain and update the overall state of the blockchain network.

[0060] In more detail, consensus nodes generate a block header, hash all transactions in the block, and combine the resulting hash values in pairs to generate further hash values until a single hash value is provided for all transactions in the block (the Merkle root hash value). This hash value is added to the block header. Consensus nodes also determine the hash value of the latest block in the blockchain (i.e., the last block added to the blockchain). Consensus nodes also add a nonce and a timestamp to the block header.

[0061] In general, PBFT provides a practical Byzantine state machine replication that tolerates Byzantine faults (e.g., faulty nodes, malicious nodes). This is achieved by assuming that faults will occur in PBFT (e.g., assuming independent node failures and / or manipulated messages sent by consensus nodes). In PBFT, consensus nodes are provided in an order that includes primary consensus nodes and backup consensus nodes. The primary consensus nodes are changed periodically. Transactions are added to the blockchain by reaching agreement on the global state of the blockchain network by all consensus nodes within the blockchain network. In this process, messages are transmitted between consensus nodes and each consensus node proves that the message was received from a specified peer node and verifies that the message was not tampered with during transmission.

[0062] In PBFT, the consensus protocol is provided in multiple phases with all consensus nodes starting with the same state. First, a client sends a request to a primary consensus node to invoke a service operation (e.g., to execute a transaction within a blockchain network). In response to receiving the request, the primary consensus node multicasts the request to backup consensus nodes. The backup consensus nodes execute the request and each sends a reply back to the client. The client waits until it receives a threshold number of replies. In some examples, the client waits until it receives f+1 replies, where f is the maximum number of faulty consensus nodes that can be tolerated within the blockchain network. The end result is that a sufficient number of consensus nodes agree on the order in which a record is added to the blockchain and the record is either accepted or rejected.

[0063] In some blockchain networks, cryptography is used to maintain the privacy of transactions. For example, if two nodes want to keep a transaction private such that other nodes in the blockchain network cannot see the details of the transaction, the two nodes can encrypt the transaction data. Examples of encryption include, but are not limited to, symmetric encryption and asymmetric encryption. Symmetric encryption is an encryption process that uses a single key for both encryption (to generate ciphertext from plaintext) and decryption (to generate plaintext from ciphertext). In symmetric encryption, the same key can be used by multiple nodes, so each node can encrypt / decrypt transaction data.

[0064] Asymmetric encryption uses key pairs, each key pair including a private key and a public key, the private key being known only to the respective node and the public key being known to any or all other nodes in the blockchain network. A node can use another node’s public key to encrypt data, and that encrypted data can be decrypted using the other node’s private key. For example, referring again to Figure 2 , participant A can use participant B’s public key to encrypt data and send the encrypted data to participant B. Participant B can use its private key to decrypt the encrypted data (ciphertext) and extract the original data (plaintext). A message encrypted using a node’s public key can only be decrypted using that node’s private key.

[0065] Asymmetric encryption is used to provide digital signatures, which enable participants in a transaction to confirm other participants in the transaction and the validity of the transaction. For example, a node can digitally sign a message, and another node can confirm that the message was sent by the node based on participant A’s digital signature. Digital signatures can also be used to ensure that a message is not tampered with during transmission. For example, referring again to Figure 2To send a message to participant B, participant A generates a hash value of the message and then encrypts the hash value using its private key to provide a digital signature as an encrypted hash value. Participant A attaches the digital signature to the message and sends the message with the digital signature to participant B. Participant B decrypts the digital signature using participant A's public key and extracts the hash value. Participant B hashes the message and compares the hash values. If the hash values are the same, participant B can confirm that the message indeed came from participant A and has not been tampered with.

[0066] Figure 3 FIG. 1 is a diagram illustrating an example of a log-structured storage system according to embodiments herein. The log-structured storage system 300 can store data for a distributed ledger system (e.g., a blockchain network) and / or a blockchain-based centralized ledger system (e.g., a general-purpose auditable ledger service system) (collectively, a blockchain-based ledger system) that stores data on one or more blockchains.

[0067] In some embodiments, the log-structured storage system 300 can be implemented by each consensus node of a blockchain network or a central node of a blockchain-based centralized ledger system. In some embodiments, the log-structured storage system 300 can be connected to a distributed storage system 340 built by a client node of the blockchain-based ledger system. As shown, the log-structured storage system 300 includes a front-end input / output (I / O) subsystem 310, a multi-layer storage subsystem 320, and a back-end data management subsystem 330. In some embodiments, the front-end I / O subsystem 310 can perform write operations to write data into data files (e.g., data log files and index log files) stored in the multi-layer storage subsystem 320 and perform read operations to access data from the data files stored in the multi-layer storage subsystem 320. In some embodiments, the back-end data management subsystem 330 can process, reorganize, and manage data in the data files according to different needs to improve the efficiency and performance of the overall system.

[0068] The front-end I / O subsystem 310 can include any suitable computing elements (e.g., one or more of a processor, a memory 315, etc.) to perform the methods described herein. In some embodiments, the front-end I / O subsystem 310 can perform front-end I / O operations including various read and write operations (e.g., insert, update, delete, query, etc.) on various data elements.

[0069] In some embodiments, all data elements (e.g., transaction data, block data, and state data) processed by the front-end I / O subsystem 310 can be stored in a log file format, whether the log file is generated from a write operation or a file generated from operations of the back-end data management subsystem 330, such as storage tiering, compaction, data compression, erasure coding, etc.

[0070] In some embodiments, data processed by the front-end I / O subsystem 310 can be stored in two types of log files: (1) data log files (e.g., data log files 390, 362, 364, 366, 372, 374, and 376) that store data such as blockchain data (e.g., transaction data, block data, state data) and self-describing metadata; and (2) index log files (e.g., index log file 380) that store index information (e.g., identifiers and offsets of data log files) that indicates the physical location of the data. In some embodiments, the data log files do not store the index information, and the index information is maintained by a separate index log file.

[0071] In some embodiments, the front-end I / O subsystem 310 can be configured to perform a write operation to write blockchain data into the data log file 390. In some embodiments, the blockchain data can include block data, transaction data, or state data generated by a blockchain network or a distributed ledger system. In some embodiments, the blockchain data can include block data and transaction data generated by a blockchain-based centralized ledger system. In some embodiments, the data written to the data log file 390 can include metadata that describes the data block, such as transaction hash values and sequence values, block hash values and block numbers, snapshot version numbers, cyclic redundancy check (CRC) codes, encryption information, etc. In some embodiments, the data log file 390 can be an append-only file.

[0072] In some embodiments, the front-end I / O subsystem 310 can be configured to generate an index that indicates the physical location where the respective data is stored in the log-structured storage system 300 (e.g., in a data log file in the multi-tier storage subsystem 320). In some embodiments, the index can be stored in the index log file 380. In some embodiments, the data log files and the index log file can be stored in the multi-tier storage subsystem 320. In some embodiments, the index can be stored in the index log file 380, which is stored in the one storage device of the multi-tier storage subsystem 320 that has the highest access speed.

[0073] In some embodiments, the data log file can be continuously updated based on data write or append operations. In some embodiments, the data log file can have a configurable maximum length, such as between 512 MB and 2 GB. In some embodiments, if it is determined that the data log file has reached the maximum length or size, the data log file can be sealed or set to read-only, and a new data log file can be allocated for new write operations.

[0074] In some embodiments, the front-end I / O subsystem 310 can perform write operations, including modifications to data stored in the log-structured storage system 300. In some embodiments, the front-end I / O subsystem 310 handles modifications to data by adding or appending data in a log format to a data log file, without overwriting the original data. In some embodiments, the data log file can form a write-ahead log (WAL) layer that can be used for crash recovery.

[0075] In some embodiments, the front-end I / O subsystem 310 stores index information in the memory 315 that indicates a mapping correspondence between data (e.g., transaction data, block data, and state data) and data log files storing the data to address or retrieve the data. In some embodiments, the index data in the memory can be organized using a log-structured merge (LSM) approach. In some embodiments, indexes for newly written data can be stored in the memory 315 and flushed to the index log file 380 when the memory usage exceeds a predetermined threshold. In this way, indexes for old data can be stored in the index log file 380 in disk storage or hard drive storage, freeing up space to cache indexes for new hot data in the memory 315.

[0076] In some embodiments, the index data can include one or more of: an index indicating a correspondence from a block hash value to a block number, an index indicating a correspondence from a block hash value to a storage location, an index indicating a correspondence from a transaction hash value to a transaction, or an index indicating a correspondence from a receipt hash value to a receipt. In some embodiments, the index data for a blockchain-based centralized ledger system can include an index indicating a correspondence from a sequence to a transaction storage location and / or an index indicating a correspondence from a time sequence to a transaction hash value.

[0077] In some embodiments, the front-end I / O subsystem 310 can include a plurality of in-memory index maps stored in the memory 315. In some embodiments, an in-memory index map can be considered any suitable component, unit, module, or data structure (e.g., a table or a structure) for maintaining indexed data in the memory of the front-end I / O subsystem 310. The in-memory index maps can be a key component of the front-end I / O subsystem 310 that determines the scalability and performance of the front-end I / O subsystem 310 and the entire log-structured storage system 300. In some embodiments, because the blockchain data is time sensitive and recently written transaction data and block data are relatively more likely to be accessed again, the log-structured storage system 300 can store indexes of hot data in the index maps in the memory 315 to improve the performance of the entire log-structured storage system 300.

[0078] In some embodiments, the in-memory index maps can maintain indexes that indicate mappings from transaction hash values to sequence values and / or indexes that indicate mappings from block hash values and block numbers to physical locations of data. In some embodiments, the front-end I / O subsystem 310 periodically persists checkpoints of the index maps in the memory 315 into index log files. For example, the front-end I / O subsystem 310 can periodically or at a certain point in time capture a snapshot of the index data in the memory 315 and store the snapshot in the index log files 380 in the multi-tier storage subsystem 320. This can create a point in time at which the log-structured storage system 300 can apply changes contained in the index log files 380 during a recovery process after an unexpected shutdown or crash of the log-structured storage system 300. In some embodiments, the front-end I / O subsystem 310 can read data (e.g., transaction data, block data, and state data) by querying the in-memory index maps and determining the current location of the requested data.

[0079] In some embodiments, a full checkpoint of the in-memory index maps can be written to an index log file when the index log file is created. In some embodiments, the index log files can be updated by batching the indexes of write operations. In some embodiments, the batch size can be dynamically configurable, such as thousands of transaction write operations or a few megabytes (MB) of write operations. In some embodiments, when an index log file has been updated for a certain number of write operations, the index log file can be sealed or set to read-only, and a new index log file can be created to write new data.

[0080] In some embodiments, to recover from an abnormal crash, the front-end I / O subsystem 310 can load the index log files (e.g., index log files 380) into the memory 315 and scan the page bottom of the data log files 390 to ensure the consistency of the data log files 390 and the index log files 380. In some embodiments, the index log files can be a few batches behind the data log files, so the recovery time can occupy limited I / O resources and time.

[0081] In some embodiments, the indexes of newly written transaction data and block data can be added to the index map and the index log files, but the indexes of existing transaction data and block data can not be modified except in replay attack and block rollback scenarios. In some embodiments, to achieve high concurrency of read-write operations, the in-memory index map can be divided into a read-only base index map 316 and a read-write delta index map 312. In some embodiments, the base index map 316 can store the indexes of cold data, while the delta index map 312 can store the indexes of newly written data. In some embodiments, the hash value index can be stored in a hash value table, while the sequence index can be stored in a B-tree.

[0082] In some embodiments, during the write operation of the front-end I / O subsystem 310, the index information of the data can be first updated to the delta index map 312. During the read operation, the front-end I / O subsystem 310 can first search the requested data in the delta index map 312. If the requested data is not found in the delta index map 312, the front-end I / O subsystem 310 can then search the base index map 316.

[0083] In some embodiments, the front-end I / O subsystem 310 can periodically flush the index data from the memory 315 to the index log files 380. In some embodiments, the basic process of index flushing can include the following operations: (1) combining the delta index map 312 and the base index map 316; (2) persisting the base index map 316 (e.g., storing the base index map to the index log files); (3) releasing part or all of the base index map 316 from the memory 315; (4) swapping the index data to the memory 315 by reading the requested index data.

[0084] In some embodiments, the front-end I / O subsystem 310 can convert the delta index map 312 in the memory 315 to immutable index maps 314, then flush them to the index log files 380, and create new delta index maps to receive the indexes generated according to new requests. In this way, the storage occupancy of the delta index map can be reduced to improve the performance of the log-structured storage system 300.

[0085] In some embodiments, to reduce the impact on the front-end I / O, the index maps in the memory can be merged asynchronously at the back-end. In some embodiments, the merging process can be triggered by at least one of the following two conditions: (1) the size of the delta index map exceeds a predetermined threshold; (2) a new snapshot is created. In some embodiments, the front-end I / O subsystem 310 can generate a merged index map to include the immutable index maps 314 to be flushed into the index log files 380. In some embodiments, the front-end I / O subsystem 310 can combine the merged index map with the current base index map 316 to generate a new base index map.

[0086] In some embodiments, during operation, the front-end I / O subsystem 310 can run with multiple base index maps and index log files. In some embodiments, when consolidation is needed in certain cases, minor compaction and major compaction can be performed periodically by combining all base index maps and delta index maps into one base index map. Major compaction mainly merges and manages indexes, which can be used for scenarios such as snapshot, garbage collection load, and index file management.

[0087] In some embodiments, major compaction can be performed by merging the base index maps and the delta index maps and generating a new base index map, and storing it into a new index log file. In some embodiments, minor compaction can be performed by combining several index log files and generating a new index log file, which can reduce the number of index log files. In some embodiments, if the size of the current index log file reaches a predetermined threshold, the current index log file can be set to a sealed or immutable state and closed, and a new index log file can be created for new index data.

[0088] In some embodiments, during read operations, if a search in the in-memory index map fails, two or more I / O operations can be needed, which can burden the log-structured storage system 300. In some embodiments, the front-end I / O subsystem 310 can provide a multi-level cache mechanism with a memory cache 313 and a block cache 317 (e.g., using flash media such as SSD cloud disks).

[0089] In some cases, the log-structured storage system 300 can receive a large read request such that the log-structured storage system 300 needs to access multiple data log files to obtain the complete request data for the client. However, accessing multiple data log files can incur non-trivial overhead. In some embodiments, the backend data management subsystem 330 can perform a compact operation to cascade logically adjacent data chunks to reduce fragmentation. In some embodiments, the compact operation can have overhead and can be performed when data fragmentation is severe.

[0090] In some embodiments, the multi-tier storage subsystem 320 can include multi-tier storage devices. A storage device can include a storage medium and a corresponding software and / or hardware interface. In some embodiments, the multi-tier storage devices can include multiple storage devices with different performance characteristics (e.g., access speed). For example, the multi-tier storage devices can include cloud disks, network-attached storage (NAS) devices, and object storage service (OSS) devices. In some embodiments, the multi-tier storage devices are tiered according to a tiering structure according to one or more performance characteristics. In some embodiments, the one or more performance characteristics can include access speed, access bandwidth, or access latency. For example, the multi-tier storage devices can include first-tier storage devices with a first performance characteristic (e.g., access speed) and second-tier storage devices with a second performance characteristic (e.g., relatively lower access speed relative to the first-tier storage devices) lower than the first performance characteristic, and so on. As shown, an example of the multi-tier storage subsystem 320 can include first-tier storage devices 350 including cloud disks or cloud-based storage devices (e.g., solid state drive (SSD) cloud disks, embedded SSD (ESSD) cloud disks), second-tier storage devices 360 including NAS devices, and third-tier storage devices 370 including OSS devices. Figure 3

[0091] In some embodiments, the multi-tier storage devices can store different types of data. In some embodiments, data can be classified as hot data 355, warm data 365, and cold data 375 based on, for example, the time at which the data is generated or received or the frequency at which the data is accessed. For example, data of recent transactions can be hot data; data of transactions yesterday can be warm data, and data of historical transactions conducted a week ago can be cold data. As another example, data in the last 10 blocks generated in a blockchain can be hot data; data in the last 10 blocks generated recently can be warm data, and data in other earlier blocks can be cold data. However, in some embodiments, the genesis block of a blockchain can be considered as hot data as it is frequently accessed.

[0092] ​​In some embodiments, the multi-tiered storage subsystem 320 can store the hot data 355, the warm data 365, and the cold data 375 into the multi-tiered storage devices, respectively. For example, the first tier storage device 350 can store the hot data 355; the second tier storage device 360 can store the warm data 365; and the third tier storage device 370 can store the cold data 375. In some embodiments, one tier storage device can store one or more types of data, e.g., based on storage space and cost. For example, the first tier storage device 350 can store the hot data 355 and some of the warm data 365, while the second tier storage device 360 can store the remaining warm data 365 and some of the cold data 375.

[0093] In some embodiments, each tier of storage devices can store a data log file that includes blockchain data generated by a blockchain-based ledger system (e.g., a distributed ledger system and / or a blockchain-based centralized ledger system). For example, the first tier storage device 350 can store a first data log file 390 that includes first blockchain data generated by a blockchain-based ledger network, and the second tier storage device 360 can store a second data log file 362 that includes second blockchain data generated by a blockchain-based ledger system, and so on.

[0094] In some embodiments, the blockchain data stored in a data log file in a relatively lower tier of storage devices can be written earlier than the blockchain data stored in a data log file in a relatively higher tier of storage devices. For example, the second blockchain data stored in the second data log file 362 on the second tier storage device 360 can be written earlier than the first blockchain data stored in the first data log file 390 on the first tier storage device 350.

[0095] In some embodiments, the first tier storage device 350 can further store one or more index log files 380 that include index data indicating physical storage locations of data in the data log files 390, 362, 364, 366, 372, 374, and 376 stored by the multi-tiered storage devices 350, 360, and 370. For example, as shown in FIG. 3, the first tier storage device 350 can store an index log file 380 that includes index data indicating physical storage locations of the blockchain data in the data log file 390 stored by the first tier storage device 350, physical storage locations of the blockchain data in the data log files 362, 364, and 366 stored by the second tier storage device 360, and physical storage locations of the blockchain data in the data log files 372, 374, and 376 stored by the third tier storage device 370. Figure 3

[0096] ​In some embodiments, one or more index log files can be stored in the second tier storage device 360 and / or the third tier storage device 370.

[0097] In some embodiments, the index log files and the data log files stored on the multi-tier storage subsystem 320 are append-only log files. In some embodiments, the blockchain data stored in the data log files can include block data, transaction data, and historical state data.

[0098] In some embodiments, a higher tier storage device can store a data log file that includes blockchain data migrated from a lower tier storage device. For example, the first tier storage device can store a data log file that includes blockchain data that is more frequently accessed than the blockchain data in the data log file of the second tier storage device and the blockchain data migrated from the second tier storage device.

[0099] In some embodiments, the storage system 300 can further include a distributed storage system 340 that includes storage media such as non-volatile memory express (NVME), SSD, hard disk drives (HDD), and shingled magnetic recording (SMR). In some embodiments, the distributed storage system 340 can be generated or extended by a client node of a blockchain-based ledger system to have better availability, partition tolerance, flexibility, and cost. For example, the distributed storage system 340 can allow scaling by adding more servers or storage nodes and thereby linearly increasing capacity and performance. It can use standard servers, drives, and networks that are inexpensive. In some embodiments, the distributed storage system 340 can improve utilization of standard servers, thus resulting in less power consumption, better cooling efficiency, better space utilization, less maintenance cost, etc.

[0100] The front-end I / O subsystem 310 can perform write operations on the blockchain data and generate index log files 380 and data log files 390, 362, 364, 366, 372, 374, and 376 stored on the multi-tier storage subsystem 320. Over time, the data stored on the multi-tier storage subsystem 320 can accumulate and aggregate, and can degrade the performance of the log-structured storage system 300. The back-end data management subsystem 330 can process and reorganize the data according to different needs, for example, to improve the performance and reduce the cost of the log-structured storage system 300. In some embodiments, the back-end data management subsystem 330 can manage the data independently from the front-end I / O subsystem 310. For example, the back-end data management subsystem 330 can perform data management operations such as tiering, compression, erasure coding, state snapshots, compaction, and validation on the sealed or read-only index log files and data log files at the back-end. In some embodiments, the back-end data management subsystem 330 can implement flow control to minimize the impact on the front-end I / O processing of the front-end I / O subsystem 310.

[0101] In some embodiments, the tasks of the back-end data management subsystem 330 can include rewriting of stored data and replacement of the index corresponding to the rewritten data. In some embodiments, the back-end data management subsystem 330 can automatically determine whether to rewrite the data log files at the back-end. In some embodiments, the back-end data management subsystem 330 can determine the placement of the rewrite based on the configuration such as tiering, compression, and erasure coding. In some embodiments, the back-end data management subsystem 330 can read data from one or more source data log files and rewrite the data to a target data log file. In some embodiments, when the rewrite is complete, the back-end data management subsystem 330 can set the target data log file to a sealed or immutable state and generate a corresponding target index log file. In some embodiments, the target index log file can include a list of data log files that can be safely deleted, as well as the data log files referenced by the target index log file. In some embodiments, the back-end data management subsystem 330 does not reclaim the old data log files that can still be used by the real-time instance of the front-end I / O subsystem 310.

[0102] In some embodiments, the back-end data management subsystem 330 can process the read-only index log files and corresponding read-only data log files generated according to the I / O operations of the front-end I / O subsystem 310. In some embodiments, the back-end data management subsystem 330 can analyze the index log files and determine, for example, the level of hot, warm, or cold of the data, the amount of data, the rate of garbage, and / or the amount of fragmentation. In some embodiments, based on the rate of garbage, the disk usage, and / or the system requests, the back-end data management subsystem 330 can perform one or more of the following tasks:

[0103] (1) Data tiering. For example, when the storage media usage approaches the safe upper limit, it can be necessary to migrate data to the storage media of the next tier or lower tier storage devices.

[0104] (2) Data compression. For example, when the storage media usage approaches the safe upper limit, it can be necessary to compress data files.

[0105] (3) Erasure coding (EC). For example, when the storage media usage approaches the safe upper limit, it can be necessary to release storage space through erasure coding.

[0106] (4) State snapshot. For example, when there is a state modification (e.g., reclaiming storage space after data deletion), a snapshot of the blockchain state can be performed.

[0107] (5) Data compaction. For example, if the garbage or fragmentation in the data log files grows to a certain size, which significantly affects the performance of the log-structured storage system 300, it can be necessary to clean up the garbage or fragmentation.

[0108] (6) Verification. For example, a cyclic redundancy check (CRC) can be performed on the data on the storage media periodically or on demand.

[0109] Data tiering:

[0110] In some embodiments, for write requests that require relatively higher performance, the write requests can be written to faster storage devices (e.g., SSD cloud disks, ESSD cloud disks, NVME, etc.) among a plurality of different storage devices. For write requests that require lower performance in exchange for lower overhead, the write requests can be written to storage device media (e.g., NAS, etc.). In some embodiments, the backend data management subsystem 330 can use a mixed set of slow and fast storage devices for data tiering and data migration. For example, the access frequency of new block data generated by the blockchain network can be relatively higher than that of old block data, and the new block data can be stored in faster storage devices. In some embodiments, a portion of the new block data with the highest access frequency can be stored in a memory cache (e.g., memory cache 313) and / or a high-speed disk cache (e.g., block cache 317).

[0111] In some embodiments, both distributed ledger systems and blockchain-based centralized ledger systems have strong hot-cold characteristics that make them suitable for tiered storage. For example, a tiered storage system such as multi-tier storage subsystem 320 can be used to include one or more of the following features: (1) a combination of fast storage media with smaller storage space and slow storage media with larger storage space improves space usage without sacrificing performance; (2) supports cold migration (e.g., automatic migration of cold data from fast media to slow media) and warm-up (e.g., migration of data from slow media to fast media); (3) scalability to reduce maintenance costs as the scale increases; (4) supports flexible configuration based on user needs; (5) supports multimedia storage pools; or (6) fast migration to new storage media.

[0112] Figure 4 is a diagram illustrating an example of a tiered storage system 400 according to embodiments herein. In some embodiments, a tiered storage system can include multiple tiers or multiple layers of storage devices, for example, based on access speed of the storage devices. For example, referring to Figure 4 , the multiple storage devices for tiering can be divided into four tiers or levels, including hot, warm, cold, and archive for storing log files based on their hot and cold characteristics. For example, the storage devices of tiered storage system 400 can be divided into four tiers or levels for storing hot log files 410, warm log files 412, cold log files 414, and archive files 416, respectively.

[0113] In some embodiments, each tier or level of storage devices can be considered as a virtual pool, and each pool can support multiple physical or virtual file systems (also referred to as storage devices). For example, tiered storage system 400 can include a first tier pool 402, a second tier pool 404, a third tier pool 406, and a fourth tier pool 408. In some embodiments, the file systems supported in the pools can include one or more of the following short-term file systems: cloud disks (e.g., block devices of virtual machines (VMs) installed on ext4 / xfs file systems); NAS (e.g., nfs file systems with posix interface); OSS low frequency (adapted to virtual file system, software development kit (SDK) system, representative state transfer (REST) interface, etc. format); and OSS archive (adapted to virtual file system, SDK system, REST interface, etc. format).

[0114] For example, as shown in Figure 4 , first tier pool 402 can include ESSD and SSD devices in a cloud storage system (e.g., multi-tier storage subsystem 320) that store hot log files 410. Second tier pool 404 can include NAS devices and cloud disks in the cloud storage system that store warm log files 412.

[0115] The third tier pool 406 can include OSS low frequency devices that store cold log files 414 in the cloud storage system.

[0116] The fourth tier pool 408 can include OSS archive devices that store archive files 416 in the cloud storage system.

[0117] In some embodiments, the file system can also include a long-term file system, such as a self-built distributed system (e.g., a distributed storage system 340 built by a client node of a blockchain-based ledger system). For example, the first tier pool 402 can also include NVME devices of the distributed storage system that store hot log files 410 generated by the client nodes of the blockchain network (e.g., as part of the distributed storage system 340). The second tier pool 404 can also include SSD devices of the distributed storage system that store hot log files 412. The third tier pool 406 can also include HDD devices of the distributed storage system that store cold log files 414. The fourth tier pool 408 can also include SMR devices of the distributed storage system that store archive files 416. In some embodiments, a unified interface can be provided for all file systems of the entire log-structured storage system 300.

[0118] In some embodiments, the tiered storage system 400 can include one or more subsystems or components, such as (1) a tiered pool manager 418, (2) a migration task manager 420, (3) a migration scheduler 422 for managing data tiering, or (4) a quality of service (QoS) manager 423. In some embodiments, each manager can include any suitable computing element (e.g., one or more of a processor, a memory, etc.) to perform the functions described herein. For example, these managers can manage data flow between multiple storage devices of different performance and cost, e.g., to improve the performance and efficiency of the entire log-structured storage system by exploiting the performance and cost differences between different storage devices.

[0119] In some embodiments, the tiered pool manager 418 can be configured to manage each tier of storage devices. In some embodiments, the tiered pool manager 418 can perform one or more of the following functions: manage storage space and pressure of the multiple-tier storage devices; provide file creation, deletion, and statistical analysis functions for a specified tier (e.g., select a storage device to create a data log file according to a system request); maintain a tier file mapping table that indicates the correspondence of data files, their storage locations in the respective tiers of storage devices, and the hotness or coldness of the data files, etc.

[0120] In some embodiments, the migration task manager 420 can manage bidirectional data migration tasks between different storage devices, manage the life cycle of tasks, call back results, perform statistical analysis, execute migration tasks, and the like.

[0121] In some embodiments, the migration scheduler 422 can support pluggable migration policies, manage data migration policies, and provide data create / query / update / delete interfaces. In some embodiments, the migration scheduler 422 can perform scheduling management on migration tasks to achieve effective flow control on the migration tasks. In some embodiments, the migration scheduler 422 can score or otherwise assign respective scores to data log files at the backend, and generate migration tasks for the data log files according to the score ranking and migration policies. In some embodiments, the data log files can be scored according to a scoring formula that takes into account tiers of storage devices, access frequency, original data creation time, migration cost, and / or other factors. In some embodiments, the migration scheduler 422 can work with other subsystems or components of the tiered storage system 400 to quickly validate different migration policies.

[0122] In some embodiments, data migration can be performed automatically according to predetermined data migration policies. For example, different data in the high-speed storage device can be scored according to a predetermined scoring scheme, and based on the respective scores of the different data, they can be migrated to the low-speed device at the backend to release cache space. In some embodiments, in certain applications, some data in the low-speed device can be determined to be hot data. The hot data can be first saved in the disk cache, and if the score of the data log file meets the requirements, it can be migrated to the high-speed device. In some embodiments, after the data file is migrated from the source storage device to the target storage device, the original data can or can not be deleted in the source storage device. For example, if the target storage device is a top-tier storage device, the data log file in the disk cache does not need to be deleted, but can be automatically replaced by other data.

[0123] In some embodiments, the QoS manager 423 can be configured to manage data flow or other performance metrics of the tiered storage system 400 to improve QoS. For example, in some cases, burst I / O writes to the high-speed storage devices can cause the high-speed storage devices in the higher tiers to be heavily occupied or used. In some embodiments, the QoS manager 423 can throttle data flow into the storage pool at a high usage rate (e.g., 85% or another threshold) to avoid the storage pool from being filled up too quickly. Flow control can prevent performance degradation of the tiered storage system 400 and can free up storage space for data migration. To improve efficiency of data migration while reducing impact on front-end I / O operations (e.g., by the front-end I / O subsystem 310), flow-controlled data migration can be performed at the back-end (e.g., by the back-end data management subsystem 330). In some embodiments, the migration speed can be positively correlated with the usage rate of the storage devices. For example, if the usage rate of the storage devices is low, flow control can be reduced to avoid excessive impact on front-end I / O. If the usage rate of the storage devices is high, flow control can be removed to speed up data migration.

[0124] In some cases, the usage rate of the high-speed storage devices can be full and front-end I / O operations can be heavily throttled. In some embodiments, data can be written directly to the lower tier storage devices without migrating the data from the higher tier storage devices to the lower tier storage devices. For example, if the first tier storage devices 350 in the tiered storage system 400 are full or reach a usage threshold, data can be written directly to the second tier storage devices 360. In some embodiments, large (e.g., having a size greater than a threshold) data such as blockchain block data can be written directly into data log files in the lower tier storage devices to save cost due to data migration. Figure 3

[0125] In some embodiments, to further reduce consumption of network resources, hard disk throughput, storage space, and other resources due to data migration, and to reduce impact on front-end I / O operations, compression and erasure coding can be performed by default when data is migrated to the lower tier storage devices.

[0126] ​Compared to high-speed storage devices, low-speed or archival storage devices have relatively poor performance. Generally, most data ends up being written to the low-speed tier of the storage device. Caching hot data on the high-speed storage device and migrating it to the high-speed storage device based on the hot and cold characteristics of the data helps improve read performance. In some embodiments, two or more of the following caches can be implemented to improve read performance: (1) memory cache (e.g., least recently used (LRU) cache 424); (2) fast disk cache (e.g., least frequently used (LFU) disk cache 426 on the high-speed storage device). In some embodiments, the total size of the memory cache 424 can be dynamically configured, for example, in hundreds of MB to a few GB. Similarly, the total size of the fast disk cache can be dynamically configured, for example, in 1 GB to tens of GB.

[0127] In some embodiments, some historical data that has been frequently accessed, such as the genesis block of a blockchain, can be placed in the LFU cache of the fast storage device.

[0128] Data compression:

[0129] For distributed ledger systems and blockchain-based centralized ledger systems, compression of data blocks can effectively reduce costs and improve the performance of the log-structured storage system. Due to its inherent characteristics and features, the log structure can facilitate compression in the log-structured storage system.

[0130] In some embodiments, data on the write front-end can not be compressed and can be directly written to the high-speed storage device (e.g., SSD cloud disk), for example, by appending the data in a data log file. In some embodiments, when the data log file reaches a certain size, it can be set to be immutable. In some embodiments, the back-end data management subsystem 330 can compress the original data log file on the back-end and replace the original data log file with the compressed data file. In this way, since the compression operation is performed on the back-end, the impact of the compression operation on the front-end I / O operation can be reduced or minimized.

[0131] Generally, when determining the compression size or capacity of the data log file, the effectiveness of compression and read amplification can need to be considered and balanced. For example, in some cases, if the compression size or capacity of the data log file is too small (e.g., less than 4 KB), the space saved due to compression can be limited and the compression performance can not be optimal. On the other hand, if the compression size or capacity of the data log file is too large, the read amplification can also become larger (e.g., to read a transaction entry, the entire compressed data log file including the transaction entry needs to be first decompressed). In some embodiments, the compression size of the data log file can be set to 16 KB - 128 KB.

[0132] In some embodiments, the compressed data log file can include a plurality of records, where each record can include a compression header and a compressed data body. In some embodiments, the metadata of the compressed data can include version information, compression algorithm, length, and CRC, etc.

[0133] For encrypted data, the randomness of the encryption itself can make the performance of data compression suboptimal. Therefore, in some embodiments, for data that needs to be encrypted (e.g., in a trusted execution environment (TEE)), the data can be compressed before encryption or after decryption.

[0134] In some embodiments, for a compressed log file, the base index mapping can encode the physical data size of the compressed log file, modify and record the corresponding index, and record the file ID of the log file, the offset of the log file, and the compressed data size of the log file.

[0135] Erasure coding

[0136] In some embodiments, the backend data management subsystem 330 can perform erasure coding on the data in the data log file. For example, the backend data management subsystem 330 can use erasure coding to write incoming data into the data log file at the backend.

[0137] For a distributed ledger system, in order to implement a Byzantine fault-tolerant log file layer among the consensus nodes of the blockchain network, erasure coding can be performed to reduce the amount of cold data stored on each consensus node of the distributed ledger system. For example, for 4 consensus nodes, 4 data replicas can be written before performing erasure coding. After performing erasure coding (e.g., an 8+3 erasure coding scheme), the 4 nodes can store less than 2 data replicas (e.g., 1.375 data replicas).

[0138] For a blockchain-based centralized ledger system, the centralized structure does not require data redundancy due to backup by multiple nodes. In some embodiments, erasure coding can be performed in a tiered storage system in a blockchain-based centralized ledger system to reduce data redundancy, e.g., in data backup in a top-tier storage device or a distributed storage system.

[0139] Data compaction

[0140] In some embodiments, transaction data, block data, and historical state data are append-only and cannot be deleted or overwritten, so data compaction is not performed on these data. In some embodiments, data compaction can be used to handle current state data. Data compaction generally includes garbage collection and data defragmentation.

[0141] In some embodiments, the back-end data management subsystem 330 can order the data log files according to their respective garbage rates and arrange them in descending order from high garbage rate to low garbage rate. In some embodiments, the back-end data management subsystem 330 can overwrite data log files with relatively high garbage rates. For example, the back-end data management subsystem 330 can overwrite data log files with garbage rates exceeding a predetermined threshold. In some embodiments, the earlier a data log file is created, the more likely the data in the data log file is to be overwritten, which means that the garbage rate of older data log files can be higher than that of newer data log files.

[0142] In some embodiments, the back-end data management subsystem 330 can implement a garbage collection mechanism that can set a maximum data amount for each overwrite. In some embodiments, multiple collection procedures can be performed in parallel, for example by multiple live instance streams of the front-end I / O subsystem 310, to improve the overall efficiency of garbage collection.

[0143] In some embodiments, a live instance stream of the front-end I / O subsystem 310 can obtain a garbage rate and report the obtained garbage rate to the back-end data management subsystem 330, which can determine an appropriate or optimal stream to overwrite data.

[0144] Defragmentation is generally a process of locating discontinuous data fragments stored on a storage device and then rearranging and restoring the fragments into fewer fragments or whole files. Defragmentation can reduce data access time and can more efficiently use storage. In some embodiments, the back-end data management subsystem 330 can perform defragmentation periodically, from time to time, or upon request.

[0145] In some embodiments, for certain types of data, such as world state data or state object data, the keys of these data have a certain level of hash property. If the keys have prefixes (e.g., different state objects have different prefixes), performing compaction on such data by putting the data into the same file or adjacent files can improve read performance.

[0146] State snapshots

[0147] A state snapshot can capture the state of a system (e.g., a distributed ledger system) at a particular point in time. In some embodiments, the backend data management subsystem 330 can perform a state snapshot operation to generate and store state data for the log-structured storage system 300. In some embodiments, the state data can include historical state data and current state data. The historical state data can include historical states of the distributed ledger system for backtracking, while the current state data can include the latest state data of the distributed ledger system. As time elapses, the size of the historical state data can grow and occupy a large amount of storage space. In some embodiments, to improve historical data backtracking and storage space usage, the backend data management subsystem 330 can perform a snapshot operation on the current state data.

[0148] The log-structured design of the log-structured storage system 300 can facilitate the snapshot operation and improve the performance and efficiency of the log-structured storage system 300. In some embodiments, the snapshot operation can be performed based on a write-once-redirect (ROW) approach, which provides efficient indexing for a data set corresponding to a snapshot.

[0149] In some embodiments, the snapshot functionality of the log-structured storage system 300 can support flash creation (e.g., at a second level) and rollback, which can have only limited or minimal impact on the front-end I / O operations. In some embodiments, the backend data management subsystem 330 can create hard links of the data log files and the index log files to avoid data duplication.

[0150] In some embodiments, when data from a write request is stored to a data log file, a record can be generated to include a snapshot identification (ID) indicating a snapshot version. In some embodiments, the backend data management subsystem 330 can perform one or more of the following operations in response to receiving a state snapshot request:

[0151] (1) write an operation log (op log) corresponding to the snapshot creation;

[0152] (2) increase the snapshot version by 1;

[0153] (3) write all new snapshot write requests to a new in-memory delta index map (redirect);

[0154] (4) after all write requests associated with the old snapshot are completed, flush the index to the current index log file, perform a compaction on all index files, merge the index files into a single index log file, and set the merged single index log file to a sealed state (the data log files are also sealed during the compaction process);

[0155] (5) create a new index file based on the new snapshot version; and

[0156] (6) Create a directory corresponding to the snapshot and create hard links to the directory for the data log files and index log files associated with the snapshot.

[0157] In some embodiments, the backend data management subsystem 330 can perform compaction on the backend to recover deleted snapshots.

[0158] In some embodiments, if snapshot upload is required, a data structure of the snapshot (e.g., bitmap representation of data ranges with 1 bit) can be maintained. For example, in the index log file created when the snapshot is created, the bitmap corresponding to the snapshot can be set to all 0s. Upon receiving a write request, the bitmap can be updated to 1, indicating that the data in that snapshot version is modified.

[0159] In some embodiments, a snapshot version number can correspond to an index log file, the snapshot version number indicating the write requests corresponding to all the indexes in the index log file.

[0160] Verification

[0161] In some embodiments, the backend data management subsystem 330 can perform a CRC check on the data recorded in the log files. In some embodiments, the backend data management subsystem 330 can perform the CRC check periodically, from time to time, or upon request.

[0162] In some embodiments, when importing the index files generated by the backend data management subsystem 330 into the live instance stream of the frontend I / O subsystem 310, the in-memory index map of the live instance stream can be more up-to-date than the index files generated by the backend data management subsystem 330, and can include location information of new data chunks and old data chunks. In some embodiments, the live instance stream can traverse the in-memory base map, replace the corresponding index entries, and then generate new index log files without referencing the old data log files. The live instance stream can then safely delete the old data log files and index log files.

[0163] In some embodiments, in a log-structured storage framework (e.g., the log-structured storage system 300), streams can be used as processing engines, components, units, or modules that operate to process I / O requests. Each stream can be adapted to different business scenarios by different configurations. In some embodiments, a stream can be implemented by one or more processors coupled with software to perform operations such as managing data log files, index log files, manifest files, request queues, and the like. In some embodiments, a live stream can refer to a live instance that processes frontend I / O operations of the frontend I / O subsystem 310. In some embodiments, there can be a corresponding dredger stream that manages the data written by the live stream on the backend by the backend data management subsystem 330.

[0164] In some embodiments, streams can include management interfaces that allow different configurations for different operations such as snapshots, statistics, and failover for different types of data. For example, streams for handling block data, state data, and transaction data can employ different configurations according to respective characteristics of the block data, state data, and transaction data. For example, a stream corresponding to blocks can be configured with tiering and / or compression functionality, but without compaction, snapshot, or table functionality.

[0165] In some embodiments, different types of data can be handled by corresponding custom or otherwise configured streams. For example, requests to write blocks can be handled by a stream corresponding to blocks.

[0166] In some embodiments, multiple streams can be combined into bundles to provide flexible implementations suitable for specific applications of distributed ledger systems and / or blockchain-based centralized ledger systems. The described techniques can support services in distributed ledger systems (e.g., blockchain networks) and / or blockchain-based centralized ledger systems. In some embodiments, the two systems can have different streams that are custom or otherwise configured according to needs of the two log-structured storage systems 300. For example, a distributed ledger system can have four types of data: transaction consensus log, blocks, state, and index. Accordingly, four streams can be configured to handle the four types of data, respectively. A blockchain-based centralized ledger system can have three types of data: transactions, blocks, and index, without state (or complex contract state). Accordingly, three streams can be configured to handle the three types of data, respectively.

[0167] In some embodiments, each stream can be configured to handle different types of data, respectively. For example, blocks, transaction consensus log, index do not need snapshots. Accordingly, streams for handling blocks, transaction consensus log, and index do not need to be configured with snapshot functionality. On the other hand, a state data stream can be configured with snapshot functionality. As another example, index data is relatively small, but it needs good performance, and does not need tiered storage. Long-term operations and large block data can need tiered storage, shared storage, and erasure coding.

[0168] In some embodiments, distributed ledger systems and blockchain-based centralized ledger systems can have different requirements for streams to perform operations such as tiering, compression, erasure coding, state snapshots, compaction, and data validation.

[0169] Table 1 provides configuration examples for different scenarios. As shown, "both" indicates that an operation on a particular type of data can be performed for both the distributed ledger system and the blockchain-based centralized ledger system. "DLS" indicates that an operation on a particular type of data can be performed for only the distributed ledger system. "Neither" indicates that an operation on a particular type of data can be performed for neither the distributed ledger system nor the blockchain-based centralized ledger system.

[0170] Table 1

[0171]

[0172]

[0173] For example, as shown in Table 1, tiered operations on transaction data and / or block data can be performed for both the distributed ledger system and the blockchain-based centralized ledger system. Tiered operations on current state data and / or historical state can be performed for only the distributed ledger system. Snapshot operations on transaction data can be performed for neither the distributed ledger system nor the blockchain-based centralized ledger system.

[0174] In some embodiments, the log-structured storage system employs a per-thread one-cycle one-queue and concurrent multi-thread fully asynchronous mechanism, thereby providing an efficient asynchronous mode and a convenient concurrent synchronous programming mode. In some embodiments, different streams can handle different types of data in parallel. For example, a stream configured for block data can write block data into a data log file allocated to store block data, while a stream configured for transaction data can read specific requested transaction data from a data log file that includes transaction data.

[0175] Figure 5 is a flowchart illustrating a process 500 for performing a write operation of a log-structured storage system according to embodiments herein. In some embodiments, some or all of the operations of the process 500 can be an example of a write procedure performed by a front-end I / O subsystem (e.g., the front-end I / O subsystem 310 of Figure 3 ). For convenience, the process 500 will be described as performed by the front-end I / O subsystem 310 of Figure 3 . However, the process 500 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in accordance with this description. For instance, a data processing and storage system (e.g., the log-structured storage system 300 of Figure 3 ) appropriately programmed can perform the process 500.

[0176] At 502, data log files (e.g., data log files 390, 362, 364, 366, 372, 374, or 376) are maintained in a data storage system (e.g., log-structured storage system 300). In some embodiments, the data log files can store data including transaction data, block data, state data, and self-describing metadata. For example, the data log files can store blockchain data generated by a blockchain network, including block data, transaction data, and / or state data. In some embodiments, the metadata in the data log files can include metadata describing data blocks, such as transaction hash values and sequence values, block hash values and block numbers, snapshot version numbers, cyclic redundancy check (CRC) codes, cryptographic information, and the like. In some embodiments, one data log file stores a single type of blockchain data, so multiple types of blockchain data are not mixed in a single data file. For example, the data storage system can maintain one or more of a data log file for transaction data, a data log file for block data, and a data log file for state data. In some embodiments, the data log files can be append-only files. In some embodiments, the data log files do not store index information. In some embodiments, the data log files can be stored in a multi-tier storage subsystem (e.g., multi-tier storage subsystem 320).

[0177] At 504, a front-end I / O subsystem (e.g., front-end I / O subsystem 310) of the data storage system receives a write request to write data to the data storage system. In some embodiments, the front-end I / O subsystem 310 can handle write operations, including modifications to data stored on the log-structured storage system 300. In some embodiments, modifications to data are handled by the front-end I / O subsystem 310 such that the original data is not overwritten. Instead, the modifications can be handled by adding or appending data to the data log files in a log form.

[0178] At 506, the front-end I / O subsystem 310 appends data to the data log files. In some embodiments, the data log files can be continuously updated based on data write or append operations. In some embodiments, the data log files can have a configurable maximum length of between 512 MB and 2 GB, or other sizes depending on the needs of the storage system or application.

[0179] At 508, the front-end I / O subsystem 310 determines whether a condition for generating a new data log file is satisfied. In some embodiments, the front-end I / O subsystem 310 can determine whether the data log file has reached a predetermined maximum length or size. If it is determined that the data log file has reached the predetermined maximum length or size, the front-end I / O subsystem 310 can determine that the condition for generating a new data log file is satisfied. If it is determined that the condition for generating a new data log file is satisfied, the process proceeds to step 510. If it is determined that the condition for generating a new data log file is not satisfied, the process returns to step 504.

[0180] At 510, if it is determined that the condition for generating a new data log file is satisfied, the front-end I / O subsystem 310 seals the data log file. In some embodiments, if it is determined that the condition for generating a new data log file is satisfied (e.g., the data log file has reached a maximum length or size), the front-end I / O subsystem 310 can set the data log file to a sealed, immutable, or read-only state.

[0181] At 512, the front-end I / O subsystem 310 generates a new data log file. In some embodiments, the new data log file can also be append-only and stored in the multi-tier storage subsystem 320.

[0182] In some embodiments, the front-end I / O subsystem 310 can determine a type of data (e.g., transaction data, block data, state data) requested to be written in the write request. In response to the determination, the front-end I / O subsystem 310 appends the data to a data log file corresponding to the type of data. In some embodiments, the front-end I / O subsystem 310 can use a respective processing engine corresponding to the type of data to perform some or all of the process 500.

[0183] For example, in response to determining that the data is transaction data, the front-end I / O subsystem 310 uses a processing engine designated for processing transaction data to append the data to a data log file for transaction data. In some embodiments, in response to determining that the data is block data, the front-end I / O subsystem 310 uses a processing engine designated for processing block data to append the data to a data log file for block data. In some embodiments, in response to determining that the data is state data, the front-end I / O subsystem 310 uses a processing engine designated for processing state data to append the data to a data log file for state data.

[0184] Figure 6is a flowchart illustrating a process 600 for generating an index related to a write operation of a log-structured storage system according to embodiments herein. In some embodiments, some or all of the operations of the process 600 can be examples of a write procedure performed by a front-end I / O subsystem (e.g., the front-end I / O subsystem 310 of the Figure 3 . For convenience, the process 600 will be described as performed by the front-end I / O subsystem 310 of the Figure 3 . However, the process 600 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in accordance with this document. For example, a data processing and storage system (e.g., the log-structured storage system 300 of the Figure 3 ) programmed appropriately can perform the process 600.

[0185] At 602, data is successfully written to a storage system (e.g., the log-structured storage system 300). In some embodiments, a front-end I / O subsystem (e.g., the front-end I / O subsystem 310) of the data storage system can write a data chunk to a data log file stored in a multi-tier storage subsystem (e.g., the multi-tier storage subsystem 320) of the data storage system.

[0186] At 604, the front-end I / O subsystem 310 generates an index indicating a physical storage location of the data in the log-structured storage system 300. In some embodiments, the index data can include: an index indicating a correspondence from a chunk hash value to a chunk number, an index indicating a correspondence from a chunk hash value to a storage location, an index indicating a correspondence from a transaction hash value to a transaction, and an index indicating a correspondence from a receipt hash value to a receipt. In some embodiments, the index data for a blockchain-based centralized ledger system can include an index indicating a correspondence from a sequence to a transaction storage location or an index indicating a correspondence from a time sequence to a transaction hash value.

[0187] At 606, the front-end I / O subsystem saves the index in an incremental index map (e.g., the incremental index map 312) in a memory (e.g., the memory 315) of the front-end I / O subsystem 310. In some embodiments, the front-end I / O subsystem 310 can include a plurality of in-memory index maps stored in the memory 315. In some embodiments, the in-memory index maps can be divided into a read-only base index map 316 and a read-write incremental index map 312. In some embodiments, the base index map 316 can store indexes for cold data (e.g., old data and / or data that is less frequently accessed), while the incremental index map 312 can store indexes for newly written data.

[0188] At 608, the front-end I / O subsystem 310 determines whether a triggering event has occurred. Triggering events may include one or more events that cause the current incremental index map to be sealed and a new incremental index map to be generated. Triggering events may include, for example, the size of the current incremental index map reaching a threshold, the storage utilization of memory 315 reaching a threshold, or a specified time arriving (e.g., the log structure storage system 300 may periodically seal incremental index maps). If it is determined that a triggering event has occurred, the process proceeds to step 610. If it is determined that no triggering event has occurred, the process returns to step 602.

[0189] At 610, if a triggering event is determined to have occurred, the front-end I / O subsystem 310 sets the incremental index mapping 312 to immutable. In some embodiments, the front-end I / O subsystem may set the incremental index mappings 312 in memory 315 to immutable index mappings 314, flush them to an index log file (e.g., index log file 380), and create new incremental index mappings 312 to receive indexes generated based on new write requests.

[0190] At 612, an index log file 380 is maintained in storage system 300. In some embodiments, the indexes of newly written transaction data and block data can be added to index maps 312 and 316 and index log file 390, but the indexes of existing transaction data and block data can be left unmodified. In some embodiments, index log file 390 can be stored together with data log files in multi-tier storage subsystem 320.

[0191] At 614, the front-end I / O subsystem 310, for example, flushes the incremental index map 312 to the index log file 380 to release the memory used by the incremental index map 312. In some embodiments, the front-end I / O subsystem 310 may create a new incremental index map 312 to receive an index generated based on a new request. In some embodiments, the front-end I / O subsystem 310 may combine the incremental index map 312 and the basic index map 316 to generate a new basic index map 316, and flush the generated basic index map 316 to the index log file 380.

[0192] In some embodiments, at 616, the front-end I / O subsystem 310 stores an index of hot data in a memory cache (e.g., memory cache 313). For example, if specific data is identified as hot data with a high probability of being accessed frequently, the index of the data can be stored in a memory cache to improve read speed.

[0193] At 618, the front-end I / O subsystem 310 determines whether a condition for generating a new index log file 380 is satisfied. In some embodiments, the condition for generating a new index log file can include a maximum length or size of the index log file 380. In some embodiments, the condition for generating a new index log file 380 can include a number of batches of write operations performed by the front-end I / O subsystem. For example, in some embodiments, the index log file 380 can be updated by batching index write operations. In some embodiments, when the index log file 380 has been updated for a particular number of batches of write operations, the index log file 380 can be sealed or set to read-only, and a new index log file 380 can be created to write new data. If it is determined that the condition for generating a new index log file 380 is satisfied, processing proceeds to step 620.

[0194] At 620, if it is determined that the condition for generating a new index log file 380 is satisfied, the front-end I / O subsystem 310 seals the index log file. For example, when the index log file 380 has reached a maximum length or size, or has been updated for a particular number of batches of write operations, the index log file 380 can be sealed or set to read-only.

[0195] At 622, the front-end I / O subsystem 310 generates a new index log file 380 after sealing the old index log file 380 to store subsequent index data.

[0196] Figure 7 is a flowchart illustrating a process 700 for performing a read operation of a log-structured storage system according to embodiments herein. In some embodiments, some or all of the operations of the process 700 can be examples of a read procedure performed by a front-end I / O subsystem (e.g., the front-end I / O subsystem 310 of the system 100, Figure 3 of FIG. 1). For convenience, the process 700 will be described as performed by the front-end I / O subsystem 310. However, the process 700 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in accordance with this description. For instance, a data processing and storage system (e.g., the log-structured storage system 300 of FIG. 1) appropriately programmed can perform the process 700. Figure 3

[0197] At 702, a front-end I / O subsystem (e.g., the front-end I / O subsystem 310) of a storage system (e.g., the log-structured storage system 300) receives a read request to read data from the storage system.

[0198] ​At 704, the front-end I / O subsystem 310 searches an incremental index map (e.g., the incremental index map 312) in the memory (e.g., the memory 315) of the front-end I / O subsystem 310 for an index corresponding to the data. In some embodiments, the index corresponding to the data can include physical location information of the data. In some embodiments, the memory 315 of the front-end I / O subsystem 310 can store a plurality of memory index maps including the read-only base index map 316 and the read-write incremental index map 312.

[0199] At 706, the front-end I / O subsystem 310 determines whether the index corresponding to the data is found in the incremental index map 312. If the index corresponding to the data is found in the incremental index map 312, the process proceeds to step 708, in which the front-end I / O subsystem 310 can locate the data based on the physical location indicated by the index. If the index corresponding to the data is not found in the incremental index map 312, the process proceeds to step 710.

[0200] At 710, if it is determined that the index corresponding to the data is not found in the incremental index map 312, the front-end I / O subsystem 310 searches the base index map 316 in the memory 315 for the index corresponding to the data.

[0201] At 712, the front-end I / O subsystem 310 determines whether the index corresponding to the data is found in the base index map 316. If it is determined that the index corresponding to the data is found in the base index map 316, the process proceeds to step 714, in which the front-end I / O subsystem 310 can locate the data based on the physical location information indicated by the index. If the index corresponding to the data is not found in the base index map 316, the process proceeds to step 716.

[0202] At 716, if it is determined that the index corresponding to the data is not found in the base index map 316, the front-end I / O subsystem 310 searches the index log file (e.g., the index log file 380) in the disk storage device for the index corresponding to the data. For example, the front-end I / O subsystem 310 can search the index log file 380 stored in the multi-tier storage subsystem (e.g., the multi-tier storage subsystem 320) of the storage system 300 for the index corresponding to the data.

[0203] In some embodiments, the front-end I / O subsystem 310 can determine a type of data (e.g., transaction data, block data, state data) that is requested to be read in the read request. In response to the determination, the front-end I / O subsystem 310 can use a respective processing engine corresponding to the type of data to perform some or all of the process 700.

[0204] Figure 8 is a flowchart illustrating a process 800 for improving read operations of a log-structured storage system according to embodiments herein. In some embodiments, some or all of the operations of the process 800 can be examples of I / O operations performed by a log-structured storage system (e.g., the log-structured storage system 300 of Figure 3 For convenience, the process 800 will be described as being performed by the log-structured storage system 300. However, the process 800 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in accordance with this document. For example, a data processing and storage system programmed appropriately can perform the process 800.

[0205] At 802, a storage system (e.g., the log-structured storage system 300 or the tiered storage system 400) maintains a plurality of tiers of storage devices (e.g., the storage devices 350, 360, and 370 of the multi-tier storage subsystem 320) and one or more tiers of cache (e.g., the memory cache 313 and the block cache 317). In some embodiments, the plurality of tiers of storage devices are tiered in a tiered structure based on one or more performance characteristics (e.g., access speed, access bandwidth, or access latency). For example, the plurality of tiers of storage devices can include a plurality of storage devices having different access speeds and storing data having different characteristics. For example, a first tier of storage devices can store blockchain data that is more frequently accessed than blockchain data stored in a second tier of storage devices.

[0206] At 804, a determination is made that a data object in a data log file (e.g., the data log files 362, 364, 366, 372, 374, 376, 390) stored in a lower tier of storage devices (e.g., the storage devices 350, 360, and 370) is an active data object, e.g., by a front-end I / O subsystem (e.g., the front-end I / O subsystem 310) or a back-end data management system (e.g., the back-end data management subsystem 330 of the storage system 300). In some embodiments, a data object can include transaction data, block data, and state data. In some embodiments, a data object can be determined to be an active data object based on one or more activity or hotness policies, e.g., if the data object has been accessed a certain number of times recently (e.g., within a predetermined time window) or if the data object has been identified as having a certain priority.

[0207] At 806, the data object is written to a cache (e.g., the memory cache 313 and the block cache 317). For example, the front-end I / O subsystem 310 can write the data object to the memory cache 313 or the disk block cache 317 of a high-speed storage medium.

[0208] At 808, an index indicating the physical storage location of the data object in the cache is generated. In some embodiments, the LSM method can be used to organize the index data in memory 315.

[0209] At 810, an incremental index map (e.g., incremental index map 312) can be saved to memory 315. In some embodiments, memory 315 may maintain multiple memory index maps, including a read-only base index map 316 and a read-write incremental index map 312. In some embodiments, incremental index map 312 may be configured to store an index of data that is accessed and / or updated more frequently than the index stored in base index map 316.

[0210] In 812, the front-end I / O subsystem 310 receives read requests for data objects.

[0211] At 814, the front-end I / O subsystem 310 searches for an index corresponding to the data object in the incremental index map 312 in memory 315. In some embodiments, the front-end I / O subsystem 310 may first search the incremental index map 312. If no index is found in the incremental index map 312, the front-end I / O subsystem 310 may subsequently search for an index corresponding to the data in the base index map 316.

[0212] At 816, the front-end I / O subsystem 310 returns a data object from the cache, which provides faster access to the requested data object compared to situations where it needs to be retrieved from a lower level of storage devices in the next cache level, main memory, or multi-level storage subsystem 320. For example, if the front-end I / O subsystem 310 successfully identifies an index corresponding to data in incremental index mapping 312 or basic index mapping 316, the front-end I / O subsystem 310 can use that index to identify the physical location of the data in the cache and retrieve the data from the cache.

[0213] In some embodiments, the front-end I / O subsystem 310 may determine the type of data requested to be read in a read request (e.g., transaction data, block data, state data). In response to this determination, the front-end I / O subsystem 310 may use the appropriate processing engine corresponding to the data type to perform some or all of the processing 800.

[0214] Figure 9 This is a flowchart illustrating a process 900 for managing data log files stored in a log-structured storage system according to embodiments of the present invention. In some embodiments, some or all of the operations of process 900 may be performed by a backend data management system of the log-structured storage system (e.g., Figure 3of the log-structured storage system 300) performs. For convenience, the process 900 will be described as being performed by the backend data management subsystem 330. However, the process 900 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in light of this description. For example, a data processing and storage system (e.g., the log-structured storage system 300) appropriately programmed can perform the process 900. Figure 3 The log-structured storage system 300) can perform the process 900.

[0215] At 902, the backend data management system (e.g., the backend data management subsystem 330) determines information of data log files (e.g., data log files 390, 362, 364, 366, 372, 374, and 376) stored in storage devices (e.g., storage devices 350, 360, and 370) from an index log file (e.g., the index log file 380). In some embodiments, the information of the data log files in the storage devices can include one or more of an activity (e.g., access frequency), size, garbage rate, or fragmentation level of the data log files.

[0216] At 904, the backend data management system 330 determines information of the storage devices. In some embodiments, the information of the storage devices can include one or more of a usage, a garbage rate, a fragmentation level, or an input / output (I / O) request of the storage devices.

[0217] At 906, the backend data management system 330 determines whether the data log files require rewrite placement. In some embodiments, the backend data management subsystem 330 can determine the rewrite placement based on the information of the data log files stored in the storage devices and / or the information of the storage devices. In some embodiments, the rewrite placement can include at least one of tiering, compression, erasure coding, state snapshot, compaction, or verification. If it is determined that the data log files require rewrite placement, the process proceeds to step 908. If it is determined that the data log files do not require rewrite placement, the process returns to step 902.

[0218] At 908, the backend data management system 330 reads the data log files from a source location and rewrites the data log files to a target location if it is determined that the data log files require rewrite placement.

[0219] At 910, the backend data management system 330 seals the data log files in the target location. For example, the backend data management system 330 can set the data log files to a sealed state or read-only after the rewrite placement is completed.

[0220] At 912, the back-end data management system 330 generates a target index log file corresponding to the data log files in the target location. In some embodiments, the target index log file can include a list of the data log files that can be safely deleted, and / or a list of the data log files referenced by the target index log file.

[0221] At 914, the back-end data management system 330 seals the target index log file. For example, the back-end data management system 330 can set the target index log file to be immutable or read-only.

[0222] At 916, the target index log file is imported into a readable index map in memory. For example, the target index log file can be imported into an incremental index map or a base index map, such that the data log files in the target location can be addressed or read.

[0223] Figure 10 is a flowchart illustrating a process 1000 for performing data migration in a log-structured storage system according to embodiments herein. In some embodiments, some or all of the operations of the process 1000 can be an example of a tiering / migration program executed by a back-end data management system of a log-structured storage system (e.g., the back-end data management subsystem 330 of the log-structured storage system 300 of Figure 3 . For convenience, the process 1000 will be described as being performed by the back-end data management subsystem 330. However, the process 1000 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately according to this description. For example, a data processing and storage system (e.g., the log-structured storage system 300 of Figure 3 ) appropriately programmed can perform the process 1000.

[0224] At 1002, the back-end data management system (e.g., the back-end data management subsystem 330) identifies one or more characteristics of the data log files (e.g., the data log files 390, 362, 364, 366, 372, 374, and 376) and one or more characteristics of the storage devices (e.g., the storage devices 350, 360, and 370). In some embodiments, the one or more characteristics of the data log files can include a data type (e.g., block data, state data, and transaction data), a creation time, a data size, an activity, a garbage rate, or a fragmentation level, etc. of the data log files. In some embodiments, the one or more characteristics of the storage devices can include an access speed, an access bandwidth, an access latency, a usage rate, a garbage rate, a fragmentation level, or an input / output (I / O) request of the storage devices.

[0225] At 1004, the backend data management system 330 determines a migration metric for the data log file based on the characteristics. In some embodiments, the backend data management system 330 can assign a score to the data log file and generate a migration task according to the score ranking and a predetermined migration policy. In some embodiments, the data log file can be scored or assigned a score according to a scoring formula that takes into account media tier, access frequency, raw data creation time, and migration cost, among others.

[0226] At 1006, the backend data management system 330 determines whether to migrate the data log file. For example, the data log file can be scored according to a predetermined scoring scheme. If the score of the data log file exceeds a predetermined threshold, the backend data management system 330 can determine that the data log file needs to be migrated. If it is determined that the data log file needs to be migrated, the process proceeds to step 1008. If it is determined that the data log file does not need to be migrated, the process returns to step 1002.

[0227] At 1008, if it is determined that the data log file needs to be migrated, the backend data management system 330 migrates the data log file from the source location to the target storage device. In some embodiments, the data log files in the high-speed storage device can be scored according to a predetermined scoring scheme and migrated to the low-speed storage device based on the score (e.g., after the scores of the data log files are sorted or ranked) to free up storage space. In some embodiments, hot data in the data log files stored in the low-speed storage device can be first saved in the disk cache, and if the score of the data log file reaches a predetermined threshold, it can be migrated to the high-speed storage device.

[0228] Figure 11 is a flowchart illustrating a process 1100 for performing data flow control in a log-structured storage system according to embodiments herein. In some embodiments, some or all of the operations of the process 1100 can be examples of a flow control / optimization procedure performed by a log-structured storage system (e.g., the log-structured storage system 300 of Figure 3 For convenience, the process 1100 will be described as being performed by a log-structured storage system. However, the process 1100 can be performed by a system of one or more computers located in one or more locations, programmed appropriately in accordance with this document. For example, a data processing and storage system programmed appropriately can perform the process 1100.

[0229] At 1102, a storage system (e.g., log-structured storage system 300) maintains a multi-tiered storage (e.g., storage 350, 360, and 370). In some embodiments, the multi-tiered storage is tiered in a hierarchy based on one or more performance characteristics (e.g., access speed, access bandwidth, or access latency). For example, the multi-tiered storage can include multiple storage devices with different access speeds, and can store data with corresponding characteristics (e.g., a first tier storage can store blockchain data that is more frequently accessed than blockchain data stored in a second tier storage).

[0230] In some embodiments, log-structured storage system 300 can assign flow control policies to the multi-tiered storage. For example, log-structured storage system 300 can assign a first flow control policy to a first tier storage and a second flow control policy to a second tier storage based on one or more characteristics (e.g., access speed, access bandwidth, access latency, usage, garbage rate, fragmentation level) of the first tier storage and the second tier storage. In some embodiments, the first flow control policy can include one or more of a first speed to write data to the first tier storage, or one or more first thresholds to adjust the first speed to write data to the first tier storage, and the second flow control policy can include one or more of a second speed to write data to the first tier storage, or one or more second thresholds to adjust the second speed to write data to the second tier storage.

[0231] At 1104, log-structured storage system 300 receives a write request for ledger data. In some embodiments, the ledger data can include blockchain data such as transaction data, block data, and state data.

[0232] At 1106, the log-structured storage system 300 identifies a type of the ledger data. For example, the backend data management subsystem 330 can determine whether the ledger data is transaction data, block data, or state data. At 1108, the log-structured storage system 300 determines whether the ledger data is block data. In some embodiments, block data has a larger size than other types of blockchain data (e.g., transaction data, state data, or index data) and can have a greater impact on the throughput of I / O operations of the log-structured storage system 300. If it is determined that the ledger data is block data, then processing proceeds to step 1110, where the backend data management subsystem 330 writes the data directly to a second tier storage device (e.g., storage device 360) and skips the first tier storage device, for example, to save the cost of performing migration later. In some embodiments, the second tier storage device is at a lower tier than the first tier storage device. For example, the access speed of the second tier storage device can be slower than the access speed of the first tier storage device. In some embodiments, the cost of the second tier storage device can be lower than the cost of the first tier storage device. In some embodiments, the storage space of the second tier storage device can be larger than the storage space of the first tier storage device. If it is determined that the ledger data is not block data, then processing proceeds to step 1112.

[0233] At 1112, if it is determined that the ledger data is not block data, the log-structured storage system 300 determines a usage rate of the first tier storage device.

[0234] At 1114, the log-structured storage system 300 determines whether the usage rate reaches or exceeds a predetermined threshold. In some embodiments, the predetermined threshold is used to determine whether the first tier storage device is substantially full. For example, if it is determined that the usage rate reaches or exceeds the threshold (e.g., 85%), the log-structured storage system 300 can determine that the first tier storage device is substantially full. If it is determined that the usage rate reaches or exceeds the predetermined threshold, then processing proceeds to step 1116, where the data is written to the second tier storage device. If it is determined that the usage rate is below the predetermined threshold, then processing proceeds to step 1118.

[0235] At 1118, the log-structured storage system 300 writes the data to the first tier storage device upon determining that the usage rate is below the predetermined threshold.

[0236] In some embodiments, the log-structured storage system 300 can adjust the speed at which data is written to the first tier storage device based on the usage of the first tier storage device. For example, if it is determined that the usage of the first tier storage device reaches or exceeds a first predetermined threshold (e.g., 65%), the log-structured storage system 300 can decrease the speed at which data is written to the first tier storage device. In some embodiments, the log-structured storage system 300 can decrease the speed at which data is written to the first tier storage device based on the usage of the first tier storage device. In some embodiments, the log-structured storage system 300 can continuously decrease the speed at which data is written to the first tier storage device as the usage of the first tier storage device increases. For example, when the usage of the first tier storage device is a first value (e.g., 70%), the log-structured storage system 300 can decrease the speed at which data is written into the first tier storage device to a first rate (e.g., 500 MB / s), and when the usage of the first tier storage device is a second value (e.g., 75%) that is greater than the first value, the speed at which data is written to the first tier storage device is decreased to a second rate (e.g., 400 MB / s) that is lower than the first rate.

[0237] In some embodiments, the log-structured storage system 300 can increase the speed at which data is written to the first tier storage device if it is determined that the usage of the first tier storage device is below a second predetermined threshold (e.g., 35%). In some embodiments, the log-structured storage system 300 can increase the speed at which data is written to the first tier storage device based on the usage of the first tier storage device. In some embodiments, the log-structured storage system 300 can continuously increase the speed at which data is written to the first tier storage device as the usage of the first tier storage device decreases. For example, when the usage of the first tier storage device is a third value (e.g., 30%), the log-structured storage system 300 can increase the speed at which data is written into the first tier storage device to a third rate (e.g., 550 MB / s), and when the usage of the first tier storage device is a fourth value (e.g., 20%) that is less than the third value, the speed at which data is written to the first tier storage device is increased to a fourth rate (e.g., 600 MB / s) that is higher than the third rate.

[0238] Figure 12 FIG. 12 is a flowchart illustrating a process 1200 that can be performed in accordance with embodiments herein. For convenience, the process 1200 will be described as being performed by the log-structured storage system 300 of Figure 3 FIG. 1. However, the process 1200 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in accordance with this document. For example, a data processing and storage system, appropriately programmed, can perform the process 1200.

[0239] At 1202, a storage system (e.g., log-structured storage system 300) receives a configuration of a plurality of processing engines. In some embodiments, the configuration can configure a respective processing engine type for processing a data type of the distributed ledger system according to characteristics of each of a plurality of data types of the distributed ledger system, e.g., Table 1. In some embodiments, storage system 300 can include a processing engine type designated for processing block data; a processing engine type designated for processing transaction data; a processing engine type designated for processing state data; a processing engine type designated for processing index data.

[0240] In some embodiments, the state data can include current state data and historical state data, and storage system 300 can include a processing engine type designated for processing current state data and a processing engine type designated for processing historical state data.

[0241] At 1204, storage system 300 receives a processing request for data of the distributed ledger system. In some embodiments, the data type of the distributed ledger system can include block data, transaction data, state data, and index data.

[0242] In some embodiments, storage system 300 can receive an I / O request for data of the distributed ledger system. In some embodiments, a respective processing engine type designated for processing a data type of the distributed ledger system can include a respective I / O processing engine type designated for performing a read or write operation on a data type of the distributed ledger system.

[0243] In some embodiments, storage system 300 can receive a data management request for data of the distributed ledger system. In some embodiments, a respective processing engine type designated for processing a data type can include a respective data management processing engine type designated for performing a data management operation on a data type in the storage system. In some embodiments, the management operation includes one or more of tiering, compaction, compression, erasure coding, or snapshotting.

[0244] At 1206, storage system 300 determines that the data belongs to a data type of a plurality of data types of the distributed ledger system. In some embodiments, the data type can be block data or transaction data. In some embodiments, the data type can be state data.

[0245] At 1208, the storage system 300 applies a processing engine type designated for processing the one type of data. In some embodiments, the processing engine type designated for processing the one type of data can be configured to have functionality including tiering, erasure coding, and compression of block data or transaction data in the storage system 300. In some embodiments, the processing engine type designated for processing the one type of data can be configured to have functionality including snapshotting and compression of state data in the storage system 300.

[0246] Figure 13 FIG. 13 is a flow diagram illustrating a process 1300 that can be performed in accordance with embodiments herein. For convenience, the process 1300 will be described as being performed by a log-structured storage system 300. Figure 3 However, the process 1300 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in accordance with the teachings herein. For example, a data processing and storage system, appropriately programmed, can perform the process 1300.

[0247] At 1302, a storage system (e.g., the log-structured storage system 300) receives a configuration of a plurality of processing engines. In some embodiments, the configuration can configure a respective processing engine type for processing each of a plurality of data types of a blockchain-based centralized ledger system in accordance with characteristics of the data types, e.g., Table 1.

[0248] In some embodiments, the storage system 300 can include a processing engine type designated for processing block data; a processing engine type designated for processing transaction data; a processing engine type designated for processing index data.

[0249] At 1304, the storage system 300 receives a processing request for data of a blockchain-based centralized ledger system. In some embodiments, data types of the blockchain-based centralized ledger system can include block data, transaction data, and index data.

[0250] In some embodiments, the storage system 300 can receive an I / O request for data of the blockchain-based centralized ledger system. In some embodiments, a respective processing engine type designated for processing one type of data of the blockchain-based centralized ledger system can include a respective I / O processing engine type designated for performing a read or write operation on the one type of data of the blockchain-based centralized ledger system. For example, the data of the blockchain-based centralized ledger system is analyzed in accordance with some or all of the operations of the processes 500, 600, 700, 1100, and 1400.

[0251] In some embodiments, storage system 300 can receive a data management request for data of a blockchain-based centralized ledger system. In some embodiments, a respective processing engine type designated for processing the one type of data can include a respective data management processing engine type designated for performing data management operations on the one type of data in the storage system. In some embodiments, a management operation can include one or more of tiering, compaction, compression, erasure coding, or snapshotting.

[0252] At 1306, storage system 300 determines that the data belongs to one type of data of a blockchain-based centralized ledger system. In some embodiments, the one type of data can be block data or transaction data.

[0253] At 1308, storage system 300 applies a processing engine type designated for processing the one type of data according to characteristics of the one type of data. In some embodiments, the processing engine type designated for processing the one type of data can be configured to have functionalities including tiering, erasure coding, and compression on block data or transaction data in the storage system. In some embodiments, storage system 300 applies the processing engine type designated for processing the one type of data according to some or all operations of processes 800, 900, 1000, and 1400.

[0254] Figure 14 FIG. 14 is a flow diagram illustrating a process 1400 that can be performed in accordance with embodiments herein. For convenience, the process 1400 will be described as being performed by a log-structured storage system 300. Figure 3 However, the process 1400 can be performed by a system of one or more computers located in one or more locations, and programmed appropriately in accordance with this document. For example, a data processing and storage system, appropriately programmed, can perform the process 1400.

[0255] At 1402, a storage system (e.g., log-structured storage system 300) receives a plurality of write requests for storing a respective plurality of blockchain data in the storage system. In some embodiments, each of the plurality of blockchain data can include a value of one or more of a block, a transaction, or a state of a blockchain network and a key corresponding to the value. In some embodiments, the key can include a hash value corresponding to the value.

[0256] At 1404, the storage system 300 appends the plurality of blockchain data to a data log file (e.g., data log files 390, 362, 364, 366, 372, 374, and 376) according to a time order of the plurality of blockchain data. For example, blockchain data received later will be appended to the data received earlier that is already stored in the data log file. In some embodiments, the data log file can be an append-only file. In some embodiments, the data log file can be stored in a first tier storage device (e.g., storage device 350) in a multi-tier storage subsystem (e.g., multi-tier storage subsystem 320) of the log-structured storage system 300 that includes a plurality of tier storage devices, and the first tier storage device has the highest access speed among the plurality of tier storage devices.

[0257] At 1406, the log-structured storage system 300 is restricted from sorting the plurality of blockchain data in the data log file according to any other metric, such as according to the corresponding keys (e.g., in KVPs) of the values in the plurality of blockchain data. In some embodiments, unlike existing storage systems that will rearrange the plurality of blockchain data in the data log file according to the corresponding keys of the values in the plurality of blockchain data, the plurality of blockchain data in the data log file of the log-structured storage system 300 is arranged according to the time that the log-structured storage system 300 generates or receives the plurality of blockchain data. At 1408, the log-structured storage system 300 generates an index indicating the corresponding physical storage locations of the plurality of blockchain data in the data log file, for example, according to the corresponding operations of the process 600.

[0258] At 1410, the log-structured storage system 300 writes the index into the first tier storage device, for example, according to the corresponding operations of the process 600.

[0259] At 1412, the log-structured storage system 300 determines the corresponding migration priority, score, or metric of the plurality of blockchain data, for example, according to the corresponding operations of the process 1000. In some embodiments, the log-structured storage system 300 determines the corresponding migration priority according to the time order of the plurality of blockchain data. In some embodiments, the migration priority of older blockchain data can be higher than the migration priority of newer blockchain data.

[0260] At 1414, the log-structured storage system 300 migrates the plurality of blockchain data stored in the first tier storage device to a second tier storage device (e.g., storage device 360) according to the corresponding migration priority. In some embodiments, the access speed of the second tier storage device can be lower than the access speed of the first tier storage device.

[0261] Figure 15 An example of modules of an apparatus 1500 according to embodiments herein is depicted. The apparatus 1500 can be a storage system (e.g.,Figure 3 An example of embodiments of the apparatus 1500 can correspond to the above-mentioned embodiments. The apparatus 1500 comprises: a receiving module 1502 configured to receive a processing request for data of a centralized ledger system, wherein a plurality of data types of the centralized ledger system comprises: block data, transaction data, and index data; a determining module 1504 configured to determine that the data belongs to one data type of the plurality of data types of the centralized ledger system; and an applying module 1506 configured to apply a processing engine type designated for processing the one data type according to a characteristic of the one data type.

[0262] In an optional embodiment, the apparatus 1500 further comprises a receiving sub-module configured to receive a configuration of a plurality of processing engines, wherein the configuration configures a corresponding processing engine type for processing each data type of the centralized ledger system according to a characteristic of the data type.

[0263] In an optional embodiment, the apparatus 1500 can comprise: a processing engine type designated for processing block data; a processing engine type designated for processing transaction data; and a processing engine type designated for processing index data.

[0264] In an optional embodiment, the apparatus 1500 further comprises a receiving sub-module configured to receive an input / output (I / O) request for data of the centralized ledger system. The corresponding processing engine type designated for processing the one data type of the centralized ledger system can comprise a corresponding I / O processing engine type designated for performing a read or write operation on the one data type of the centralized ledger system.

[0265] In an optional embodiment, the apparatus 1500 further comprises: a receiving sub-module configured to receive a data management request for data of the centralized ledger system. The corresponding processing engine type designated for processing the one data type can comprise a corresponding data management processing engine type designated for performing a data management operation on the one data type in the storage system. The management operation comprises one or more of tiering, compaction, compression, erasure coding, or snapshot.

[0266] In an optional embodiment, the one data type is block data or transaction data, and the processing engine type designated for processing the one data type is configured with a function comprising tiering, erasure coding, and compression on the block data or transaction data in the storage system.

[0267] The systems, apparatuses, modules, or units shown in the previous implementations can be realized by using a computer chip or entity, or can be realized by using a product with specific functions. A typical implementation device is a computer, which can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email receiving and sending device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0268] For the implementation process of the functions and roles of each unit in the device, reference can be made to the implementation process of the corresponding steps in the previous method. For the sake of simplicity, the details are omitted here.

[0269] Since the device embodiments basically correspond to the method embodiments, for the relevant parts, reference can be made to the relevant description in the method embodiments. The foregoing device embodiments are merely examples. The modules described as separate parts can or can not be physically separated, and the parts shown as modules can or can not be physical modules, can be located in one location, or can be distributed on multiple network modules. Some or all modules can be selected based on actual needs to achieve the goal of the scheme herein. Those of ordinary skill in the art can understand and implement the embodiments of the present application without creative effort.

[0270] Again referring to Figure 15 , it can be interpreted to show the internal functional modules and structures of a data processing and storage device. The data processing and storage device can be an example of a log-structured storage system (e.g., the log-structured storage system 300 of Figure 3 In essence, the execution subject can essentially be an electronic device comprising the following: one or more processors; one or more computer-readable memories configured to store executable instructions of the one or more processors. In some embodiments, the one or more computer-readable memories are coupled to the one or more processors and have program instructions stored thereon that are executable by the one or more processors to perform the algorithms, methods, functions, processes, flows, and procedures described herein.

[0271] Embodiments of the described subject matter can include one or more of the following features alone or in combination. In a first embodiment, a method includes: a storage system receiving a processing request for data of a centralized ledger system, wherein a plurality of data types of the centralized ledger system includes block data, transaction data, and index data; the storage system determining that the data belongs to one data type of the plurality of data types of the centralized ledger system. The storage system applies a processing engine type designated for processing the one data type according to a characteristic of the one data type.

[0272] The foregoing and other described embodiments can each optionally include one or more of the following features:

[0273] A first feature, combinable with any of the previous or following features, provides that the method further includes the storage system receiving a configuration of a plurality of processing engines, wherein the configuration configures a corresponding processing engine type for processing each data type of the centralized ledger system according to characteristics of the data type.

[0274] A second feature, combinable with any of the previous or following features, provides that the storage system further includes a processing engine type designated for processing block data, a processing engine type designated for processing transaction data, and a processing engine type designated for processing index data.

[0275] A third feature, combinable with any of the previous or following features, provides that the storage system receiving a processing request for data of the centralized ledger system includes the storage system receiving an input / output (I / O) request for data of the centralized ledger system, and the corresponding processing engine type designated for processing the one data type of the centralized ledger system can include a corresponding input / output (I / O) processing engine type designated for performing a read or write operation on the one data type of the centralized ledger system.

[0276] A fourth feature, combinable with any of the previous or following features, provides that the storage system receiving a processing request for data of the centralized ledger system includes the storage system receiving a data management request for data of the centralized ledger system, and the corresponding processing engine type designated for processing the one data type includes a corresponding data management processing engine type designated for performing a data management operation on the one data type in the storage system. The management operation includes one or more of tiering, compaction, compression, erasure coding, or snapshotting.

[0277] A fifth feature, combinable with any of the previous features, provides that the one data type is block data or transaction data, and the processing engine type designated for processing the one data type is configured with functionality including tiering, erasure coding, and compression of the block data or transaction data in the storage system.

[0278] Embodiments of the subjects, actions, and operations described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or combinations thereof. Embodiments of the subjects described herein may be implemented as one or more computer programs, such as one or more computer program instruction modules encoded on a computer program carrier for operations performed by or controlling data processing by a data processing apparatus. For example, a computer program carrier may include one or more computer-readable storage media on which instructions are encoded or stored. The carrier may be a tangible, non-transitory computer-readable medium, such as a magnetic disk, magneto-optical disk or optical disk, a solid-state drive, random access memory (RAM), read-only memory (ROM), or other media types. Optionally or additionally, the carrier may be an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus. A computer storage medium may be, or can be in part, a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. A computer storage medium is not a propagation signal.

[0279] A computer program may also be referred to or described as a program, software, software application, app, module, software module, engine, script, or code, and may be written in any form of programming language, including compiled or deductive languages, declarative or procedural languages; it may be configured in any form, including as a standalone program, or as a module, component, engine, subroutine, or other unit suitable for execution in a computing environment, which may include one or more computers interconnected by a data communication network in one or more locations.

[0280] A computer program may, but is not required to, correspond to a file in a file system. A computer program may be stored as: a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document; a single file dedicated to the program in question; or multiple coordination files, such as multiple files that store one or more modules, subroutines, or code sections.

[0281] For example, processors used to execute computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, a processor receives instructions for executing a computer program, and also receives data from a non-transitory computer-readable medium coupled to the processor.

[0282] The term“data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The data processing apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) or GPU (graphics processing unit), as well as a general purpose logic circuitry. The apparatus can also include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0283] The processes and logic flows described in this specification can be performed by one or more computers or processors executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0284] Computers suitable for the execution of a computer program can be based on general and / or special purpose microprocessors, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The elements of a computer can include a central processing unit for performing instructions and one or more memory devices for storing instructions and data. A central processing unit and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0285] Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, one or more mass storage devices for storing data. Mass storage devices can be, for example, magnetic, magneto-optical disks, or optical disks, solid state drives, or any other type of non-transitory computer-readable media. A computer need not have such devices. Thus, a computer can be coupled to one or more storage devices, which need not be local or even remote. For example, a computer can include one or more local storage devices, which can be integral to the computer, or a computer can be coupled to one or more remote storage devices, which can be coupled to a cloud network. Further, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable memory device such as a universal serial bus (USB) flash drive, to name just a few.

[0286] Components can be "coupled" to each other if they are in electronic communication or

[0287] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a LCD (liquid crystal display) monitor, for displaying information to the user and a input device, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball, or a touchpad, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device of the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser, or by interacting with an app running on a user device, e.g., a smart phone or an electronic tablet computer. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smart phone that is running a messaging app, and receiving responsive messages from the user.

[0288] The term "configured to" is used herein with respect to systems, apparatuses, and computer program components. With respect to a system of one or more computers configured to perform particular operations or actions, that the system has installed on it software, firmware, hardware, or a combination thereof that in operation causes the system to perform the operations or actions. With respect to one or more computer programs configured to perform particular operations or actions, that the program includes instructions that when executed by data processing apparatus cause the apparatus to perform the operations or actions. With respect to a special purpose logic circuit configured to perform particular operations or actions, that the circuit has electronic logic that performs the operations or actions.

[0289] Although numerous details of implementation are included herein, these should not be construed as limiting the scope of the claims to the particular forms disclosed, but rather should be understood to include all specific embodiments falling within the scope of the claims. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any appropriate subcombination. Moreover, although features can be described above as acting in particular combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0290] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring or implying that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the division of various system modules and components in the embodiments described above should not be understood as requiring such division in all embodiments, but rather should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0291] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures need not be performed in the particular order described and illustrated, or in sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method, comprising: The storage system receives configurations from multiple processing engines. The configuration specifies the corresponding processing engine type for processing each data type in the centralized ledger system based on its characteristics. The centralized ledger system includes various data types such as block data, transaction data, and index data. These data types have different storage requirements, and the processing engine type configured for each data type is configured according to the storage requirements of the corresponding data type. The storage system receives processing requests for data from the centralized ledger system; The storage system determines that the data belongs to one of the multiple data types in the centralized ledger system; The storage system applies a processing engine type designated for processing the data type based on the characteristics of the data type.

2. The computer-implemented method as described in claim 1, wherein the centralized ledger system includes a central trusted authority; the central trusted authority stores data in blocks of a blockchain data structure, and the data type of the centralized ledger system includes block data and index data; wherein, The block data includes at least one block in the blockchain structure, and any one of the at least one blocks contains the hash value of the previous block; The storage system applies a processing engine type designated for processing the data type based on the characteristics of the data type, including: When the data received by the storage system is block data in the data type of the centralized ledger system, the storage system application is assigned a processing engine type for processing the block data; wherein the processing engine type assigned for processing the block data is configured based on the characteristics of the block data; When the data received by the storage system is the index data in the data type of the centralized ledger system, the storage system application is assigned a processing engine type for processing the index data; wherein the processing engine type assigned for processing the index data is configured based on the characteristics of the index data.

3. The computer-implemented method as described in claim 1, wherein, The storage system includes: The type of processing engine specified for handling block data; The type of processing engine specified for handling transaction data; The processing engine type specified for handling index data.

4. The computer-implemented method as described in claim 1, wherein, The storage system receiving processing requests for the data of the centralized ledger system includes: the storage system receiving input / output I / O requests for the data of the centralized ledger system; and The processing engine type designated for processing the data type of the centralized ledger system includes: an input / output I / O processing engine type designated for performing read or write operations on the data type of the centralized ledger system.

5. The computer-implemented method as described in claim 4, wherein, The processing engine types designated for processing the index data include: the corresponding input / output I / O processing engine types designated for performing read or write operations on the index data.

6. The computer-implemented method as described in claim 1, wherein: The storage system receiving a processing request for the data of the centralized ledger system includes: the storage system receiving a data management request for the data of the centralized ledger system; The processing engine types designated for processing the aforementioned data type include: data management processing engine types designated for performing data management operations on the aforementioned data type in the storage system; and The data management operations include one or more of the following: layering, compression, erasure coding, or snapshotting.

7. The computer-implemented method as described in claim 6, wherein: The processing engine types designated for processing the index data include: data management processing engine types designated for performing data management operations on the index data in the storage system.

8. The computer-implemented method as described in claim 1, wherein, One of the data types is block data or transaction data, and The processing engine type designated for processing the aforementioned data type is configured with functions including layering, erasure coding, and compression of the block data or transaction data in the storage system.

9. An apparatus for data processing and storage, the apparatus comprising a plurality of modules for performing the method as described in any one of claims 1 to 8.

10. A system for data processing and storage, comprising: One or more processors; as well as One or more computer-readable storage devices coupled to the one or more processors and storing instructions thereon, the instructions being executable by the one or more processors to perform a computer-implemented method as claimed in any one of claims 1 to 8.

Images