Log structure storage system

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing and storage are disclosed herein. One of the methods includes a storage system receiving a plurality of write requests for storing a corresponding plurality of blockchain data in the storage system. Each of the plurality of blockchain data includes: a value that is one or more of a state of a block, a transaction, or a blockchain network; and, a key corresponding to the value. And the storage system adds the multiple pieces of block chain data to a data log file according to a time sequence of the multiple pieces of block chain data, and limits sorting of the multiple pieces of block chain data in the data log file according to corresponding keys of values in the multiple pieces of block chain data.

Description

Technical Field

[0001] This article deals with log-structured storage systems. Background Art

[0002] A distributed ledger system (DLS), which may also be referred to as a consensus network and / or a blockchain network, enables participating entities to store data securely and immutably. Without reference to any specific use case, a DLS is often referred to as a blockchain network. Examples of types of blockchain networks may include public blockchain networks, private blockchain networks, and consortium blockchain networks. A consortium blockchain network is provided for a selected group of entities, which control the consensus process and include an access control layer.

[0003] Typically, each node of a DLS (e.g., a blockchain network node) stores or has a complete backup of the blockchain network data, so that each node can be independent and the local data at each node can be trusted to provide services. However, this storage scheme imposes demanding storage requirements and increases the storage cost of each node, especially as the DLS reaches scale. Therefore, a solution for improving efficiency and reducing the cost of storage systems is desired. Summary of the invention

[0004] Described herein are techniques for a log-structured storage system for storing data in, for example, a distributed ledger system (e.g., a blockchain network) and / or a centralized ledger system based on a blockchain (e.g., a general auditable ledger service system), wherein the distributed ledger system and / or the centralized ledger system based on a blockchain employs the data structure of the blockchain to exploit the immutability, reliability, and trustworthiness of the data stored on the blockchain.

[0005] The present invention also provides one or more non-transitory computer-readable storage media coupled to one or more processors and having instructions stored thereon, which, when executed by the one or more processors, will cause the one or more processors to perform operations in accordance with an embodiment of the method provided herein.

[0006] The present invention also provides a system for implementing the method provided herein. The log structure storage system includes one or more processors and a computer readable storage medium coupled to the one or more processors and having instructions stored thereon, and when the instructions are executed by the one or more processors, the instructions will cause the one or more processors to perform operations according to the embodiments of the method provided herein.

[0007] It should be understood that the method according to this article may include any combination of aspects and features described herein. That is, the method according to this article is not limited to the combination of aspects and features specifically described herein, but also includes any combination of aspects and features provided.

[0008] The details of one or more embodiments of the present invention are set forth in the accompanying drawings and the description below. Other features and advantages of the present invention will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram illustrating an example of an environment that can be used to perform embodiments herein.

[0010] Figure 2 is a diagram illustrating an example of an architecture according to an embodiment of the present invention.

[0011] Figure 3 is a diagram showing an example of a blockchain-based log-structured storage system according to an embodiment of this document.

[0012] Figure 4 is a diagram illustrating an example of a tiered storage system according to an embodiment of this document.

[0013] Figure 5 is a flow chart illustrating an example of a process for performing a write operation of a log-structured storage system according to an embodiment of the present invention.

[0014] Figure 6 is a flow chart illustrating an example of a process for generating an index related to a write operation of a log-structured storage system according to an embodiment of the present invention.

[0015] Figure 7 is a flowchart illustrating an example of a process for performing a read operation of a log-structured storage system according to an embodiment of the present invention.

[0016] Figure 8 is a flow chart illustrating an example of a process for improving read operations of a log-structured storage system according to an embodiment of the present invention.

[0017] Fig. 9 is a flow chart illustrating an example of a process for managing data log files stored in a log-structured storage system according to an embodiment of the present invention.

[0018] Fig.10 is a flow chart illustrating an example of a process for performing data migration in a log-structured storage system according to an embodiment of the present invention.

[0019] Fig.11 is a flow chart illustrating an example of a process for performing data flow control in a log-structured storage system according to an embodiment of the present invention.

[0020] Fig.12 is a flow chart illustrating an example of a process that may be performed according to embodiments herein.

[0021] Fig.13 is a flow chart illustrating an example of a process that may be performed according to embodiments herein.

[0022] Fig.14 is a flow chart illustrating an example of a process that may be performed according to embodiments herein.

[0023] Fig.15 is a diagram showing an example of modules of a device according to an embodiment of this document.

[0024] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0025] This article describes a log-structured storage system for storing data in, for example, a distributed ledger system (e.g., a blockchain network) and / or a centralized ledger system based on a blockchain (e.g., a general auditable ledger service system), wherein the distributed ledger system and / or the centralized ledger system based on a blockchain adopts the data structure of the blockchain to exploit the immutability, reliability, and trustworthiness of the data stored on the blockchain. In some embodiments, the distributed ledger system and the centralized ledger system based on the blockchain may be collectively referred to as a ledger system based on the blockchain.

[0026] In some embodiments, the centralized ledger system based on blockchain can be a centralized ledger system that can provide cryptographically verifiable state-independent data ledger storage with time-critical audit (with non-repudiation and tamper-proof). In some embodiments, the centralized ledger system based on blockchain can provide ledger services based on a cloud platform, which is characterized by centralized endorsement with credibility and neutrality. The centralized ledger system based on blockchain can provide a highly reliable and high-performance auditable running ledger service, which combines the high credibility of the blockchain system with the high performance and low latency of the centralized system to handle various data and logs with audit requirements, traceability and tracking.

[0027] The techniques described herein produce several technical effects. In some embodiments, the described techniques can be applied in various applications and scenarios to provide efficient, reliable, scalable, cost-effective, and high-performance data storage and management. The described techniques can provide a simple and well-defined set of application programming interfaces (APIs) for blockchain data including, for example, transaction data, block data, state data, and index data.

[0028] The described technology provides a log-structured storage system that not only provides I / O services, but also takes into account cost and personalized needs to provide functions such as tiering, data compression, shared storage, error correction coding, and state snapshots, especially after the amount of data stored in the blockchain system reaches scale. The log-structured storage system can provide features such as log-structured data storage and asynchronous and / or concurrent processing to achieve performance optimization, efficient processing, trusted environment, versatility (for example, both distributed ledger systems and blockchain-based centralized ledger systems are available), and improved storage solutions. The described technology can provide an overall framework or architecture for providing such functions and features.

[0029] Typically, data generated and / or stored in a distributed ledger system (e.g., a blockchain network) may be referred to as blockchain data. Blockchain data may include or be classified as transaction data, block data, state data, and index data. In some embodiments, data generated and / or stored in a centralized ledger system based on blockchain (e.g., a universal auditable ledger service system) may include or be classified as transaction data, block data, and index data.

[0030] In some embodiments, it can be expressed as<hash(value),value> The blockchain receives various blockchain data in the form of a key-value pair (KVP). The value can be actual data representing one or more of a block, a transaction, or the state of the blockchain network. The key can be a hash of the value.

[0031] In some embodiments, for block data, each block may include a block header and a block body. The block header may include identity information of a specific block, and the block body may include transactions confirmed with the block. In some embodiments, block data is a data structure in a blockchain system and generally has one or more of the following characteristics. For example, (1) After consensus is reached in the blockchain network, the content of the block data stored in each node in the blockchain network is theoretically consistent. (2) Block numbers increase densely. (3) There is hash entanglement between consecutive blocks. (4) Block data is append-only. That is, once consensus is reached, historical block data will not be modified. (5) The access frequency of block data is generally low. The storage space occupied by block data is generally large.

[0032] In some embodiments, the state data can be assembled into a global shared state (also called a world state). The world state can include a mapping between an account address and an account state. The world state can be stored in a data structure such as a Merkle Patricia tree (MPT). In some embodiments, for example, in a smart contract scenario, the state data can be designed based on the content of the Merkle tree. It is an incremental content-addressed data set. The storage space occupied by the state data is usually large.

[0033] In some embodiments, the state data can be further classified into current state and historical state. In some embodiments, the current state is the state data corresponding to the latest block and is the data source when the latest transaction is executed on the blockchain network. In some embodiments, the historical state is a content-addressed data set that stores all state data from the genesis block to the latest block. In some embodiments, the historical state data is stored in a historical state tree. The historical state tree can store state information as a content-addressable representation as<hash(node value),node value> The value or node value may be the account state of an account associated with a blockchain node, and the key may be a hash value of the corresponding account state. In some embodiments, the current state data is stored in a current state tree. In some embodiments, the current state tree may be location-addressed based on one or more location-related identifiers (IDs). For example, the current state tree may store state information as a hash represented as<node ID,node value> KVP, where node values ​​can be addressed according to the corresponding node ID.

[0034] In some embodiments, transaction data may include data related to the input and output of a series of operations. In some embodiments, transaction data may include data related to the exchange of valuables (e.g., assets, products, services, currencies).

[0035] In some embodiments, the index data may indicate a mapping correspondence between data (e.g., transaction data, block data, and status data) and a data log file storing the data for addressing or retrieving the data. In some embodiments, the index data may indicate a physical location where the corresponding data is stored in a storage system. In some embodiments, the index data may include one or more of the following: an index indicating a correspondence from a block hash value to a block number, an index indicating a correspondence from a block hash value to a storage location, an index indicating a correspondence from a transaction hash value to a transaction, or an index indicating a correspondence from a receipt hash value to a receipt. In some embodiments, the index data does not include the content of the blockchain data.

[0036] When more and more transactions are input into the blockchain, the size of blockchain data (such as state data and block data) will become larger and larger. In DLS, each node of DLS stores an entire copy of the blockchain, which may take up a lot of storage space even if some old block data or state data is not frequently accessed.

[0037] In some embodiments, blockchain data is stored in data files by a log structure system, and the data files are continuously appended and divided based on time. In some embodiments, data may not be rearranged according to key classification (e.g., data is not arranged by key value or other metric, so that hot data and cold data are not mixed in multiple data log files), thereby greatly reducing the technical challenges of layered implementation.

[0038] In some embodiments, the log-structured storage system uses two types of append-only data files to store blockchain data to provide data persistence: a data log file and an index log file. For example, block data, transaction data, state data, and additional self-describing data may be stored in a data log file, and index data (e.g., an identifier and offset of a data log file) indicating the storage location of transaction data, block data, and state data may be stored in an index log file.

[0039] In blockchain data, transaction data and block data may be log-structure friendly and may include append-only data such that such data may be written to a data log file by directly adding or appending such data to the corresponding data log file. In some embodiments, the writing of transaction data and block data may not require a large amount of compaction. For example, it may require a relatively small number of transaction replays and may not require a block rollback. In some embodiments, state data may be log-structure friendly data such that historical state data may be increased without the need for compaction.

[0040] In some embodiments, the log structure storage system can support multi-level data tiering and support a variety of storage devices, such as cloud disks, network attached systems (NAS) and object storage services (OSS) (low frequency, archive). For example, log files can be stored in a cloud-based storage system, NAS or OSS devices, or a self-built distributed storage system.

[0041] In some embodiments, different types of log files may have different storage strategies. For example, data log files that have not been accessed for a relatively long time may be stored in a relatively inexpensive and relatively low-speed storage device such as NAS / OSS, and may be processed using compression and error correction coding for storage. As another example, index log files may be stored on a high-speed storage device such as a cloud disk.

[0042] In some embodiments, a log-structured storage system may perform data tiering by using a least recently used (LRU) memory cache and a disk cache to optimize read performance of slow storage devices.

[0043] In some embodiments, a log-structured storage system may provide a hierarchical pool manager that manages multiple levels of storage device pools. In some embodiments, each pool supports multiple disks or storage devices in a cluster. The hierarchical pool manager may manage the space, pressure, and operating status of the pool. In some embodiments, a log-structured storage system may provide a migration task manager that manages bidirectional migration tasks of data between storage devices of different levels; manages result callbacks, statistical information, the life cycle of migration tasks, etc. In some embodiments, a log-structured storage system may provide a migration scheduler that supports pluggable policies, manages data migration policies, and provides data create / query / update / delete interfaces.

[0044] The disclosed log structure storage system adopts the idea of ​​merge tree (LSM-Tree) architecture. In some embodiments, the log structure storage system may include multiple log structure storage instances (or streams), each of which is responsible for storing and managing data of a distributed ledger system (e.g., a blockchain system) or a centralized ledger system based on blockchain. In some embodiments, the log structure storage system may convert random write operations into ordered append operations to alleviate write amplification problems caused by frequent "dirty" page refreshes due to a large number of random write operations. In some embodiments, the log structure storage system may delay write refresh operations in high-performance scenarios and reduce the number of synchronization operations to improve the efficiency and performance of the entire system.

[0045] To provide further context for embodiments herein, and as described above, a distributed ledger system (DLS), which may also be referred to as a consensus network (e.g., composed of peer-to-peer nodes) and a blockchain network, enables participating entities to conduct transactions and store data securely and immutably. Although the term blockchain is often associated with a specific network and / or use case, blockchain as used herein generally refers to a DLS without reference to any specific use case.

[0046] Blockchain is a data structure that stores transactions in a way that transactions cannot be tampered with. Therefore, the transactions recorded on the blockchain are reliable and trustworthy. The blockchain includes one or more blocks. Each block in the chain is linked to the previous block by including the cryptographic hash of the previous block in the chain. Each block also includes a timestamp, its own cryptographic hash value, and one or more transactions. Transactions that have been verified by nodes in the blockchain network are hashed and compiled into a Merkle tree. The Merkle tree is a data structure that hashes the data at the leaf nodes of the tree, and all hash values ​​in each branch of the tree are cascaded at the root of the branch. The process continues along the tree until the root of the entire tree, and the hash values ​​representing all the data in the tree are stored at the root of the entire tree. By determining whether the hash value is consistent with the structure of the tree, it is possible to quickly verify whether the hash value is the hash value of the transaction stored in the tree.

[0047] In some embodiments, for example, in a blockchain network as a network of computing nodes, a blockchain can be implemented in a distributed or decentralized or at least partially decentralized manner for storing transactions. Each computing node (also known as a blockchain network node) can manage, update, and maintain one or more blockchains by broadcasting transactions, verifying transactions, and confirming transaction validity. As described above, a blockchain network can be provided as a public blockchain network, a private blockchain network, or a consortium blockchain network. The embodiments of this article are further described in detail herein with reference to a consortium blockchain network. However, it is contemplated that the embodiments of this article can be implemented in any suitable type of blockchain network.

[0048] Typically, consortium blockchain networks are private among participating entities. In a consortium blockchain network, consensus processing is controlled by an authorized set of nodes, which may be referred to as consensus nodes, with one or more consensus nodes operated by the respective entity (e.g., financial institution, insurance company). For example, a consortium of 10 entities (e.g., financial institutions, insurance companies) may operate a consortium blockchain network, with each entity operating at least one node in the consortium blockchain network.

[0049] In some examples, within a consortium blockchain network, a global blockchain is provided as a blockchain that is replicated across all nodes. That is, for the global blockchain, all consensus nodes are in a state of full consensus. In order to reach consensus (e.g., agree to add a block to the blockchain), a consensus protocol is implemented within the consortium blockchain network. For example, the consortium blockchain network can implement a practical Byzantine fault tolerance (PBFT) consensus, which is described in further detail below.

[0050] In some embodiments, the centralized ledger system can also adopt the data structure of blockchain to take advantage of the immutability, reliability and credibility of the data stored on the blockchain. In some embodiments, for example, the centralized ledger system can be referred to as a centralized ledger system based on blockchain or a general auditable ledger service system. In some embodiments, the centralized ledger system based on blockchain can include a central trusted institution that provides transparent, immutable and cryptographically verifiable data stored in blocks of the blockchain data structure. The stored data can be in log format, for example, including not only transaction logs, but also other transaction data and block data. Due to the presence of a central trusted institution, a centralized ledger system based on blockchain can establish trust without performing consensus processing. In some embodiments, a centralized ledger system based on blockchain can be more efficient than a typical distributed or decentralized ledger system based on blockchain. In some embodiments, a centralized ledger system based on blockchain can provide cloud-based storage services with enhanced trust, efficiency and storage performance.

[0051] Figure 1 1 is a diagram showing an example of an environment 100 that can be used to perform embodiments of the present invention. In some examples, the environment 100 enables entities to participate in a consortium blockchain network 102. The environment 100 includes computing devices 106, 108 and a network 110. In some examples, the network 110 includes a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and connects websites, user devices (e.g., computing devices), and back-end systems. In some examples, the network 110 can be accessed through a wired and / or wireless communication link. In some examples, the network 110 enables communication with the consortium blockchain network 102 or communication within the consortium blockchain network 102. In general, the network 110 represents one or more communication networks. In some cases, the computing devices 106, 108 can be nodes of a cloud computing system (not shown), or each computing device 106, 108 can be a separate cloud computing system that includes multiple computers interconnected by a network and used as a distributed processing system.

[0052] In the depicted example, computing devices 106, 108 may each include any suitable computing system capable of participating as a node in consortium blockchain network 102. Examples of computing devices include, but are not limited to, servers, desktop computers, laptops, tablets, and smartphones. In some examples, computing devices 106, 108 host one or more computer-implemented services for interacting with consortium blockchain network 102. For example, computing device 106 may host a computer-implemented service, such as a transaction management system, for a first entity (e.g., user A), for example, the first entity uses the transaction management system to manage its transactions with one or more other entities (e.g., other users). Computing device 108 may host a computer-implemented service, such as a transaction management system, for a second entity (e.g., user B), for example, the second entity uses the transaction management system to manage its transactions with one or more other entities (e.g., other users). Figure 1 In the example of , the consortium blockchain network 102 is represented as a peer-to-peer network of nodes, and the computing devices 106 and 108 provide nodes of a first entity and a second entity participating in the consortium blockchain network 102, respectively.

[0053] Figure 2 2 is a diagram illustrating an architecture 200 according to an embodiment of the present invention. The exemplary conceptual architecture 200 includes participant systems 202, 204, 206 corresponding to participant A, participant B, and participant C, respectively. Each participant (e.g., user, enterprise) participates in a blockchain network 212 provided as a peer-to-peer network, which includes multiple nodes 214, at least some of which record information in an unalterable manner in a blockchain 216. As further detailed in the figure, although a single blockchain 216 is schematically depicted in the blockchain network 212, multiple copies of the blockchain 216 are provided and maintained on the blockchain network 212.

[0054] In the depicted example, each participant system 202, 204, 206 is provided by or on behalf of participant A, participant B, and participant C, respectively, and functions as a respective node 214 in the blockchain network. As used herein, a node generally refers to an individual system (e.g., computer, server) that is connected to the blockchain network 212 and enables the corresponding participant to participate in the blockchain network. Figure 2In the example of , a participant corresponds to each node 214. However, it is contemplated that one participant may operate multiple nodes 214 within the blockchain network 212, and / or multiple participants may share a node 214. In some examples, the participant systems 202, 204, 206 communicate with or through the blockchain network 212 using a protocol (e.g., Hypertext Transfer Protocol Secure (HTTPS)) and / or using remote procedure calls (RPCs).

[0055] Nodes 214 may have different levels of participation within blockchain network 212. For example, some nodes 214 may participate in the consensus process (e.g., as monitoring nodes for adding blocks to blockchain 216), while other nodes 214 do not participate in this consensus process. As another example, some nodes 214 store a complete copy of blockchain 216, while other nodes 214 store only a copy of a portion of blockchain 216. For example, data access privileges may limit the blockchain data that a respective participant stores within their respective system. Figure 2 In the example of FIG. 2 , participant systems 202 , 204 , and 206 store respective complete copies 216 ′, 216 ″, and 216 ′″ of blockchain 216 .

[0056] Blockchain (e.g. Figure 2 The blockchain 216) consists of a series of blocks, each of which stores data. Examples of data include transaction data representing a transaction between two or more participants. Although "transaction" is used herein by way of non-limiting example, it is contemplated that any suitable data may be stored in the blockchain (e.g., documents, images, videos, audio). Examples of transactions may include (but are not limited to) the exchange of valuables (e.g., assets, products, services, currencies). Transaction data is stored in an immutable manner in the blockchain. That is, the transaction data cannot be changed.

[0057] Before the transaction data is stored in the block, the transaction data is hashed. Hashing is the process of converting the transaction data (provided as string data) into a fixed-length hash value (also provided as string data). It is impossible to un-hash the hash value to obtain the transaction data. Hashing ensures that even a slight change in the transaction data will result in a completely different hash value. In addition, as mentioned above, the hash value has a fixed length. That is, regardless of the size of the transaction data, the length of the hash value is fixed. Hashing includes processing the transaction data through a hash function to generate a hash value. Examples of hash functions include (but are not limited to) the Secure Hash Algorithm (SHA)-256, which outputs a 256-bit hash value.

[0058] The transaction data of multiple transactions are hashed and stored in a block. For example, the hash of two transactions is provided and they themselves are hashed to provide another hash. This process is repeated until a single hash is provided for all transactions to be stored in a block. This hash is called the Merkle root hash and is stored in the header of the block. Changes in any transaction will cause its hash to change and ultimately the Merkle root hash to change.

[0059] Blocks are added to the blockchain through a consensus protocol. Multiple nodes in the blockchain network participate in the consensus protocol and compete to add blocks to the blockchain. Such nodes are called consensus nodes. The PBFT introduced above is used as a non-limiting example of a consensus protocol. Consensus nodes execute the consensus protocol to add transactions to the blockchain and update the overall state of the blockchain network.

[0060] In more detail, the consensus node generates a block header, hashes all transactions in the block, and combines the resulting hashes in pairs to generate further hashes until a single hash (Merkle root hash) is provided for all transactions in the block. This hash is added to the block header. The consensus node also determines the hash of the latest block in the blockchain (i.e., the last block added to the blockchain). The consensus node also adds a random number (nonce) and a timestamp to the block header.

[0061] In general, PBFT provides practical Byzantine state machine replication that tolerates Byzantine failures (e.g., faulty nodes, malicious nodes). This is achieved by assuming in PBFT that failures will occur (e.g., assuming that there are independent node failures and / or manipulated messages sent by consensus nodes). In PBFT, consensus nodes are provided in an order that includes a primary consensus node and a backup consensus node. The primary consensus node is changed periodically. Transactions are added to the blockchain by reaching a consensus on the global state of the blockchain network by all consensus nodes within the blockchain network. In this process, messages are transmitted between consensus nodes, and each consensus node proves that the message is received from a designated peer node and verifies that the message has not been tampered with during transmission.

[0062] In PBFT, the consensus protocol is provided in multiple stages with all consensus nodes starting in the same state. First, the client sends a request to the primary consensus node to invoke a service operation (e.g., execute a transaction within the blockchain network). In response to receiving the request, the primary consensus node multicasts the request to the backup consensus nodes. The backup consensus nodes execute the request and each sends a reply to the client. The client waits until a threshold number of replies are received. In some examples, the client waits until f+1 replies are received, where f is the maximum number of faulty consensus nodes that can be tolerated within the blockchain network. The end result is that a sufficient number of consensus nodes agree on the order in which records are added to the blockchain, and the record is either accepted or rejected.

[0063] In some blockchain networks, cryptography is used to maintain the privacy of transactions. For example, if two nodes want to keep a transaction private so that other nodes in the blockchain network cannot see the details of the transaction, the two nodes can encrypt the transaction data. Examples of encryption include, but are not limited to, symmetric encryption and asymmetric encryption. Symmetric encryption refers to the encryption process that uses a single key to both encrypt (generate ciphertext from plaintext) and decrypt (generate plaintext from ciphertext). In symmetric encryption, the same key can be used for multiple nodes, so each node can encrypt / decrypt the transaction data.

[0064] Asymmetric encryption uses key pairs, each of which consists of a private key and a public key, where the private key is known only to the corresponding node and the public key is known to any or all other nodes in the blockchain network. A node can encrypt data using the public key of another node, and the encrypted data can be decrypted using the private key of the other node. For example, referring again to Figure 2 , participant A can use participant B's public key to encrypt data and send the encrypted data to participant B. Participant B can use its private key to decrypt the encrypted data (ciphertext) and extract the original data (plaintext). Messages encrypted using a node's public key can only be decrypted using that node's private key.

[0065] Asymmetric encryption is used to provide digital signatures, which enables participants in a transaction to confirm the other participants in the transaction and the validity of the transaction. For example, a node can digitally sign a message, and another node can confirm that the message was sent by the node based on the digital signature of participant A. Digital signatures can also be used to ensure that the message has not been tampered with during transmission. For example, again referring to Figure 2, Participant A will send a message to Participant B. Participant A generates a hash value of the message and then encrypts the hash value using its private key to provide a digital signature that is an encrypted hash value. Participant A appends the digital signature to the message and sends the message with the digital signature to Participant B. Participant B decrypts the digital signature using Participant A's public key and extracts the hash value. Participant B hashes the message and compares the hash values. If the hash values ​​are the same, Participant B can confirm that the message is indeed from Participant A and has not been tampered with.

[0066] Figure 3 300 is a diagram showing an example of a log structure storage system according to an embodiment of the present invention. The log structure storage system 300 can store data of a distributed ledger system (e.g., a blockchain network) and / or a centralized ledger system based on blockchain (e.g., a general auditable ledger service system) (collectively referred to as a blockchain-based ledger system) that stores data on one or more blockchains.

[0067] In some embodiments, the log structure storage system 300 can be implemented by each consensus node of the blockchain network or the central node of the centralized ledger system based on the blockchain. In some embodiments, the log structure storage system 300 can be connected to a distributed storage system 340 constructed by the client nodes of the ledger system based on the blockchain. As shown in the figure, the log structure storage system 300 includes a front-end input / output (I / O) subsystem 310, a multi-layer storage subsystem 320, and a back-end data management subsystem 330. In some embodiments, the front-end I / O subsystem 310 can perform a write operation to write data to a data file (e.g., a data log file and an index log file) stored in the multi-layer storage subsystem 320, and perform a read operation to access data from a data file stored in the multi-layer storage subsystem 320. In some embodiments, the back-end data management subsystem 330 can process, reorganize, and manage the data in the data file according to different requirements to improve the efficiency and performance of the entire system.

[0068] The front-end I / O subsystem 310 may include any suitable computing elements (e.g., one or more of a processor, memory 315, etc.) to perform the methods described herein. In some embodiments, the front-end I / O subsystem 310 may perform front-end I / O operations including various read and write operations (e.g., insert, update, delete, query, etc.) on a variety of data elements.

[0069] In some embodiments, all data elements processed by the front-end I / O subsystem 310 (e.g., transaction data, block data, and status data) can be stored in a log file format, regardless of whether the log file is generated based on a write operation or a file generated based on operations of the back-end data management subsystem 330, such as storage tiering, compaction, data compression, erasure coding, etc.

[0070] In some embodiments, data processed by the front-end I / O subsystem 310 may be stored in the following two types of log files: (1) a data log file (e.g., data log files 390, 362, 364, 366, 372, 374, and 376) storing data such as blockchain data (e.g., transaction data, block data, state data) and self-describing metadata; and (2) an index log file (e.g., index log file 380) storing index information indicating the physical location of the data (e.g., an identifier and offset of the data log file). In some embodiments, the data log file does not store the index information, and the index information is maintained by a separate index log file.

[0071] In some embodiments, the front-end I / O subsystem 310 may be configured to perform a write operation to write blockchain data into the data log file 390. In some embodiments, the blockchain data may include block data, transaction data, or state data generated by a blockchain network or a distributed ledger system. In some embodiments, the blockchain data may include block data and transaction data generated by a centralized ledger system based on blockchain. In some embodiments, the data written to the data log file 390 may include metadata describing the data block, such as a transaction hash value and sequence value, a block hash value and a block number, a snapshot version number, a cyclic redundancy check (CRC) code, encryption information, etc. In some embodiments, the data log file 390 may be an append-only file.

[0072] In some embodiments, the front-end I / O subsystem 310 may be configured to generate an index indicating a physical location where corresponding data is stored in the log-structured storage system 300 (e.g., in a data log file in the multi-layer storage subsystem 320). In some embodiments, the index may be stored in an index log file 380. In some embodiments, the data log file and the index log file may be stored in the multi-layer storage subsystem 320. In some embodiments, the index may be stored in an index log file 380, which is stored in one of the storage devices of the multi-layer storage subsystem 320 with the highest access speed.

[0073] In some embodiments, the data log file can be continuously updated based on data write or append operations. In some embodiments, the data log file can have a configurable maximum length, such as between 512MB and 2GB. In some embodiments, if it is determined that the data log file has reached the maximum length or size, the data log file can be sealed or set to read-only, and a new data log file can be allocated for a new write operation.

[0074] In some embodiments, the front-end I / O subsystem 310 may perform write operations, including modifications to data stored in the log structure storage system 300. In some embodiments, the front-end I / O subsystem 310 processes modifications to data by adding or appending the data to a data log file in a log format, thereby not overwriting the original data. In some embodiments, the data log file may form a write-ahead log (WAL) layer that may be used for crash recovery.

[0075] In some embodiments, the front-end I / O subsystem 310 stores index information in the memory 315, which indicates the mapping correspondence between data (e.g., transaction data, block data, and status data) and the data log file storing the data to address or retrieve the data. In some embodiments, a log structure merge (LSM) method can be used to organize the index data in the memory. In some embodiments, the index of the newly written data can be stored in the memory 315 and flushed to the index log file 380 when the memory usage exceeds a predetermined threshold. In this way, the index of the old data can be stored in the index log file 380 in the disk storage device or the hard disk drive storage device, thereby freeing up space to cache the index of the new hot data in the memory 315.

[0076] In some embodiments, the index data may include one or more of the following: an index indicating a correspondence from a block hash value to a block number, an index indicating a correspondence from a block hash value to a storage location, an index indicating a correspondence from a transaction hash value to a transaction, or an index indicating a correspondence from a receipt hash value to a receipt. In some embodiments, the index data for a centralized ledger system based on a blockchain may include an index indicating a correspondence from a sequence to a transaction storage location and / or an index indicating a correspondence from a time sequence to a transaction hash value.

[0077] In some embodiments, the front-end I / O subsystem 310 may include a plurality of memory index maps stored in the memory 315. In some embodiments, the memory index map may be considered to be any suitable component, unit, module, or data structure (e.g., table or structure) for maintaining index data in the memory of the front-end I / O subsystem 310. The memory index map may be a key component of the front-end I / O subsystem 310 that determines the scalability and performance of the front-end I / O subsystem 310 and the entire log-structured storage system 300. In some embodiments, because blockchain data is highly time-sensitive and the chances of recently written transaction data and block data being accessed again are relatively high, the log-structured storage system 300 may store the index of hot data in the index map in the memory 315 to improve the performance of the entire log-structured storage system 300.

[0078] In some embodiments, the memory index map may maintain an index indicating a mapping from a transaction hash value to a sequence value and / or an index indicating a mapping from a block hash value and a block number to a physical location of the data. In some embodiments, the front-end I / O subsystem 310 periodically persists a checkpoint of the index map in the memory 315 into an index log file. For example, the front-end I / O subsystem 310 may periodically or at a point in time capture a snapshot of the index data in the memory 315 and store the snapshot in an index log file 380 in the multi-layer storage subsystem 320. This may create a point in time at which the log-structured storage system 300 may apply changes contained in the index log file 380 during a recovery process after an unexpected shutdown or crash of the log-structured storage system 300. In some embodiments, the front-end I / O subsystem 310 may read data (e.g., transaction data, block data, and status data) by querying the memory index map and determining the current location of the requested data.

[0079] In some embodiments, a complete checkpoint of the memory index map can be written to the index log file when the index log file is created. In some embodiments, the index log file can be updated by batching the index of the write operations. In some embodiments, the batch size can be dynamically configurable, such as thousands of transaction write operations or several megabytes (MB) of write operations. In some embodiments, when the index log file has been updated for a specific batch number of write operations, the index log file can be sealed or set to read-only, and a new index log file can be created to write new data.

[0080] In some embodiments, to recover from an abnormal crash, the front-end I / O subsystem 310 may load an index log file (e.g., index log file 380) into the memory 315 and scan the bottom of the page of the data log file 390 to ensure the consistency of the data log file 390 and the index log file 380. In some embodiments, the index log file may lag behind the data log file by several batches, so the recovery time may occupy limited I / O resources and time.

[0081] In some embodiments, the indexes of newly written transaction data and block data may be added to the index map and index log file, but the indexes of existing transaction data and block data may not be modified except in replay attack and block rollback scenarios. In some embodiments, in order to achieve high concurrency of read and write operations, the memory index map may be divided into a read-only basic index map 316 and a read-write incremental index map 312. In some embodiments, the basic index map 316 may store the index of cold data, and the incremental index map 312 may store the index of newly written data. In some embodiments, the hash value index may be stored in a hash value table, and the sequence index may be stored in a B-tree.

[0082] In some embodiments, during a write operation of the front-end I / O subsystem 310, the index information of the data may first be updated to the incremental index map 312. During a read operation, the front-end I / O subsystem 310 may first search for the requested data in the incremental index map 312. If the requested data is not found in the incremental index map 312, the front-end I / O subsystem 310 may then search the base index map 316.

[0083] In some embodiments, the front-end I / O subsystem 310 may periodically flush index data from the memory 315 to the index log file 380. In some embodiments, the basic process of index refreshing may include the following operations: (1) combining the incremental index map 312 and the basic index map 316; (2) persisting the basic index map 316 (e.g., storing the basic index map in the index log file); (3) releasing part or all of the basic index map 316 from the memory 315; and (4) exchanging the index data to the memory 315 by reading the requested index data.

[0084] In some embodiments, the front-end I / O subsystem 310 can convert the incremental index map 312 in the memory 315 into an immutable index map 314, and then flush them to the index log file 380, and create a new incremental index map to receive the index generated according to the new request. In this way, the storage occupation of the incremental index map can be reduced to improve the performance of the log structure storage system 300.

[0085] In some embodiments, to reduce the impact on the front-end I / O, the index mappings in the memory can be merged asynchronously at the back end. In some embodiments, the merge process can be triggered by at least one of the following two conditions: (1) the size of the incremental index mapping exceeds a predetermined threshold; (2) a new snapshot is created. In some embodiments, the front-end I / O subsystem 310 can generate a merged index mapping to include an immutable index mapping 314 to be flushed to the index log file 380. In some embodiments, the front-end I / O subsystem 310 can combine the merged index mapping with the current basic index mapping 316 to generate a new basic index mapping.

[0086] In some embodiments, during operation, the front-end I / O subsystem 310 can run with multiple base index maps and index log files. In some embodiments, when compaction is required in certain situations, minor compaction and major compaction can be performed periodically by combining all base index maps and incremental index maps into one base index map. Major compaction mainly merges and manages indexes, which can be used in scenarios such as snapshots, garbage collection loading, and index file management.

[0087] In some embodiments, a major compaction may be performed by merging a base index map and an incremental index map and generating a new base index map, and storing it in a new index log file. In some embodiments, a minor compaction may be performed by combining several index log files and generating a new index log file, which may reduce the number of index log files. In some embodiments, if the size of the current index log file reaches a predetermined threshold, the current index log file may be set to a sealed or immutable state and closed, and a new index log file may be created for the new index data.

[0088] In some embodiments, during a read operation, if a search in the memory index map fails, two or more I / O operations may be required, which may burden the log-structured storage system 300. In some embodiments, the front-end I / O subsystem 310 may provide a multi-level cache mechanism having a memory cache 313 and a block cache 317 (e.g., using flash media (e.g., SSD cloud disk)).

[0089] In some cases, the log structure storage system 300 may receive a large read request, so that the log structure storage system 300 needs to access multiple data log files to obtain complete request data for the client. However, accessing multiple data log files may cause considerable overhead. In some embodiments, the back-end data management subsystem 330 may perform a compression operation to cascade logically adjacent data blocks to reduce fragmentation. In some embodiments, the compression operation may have overhead and may be performed when data fragmentation is severe.

[0090] In some embodiments, the multi-layer storage subsystem 320 may include multi-layer storage devices. The storage device may include a storage medium and corresponding software and / or hardware interfaces. In some embodiments, the multi-layer storage device may include multiple storage devices with different performance characteristics (e.g., access speeds). For example, the multi-layer storage device may include a cloud disk, a network attached storage (NAS) device, and an object storage service (OSS) device. In some embodiments, the multi-layer storage device is layered in a hierarchical structure according to one or more performance characteristics. In some embodiments, one or more performance characteristics may include access speed, access bandwidth, or access latency. For example, the multi-layer storage device may include a first-tier storage device having a first performance characteristic (e.g., access speed) and a second-tier storage device having a second performance characteristic lower than the first performance characteristic (e.g., a relatively lower access speed relative to the first-tier storage device), and so on. Figure 3 As shown, an example of a multi-layer storage subsystem 320 may include a first-layer storage device 350, a second-layer storage device 360, and a third-layer storage device 370, wherein the first-layer storage device 350 includes a cloud disk or a cloud-based storage device (e.g., a solid-state drive (SSD) cloud disk, an embedded SSD (ESSD) cloud disk), the second-layer storage device 360 ​​includes a NAS device, and the third-layer storage device 370 includes an OSS device.

[0091] In some embodiments, the multi-layer storage device can store different types of data. In some embodiments, data can be classified into hot data 355, warm data 365, and cold data 375 based on, for example, the time when the data was generated or received or the frequency with which the data is accessed. For example, the data of the latest transaction can be hot data; yesterday's transaction data can be warm data, and historical transaction data conducted a week ago can be cold data. For another example, the data in the most recently generated 10 blocks in the blockchain can be hot data; the most recently generated The data in the first block may be warm data, and the data in other earlier blocks may be cold data. However, in some embodiments, the genesis block of the blockchain may be considered hot data because it is frequently accessed.

[0092] In some embodiments, the multi-tier storage subsystem 320 may store hot data 355, warm data 365, and cold data 375 in the multi-tier storage devices, respectively. For example, the first tier storage device 350 may store hot data 355; the second tier storage device 360 ​​may store warm data 365; and the third tier storage device 370 may store cold data 375. In some embodiments, a tier storage device may store one or more types of data, for example, based on storage space and cost. For example, the first tier storage device 350 may store hot data 355 and some warm data 365, while the second tier storage device 360 ​​may store the rest of the warm data 365 and some cold data 375.

[0093] In some embodiments, each layer of storage devices may store a data log file including blockchain data generated by a blockchain-based ledger system (e.g., a distributed ledger system and / or a centralized blockchain-based ledger system). For example, the first layer storage device 350 may store a first data log file 390 including first blockchain data generated by a blockchain-based ledger network, and the second layer storage device 360 ​​may store a second data log file 362 including second blockchain data generated by a blockchain-based ledger system, and so on.

[0094] In some embodiments, blockchain data stored in a data log file in a relatively lower layer of a storage device may be written earlier than blockchain data stored in a data log file in a relatively higher layer of the storage device. For example, second blockchain data stored in a second data log file 362 on a second layer of storage device 360 ​​may be written earlier than first blockchain data stored in a first data log file 390 on a first layer of storage device 350.

[0095] In some embodiments, the first tier storage device 350 may further store one or more index log files 380, which include index data for indicating the physical storage location of data in the data log files 390, 362, 364, 366, 372, 374, and 376 stored in the multi-tier storage devices 350, 360, and 370. For example, Figure 3 As shown, the first-layer storage device 350 can store an index log file 380, which includes index data indicating the physical storage location of the blockchain data in the data log file 390 stored in the first-layer storage device 350, the physical storage location of the blockchain data in the data log files 362, 364 and 366 stored in the second-layer storage device 360, and the physical storage location of the blockchain data in the data log files 372, 374 and 376 stored in the third-layer storage device 370.

[0096] In some embodiments, one or more index log files may be stored in the second tier storage device 360 ​​and / or the third tier storage device 370 .

[0097] In some embodiments, the index log file and the data log file stored on the multi-layer storage subsystem 320 are append-only log files. In some embodiments, the blockchain data stored in the data log file may include block data, transaction data, and historical state data.

[0098] In some embodiments, a higher-layer storage device may store a data log file including blockchain data migrated from a lower-layer storage device. For example, a first-layer storage device may store a data log file including blockchain data that is accessed more frequently than blockchain data in a data log file of a second-layer storage device and blockchain data migrated from the second-layer storage device.

[0099] In some embodiments, the storage system 300 may further include a distributed storage system 340, which includes storage media such as non-volatile memory express (NVME), SSD, hard disk drive (HDD) and laminated magnetic recording (SMR). In some embodiments, the distributed storage system 340 can be generated or expanded by client nodes of a blockchain-based ledger system to have better availability, partition tolerance, flexibility and cost. For example, the distributed storage system 340 can allow expansion by adding more servers or storage nodes and thereby linearly increasing capacity and performance. It can use inexpensive standard servers, drives and networks. In some embodiments, the distributed storage system 340 can improve the utilization of standard servers, thereby resulting in less power consumption, better cooling efficiency, better space utilization, less maintenance costs, etc.

[0100] The front-end I / O subsystem 310 may perform write operations on blockchain data and generate index log files 380 and data log files 390, 362, 364, 366, 372, 374, and 376 stored on the multi-layer storage subsystem 320. Over time, the data stored on the multi-layer storage subsystem 320 may accumulate and aggregate, and may reduce the performance of the log structure storage system 300. The back-end data management subsystem 330 may process and reorganize the data according to different needs, for example, thereby improving the performance of the log structure storage system 300 and reducing its cost. In some embodiments, the back-end data management subsystem 330 may manage data independently of the front-end I / O subsystem 310. For example, the back-end data management subsystem 330 may perform data management operations such as tiering, compression, erasure coding, state snapshots, compaction, and verification on sealed index log files or read-only index log files and data log files at the back end. In some embodiments, the back-end data management subsystem 330 may implement flow control to minimize the impact on the front-end I / O processing of the front-end I / O subsystem 310.

[0101] In some embodiments, the tasks of the back-end data management subsystem 330 may include rewriting the stored data and replacing the index corresponding to the rewritten data. In some embodiments, the back-end data management subsystem 330 may automatically determine whether the data log file needs to be rewritten on the back end. In some embodiments, the back-end data management subsystem 330 may determine the placement of the rewrite based on configurations such as layering, compression, and erasure coding. In some embodiments, the back-end data management subsystem 330 may read data from one or more source data log files and rewrite the data to the target data log file. In some embodiments, when the rewrite is completed, the back-end data management subsystem 330 may set the target data log file to a sealed or immutable state and generate a corresponding target index log file. In some embodiments, the target index log file may include a list of data log files that can be safely deleted, and the data log files referenced by the target index log file. In some embodiments, the back-end data management subsystem 330 does not recycle old data log files that can still be used by real-time instances of the front-end I / O subsystem 310.

[0102] In some embodiments, the back-end data management subsystem 330 can process read-only index log files and corresponding read-only data log files generated according to I / O operations of the front-end I / O subsystem 310. In some embodiments, the back-end data management subsystem 330 can analyze the index log files and determine, for example, the level of heat, warm or cold of the data, the amount of data, the garbage rate and / or the amount of fragmentation. In some embodiments, based on the garbage rate, disk usage and / or system requests, the back-end data management subsystem 330 can perform one or more of the following tasks:

[0103] (1) Data tiering: For example, when the storage medium usage rate approaches the safety upper limit, data may need to be migrated to the storage medium of the next or lower storage device.

[0104] (2) Data compression: For example, when the storage medium usage is close to the safety upper limit, it may be necessary to compress the data file.

[0105] (3) Erasure Coding (EC): For example, when the storage medium usage rate is close to the safety upper limit, it may be necessary to release storage space through erasure coding.

[0106] (4) State snapshot. For example, when there is a state modification (e.g., reclaiming storage space after data deletion), a snapshot of the blockchain state can be performed.

[0107] (5) Data compaction: For example, if the garbage or fragments in the data log file grow to a certain size and thus significantly affect the performance of the log structure storage system 300, it may be necessary to clean up the garbage or fragments.

[0108] (6) Verification. For example, a cyclic redundancy check (CRC) may be performed on the data on the storage medium periodically or as needed.

[0109] Data stratification:

[0110] In some embodiments, for write requests that require relatively high performance, the write request can be written to a faster storage device (e.g., SSD cloud disk, ESSD cloud disk, NVME, etc.) among multiple different storage devices. For write requests that require lower performance in exchange for lower overhead, the write request can be written to a storage device medium (e.g., NAS, etc.). In some embodiments, the back-end data management subsystem 330 can use a mixed set of slow and fast storage devices for data tiering and data migration. For example, the access frequency of new block data generated by the blockchain network may be relatively higher than that of old block data, and the new block data can be stored in a faster storage device. In some embodiments, a portion of the new block data with the highest access frequency can be stored in a memory cache (e.g., memory cache 313) and / or a high-speed disk cache (e.g., block cache 317).

[0111] In some embodiments, both the distributed ledger system and the centralized ledger system based on blockchain have strong hot and cold characteristics, which makes them suitable for tiered storage. For example, a tiered storage system such as the multi-layer storage subsystem 320 can be used to include one or more of the following features: (1) a combination of fast storage media with smaller storage space and slow storage media with larger storage space improves space utilization without sacrificing performance; (2) supports cold migration (e.g., cold data is automatically migrated from fast media to slow media) and warm-up (e.g., data is migrated from slow media to fast media); (3) scalability to reduce maintenance costs when the scale increases; (4) supports flexible configuration based on user needs; (5) supports multimedia storage pools; or (6) quickly migrates to new storage media.

[0112] Figure 4 4 is a diagram showing an example of a tiered storage system 400 according to an embodiment of the present invention. In some embodiments, the tiered storage system may include multiple levels or layers of storage devices, for example, based on the access speed of the storage devices. For example, referring to Figure 4 , the plurality of storage devices used for tiering can be divided into four layers or levels, including hot, warm, cold, and archive for storing log files based on the hot and cold characteristics of the log files. For example, the storage devices of the tiered storage system 400 can be divided into four layers or levels for storing hot log files 410, warm log files 412, cold log files 414, and archive files 416, respectively.

[0113] In some embodiments, each layer or level of the storage device can be regarded as a virtual pool, and each pool can support multiple physical or virtual file systems (also referred to as storage devices). For example, the tiered storage system 400 may include a first-level pool 402, a second-level pool 404, a third-level pool 406, and a fourth-level pool 408. In some embodiments, the file systems supported in the pool may include one or more of the following short-term file systems: cloud disks (e.g., block devices of virtual machines (VMs) mounted on ext4 / xfs file systems); NAS (e.g., nfs file systems with posix interfaces); OSS low frequency (applicable to virtual file systems, software development kit (SDK) systems, representative state transfer (REST) ​​interface, etc. formats); and OSS archives (applicable to virtual file systems, SDK systems, REST interfaces, etc. formats).

[0114] For example, Figure 4 As shown, the first pool 402 may include ESSD and SSD devices in the cloud storage system (eg, multi-tier storage subsystem 320) storing hot log files 410. The second pool 404 may include NAS devices and cloud disks in the cloud storage system storing warm log files 412.

[0115] The third-level pool 406 may include OSS low-frequency devices that store cold log files 414 in the cloud storage system.

[0116] The fourth-level pool 408 may include an OSS archive device that stores archive files 416 in a cloud storage system.

[0117] In some embodiments, the file system may also include a long-term file system, such as a self-built distributed system (e.g., a distributed storage system 340 built by a client node of a blockchain-based ledger system). For example, the first-level pool 402 may also include an NVME device (e.g., as part of the distributed storage system 340) storing hot log files 410 of a distributed storage system generated by a client node of the blockchain network. The second-level pool 404 may also include an SSD device storing hot log files 412 of a distributed storage system. The third-level pool 406 may also include an HDD device storing cold log files 414 of a distributed storage system. The fourth-level pool 408 may also include an SMR device storing archive files 416 of a distributed storage system. In some embodiments, a unified interface with the entire log structure storage system 300 may be provided for all file systems.

[0118] In some embodiments, the tiered storage system 400 may include one or more subsystems or components, such as (1) a tiered pool manager 418, (2) a migration task manager 420, (3) a migration scheduler 422 for managing data tiering, or (4) a quality of service (QoS) manager 423. In some embodiments, each manager may include any suitable computing element (e.g., one or more of a processor, a memory, etc.) to perform the functions described herein. For example, these managers may manage data flows between multiple storage devices of different performance and cost, for example, by utilizing the performance and cost differences between different storage devices to improve the performance and efficiency of the entire log-structured storage system.

[0119] In some embodiments, the tiered pool manager 418 may be configured to manage each tier of the storage device. In some embodiments, the tiered pool manager 418 may perform one or more of the following functions: managing storage space and pressure of multi-tiered storage devices; providing file creation, deletion, and statistical analysis functions for a specified tier (e.g., selecting a storage device to create a data log file based on a system request); maintaining a tier file mapping table that indicates the correspondence of data files, their storage locations in the corresponding tiers of the storage device, and the hotness or coldness of the data files, etc.

[0120] In some embodiments, the migration task manager 420 can manage bidirectional data migration tasks between different storage devices, manage task lifecycles, call back results, perform statistical analysis, execute migration tasks, and so on.

[0121] In some embodiments, the migration scheduler 422 can support pluggable migration strategies, manage data migration strategies, and provide data creation / query / update / deletion interfaces. In some embodiments, the migration scheduler 422 can perform scheduling management on migration tasks to achieve effective flow control of migration tasks. In some embodiments, the migration scheduler 422 can score data log files at the back end or otherwise assign corresponding scores, and generate migration tasks for data log files based on score rankings and migration strategies. In some embodiments, data log files can be scored according to a scoring formula that takes into account the layer of the storage device, access frequency, original data creation time, migration cost, and / or other factors. In some embodiments, the migration scheduler 422 can work with other subsystems or components of the tiered storage system 400 to quickly verify different migration strategies.

[0122] In some embodiments, data migration can be automatically performed according to a predetermined data migration strategy. For example, different data in a high-speed storage device can be scored according to a predetermined scoring scheme, and migrated to a low-speed device at the back end based on the corresponding scores of the different data to free up cache space. In some embodiments, in some applications, some data in a low-speed device can be determined as hot data. The hot data can be first saved in the disk cache, and if the score of the data log file meets the requirements, it can be migrated to a high-speed device. In some embodiments, after the data file is migrated from the source storage device to the target storage device, the original data may or may not be deleted in the source storage device. For example, if the target storage device is a top-level storage device, it is not necessary to delete the data log file in the disk cache, but it can be automatically replaced with other data.

[0123] In some embodiments, the QoS manager 423 may be configured to manage data flows or other performance indicators of the tiered storage system 400 to improve QoS. For example, in some cases, a burst of I / O writes to a high-speed storage device may cause a high-speed storage device in a higher layer to be heavily occupied or used. In some embodiments, the QoS manager 423 may control the flow of data entering the storage pool at a high utilization rate (e.g., 85% or another threshold) to avoid the storage pool from being filled up too quickly. Flow control can prevent the performance of the tiered storage system 400 from being degraded and can free up storage space for data migration. In order to improve the efficiency of data migration while reducing the impact on the front-end I / O operation (e.g., by the front-end I / O subsystem 310), flow-controlled data migration can be performed at the back end (e.g., by the back-end data management subsystem 330). In some embodiments, the migration speed may be positively correlated with the utilization of the storage device. For example, if the utilization of the storage device is low, flow control may be reduced to avoid excessive impact on the front-end I / O. If the utilization of the storage device is high, flow control may be canceled to speed up data migration.

[0124] In some cases, the high-speed storage device may be fully utilized and front-end I / O operations may be severely limited. In some embodiments, data may be written directly to the lower-tier storage device without migrating the data from the higher-tier storage device to the lower-tier storage device. For example, if Figure 3 If the first-tier storage device 350 in the lower-tier storage device is full or reaches a usage threshold, the data can be written directly to the second-tier storage device 360. In some embodiments, large data (e.g., having a size greater than a threshold) such as blockchain block data can be written directly to the data log file in the lower-tier storage device to save costs caused by data migration.

[0125] In some embodiments, in order to further reduce the consumption of network resources, hard disk throughput, storage space and other resources caused by data migration, and to reduce the impact on front-end I / O operations, compression and erasure coding can be performed by default settings when data is migrated to low-level storage devices.

[0126] Compared to high-speed storage devices, low-speed or archive storage devices have relatively poor performance. Typically, most data is eventually written to the low-speed tier of the storage device. Caching hot data on high-speed storage devices and migrating it to high-speed storage devices based on the hot and cold characteristics of the data helps improve read performance. In some embodiments, two or more of the following caches can be implemented to improve read performance: (1) memory cache (e.g., least recently used (LRU) cache 424); (2) fast disk cache (e.g., least frequently used (LFU) disk cache 426 on a high-speed storage device). In some embodiments, the total size of the memory cache 424 can be dynamically configured, for example, in hundreds of MB to several GB. Similarly, the total size of the fast disk cache can be dynamically configured, for example, in 1 GB to tens of GB.

[0127] In some embodiments, some historical data that has been frequently accessed, such as the genesis block of a blockchain, may be placed in an LFU cache of a fast storage device.

[0128] Data Compression:

[0129] For distributed ledger systems and centralized ledger systems based on blockchain, compression of data blocks can effectively reduce costs and improve the performance of log-structured storage systems. Log structures can facilitate compression in log-structured storage systems due to their inherent characteristics and features.

[0130] In some embodiments, the data written on the front end may not be compressed, and may be written directly to a high-speed storage device (e.g., an SSD cloud disk), for example, by appending the data to a data log file. In some embodiments, when the data log file reaches a certain size, it may be set to be immutable. In some embodiments, the back-end data management subsystem 330 may compress the original data log file on the back end, and replace the original data log file with a compressed data file. In this way, since the compression operation is performed at the back end, the impact of the compression operation on the front-end I / O operation may be reduced or minimized.

[0131] Typically, when determining the compressed size or capacity of a data log file, it may be necessary to consider the effectiveness of compression and read amplification and reach a balance. For example, in some cases, if the compressed size or capacity of a data log file is too small (e.g., less than 4KB), the space saved due to compression may be limited, and the compression performance may not be optimal. On the other hand, if the compressed size or capacity of the data log file is too large, the read amplification may also become larger (e.g., to read a transaction entry, the entire compressed data log file including the transaction entry needs to be decompressed first). In some embodiments, the compressed size of the data log file can be set to 16KB-128KB.

[0132] In some embodiments, the compressed data log file may include multiple records, each of which may include a compression header and a compressed data body. In some embodiments, the metadata of the compressed data may include version information, compression algorithm, length, CRC, etc.

[0133] For encrypted data, the randomness of the encryption itself can make the performance of data compression unsatisfactory. Therefore, in some embodiments, for data that needs to be encrypted (e.g., in a trusted execution environment (TEE)), the data can be compressed before encryption or after decryption.

[0134] In some embodiments, for a compressed log file, the basic index map may encode the physical data size of the compressed log file, modify and record the corresponding index, and record the file ID of the log file, the offset of the log file, and the compressed data size of the log file.

[0135] Erasure Coding

[0136] In some embodiments, the backend data management subsystem 330 may perform erasure coding on the data in the data log file. For example, the backend data management subsystem 330 may write incoming data to the data log file using erasure coding at the backend.

[0137] For a distributed ledger system, in order to implement a Byzantine fault-tolerant log file layer between consensus nodes of a blockchain network, erasure coding can be performed to reduce the amount of cold data stored on each consensus node of the distributed ledger system. For example, for 4 consensus nodes, 4 copies of data can be written before performing erasure coding. After performing erasure coding (e.g., an 8+3 erasure coding scheme), these 4 nodes can store less than 2 copies of data (e.g., 1.375 copies of data).

[0138] For a centralized ledger system based on blockchain, the centralized structure does not require data redundancy due to backup of multiple nodes. In some embodiments, erasure coding can be performed in a hierarchical storage system in a centralized ledger system based on blockchain to reduce data redundancy, for example, in data backup in a top-level storage device or a distributed storage system.

[0139] Data compression

[0140] In some embodiments, transaction data, block data, and historical state data are append-only and cannot be deleted or overwritten, so compaction is not performed on these data. In some embodiments, data compaction can be used to process current state data. Data compaction typically includes garbage collection and data defragmentation.

[0141] In some embodiments, the back-end data management subsystem 330 can sort the data log files according to their respective garbage rates, and arrange them in descending order from high garbage rates to low garbage rates. In some embodiments, the back-end data management subsystem 330 can rewrite data log files with relatively high garbage rates. For example, the back-end data management subsystem 330 can rewrite data log files whose garbage rates exceed a predetermined threshold. In some embodiments, the earlier a data log file is created, the more likely the data in the data log file is to be overwritten, which means that the garbage rate of an earlier data log file may be higher than the garbage rate of a new data log file.

[0142] In some embodiments, the back-end data management subsystem 330 can implement a garbage collection mechanism that can set a maximum amount of data for each rewrite. In some embodiments, for example, multiple real-time instance streams of the front-end I / O subsystem 310 can execute multiple recovery programs in parallel to improve the overall efficiency of garbage collection.

[0143] In some embodiments, the real-time instance flow of the front-end I / O subsystem 310 can obtain the garbage rate and report the obtained garbage rate to the back-end data management subsystem 330, and the back-end data management subsystem 330 can determine the appropriate or optimal flow to rewrite the data.

[0144] Defragmentation is generally the process of locating discontinuous data fragments stored on a storage device, and then rearranging and restoring the fragments into fewer fragments or entire files. Defragmentation can reduce data access time and can use storage more efficiently. In some embodiments, the back-end data management subsystem 330 can perform defragmentation periodically, from time to time, or upon request.

[0145] In some embodiments, for certain types of data, such as world state data or state object data, the keys of these data have a certain level of hashing properties. If the keys have prefixes (e.g., different state objects have different prefixes), compacting such data by putting the data in the same file or adjacent files can improve read performance.

[0146] Status Snapshot

[0147] A state snapshot can capture the state of a system (e.g., a distributed ledger system) at a specific point in time. In some embodiments, the back-end data management subsystem 330 can perform a state snapshot operation to generate and store state data of the log structure storage system 300. In some embodiments, the state data may include historical state data and current state data. The historical state data may include the historical state of the distributed ledger system for backtracking, and the current state data may include the latest state data of the distributed ledger system. Over time, the size of the historical state data may become larger and take up a lot of storage space. In some embodiments, in order to improve historical data backtracking and storage space utilization, the back-end data management subsystem 330 may perform a snapshot operation on the current state data.

[0148] The log structure design of the log structure storage system 300 can facilitate snapshot operations and improve the performance and efficiency of the log structure storage system 300. In some embodiments, snapshot operations can be performed based on a redirect-on-write (ROW) method that provides efficient indexing for data sets corresponding to the snapshot.

[0149] In some embodiments, the snapshot function of the log structure storage system 300 can support flash creation (e.g., in seconds) and rollback, which may have only limited or minimal impact on front-end I / O operations. In some embodiments, the back-end data management subsystem 330 can create hard links between data log files and index log files to avoid data duplication.

[0150] In some embodiments, when data from a write request is stored to a data log file, a record may be generated to include a snapshot identifier (ID) indicating a snapshot version. In some embodiments, the backend data management subsystem 330 may perform one or more of the following operations in response to receiving a state snapshot request:

[0151] (1) Write the operation log (op log) corresponding to the snapshot creation;

[0152] (2) Increase the snapshot version by 1;

[0153] (3) Write all new snapshot write requests to the new memory incremental index map (redirection);

[0154] (4) After all write requests associated with the old snapshot are completed, the index is flushed to the current index log file, all index files are compacted, the index files are merged into a single index log file, and the merged single index log file is set to a sealed state (the data log file is also sealed during the compaction process);

[0155] (5) creating a new index file based on the new snapshot version; and

[0156] (6) Create a directory corresponding to the snapshot and create hard links to the directory for the data log file and index log file associated with the snapshot.

[0157] In some embodiments, the backend data management subsystem 330 may perform a compaction on the backend to restore the deleted snapshot.

[0158] In some embodiments, if a snapshot upload is required, the data structure of the snapshot can be maintained (e.g., a bitmap with 1 bits representing data ranges). For example, in the index log file created when the snapshot is created, the bitmap corresponding to the snapshot can be set to all 0s. After receiving a write request, the bitmap can be updated to 1, indicating that the data in the snapshot version has been modified.

[0159] In some embodiments, a snapshot version number may correspond to an index log file, the snapshot version number indicating write requests corresponding to all indexes in the index log file.

[0160] verify

[0161] In some embodiments, the backend data management subsystem 330 may perform a CRC check on the data recorded in the log file. In some embodiments, the backend data management subsystem 330 may perform a CRC check periodically, from time to time, or upon request.

[0162] In some embodiments, when the index file generated by the backend data management subsystem 330 is imported into the real-time instance stream of the front-end I / O subsystem 310, the memory index map of the real-time instance stream can be updated compared to the index file generated by the backend data management subsystem 330, and can include location information of new data blocks and old data blocks. In some embodiments, the real-time instance stream can traverse the memory base map, replace the corresponding index entries, and then generate a new index log file without reference to the old data log file. Then, the real-time instance stream can safely delete the old data log file and the index log file.

[0163] In some embodiments, in a log structure storage framework (e.g., log structure storage system 300), streams can be used to operate as processing engines, components, units or modules for processing I / O requests. Each stream can be adapted to different business scenarios by different configurations. In some embodiments, streams can be implemented by one or more processors coupled to software to perform operations such as management data log files, index log files, manifest files, request queues, etc. In some embodiments, real-time streams can refer to real-time instances of front-end I / O operations that process front-end I / O subsystems 310. In some embodiments, there may be corresponding dredger streams of data written by real-time streams managed at the back end by the back-end data management subsystem 330.

[0164] In some embodiments, a stream may include a management interface that allows different configurations for different operations such as snapshots, statistics, and fault recovery for different types of data. For example, streams for processing block data, state data, and transaction data may be configured differently depending on the characteristics of the block data, state data, and transaction data. For example, a stream corresponding to a block may be configured to have layering and / or compression capabilities, but not compaction, snapshot, or table capabilities.

[0165] In some embodiments, different types of data can be processed by corresponding customized or otherwise configured streams. For example, a request to write a block can be processed by a stream corresponding to the block.

[0166] In some embodiments, multiple streams can be combined into bundles to provide flexible implementations suitable for specific applications of distributed ledger systems and / or centralized ledger systems based on blockchains. The described technology can support services in distributed ledger systems (e.g., blockchain networks) and / or centralized ledger systems based on blockchains. In some embodiments, the two systems can have different streams that are customized or otherwise configured according to the needs of the two log structure storage systems 300. For example, a distributed ledger system can have four types of data: transaction consensus logs, blocks, states, and indexes. Therefore, the four streams can be configured to process the four types of data separately. A centralized ledger system based on blockchains can have three types of data: transactions, blocks, and indexes, but no state (or complex contract state). Therefore, the three streams can be configured to process the three types of data separately.

[0167] In some embodiments, each stream can be configured to process different types of data. For example, blocks, transaction consensus logs, and indexes do not require snapshots. Therefore, streams for processing blocks, transaction consensus logs, and indexes do not need to be configured with snapshot functions. On the other hand, state data streams can be configured with snapshot functions. As another example, index data is relatively small, but it requires good performance and does not require hierarchical storage. Long-term operations and large amounts of block data may require hierarchical storage, shared storage, and erasure coding.

[0168] In some embodiments, distributed ledger systems and blockchain-based centralized ledger systems may have different requirements for streams to perform operations such as layering, compression, erasure coding, state snapshots, compaction, and data verification.

[0169] Table 1 provides configuration examples for different scenarios. As shown in the figure, "Both" means that operations on specific types of data can be performed on both the distributed ledger system and the centralized ledger system based on blockchain. "DLS" means that operations on specific types of data can be performed only on the distributed ledger system. "Neither" means that operations on specific types of data can be performed on neither the distributed ledger system nor the centralized ledger system based on blockchain.

[0170] Table 1

[0171]

[0172]

[0173] For example, as shown in Table 1, the layering operation on transaction data and / or block data may be performed for the distributed ledger system and the centralized ledger system based on blockchain. The layering operation on current state data and / or historical state may be performed only for the distributed ledger system. The snapshot operation on transaction data may not be performed for both the distributed ledger system and the centralized ledger system based on blockchain.

[0174] In some embodiments, the log structure storage system adopts a multi-threaded fully asynchronous mechanism based on one loop per queue and concurrency for each thread, thereby providing an efficient asynchronous mode and a convenient concurrent synchronous programming mode. In some embodiments, different streams can process different types of data in parallel. For example, a stream configured for block data can write block data to a data log file allocated to store block data, while a stream configured for transaction data can read specific request transaction data from a data log file including transaction data.

[0175] Figure 5 5 is a flow chart showing a process 500 for performing write operations on a log structure storage system according to an embodiment of the present invention. In some embodiments, some or all of the operations of process 500 may be performed by a front-end I / O subsystem (e.g., Figure 3 For convenience, the process 500 will be described as being performed by Figure 3 However, process 500 may be performed by a system of one or more computers located in one or more locations and appropriately programmed according to the present invention. For example, an appropriately programmed data processing and storage system (e.g., Figure 3 The log structure storage system 300) can perform processing 500.

[0176] At 502, a data log file (e.g., data log file 390, 362, 364, 366, 372, 374, or 376) is maintained in a data storage system (e.g., log structure storage system 300). In some embodiments, the data log file may store data including transaction data, block data, state data, and self-describing metadata. For example, the data log file may store blockchain data generated by a blockchain network, including block data, transaction data, and / or state data. In some embodiments, the metadata in the data log file may include metadata describing the data block, such as transaction hash values ​​and sequence values, block hash values ​​and block numbers, snapshot version numbers, cyclic redundancy check (CRC) codes, encryption information, etc. In some embodiments, a data log file stores a single type of blockchain data, so that multiple blockchain data are not mixed in a single data file. For example, the data storage system may maintain one or more of a data log file for transaction data, a data log file for block data, and a data log file for state data. In some embodiments, the data log file may be an append-only file. In some embodiments, the data log file does not store index information. In some embodiments, the data log file may be stored in a multi-tier storage subsystem (eg, multi-tier storage subsystem 320).

[0177] At 504, a front-end I / O subsystem (e.g., front-end I / O subsystem 310) of the data storage system receives a write request for writing data to the data storage system. In some embodiments, the front-end I / O subsystem 310 can process the write operation, including the modification of the data stored on the log structure storage system 300. In some embodiments, the modification of the data is processed by the front-end I / O subsystem 310 so that the original data is not overwritten. Instead, the modification can be processed by adding or appending the data to the data log file in the form of a log.

[0178] At 506, the front-end I / O subsystem 310 appends the data to the data log file. In some embodiments, the data log file may be continuously updated based on the data write or append operation. In some embodiments, the data log file may have a configurable maximum length between 512MB and 2GB, or other sizes depending on the needs of the storage system or the application.

[0179] At 508, the front-end I / O subsystem 310 determines whether the conditions for generating a new data log file are met. In some embodiments, the front-end I / O subsystem 310 may determine whether the data log file has reached a predetermined maximum length or size. If it is determined that the data log file has reached a predetermined maximum length or size, the front-end I / O subsystem 310 may determine that the conditions for generating a new data log file are met. If it is determined that the conditions for generating a new data log file are met, the process proceeds to step 510. If it is determined that the conditions for generating a new data log file are not met, the process returns to step 504.

[0180] At 510, if it is determined that the conditions for generating a new data log file are met, the front-end I / O subsystem 310 seals the data log file. In some embodiments, if it is determined that the conditions for generating a new data log file are met (e.g., the data log file has reached a maximum length or size), the front-end I / O subsystem 310 may set the data log file to a sealed, immutable, or read-only state.

[0181] At 512 , the front-end I / O subsystem 310 generates a new data log file. In some embodiments, the new data log file may also be append-only and stored in the multi-tier storage subsystem 320 .

[0182] In some embodiments, the front-end I / O subsystem 310 may determine the type of data requested to be written in the write request (e.g., transaction data, block data, status data). In response to the determination, the front-end I / O subsystem 310 appends the data to a data log file corresponding to the type of data. In some embodiments, the front-end I / O subsystem 310 may perform some or all of the process 500 using a corresponding processing engine corresponding to the data type.

[0183] For example, in response to determining that the data is transaction data, the front-end I / O subsystem 310 uses a processing engine designated for processing transaction data to append the data to a data log file for transaction data. In some embodiments, in response to determining that the data is block data, the front-end I / O subsystem 310 uses a processing engine designated for processing block data to append the data to a data log file for block data. In some embodiments, in response to determining that the data is state data, the front-end I / O subsystem 310 uses a processing engine designated for processing state data to append the data to a data log file for state data.

[0184] Figure 6600 is a flowchart illustrating a process 600 for generating an index related to a write operation of a log structure storage system according to an embodiment of the present invention. In some embodiments, some or all operations of process 600 may be performed by a front-end I / O subsystem (e.g., Figure 3 For convenience, the process 600 will be described as being performed by Figure 3 However, process 600 may be performed by a system of one or more computers located in one or more locations and appropriately programmed according to the present invention. For example, an appropriately programmed data processing and storage system (e.g., Figure 3 The log structure storage system 300) can perform processing 600.

[0185] At 602, data is successfully written to a storage system (e.g., log-structured storage system 300). In some embodiments, a front-end I / O subsystem (e.g., front-end I / O subsystem 310) of a data storage system may write data chunks to a data log file stored in a multi-tier storage subsystem (e.g., multi-tier storage subsystem 320) of the data storage system.

[0186] At 604, the front-end I / O subsystem 310 generates an index indicating a physical storage location of the data in the log-structured storage system 300. In some embodiments, the index data may include: an index indicating a correspondence from a block hash value to a block number, an index indicating a correspondence from a block hash value to a storage location, an index indicating a correspondence from a transaction hash value to a transaction, and an index indicating a correspondence from a receipt hash value to a receipt. In some embodiments, the index data for a centralized ledger system based on a blockchain may include an index indicating a correspondence from a sequence to a transaction storage location or an index indicating a correspondence from a time sequence to a transaction hash value.

[0187] At 606, the front-end I / O subsystem saves the index to a delta index map (e.g., delta index map 312) in a memory (e.g., memory 315) of the front-end I / O subsystem 310. In some embodiments, the front-end I / O subsystem 310 may include a plurality of memory index maps stored in the memory 315. In some embodiments, the memory index map may be divided into a read-only base index map 316 and a read-write delta index map 312. In some embodiments, the base index map 316 may store an index for cold data (e.g., old data and / or data that is less frequently accessed), while the delta index map 312 may store an index for newly written data.

[0188] At 608, the front-end I / O subsystem 310 determines whether a triggering event occurs. The triggering event may include one or more events that cause the current incremental index map to be sealed and a new incremental index map to be generated. The triggering event may include, for example, the size of the current incremental index map reaches a threshold, the storage usage of the memory 315 reaches a threshold, or a specified time arrives (for example, the log structure storage system 300 may periodically seal the incremental index map). If it is determined that the triggering event occurs, the process proceeds to step 610. If it is determined that the triggering event does not occur, the process returns to step 602.

[0189] At 610, if it is determined that a triggering event occurs, the front-end I / O subsystem 310 sets the incremental index map 312 to be immutable. In some embodiments, the front-end I / O subsystem may set the incremental index map 312 in the memory 315 to be an immutable index map 314, flush them to an index log file (e.g., index log file 380), and create a new incremental index map 312 to receive the index generated according to the new write request.

[0190] At 612, the index log file 380 is maintained in the storage system 300. In some embodiments, the indexes of the newly written transaction data and block data may be added to the index maps 312 and 316 and the index log file 390, but the indexes of the existing transaction data and block data may not be modified. In some embodiments, the index log file 390 may be stored in the multi-layer storage subsystem 320 together with the data log file.

[0191] At 614, the front-end I / O subsystem 310 flushes the incremental index map 312 to the index log file 380, for example, to release the memory used by the incremental index map 312. In some embodiments, the front-end I / O subsystem 310 can create a new incremental index map 312 to receive the index generated according to the new request. In some embodiments, the front-end I / O subsystem 310 can combine the incremental index map 312 and the basic index map 316, and generate a new basic index map 316, and flush the generated basic index map 316 to the index log file 380.

[0192] In some embodiments, the front-end I / O subsystem 310 saves the index of the hot data in a memory cache (e.g., memory cache 313) at 616. For example, if specific data is determined to be hot data with a likelihood of being frequently accessed, the index of the data may be saved in the memory cache to increase the read speed.

[0193] At 618, the front-end I / O subsystem 310 determines whether the conditions for generating a new index log file 380 are met. In some embodiments, the conditions for generating a new index log file may include the maximum length or size of the index log file 380. In some embodiments, the conditions for generating a new index log file 380 may include the number of batches of write operations performed by the front-end I / O subsystem. For example, in some embodiments, the index log file 380 can be updated by batch processing an index of write operations. In some embodiments, when the index log file 380 has been updated for a specific number of batches of write operations, the index log file 380 can be sealed or set to read-only, and a new index log file 380 can be created to write new data. If it is determined that the conditions for generating a new index log file 380 are met, processing proceeds to step 620.

[0194] At 620, if it is determined that the conditions for generating a new index log file 380 are met, the front-end I / O subsystem 310 seals the index log file. For example, when the index log file 380 has reached a maximum length or size, or has been updated for a specific number of batches of write operations, the index log file 380 can be sealed or set to read-only.

[0195] At 622 , after sealing the old index log file 380 , the front-end I / O subsystem 310 generates a new index log file 380 to store subsequent index data.

[0196] Figure 7 700 is a flowchart illustrating a process 700 for performing a read operation of a log structure storage system according to an embodiment of the present invention. In some embodiments, some or all operations of process 700 may be performed by a front-end I / O subsystem (e.g., Figure 3 For convenience, process 700 will be described as being performed by front-end I / O subsystem 310. However, process 700 may be performed by a system of one or more computers located in one or more locations and appropriately programmed according to the present invention. For example, an appropriately programmed data processing and storage system (e.g., Figure 3 The log structure storage system 300) can perform processing 700.

[0197] At 702, a front-end I / O subsystem (eg, front-end I / O subsystem 310) of a storage system (eg, log-structured storage system 300) receives a read request to read data from the storage system.

[0198] At 704, the front-end I / O subsystem 310 searches for an index corresponding to the data in an incremental index map (e.g., incremental index map 312) in a memory (e.g., memory 315) of the front-end I / O subsystem 310. In some embodiments, the index corresponding to the data may include physical location information of the data. In some embodiments, the memory 315 of the front-end I / O subsystem 310 may store a plurality of memory index maps including a read-only basic index map 316 and a read-write incremental index map 312.

[0199] At 706, the front-end I / O subsystem 310 determines whether an index corresponding to the data is found in the incremental index map 312. If an index corresponding to the data is found in the incremental index map 312, the process proceeds to step 708, where the front-end I / O subsystem 310 can locate the data based on the physical location indicated by the index. If an index corresponding to the data is not found in the incremental index map 312, the process proceeds to step 710.

[0200] At 710 , if it is determined that the index corresponding to the data is not found in the incremental index map 312 , the front-end I / O subsystem 310 searches for the index corresponding to the data in the basic index map 316 in the memory 315 .

[0201] At 712, the front-end I / O subsystem 310 determines whether an index corresponding to the data is found in the basic index map 316. If it is determined that an index corresponding to the data is found in the basic index map 316, the process proceeds to step 714, where the front-end I / O subsystem 310 can locate the data based on the physical location information indicated by the index. If an index corresponding to the data is not found in the basic index map 316, the process proceeds to step 716.

[0202] At 716, if it is determined that the index corresponding to the data is not found in the basic index map 316, the front-end I / O subsystem 310 searches for the index corresponding to the data in the index log file (e.g., index log file 380) in the disk storage device. For example, the front-end I / O subsystem 310 may search for the index corresponding to the data in the index log file 380 stored in the multi-layer storage subsystem (e.g., multi-layer storage subsystem 320) of the storage system 300.

[0203] In some embodiments, the front-end I / O subsystem 310 may determine the type of data requested to be read in the read request (e.g., transaction data, block data, status data). In response to the determination, the front-end I / O subsystem 310 may use a corresponding processing engine corresponding to the data type to perform some or all of the process 700.

[0204] Figure 8 800 is a flowchart illustrating a process 800 for improving read operations of a log structured storage system according to an embodiment of the present invention. In some embodiments, some or all operations of process 800 may be performed by a log structured storage system (e.g., Figure 3 800). For convenience, process 800 will be described as being performed by log structured storage system 300. However, process 800 may be performed by a system of one or more computers located in one or more locations and appropriately programmed according to the present invention. For example, an appropriately programmed data processing and storage system may perform process 800.

[0205] At 802, a storage system (e.g., a log-structured storage system 300 or a tiered storage system 400) maintains multiple layers of storage devices (e.g., storage devices 350, 360, and 370 of a multi-layer storage subsystem 320) and one or more layers of cache (e.g., a memory cache 313 and a block cache 317). In some embodiments, the multiple layers of storage devices are tiered in a hierarchical structure based on one or more performance characteristics (e.g., access speed, access bandwidth, or access latency). For example, the multiple layers of storage devices may include multiple storage devices having different access speeds and storing data having different characteristics. For example, a first-tier storage device may store blockchain data that is more frequently accessed than blockchain data stored in a second-tier storage device.

[0206] At 804, for example, a front-end I / O subsystem (e.g., front-end I / O subsystem 310) or a back-end data management system (e.g., back-end data management subsystem 330 of storage system 300) determines that a data object in a data log file (e.g., data log files 362, 364, 366, 372, 374, 376, 390) stored in a lower-tier storage device (e.g., storage devices 350, 360, and 370) is an active data object. In some embodiments, the data object may include transaction data, block data, and state data. In some embodiments, for example, if the data object has been recently accessed a certain number of times (e.g., accessed a certain number of times within a predetermined time window) or if the data object has been identified as having a particular priority, the data object may be determined to be an active data object based on one or more activity or heat policies.

[0207] At 806 , the data object is written to a cache (eg, the memory cache 313 and the block cache 317 ). For example, the front-end I / O subsystem 310 may write the data object to the memory cache 313 or the disk block cache 317 of a high-speed storage medium.

[0208] At 808, an index is generated that indicates the physical storage location of the data object in the cache. In some embodiments, the index data in the memory 315 may be organized using an LSM method.

[0209] At 810, the index may be saved to a delta index map (e.g., delta index map 312) in memory 315. In some embodiments, memory 315 may maintain multiple in-memory index maps, including a read-only base index map 316 and a read-write delta index map 312. In some embodiments, delta index map 312 may be configured to store indexes for data that are more frequently accessed and / or updated than the indexes stored in base index map 316.

[0210] At 812 , the front-end I / O subsystem 310 receives a read request for a data object.

[0211] At 814, the front-end I / O subsystem 310 searches for an index corresponding to the data object in the incremental index map 312 in the memory 315. In some embodiments, the front-end I / O subsystem 310 may first search the incremental index map 312. If the index is not found in the incremental index map 312, the front-end I / O subsystem 310 may then search the base index map 316 for an index corresponding to the data.

[0212] At 816, the front-end I / O subsystem 310 returns the data object from the cache, which may provide faster access to the requested data object than if the requested data object had to be obtained from the next cache level, main memory, or a lower layer of storage devices of the multi-layer storage subsystem 320. For example, if the front-end I / O subsystem 310 successfully identifies an index corresponding to the data in the delta index map 312 or the base index map 316, the front-end I / O subsystem 310 may use the index to identify the physical location of the data in the cache and retrieve the data from the cache.

[0213] In some embodiments, the front-end I / O subsystem 310 may determine the type of data requested to be read in the read request (e.g., transaction data, block data, status data). In response to the determination, the front-end I / O subsystem 310 may use a corresponding processing engine corresponding to the data type to perform some or all of the process 800.

[0214] Fig. 9 900 is a flowchart illustrating a process 900 for managing data log files stored in a log structure storage system according to an embodiment of the present invention. In some embodiments, some or all of the operations of process 900 may be performed by a backend data management system (e.g., Figure 3900 is an example of a rewrite placement program executed by the backend data management subsystem 330 of the log structure storage system 300. For convenience, the process 900 will be described as being performed by the backend data management subsystem 330. However, the process 900 can be performed by a system of one or more computers located in one or more locations and appropriately programmed according to the present invention. For example, a suitably programmed data processing and storage system (e.g., Figure 3 The log structure storage system 300) can perform processing 900.

[0215] At 902, the backend data management system (e.g., the backend data management subsystem 330) determines information of data log files (e.g., data log files 390, 362, 364, 366, 372, 374, and 376) stored in storage devices (e.g., storage devices 350, 360, and 370) based on the index log file (e.g., the index log file 380). In some embodiments, the information of the data log files in the storage device may include one or more of the activity (e.g., access frequency), size, garbage rate, or fragmentation level of the data log files.

[0216] At 904, the backend data management system 330 determines the information of the storage device. In some embodiments, the information of the storage device may include one or more of a usage rate, a garbage rate, a fragmentation level, or an input / output (I / O) request of the storage device.

[0217] At 906, the backend data management system 330 determines whether the data log file needs to be rewritten. In some embodiments, the backend data management subsystem 330 can determine the rewrite placement based on the information of the data log file stored in the storage device and / or the information of the storage device. In some embodiments, the rewrite placement can include at least one of layering, compression, erasure coding, state snapshot, compression, or verification. If it is determined that the data log file needs to be rewritten, the process proceeds to step 908. If it is determined that the data log file does not need to be rewritten, the process returns to step 902.

[0218] At 908 , the backend data management system 330 reads the data log file from the source location, and if it is determined that the data log file needs to be rewritten, rewrites the data log file to the target location.

[0219] At 910, the backend data management system 330 seals the data log file in the target location. For example, the backend data management system 330 can set the data log file to a sealed state or read-only after the rewrite placement is completed.

[0220] At 912, the backend data management system 330 generates a target index log file corresponding to the data log file in the target location. In some embodiments, the target index log file may include a list of data log files that can be safely deleted and / or a list of data log files referenced by the target index log file.

[0221] At 914, the backend data management system 330 seals the target index log file. For example, the backend data management system 330 can set the target index log file to be immutable or read-only.

[0222] The target index log file is imported into a readable index map in memory at 916. For example, the target index log file may be imported into a delta index map or a base index map so that the data log file in the target location may be addressed or read.

[0223] Fig.10 1 is a flow chart showing a process 1000 for performing data migration in a log-structured storage system according to an embodiment of the present invention. In some embodiments, some or all operations of the process 1000 may be performed by a backend data management system (e.g., Figure 3 1000) is an example of a tiering / migration procedure performed by a backend data management subsystem 330 of a log structured storage system 300. For convenience, process 1000 will be described as being performed by backend data management subsystem 330. However, process 1000 may be performed by a system of one or more computers located in one or more locations and appropriately programmed in accordance with the present disclosure. For example, an appropriately programmed data processing and storage system (e.g., Figure 3 The log structure storage system 300) can perform processing 1000.

[0224] At 1002, a backend data management system (e.g., backend data management subsystem 330) identifies one or more characteristics of data log files (e.g., data log files 390, 362, 364, 366, 372, 374, and 376) and one or more characteristics of storage devices (e.g., storage devices 350, 360, and 370). In some embodiments, the one or more characteristics of the data log files may include data types (e.g., block data, state data, and transaction data), creation time, data size, activity, garbage rate, or fragmentation level of the data log files, etc. In some embodiments, the one or more characteristics of the storage devices may include access speed, access bandwidth, access latency, usage rate, garbage rate, fragmentation level, or input / output (I / O) requests of the storage devices.

[0225] At 1004, the backend data management system 330 determines a migration metric for the data log file based on the characteristics. In some embodiments, the backend data management system 330 may assign a score to the data log file and generate a migration task based on the score ranking and a predetermined migration strategy. In some embodiments, the data log file may be scored or assigned a score based on a scoring formula that takes into account media level, access frequency, original data creation time, and migration cost, etc.

[0226] At 1006, the backend data management system 330 determines whether to migrate the data log file. For example, the data log file may be scored according to a predetermined scoring scheme. If the score of the data log file exceeds a predetermined threshold, the backend data management system 330 may determine that the data log file needs to be migrated. If it is determined that the data log file needs to be migrated, the process proceeds to step 1008. If it is determined that the data log file does not need to be migrated, the process returns to step 1002.

[0227] At 1008, if it is determined that the data log file needs to be migrated, the back-end data management system 330 migrates the data log file from the source location to the target storage device. In some embodiments, the data log files in the high-speed storage device can be scored according to a predetermined scoring scheme, and migrated to the low-speed storage device based on the score (for example, after sorting or ranking the scores of the data log files) to free up storage space. In some embodiments, the hot data in the data log file stored in the low-speed storage device can be first saved in the disk cache, and if the score of the data log file reaches a predetermined threshold, it can be migrated to the high-speed storage device.

[0228] Fig.11 1 is a flow chart illustrating a process 1100 for performing data flow control in a log structured storage system according to an embodiment of the present invention. In some embodiments, some or all of the operations of process 1100 may be performed by a log structured storage system (e.g., Figure 3 1100). For convenience, process 1100 will be described as being performed by a log structured storage system. However, process 1100 may be performed by a system of one or more computers located in one or more locations and appropriately programmed according to the present invention. For example, an appropriately programmed data processing and storage system may perform process 1100.

[0229] At 1102, a storage system (e.g., log-structured storage system 300) maintains a multi-tier storage device (e.g., storage devices 350, 360, and 370). In some embodiments, the multi-tier storage device is tiered in a tiered structure based on one or more performance characteristics (e.g., access speed, access bandwidth, or access latency). For example, the multi-tier storage device may include a plurality of storage devices having different access speeds, and may store data having corresponding characteristics (e.g., a first tier storage device may store blockchain data that is more frequently accessed than blockchain data stored in a second tier storage device).

[0230] In some embodiments, the log-structured storage system 300 may assign flow control policies to multi-layer storage devices. For example, the log-structured storage system 300 may assign a first flow control policy to a first-layer storage device and a second flow control policy to a second-layer storage device based on one or more characteristics of the first-layer storage device and the second-layer storage device (e.g., access speed, access bandwidth, access latency, usage rate, garbage rate, and fragmentation level). In some embodiments, the first flow control policy may include one or more of the following: a first speed at which data is written to the first-layer storage device, or one or more first thresholds for adjusting the first speed at which data is written to the first-layer storage device, and the second flow control policy may include one or more of the following: a second speed at which data is written to the first-layer storage device, or one or more second thresholds for adjusting the second speed at which data is written to the second-layer storage device.

[0231] At 1104, the log-structured storage system 300 receives a write request for ledger data. In some embodiments, the ledger data may include blockchain data such as transaction data, block data, and state data.

[0232] At 1106, the log-structured storage system 300 identifies the type of the ledger data. For example, the backend data management subsystem 330 may determine whether the ledger data is transaction data, block data, or state data. At 1108, the log-structured storage system 300 determines whether the ledger data is block data. In some embodiments, block data has a larger size than other types of blockchain data (e.g., transaction data, state data, or index data) and may have a greater impact on the throughput of I / O operations of the log-structured storage system 300. If it is determined that the ledger data is block data, the process proceeds to step 1110, where the backend data management subsystem 330 writes the data directly to a second-tier storage device (e.g., storage device 360) and skips the first-tier storage device, for example, to save the cost of performing migration later. In some embodiments, the second-tier storage device is located at a lower layer than the first-tier storage device. For example, the access speed of the second-tier storage device may be lower than the access speed of the first-tier storage device. In some embodiments, the cost of the second-tier storage device may be lower than the cost of the first-tier storage device. In some embodiments, the storage space of the second-tier storage device may be larger than the storage space of the first-tier storage device. If it is determined that the ledger data is not block data, the process proceeds to step 1112 .

[0233] At 1112 , if it is determined that the ledger data is not block data, the log-structured storage system 300 determines the usage rate of the first-tier storage device.

[0234] At 1114, the log-structured storage system 300 determines whether the usage rate reaches or exceeds a predetermined threshold. In some embodiments, the predetermined threshold is used to determine whether the first-tier storage device is substantially full. For example, if it is determined that the usage rate reaches or exceeds a threshold (e.g., 85%), the log-structured storage system 300 can determine that the first-tier storage device is substantially full. If it is determined that the usage rate reaches or exceeds the predetermined threshold, the process proceeds to step 1116, where the data is written to the second-tier storage device. If it is determined that the usage rate is below the predetermined threshold, the process proceeds to step 1118.

[0235] At 1118, the log-structured storage system 300 writes the data to the first-tier storage device if it is determined that the usage rate is lower than a predetermined threshold.

[0236] In some embodiments, the log structure storage system 300 can adjust the speed of writing data to the first-tier storage device based on the utilization rate of the first-tier storage device. For example, if it is determined that the utilization rate of the first-tier storage device reaches or exceeds a first predetermined threshold (e.g., 65%), the log structure storage system 300 can reduce the speed of writing data to the first-tier storage device. In some embodiments, the log structure storage system 300 can reduce the speed of writing data to the first-tier storage device based on the utilization rate of the first-tier storage device. In some embodiments, the log structure storage system 300 can continue to reduce the speed of writing data to the first-tier storage device as the utilization rate of the first-tier storage device increases. For example, when the utilization rate of the first-tier storage device is a first value (e.g., 70%), the log structure storage system 300 can reduce the speed of writing data to the first-tier storage device to a first rate (e.g., 500MB / s), and when the utilization rate of the first-tier storage device is a second value (e.g., 75%) greater than the first value, the speed of writing data to the first-tier storage device is reduced to a second rate (e.g., 400MB / s) lower than the first rate.

[0237] In some embodiments, if it is determined that the utilization rate of the first-tier storage device is lower than a second predetermined threshold value (e.g., 35%), the log-structured storage system 300 may increase the speed of writing data to the first-tier storage device. In some embodiments, the log-structured storage system 300 may increase the speed of writing data to the first-tier storage device based on the utilization rate of the first-tier storage device. In some embodiments, the log-structured storage system 300 may continue to increase the speed of writing data to the first-tier storage device as the utilization rate of the first-tier storage device decreases. For example, when the utilization rate of the first-tier storage device is a third value (e.g., 30%), the log-structured storage system 300 may increase the speed of writing data to the first-tier storage device to a third rate (e.g., 550 MB / s), and when the utilization rate of the first-tier storage device is a fourth value (e.g., 20%) that is smaller than the third value, the speed of writing data to the first-tier storage device is increased to a fourth rate (e.g., 600 MB / s) that is higher than the third rate.

[0238] Fig.12 1 is a flow chart illustrating a process 1200 that may be performed according to embodiments of the present invention. For convenience, the process 1200 will be described as consisting of Figure 3 The log structure storage system 300 of the embodiment of the present invention is performed. However, the process 1200 can be performed by a system of one or more computers located in one or more locations and is appropriately programmed according to the present invention. For example, a suitably programmed data processing and storage system can perform the process 1200.

[0239] At 1202, a storage system (e.g., log-structured storage system 300) receives configurations of multiple processing engines. In some embodiments, the configuration may configure a corresponding processing engine type for processing each data type in a plurality of data types of a distributed ledger system, such as in Table 1, based on characteristics of such data types. In some embodiments, storage system 300 may include a processing engine type designated for processing block data; a processing engine type designated for processing transaction data; a processing engine type designated for processing state data; and a processing engine type designated for processing index data.

[0240] In some embodiments, the state data may include current state data and historical state data, and storage system 300 may include a processing engine type designated for processing the current state data and a processing engine type designated for processing the historical state data.

[0241] At 1204, the storage system 300 receives a processing request for data of the distributed ledger system. In some embodiments, the data types of the distributed ledger system may include block data, transaction data, state data, and index data.

[0242] In some embodiments, the storage system 300 may receive an I / O request for data of the distributed ledger system. In some embodiments, the corresponding processing engine type designated for processing a data type of the distributed ledger system may include a corresponding I / O processing engine type designated for performing a read or write operation on a data type of the distributed ledger system.

[0243] In some embodiments, the storage system 300 may receive a data management request for data of the distributed ledger system. In some embodiments, the corresponding processing engine type designated for processing a data type may include a corresponding data management processing engine type designated for performing a data management operation on a data type in the storage system. In some embodiments, the management operation includes one or more of tiering, compaction, compression, erasure coding, or snapshot.

[0244] At 1206, the storage system 300 determines that the data belongs to one of a plurality of data types of the distributed ledger system. In some embodiments, the one data type may be block data or transaction data. In some embodiments, the one data type may be state data.

[0245] At 1208, the storage system 300 applies a processing engine type designated for processing the one data type. In some embodiments, the processing engine type designated for processing the one data type may be configured to have functions including layering, erasure coding, and compression of block data or transaction data in the storage system 300. In some embodiments, the processing engine type designated for processing the one data type may be configured to have functions including snapshotting and compressing state data in the storage system 300.

[0246] Fig.13 1300 is a flow chart illustrating a process 1300 that may be performed according to embodiments of the present invention. For convenience, the process 1300 will be described as consisting of Figure 3 The log structure storage system 300 of the embodiment of the present invention is performed. However, the process 1300 can be performed by a system of one or more computers located in one or more locations and is appropriately programmed according to the present invention. For example, an appropriately programmed data processing and storage system can perform the process 1300.

[0247] At 1302, a storage system (e.g., log-structured storage system 300) receives configurations of multiple processing engines. In some embodiments, the configurations may configure corresponding processing engine types for processing such data types based on the characteristics of each of the multiple data types in the centralized ledger system based on blockchain, such as Table 1.

[0248] In some embodiments, storage system 300 may include a processing engine type designated for processing block data; a processing engine type designated for processing transaction data; and a processing engine type designated for processing index data.

[0249] At 1304, the storage system 300 receives a processing request for data of a centralized ledger system based on blockchain. In some embodiments, the data types of the centralized ledger system based on blockchain may include block data, transaction data, and index data.

[0250] In some embodiments, the storage system 300 may receive an I / O request for data of a centralized ledger system based on blockchain. In some embodiments, the corresponding processing engine type designated for processing a data type of a centralized ledger system based on blockchain may include a corresponding I / O processing engine type designated for performing a read or write operation on the data type of the centralized ledger system based on blockchain. For example, according to some or all of the operations of processing 500, 600, 700, 1100, and 1400, the data of the centralized ledger system based on blockchain is analyzed.

[0251] In some embodiments, the storage system 300 may receive a data management request for data of a centralized ledger system based on a blockchain. In some embodiments, the corresponding processing engine type designated to process the one data type may include a corresponding data management processing engine type designated to perform a data management operation on the one data type in the storage system. In some embodiments, the management operation may include one or more of tiering, compaction, compression, erasure coding, or snapshots.

[0252] At 1306, the storage system 300 determines that the data belongs to a data type in a centralized ledger system based on blockchain. In some embodiments, the data type may be block data or transaction data.

[0253] At 1308, the storage system 300 applies a processing engine type designated for processing a data type according to characteristics of the data type. In some embodiments, the processing engine type designated for processing the data type may be configured to have functions including layering, erasure coding, and compression of block data or transaction data in the storage system. In some embodiments, the storage system 300 applies a processing engine type designated for processing the data type according to some or all of the operations of processes 800, 900, 1000, and 1400.

[0254] Fig.14 1400 is a flowchart illustrating a process 1400 that may be performed according to embodiments of the present invention. For convenience, the process 1400 will be described as consisting of Figure 3 The log structure storage system 300 of the embodiment of the present invention is performed. However, the process 1400 can be performed by a system of one or more computers located in one or more locations and is appropriately programmed according to the present invention. For example, a suitably programmed data processing and storage system can perform the process 1400.

[0255] At 1402, a storage system (e.g., log-structured storage system 300) receives a plurality of write requests for storing a corresponding plurality of blockchain data in the storage system. In some embodiments, each of the plurality of blockchain data may include a value of one or more of a block, a transaction, or a state of a blockchain network and a key corresponding to the value. In some embodiments, the key may include a hash value corresponding to the value.

[0256] At 1404, the storage system 300 appends the plurality of blockchain data to a data log file (e.g., data log files 390, 362, 364, 366, 372, 374, and 376) according to the time order of the plurality of blockchain data. For example, the blockchain data received later will be appended to the previously received data stored in the data log file. In some embodiments, the data log file may be an append-only file. In some embodiments, the data log file may be stored in a first-tier storage device (e.g., storage device 350) in a multi-tier storage subsystem (e.g., multi-tier storage subsystem 320) including multi-tier storage devices of the log-structured storage system 300, and the access speed of the first-tier storage device is the highest among the multi-tier storage devices.

[0257] At 1406, the log-structured storage system 300 is restricted to sorting the plurality of blockchain data in the data log file according to any other metric, such as according to corresponding keys (e.g., in KVP) of values ​​in the plurality of blockchain data. In some embodiments, unlike existing storage systems that rearrange the plurality of blockchain data in the data log file according to corresponding keys of values ​​in the plurality of blockchain data, the plurality of blockchain data in the data log file of the log-structured storage system 300 is arranged according to the time when the log-structured storage system 300 generates or receives the plurality of blockchain data. At 1408, the log-structured storage system 300 generates an index indicating corresponding physical storage locations of the plurality of blockchain data in the data log file, for example, according to corresponding operations of process 600.

[0258] At 1410 , the log-structured storage system 300 writes the index into the first-tier storage device, for example according to the corresponding operation of process 600 .

[0259] At 1412, the log-structured storage system 300 determines corresponding migration priorities, scores, or metrics for the plurality of blockchain data, for example, based on corresponding operations of the process 1000. In some embodiments, the log-structured storage system 300 determines corresponding migration priorities based on a time sequence of the plurality of blockchain data. In some embodiments, the migration priority of older blockchain data may be higher than the migration priority of newer blockchain data.

[0260] At 1414, the log structure storage system 300 migrates the plurality of blockchain data stored in the first-layer storage device to the second-layer storage device (e.g., storage device 360) according to the corresponding migration priority. In some embodiments, the access speed of the second-layer storage device may be lower than the access speed of the first-layer storage device.

[0261] Fig.15 An example of a module of an apparatus 1500 according to an embodiment of the present invention is depicted. The apparatus 1500 may be a storage system (e.g., Figure 3 The device 1500 may correspond to the above embodiment, and the device 1500 includes: a receiving module 1502, which is used to receive multiple write requests for storing corresponding multiple blockchain data in the storage system, wherein the multiple blockchain data each includes: a value of one or more of a block, a transaction, or a state of a blockchain network, and a key corresponding to the value; an appending module 1504, which is used to append the multiple blockchain data to the data log file according to the time sequence of the multiple blockchain data; and a restriction module 1506, which is used to restrict the sorting of the multiple blockchain data in the data log file according to the corresponding key of the value in the multiple blockchain data.

[0262] In an alternative embodiment, the key comprises a hash value corresponding to the value.

[0263] In an alternative embodiment, the data log file is an append-only file.

[0264] In an optional embodiment, the data log file is stored in a first-tier storage device in a multi-tier storage subsystem of the storage system including multi-tier storage devices, and the access speed of the first-tier storage device is the highest among the multi-tier storage devices.

[0265] In an optional embodiment, the apparatus 1500 further includes: a generating submodule, configured to generate an index indicating corresponding physical storage locations of a plurality of blockchain data in a data log file; and a writing submodule, configured to write the index into a first-layer storage device.

[0266] In an optional embodiment, the device 1500 also includes: a determination submodule, used to determine the corresponding migration priorities of multiple blockchain data according to the time sequence of the multiple blockchain data; and a migration submodule, used to migrate the multiple blockchain data stored in the first-layer storage device to the second-layer storage device according to the corresponding migration priorities, wherein the access speed of the second-layer storage device is lower than the access speed of the first-layer storage device.

[0267] In an optional embodiment, the migration priority of older blockchain data is higher than the migration priority of newer blockchain data.

[0268] The systems, devices, modules or units shown in the previous implementations may be implemented by using a computer chip or entity, or may be implemented by using a product having specific functions. A typical implementation device is a computer, which may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email receiving and sending device, a game console, a tablet computer, a wearable device or any combination of these devices.

[0269] For the implementation process of the functions and roles of each unit in the device, reference may be made to the implementation process of the corresponding steps in the previous method. For simplicity, the details are omitted here.

[0270] Since the device embodiment corresponds substantially to the method embodiment, for the relevant parts, reference may be made to the relevant description in the method embodiment. The aforementioned device embodiment is merely an example. The modules described as separate parts may or may not be physically separated, and the parts shown as modules may or may not be physical modules, may be located in one location, or may be distributed on multiple network modules. Some or all modules may be selected based on actual needs to achieve the goal of the scheme herein. Those of ordinary skill in the art can understand and implement the embodiments of the present application without making creative efforts.

[0271] See again Fig.15 , which can be interpreted as showing the internal functional modules and structure of the data processing and storage device. The data processing and storage device can be a log structure storage system (e.g., Figure 3 In essence, the execution subject can be an electronic device that includes the following: one or more processors; one or more computer-readable memories configured to store executable instructions for one or more processors. In some embodiments, the one or more computer-readable memories are coupled to one or more processors and have program instructions stored thereon, which can be executed by one or more processors to perform the algorithms, methods, functions, processes, procedures, and programs described herein.

[0272] Embodiments of the described subject matter may include one or more features, individually or in combination. For example, in a first embodiment, a method includes: a storage system receives multiple write requests for storing corresponding multiple blockchain data in the storage system. The multiple blockchain data each include: a value of one or more of a block, a transaction, or a state of a blockchain network, and a key corresponding to the value. The storage system appends the multiple blockchain data to a data log file according to a time order of the multiple blockchain data, and limits the sorting of the multiple blockchain data in the data log file according to the corresponding keys of the values ​​in the multiple blockchain data.

[0273] The foregoing and other described embodiments may each optionally include one or more of the following features:

[0274] The first feature, which can be combined with any of the following features, specifies that the key includes a hash value corresponding to the value.

[0275] The second feature, which can be combined with any of the previous or following features, specifies that the data log file is an append-only file.

[0276] The third feature, which can be combined with any previous or following feature, specifies that the data log file is stored in a first-tier storage device in a multi-tier storage subsystem of the storage system including multi-tier storage devices, and the access speed of the first-tier storage device is the highest among the multi-tier storage devices.

[0277] The fourth feature, which can be combined with any previous or following features, specifies that the method also includes: the storage system generates an index indicating the corresponding physical storage locations of the plurality of blockchain data in the data log file; and the storage system writes the index to the first-tier storage device.

[0278] The fifth feature, which can be combined with any previous or following feature, specifies that the method also includes: determining corresponding migration priorities of multiple blockchain data according to the time order of the multiple blockchain data; and migrating the multiple blockchain data stored in the first-layer storage device to the second-layer storage device according to the corresponding migration priorities, wherein the access speed of the second-layer storage device is lower than the access speed of the first-layer storage device.

[0279] The sixth feature, which can be combined with any of the previous or following features, specifies that the migration priority of older blockchain data is higher than the migration priority of newer blockchain data.

[0280] The embodiments of the subject matter, actions and operations described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described herein may be implemented as one or more computer programs, for example, one or more computer program instruction modules, encoded on a computer program carrier, for operation of data processing or control by a data processing device. For example, a computer program carrier may include one or more computer-readable storage media with instructions encoded or stored thereon. The carrier may be a tangible, non-transitory computer-readable medium, such as a magnetic disk, a magneto-optical disk or an optical disk, a solid-state drive, a random access memory (RAM), a read-only memory (ROM), or other media types. Alternatively or additionally, the carrier may be an artificially generated propagation signal, for example, a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be or may be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagation signal.

[0281] A computer program may also be referred to or described as a program, software, software application, app, module, software module, engine, script or code, and may be written in any form of programming language, including compiled or deductive languages, declarative or procedural languages; it may be configured in any form, including as a stand-alone program, or as a module, component, engine, subroutine or other unit suitable for execution in a computing environment, which may include one or more computers at one or more locations interconnected by a data communications network.

[0282] A computer program may, but need not, correspond to a file in a file system. A computer program may be stored in: a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document; a single file dedicated to the program in question; or multiple coordinated files, such as multiple files storing one or more modules, subroutines, or code portions.

[0283] For example, processors for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions for executing a computer program and receive data from a non-transitory computer readable medium coupled to the processor.

[0284] The term "data processing apparatus" includes all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The data processing apparatus may include special-purpose logic circuits such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit) or a GPU (graphics processing unit). In addition to hardware, the apparatus may also include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0285] The processes and logic flows described herein may be performed by one or more computers or processors executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows may also be performed by a special purpose logic circuit such as an FPGA, ASIC, or GPU, or a combination of a special purpose logic circuit and one or more programmed computers.

[0286] Computers suitable for executing computer programs may be based on general and / or special purpose microprocessors, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The elements of a computer may include a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented with or integrated in a special purpose logic circuit.

[0287] Typically, the computer will also include or be operably coupled to one or more mass storage devices to receive data from or transfer data to one or more mass storage devices. The storage device may be, for example, a magnetic disk, a magneto-optical disk or an optical disk, a solid-state drive, or any other type of non-transient computer-readable medium. However, the computer does not need to have such a device. Therefore, the computer can be coupled to one or more storage devices such as one or more memories, local and / or remote. For example, the computer may include one or more local memories as an integrated component of the computer, or the computer may be coupled to one or more remote memories in a cloud network. In addition, the computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0288] Components may be "coupled" to one another by being communicatively connected, such as directly, or by being electrically or optically connected to one another through one or more intermediate components. Components may also be "coupled" to one another if one of the components is integrated into another component. For example, a storage component integrated into a processor (e.g., an L2 cache component) is "coupled to" the processor.

[0289] To provide interaction with a user, embodiments of the subject matter described herein may be implemented on or configured to communicate with a computer having: a display device, such as an LCD (liquid crystal display) monitor, for displaying information to a user; and an input device through which a user can provide input to the computer, such as a keyboard and a pointer device such as a mouse, trackball, or touchpad. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form of input from the user may be received, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser, or by interacting with an application (app) running on a user's device, such as a smart phone or electronic tablet. In addition, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application) in turn and receiving a response message from the user.

[0290] The term "configured to" is used herein in relation to systems, devices and computer program components. For a system of one or more computers configured to perform a specific operation or action, it means that the system has installed thereon software, firmware, hardware or a combination thereof that causes the system to perform the operation or action in operation. For one or more computer programs configured to perform a specific operation or action, it means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action. For a dedicated logic circuit configured to perform a specific operation or action, it means that the circuit has electronic logic to perform the operation or action.

[0291] Although many specific implementation details are included herein, these should not be interpreted as limitations on the scope of the claims defined by the claims themselves, but rather as descriptions of specific features of specific embodiments. Certain features described in the context of individual embodiments herein may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features from the combination may be deleted from the claimed combination, and claims may be directed to a sub-combination or a variant of a sub-combination.

[0292] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence in order to achieve the desired results, or requiring that all of the operations shown be performed. In some cases, multitasking and parallel processing may be advantageous. In addition, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, but it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0293] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method comprising: Receiving a plurality of blockchain data for storage in a storage system; wherein the storage system comprises: a multi-layer storage device; the multi-layer storage device comprises a plurality of storage devices having different performance characteristics; the multi-layer storage device is used to store different types of blockchain data; storing the plurality of blockchain data respectively in a data log file of the multi-layer storage device based on the time sequence of the plurality of blockchain data; restricting the plurality of blockchain data in the data log file to be sorted according to corresponding keys of values ​​in the plurality of blockchain data, determining corresponding migration priorities of the plurality of blockchain data according to the time sequence of the plurality of blockchain data, and assigning scores to the plurality of blockchain data in the data log file; According to the corresponding migration priority and the score, the multiple blockchain data stored in the data log file are migrated to storage devices at each layer.

2. The computer-implemented method of claim 1, wherein: The key includes a hash value corresponding to the value.

3. The computer-implemented method of claim 1, wherein: The data log file is an append-only file.

4. The computer-implemented method of claim 1, wherein: The data log file is stored in a first-tier storage device in a multi-tier storage subsystem of the storage system including multi-tier storage devices, and the access speed of the first-tier storage device is the highest among the multi-tier storage devices.

5. The computer-implemented method of claim 1, assigning scores to a plurality of blockchain data in a data log file, comprising: Scores are assigned to multiple blockchain data in the data log file according to a scoring formula; wherein the scoring formula considers one or a combination of the following factors: Media level, access frequency, original data creation time and migration cost.

6. The computer-implemented method of claim 1, storing the plurality of blockchain data in a data log file of the multi-layer storage device based on a time sequence of the plurality of blockchain data, respectively, comprising: Based on the time sequence of the blockchain data, classify the blockchain data into at least hot data and cold data; The hot data and the cold data are stored in the data log file of the multi-layer storage device respectively; wherein the time sequence includes any one of the following: The time when the blockchain data was generated, the time when the blockchain data was received, and the frequency with which the blockchain data was accessed.

7. The computer-implemented method of claim 4, further comprising: The storage device generates an index for indicating corresponding physical storage locations of the plurality of blockchain data in the data log file; as well as The storage system writes the index into the first-tier storage device.

8. The computer-implemented method of claim 1, wherein: The migration priority of older blockchain data is higher than that of newer blockchain data.

9. An apparatus for data processing and storage, the apparatus comprising a plurality of modules for executing the method according to any one of claims 1 to 8.

10. A storage system comprising: one or more processors; as well as One or more computer-readable memories coupled to the one or more processors and having instructions stored thereon, the instructions being executable by the one or more processors to perform the computer-implemented method of any one of claims 1 to 8.

Images