A data semantic separation storage method and system

Abstract

The present invention provides a data semantic separation storage method and system, including: obtaining Ethereum data; inserting the Ethereum data into a corresponding type of skip list structure in a memory component according to the type of the Ethereum data; wherein, when the Ethereum data stored in the skip list structure reaches the upper capacity limit, the skip list structure will be frozen, and the frozen skip list structure will be converted into a file according to the type of the stored Ethereum data and then written into the corresponding type of file area in the disk component. The Ethereum data is divided into two weakly related types of data, which realizes the independent management and storage of different types of data, saves a part of the read-write amplification overhead, and can accelerate the data storage and retrieval process.

Description

Technical Field

[0001] The present invention belongs to the technical field of Ethereum key-value storage, and in particular to a data semantic separation storage method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As one of the representative works of blockchain 2.0, Ethereum has broadened the application scenarios by supporting smart contracts, making programming possible on any blockchain platform. In order to better serve different application scenarios, Ethereum maintains a global state MPT (Merkle Patricia Trie) to manage account data. As transactions are continuously packaged into blocks, the global state is continuously updated by transactions. However, as more and more applications are built on Ethereum, the complex logic of smart contracts and the increasing amount of data make the expansion of Ethereum increasingly difficult.

[0004] The expansion of Ethereum decentralized applications (DApps) requires two basic conditions: 1) low latency in transaction packaging and on-chain; 2) high efficiency in transaction execution. With the introduction of PoS (Proof of Stake) consensus protocols and sharding strategies by Ethereum, the throughput problem of transaction on-chain has been basically solved. However, the massive amount of data makes it difficult to guarantee the speed of transaction entry. At the same time, due to the large size of the database, the inefficient data request speed will also limit the further expansion of DApps.

[0005] In order to meet the massive big data storage and reading needs of Ethereum, the underlying Ethereum uses a key-value database (Key-Value Store, KV Store) based on the Log Structure Merge (LSM) to handle its write-intensive load. KV Store maintains a huge buffer pool in memory, which ensures data writing performance by converting a large number of random writes into sequential writes. At the same time, the underlying storage of KV Store is designed to be multi-layered, and the key-value pairs written are globally sorted on each layer according to the lexicographic order of their keys, thereby ensuring low query overhead.

[0006] However, KV Store is less resistant to massive amounts of data. As massive amounts of data are continuously written, the read and write performance of KVStore shows a trend of gradual decline. The fundamental reasons are: (1) As the subsequent background merging (compaction) process sorts and rewrites massive amounts of data, it occupies too much CPU resources and disk bandwidth. This phenomenon becomes more serious as the data scale increases; (2) KV Store, which is more inclined to write-intensive loads, performs poorly for a large number of point query operations. Since each query needs to traverse all data blocks layer by layer, querying a piece of data becomes very expensive. To address such problems, traditional relational databases achieve refined data management through table sharding, partitioning, and database sharding. For Ethereum, non-redundant distributed storage of Ethereum data is bound to reduce the security of the entire system. Distributed storage of data is a costly approach. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a data semantic separation storage method and system, which divides Ethereum data into two types of weakly related data, realizes independent management and storage of different types of data, saves part of the read-write amplification overhead, and can accelerate the data warehousing and retrieval process.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A first aspect of the present invention provides a data semantic separation storage method, which comprises:

[0010] Get Ethereum data;

[0011] According to the type of the Ethereum data, the Ethereum data is inserted into a corresponding type of jump table structure in the memory component;

[0012] Among them, when the Ethereum data stored in the skip list structure reaches the capacity limit, the skip list structure will be frozen. The frozen skip list structure is converted into a file according to the type of Ethereum data stored and written to the file area of ​​the corresponding type in the disk component.

[0013] Furthermore, if the skip list structure is frozen, a region is reallocated from the memory component as a new skip list structure to support the insertion of Ethereum data.

[0014] Furthermore, before the frozen skip list structure is written into the file area, the minimum key and the maximum key of the frozen skip list structure are recorded and used as an index resident memory component.

[0015] Furthermore, the types of Ethereum data include state data and non-state data;

[0016] The state data consists entirely of account data;

[0017] The non-state data includes transaction blocks and all metadata.

[0018] Furthermore, when requesting a piece of Ethereum data, according to the type of the Ethereum data to be queried, it is searched in the corresponding type of jump table structure in the memory component, or in the corresponding type of file area in the disk component.

[0019] Furthermore, if searching in a file area of ​​a corresponding type in a disk component, the specific method is: based on the index in the memory component, the file in the disk component is searched by a binary search method to query the requested Ethereum data.

[0020] A second aspect of the present invention provides a data semantic separation storage system, comprising:

[0021] A data acquisition module, configured to: acquire Ethereum data;

[0022] A data storage module, which is configured to: insert the Ethereum data into a corresponding type of jump table structure in the memory component according to the type of the Ethereum data;

[0023] Among them, when the Ethereum data stored in the skip list structure reaches the capacity limit, the skip list structure will be frozen. The frozen skip list structure is converted into a file according to the type of Ethereum data stored and written to the file area of ​​the corresponding type in the disk component.

[0024] Furthermore, before the frozen skip list structure is written into the file area, the minimum key and the maximum key of the frozen skip list structure are recorded and used as an index resident memory component.

[0025] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a data semantic separation storage method as described above.

[0026] The fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a data semantic separation storage method as described above are implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention provides a data semantic separation storage method, which analyzes the upper-layer data types and underlying storage characteristics of Ethereum from the perspective of data and storage, redefines and divides Ethereum data into two weakly related data types, and opens up a semantic bridge between the Ethereum data layer and the storage layer by separating and storing different types of data, thereby realizing independent management and storage of different types of data, saving a part of the read-write amplification overhead, accelerating the data warehousing and retrieval process, and further promoting the chaining and execution of DApp transactions, and realizing the overall optimization of the Ethereum system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1 It is a schematic diagram of a data semantic separation storage method according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0034] Embodiment 1

[0035] This embodiment provides a data semantic separation storage method, based on the principle that data of the same semantic type has strong locality, and proposes state separation for read-write load-intensive applications, effectively alleviating the current problem of slow data read and write speed in Ethereum. Figure 1 As shown, the specific steps include:

[0036] Step 1. Get Ethereum data.

[0037] Step 2: According to the type of Ethereum data (state data and non-state data), the Ethereum data is inserted into the corresponding type of skip list structure (state skip list structure and non-state skip list structure) in the memory component. When the Ethereum data stored in the skip list structure reaches the upper limit of the capacity, the skip list structure will be frozen; if the skip list structure is frozen, a new area is re-divided from the memory component as a new skip list structure to support the insertion of Ethereum data. If the skip list structure is frozen, the minimum key and maximum key of the frozen skip list structure are recorded and stored in the memory component as an index, and according to the type of Ethereum data stored in the skip list structure, the frozen skip list structure is converted into a file and written to the corresponding type of file area (state file area and non-state file area) in the disk component.

[0038] In traditional database design, in order to shorten the changes in disk data areas involved in a read or write, partitioning or table partitioning is often used to avoid high read and write amplification. To match this feature, data with the same attributes should be divided into multiple independently managed data areas. In fact, Ethereum data consists of three parts: transaction blocks, account data, and some metadata used to assist in queries. At the same time, Ethereum data queries are mainly composed of two categories: account queries and transaction queries. Analysis of Ethereum data semantics found that: (1) Account queries are independent and do not require metadata data. Querying an account only requires the state MPT maintained within Ethereum to complete; (2) Querying a transaction requires the assistance of metadata. It first needs to locate the block where the transaction is located based on the metadata, and then complete it by traversing all transactions in the block. Based on this, the present invention redefines Ethereum data and divides it into two types: state data and non-state data. State data is completely composed of account data, while non-state data contains transaction blocks and all metadata.

[0039] After the present invention re-divides all data types according to data semantics, the data is divided into two independent data areas. Next, consider the combination of upper-layer data semantics and underlying storage engine. For the structure of KV Store, all its data is indiscriminately written into the LSM-tree (Log Structured Merge Trees) structure, which will cause a large amount of movement, rewriting and query overhead in the storage layer. Therefore, the present invention realizes independent management of data by dividing one LSM-tree into two LSM-trees.

[0040] The present invention designs two data storage interfaces, namely Put_s() and Put_ns(), the former is responsible for writing status data, and the latter is responsible for writing non-status data.

[0041] When writing a state data, the state data is first written to the state mem through Put_s(). The state mem is a skip table structure, and the written key-value pairs are sorted in their lexicographic order. That is, each state data is a tuple containing a key and a value (key, value). When a new state data arrives, it will first be inserted into the state skip table structure - state mem. State mem supports single insertion and batch insertion. When the state mem reaches its capacity limit, this state mem will be frozen and turned into an unwritable state immu (frozen state skip table structure) and prepared for serialization. At the same time, an area will be re-allocated from the memory pool for the new state mem to support the insertion of new data. When the state immu appears, the Flush process running in the background will record the minimum key (min key) and the maximum key (max key) in the state immu and store them in memory as indexes. Then, this state immu will be converted into an SST file and written to the state file area in the disk component. Because the state data is managed separately in the state file area, when the compaction thread is triggered in the disk component, only the data in the same file area will be merged.

[0042] The process of writing non-state data is the same as that of state data. Non-state data is first written to non-state mem through Put_ns(). Non-state mem is a skip table structure, and the key-value pairs written are sorted in their lexicographic order. That is, each piece of non-state data is a tuple containing a key and a value (key, value). When a new piece of non-state data arrives, it will first be inserted into the non-state skip table structure - non-state mem. Non-state mem supports single insertion and batch insertion. When the non-state mem reaches its capacity limit, this non-state mem will be frozen and turned into a non-writable non-state immu (frozen non-state skip table structure) and prepared for serialization. At the same time, an area will be re-allocated from the memory pool for the new non-state mem to support the insertion of new data. When the non-state immu appears, the Flush process running in the background will record the min key and max key in the non-state immu and store them in memory as indexes. Then, this non-state immu will be converted into an SST file and written to the non-state file area in the disk component. Since non-state data is managed separately in non-state file areas, when the compaction thread is triggered in the disk component, only data in the same file area will be merged.

[0043] The disk components are generally divided into two independently managed areas, namely the state file area and the non-state file area, which are transparent to each other. Therefore, for the compaction process occurring in a certain area, the data files participating in the compaction are restricted to the same type. Therefore, the entire system is logically divided into two independently managed subsystems, each of which only manages the same type of data. With this in mind, a smaller data range and smaller data volume achieve smaller write amplification during data writing, increasing the overall write throughput of the Ethereum system.

[0044] After state separation, the original large system is divided into two small subsystems, and the two subsystems are transparent to each other. Therefore, the read operations occurring in any subsystem are also transparent to each other. When requesting an Ethereum data, according to the type of Ethereum data to be queried, it is searched in the corresponding type of jump table structure in the memory component, or in the corresponding type of file area in the disk component. If searching in the corresponding type of file area in the disk component, the specific method is: based on the index in the memory component, the file in the disk component is searched by binary search to query the requested Ethereum data.

[0045] The data request process is similar to that of writing. The present invention implements two request interfaces, Get_s() and Get_ns(), to realize independent retrieval of different types of Ethereum data.

[0046] When requesting an account data (status data), first search the status mem and status immu in the memory component based on the key of the status data to be queried to see if there is a value of the status data to be queried; if the status data to be queried is not retrieved, search the status file area in the disk component. Specifically, the index controls access to the status file area and traverses it layer by layer. The status file partition on the disk component consists of multiple levels, and the query of a single data must be queried layer by layer from top to bottom. The index control residing in the memory maintains the metadata of all SST files in the two file partitions (including minkey, max key, filenumber, level and other information). This index will retrieve the corresponding SST query value on the disk through a binary search method.

[0047] The request for non-state data is consistent with that for state data. When requesting a piece of non-state data, firstly search the non-state mem and non-state immu in the memory component based on the key of the state data to be queried to see whether the value of the non-state data to be queried exists; if the non-state data to be queried is not found, search the non-state file area in the disk component. Specifically, the index controls the access to the non-state file area and traverses it layer by layer. The non-state file partition on the disk component consists of multiple levels, and the query of a single piece of data must be queried layer by layer from top to bottom.

[0048] After state separation, queries for a single data type only need to retrieve a portion of the SST files instead of traversing all the files. That is, the state separation strategy proposed in the present invention ensures that the reading of a transaction or account only involves a small amount of data, which greatly reduces the read amplification during the query process and improves the overall throughput.

[0049] The present invention re-divides the data in Ethereum into two weakly correlated areas and realizes independent management and storage. Compared with the current Ethereum storage method, it saves a part of the read-write amplification overhead, can accelerate the data storage and retrieval process, and further promotes the chaining and execution of Dapp transactions, realizing the overall optimization of the Ethereum system.

[0050] Embodiment 2

[0051] This embodiment provides a data semantic separation storage system, which specifically includes the following modules:

[0052] A data acquisition module, configured to: acquire Ethereum data;

[0053] A data storage module, which is configured to: insert the Ethereum data into a corresponding type of jump table structure in the memory component according to the type of the Ethereum data;

[0054] When the Ethereum data stored in the skip list structure reaches the upper limit of capacity, the skip list structure will be frozen. The frozen skip list structure is converted into a file according to the type of Ethereum data stored and written to the file area of ​​the corresponding type in the disk component. Before the frozen skip list structure is written to the file area, the minimum key and maximum key of the frozen skip list structure are recorded and stored in the memory component as an index.

[0055] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0056] Embodiment 3

[0057] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the data semantic separation storage method described in the above-mentioned embodiment 1 are implemented.

[0058] Embodiment 4

[0059] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in a data semantic separation storage method as described in the first embodiment above are implemented.

[0060] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0061] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data semantic separation storage method, characterized in that: include: Get Ethereum data; According to the type of the Ethereum data, the Ethereum data is inserted into a corresponding type of jump table structure in the memory component; When the Ethereum data stored in the skip table structure reaches the upper limit of the capacity, the skip table structure will be frozen. The frozen skip table structure will be converted into a file according to the type of Ethereum data stored and then written into the file area of ​​the corresponding type in the disk component; The types of Ethereum data include state data and non-state data; The state data consists entirely of account data; The non-state data includes transaction blocks and all metadata; When requesting a sum of Ethereum data, according to the type of Ethereum data to be queried, the corresponding type of skip list structure in the memory component is searched, or the corresponding type of file area in the disk component is searched; If searching in the file area of ​​the corresponding type in the disk component, the specific method is: based on the index in the memory component, the file in the disk component is searched by binary search to query the requested Ethereum data.

2. A method for storing data with semantic separation as claimed in claim 1, characterized in that: If the skip list structure is frozen, a new skip list structure is created from the memory component to support the insertion of Ethereum data.

3. A method for storing data with semantic separation as claimed in claim 1, characterized in that: Before the frozen skip list structure is written into the file area, the minimum key and the maximum key of the frozen skip list structure are recorded and used as index resident memory components.

4. A data semantic separation storage system, characterized in that: include: A data acquisition module, configured to: acquire Ethereum data; A data storage module, which is configured to: insert the Ethereum data into a corresponding type of jump table structure in the memory component according to the type of the Ethereum data; When the Ethereum data stored in the skip table structure reaches the upper limit of the capacity, the skip table structure will be frozen. The frozen skip table structure will be converted into a file according to the type of Ethereum data stored and then written into the file area of ​​the corresponding type in the disk component; The types of Ethereum data include state data and non-state data; The state data consists entirely of account data; The non-state data includes transaction blocks and all metadata; When requesting a sum of Ethereum data, according to the type of Ethereum data to be queried, the corresponding type of skip list structure in the memory component is searched, or the corresponding type of file area in the disk component is searched; If searching in the file area of ​​the corresponding type in the disk component, the specific method is: based on the index in the memory component, the file in the disk component is searched by binary search to query the requested Ethereum data.

5. A data semantic separation storage system as claimed in claim 4, characterized in that: Before the frozen skip list structure is written into the file area, the minimum key and the maximum key of the frozen skip list structure are recorded and used as index resident memory components.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a data semantic separation storage method as described in any one of claims 1 to 3 are implemented.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the data semantic separation storage method as described in any one of claims 1-3 are implemented.

Images