Decentralized database solution based on ipfs

Abstract

The application discloses a decentralized database solution method based on IPFS, and comprises the following steps: abstracting a representation format of a logical directory structure of Table data based on an IPFS decentralized file system; representing the meta information of the logical directory structure of the Table data based on the functions of the IPFS and a directed acyclic graph, and obtaining the directed acyclic graph meta information of the Table data; after obtaining the directed acyclic graph meta information of the Table data, each node retains the directed acyclic graph meta information of the logical directory structure of the Table data; and if the data of a node is updated, the change of the directed acyclic graph of the node is synchronized to other nodes. The application solves the shortcomings of the prior art, has the advantages of not being prone to single-point capacity bottlenecks, good scalability, low storage cost and good synchronization performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of file storage, in particular, to a decentralized database solution based on IPFS. BACKGROUND

[0002] IPFS is a decentralized file storage system, which is an important part of the Web3 technology stack. Many Dapp decentralized applications use IPFS as the file storage layer, but a decentralized database is usually needed to serve as the CRUD (CRUD is an acronym for Create, Read, Update, and Delete) of business data. This is the same layered approach as traditional Web2.

[0003] In Web3, existing decentralized database solutions based on IPFS include OrbitDB, AvonDB, and ThreadDB. The data storage architecture of these existing technologies can be understood as each node in the decentralized network having a complete data copy. Each node can perform CRUD, and the updates are synchronized to other nodes through the PubSub protocol (a p2p protocol similar to gossip) of IPFS. When other nodes receive an update data request, they will merge the updated data into the local copy. Eventually, the data in each node is consistent. The basic data architecture is as follows Figures 13-14 . Figure 13 As can be seen, each node has a complete data copy, and when multiple nodes update, they are synchronized to other nodes, Figure 14 As can be seen, after synchronization is completed, each node maintains the latest complete data copy.

[0004] The existing IPFS-based solution stores a copy on each node, and when writing updates, it uses the PubSub protocol for decentralized network data propagation and synchronization. This solution has the following disadvantages:

[0005] 1) Single point of storage capacity bottleneck: Because each node stores a complete data copy, when the data volume is large, the storage bottleneck of a single node will soon be reached. Moreover, many IPFS nodes may not be in the form of a server, and there may be significant differences in server performance and specifications. In some cases, the specifications of some nodes are not high, which has a significant impact.

[0006] 2) Poor scalability: When a new node is added, it needs to copy the complete data copy to the local node, which is low in performance. Moreover, the newly added node may have small specifications, which makes it slower to scale.

[0007] 3) Storage cost is high: the same data is stored N times, and N is the number of P2P network nodes added to the table. For example, a 10-node P2P network creates a testtable01 with a size of 10G, and the entire network will have 10*10G of data distributed on each node, which is a serious waste of storage space.

[0008] 4) Poor synchronization performance: each time a batch of data is inserted, the batch of data also needs to be synchronized to all nodes, which also causes serious waste of bandwidth. For example, a 10-node P2P network has a testtable01 table, and when 1w rows of data are inserted, the 1w rows of data will be distributed to the other 9 nodes, and the total network flow is 1w*10 times = 10w rows of data, which cannot be ignored. And as the amount of data to be synchronized increases, the synchronization time will inevitably slow down as the amount of data increases.

[0009] At present, there is no effective solution to the problems in the related art. SUMMARY

[0010] In view of the problems in the related art, the present application proposes a decentralized database solution based on IPFS to overcome the above technical problems existing in the prior art.

[0011] To this end, the specific technical solutions adopted by the present application are as follows:

[0012] The decentralized database solution based on IPFS includes the following steps:

[0013] S1, based on the IPFS decentralized file system, abstract a layer of representation format of the logical directory structure of the Table data;

[0014] S2, based on the IPFS function and the directed acyclic graph, representing the meta information of the logical directory structure of the Table data, and obtaining the directed acyclic graph meta information of the Table data;

[0015] S3, after obtaining the directed acyclic graph meta information of the Table data, each node retains the directed acyclic graph meta information of the logical directory structure of the Table data;

[0016] S4, if the data of a node is updated, the change of the directed acyclic graph of the node is synchronized to other nodes.

[0017] Further, the logical file object in the logical directory structure of the Table data is split into several Blocks and stored on the IPFS decentralized file system.

[0018] Furthermore, IPFS stores data in key-value pairs, where the key is associated with the content, and if the keys are the same, the data content is the same. At the same time, other key values ​​are stored in the data content, forming a chain or a directed acyclic graph.

[0019] Furthermore, after obtaining the directed acyclic primitive information of the Table data, each node retains the directed acyclic primitive information of the logical directory structure of the Table data, which includes the following steps:

[0020] When processing data, the directed acyclic graph primitive information of the data directory is managed, and the bottom node of the directed acyclic graph tree is made to point to a file node.

[0021] The file is split into several blocks and stored in a distributed manner on a decentralized file system based on IPFS;

[0022] This enables each node in a P2P application to store a portion of the Table data in a logical directory containing directed acyclic graph primitive information.

[0023] Furthermore, after the directed acyclic graph is obtained, and before data is inserted into a table, each node has a directed acyclic data directory of the current state of a table. Figure 1 Information, and obtain directed acyclic variables. Figure 1 The tableDirCid is obtained, and the addresses of all subdirectories are retrieved. At the same time, the contents of various subdirectories and subfiles under all logical directories of this table are found.

[0024] Furthermore, after the directed acyclic graph is obtained, if a table inserts or updates data and synchronizes, all nodes maintain a directed acyclic relationship. Figure 2 Meta-information.

[0025] Furthermore, if the data of one node is updated, the change in the directed acyclic graph of that node is synchronized to other nodes, which includes the following steps:

[0026] Before data is inserted, the metadata of the current table in all nodes is directed acyclic. Figure 1 ;

[0027] After the user successfully inserts data into node 1, the directed acyclic node 1... Figure 1 Updated to directed acyclic. Figure 2 ;

[0028] Node 1 sends synchronization change information to other nodes in the entire IPFS-based P2P network, and the message event is a directed acyclic message. Figure 1 Updated to directed acyclic. Figure 2 ;

[0029] After each node receives the directed acyclic graph update message, the local directed acyclic graph is updated Figure 1 The change is synchronized to the directed acyclic graph Figure 2 , so that the entire P2P network reaches a consistent state.

[0030] Further, when the change of the directed acyclic graph of the node is synchronized to other nodes, the data represented by the directed acyclic graph is stored in the IPFS-based decentralized file system.

[0031] Further, the local directed acyclic graph of each node is updated Figure 1 The change is synchronized to the directed acyclic graph Figure 2 , and if a conflict occurs, it is resolved according to specific circumstances.

[0032] Further, if a conflict occurs, it is resolved according to specific circumstances, including the following steps:

[0033] If the current content identifier of the node is equal to the old content identifier, update the current content identifier to the old content identifier;

[0034] If the current content identifier, the old content identifier and the new content identifier are not equal, and the old content identifier and the new content identifier are in the history content identifier list, ignore them;

[0035] If the current content identifier is equal to the new content identifier, ignore it;

[0036] If the current content identifier, the old content identifier and the new content identifier are not equal, and the old content identifier and the new content identifier are not in the history content identifier list, update the current content identifier to the new content identifier.

[0037] The beneficial effects of the present application are:

[0038] (1) The present application is based on the table logic metadata directory designed on IPFS, which uses a directed acyclic graph to represent it, and designs the update and merging strategy of the CURD process of the table, solves the shortcomings of the prior art, has the advantages of not easy to reach single point capacity bottleneck, good scalability, low storage cost and good synchronization performance.

[0039] (2) The existing scheme is that each node saves a complete data copy, and the present application only represents a small amount of directory information, which is more difficult to reach the capacity bottleneck of a single node.

[0040] (3) The scalability of the present application is closely related to the way of metadata and data storage. In the case that the Node of the present application does not need to store a complete copy of the data, the expansion and contraction are more simple and fast, and the storage cost is also reduced.

[0041] (4) The present application only synchronizes a DAG metadata to other nodes regardless of whether a new node is added or data is inserted, greatly reducing the data amount of data synchronization, and thus greatly improving performance and latency. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0043] Figure 1 is a flowchart of the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0044] Figure 2 is a logical data directory structure diagram in the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0045] Figure 3 is a tree representation diagram of the directory structure in the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0046] Figure 4 is a table logical directory tree using IPFS DAG representation form diagram in the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0047] Figure 5 is one of the change situation diagrams of the DAG of the data logical directory in the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0048] Figure 6 is a DAG1 tree diagram of the data directory of the table before inserting new data in the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0049] Figure 7 is the second change situation diagram of the DAG of the data logical directory in the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0050] Figure 8 is a DAG2 tree diagram of the data directory of the table after inserting new data in the IPFS-based decentralized database solution method according to the embodiment of the present application;

[0051] Figure 9is one of the step flowcharts of data synchronization in the IPFS-based decentralized database solution method according to an embodiment of the present application;

[0052] Figure 10 is one of the step flowcharts of data synchronization in the IPFS-based decentralized database solution method according to an embodiment of the present application;

[0053] Figure 11 is one of the step flowcharts of data synchronization in the IPFS-based decentralized database solution method according to an embodiment of the present application;

[0054] Figure 12 is one of the step flowcharts of data synchronization in the IPFS-based decentralized database solution method according to an embodiment of the present application;

[0055] Figure 13 is a schematic diagram of the network data in the case of inserting kv3 / kv4.

[0056] Figure 14 is a schematic diagram of the kv3 / kv4 synchronization completion state. DETAILED DESCRIPTION

[0057] To further illustrate the embodiments, the present application provides drawings, which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. Those of ordinary skill in the art should understand other possible implementations and advantages of the present application by referring to these contents. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0058] According to an embodiment of the present application, an IPFS-based decentralized database solution method is provided.

[0059] The present application will be further described in conjunction with the drawings and specific embodiments. As shown in the drawings, the IPFS-based decentralized database solution method according to an embodiment of the present application includes the following steps: Figure 1

[0060] S1, based on the IPFS decentralized file system, abstract a layer of logical directory structure representation format of Table data; for example:

[0061] <base> / db1 / table1 / .mando / state_1669556239214_0001.json. <base> / db1 / table1 / 18 / fileId1_001.parquet# data file. ​<base> / db1 / table1 / 19 / fileId2_001.parquet#data file.

[0062] <base> / db1 / table1 / 20 / fileId3_001.parquet#data file.

[0063] <base> / db1 / table1 / 20 / fileId3_001.parquet.delete_1669558239012#delete marker file.

[0064] The above table is the regular logical data directory architecture for db1.table1 table. The ".mando" directory is the current state file of the table, which can be understood as snapshot information. The state json file saves which data files are composed under the current state of db1.table1. The directories "18", "19", "20" are logical partition directories, and the final parquet file and delete marker file are logical file objects on IPFS. This file object will eventually be split into many Blocks and exist on the IPFS decentralized file storage. "Table" refers to the concept of a database table.

[0065] The above table is the regular logical data directory architecture for db1.table1 table. The ".mando" directory is the current state file of the table, which can be understood as snapshot information. The state json file saves which data files are composed under the current state of db1.table1. The directories "18", "19", "20" are logical partition directories, and the final parquet file and delete marker file are logical file objects on IPFS. This file object will eventually be split into many Blocks and exist on the IPFS decentralized file storage. "Table" refers to the concept of a database table.

[0066] S2, based on IPFS functions and directed acyclic graph (DAG), representing the meta information of the logical directory structure of Table data, and obtaining the directed acyclic graph meta information of Table data;

[0067] IPFS data is stored in the form of key-value. Assuming that a file content is represented as byte[] type data, then key = hash(data), value = encode(data). The hash algorithm can be sha256 or other configurable algorithms. Here, the key is associated with the content, so the data under the same key must be the same content, which guarantees their immutable characteristics.

[0068] In addition, there is a feature that can use the data content to save more other keys, forming a chain or a directed acyclic graph. For example, we use this feature to design the logical directory structure of table data in the form of a directed acyclic graph as follows Figure 2 .

[0069] As Figures 2-4 illustrate how to use IPFS DAG to represent the directory tree of table data by example. Figure 2 The root directory of the data of the example db1.table1 is shown in the following figure: Figure 3 The direct representation of the directory tree of the logical directory of db1.table1 is shown in the following figure: Figure 4 The DAG directed acyclic graph representation of the directory tree is shown in the following figure: The core of this DAG is the 5 nodes in the black box, which respectively correspond to the root directory of db1.table1, “.mando” and the three partition directory nodes 18, 19 and 20. The directory and the files under the directory are described by the way of connection, for example, there are directories “.mando, 18, 19, 20” under the root directory of db1.table1, which are represented by <tableDirCid, tableDirContent = {mandoDirCid, 18DirCid, 19dirCid, 20DirCid}>, where 18DirCid represents the node address of the subdirectory “18” and is linked to its child node by a dashed line. Here, the dashed line is used because it is a logical chain and not a real directory and subdirectory relationship. In the same way, the relationship between other directories and subdirectories is represented in this way. In this way, it is the logical directory structure information of the table data according to the DAG of IPFS.

[0070] In one embodiment, the logical file objects in the logical directory structure of the table data are split into several blocks and stored on the IPFS decentralized file system.

[0071] In one embodiment, the IPFS stores data in the form of key-value, where the key is associated with the content, and if the key is the same, the data content is the same, and the data content saves other key values and forms a chain or a directed acyclic graph.

[0072] S3, after obtaining the DAG meta information of the table data, each node retains the DAG meta information of the logical directory structure of the table data;

[0073] By the above definition, for a large amount of data, we only need to manage the directory DAG meta information, and the bottom node of the DAG tree points to a file node, which is a normal IPFS file object, which is automatically divided into multiple blocks and stored in the IPFS decentralized P2P network file storage system. Each node in the P2P network only needs to save a small amount of table logical directory DAG meta information.

[0074] After using DAG technology, the changes in the DAG of the data logical directory when inserting data into a table are as follows Figures 5-6 .

[0075] As Figure 5 : Before insertion, each node has the data directory DAG1 information of the current state of the table, and the structure of the DAG1 information is as shown in Figure 6 . That is, knowing the tableDirCid of DAG1 (tableDirCid refers to the unique number of the table root directory node, and through this number, the content of the node can be obtained, which is a list of numbers of other nodes, and through this way, the structure of a node directed acyclic graph can be obtained. ), can know the addresses of all subdirectory nodes, and can find all the logical directory contents of the table and the contents of the subdirectories and subfiles. In fact, these file contents can be divided into blocks and stored in the IPFS decentralized file storage system.

[0076] In an embodiment, as Figures 7-8 , after obtaining the DAG meta information of the table data, each node retains the DAG meta information of the logical directory structure of the table data, including the following steps:

[0077] When processing data, manage the DAG meta information of the data directory, and make the bottom node of the DAG tree point to a file node;

[0078] Divide the file into blocks and store it in the IPFS decentralized file system;

[0079] Make each node in the P2P network save part of the DAG meta information of the logical directory of the table data.

[0080] In an embodiment, after obtaining the DAG, and before inserting data into a table, each node has the DAG meta information of the current state of the table data directory Figure 1 (DAG Figure 1 information structure as shown inFigure 6 The DAG represents the structure of the table, and the tableDirCid is obtained, and all subdirectory node addresses are obtained, and all subdirectories and subfile contents under the logical directory of the table are found. The actual file contents can be distributed on the IPFS decentralized file storage system. Figure 1

[0081] In an embodiment, after the DAG is obtained, if a table insertion update data and synchronization occur, each node maintains the latest DAG meta information. The DAG represents the logical data directory structure as shown in Figure 2 Figure 2 Figure 8 It can be seen that data insertion does not matter how large the data file is, each node only needs to update the DAG information, and the size of the specific data file has little to do with it.

[0082] In summary Figures 5-8 It can be seen that the present scheme is different from OrbitDB, ThreadDB, and AvonDB, and uses the DAG of IPFS to represent the logical directory structure of a table data, each node only stores the DAG information of the logical directory structure, the DAG meta information, and the size of the DAG is relatively stable. Compared with the data file size of the table, there may be a difference in data size, GB level or above, the storage advantage begins to be reflected, especially TB level or above, which clearly demonstrates the storage and synchronization performance advantage of the present scheme. For example, a table has 300G of data, and the metadata DAG is only 5MB. Compared with existing schemes, the storage occupancy of a single node is only 5MB, and a large amount of data file content is distributed on the decentralized IPFS system, fully utilizing the storage capacity of IPFS, and each node has complete copy data CURD capability.

[0083] S4, if the data of a node is updated, the change of the DAG of the node is synchronized to other nodes.

[0084] Since we stipulate that each node only saves the DAG metadata information of the Table data directory, this makes the information transmitted and how to handle the received update information different when we perform data synchronization. The information we transmit is not complete insertion data, but only the latest DAG information. The following describes the data synchronization step process, as shown in Figure 9 .

[0085] In an embodiment, as shown in Figures 9-12 , if the data of a node is updated, the change of the DAG of the node is synchronized to other nodes, including the following steps: ​​​

[0086] Before data is inserted, the metadata of the current table in all nodes is directed acyclic. Figure 1 ;

[0087] After the user successfully inserts data into node 1, the directed acyclic node 1... Figure 1 Updated to directed acyclic. Figure 2 ;

[0088] Node 1 sends synchronization change information to other nodes in the entire IPFS-based P2P network, and the message event is a directed acyclic message. Figure 1 Updated to directed acyclic. Figure 2 ;

[0089] After each node receives the directed acyclic graph update message, it will update the local directed acyclic graph. Figure 1 Synchronous change to directed acyclic Figure 2 This ensures that the entire P2P network reaches a consistent state.

[0090] As you can see, this synchronization process refers to synchronizing the DAG, and the data represented by the DAG is stored in the IPFS decentralized file storage system.

[0091] In one embodiment, when changes to the directed acyclic graph of a node are synchronized to other nodes, the data represented by the directed acyclic graph is stored in a decentralized file system based on IPFS.

[0092] In one embodiment, the local directed acyclic graph (DAG) is... Figure 1 Synchronous change to directed acyclic Figure 2 If a conflict arises, it will be resolved according to the specific circumstances.

[0093] In one embodiment, resolving conflicts as appropriate includes the following steps:

[0094] If the current content identifier of a node is equal to the old content identifier, then update the current content identifier to the old content identifier;

[0095] If the current content identifier, the old content identifier, and the new content identifier are all different, and both the old content identifier and the new content identifier are in the list of historical content identifiers, then ignore them;

[0096] If the current content identifier is equal to the new content identifier, then ignore it;

[0097] If the current content identifier, the old content identifier, and the new content identifier are all different, and neither the old content identifier nor the new content identifier is in the historical content identifier list, then the current content identifier will be updated to the new content identifier.

[0098] For example, the event that Node1 updates DAG1 to DAG2 is defined as event{table root oldCid, table root newCid}, and table at current cid of each node is table root currentCid, then we discuss the conflict resolution according to how the receiver handles an update message:

[0099] Node2{table root currentCid} receives event{oldCid, newCid}.

[0100] Case 1: currentCid = oldCid, at this time, there is no conflict in table name, and the current currentCid is directly updated to newCid.

[0101] Case 2: currentCid is not equal to oldCid and newCid, and oldCid and newCid are in the historical cid list, that is, the update time is older, that is, oldCid < newCid < currentCid, which indicates that this update message is an older message, which may be sent by another node, at this time, the message can be ignored.

[0102] Case 3: currentCid = newCid, at this time, it indicates that this may be a repeated update event sent by another node, which can be ignored. Since it is a P2P network, the gossip protocol is used for message update synchronization, and it is inevitable to receive repeated messages, so it belongs to the normal phenomenon.

[0103] Case 4: When currentCid is not equal to oldCid and newCid, and newCid and oldCid are not in the history cid list, it means that this is an advanced update message, that is, currentCid < oldCid < newCid, which may be because the node missed many intermediate messages for some reason, for example, the node is offline for a period of time and then online, which may encounter similar situations. At this time, only the currentCid needs to be updated to newCid. We know that there may be many events that are not received in the middle, and the received many {oldCid, newCid} event messages are not equal to currentCid, but here as long as all the messages are eventually received, he will become a finally consistent state. For example, currentCid=cid0, and then received update events {cid2, cid3}, {cid4, cid5}, {cid1, cid2}, {cid3, cid4}, since the received messages are not in order, but we can simply use the strategy of new covering old and old not covering new to achieve the final currentCid updated to cid5, thereby achieving a consistent state.

[0104] The advantages of the present application are as follows:

[0105] Table 1 Advantage Summary

[0106]

[0107] As can be seen from Table 1, the present application solves several shortcomings of the prior art solution, and the reasons and evidence described in the table are described again below.

[0108] (1) Whether it is easy to reach a single-point capacity bottleneck problem

[0109] The existing solution is that each node saves a complete data copy. Assuming that the total data of a table is 300G, even if this P2P network has 10 nodes, each node has different capacity sizes, and the capacity specifications may be between (10G-500G), then for those nodes with a capacity less than 300G, the data copy of this table cannot be accommodated. For example, such small capacity specification network nodes may be javascript version Node program browsers, or small specification capacity 100G specification cloud servers.

[0110] And the solution of the present application, the metadata of a 300G data, may only have a few directory information, for example

[0111] / db / table / p1 / data_file1.parquet / / 128M

[0112] / db / table / p1 / data_file2.parquet / / 128M

[0113] / db / table / p1 / ...

[0114] Assuming that each piece of metadata directory can represent a File Object can be 128M, then 300G of data only need about 2345 metadata records, each record plus the average length of various characters is 100 bytes, then 2345*100 = 234K less than 1MB. Plus other meta image will not be too much. And the real data is stored in the form of IPFS file object, a file object is cut into many blocks, scattered to the nodes of IPFS. IPFS will allocate blocks according to the capacity of each node. So in a 10-node network, even if the capacity of these nodes is between (10G-500G), it can also accommodate the DAG meta information of the Table data directory. Therefore, in the present application, it is more difficult to reach the capacity bottleneck of a single node.

[0115] (2) scalability

[0116] Scalability is closely related to the way metadata and data are stored. When the Node of the present application does not need to store a complete copy of the data, it is more simple and fast to expand and shrink. For example, when a new node is added, it does not need to copy all the data of the entire table 300G to the new node, but only needs to synchronize the 1MB metadata of the table to the new node.

[0117] (3) storage cost

[0118] This is simple, as in the above example, a table of 300G data is added to a P2P network with 10 nodes, so it needs to store 3T of data capacity consumption. For the present application, only 1MB of metadata is stored for 10 times, about 10MB, and the data of the entire table is stored in the form of IPFS blocks, so the capacity consumption is only 10M+300G.

[0119] (4) synchronization performance

[0120] Synchronization performance is even simpler. One is when a new node is added, the existing solution needs to copy the 300G table data to the new node. Two is to insert 100M data into a node, then the 100M data also needs to be distributed to the other 9 nodes, and the network traffic also needs to be linearly increased.

[0121] While the scheme of the present application will synchronize only one DAG 1MB metadata to other nodes whether it is a new node or inserting 100M data, the data volume of data synchronization is far lower than execution, thereby greatly improving performance and latency.

[0122] Related terms are explained as follows:

[0123] Web3: is a derivative of the underlying protocol of the current Internet "World Wide Web". Web3.0 is the expectation of the next generation of Internet, which means that machines can read any information (i.e. semantic web), websites can provide intelligent selection and better information based on information (artificial intelligence), the Internet is ubiquitous (Internet of Things), and more importantly, the ownership of Internet data will be decentralized.

[0124] IPFS: InterPlanetary File System, short for InterPlanetary File System. It is a decentralized storage system that supports multiple access protocols, and its data is immutable because it uses content-generated hash values for routing. If the content changes its routing, it is not the same content as before. Inserting new values cannot replace the original values, and the original values still exist in the system.

[0125] IPLD: InterPlanetary Linked Data, which is the data model of the decentralized Internet. It connects all data through encryption hash, making it easier to traverse and link to the decentralized Web.

[0126] OrbitDB: a database based on IPFS written in javascript, a serverless, distributed, peer-to-peer database.

[0127] ThreadDB: an unstructured peer-to-peer database built on IPFS.

[0128] AvonDB: a wrapper built on OrbitDB, aiming to encapsulate it as a MongoDB-like application data model.

[0129] Decentralized Application (DApp): DApp is the abbreviation of Decentralized Application, which can be translated as decentralized application in Chinese, or distributed application.

[0130] In summary, the present application is based on the table logic metadata directory designed on IPFS, using directed acyclic graph to represent it, and designing the update and merging strategy of the CURD process of the table, solving the shortcomings of the prior art, having the advantages of not easy to reach single point capacity bottleneck, good scalability, low storage cost, good synchronization performance. The existing scheme is that each node saves a complete data copy, and the present application only represents a small amount of directory information, which is more difficult to reach the capacity bottleneck of a single node. The scalability of the present application is closely related to the way of metadata and data storage. In the case that the Node of the present application does not need to store a complete copy of the data, the expansion and contraction are simpler and faster, and the storage cost is also reduced. Whether it is a newly added node or an inserted data, the present application only synchronizes a DAG metadata to other nodes, greatly reducing the data amount of data synchronization, thereby greatly improving the performance and latency.

[0131] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An IPFS-based decentralized database solution method, characterized in that, The method comprises the following steps: S1, abstracting a representation format of a logical directory structure of Table data based on an IPFS decentralized file system; S2, representing the meta information of the logical directory structure of Table data based on the functions and directed acyclic graph of IPFS, and obtaining the directed acyclic graph meta information of Table data; S3, after obtaining the directed acyclic graph meta information of Table data, each node retains the directed acyclic graph meta information of the logical directory structure of Table data, specifically comprising: When processing data, the directed acyclic graph meta information of the data directory is managed, and the nodes at the bottom of the directed acyclic graph tree point to a file node; the file is divided into several blocks and stored on the IPFS decentralized file system; and each node of the P2P network saves part of the logical directory directed acyclic graph meta information of Table data; After obtaining the directed acyclic graph, and before inserting data into a table, each node has the data directory directed acyclic graph 1 information of the current state of the table, and obtains the tableDirCid of the directed acyclic graph 1 and the addresses of all subdirectory nodes, and finds all the logical directories and subfile contents of the table; S4, if the data of a node is updated, the change of the directed acyclic graph of the node is synchronized to other nodes. 2.The IPFS-based decentralized database solution method of claim 1, wherein, The logical file objects in the logical directory structure of the Table data are split into several blocks and stored on the IPFS decentralized file system. 3.The IPFS-based decentralized database solution method of claim 2, wherein, The IPFS stores data in the form of key-value, wherein the key is associated with the content, and if the key is the same, the data content is the same, and other key values are saved in the data content, forming a chain or a directed acyclic graph. 4.The IPFS-based decentralized database solution method of claim 1, wherein, After obtaining the directed acyclic graph, if a table inserts and updates data and synchronizes, each node retains the directed acyclic graph 2 meta information. 5.The IPFS-based decentralized database solution method of claim 4, wherein, If the data of a node is updated, the change of the directed acyclic graph of the node is synchronized to other nodes, comprising the following steps: Before inserting data, the metadata of the current Table in all nodes is directed acyclic graph 1; After the user successfully inserts data into node 1, the directed acyclic graph 1 of node 1 is updated to directed acyclic graph 2; Node 1 sends a synchronization change message to other nodes of the entire P2P network based on IPFS, and the message event is that the directed acyclic graph 1 is updated to the directed acyclic graph 2; After each node receives the directed acyclic graph update message, the local directed acyclic graph 1 is synchronized and changed to the directed acyclic graph 2, so that the entire P2P network reaches a consistent state. 6.The IPFS-based decentralized database solution method of claim 5, wherein, When the directed acyclic graph of the node is changed and synchronized to other nodes, the data represented by the directed acyclic graph is stored in the IPFS decentralized file system. 7.The IPFS-based decentralized database solution method of claim 6, wherein, When the local directed acyclic graph 1 is synchronized and changed to the directed acyclic graph 2, if a conflict occurs, it is solved according to the specific circumstances. 8.The IPFS-based decentralized database solution method of claim 7, wherein, If the current content identifier of the node is equal to the old content identifier, the current content identifier is updated to the old content identifier. ​ if the current content identifier, the old content identifier, and the new content identifier are all not equal, and the old content identifier and the new content identifier are both in the history content identifier list, then ignore; if the current content identifier is equal to the new content identifier, then ignore; if the current content identifier, the old content identifier, and the new content identifier are all not equal, and the old content identifier and the new content identifier are both not in the history content identifier list, then update the current content identifier to the new content identifier.

Images