A distributed storage search method and system based on secondary routing

Abstract

The present invention discloses a distributed storage search method and system based on secondary routing. The method comprises the following steps: setting the composition attributes of a two-level routing table; adding two-level routing tables respectively when storing data; and searching the secondary routing table when reading data. The technical solution of the present invention improves the DHT routing table and adds secondary routing on the basis of DHT. It not only speeds up the data search speed, but also improves the hit rate of data access. It technically improves the user request response speed. In the overall distributed network storage, it avoids excessive backup redundant data and increases the data storage capacity.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to a distributed storage search method and system based on secondary routing. Background Art

[0002] Blockchain technology is a new distributed infrastructure and computing method that uses block chain data structures to verify and store data, distributed node consensus algorithms to generate and update data, cryptography to ensure the security of data transmission and access, and smart contracts composed of automated script codes to program and operate data.

[0003] The following concepts exist in blockchain-related technologies:

[0004] 1.P2P peer-to-peer network technology;

[0005] P2P network is a peer-to-peer network / peer-to-peer computer network: it is a distributed application architecture that distributes tasks and workloads among peers. It is a networking or network form formed by the peer computing model at the application layer. "Peer" means "peer, partner, and opposite end" in English. Therefore, literally, P2P can be understood as peer computing or peer network. Some domestic media translate P2P into "point-to-point" or "end-to-end". The academic community uniformly calls it peer-to-peer networking or peer-to-peer computing, which can be defined as: the participants of the network share part of their hardware resources (processing power, storage capacity, network connection capacity, printers, etc.), these shared resources provide services and content through the network, and can be directly accessed by other peer nodes (Peer) without going through an intermediate entity. The participants in this network are both providers of resources, services and content (Server) and acquirers of resources, services and content (Client).

[0006] 2. Distributed storage technology;

[0007] InterPlanetary File System (IPFS) is a network transmission protocol designed to create persistent and distributed storage and sharing of files. The technology is a content-addressable peer-to-peer hypermedia distribution protocol. The nodes in the IPFS network will form a distributed file system. It is an open source project that has been developed by Protocol Labs with the help of the open source community since 2014. It was originally designed by Juan Benet.

[0008] IPFS is a peer-to-peer distributed file system that attempts to connect all computing devices to the same file system. In some ways, IPFS is similar to the World Wide Web, but it can also be viewed as a separate BitTorrent group exchanging objects in the same Git repository. In other words, IPFS provides a high-throughput, content-addressed block storage model with hyperlinks associated with the content. This forms a generalized Merkle directed acyclic graph (DAG). IPFS combines a distributed hash table, encouraged block exchange, and a self-certified namespace. IPFS has no single point of failure, and nodes do not need to trust each other. Distributed content delivery can save bandwidth and prevent DDoS attacks that HTTP solutions may encounter.

[0009] The file system can be accessed in a variety of ways, including FUSE and HTTP. Adding local files to the IPFS file system makes them available to the world. File representation is based on its hash, so it is easy to cache. Files are distributed using a BitTorrent-based protocol. Other users who view the content also help to make the content available to others on the network. IPFS has a name service called IPNS, which is a PKI-based global namespace for building a trust chain, which is compatible with other NS and can map DNS, onion, bit, etc. to IPNS.

[0010] Each file and all the blocks in it are given a unique fingerprint called a cryptographic hash. IPFS deletes files with the same hash value through the network, and can determine which files are redundant through calculation. And track the version history of each file. When looking for a file, you can find the file on the network by looking for the hash value of the content and find the desired file. Using IPNS (decentralized naming system), each file can be collaboratively named with an easy-to-read name, so that you can easily find the desired file.

[0011] 3. Distributed Hash Table

[0012] DHT (Distributed Hash Table) is a network similar to Tracker that returns seed information based on seed signatures. DHT stands for Distributed Hash Table, which is a distributed storage method. Without the need for a server, each client is responsible for routing in a small range and storing a small portion of data, thereby achieving addressing and storage of the entire DHT network. The new version of BitComet allows simultaneous connection to the DHT network and Tracker, which means that even without connecting to the Tracker server at all, it can still download well because it can find other users downloading the same file in the DHT network.

[0013] In the related art, in the overall distributed network storage, the backup data is redundant and the data is difficult to retrieve quickly.

[0014] Therefore, it is necessary to provide a new distributed storage search method and system based on secondary routing to solve the above technical problems. Summary of the invention

[0015] The main purpose of the present invention is to provide a distributed storage search method and system based on secondary routing, aiming to solve the technical problems of redundant backup data and slow data search speed in related technologies.

[0016] To achieve the above object, the present invention provides a distributed storage search method based on secondary routing, comprising the following steps:

[0017] S1, set the composition attributes of the two-level routing table;

[0018] S2, when storing data, adds two levels of routing tables;

[0019] S3, when reading data, the secondary routing table is searched.

[0020] Preferably, the step of setting the composition attributes of the two-level routing table specifically includes the following:

[0021] S11, setting the composition attributes of the second-level router according to the node hash function and the data hash function;

[0022] S12, the first-level routing table is composed according to the Kademlia algorithm of DHT.

[0023] Preferably, when storing data, the steps of adding two levels of routing tables respectively include the following:

[0024] S21, adding the second level routing table;

[0025] S22, adding the first-level routing table.

[0026] Preferably, when reading data, the step of searching the secondary routing table specifically includes the following:

[0027] S31, searching the second level routing table;

[0028] S32, searching the first-level routing table.

[0029] In order to solve the above technical problems, the present invention also provides a distributed storage search system based on secondary routing, comprising:

[0030] A setting module, the setting module is used to set the composition attributes of the two-level routing table;

[0031] A storage module, wherein when the storage module is used to store data, two levels of routing tables are added respectively;

[0032] A search module is used to search the secondary routing table when reading data.

[0033] Preferably, the setting module is specifically used for:

[0034] Setting composition attributes of the second-level router according to the node hash function and the data hash function;

[0035] The first-level routing table is composed according to the Kademlia algorithm of DHT.

[0036] Preferably, the storage module is specifically used for:

[0037] the increase of the second level routing table;

[0038] The first level routing table is increased.

[0039] Preferably, the search module specifically includes:

[0040] Searching the second level routing table;

[0041] The first level routing table is looked up.

[0042] The distributed storage search method and system based on secondary routing provided by the present invention improves the DHT routing table and adds secondary routing on the basis of DHT. It not only speeds up the data search speed, but also improves the hit rate of data access. It technically improves the user request response speed. In the overall distributed network storage, it avoids excessive backup redundant data and increases the data storage capacity. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a preferred embodiment of the distributed storage search method based on secondary routing provided by the present invention;

[0044] Figure 2 for Figure 1 The work flow diagram of step S1 shown;

[0045] Figure 3 for Figure 1 The work flow diagram in step S2 shown;

[0046] Figure 4 for Figure 1 The work flow diagram in step S3 shown;

[0047] Figure 5 for Figure 1The secondary routing topology diagram in the distributed storage search method based on secondary routing is shown;

[0048] Figure 6 for Figure 1 The first-level routing topology diagram in the distributed storage search method based on second-level routing is shown;

[0049] Figure 7 An architectural diagram of the distributed storage search system based on secondary routing provided by the present invention.

[0050] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0052] The invention provides a distributed storage search method based on secondary routing.

[0053] Please refer to Figure 1-6 To achieve the above object, in one embodiment of the present invention, a distributed storage search method based on secondary routing is characterized by comprising the following steps:

[0054] S1, set the composition attributes of the two-level routing table;

[0055] S2, when storing data, adds two levels of routing tables;

[0056] S3, when reading data, the secondary routing table is searched.

[0057] The distributed storage search method based on secondary routing provided by the present invention increases secondary routing on the basis of DHT by improving the DHT routing table. It not only speeds up the data search speed, but also improves the hit rate of data access. It technically improves the user request response speed. In the overall distributed network storage, it avoids excessive backup redundant data and increases the data storage capacity.

[0058] The step S1 specifically includes the following:

[0059] S11, setting the composition attributes of the second-level router according to the node hash function and the data hash function; wherein the composition attributes of the second-level routing table are set as follows:

[0060] Distributed network nodes are evenly dispersed according to the node hash function F(IP, Port) = (IP + Port) % mod; IP is the IPv4 address, Port is the port, and Mod is the maximum hash space;

[0061] The node hash function F(IP, Port) uses integer division modulus method to convert the node's IPv4 and Port into integers, and then calculates with the maximum hash space (mod, i.e. the number of all network nodes) to obtain the hash value (nodehash) of the network node;

[0062] Then, the hash values ​​of the network nodes are grouped according to the maximum number of links of a single node;

[0063] Data hash function F(merkle root hash) = merkle root hash%mod;

[0064] According to the Merkle Tree algorithm, the Merkle root hash of the stored data is calculated;

[0065] The data hash function F (merkle root hash) uses the integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space (mod, that is, the number of all network nodes) to calculate the hash value (data hash) of the stored data.

[0066] S12, the first-level routing table is composed according to the Kademlia algorithm of DHT. The composition attributes of the first-level routing table are as follows:

[0067] The first-level routing table is composed of the DHT distributed hash table; the first-level routing table is composed according to the DHT Kademlia algorithm.

[0068] The step S2 specifically includes the following:

[0069] S21, adding the second level routing table;

[0070] Specifically, according to the Merkle Tree algorithm, the Merkle root hash of the stored data is calculated;

[0071] Then calculate the hash value of the data according to the data hash function F (merkle root hash) = merkle root hash% mod;

[0072] The data hash function F (merkle root hash) uses integer division modulo method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space (mod, i.e. the number of all network nodes) to calculate the hash value (data hash) of the stored data;

[0073] The node hash value (node ​​hash) is mapped to the data hash value (data hash) to increase the secondary routing table; specifically, the data hash value (data hash) is used to find a similar node hash value (node ​​hash), thereby finding the node and saving the data to this node.

[0074] S22, adding the first-level routing table, that is, adding routing table entries according to the Kademlia algorithm of DHT.

[0075] The step S3 specifically includes the following:

[0076] S31, searching the second level routing table;

[0077] Specifically, the Merkle root hash is used to search for data;

[0078] Then, according to the hash function F (merkle root hash) = merkle root hash% mod, calculate the hash value of the data (data hash);

[0079] In the second-level routing table, use the data hash value to find the node hash value close to it;

[0080] When a similar node is found, data is read from this node;

[0081] S32, searching the first-level routing table.

[0082] When no similar node is found, the node corresponding to the Merkle root hash of the data is searched from the first-level routing table, and the data is read from this node; wherein, the search of the first-level routing table: search according to the routing table of DHT.

[0083] The distributed storage search method based on secondary routing provided by the present invention has the following innovative features:

[0084] Innovation 1: In the distributed network, based on the current DHT routing, add the second-level routing. The hashing method of the second-level routing table is:

[0085] (1) Node hash function F(IP,Port) = (IP+Port)%mod;

[0086] The hash function F(IP,Port) uses the integer division modulo method to convert the node's IPv4 and Port into integers, and then calculates them with the maximum hash space (mod, that is, the number of all network nodes) to obtain the hash value (nodehash) of the network node.

[0087] (2) Data hash function F (merkle root hash) = merkle root hash% mod;

[0088] ① According to the Merkle Tree algorithm, the Merkle root hash value of the stored data is calculated;

[0089] ② The hash function F (merkle root hash) uses the integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space (mod, that is, the number of all network nodes) to calculate the hash value (data hash) of the stored data.

[0090] Innovation point 2: In a distributed network, a query method based on two-level routing: (1) query when writing data and (2) query when reading data.

[0091] The present invention also provides a distributed storage search system based on secondary routing.

[0092] See also Figure 7 ,The distributed storage search system based on secondary routing includes:

[0093] A setting module, the setting module is used to set the composition attributes of the two-level routing table;

[0094] A storage module, wherein when the storage module is used to store data, two levels of routing tables are added respectively;

[0095] A search module is used to search the secondary routing table when reading data.

[0096] The distributed storage search system based on secondary routing provided by the present invention increases secondary routing on the basis of DHT by improving the DHT routing table. It not only speeds up the data search speed, but also improves the hit rate of data access. It technically improves the user request response speed. In the overall distributed network storage, it avoids excessive backup redundant data and increases the data storage capacity.

[0097] The setting module is specifically used for:

[0098] The composition attributes of the second-level router are set according to the node hash function and the data hash function; wherein the composition attributes of the second-level routing table are set as follows:

[0099] Distributed network nodes are evenly dispersed according to the node hash function F(IP, Port) = (IP + Port) % mod; IP is the IPv4 address, Port is the port, and Mod is the maximum hash space;

[0100] The node hash function F(IP, Port) uses integer division modulus method to convert the node's IPv4 and Port into integers, and then calculates with the maximum hash space (mod, i.e. the number of all network nodes) to obtain the hash value (nodehash) of the network node;

[0101] Then, the hash values ​​of the network nodes are grouped according to the maximum number of links of a single node;

[0102] Data hash function F(merkle root hash) = merkle root hash%mod;

[0103] According to the Merkle Tree algorithm, the Merkle root hash of the stored data is calculated;

[0104] The data hash function F (merkle root hash) uses the integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space (mod, that is, the number of all network nodes) to calculate the hash value (data hash) of the stored data.

[0105] The first-level routing table is composed according to the Kademlia algorithm of DHT. The composition attributes of the first-level routing table are as follows:

[0106] The first-level routing table is composed of the DHT distributed hash table; the first-level routing table is composed according to the DHT Kademlia algorithm.

[0107] The storage module is specifically used for:

[0108] the increase of the second level routing table;

[0109] Specifically, according to the Merkle Tree algorithm, the Merkle root hash of the stored data is calculated;

[0110] Then calculate the hash value of the data according to the data hash function F (merkle root hash) = merkle root hash% mod;

[0111] The data hash function F (merkle root hash) uses integer division modulo method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space (mod, i.e. the number of all network nodes) to calculate the hash value (data hash) of the stored data;

[0112] The node hash value (node ​​hash) is mapped to the data hash value (data hash) to increase the secondary routing table; specifically, the data hash value (data hash) is used to find a similar node hash value (node ​​hash), thereby finding the node and saving the data to this node.

[0113] The first-level routing table is increased. Specifically, routing table entries are increased according to the Kademlia algorithm of DHT.

[0114] The search module specifically includes:

[0115] Searching the second level routing table;

[0116] Specifically, the Merkle root hash is used to search for data;

[0117] Then, according to the hash function F (merkle root hash) = merkle root hash% mod, calculate the hash value of the data (data hash);

[0118] In the second-level routing table, use the data hash value to find the node hash value close to it;

[0119] When a similar node is found, data is read from this node;

[0120] The first level routing table is looked up.

[0121] Specifically, when no similar node is found, the node corresponding to the Merkle root hash of the data is searched from the first-level routing table, and the data is read from this node; wherein, the search of the first-level routing table: search according to the routing table of DHT.

[0122] The distributed storage search system based on secondary routing provided by the present invention has the following innovative features:

[0123] Innovation 1: In the distributed network, based on the current DHT routing, add the second-level routing. The hashing method of the second-level routing table is:

[0124] (1) Node hash function F(IP,Port) = (IP+Port)%mod;

[0125] The hash function F(IP,Port) uses the integer division modulo method to convert the node's IPv4 and Port into integers, and then calculates them with the maximum hash space (mod, that is, the number of all network nodes) to obtain the hash value (nodehash) of the network node.

[0126] (2) Data hash function F (merkle root hash) = merkle root hash% mod;

[0127] ① According to the Merkle Tree algorithm, the Merkle root hash value of the stored data is calculated;

[0128] ② The hash function F (merkle root hash) uses the integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space (mod, that is, the number of all network nodes) to calculate the hash value (data hash) of the stored data.

[0129] Innovation point 2: In a distributed network, a query method based on two-level routing: (1) query when writing data and (2) query when reading data.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to enter the method described in each embodiment of the present invention.

[0131] In the description of this specification, the description with reference to the terms "an embodiment", "another embodiment", "other embodiments", or "first embodiment to Xth embodiment" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, method steps or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0132] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0133] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0134] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A distributed storage search method based on secondary routing, It is characterized in that The steps include: S1, set the composition attributes of the two-level routing table; S2, when storing data, adds two levels of routing tables; S3, when reading data, the secondary routing table is searched; The steps of setting the composition attributes of the two-level routing table specifically include the following: S11, setting the composition attributes of the second-level router according to the node hash function and the data hash function; wherein the composition attributes of the second-level routing table are set as follows: Distributed network nodes are evenly dispersed according to the node hash function F(IP, Port) = (IP + Port) % mod; IP is the IPv4 address, Port is the port, and Mod is the maximum hash space; The node hash function F(IP, Port) uses integer division and remainder method to convert the node's IPv4 and Port into integers, and then calculates with the maximum hash space to obtain the hash value of the network node; Then group the hash values ​​of the network nodes according to the maximum number of links of a single node; Data hash function F(merkle root hash) = merkle root hash%mod; According to the Merkle tree algorithm, the Merkle root hash value of the stored data is calculated; The data hash function F (merkle root hash) uses integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space to calculate the hash value of the stored data; S12, the first-level routing table is composed according to the Kademlia algorithm of DHT; the composition attributes of the first-level routing table are as follows: The first-level routing table is composed of DHT distributed hash tables; the first-level routing table is composed according to the DHT Kademlia algorithm; When storing data, the steps of adding two levels of routing tables respectively include the following: S21, adding the second-level routing table; specifically, calculating the Merkle root hash value of the stored data according to the Merkle tree algorithm; Then calculate the hash value of the data according to the data hash function F (merkle root hash) = merkle root hash% mod; The data hash function F (merkle root hash) uses integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space to calculate the hash value of the stored data; The hash value of the node is mapped to the hash value of the data, and a secondary routing table is added. Specifically, the data hash value is used to find a similar node hash value, thereby finding the node and saving the data to this node. S22, the first level routing table is increased; that is, the routing table items are increased according to the Kademlia algorithm of DHT; when reading the data, the steps of searching the secondary routing table specifically include the following: S31, searching the second-level routing table; specifically, using the Merkle root hash value to search for data; Then calculate the hash value of the data according to the hash function F (merkle root hash) = merkle root hash% mod; In the second-level routing table, the data hash value is used to find the node hash value close to it; When a similar node is found, data is read from this node; S32, searching the first-level routing table; when no similar node is found, searching the first-level routing table for the node corresponding to the Merkle root hash of the data, and reading data from this node; wherein, searching the first-level routing table: searching according to the routing table of DHT.

2. A distributed storage search system based on secondary routing, It is characterized in that include: A setting module, the setting module is used to set the composition attributes of the two-level routing table; A storage module, wherein when the storage module is used to store data, two levels of routing tables are added respectively; A search module, which is used to search the secondary routing table when reading data; The setting module is specifically used for: The composition attributes of the second-level router are set according to the node hash function and the data hash function; wherein the composition attributes of the second-level routing table are set as follows: Distributed network nodes are evenly dispersed according to the node hash function F(IP, Port) = (IP + Port) % mod; IP is the IPv4 address, Port is the port, and Mod is the maximum hash space; The node hash function F(IP, Port) uses integer division and remainder method to convert the node's IPv4 and Port into integers, and then calculates with the maximum hash space to obtain the hash value of the network node; Then group the hash values ​​of the network nodes according to the maximum number of links of a single node; Data hash function F(merkle root hash) = merkle root hash%mod; According to the Merkle tree algorithm, the Merkle root hash value of the stored data is calculated; The data hash function F (merkle root hash) uses integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space to calculate the hash value of the stored data; The first-level routing table is composed according to the Kademlia algorithm of DHT. The composition attributes of the first-level routing table are as follows: The first-level routing table is composed of DHT distributed hash tables; the first-level routing table is composed according to the DHT Kademlia algorithm; The storage module is specifically used for: The second-level routing table is increased; specifically, according to the Merkle tree algorithm, the Merkle root hash value of the stored data is calculated; Then calculate the hash value of the data according to the data hash function F (merkle root hash) = merkle root hash% mod; The data hash function F (merkle root hash) uses integer division remainder method to convert the merkle root hash into an integer, and then calculates it with the maximum hash space to calculate the hash value of the stored data; The hash value of the node is mapped to the hash value of the data, and a secondary routing table is added. Specifically, the data hash value is used to find a similar node hash value, thereby finding the node and saving the data to this node. The first-level routing table is increased; that is, routing table entries are increased according to the Kademlia algorithm of DHT; The search module specifically includes: The second-level routing table is searched; specifically, the data is searched using the Merkle root hash value; Then calculate the hash value of the data according to the hash function F (merkle root hash) = merkle root hash% mod; In the second-level routing table, the data hash value is used to find the node hash value close to it; When a similar node is found, read the data from this node; The first-level routing table is searched; when no similar node is found, the node corresponding to the Merkle root hash of the data is searched from the first-level routing table, and the data is read from this node; wherein, the first-level routing table is searched: according to the routing table search of DHT.

Images