A distributed storage network construction method based on IPFS and a distributed storage network

Abstract

The embodiment of the present application discloses a distributed storage network construction method and a distributed storage network based on IPFS. It is used to solve the problems of low disk utilization and low file security in the existing distributed storage system. The method includes: the NAS server sends the Interstellar File System IPFS installation file and the deployment script to the node to be deployed; wherein the node to be deployed is a user terminal in the current IPFS local area network; the node to be deployed automatically installs the IPFS client based on the IPFS installation package; the node to be deployed sets the shared storage path of the node to be deployed based on the deployment script; the NAS server verifies the identity of the node to be deployed based on the key in the USB KEY corresponding to the node to be deployed; if the identity of the node to be deployed is legal, the node to be deployed is determined as an IPFS node, and the IPFS node is started. The present application shares the storage space of the user terminal, which is extremely secure, compatible, fast, and easy to use.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for constructing a storage network based on IPFS. Background Art

[0002] With the advancement of intelligent social management, the demand for data storage is increasing, and the Storage Area Network (SAN) system and Network Attached Storage (NAS) system are good choices for long-term storage and protection of data. These two systems have been widely used in communities, small businesses, residential areas and multi-person families, providing them with professional and reliable solutions for hot and cold data storage.

[0003] However, both SAN and NAS systems are highly dependent on RAID (Redundant Arrays of Independent Disks), while RAID systems have low utilization of disks, which results in low utilization of disks in SAN and NAS systems, and in turn increases the cost of storage. In addition, the communication between multiple nodes of SAN and NAS systems does not authenticate user identities, which poses a security risk and cannot guarantee the security of file storage. Summary of the invention

[0004] The embodiments of the present application provide a storage network construction method and device based on IPFS, which are used to solve the following technical problems: the existing distributed storage system has low disk utilization and low file security.

[0005] The present application embodiment adopts the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a method for constructing a distributed storage network based on IPFS, the method comprising: a NAS server sends an InterPlanetary File System IPFS installation file and a deployment script to a node to be deployed; wherein the node to be deployed is a user terminal that needs to join a distributed storage network; the node to be deployed automatically installs an IPFS client based on the IPFS installation file; the node to be deployed sets a shared storage path of the node to be deployed based on the deployment script; the NAS server verifies the identity of the node to be deployed based on a key in a USB KEY corresponding to the node to be deployed; if the identity of the node to be deployed is legal, the node to be deployed is determined as an IPFS node, and the IPFS node is started.

[0007] In a feasible implementation, the node to be deployed sets the shared storage path of the node to be deployed based on the deployment script, specifically including: the node to be deployed reads the storage shared path name in the deployment script, and sets the path corresponding to the storage shared path name as the storage shared path of the node to be deployed; if the path corresponding to the storage shared path name does not exist in the node to be deployed, then creates the storage shared path according to the storage shared path name.

[0008] In a feasible implementation manner, the NAS server verifies the identity of the node to be deployed based on the key in the USB KEY corresponding to the node to be deployed, specifically including: the node to be deployed sends a verification request to the NAS server; the NAS server returns a random number to the USB KEY based on the verification request; the NAS server operates the random number and a pre-stored first key through a one-way hash algorithm RSA to obtain a first RSA operation result; the USB KEY operates the random number and a pre-stored second key through a one-way hash algorithm RSA to obtain a corresponding second RSA operation result; and returns the second RSA operation result to the NAS server; the NAS server determines whether the identity of the node to be deployed is legal based on the first RSA operation result and the second RSA operation result.

[0009] In a feasible implementation manner, the NAS server determines whether the identity of the node to be deployed is legal based on the first RSA operation result and the second RSA operation result, specifically including: the NAS server compares the first RSA operation result with the second RSA operation result; if the first RSA operation result is the same as the second RSA operation result, the NAS server determines that the identity of the node to be deployed is legal; if the first RSA operation result is different from the second RSA operation result, the NAS server determines that the identity of the node to be deployed is illegal.

[0010] In a feasible implementation, after determining the node to be deployed as an IPFS node and starting the IPFS node, the method further includes: the NAS server sets the determined first IPFS node as a bootstrap node, and adds the bootstrap node to the IPFS local area network; the NAS server adds the started second and subsequent nodes to be deployed as IPFS nodes to the IPFS local area network, and together with the bootstrap node, constitutes a distributed storage network.

[0011] In a feasible implementation manner, after the NAS server adds the second and subsequent nodes to be deployed started as IPFS nodes to the IPFS local area network, and together with the bootstrap node, forms a distributed storage network, the method further includes: the file service module encapsulates the upload interface and download interface of the distributed storage network through a Merkle tree algorithm to verify the data uploaded or downloaded by the file service module; if any user terminal sends a file upload request to the distributed storage network, the file service module calls the upload interface of the distributed storage network based on the file upload request; wherein each user terminal is also an IPFS node; the file service module divides the original file uploaded by the user terminal into several data blocks of preset sizes, and uploads the data blocks to several IPFS nodes of the distributed storage network through the upload interface.

[0012] In a feasible implementation, after uploading the data blocks to several IPFS nodes of the distributed storage network respectively, the method further includes: the IPFS node receiving the data block calculates the hash value of the data block based on a preset hash function, and sends the hash value to the file service module; the file service module stores the received several hash values ​​in the distributed hash table corresponding to the user terminal.

[0013] In a feasible implementation, after the file service module stores the received hash values ​​in the distributed hash table corresponding to the user terminal, the method further includes: the user terminal sends a file download request to the file service module; the file service module searches for the IPFS node where each data block is located in the distributed hash table according to the file download request; and calls the download interface of the distributed storage network to obtain the data block in the corresponding IPFS node, and performs integrity check on the downloaded data; and splices the obtained data blocks into the original file, and sends it to the user terminal.

[0014] On the other hand, an embodiment of the present application also provides a distributed storage network based on IPFS, including: a NAS server, used to send IPFS installation files and deployment scripts to the nodes to be deployed; and based on the key in the USB KEY corresponding to the nodes to be deployed, verify the identity of the nodes to be deployed; a number of IPFS nodes, used to store files; wherein the IPFS node is also a user terminal.

[0015] In a feasible implementation, the distributed storage network also includes: a file service module, which is used to call the upload interface of the distributed storage network based on the file upload request of the user terminal; and divide the file uploaded by the user terminal into several data blocks of preset sizes, and upload the data blocks to several IPFS nodes respectively; store several hash values ​​returned by several IPFS nodes in the distributed hash table of the user terminal; the file service module is also used to search for the corresponding IPFS node in the distributed hash table according to the file download request of the user terminal, and call the download interface of the distributed storage network to obtain the data block in the corresponding IPFS node, and perform integrity check on the downloaded data; splice the obtained data blocks into the original file and send it to the user terminal.

[0016] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0017] 1. Use USB KEY peripherals as node mutual trust tools. The keys pre-stored in USB KEY cannot be read. Theoretically, the probability of forgery is about 1 in 2 to the 80th power. It has extremely high security, good compatibility, fast speed and is easy to use.

[0018] 2. Make full use of the characteristics of the IPFS network to safely build a local area network and improve the disk utilization rate of each node.

[0019] 3. Effectively reduce the use cost of NAS. Based on the NAS server disk, share the storage space of the user terminal, add the user terminal as an IPFS node to the storage network, and improve the scalability of the storage space.

[0020] 4. The file service module can support large concurrent scenarios, allowing large files to be transmitted in fragments, fully occupying the network bandwidth and improving the efficiency of file transfer between nodes.

[0021] 5. The NAS server sends a unified installation package and deployment script to each node. Each node automatically installs the IPFS client and automatically runs the deployment script, greatly reducing the cost of deploying the IPFS local area network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0023] Figure 1A flow chart of a distributed storage network construction method based on IPFS provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a distributed storage network based on IPFS provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0026] The present application embodiment provides a method for constructing a distributed storage network based on IPFS, such as Figure 1 As shown, the method for constructing a distributed storage network based on IPFS specifically includes steps S101-S106:

[0027] S101. The NAS server sends the IPFS installation file and deployment script to the node to be deployed.

[0028] Specifically, first deploy IPFS service and file service on a NAS server, and pre-store a key for subsequent node identity authentication. The deployed NAS server is used as the master node of the distributed storage network.

[0029] Furthermore, after the IPFS installation file is decompressed through the NAS server, it is packaged together with the deployment script and sent to each node to be deployed. Among them, the deployment script is a shell script, and the node to be deployed is the user terminal that needs to be built in the distributed storage network.

[0030] For example, if user terminals 1, 2, 3, and 4 need to be built in a distributed storage network, then the IPFS installation file and deployment script are sent to user terminals 1, 2, 3, and 4 through the NAS server.

[0031] As a feasible implementation method, the nodes to be deployed can be terminals such as servers, minicomputers, personal computers, micro storage devices, etc., and the IPFS environment can be deployed in Windows systems, various distribution versions of minimallinux systems, and operating systems developed by manufacturers such as Synology based on the Linux kernel.

[0032] S102. The node to be deployed automatically installs the IPFS client based on the IPFS installation file, and sets the shared storage path of the node to be deployed based on the deployment script.

[0033] Specifically, after the node to be deployed receives the IPFS installation file and deployment script sent by the NAS server, it automatically installs the IPFS client through the IPFS installation file.

[0034] Furthermore, after the client is installed on the node to be deployed, the storage sharing path name in the deployment script is read, and the path corresponding to the storage sharing path name is set as the storage sharing path of the node to be deployed. If the path corresponding to the storage sharing path name does not exist in the node to be deployed, a storage sharing path is created according to the storage sharing path name. By specifying the storage sharing path in the deployment script, the NAS server sets a part of the storage space in each user terminal as the storage space of the distributed storage network, thereby sharing the user's storage space and improving the scalability of the NAS storage space.

[0035] For example, after receiving the deployment script, a user terminal reads the storage sharing path as C:\gongxiang, and first checks whether there is a gongxiang folder on its C drive. If so, this folder is used as a shared storage space. If not, a new gongxiang folder is automatically created on the C drive as a shared storage space.

[0036] It should be noted that there are other deployment requirements for the node environment to be deployed in the deployment script, which are not discussed in this application. In actual applications, the deployment script can be written according to actual needs.

[0037] S103: The NAS server verifies the identity of the node to be deployed based on the key in the USB KEY corresponding to the node to be deployed.

[0038] Specifically, after the deployment of the node to be deployed is completed, a USB KEY can be manually inserted into the node to be deployed, and the USB KEY pre-stores the same key as that in the NAS server.

[0039] Then the node to be deployed sends a verification request to the NAS server. After receiving the verification request, the NAS server returns a random number to the USB KEY inserted in the node to be deployed. At the same time, the NAS server uses the RSA algorithm to calculate the random number and its pre-stored key (i.e., the first key) to obtain the first RSA calculation result.

[0040] At the same time, after receiving the random number, the USB KEY uses the RSA algorithm to calculate the random number and the key (i.e., the second key) pre-stored in the USB KEY to obtain the corresponding second RSA calculation result, and then transmits the second RSA calculation result back to the NAS server.

[0041] Furthermore, the NAS server determines whether the identity of the node to be deployed is legal based on the first RSA operation result and the second RSA operation result. Specifically, the NAS server compares the first RSA operation result with the second RSA operation result. If the first RSA operation result is the same as the second RSA operation result, the identity of the node to be deployed is determined to be legal; if the first RSA operation result is different from the second RSA operation result, the identity of the node to be deployed is determined to be illegal.

[0042] S104. If the identity of the node to be deployed is legal, the node to be deployed is determined as an IPFS node, and the IPFS node is started.

[0043] Specifically, if the identity of the node to be deployed is determined to be legitimate, it means that the node to be deployed is a trustworthy node, so the node to be deployed is started as an IPFS node.

[0044] If the identity of the node to be deployed is illegal, the node to be deployed will not be started.

[0045] S105. The NAS server sets the first determined IPFS node as the bootstrap node, and adds the second and subsequent nodes to be deployed as IPFS nodes to the IPFS local area network, forming a distributed storage network together with the bootstrap node.

[0046] Specifically, the NAS server sets the first determined IPFS node as the bootstrap node, and adds the bootstrap node to the IPFS local area network. The second and subsequent nodes to be deployed are added to the IPFS local area network as ordinary IPFS nodes, and together with the bootstrap node, they form a distributed storage network.

[0047] In one embodiment, since the first IPFS node started is a NAS server, generally speaking, the NAS server is the bootstrap node of the distributed storage network.

[0048] S106. The file service module uploads and downloads files in the distributed storage network through a distributed hash table.

[0049] Specifically, when the file service is deployed in the NAS server, the upload interface and the download interface of the NAS server are encapsulated by the Merkle tree algorithm to verify the uploaded or downloaded data.

[0050] After the distributed storage network is successfully constructed, if any user terminal sends a file upload request to the distributed storage network, the file service module will call the upload interface of the NAS server in the distributed storage network based on the file upload request.

[0051] The file service module then divides the original file uploaded by the user terminal into several data blocks of preset sizes, and uploads the data blocks to several IPFS nodes of the distributed storage network through the upload interface. The IPFS node that receives the data block calculates the hash value of the received data block based on the preset hash function and sends the hash value to the file service module.

[0052] Furthermore, the file service module stores the received hash values ​​in a distributed hash table corresponding to the user terminal.

[0053] In one embodiment, if user terminal 1 sends a request to upload a file to the distributed storage network, the file service module will divide the file uploaded by user terminal 1 into 4 data blocks of equal or different sizes, and then store each data block in IPFS nodes 1, 2, 3, and 4 respectively. At the same time, IPFS nodes 1, 2, 3, and 4 respectively calculate the hash values ​​corresponding to the received data blocks, and then return the calculated hash values ​​to the file service module. The file service module saves these hash values ​​in a distributed hash table. For example, the hash value calculated by IPFS node 1 is KEY1, the hash value calculated by IPFS node 2 is KEY2, the hash value calculated by IPFS node 3 is KEY3, and the hash value calculated by IPFS node 4 is KEY4. Then the content in the distributed hash table is:

[0054] <IPFS1,KEY1> ,<IPFS2,KEY2> ,<IPFS3,KEY3> ,<IPFS4,KEY4> .

[0055] Among them, storing each data block in IPFS nodes 1, 2, 3, and 4 respectively means storing each data block in the shared storage path of user terminals 1, 2, 3, and 4 respectively. Using user terminals as IPFS nodes for distributed storage not only facilitates the storage of large files for users, but also makes full use of the storage space of user terminals.

[0056] Furthermore, when the user terminal sends a file download request to the file service module, the file service module searches the IPFS node where each data block is located in the distributed hash table according to the file download request, and calls the download interface of the distributed storage network to obtain the corresponding data block in the corresponding IPFS node. Since the download interface encapsulates the Merkle tree algorithm, the integrity of the downloaded data can be checked.

[0057] Furthermore, the file service module splices the acquired data blocks into an original file and sends it to the user terminal.

[0058] In one embodiment, if the user terminal 1 sends a request to download a file to the distributed storage network, the file service module will search for the IPFS node where each data block is located in turn according to the distributed hash table of the user terminal 1, and then download the data block from each IPFS node, and finally splice it into a complete file. The advantage of shard storage is that when an error occurs in a downloaded data block, there is no need to download the entire file, only the data block with the error needs to be downloaded, which greatly saves download time.

[0059] In addition, this application also provides a distributed storage network based on IPFS, such as Figure 2 As shown, the IPFS-based distributed storage network 200 includes: a NAS server 210, several IPFS nodes 220 and a file service module 230.

[0060] The NAS server 210 is used to send the IPFS installation file and the deployment script to the node to be deployed, and to verify the identity of the node to be deployed based on the key in the USB KEY corresponding to the node to be deployed.

[0061] The number of IPFS nodes 220 includes many IPFS nodes, such as Figure 2 IPFS node 1, IPFS node 2, etc. Each IPFS node can store files. And each IPFS node is also a user terminal, such as Figure 2 As shown, user terminal 1 is IPFS node 1, and user terminal 2 is IPFS node 2. It should be noted that Figure 2 Only two IPFS nodes and two user terminals are listed as examples, but the several IPFS nodes in this application are not limited to two nodes, but are constructed according to actual needs.

[0062] The file service module 230 includes a file segment upload module 231 and a file download module 232 .

[0063] The file block upload module 231 is used to call the upload interface of the distributed storage network based on the file upload request of the user terminal, and divide the file uploaded by the user terminal into a number of data blocks of preset sizes, and upload these data blocks to each IPFS node in the number of IPFS nodes 220. Then, the hash value returned by each IPFS node is stored in the distributed hash table.

[0064] The file download module 232 is used to search the IPFS node where each data block is located in the distributed hash table according to the file download request of the user terminal, call the download interface of the distributed storage network, obtain the data block in the corresponding IPFS node, and perform integrity check on the downloaded data; finally, all the obtained data blocks are spliced ​​into the original file and sent to the user terminal.

[0065] This application builds a secure storage network through IPFS, reducing the local disk cost of NAS. The security of file transfer is ensured by using USB KEY. Through the distributed storage network construction method provided by this application, the user's storage space can be safely shared, making the user's storage space part of NAS storage.

[0066] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0067] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0068] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for constructing a distributed storage network based on IPFS, characterized in that: The method comprises: The NAS server sends the Interplanetary File System IPFS installation file and deployment script to the node to be deployed; wherein the node to be deployed is a user terminal that needs to be added to the distributed storage network; The node to be deployed automatically installs the IPFS client based on the IPFS installation file; The node to be deployed sets a shared storage path of the node to be deployed based on the deployment script; The NAS server verifies the identity of the node to be deployed based on the key in the USB KEY corresponding to the node to be deployed, specifically including: The node to be deployed sends a verification request to the NAS server; The NAS server returns a random number to the USB KEY based on the verification request; The NAS server operates the random number and the pre-stored first key through a one-way hash algorithm RSA to obtain a first RSA operation result; The USB KEY calculates the random number and the pre-stored second key through a one-way hash algorithm RSA to obtain a corresponding second RSA calculation result; and transmits the second RSA calculation result back to the NAS server; The NAS server determines whether the identity of the node to be deployed is legal based on the first RSA calculation result and the second RSA calculation result, specifically including: The NAS server compares the first RSA operation result with the second RSA operation result; if the first RSA operation result is the same as the second RSA operation result, the NAS server determines that the identity of the node to be deployed is legal; if the first RSA operation result is different from the second RSA operation result, the NAS server determines that the identity of the node to be deployed is illegal; If the identity of the node to be deployed is legal, the node to be deployed is determined as an IPFS node, and the IPFS node is started; The NAS server sets the first determined IPFS node as the bootstrap node, and adds the bootstrap node to the IPFS local area network; The NAS server adds the second and subsequent nodes to be deployed as IPFS nodes to the IPFS local area network, and together with the bootstrap node, forms a distributed storage network; The file service module encapsulates the upload interface and download interface of the distributed storage network through the Merkle tree algorithm to verify the data uploaded or downloaded by the file service module; If any user terminal sends a file upload request to the distributed storage network, the file service module calls the upload interface of the distributed storage network based on the file upload request; wherein each user terminal is also an IPFS node; The file service module divides the original file uploaded by the user terminal into a number of data blocks of preset sizes, and uploads the data blocks to a number of IPFS nodes of the distributed storage network through the upload interface; The IPFS node that receives the data block calculates a hash value of the data block based on a preset hash function, and sends the hash value to the file service module; The file service module stores the received hash values ​​in a distributed hash table corresponding to the user terminal; The user terminal sends a file download request to the file service module; The file service module searches the distributed hash table for the IPFS node where each data block is located according to the file download request; and calls the download interface of the distributed storage network to obtain the data block in the corresponding IPFS node, and performs integrity check on the downloaded data; The acquired data blocks are spliced ​​into the original file, and sent to the user terminal.

2. According to a method for constructing a distributed storage network based on IPFS according to claim 1, it is characterized in that: The node to be deployed sets a shared storage path of the node to be deployed based on the deployment script, specifically including: The node to be deployed reads the storage shared path name in the deployment script, and sets the path corresponding to the storage shared path name as the storage shared path of the node to be deployed; If the path corresponding to the storage sharing path name does not exist in the node to be deployed, the storage sharing path is created according to the storage sharing path name.

3. A distributed storage network using the distributed storage network construction method based on IPFS as claimed in claim 1, characterized in that: The distributed storage network includes: The NAS server is used to send the IPFS installation file and the deployment script to the node to be deployed; and verify the identity of the node to be deployed based on the key in the USB KEY corresponding to the node to be deployed; Several IPFS nodes are used to store files; wherein the IPFS node is also a user terminal.

4. The distributed storage network according to claim 3, characterized in that: The distributed storage network also includes: The file service module is used to call the upload interface of the distributed storage network based on the file upload request of the user terminal; and divide the file uploaded by the user terminal into a number of data blocks of preset sizes, and upload the data blocks to a number of IPFS nodes respectively; and store a number of hash values ​​returned by the number of IPFS nodes in the distributed hash table of the user terminal; The file service module is also used to search for the corresponding IPFS node in the distributed hash table according to the file download request of the user terminal, call the download interface of the distributed storage network, obtain the data block in the corresponding IPFS node, and perform integrity check on the downloaded data; splice the obtained data blocks into the original file and send it to the user terminal.

Images