A method for optimizing IPFS data transmission based on network multipath

Abstract

The present invention discloses an IPFS data transmission optimization method based on network multipath, and relates to the technical field of IPFS data transmission. The present invention at least includes the following steps: embedding the MPQUIC protocol into IPFS as an extension plug-in; the node of the IPFS that integrates the MPQUIC protocol chooses whether to start the monitoring of the MPQUIC protocol according to the physical environment; IPFS establishes a connection with the IPFS peer node according to the MPQUIC protocol keyword and the node identity information; and selects the user file and performs the file transfer operation. The present invention greatly utilizes the network bandwidth resources during data transmission. At the upload end, the upload time is shortened due to the integration of the bandwidth of multiple network paths; the present invention will not conflict with other versions of IPFS, and the MPQUIC protocol introduced by the integration is compatible with the QUIC protocol natively supported by IPFS.

Description

Technical Field

[0001] The present invention relates to the technical field of IPFS data transmission, and in particular to an IPFS data transmission optimization method based on network multipath. Background Art

[0002] IPFS (InterPlanetary File System) is a content-addressable, peer-to-peer, distributed file system. IPFS uses content-address addressing technology, which means that it searches by file content rather than by the network address of the file. Multiple nodes running IPFS programs constitute the IPFS storage network. The data stored in the IPFS network is divided into multiple blocks and stored in different nodes. When a node requests data in the network, it caches the file locally in the node. That is, each node saves the cache of the files it has downloaded to ensure that even if a node that stores the resource is removed from the IPFS network, the resource can still be accessed by other nodes.

[0003] When a user uploads a file to an IPFS node for storage, the node will store the file in blocks. Each file block is organized in the form of a Merkle directed acyclic graph, and the root hash of the Merkle directed acyclic graph is used to represent the file. As an important peer-to-peer, decentralized file system, IPFS enables users to store and share data on their devices in a censorship-resistant way. It has received increasing attention as a storage solution due to the rapid growth of social data (images, videos, etc.), which are often used by decentralized applications such as blockchain online social media Peepeth, Akasha, and DTube.

[0004] Research has shown that even small improvements in network performance can have a significant positive impact on the customer base. Therefore, increased bandwidth utilization plays an important role in speeding up IPFS data transfer rates.

[0005] Nowadays, more and more communication devices are equipped with multiple network interfaces, such as laptops with LAN interfaces and WiFi interfaces. In order to utilize the resources on multiple network interfaces, Multipath QUIC (MPQUIC) as a multipath extension of the QUIC protocol implements concurrent multipath transmission on multiple network paths. It inherits the flexibility of QUIC and is expected to have great potential for widespread deployment.

[0006] However, in the process of file transfer, the existing IPFS technology does not adopt any processing measures for devices equipped with multiple network interfaces, and cannot meet the customer's demand for fully utilizing the network bandwidth of multi-network interface devices. To this end, the present invention provides an optimization method for IPFS data transmission based on network multi-path to solve the problem that the network bandwidth of multi-network interface devices cannot be fully utilized in the existing technology.

[0007] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0008] The purpose of the present invention is to provide an IPFS data transmission optimization method based on network multi-path, which greatly utilizes network bandwidth resources during data transmission. At the upload end, the upload time is shortened due to the integration of the bandwidth of multiple network paths. The present invention will not conflict with other versions of IPFS. The MPQUIC protocol introduced by the integration is compatible with the QUIC protocol natively supported by IPFS to solve the problem that the network bandwidth of multiple network interface devices cannot be fully utilized in the prior art.

[0009] To achieve the above object, the present invention provides the following technical solution: an IPFS data transmission optimization method based on network multipath, comprising at least the following steps:

[0010] Embed the MPQUIC protocol into IPFS as an extension plug-in;

[0011] The IPFS node that integrates the MPQUIC protocol chooses whether to start monitoring the MPQUIC protocol based on the physical environment;

[0012] IPFS establishes a connection with the IPFS peer node based on the MPQUIC protocol keywords and node identity information;

[0013] Select the user files and transfer them.

[0014] Further preferably, the MPQUIC protocol is embedded in IPFS as an extension plug-in, comprising at least the following steps:

[0015] Define a new Protocol structure variable for the MPQUIC protocol and register it in the node component of IPFS;

[0016] Use the MultiAddress structure to represent the new network address of the IPFS node used for the MPQUIC connection;

[0017] Extend the listen method in the Swarm module in IPFS to listen for MPQUIC requests;

[0018] The dial method in Swarm is similarly extended to request a connection and establish an MPQUIC connection after listen handles its request.

[0019] Further preferably, the extension of the listen method in the Swarm module in IPFS to listen to the MPQUIC request includes at least the following steps:

[0020] When the MPQUIC protocol is configured and used to initiate a connection, the header of each data packet contains some necessary information about the transport protocol, including at least the name and version;

[0021] Use libp2p to analyze and extract this information from the data packet to identify MPQUIC requests;

[0022] The listen method in the Bitswap component of IPFS is used to call the MPQUIC-based transport implementation method to start the listen daemon process, thereby ensuring correct listening of MPQUIC requests;

[0023] When incoming requests are identified as MPQUIC requests, they are processed according to the MPQUIC protocol specification.

[0024] Further preferably, the dial method of the Swarm module in IPFS is extended, comprising at least the following steps:

[0025] Initiate a connection request based on the MPQUIC protocol to the peer end, and establish an MPQUIC connection after the peer end listens to process its request;

[0026] All established connections are added to the connection pool in the Swarm for incoming data transfer.

[0027] Further preferably, the MPQUIC-based transport implementation method comprises at least the following steps:

[0028] Define a reuseConn type, which mainly consists of UDP connection, mutex variable and reference count, for connection reuse;

[0029] Use listen and dial as methods in the Transport interface. The listen method is used by the target node to listen to and accept incoming requests, and the dial method is used by the connection initiator to obtain a connection.

[0030] In the process of implementing dial, UDP incoming requests are listened first, and then the corresponding connection is established through net.ListenUDP;

[0031] After that, the established UDP connection is first encapsulated in the variable reuseConn and used to create an MPQUIC session with multipath configuration to achieve MPQUIC connection;

[0032] To be compatible with connections in IPFS, the session is encapsulated in a variable of type conn and returned by dial;

[0033] To implement listen, first use net.ListenUDP to listen and accept UDP connections, which are encapsulated in the reuseConn variable;

[0034] Then activate the listener provided by MPQUIC through multipath configuration to listen for data packets from the UDP connection.

[0035] Further preferably, the IPFS node starts the monitoring of the MPQUIC protocol by default, mainly including:

[0036] When the IPFS node starts, it detects whether the physical host where the IPFS node is running has multiple network interfaces available. If the physical device has multiple network interfaces, the listen method in the Swarm module of IPFS listens for connection establishment requests containing the MPQUIC keyword as the node is established.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention detects the physical environment of the IPFS node. If there are multiple network interfaces available, the present invention will fully utilize the network bandwidth resources of the multiple network interfaces to transfer files. On the upload end, files are transmitted through multiple network interfaces, so that the upload time is greatly shortened, and the utilization rate of network resources is also greatly improved. Moreover, for the IPFS network cluster, the increase in bandwidth utilization has a more efficient improvement in accelerating the data transmission rate of IPFS.

[0039] 2. This invention will not conflict with other versions of IPFS. The MPQUIC protocol introduced by the integration is compatible with the QUIC protocol natively supported by IPFS. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0041] Figure 1It is the network topology diagram of multi-port devices;

[0042] Figure 2 This is the difference diagram of IPFS transmission between native IPFS and IPFS integrated with MPQUIC;

[0043] Figure 3 Flowchart of integrating MPQUIC for IPFS. Implementation

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Example

[0045] Please refer to Figure 1 , Figure 1 The physical multi-port device network topology of an IPFS node is shown. Experimental results show that MPQUIC brings nearly 100% performance improvement by aggregating the bandwidth of two identical networks compared to a single path.

[0046] Please refer to Figure 2 , Figure 2 The difference between native IPFS and an IPFS data transmission optimization method based on network multipath provided by the present invention is shown.

[0047] Please refer to Figure 3 , Figure 3 The workflow of data transmission in IPFS and the expansion of the present invention to the merger of IPFS and MPQUIC are shown. The present invention provides an IPFS data transmission optimization method based on network multipath, and the specific steps include:

[0048] In order to incorporate MPQUIC into IPFS, the present invention first defines a new Protocol structure variable for the MPQUIC protocol and registers it with the node component. Therefore, each time an IPFS node starts, the node can configure the MPQUIC protocol for incoming data transmission. In addition, the Multi-Address structure is used to support multiple network protocols in IPFS. The present invention also uses this structure to represent the new network address of the IPFS node that is used for MPQUIC connections by assigning a unique port number and the keyword MPQUIC to the corresponding fields in the structure;

[0049] The present invention extends listen in Bitswap to listen for MPQUIC requests. When the MPQUIC protocol is configured and used to initiate a connection, the header of each data packet contains some necessary information about the transport protocol, such as its name and version. libp2p can analyze and extract this information from the data packet to identify MPQUIC requests. After the incoming requests are identified as MPQUIC requests, it can process these requests according to the specification of the MPQUIC protocol. Dial in Swarm is also extended in a similar way to establish an MPQUIC connection after listen processes its request. Afterwards, all established connections are added to the connection pool in Swarm for incoming data transmission.

[0050] The MPQUIC-based Transport interface implementation is the most critical component of the combination of IPFS and MPQUIC. It is similar to the QUIC-based implementation of Transport. First, define a reuseConn type, which is mainly composed of UDP connections, mutex variables, and reference counts for connection reuse. Since MPQUIC is based on UDP, the UDP connection is the core of the MPQUIC connection, represented by the Session interface type in MPQUIC, and IPFS uses an abstract connection from libp2p, the interface type of CapableConn, which supports various transport protocols. Therefore, we define another type conn and a set of methods of this type to implement data transmission on MPQUIC connections, such as stream multiplexing, encryption, peer authentication, etc. This set has implemented all the methods required by CapableConn. Therefore, the MPQUIC connection is regarded as a CapableConn connection and is correctly used for data transmission in IPFS.

[0051] listen and dial are the most common methods in the Transport interface. The former is used by the target node to listen to and accept incoming requests, and the latter is used by the connection initiator to obtain a connection. To implement dial, we listen for incoming UDP requests and then establish the corresponding connection through net.ListenUDP. After that, the established UDP connection is first encapsulated in the variable reuseConn and used to create an MPQUIC session with multipath configuration, that is, an MPQUIC connection. In order to be compatible with the connections in IPFS, the session is encapsulated in a variable of type conn and returned by dial. To implement listen, we first use net.ListenUDP to listen and accept the UDP connection, which is encapsulated in the reuseConn variable. We activate the listener provided by MPQUIC through the multipath configuration to listen for packets from the UDP connection. Based on the listener, we can also accept incoming requests, resulting in an MPQUIC connection.

[0052] Specifically, to further illustrate the present invention, the following specific embodiments are provided: Example

[0053] When users upload files to be shared or downloaded in the IPFS network, they first need to establish a connection with the owner of the resource. For the sender (owner of the resource), as the IPFS node is started, the node will automatically pull up the monitoring of the supported transmission protocols, such as TCP, QUIC, MPQUIC, etc.; for the receiver (resource acquirer), when initiating the connection, the MPQUIC protocol will be selected first to establish the connection. If the sender supports the MPQUIC protocol, the MPQUIC connection will be successfully established. Figure 2 As shown in (a), if the sender does not support the MPQUIC protocol, a connection with TCP, QUIC or other protocols is established. Figure 2 (b) as shown.

[0054] After successfully establishing an MPQUIC connection, the sender will transfer files through multiple network paths.

[0055] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for optimizing IPFS data transmission based on network multipath, characterized in that: At least the following steps are included: Embed the MPQUIC protocol into IPFS as an extension plug-in; The MPQUIC protocol is embedded in IPFS as an extension plug-in, which includes at least the following steps: Define a new Protocol structure variable for the MPQUIC protocol and register it in the node component of IPFS; Use the MultiAddress structure to represent the new network address of the IPFS node used for the MPQUIC connection; Extend the listen method in the Swarm module in IPFS to listen for MPQUIC requests; The dial method in Swarm is also extended in this way to request a connection and establish an MPQUIC connection after the listen method on the other end processes its request. The IPFS node that integrates the MPQUIC protocol chooses whether to start monitoring the MPQUIC protocol based on the physical environment; IPFS establishes a connection with the IPFS peer node based on the MPQUIC protocol keywords and node identity information; Select the user files and transfer them.

2. According to claim 1, a method for optimizing IPFS data transmission based on network multipath, characterized in that: The method of extending the listen method in the Swarm module in IPFS to listen to MPQUIC requests includes at least the following steps: When the MPQUIC protocol is configured and used to initiate a connection, the header of each data packet contains some necessary information about the transport protocol, including at least the name and version; Use libp2p to analyze and extract this information from the data packet to identify MPQUIC requests; The listen method in the Bitswap component of IPFS is used to call the MPQUIC-based transport implementation method to start the listen daemon process, thereby ensuring correct listening of MPQUIC requests; When incoming requests are identified as MPQUIC requests, they are processed according to the MPQUIC protocol specification.

3. According to claim 1, a method for optimizing IPFS data transmission based on network multipath, characterized in that: The dial method of the Swarm module in the extended IPFS includes at least the following steps: Initiate a connection request based on the MPQUIC protocol to the peer end, and establish an MPQUIC connection after the listen method on the peer end processes its request; All established connections are added to the connection pool in the Swarm for incoming data transfer.

4. According to claim 2, a method for optimizing IPFS data transmission based on network multipath, characterized in that: The MPQUIC-based transport implementation method includes at least the following steps: Define a reuseConn type, which mainly consists of UDP connection, mutex variable and reference count, for connection reuse; Use listen and dial as methods in the Transport interface. The listen method is used by the target node to listen to and accept incoming requests, and the dial method is used by the connection initiator to obtain a connection. In the process of implementing dial, UDP incoming requests are listened first, and then the corresponding connection is established through net.ListenUDP; After that, the established UDP connection is first encapsulated in the variable reuseConn and used to create an MPQUIC session with multipath configuration to achieve MPQUIC connection; To be compatible with connections in IPFS, the session is encapsulated in a variable of type conn and returned by dial; To implement listen, first use net.ListenUDP to listen and accept UDP connections, which are encapsulated in the reuseConn variable; Then activate the listener provided by MPQUIC through multipath configuration to listen for data packets from the UDP connection.

Images