A method for optimizing IPFS data transmission based on data compression

Abstract

The present invention relates to the technical field of IPFS data transmission, and in particular to an IPFS data transmission optimization method based on data compression. The steps include: when a user uploads or downloads a file in an IPFS network, he selects his preference; according to the characteristics of the transmitted file and the user's preference, the optimal compression algorithm is selected from multiple candidate compression algorithms using a file cost minimization strategy; the user file is compressed by the selected optimal compression algorithm, and an output transmission operation is performed. The present invention performs compression processing in advance during data transmission. At the upload end, the upload time is shortened, saving disk space; at the download end, after the file becomes smaller, the download time is also shortened and the network bandwidth is saved; the present invention also supports user compression preference selection, and is more user-friendly when performing compression processing, which can effectively meet the compression needs of users with different focuses.

Description

Technical Field

[0001] The present invention relates to the technical field of IPFS data transmission, and in particular to an IPFS data transmission optimization method based on data compression. Background Art

[0002] IPFS (InterPlanetary File System) is a content-addressable, peer-to-peer, distributed file system. IPFS uses content-address addressing technology, that is, it searches through file content rather than the network address of the file. Multiple nodes running IPFS programs constitute the IPFS storage network. The data stored in the IPFS network is divided into multiple blocks and stored in different nodes. When a node requests data in the network, it caches the file locally on the node. That is, each node saves a cache of the files it has downloaded to ensure that even if a node that stores the resource is removed from the IPFS network, the resource can still be accessed by other nodes. When a user uploads a file to the IPFS node for storage, the node will divide the file into blocks and store it. Each file block is organized in the form of a Merkle directed acyclic graph, and the root hash of the Merkle directed acyclic graph is used to represent the file.

[0003] In the process of file transmission, the prior art does not adopt any data compression processing measures, and cannot meet the customer's requirements for file upload time, CPU utilization, disk size, and memory utilization. To this end, the present invention provides an optimization algorithm for IPFS data transmission based on data compression to solve the additional unnecessary overhead caused by the uncompressed data processing in the prior art. Summary of the invention

[0004] The purpose of the present invention is to provide an IPFS data transmission optimization method based on data compression to solve the problem that the prior art does not adopt any data compression processing measures and cannot meet customers' requirements for file upload time, CPU utilization, disk size, and memory utilization.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides an IPFS data transmission optimization method based on data compression, comprising the following steps:

[0007] Users select their preferences when uploading or downloading files in the IPFS network;

[0008] According to the characteristics of the transmitted files and the user's preferences, the optimal compression algorithm is selected from multiple candidate compression algorithms using the minimum file cost strategy;

[0009] Compress user files using the selected optimal compression algorithm and perform output transmission operations.

[0010] Further preferably, the preferences include:

[0011] Prioritize disk space, upload or download time, CPU utilization, and memory utilization.

[0012] Further preferably, the compression algorithms to be selected include at least two of the following:

[0013] LZ4, LZMA2, LZO, ZLIB, ZIP, ZSTD, SNAPPY.

[0014] Further preferably, according to the characteristics of the transmitted file and the user's preference, the optimal compression algorithm is selected from multiple candidate compression algorithms using a file cost minimization strategy, mainly including:

[0015] Set the cost formula:

[0016] Cost=α*filesize_weight+β*time_weight+γ*cpuRatio_weight+σ*memRatio_weight

[0017] Among them, α, β, γ, σ are parameters that characterize the file size after being processed by various compression algorithms, the time required for uploading or downloading, the CPU utilization rate when the compression algorithm processes the file, and the memory utilization rate when the compression algorithm processes the file, and the value range of these four parameters is between (0,1); fileSize_weight, time_weight, cpuRatio_weight, memRatio_weight are the weights of α, β, γ, σ respectively, and fileSize_weight+time_weight+cpuRatio_weight+memRatio_weight=1,

[0018] Dynamically adjust the values ​​of fileSize_weight, time_weight, cpuRatio_weight, and memRatio_weight according to user needs and substitute them into the cost formula;

[0019] Obtain the cost value of the cost formula of each compression algorithm;

[0020] The compression algorithm with the smallest Cost value is selected as the optimal compression algorithm.

[0021] Further preferably, the method in which α, β, γ, and σ characterize the file size after being processed by various compression algorithms, the time required for uploading or downloading, the CPU utilization rate when the compression algorithm processes the file, and the memory utilization rate when the compression algorithm processes the file is as follows:

[0022] Embed various compression algorithms into IPFS;

[0023] Obtain a horizontal comparison of various indicators after combining various compression algorithms with IPFS;

[0024] Test the appropriate compression data set, divide the test files by size and type, and collect statistics on various indicators: the size of the compressed file by the compression algorithm, the time required for uploading or downloading, the CPU utilization rate when the compression algorithm processes the file, and the memory utilization rate when the compression algorithm processes the file;

[0025] The clustering algorithm is used to classify the measured data, and the category with the smallest data value is the compression algorithm of the score;

[0026] Count the number of files in the smallest class in the dataset;

[0027] The initial values ​​of the same indicator of all compression algorithms are normalized so that the value range is within (0, 1).

[0028] Further preferably, the clustering algorithms include: AP clustering, K-means clustering, and percentage clustering.

[0029] The present invention has at least the following beneficial effects:

[0030] The present invention compresses the files to be uploaded to IPFS in advance according to their file characteristics: at the upload end, the size of the processed file will become smaller, so that the upload time is shortened and the disk space is saved. Moreover, when the unprocessed file data is being uploaded, the CPU is mostly waiting for I or O operations, and the processed file data also shortens the CPU waiting time and thus reduces the CPU utilization; at the download end, after the file becomes smaller, the download time is also shortened and the network bandwidth is saved.

[0031] The present invention also supports users to make preference choices of: upload or download time priority, CPU utilization priority, disk space priority, and memory utilization priority, thereby making the compression process more humane and effectively meeting the compression needs of users with different emphases. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 This is the flow chart of the IPFS built-in compression algorithm;

[0035] Figure 3 A roadmap for selecting characterization parameters. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] Please refer to Figure 1 The present invention provides an IPFS data transmission optimization method based on data compression, and the specific steps include:

[0038] Users upload files to be shared or downloaded in the IPFS (InterPlanetary File System) network and select their preferences for file upload and download processes;

[0039] In one embodiment, the preferences include: upload or download time priority, CPU utilization priority, disk space priority, and memory utilization priority.

[0040] The IPFS data transmission optimization method based on data compression provided by the present invention selects an optimal candidate compression algorithm to compress the file according to the characteristics of the input file and the user's preference, and then uploads the compressed data to the IPFS network.

[0041] In a specific embodiment, the compression algorithms to be selected include at least the following seven compression algorithms: LZ4, LZMA2, LZO, ZLIB, ZIP, ZSTD, and SNAPPY.

[0042] Specifically, when selecting the optimal candidate compression algorithm, the present invention adopts a file cost minimum strategy for selection.

[0043] Specifically, the file cost minimization strategy is to select the most appropriate compression algorithm to process the file according to minimizing the cost formula, and the cost formula is:

[0044] Cost=α*filesize_weight+β*time_weight+γ*cpuRatio_weight+σ*memRatio_weight (1)

[0045] Among them, α, β, γ, σ are parameters that characterize the file size after processing by various compression algorithms, the time required for uploading or downloading, the CPU utilization when the compression algorithm processes the file, and the memory utilization when the compression algorithm processes the file, and the value range of these four parameters is between (0,1) (the source of these parameters is based on the following); fileSize_weight, time_weight, cpuRatio_weight, memRatio_weight are the weights of the corresponding items (α, β, γ, σ), and fileSize_weight+time_weight+cpuRatio_weight+memRatio_weight=1; the values ​​of fileSize_weight, time_weight, cpuRatio_weight, and memRatio_weight are dynamically adjusted according to user needs.

[0046] Further explanation of how α, β, γ, and σ characterize the file size after being processed by various compression algorithms, the time required for uploading or downloading, the CPU utilization rate when the compression algorithm processes the file, and the memory utilization rate when the compression algorithm processes the file:

[0047] First, various compression algorithms need to be embedded in IPFS (the specific steps are as follows Figure 2 As shown in the figure, so that a horizontal comparison of various indicators after combining various compression algorithms with IPFS can be obtained. Then, a suitable compressed data set (the data set is a whole composed of multiple files) is tested (the specific steps are as follows Figure 3As shown), the test files are divided according to size and type, and various indicators are counted, namely, the compressed file size, upload or download time, CPU utilization when the compression algorithm processes the file, and memory utilization when the compression algorithm processes the file corresponding to the LZ4, LZMA2, LZO, ZLIB, ZIP, ZSTD, and SNAPPY compression algorithms of each file in the data set. Since there are errors in the test process and the measured indicators need to be graded, a clustering algorithm is used to classify the measured data. The clustering algorithms used are: AP clustering (Affinity propagation), K-means clustering, percentage clustering (according to 5%, 10% and other minimum values), the class with the smallest data value is the compression algorithm with the score, and finally the number of files in the smallest class of each file in the data set is counted. For example, if the number of compression ratio indicators of compression algorithm LZ4 in the smallest class in the entire data set is m, and the number of files in the entire data set is n, then the initial value of parameter α corresponding to compression algorithm LZ4 is: α = m / n. If the number of compression ratio indicators of LZ4 in the smallest class in the entire data set is 0, then the initial value of parameter α corresponding to compression algorithm LZ4 is α = 2*max(m / n) (m∈all compression algorithms). The initial values ​​of the same indicator of all compression algorithms are normalized so that the value range is within (0, 1).

[0048] Specifically, to further illustrate the present invention, the following specific embodiments are provided:

[0049] Embodiment 1:

[0050] When users upload files to be shared or downloaded in the IPFS network, they first select their preferences as follows: upload or download time priority, CPU utilization priority;

[0051] Then the system sets α=σ=0, and β and γ are obtained from the above-mentioned steps of characterizing parameter values.

[0052] but:

[0053] Cost=β*time_weight+γ*cpuRatio_weight (2)

[0054] At the same time, let time_weight+cpuRatio_weight=1, time_weight=0.5, cpuRatio_weight=0.5.

[0055] The file transfer optimization algorithm of IPFS will use the compression algorithm corresponding to the minimum value of Cost as the compression algorithm for this operation.

[0056] Select the compression algorithm with the smallest cost value to compress the file, and then upload or download it.

[0057] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. An IPFS data transmission optimization method based on data compression, It is characterized in that The following steps are involved: Users select their preferences when uploading or downloading files in the IPFS network; According to the characteristics of the transmitted files and the user's preferences, the optimal compression algorithm is selected from multiple candidate compression algorithms using the minimum file cost strategy; Compress user files using the selected optimal compression algorithm and perform output transmission operations; Among them, according to the characteristics of the transmitted files and the user's preferences, the file cost minimum strategy is adopted to select the optimal compression algorithm from multiple candidate compression algorithms, mainly including: Set the cost formula: ， in, They are parameters that characterize the file size after being processed by various compression algorithms, the time required for uploading or downloading, the CPU utilization rate when the compression algorithm processes the file, and the memory utilization rate when the compression algorithm processes the file. The value range of these four parameters is between (0,1); They are The weight and ， Dynamically adjust according to user needs , and substitute it into the cost formula; Obtain the cost formula of each compression algorithm value; Select The compression algorithm with the smallest value is taken as the optimal compression algorithm; Said The method of characterizing the file size after being processed by various compression algorithms, the time required for uploading or downloading, the CPU utilization rate when the compression algorithm processes the file, and the memory utilization rate when the compression algorithm processes the file is as follows: Embed various compression algorithms into IPFS; Obtain a horizontal comparison of various indicators after combining various compression algorithms with IPFS; Test the appropriate compression data set, divide the test files by size and type, and collect statistics on various indicators: the size of the compressed file by the compression algorithm, the time required for uploading or downloading, the CPU utilization rate when the compression algorithm processes the file, and the memory utilization rate when the compression algorithm processes the file; The clustering algorithm is used to classify the measured data, and the class with the smallest data value is the compression algorithm of the score; Count the number of files in the smallest class in the dataset; The initial values ​​of the same indicator of all compression algorithms are normalized so that the value range is within (0, 1).

2. According to claim 1, a method for optimizing IPFS data transmission based on data compression, It is characterized in that The preferences include: Disk space usage priority, upload or download time priority, CPU utilization priority, memory utilization priority.

3. According to claim 1, a method for optimizing IPFS data transmission based on data compression, It is characterized in that The compression algorithms to be selected include at least two of the following: LZ4, LZMA2, LZO, ZLIB, ZIP, ZSTD, SNAPPY.

4. According to claim 1, a method for optimizing IPFS data transmission based on data compression, It is characterized in that The clustering algorithms include: AP clustering, K-means clustering, and percentage clustering.

Images