A file system adaptive compression method and system

Abstract

The present invention relates to the technical field of file system file compression, specifically disclosing a file system adaptive compression method and system, comprising the steps of: obtaining a group of files to be compressed, and setting a maximum allowable delay T for compressing and writing the group of files to disk; selecting the most appropriate compression algorithm for each file in the group of files to be compressed, using the maximum delay T as a constraint and minimizing storage capacity as an optimization goal; compressing each file in the group of files to be compressed according to the selected compression algorithm and writing it to disk via the file system. The present invention can select the compression algorithm that saves the most storage space based on different delay requirements; and using prediction technology to obtain the compression time and compression ratio of a compression algorithm for a file without actually compressing the file, thereby reducing the time overhead of selecting a traditional compression algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of file system file compression, and in particular to a file system adaptive compression method and compression system. Background Art

[0002] With the continuous advancement of internet technology, the demand for network bandwidth and storage capacity from applications continues to increase, especially for the transmission of large amounts of data. To address this challenge, new technologies are constantly emerging, one of which is data compression. Data compression eliminates redundant information in large data sets, thereby saving storage space and improving transmission efficiency. This technological development enables more data to be transmitted within limited bandwidth, while taking up less storage space.

[0003] Compression algorithms are mainly divided into two types: lossy and lossless. Lossy compression reduces data size by removing information that is imperceptible to humans. It is mainly used for compressing files such as multimedia files where the original data does not need to be completely preserved. In contrast, lossless compression ensures data integrity, but is less efficient and more complex. Lossless compression is often used when minimizing information loss is not an option, such as in text files. The performance of a compression algorithm is typically evaluated by compression speed, decompression speed, and compression ratio. Existing compression algorithms often focus on different performance factors during design. For example, LZO and LZ4 focus on compression and decompression speed, while GZip and Zlib focus more on compression ratio. This results in different advantages and disadvantages for different data types and application scenarios.

[0004] Because different data formats and workloads have different characteristics, a single compression algorithm cannot effectively handle the compression needs of all data and scenarios. Therefore, to maximize compression benefits, it is necessary to select the appropriate compression algorithm for different workloads. However, if the selection of compression algorithms is made manually, manual decisions are often not fast or accurate enough when data changes frequently. To address this problem, adaptive compression systems have emerged. Adaptive compression systems leverage the high-speed analysis and decision-making capabilities of computers to dynamically select the most appropriate compression algorithm based on data characteristics and workload requirements, thereby saving resources such as time, space, and energy. The core of adaptive compression systems lies in developing a mechanism for dynamically selecting compression algorithms to achieve efficient compression and decompression processes. Through adaptive compression systems, the optimal compression algorithm can be flexibly selected in different scenarios, achieving better performance and results in different data scenarios.

[0005] Currently, some file systems, such as Cramfs, EROFS, and ZFS, already support transparent file compression, often with multiple compression algorithms. In these file systems, users must manually configure the compression algorithm to compress files. However, some file systems have introduced the concept of adaptive compression, which automatically selects different compression algorithms to optimize storage and performance. For example, researchers at the University of the Chinese Academy of Sciences proposed the Prediction-Based Adaptive Compression Model (PACM), which uses different compression algorithms based on different data formats and workloads, thereby improving storage system performance. This model has been applied to the distributed file system HDFS to improve data transfer throughput. Furthermore, researchers at the University of Minnesota proposed an adaptive compression (AC) system that selects a compression algorithm based on a list of available compression algorithms, combined with information such as CPU usage, disk bandwidth, and the current write queue length. The system also uses "byte counts" to estimate the compression ratio. Ultimately, this AC system was integrated into the ZFS write pipeline to reduce the file system's overall write time. Furthermore, researchers at the University of Electronic Science and Technology of China have integrated an elastic data compression solution (EDC) into a client-side file system for distributed storage. The system calculates a file's compression metric by examining factors such as file type, compressibility, and client load, and uses this metric to determine whether the file needs to be compressed. If necessary, the system applies high, medium, or low compression algorithms within a specified threshold range, thereby achieving high I / O performance while maintaining high storage efficiency. The introduction of these adaptive compression systems enables the file system to automatically select the most appropriate compression algorithm based on different data characteristics and workload requirements, thereby optimizing performance while improving storage efficiency. This provides the file system with a more flexible and efficient data compression solution for different application scenarios.

[0006] Research has revealed that predicting compression ratios is a key technology in AC system implementation. This is because in order to select the compression algorithm most suitable for the data, it is necessary to understand its performance under different compression algorithms (usually focusing on compression ratios). Directly compressing the data and then comparing performance differences is computationally expensive, so it is necessary to understand the performance of various algorithms on the corresponding files without actually compressing the data. Currently, there is some research on compression ratio prediction. Existing AC systems typically use Shannon entropy, byte counts, or file extensions to predict file compression ratios.

[0007] Existing compression file systems currently have several shortcomings. For one thing, some file systems lack adaptive compression support, requiring users to manually select a compression algorithm, which lacks intelligence. On the other hand, some file systems that do support adaptive compression primarily focus on optimizing system throughput, making them suitable for data transmission and communications. However, they are not suitable for specific application scenarios with limited computing and storage resources (such as edge computing, the Internet of Things, and embedded systems), necessitating new solutions. Summary of the Invention

[0008] The present invention provides a file system adaptive compression method and system, which solves the technical problem of how to select the most appropriate compression algorithm for each file and perform compression with the optimization goal of maximizing storage capacity reduction under certain time constraints.

[0009] To solve the above technical problems, the present invention provides a file system adaptive compression method, comprising the steps of:

[0010] S1. Obtain a file group to be compressed, and set a maximum delay T allowed for compressing the file group to be compressed and writing it to a disk;

[0011] S2. Using the maximum delay T as a constraint and minimizing storage capacity as an optimization goal, select the most appropriate compression algorithm for each file in the group of files to be compressed;

[0012] S3. Compress each file in the to-be-compressed file group according to the compression algorithm selected in step S2 and write the file to a disk through the file system.

[0013] Furthermore, in step S2, an optimization problem is constructed and solved to select the most appropriate compression algorithm for each file in the group of files to be compressed;

[0014] The optimization problem is formulated as:

[0015]

[0016]

[0017]

[0018] Among them, a ij Represents the state function. If file i is compressed using the jth compression algorithm, then a ij =1, otherwise a ij =0; represents the estimated delay of file i using the jth compression algorithm, Predicted by a compression time prediction model based on machine learning; represents the estimated delay of writing file i to the storage medium after using the jth compression algorithm, Calculated based on disk writing speed and compressed file size; S i,j Indicates the size of file i after using the jth compression algorithm, S i,j It is calculated based on the predicted compression rate and the original size of the file, where the predicted compression rate is predicted by a compression rate prediction model; i∈F, F represents a set of n files in the file group to be compressed; j∈A, A represents a set of m compression algorithms.

[0019] Furthermore, the step S2 specifically includes the steps of:

[0020] S21, selecting a file in the group of files to be compressed;

[0021] S22, using a compression ratio prediction model, predicting the compression ratio of each compression algorithm in the compression library when compressing the file;

[0022] S23, using the compression time prediction model, the time required for each compression algorithm in the compression library to compress the file, that is, Make predictions;

[0023] S24, calculate the compressed file size S according to the compression ratio predicted in step S22 and the original file size i ,j;

[0024] S25. Use the iostat system call to obtain the current disk write speed;

[0025] S26, using disk write speed and compressed file size S i,j Calculate disk write time

[0026] S27, repeating steps S21 to S26 for all files in the compressed file group to obtain the compression time, disk writing time, and compressed file size of each file under different compression algorithms;

[0027] S28. Calculate the delay of all files under all compression algorithm selection combinations And the corresponding compressed total file size

[0028] S29. Solve the optimization problem to obtain a compression algorithm combination that satisfies the requirement that the total delay is less than or equal to T and the total file size after compression is the smallest.

[0029] Furthermore, in step S22, the compression rate prediction model predicts the compression rate of the file based on the file type, file size, Shannon entropy, data pair random distribution distance, and compression algorithm of the input file. The data pair random distribution distance is calculated by the following formula:

[0030]

[0031] Where coreset is the unique symbol set that makes up most of the data, a and b are two different symbols in coreset, size of sample represents the size of coreset, freq(a) represents the frequency of symbol a in the data, freq(b) represents the frequency of symbol b in the data, and freq(a,b) represents the frequency of symbol a,b.

[0032] The number of data pairs that appear at the same time. Number of pairs indicates the number of data pairs into which the data is divided.

[0033] Furthermore, training the compression rate prediction model in step S2 includes the following steps:

[0034] S221. Select a representative data set as a training set for the compression rate prediction model;

[0035] S222, select a file from the training set;

[0036] S223, obtaining the file type and file size according to the file attributes;

[0037] S224. Calculate the Shannon entropy and the data-to-random distribution distance of the file according to the file content and the calculation formula of the Shannon entropy and the data-to-random distribution distance;

[0038] S225, sequentially selecting each compression algorithm in the compression algorithm library to compress the file, and obtaining an actual compression ratio of the file under each compression algorithm;

[0039] S226. Combining the file type, file size, Shannon entropy, data pair random distribution distance, compression algorithm, and actual compression rate into a piece of training data;

[0040] S227. Repeat steps S222 to S226 for other files in the training set to obtain all data required for training.

[0041] S228. Build a regression neural network model;

[0042] S229: Use the data obtained in step S227 as a training data set for the neural network to train the constructed regression neural network model.

[0043] Furthermore, training the compression time prediction model in step S3 includes the following steps:

[0044] S231. Select a representative data set as a training set for the compression time prediction model;

[0045] S232: Select a compression algorithm from the compression algorithm library, compress each file in the training set, and obtain the compression ratio and compression time of all files in the training set under the compression algorithm;

[0046] S233. Perform function fitting based on the obtained relationship between the compression rate and the compression time, thereby obtaining a fitting function of the compression rate and the compression time of the algorithm when compressing the file on the machine;

[0047] S234. Repeat steps S232 to S233 for other compression algorithms in the compression library to obtain fitting functions of all compression algorithms, and use them as compression time prediction models.

[0048] Furthermore, the step S1 specifically includes the steps of:

[0049] S11, obtaining a group of files to be compressed from a user as a group of files to be compressed by the adaptive compression system;

[0050] S12. The user sets a maximum delay T allowed for compressing the to-be-compressed file group and writing it to disk.

[0051] Furthermore, the step S3 specifically includes the steps of:

[0052] S31, compressing each file in the to-be-compressed file group according to the compression algorithm selected in step S2 to obtain an actual compressed file;

[0053] S32, combining the actual compressed file and the type code of the adopted compression algorithm to obtain a final compressed file;

[0054] S33. Write all final compressed files to disk through the file system.

[0055] Furthermore, in step S32, a multi-bit binary number is used for encoding according to the type of compression algorithm to obtain a type code of each compression algorithm.

[0056] The present invention also provides a file system adaptive compression system, the key of which is that it includes a user layer and an adaptive compression layer, the user layer is used to execute step S1 in the above method, and the adaptive compression layer is used to execute steps S2 and S3 in the above method.

[0057] The present invention provides a file system adaptive compression method and system, which can select the compression algorithm that saves the most storage space according to different delay requirements; through prediction technology, the compression time and compression rate of the compression algorithm on the file are obtained without actually compressing the file, thereby reducing the time overhead of traditional compression algorithm selection; at the same time, the present invention also introduces a machine learning method to predict the file compression rate, thereby improving the accuracy of compression algorithm selection during adaptive compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a file system adaptive compression method provided by an embodiment of the present invention;

[0059] Figure 2 The embodiment of the present invention provides Figure 2 is the relationship between the entropy and compression rate of three types of files, where Figure 2 (a) corresponds to the binary file, Figure 2 (b) corresponding executable file, Figure 2 (c) corresponding text files;

[0060] Figure 3 Schematic diagram of a compression rate prediction model provided by an embodiment of the present invention;

[0061] Figure 4 Schematic diagram of a compression time prediction model provided by an embodiment of the present invention;

[0062] Figure 5 It is a structural diagram of a file system adaptive compression system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings. The embodiments are provided for illustrative purposes only and are not to be construed as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of protection of the present invention. Many changes may be made to the present invention without departing from the spirit and scope of the present invention.

[0064] The embodiment of the present invention provides a file system adaptive compression method, which selects different compression algorithms to process data according to the change of delay limit, thereby saving storage space to the maximum extent while meeting the delay requirement. Figure 1 As shown, the steps include:

[0065] S1. Obtain a file group to be compressed and set a maximum delay T allowed for compressing the file group and writing it to disk.

[0066] S2. Using the maximum delay T as a constraint and minimizing storage capacity as an optimization goal, select the most appropriate compression algorithm for each file in the group of files to be compressed.

[0067] S3. Compress each file in the to-be-compressed file group according to the compression algorithm selected in step S2 and write the file to the disk through the file system.

[0068] Specifically, step S1 includes the following steps:

[0069] S11, obtaining a group of files to be compressed from a user as a group of files to be compressed by the adaptive compression system;

[0070] S12. The user sets the maximum delay T allowed for compressing the file group to be compressed and writing it to disk.

[0071] In step S2, an optimization problem is constructed and solved to select the most appropriate compression algorithm for each file in the group of files to be compressed. The optimization problem is constructed as follows:

[0072]

[0073]

[0074]

[0075] Among them, a ij Represents the state function. If file i is compressed using the jth compression algorithm, then a ij =1, otherwise a ij =0; represents the estimated delay of file i using the jth compression algorithm, It is related to the file itself, compression algorithm, and CPU performance, and is predicted by a compression time prediction model based on machine learning. represents the estimated delay of writing file i to the storage medium after using the jth compression algorithm, Calculated based on disk write speed (machine I / O throughput) and compressed file size; S i,j Indicates the size of file i after using the jth compression algorithm, S i,j It is calculated based on the predicted compression rate and the original size of the file, where the predicted compression rate is predicted by the compression rate prediction model; i∈F, F represents the set of n files in the file group to be compressed; j∈A, A represents the set of m compression algorithms.

[0076] By obtaining compression time and compression ratio data for each file using various algorithms in the compression algorithm library, the system can use this information, combined with some basic information, to select the appropriate compression algorithm for each file. By comparing the compression times and compression ratios of different algorithms, the system can select the compression algorithm that maximizes storage space savings while meeting latency constraints. This effectively optimizes the file system's storage space utilization and improves system performance and efficiency.

[0077] In general, step S2 specifically includes the steps of:

[0078] S21, selecting a file in the group of files to be compressed;

[0079] S22, using a compression ratio prediction model, predicting the compression ratio of each compression algorithm in the compression library when compressing the file;

[0080] S23, using the compression time prediction model, the time required for each compression algorithm in the compression library to compress the file, that is, Make predictions;

[0081] S24, calculate the compressed file size S according to the compression ratio predicted in step S22 and the original file size i ,j;

[0082] S25. Use the iostat system call to obtain the current disk write speed;

[0083] S26, using disk write speed and compressed file size S i,j Calculate disk write time

[0084] S27, repeating steps S21 to S26 for all files in the compressed file group to obtain the compression time, disk writing time, and compressed file size of each file under different compression algorithms;

[0085] S28. Calculate the delay of all files under all compression algorithm selection combinations And the corresponding compressed total file size

[0086] S29. Solve the optimization problem to obtain a compression algorithm combination that satisfies the requirement that the total delay is less than or equal to T and the total file size after compression is the smallest.

[0087] The implementation of the compression rate prediction model in step S22 is based on the following analysis: Some adaptive compression systems use a single mathematical characteristic that describes data redundancy, such as Shannon entropy, when predicting data compression rate. Shannon entropy is used to measure the number of bits required to encode (represent) a symbol without losing information. Generally speaking, the greater the entropy, the lower the compression potential of the data. Entropy is calculated by solving the weighted average of the bits required to encode a symbol to obtain the entropy of a group of symbols. Its calculation formula is: Among them, P i is the frequency of the i-th symbol in the data. However, experiments have shown that for various types of files, a single feature often cannot accurately predict the data compression rate. Figure 2 The relationship between the entropy (Entorpy) and compression ratio (Compress Ratio) of three types of files is shown in the figure. Figure 2 (a) corresponds to the binary file, Figure 2 (b) corresponding executable file, Figure 2 (c) Corresponding text file. Figure 2 For binary and executable files, since there is an obvious correlation between entropy and the compression rate of files, the compression rate of data can be predicted well. However, for text files, the predictive ability of entropy will obviously fail. Therefore, in order to still be able to predict the compression rate of files when entropy fails, more features are needed. Through the analysis of the principles of compression algorithms and different data characteristics, this example selects the following features and combines them with the regression neural network in machine learning to predict the compression rate of data: file type (indicated by file extension), file size, Shannon entropy, and data pair random distribution distance. The data pair random distribution distance is designed to distinguish randomly sorted data from data containing duplicates. It allows the system to distinguish compressible data from incompressible data when entropy fails. Its calculation formula is:

[0088]

[0089] Where coreset is the unique set of symbols that make up most of the data, a and b are two different symbols in coreset, sizeofsample represents the size of coreset, freq(a) represents the frequency of symbol a in the data, freq(b) represents the frequency of symbol b in the data, freq(a,b) represents the number of data pairs where symbols a and b appear simultaneously, and numberofpairs represents the number of data pairs into which the data is divided.

[0090] Based on the above analysis, step S22 specifically includes the following steps:

[0091] S221. Select a representative data set as a training set for the compression rate prediction model;

[0092] S222, select a file from the training set;

[0093] S223, obtaining the file type and file size according to the file attributes;

[0094] S224. Calculate the Shannon entropy and the data-to-random distribution distance of the file according to the file content and the calculation formula of the Shannon entropy and the data-to-random distribution distance;

[0095] S225, sequentially selecting each compression algorithm in the compression algorithm library to compress the file, and obtaining an actual compression ratio of the file under each compression algorithm;

[0096] S226. Combining the file type, file size, Shannon entropy, data pair random distribution distance, compression algorithm, and actual compression rate into a piece of training data;

[0097] S227. Repeat steps S222 to S226 for other files in the training set to obtain all data required for training.

[0098] S228. Build a regression neural network model;

[0099] S229: Using the data obtained in step S227 as a training data set for the neural network, the constructed regression neural network model is trained;

[0100] S2210, obtaining the file type (indicated by the extension), file size, Shannon entropy, random distribution distance of data pairs, and compression algorithm characteristics of the file to be compressed;

[0101] S2211. Input the features into the regression neural network model to obtain the output prediction result (compression rate).

[0102] Among them, steps S221 to S229 are called the training phase of the compression rate prediction model. Figure 3 In the upper part, steps S2210 to S2211 are the prediction stage of the model, such as Figure 3 Lower part.

[0103] Specifically, step S23 specifically includes the following steps:

[0104] S231. Select a representative data set as a training set for the compression time prediction model;

[0105] S232: Select a compression algorithm from the compression algorithm library, compress each file in the training set, and obtain the compression ratio and compression time of all files in the training set under the compression algorithm;

[0106] S233. Perform function fitting based on the obtained relationship between the compression rate and the compression time, thereby obtaining a fitting function of the compression rate and the compression time of the algorithm when compressing the file on the machine;

[0107] S234: Repeat steps S232 to S233 for other compression algorithms in the compression library to obtain fitting functions of all compression algorithms, and use them as compression time prediction models;

[0108] S235 , by taking the compression rate predicted in step S22 and the compression algorithm currently to be predicted as input, performing calculations through a compression time prediction model, thereby obtaining the compression time of the file under a certain compression algorithm.

[0109] Among them, steps S231 to S234 are called the training phase of the model. Figure 4 In the upper part, step S235 is the prediction stage of the model, such as Figure 4 Lower part. It is worth noting that since the execution time of the compression algorithm is closely related to the performance of the machine, this example uses the model training phase of steps S231 to S234 as the installation process of the adaptive compression system. When running the adaptive system on a new machine, the above steps need to be repeated. In addition, when a new compression algorithm is added to the compression algorithm library, it is also necessary to train the data set for the new compression algorithm to obtain its fitting function.

[0110] Step S2 uses a regression neural network approach to predict the file compression ratio, leveraging file attributes and file content features related to file redundancy. Furthermore, the file compression time is predicted based on the relationship between the file compression ratio and the file compression time. Using these two predicted values, combined with the optimization objectives of the system model, the most appropriate compression algorithm is selected for all files in the group to be compressed.

[0111] Next, step S3 is used to actually compress the file and write it to disk. Step S3 specifically includes the following steps:

[0112] S31, compressing each file in the to-be-compressed file group according to the compression algorithm selected in step S2 to obtain an actual compressed file;

[0113] S32, combining the actual compressed file and the type code of the compression algorithm used (so as to know which compression algorithm to use for decompression during decompression) to obtain a final compressed file;

[0114] S33. Write all final compressed files to disk through the file system.

[0115] The compression library provides a flexible selection of compression algorithms based on your needs. In this example, the library provides three compression algorithms: lZ4, Zlib, and LZO. The compression speed is: lZ4 > LZO > Zlib, and the compression ratio (the higher the compression ratio, the smaller the compressed file size) is: Zlib > LZO > LZ4. In this example, since there are three compression algorithms, a two-bit binary number is used for encoding. Specifically, the LZ4 compression algorithm is encoded as 00, the LZO compression algorithm is encoded as 01, and the Zlib compression algorithm is encoded as 11.

[0116] This embodiment also provides a file system adaptive compression system, such as Figure 5 As shown, it includes a user layer and an adaptive compression layer. The user layer is used to execute step S1 in the above method, and the adaptive compression layer is used to execute steps S2 and S3 in the above method.

[0117] In summary, the embodiments of the present invention provide a file system adaptive compression method and system that maximizes storage space savings by selecting an appropriate compression algorithm under latency constraints. Furthermore, this system utilizes a regression network model combined with file data characteristics to improve the accuracy of file compression rate prediction within the adaptive system, thereby improving the accuracy of file compression algorithm selection.

[0118] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A file system adaptive compression method, characterized in that: Including steps: S1. Obtain a file group to be compressed, and set a maximum delay T allowed for compressing the file group to be compressed and writing it to a disk; S2. Taking the maximum delay T as a constraint and minimizing storage capacity as an optimization goal, select the most appropriate compression algorithm for each file in the group of files to be compressed. Step S2 specifically includes the following steps: S21, selecting a file in the group of files to be compressed; S22. Use a compression ratio prediction model to predict the compression ratio of each compression algorithm in the compression library when compressing the file; the compression ratio prediction model predicts the compression ratio of the file based on the file type, file size, Shannon entropy, random distribution distance of the data pair, and the compression algorithm of the input file, and the random distribution distance of the data pair is calculated by the following formula: Where coreset is the unique symbol set that makes up most of the data, a and b are two different symbols in coreset, sizeofsample represents the size of coreset, freq(a) represents the frequency of symbol a in the data, freq(b) represents the frequency of symbol b in the data, freq(a,b) represents the number of data pairs where symbols a and b appear simultaneously, and numberofpairs represents the number of data pairs into which the data is divided. S3. Compress each file in the to-be-compressed file group according to the compression algorithm selected in step S2 and write the file to a disk through the file system.

2. A file system adaptive compression method according to claim 1, characterized in that: In step S2, an optimization problem is constructed and solved to select the most appropriate compression algorithm for each file in the group of files to be compressed; The optimization problem is formulated as: Among them, a ij Represents the state function. If file i is compressed using the jth compression algorithm, then a ij =1, otherwise a ij =0; represents the estimated delay of file i using the jth compression algorithm, Predicted by a compression time prediction model based on machine learning; represents the estimated delay of writing file i to the storage medium after using the jth compression algorithm, Calculated based on disk writing speed and compressed file size; S i,j Indicates the size of file i after using the jth compression algorithm, S i,j It is calculated based on the predicted compression rate and the original size of the file, where the predicted compression rate is predicted by a compression rate prediction model; i∈F, F represents a set of n files in the file group to be compressed; j∈A, A represents a set of m compression algorithms.

3. A file system adaptive compression method according to claim 2, characterized in that: The step S2 specifically further includes the following steps: S23, using the compression time prediction model, the time required for each compression algorithm in the compression library to compress the file, that is, Make predictions; S24, calculate the compressed file size S according to the compression ratio predicted in step S22 and the original file size i,j ; S25. Use the iostat system call to obtain the current disk write speed; S26, using disk write speed and compressed file size S i,j Calculate disk write time S27, repeating steps S21 to S26 for all files in the compressed file group to obtain the compression time, disk writing time, and compressed file size of each file under different compression algorithms; S28. Calculate the delay of all files under all compression algorithm selection combinations And the corresponding compressed total file size S29. Solve the optimization problem to obtain a compression algorithm combination that satisfies the requirement that the total delay is less than or equal to T and the total file size after compression is the smallest.

4. A file system adaptive compression method according to claim 3, characterized in that: The compression rate prediction model in training step S22 includes the following steps: S221. Select a representative data set as a training set for the compression rate prediction model; S222, select a file from the training set; S223, obtaining the file type and file size according to the file attributes; S224. Calculate the Shannon entropy and the data-to-random distribution distance of the file according to the file content and the calculation formula of the Shannon entropy and the data-to-random distribution distance; S225, sequentially selecting each compression algorithm in the compression algorithm library to compress the file, and obtaining an actual compression ratio of the file under each compression algorithm; S226. Combining the file type, file size, Shannon entropy, data pair random distribution distance, compression algorithm, and actual compression rate into a piece of training data; S227. Repeat steps S222 to S226 for other files in the training set to obtain all data required for training. S228. Build a regression neural network model; S229: Use the data obtained in step S227 as a training data set for the neural network to train the constructed regression neural network model.

5. The file system adaptive compression method according to claim 3, characterized in that: The compression time prediction model in step S23 is trained, comprising the following steps: S231. Select a representative data set as a training set for the compression time prediction model; S232: Select a compression algorithm from the compression algorithm library, compress each file in the training set, and obtain the compression ratio and compression time of all files in the training set under the compression algorithm; S233. Perform function fitting based on the obtained relationship between the compression rate and the compression time, thereby obtaining a fitting function of the compression rate and the compression time of the algorithm when compressing the file on the machine; S234. Repeat steps S232 to S233 for other compression algorithms in the compression library to obtain fitting functions of all compression algorithms, and use them as compression time prediction models.

6. The file system adaptive compression method according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11, obtaining a group of files to be compressed from a user as a group of files to be compressed by the adaptive compression system; S12. The user sets a maximum delay T allowed for compressing the to-be-compressed file group and writing it to disk.

7. The file system adaptive compression method according to claim 1, characterized in that: The step S3 specifically includes the following steps: S31, compressing each file in the to-be-compressed file group according to the compression algorithm selected in step S2 to obtain an actual compressed file; S32, combining the actual compressed file and the type code of the adopted compression algorithm to obtain a final compressed file; S33. Write all final compressed files to disk through the file system.

8. The file system adaptive compression method according to claim 7, characterized in that: In step S32, a multi-bit binary number is used for encoding according to the type of compression algorithm to obtain a type code of each compression algorithm.

9. A file system adaptive compression system, characterized by: It includes a user layer and an adaptive compression layer, wherein the user layer is used to execute step S1 described in any one of claims 1 to 8, and the adaptive compression layer is used to execute steps S2 and S3 described in any one of claims 1 to 8.

Images