Distributed adaptive storage method and system based on file access characteristics

Abstract

The present invention proposes a distributed adaptive storage method and system based on file access characteristics, the purpose of which is to implement an adaptive hybrid storage strategy of replicas and erasure codes for hot and cold files of different sizes in the existing distributed storage system, so as to improve the storage efficiency and performance of the cluster. The advantage of the present invention is that according to the access characteristics of different types of files in the distributed system, the file heat is calculated by the size of the data block occupied by the file and the user's access frequency to the file, so that the file has a more suitable storage method. And we will make adaptive storage strategy changes to the file in each cycle according to the change in the heat of the file and the overall storage efficiency of the system. Ultimately, the performance of the system is improved and the overall storage efficiency of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing and the technical field of data storage strategies in distributed storage, and in particular to a distributed adaptive storage method and system based on file access characteristics. Background Art

[0002] With the widespread use and increasing popularity of big data distributed file systems, the storage overhead of distributed storage systems, storage efficiency, data availability, and persistence have become issues of increasing concern. The traditional three-copy storage strategy causes excessive storage overhead, thus wasting storage resources. Erasure coding is considered to be the best alternative to the copy strategy.

[0003] However, using erasure codes alone for data storage will cause a decrease in the access performance of distributed systems. This is because compared to the replica replication strategy, erasure codes consume more resources than replica storage, both in terms of bandwidth consumption across racks and CPU / IO resource utilization. Therefore, hybrid storage of replicas and erasure codes currently seems to be the best solution for distributed systems in terms of storage and computing.

[0004] If you simply use a simple mixture of replicas and erasure codes without considering the access characteristics of files, it will cause a waste of resources. Distributed systems store many files, both large and small. Most of the files are large files, of which large files may be divided into multiple data blocks, while small files may only be divided into one or two data blocks. In addition, the "hotness" of files in distributed systems is not the same. The access frequency of different files can vary greatly. After the access frequency of files is counted, hot files and cold files appear.

[0005] For example, the hybrid storage of replicas and RS, which mainly considers file size, represented by DECPA, will have the following technical problems: 1) The RS erasure code storage strategy is used for files with high access popularity, resulting in users frequently accessing files, but the erasure code storage is not sufficient to support such high-frequency access. 2) The replica replication storage strategy is used for files with low access popularity, resulting in a serious waste of storage space, making the overall storage overhead of the system too high. Summary of the invention

[0006] Specifically, the present application provides a distributed adaptive storage method based on file access characteristics, which includes:

[0007] Step 1: Obtain the average access frequency of each file in the distributed storage system within a preset period, and calculate the heat of each file based on the average access frequency;

[0008] Step 2: According to the popularity of each file in the current cycle, all files are divided into hot data and cold data. According to the number of data blocks occupied by each hot data, all the hot data are further divided into small hot data and large hot data. According to the number of data blocks occupied by each cold data, all the cold data are divided into small cold data and large cold data.

[0009] Step 3: Use a replica replication strategy to store the small hot data, use LRC (6, 2, 2) encoding to store the large hot data, use RS (3, 2) encoding to store the small cold data, and use RS (6, 3) to store the large cold data.

[0010] The distributed adaptive storage method based on file access characteristics includes:

[0011] Step 4: Execute step 1 multiple times to periodically count the heat of each file, use the hot data in the latest cycle to compare the files with the hot data in the previous cycle, and obtain the new hot files newly generated in the latest cycle and the new cold files that have turned from hot to cold in the previous cycle;

[0012] Step 5: Change the storage strategy of the new hot file to LRC (6, 2, 2) encoding or copy replication storage, and change the storage strategy of the new cold file to RS (6, 3) encoding or RS (3, 2) encoding storage.

[0013] The distributed adaptive storage method based on file access characteristics, wherein step 4 includes: determining whether the current storage overhead of the distributed storage system is lower than a threshold, if so, executing step 5; otherwise, converting the file at the bottom of the hot data heat ranking in the latest cycle into cold data, and then executing step 5.

[0014] The distributed adaptive storage method based on file access characteristics, wherein step 1 comprises:

[0015] The preset period is divided into multiple time periods, and one of the time periods is assumed to be the current heat statistics moment, and the other time period before the current heat statistics moment is assumed to be the previous heat statistics moment. In the preset period, the file access weight value closer to the current heat statistics moment is higher, and the file access weight value closer to the previous heat statistics moment is lower; the heat H of each file is calculated by the following formula:

[0016] H=(1-β)I+βf avg (0<β<1)

[0017] f avg is the average access frequency, I is the weight value, and β is the balance factor.

[0018] The present invention also proposes a distributed adaptive storage system based on file access characteristics, which includes:

[0019] The heat calculation module is used to obtain the average access frequency of each file in the distributed storage system within a preset period, and calculate the heat of each file based on the average access frequency;

[0020] The file classification module is used to classify all files into hot data and cold data according to the popularity of each file in the current cycle, and further classify all the hot data into small hot data and large hot data according to the number of data blocks occupied by each hot data, and classify all the cold data into small cold data and large cold data according to the number of data blocks occupied by each cold data;

[0021] The file storage module is used to store the small hot data using a copy replication strategy, store the large hot data using LRC (6, 2, 2) encoding, store the small cold data using RS (3, 2) encoding, and store the large cold data using RS (6, 3).

[0022] The distributed adaptive storage system based on file access characteristics includes:

[0023] A comparison module is used to call the module 1 multiple times to periodically count the heat of each file, use the hot data in the latest cycle to compare the files with the hot data in the previous cycle, and obtain the new hot files newly generated in the latest cycle and the new cold files that have turned from hot to cold in the previous cycle;

[0024] The storage strategy change module is used to change the storage strategy of new hot files to LRC (6, 2, 2) encoding or copy replication storage, and change the storage strategy of new cold files to RS (6, 3) encoding or RS (3, 2) encoding storage.

[0025] The distributed adaptive storage method based on file access characteristics, wherein the comparison module includes: determining whether the current storage overhead of the distributed storage system is lower than a threshold, if so, executing the storage policy change module; otherwise, converting the file at the bottom of the hot data heat ranking in the current latest cycle into cold data, and then executing the storage policy change module.

[0026] The distributed adaptive storage system based on file access characteristics, wherein the heat calculation module includes:

[0027] The preset period is divided into multiple time periods, and one of the time periods is assumed to be the current heat statistics moment, and the other time period before the current heat statistics moment is assumed to be the previous heat statistics moment. In the preset period, the file access weight value closer to the current heat statistics moment is higher, and the file access weight value closer to the previous heat statistics moment is lower; the heat H of each file is calculated by the following formula:

[0028] H=(1-β)I+βf avg (0<β<1)

[0029] f avg is the average access frequency, I is the weight value, and β is the balance factor.

[0030] The present invention also proposes a storage medium for storing a program for executing any one of the distributed adaptive storage methods based on file access characteristics.

[0031] The present invention also proposes a client, which is used for any of the distributed adaptive storage systems based on file access characteristics.

[0032] It can be seen from the above scheme that the advantages of the present invention are:

[0033] The purpose of the present invention is to implement an adaptive hybrid storage strategy of replicas and erasure codes for hot and cold files of different sizes in the existing distributed storage system, so as to improve the storage efficiency and performance of the cluster. The advantage of the present invention is that according to the access characteristics of different types of files in the distributed system, the file heat is calculated by the size of the data block occupied by the file and the user's access frequency to the file, so that the file has a more suitable storage method. And we will make adaptive storage strategy changes to the file in each cycle according to the change of the file's heat and the overall storage efficiency of the system. Ultimately, the performance of the system is improved and the overall storage efficiency of the system is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of the adaptive storage strategy of replication and erasure coding based on file popularity and size;

[0035] Figure 2 This is a storage strategy diagram for different similar data according to the present invention. DETAILED DESCRIPTION

[0036] Specifically, the present invention comprises the following steps:

[0037] A. Calculate the heat of the file:

[0038] A1. Calculate the average access frequency f of each file avg :

[0039] Set the time interval between two heat statistics as period T, the number of visits within the period as N, and the average visit frequency within the period as f avg ,

[0040] A2. Calculate the time access inclination I of each file:

[0041] Then each cycle T is divided into {t 1 ,t 2 ,...t n}n time periods, assuming that t 0 is the heat statistics time, t i is the last heat statistics time, access weight λ=1 / i,t i The access frequency of the time period is f ti That is, the closer the access weight is to the current heat statistics time, the higher the access weight is, and the closer the access weight is to the last heat statistics time, the lower the access weight is. Set the access inclination rate to 1, and the formula is as follows:

[0042]

[0043] A3. Calculate the heat H of each file:

[0044] For the average access frequency of a file, f avg Assign a balance factor β (0<β<1), then the balance factor of the file's time access inclination rate I is (1-β), and the file's heat is set to H. The formula is as follows:

[0045] H=(1-β)I+βf avg (0<β<1)

[0046] B. Use different storage methods based on the popularity and size of the file:

[0047] B1. Four categories of documents:

[0048] The classification criteria for adopting different types of storage strategies for files are heat and size. We classify files into hot files and cold files based on their heat. We classify files into large files and small files based on their size.

[0049] B1-1. File popularity classification method

[0050] In daily life, there are many phenomena that conform to the Zipf distribution law, including the frequency of word occurrence, the frequency of web page access, etc. In large-scale distributed storage systems, this law is also followed. Often, 80% of the data access to the cluster by users is concentrated on 20% of the data. Therefore, in our design, we use the heat calculation formula mentioned above to calculate the heat H of the file within a period T, and then define the top 20% of the heat ranking data as hot data, and the remaining 80% of the data as cold data.

[0051] B1-2. File size classification

[0052] For file size classification, we base the classification on the popularity of the files.

[0053] For hot files, files with less than 2 data blocks (the default size of a data block in Hadoop 3.0 is 128MB), we use a relatively simple copy replication strategy, because when the number of data blocks is small, it will not generate too much storage space, but it can greatly improve the parallel reading and writing capabilities of the file and reduce the reading delay of the file, which is in line with the characteristics of hot files that need to be read and written frequently; for files with more than 2 data blocks, we use LRC (6, 2, 2) encoding to store them. Because in erasure codes, LRC erasure codes have excellent ability to handle frequent data access. And through our research on distributed storage systems, we found that more than 95% of data loss occurs on a single data block. When dealing with data loss of less than 2 data blocks, LRC erasure codes can directly call local check blocks for fast data recovery without calling global check blocks, which greatly improves the data recovery speed and reduces the data reconstruction time. Hot files are often frequently accessed. If the data reconstruction efficiency is fast enough when data is lost, it can also improve the performance of the system.

[0054] For cold files, we use RS (3, 2) to store files smaller than 3 data blocks. If the file size is smaller than 3 data blocks, only one stripe is needed for encoding during RS (3, 2) encoding. If it is larger than 3 data blocks, multiple stripes are needed for storage, which will cause waste of storage of blank blocks. For small files, the wasted storage space is relatively high. If the file size is larger than 3 data blocks, we use RS (6, 3) to store it. In this way, even if the storage overhead of blank blocks occurs for larger files, it is within an acceptable range.

[0055] B1-3. Four types of combinations of heat and size

[0056] As mentioned in B1-1 and B1-2, after we classify the files by popularity and size, we have the following four combinations:

[0057] HOT--BIG

[0058] HOT--SMALL

[0059] COLD--BIG

[0060] COLD--SMALL

[0061] B2. Four storage strategies for files:

[0062] like Figure 2 As shown in the figure, after having the above four types of file classification combinations, we adopt different storage strategies according to each type of file.

[0063] C. Periodically count the popularity of files and estimate the overall storage overhead of the system:

[0064] C1. Statistics on file popularity changes:

[0065] After a cycle of user access, the access distribution of C1-1 roughly conforms to Zipf's law, but the user's access frequency and time period for each file will be significantly different. Some files may have more access at the beginning of this cycle, and the access frequency and number of accesses will gradually decrease in the middle and late stages, while the access frequency and number of accesses of some files may remain relatively "hot". We will first calculate the heat of the latest cycle according to the heat H calculation formula mentioned in A3.

[0066] C1-2 then counts the top 20% of the most popular files in the latest cycle i and puts them into the set G(i)={F 1 ,F 2 ,...F n}, and the set G(i-1) of the previous cycle = {F 1 ,F 2 ,...F m} to perform file matching. The files that appear in set G(i) but not in set G(i-1) are put into set H = {F 1 ,F 2 ,...F i}, these files are the new hot files counted in the latest cycle. Files that appear in set G(i-1) but not in G(i) are put into set C = {F 1 ,F 2 ,...F j}, these files are the new cold files that have been counted as being changed from hot to cold in the latest cycle.

[0067] C1-3 Since new files are often accessed more frequently, we store them as hot files first. In this way, when we count the heat, we only need to record whether the new file is in the set G(i) in this period. If not, add it to the set C.

[0068] C2. Estimate the overall storage cost of the system

[0069] C2-1 After investigation, it is found that when different types of erasure codes are mixed for storage, the optimal storage overhead of the system is between 1.5x-1.7x. Since our system uses replicas to store a small amount of hot data, we have chosen to improve the efficiency of parallel access and reduce read latency to improve the performance of the system. In theory, the storage overhead will be higher than that of a system with mixed storage of erasure codes. However, since we have more suitable processing for files with different characteristics, the estimated storage overhead of the system is not very high. (Because most of the distributed storage systems in actual scenarios store large files, in our estimated calculations, the number of large files is simulated as 80% of the total number of files)

[0070] C2-2 The following are the storage efficiencies of several storage methods:

[0071]

[0072] C2-3 Assume that the overall storage cost of the system is E, and the calculation formula is as follows:

[0073] E=HOT*0.2+COLD*0.8

[0074] =3*0.2*0.2+1.66*0.2*0.8+1.66*0.8*0.2+1.5*0.8*0.8

[0075] =1.61

[0076] D. Change the file storage strategy based on popularity changes and system storage overhead

[0077] D1. Determine whether some file storage strategies need to be adjusted

[0078] D.1-1 Since the system storage overhead is 1.61, it is within the ideal storage efficiency. However, in practice, the system storage overhead depends on the size of the user's stored files. If the user stores more small files, the system storage overhead will increase. For example, when the ratio of small files to large files is 1:1, the estimated system overhead is 1.73. This exceeds the optimal storage efficiency we set. (If the system storage overhead is greater than 1.7x, it is a rare case, because it is almost impossible for the ratio of small files to large files to be 1:1 in a distributed storage system, and most of them are large files.)

[0079] D.1-2 We designed that when C1-2 obtains the G(i) set of the latest cycle, the system storage overhead will be calculated first. If the storage overhead is less than 1.7x, then no additional operations are required, and the storage strategy of the files whose popularity has changed can be changed directly according to the method in D2.

[0080] D.1-3 If the system storage overhead is greater than 1.7x with a very small probability, the file with the smallest heat H in the G(i) set will be deleted from the set to meet the system storage space setting. Then the system storage overhead is determined until the optimal system storage efficiency is met, and then the storage strategy of the file with changed heat is changed according to the method in D2.

[0081] D2. Change file storage strategy

[0082] After obtaining the heat set G(i) and determining the storage cost:

[0083] We change the H set, that is, the new hot files, to the new storage strategy LRC (6, 2, 2) or replica replication according to the file size.

[0084] We change the new cold files in set C to the new storage strategy RS(6,3) or RS(3,2) according to the file size.

[0085] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0086] The following is combined with Figure 1 , further describing the steps of the present invention, such as Figure 1 The steps of the present invention mainly include: A. calculating the popularity of files; B. classifying file types; C. calculating file popularity changes and storage overheads; and D. adjusting file storage strategies.

[0087] Specifically, the present invention comprises the following steps:

[0088] A. Calculate the heat of the file:

[0089] A1. Calculate the average access frequency f of each file avg :

[0090]

[0091] A2. Calculate the time access inclination I of each file:

[0092]

[0093] A3. Calculate the heat H of each file:

[0094] H=(1-β)I+βf avg (0<β<1)

[0095] B. File type classification:

[0096] B1. Four categories of documents:

[0097] The classification criteria for different types of file storage methods we adopt are heat and size. We classify files into hot files and cold files based on the heat of the files. We classify files into large files and small files based on the size of the files.

[0098] B2. Four storage strategies for files:

[0099] After having the above four file classification combinations, we can classify the file storage strategies into the following four categories:

[0100] HOT--BIG uses LRC (6, 2, 2), which means that 2 local check blocks are generated for every 6 data blocks.

[0101] 2 global checks are fast;

[0102] HOT--SMALL uses copy replication;

[0103] COLD--BIG uses RS (6, 3), that is, 3 parity check blocks are generated for every 6 data blocks;

[0104] COLD--SMALL uses RS(3,2), that is, 2 parity blocks are generated for every 3 data blocks.

[0105] C. Periodically count the popularity of files and estimate the overall storage overhead of the system:

[0106] C1. Statistics on file popularity changes:

[0107] After a cycle of user access, the access distribution of C1-1 will roughly conform to Zipf's law, but the user's access frequency and time period for each file will be significantly different. Some files may have more access at the beginning of this cycle, and the access frequency and number of accesses will gradually decrease in the middle and late stages, while the access frequency and number of accesses of some files may remain relatively "hot". We will first calculate the heat of the latest cycle according to the heat H calculation formula mentioned in A3.

[0108] C1-2 then counts the top 20% of the most popular files in the latest cycle and puts them into the set G(i)={F 1 ,F 2 ,...F n}, and the set G(i-1) of the previous cycle = {F 1 ,F 2 ,...F m} to perform file matching. The files that appear in set G(i) but not in set G(i-1) are put into set H = {F 1 ,F 2 ,...F i}, these files are the new hot files counted in the latest cycle. Files that appear in set G(i-1) but not in G(i) are put into set C = {F 1 ,F 2 ,...F j}, these files are the new cold files that have been counted as being changed from hot to cold in the latest cycle.

[0109] C2. Estimate the overall storage cost of the system

[0110] Assume that the overall storage cost of the system is E, and the calculation formula is as follows:

[0111] E=HOT*0.2+COLD*0.8

[0112] =3*0.2*0.2+1.66*0.2*0.8+1.66*0.8*0.2+1.5*0.8*0.8

[0113] =1.61

[0114] In the formula, 3 represents the storage overhead of replica replication; 1.66 represents the storage overhead of LRC (6, 2, 2). The two types of storage occupy 20% and 80% of the files respectively. And HOT files occupy 20% of the entire system.

[0115] The 1.66 in the formula represents the storage overhead of RS(3,2); 1.5 represents the storage overhead of RS(6,3). The two storages occupy 20% and 80% of the files respectively. And COLD files occupy 80% of the entire system.

[0116] D. Change the file storage strategy based on popularity changes and system storage overhead;

[0117] D1. Determine whether some file storage strategies need to be adjusted;

[0118] D.1-1 We designed that when C1-2 obtains the G(i) set of the latest period, the system storage overhead will be calculated first. If the storage overhead is less than 1.7x, then no additional operations are required, and the storage strategy of the files whose popularity has changed can be changed directly according to the method in D2.

[0119] D.1-2 If the system storage overhead is greater than 1.7x with a very small probability, the file with the smallest heat H in the G(i) set is deleted from the set, and the deleted data is set as cold data. This satisfies the system storage space setting. Then the system storage overhead is determined until the optimal system storage efficiency is met, and then the storage strategy of the file with changed heat is changed according to the method in D2.

[0120] D2. Change the file storage strategy;

[0121] After obtaining the heat set G(i) and determining the storage cost:

[0122] We change the H set, that is, the new hot files, to the new storage strategy LRC (6, 2, 2) or replica replication according to the file size.

[0123] We change the new cold files in set C to the new storage strategy RS(6,3) or RS(3,2) according to the file size.

[0124] The advantage of the present invention is that the file has a more suitable storage method according to the access characteristics of different types of files in the distributed system, and the file heat is calculated by the size of the data block occupied by the file and the user's access frequency to the file. In addition, we will make adaptive storage strategy changes to the file in each cycle according to the change of the file heat and the overall storage efficiency of the system. Ultimately, the performance of the system is improved and the overall storage efficiency of the system is improved.

[0125] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0126] The present invention also proposes a distributed adaptive storage system based on file access characteristics, which includes:

[0127] The heat calculation module is used to obtain the average access frequency of each file in the distributed storage system within a preset period, and calculate the heat of each file based on the average access frequency;

[0128] The file classification module is used to classify all files into hot data and cold data according to the popularity of each file in the current cycle, and further classify all the hot data into small hot data and large hot data according to the number of data blocks occupied by each hot data, and classify all the cold data into small cold data and large cold data according to the number of data blocks occupied by each cold data;

[0129] The file storage module is used to store the small hot data using a copy replication strategy, store the large hot data using LRC (6, 2, 2) encoding, store the small cold data using RS (3, 2) encoding, and store the large cold data using RS (6, 3).

[0130] The distributed adaptive storage system based on file access characteristics includes:

[0131] A comparison module is used to call the module 1 multiple times to periodically count the heat of each file, use the hot data in the latest cycle to compare the files with the hot data in the previous cycle, and obtain the new hot files newly generated in the latest cycle and the new cold files that have turned from hot to cold in the previous cycle;

[0132] The storage strategy change module is used to change the storage strategy of new hot files to LRC (6, 2, 2) encoding or copy replication storage, and change the storage strategy of new cold files to RS (6, 3) encoding or RS (3, 2) encoding storage.

[0133] The distributed adaptive storage method based on file access characteristics, wherein the comparison module includes: determining whether the current storage overhead of the distributed storage system is lower than a threshold, if so, executing the storage policy change module; otherwise, converting the file at the bottom of the hot data heat ranking in the current latest cycle into cold data, and then executing the storage policy change module.

[0134] The distributed adaptive storage system based on file access characteristics, wherein the heat calculation module includes:

[0135] The preset period is divided into multiple time periods, and one of the time periods is assumed to be the current heat statistics moment, and the other time period before the current heat statistics moment is assumed to be the previous heat statistics moment. In the preset period, the file access weight value closer to the current heat statistics moment is higher, and the file access weight value closer to the previous heat statistics moment is lower; the heat H of each file is calculated by the following formula:

[0136] H=(1-β)I+βf avg (0<β<1)

[0137] f avg is the average access frequency, I is the weight value, and β is the balance factor.

[0138] The present invention also proposes a storage medium for storing a program for executing any one of the distributed adaptive storage methods based on file access characteristics.

[0139] The present invention also proposes a client, which is used for any of the distributed adaptive storage systems based on file access characteristics.

Claims

1. A distributed adaptive storage method based on file access characteristics, characterized in that: include: Step 1: Obtain the average access frequency of each file in the distributed storage system within a preset period, and calculate the heat of each file based on the average access frequency; Step 2: According to the popularity of each file in the current cycle, all files are divided into hot data and cold data. According to the number of data blocks occupied by each hot data, all the hot data are further divided into small hot data and large hot data. According to the number of data blocks occupied by each cold data, all the cold data are divided into small cold data and large cold data. Step 3: The small hot data is stored using a copy replication strategy, the large hot data is stored using an LRC (6, 2, 2) encoding method, the small cold data is stored using an RS (3, 2) encoding method, and the large cold data is stored using an RS (6, 3) encoding method; Step 4: Execute step 1 multiple times to periodically count the heat of each file, use the hot data in the latest cycle to compare the files with the hot data in the previous cycle, and obtain the new hot files newly generated in the latest cycle and the new cold files that have turned from hot to cold in the previous cycle; Step 5: Change the storage strategy of the new hot file to LRC (6, 2, 2) encoding or copy replication storage, and change the storage strategy of the new cold file to RS (6, 3) encoding or RS (3, 2) encoding storage.

2. The distributed adaptive storage method based on file access characteristics according to claim 1, characterized in that: The step 4 includes: determining whether the current storage overhead of the distributed storage system is lower than a threshold value; if so, executing the step 5; otherwise, converting the file at the end of the hot data heat ranking in the latest cycle into cold data, and then executing the step 5.

3. The distributed adaptive storage method based on file access characteristics according to claim 1, characterized in that: This step 1 includes: The preset period is divided into multiple time periods, and one of the time periods is assumed to be the current heat statistics moment, and the other time period before the current heat statistics moment is assumed to be the previous heat statistics moment. In the preset period, the file access weight value closer to the current heat statistics moment is higher, and the file access weight value closer to the previous heat statistics moment is lower; the heat H of each file is calculated by the following formula: H=(1-β)I+βf avg (0<β<1) f avg is the average access frequency, I is the weight value, and β is the balance factor.

4. A distributed adaptive storage system based on file access characteristics, characterized in that: include: The heat calculation module is used to obtain the average access frequency of each file in the distributed storage system within a preset period, and calculate the heat of each file based on the average access frequency; The file classification module is used to classify all files into hot data and cold data according to the popularity of each file in the current cycle, and further classify all the hot data into small hot data and large hot data according to the number of data blocks occupied by each hot data, and classify all the cold data into small cold data and large cold data according to the number of data blocks occupied by each cold data; The file storage module is used to store the small hot data by adopting a copy replication strategy, store the large hot data by adopting an LRC (6, 2, 2) encoding method, store the small cold data by adopting an RS (3, 2) encoding method, and store the large cold data by adopting an RS (6, 3) encoding method; A comparison module is used to call the module 1 multiple times to periodically count the heat of each file, use the hot data in the latest cycle to compare the files with the hot data in the previous cycle, and obtain the new hot files newly generated in the latest cycle and the new cold files that have turned from hot to cold in the previous cycle; The storage strategy change module is used to change the storage strategy of new hot files to LRC (6, 2, 2) encoding or copy replication storage, and change the storage strategy of new cold files to RS (6, 3) encoding or RS (3, 2) encoding storage.

5. The distributed adaptive storage method based on file access characteristics according to claim 4, characterized in that: The comparison module includes: determining whether the current storage overhead of the distributed storage system is lower than a threshold; if so, executing the storage policy change module; otherwise, converting the file at the end of the hot data heat ranking in the latest cycle into cold data, and then executing the storage policy change module.

6. The distributed adaptive storage system based on file access characteristics according to claim 4, characterized in that: The heat calculation module includes: The preset period is divided into multiple time periods, and one of the time periods is assumed to be the current heat statistics moment, and the other time period before the current heat statistics moment is assumed to be the previous heat statistics moment. In the preset period, the file access weight value closer to the current heat statistics moment is higher, and the file access weight value closer to the previous heat statistics moment is lower; the heat H of each file is calculated by the following formula: H=(1-β)I+βf avg (0<β<1) f avg is the average access frequency, I is the weight value, and β is the balance factor.

7. A storage medium for storing a program for executing the distributed adaptive storage method based on file access characteristics as claimed in any one of claims 1 to 3.

8. A client, used in the distributed adaptive storage system based on file access characteristics as claimed in any one of claims 4 to 6.

Images