Space distribution method for distributed storage
By creating cached hard disk pools and data hard disk pools in distributed storage systems and selecting write hard disk pools based on data length and redundant policies, the space waste and performance problems in distributed storage space allocation are solved due to inappropriate data length in distributed storage space allocation. , achieving more efficient space utilization and reading performance.
Patent Information
- Application Number
- CN202411493903.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-18
AI Technical Summary
The existing distributed storage space allocation method results in wasted space when the write data length is less than or not the integer multiple of the minimum allocation unit, or when the write data length is much larger than the minimum allocation unit, the data is stored in multiple discontinuous storage spaces. , read performance worsens and adds metadata.
Create cached hard disk pools and data hard disk pools and configure them with different minimum allocation units. Select the written hard disk pool according to the data length and the redundancy policy of the data hard disk pool, and then the data of the cached hard disk pool is merged and written to the data hard disk pool.
It effectively reduces space waste, improves the utilization rate of storage space, reduces the increase in space fragmentation and metadata, and improves the performance of data reading.
Smart Images
Figure CN119473136A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of storage space allocation, and in particular relates to a distributed storage space allocation method. Background Art
[0002] Distributed storage is a computer data storage architecture that stores data on multiple servers to achieve high reliability, scalability, and performance. At the same time, distributed storage forms a virtual storage device with dispersed storage resources. In distributed storage, each server can be called a storage node, and they communicate and collaborate with each other through network connections to achieve distributed storage and management of data.
[0003] The storage system has a minimum allocation unit min_alloc_size for allocating storage space for written data. When a user writes data, the storage system allocates a size of n*min_alloc_size, where n is a positive integer. When the range of written data length varies little, configuring an appropriate min_alloc_size will prevent problems such as large space waste, space fragmentation, and excessive metadata.
[0004] However, with the development of storage technology, the data in storage systems is becoming more and more complex, and the range of written data length varies greatly. If min_alloc_size is configured too large, when the written data is small, the allocated storage space is much larger than the written data, resulting in a waste of space. If min_alloc_size is configured too small, when the written data is large, the allocated storage space requires multiple minimum allocation units, and the data may be stored in multiple discontinuous storage spaces. The performance of reading the data deteriorates, and the metadata used to describe the data storage location information also increases. The length of the written data is less than the minimum allocation unit or is not an integer multiple of the minimum allocation unit, resulting in a waste of storage space. The length of the written data is much larger than the minimum allocation unit, and the data is stored in multiple discontinuous storage spaces. The performance of reading the data deteriorates, and the metadata used to describe the data storage location information also increases.
[0005] Therefore, how to improve the existing distributed storage space allocation process to avoid the situation where the length of written data is less than the minimum allocation unit or is not an integer multiple of the minimum allocation unit, resulting in waste of storage space or the length of written data is much larger than the minimum allocation unit, and the data is stored in multiple discontinuous storage spaces, resulting in poor performance in reading the data, is a technical problem that urgently needs to be solved. Summary of the invention
[0006] The purpose of the present invention is to provide a distributed storage space allocation method to improve the existing distributed storage space allocation process to avoid the situation where the length of written data is less than the minimum allocation unit or is not an integer multiple of the minimum allocation unit, resulting in waste of storage space or the length of written data is much larger than the minimum allocation unit, the data is stored in multiple discontinuous storage spaces, and the performance of reading the data deteriorates.
[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:
[0008] A distributed storage space allocation method comprises the following steps:
[0009] S1: Create a cache disk pool and a data disk pool;
[0010] S2: Configure different minimum allocation units for the cache disk pool and data disk pool;
[0011] S3: Select the hard disk pool to be written based on the data length and the redundancy policy of the data hard disk pool;
[0012] S4: The data in the cache disk pool is merged and then written to the data disk pool.
[0013] Preferably, the specific process of creating the cache hard disk pool and the data hard disk pool in step S1 is as follows:
[0014] S11: creating a cache hard disk pool and setting a cache hard disk pool redundancy policy, wherein the cache hard disk pool is a solid state hard disk (SSD) used for temporarily storing data with a small data length;
[0015] S12: Create a data hard disk pool and set a data hard disk pool redundancy policy. The data hard disk pool is a mechanical hard disk HHD, which is used to store data with a large length or data after merging multiple data.
[0016] Preferably, when the cache hard disk pool is storing data, the data storage redundancy strategy of the cache hard disk pool is:
[0017] n copies are set during storage. n copies means that the data to be stored is copied n times and stored in different storage nodes. The number of SSDs d1 in each storage node is determined based on business performance.
[0018] When the business requires k IOPS of read and write operations per second, the number of storage nodes is N, and each SSD can provide m IOPS of read and write operations per second, each storage node selects d1 SSDs, d1 = (k*n) / m. If d1 is not an integer, it is rounded up.
[0019] Preferably, the data hard disk pool is preset with a redundancy strategy when storing data, and the data storage redundancy strategy of the data hard disk pool is: n copies or erasure code;
[0020] Erasure coding divides data into K data blocks, encodes them, and generates M check blocks. There are any K data blocks or check blocks to reconstruct the complete data. The number of HDDs d2 in each storage node is determined according to the amount of stored data.
[0021] When the estimated amount of data to be stored is total, the number of storage nodes is N, the capacity of each hard disk is S, and the data utilization rate of the redundancy strategy is util, each storage node selects d2 hard disks, d2 = total / (S*N*util)); if d2 is not an integer, it is rounded up.
[0022] Preferably, the process of configuring different minimum allocation units for the cache hard disk pool in step S2 is: based on the fact that the data length of temporary data stored in the cache hard disk pool is relatively small, a smaller minimum allocation unit is configured for the cache hard disk pool, wherein the smaller minimum allocation unit configured for the cache hard disk pool is allocate_1, and the size of allocate_1 is 4KB, 8KB, 16KB, or 32KB;
[0023] The minimum allocation unit of the data disk pool is configured as allocate_2, and the size of allocate_2 is 64KB, 128KB, 256KB, or 512KB.
[0024] Preferably, the specific process of selecting the hard disk pool to be written according to the data length and the redundancy strategy of the data hard disk pool in step S3 is as follows:
[0025] S31: when the length of the written data is L, the space waste rate W of the data written into the data hard disk pool is calculated;
[0026] S32: Compare the calculated space waste rate W with the preset space waste rate threshold W t Make a comparison;
[0027] S33: When W≤Wt, the data is written to the data hard disk pool; when W>Wt, the data is written to the cache hard disk pool.
[0028] Preferably, in step S31, when the redundancy strategy is n copies, the specific process of calculating the space waste rate W of writing data into the data hard disk pool is as follows:
[0029] S311: Calculate the actual occupied space R1. The specific calculation formula is as follows:
[0030] R1=[(L+allocate_2-1) / allocate_2]*allocate_2*n;
[0031] S312: Calculate the actual space utilization util_R1. The specific calculation formula is as follows:
[0032] util_R1=(R1-L) / R1*100%;
[0033] S313: Calculate theoretical space utilization util_M1. The specific calculation formula is as follows:
[0034] util_M1=1 / n*100%;
[0035] S314: Calculate the space waste rate W. The specific calculation formula is as follows:
[0036] W=(util_M1-util_R1) / util_M1*100%;
[0037] Among them, [x] represents the rounding operation, that is, taking the integer part of x.
[0038] Preferably, in step S31, when the redundancy strategy is EC K+M, the specific process of calculating the space waste rate W of writing data into the data hard disk pool is as follows:
[0039] Calculate the actual occupied space R2,
[0040] R2=[(L+K*allocate_2-1) / (K*allocate_2)]*(K+M)*allocate_2;
[0041] Calculate the actual space utilization util_R2. The specific calculation formula is as follows:
[0042] util_R2=(R2-L) / R2*100%;
[0043] Calculate the theoretical space utilization util_M2. The specific calculation formula is:
[0044] util_M2=K / (K+M)*100%
[0045] Calculate the space waste rate W. The specific calculation formula is:
[0046] W=(util_M2-util_R2) / util_M2*100%.
[0047] Preferably, the specific process of merging the data in the cache hard disk pool in step S4 and writing the merged data into the data hard disk pool is as follows:
[0048] S41: Read c pieces of data, which are a_1, a_2, a_3, ..., a_c in sequence, where c is a positive integer;
[0049] S42: Traverse a_1, a_2, a_3, ..., a_c, the cumulative length of the first p data is Calculate the space waste rate W_p. If W_p≤Wt, end the traversal.
[0050] S43: The data of a_1, a_2, ..., a_p are merged and written into the data hard disk pool, and the metadata describing the location information of a_1, a_2, ..., a_p is modified to reclaim the space of the cache hard disk pool.
[0051] The beneficial effects of the present invention include:
[0052] The space allocation method for distributed storage provided by the present invention creates a cache hard disk pool and a data hard disk pool; configures different minimum allocation units for the cache hard disk pool and the data hard disk pool; selects the hard disk pool to be written according to the data length and the redundancy strategy of the data hard disk pool; merges the data in the cache hard disk pool and writes the merged data to the data hard disk pool. The above storage space allocation process does not need to rely on the length of the data written by the user. The data hard disk pool adopts a larger allocation unit, which reduces space fragmentation and metadata. At the same time, data is merged in combination with a merger termination condition based on the space waste rate. The data merging condition and the termination condition of the merged data are both the space waste rate. The length of the merged data is less than and close to an integer multiple of the allocation unit, which effectively reduces space waste.
[0053] First, when the length of the written data is L, the space waste rate W of the data written into the data hard disk pool is calculated and compared with the preset space waste rate threshold W. t For comparison, when W≤Wt, data is written to the data hard disk pool; when W>Wt, data is written to the cache hard disk pool, and the calculation result based on the space waste rate is written to the corresponding hard disk pool. This eliminates the need to rely on the length of data written by the user, and the data hard disk pool uses a larger allocation unit, which reduces space fragmentation and metadata. At the same time, data is merged in combination with the merger termination condition based on the space waste rate. The length of the merged data is less than and close to an integer multiple of the allocation unit, which reduces space waste and effectively improves space utilization.
[0054] Secondly, the cache disk pool and the data disk pool use different minimum allocation units. The data disk pool uses a larger minimum allocation unit, combined with data merging based on space waste rate, to reduce space fragmentation and metadata describing data storage information, and improve the access performance of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a schematic diagram of the composition structure of the hard disk pool of the present invention.
[0056] Figure 2Schematic diagram of the principle of the erasure code of the present invention.
[0057] Figure 3 Schematic diagram of the principle of the erasure code with EC K=2, M=1 of the present invention.
[0058] Figure 4 The present invention is a flowchart of selecting a hard disk pool to be written according to the data length and the redundancy strategy of the data hard disk pool. DETAILED DESCRIPTION
[0059] The following is combined with Figure 1 to Figure 4 The present invention is further described in detail:
[0060] Example 1
[0061] See attached Figure 1 As shown, a distributed storage space allocation method includes the following steps:
[0062] S1: Create a cache disk pool and a data disk pool. The cache disk pool is used to temporarily store data with a short length, and the data disk pool is used to store data with a long length or data after merging multiple data.
[0063] S2: Different minimum allocation units are configured for the cache disk pool and the data disk pool. The cache disk pool is configured with a relatively small minimum allocation unit, while the data disk pool is configured with a relatively large minimum allocation unit.
[0064] S3: Select a hard disk pool to write data to based on the data length and the redundancy policy of the data hard disk pool, calculate the space waste rate corresponding to the data length based on the redundancy policy, and then select a hard disk pool to write data to based on the calculation result of the corresponding space waste rate;
[0065] S4: The data in the cache disk pool is merged and then written to the data disk pool.
[0066] A disk pool is a collection of disks from multiple storage nodes. Figure 1 As shown in the figure, a hard disk pool is a unit for storing data in a distributed storage system. A hard disk pool can provide storage space for a distributed storage system, and data can be stored in one or more hard disk pools. Read and write requests between hard disk pools do not affect each other. When a hard disk pool fails, other hard disk pools can still read and write data.
[0067] Example 2
[0068] Based on Example 1, the specific process of creating a cache hard disk pool and a data hard disk pool in step S1 is as follows:
[0069] S11: Create a cache hard disk pool and set a cache hard disk pool redundancy policy. The cache hard disk pool is a solid state drive (SSD) used to temporarily store data with a small data length.
[0070] When the cache hard disk pool is storing data, the data storage redundancy strategy of the cache hard disk pool is:
[0071] When storing, set n copies. n copies means that the data to be stored is copied n times and stored in different storage nodes. The number of SSDs d1 of each storage node is determined according to the business performance. When the business requires k IOPS of read and write operations per second, the number of storage nodes is N, and each SSD can provide m IOPS of read and write operations per second, then each storage node selects d1 SSDs, d1 = (k*n) / m. If d1 is not an integer, it is rounded up.
[0072] S12: Create a data hard disk pool and set a data hard disk pool redundancy policy. The data hard disk pool is a mechanical hard disk HHD, which is used to store data with a large length or data after merging multiple data.
[0073] The data hard disk pool has a preset redundancy strategy when storing data. The data storage redundancy strategy of the data hard disk pool is: n copies or erasure code. Erasure code divides data into K data blocks and encodes them to generate M check blocks. There are any K data blocks or check blocks to reconstruct the complete data. The number of HDDs d2 of each storage node is determined according to the amount of stored data.
[0074] When the estimated amount of data to be stored is total, the number of storage nodes is N, the capacity of each hard disk is S, and the data utilization rate of the redundancy strategy is util, each storage node selects d2 hard disks, d2 = total / (S*N*util)); if d2 is not an integer, it is rounded up.
[0075] In this embodiment, the redundancy strategy of n copies refers to storing multiple copies of the same data on hard disks of different nodes. When a disk is damaged or data is lost, data can be restored through data on other disks.
[0076] Erasure Coding (EC) is a coding fault-tolerant technology. Data is divided into data blocks, and check blocks are generated by calculation. Usually, the number of data blocks is represented by K, and the number of check blocks is represented by M. When no more than M storage blocks (data blocks or check blocks) are damaged, the original data can be calculated from the data of the remaining blocks.
[0077] Common terms used in EC include: Chunk: When encoding data based on erasure codes, each encoding will generate blocks of the same size (including data blocks and check blocks). These blocks are in order. Blocks with the same serial number are stored on the same hard disk. Stripe: If the data is too large to be encoded in one go, it can be divided into multiple times. Each encoded part is called a stripe. Stripes within the same data are in order. See Erasure Code Principles for more information. Figure 2 , Taking EC K=2, M=1 as an example: the original data is divided into 4 data blocks, and a check block is calculated from every 2 data blocks to form a stripe.
[0078] See also Figure 3 , distributed storage data is ultimately stored on a single hard disk, and the hard disk has its own space allocation rules. If the length of the data written by the user is D, in the case of multiple copies, each hard disk stores the same data of length D; in the case of EC K+M, each hard disk stores a data block of length D / K. Each hard disk has a minimum allocation unit min_alloc_size. When the user writes data, the size allocated by the hard disk is n*min_alloc_size, where n is a positive integer. After allocating space for the data, it is necessary to additionally record metadata that describes the data storage location information. When the data length is less than the minimum allocation unit, the data occupies the space of the minimum allocation unit size, and there is a certain amount of space waste. When the data length is greater than the minimum allocation unit, the data occupies multiple minimum allocation units, and the metadata that describes the data storage location information increases. At the same time, the several segments of the minimum allocation unit may not be continuous.
[0079] Example 3
[0080] On the basis of Example 1 or Example 2, the process of configuring different minimum allocation units for the cache hard disk pool in step S2 is: based on the fact that the data length of temporary data stored in the cache hard disk pool is small, a smaller minimum allocation unit is configured for the cache hard disk pool, and the smaller minimum allocation unit configured for the cache hard disk pool is allocate_1, and the size of allocate_1 is 4KB or 8KB or 16KB or 32KB, and the default is 32KB. The minimum allocation unit of the data hard disk pool is configured as allocate_2, and the size of allocate_2 is 64KB or 128KB or 256KB or 512KB, and the default is 512KB.
[0081] Example 4
[0082] On the basis of Example 1 or Example 2 or Example 3, see Figure 4 The specific process of selecting the hard disk pool to be written according to the data length and the redundancy strategy of the data hard disk pool in step S3 is as follows:
[0083] S31: when the length of the written data is L, the space waste rate W of the data written into the data hard disk pool is calculated;
[0084] S32: Compare the calculated space waste rate W with the preset space waste rate threshold W t Make a comparison;
[0085] S33: When W≤Wt, the data is written to the data hard disk pool; when W>Wt, the data is written to the cache hard disk pool.
[0086] In this embodiment, in step S31, when the redundancy strategy is n copies, the specific process of calculating the space waste rate W of writing data into the data hard disk pool is as follows:
[0087] S311: Calculate the actual occupied space R1. The specific calculation formula is as follows:
[0088] R1=[(L+allocate_2-1) / allocate_2]*allocate_2*n;
[0089] S312: Calculate the actual space utilization util_R1. The specific calculation formula is as follows:
[0090] util_R1=(R1-L) / R1*100%;
[0091] S313: Calculate theoretical space utilization util_M1. The specific calculation formula is as follows:
[0092] util_M1=1 / n*100%;
[0093] S314: Calculate the space waste rate W. The specific calculation formula is as follows:
[0094] W=(util_M1-util_R1) / util_M1*100%;
[0095] Among them, [x] represents the rounding operation, that is, taking the integer part of x.
[0096] In step S31, when the redundancy strategy is EC K+M, the specific process of calculating the space waste rate W of writing data into the data hard disk pool is as follows:
[0097] Calculate the actual occupied space R2,
[0098] R2=[(L+K*allocate_2-1) / (K*allocate_2)]*(K+M)*allocate_2;
[0099] Calculate the actual space utilization util_R2. The specific calculation formula is as follows:
[0100] util_R2=(R2-L) / R2*100%;
[0101] Calculate the theoretical space utilization util_M2. The specific calculation formula is:
[0102] util_M2=K / (K+M)*100%
[0103] Calculate the space waste rate W. The specific calculation formula is:
[0104] W=(util_M2-util_R2) / util_M2*100%.
[0105] Existing data merging is based on the data length L and the threshold Lt, that is, L≤Lt, then the data is merged. If L>Lt, it is written directly. At the same time, the termination condition for merging data is also to merge to the specified data length, without considering the space utilization. The data merging condition and the termination condition for merging data of the present invention are both space waste rate, which improves space utilization. The minimum allocation unit of all hard disk pools in most storage systems is the same. If the minimum allocation unit is too large, it will lead to space waste, and if the minimum allocation unit is too small, it will lead to space fragmentation and metadata increase. For example, a granular space allocation method, that is, a single hard disk has multiple allocation granularities (such as 32KB, 64K, 128K, 256K, etc. If the written data is 31KB, 32KB is allocated, if the written data is 63KB, 64KB is allocated, if 94KB is written, 1 32KB and 1 64KB are allocated. This patent solves the problem of space waste and space fragmentation to a certain extent. However, space waste and space fragmentation still depend on the length of data written by the user. The present invention does not depend on the length of data written by the user. The data hard disk pool adopts a larger allocation unit, which reduces space fragmentation and metadata. At the same time, data is merged in combination with a merge termination condition based on the space waste rate. The length of the merged data is less than and close to an integer multiple of the allocation unit, thereby reducing space waste.
[0106] In step S4, the data in the cache hard disk pool is merged and written into the data hard disk pool after merging. The specific process is as follows:
[0107] S41: Read c pieces of data, which are a_1, a_2, a_3, ..., a_c in sequence, where c is a positive integer;
[0108] S42: Traverse a_1, a_2, a_3, ..., a_c, the cumulative length of the first p data is Calculate the space waste rate W_p. If W_p≤Wt, end the traversal.
[0109] S43: The data of a_1, a_2, ..., a_p are merged and written into the data hard disk pool, and the metadata describing the location information of a_1, a_2, ..., a_p is modified to reclaim the space of the cache hard disk pool.
[0110] In summary, the space allocation method for distributed storage provided by the present invention creates a cache hard disk pool and a data hard disk pool; configures different minimum allocation units for the cache hard disk pool and the data hard disk pool; selects the hard disk pool to be written according to the data length and the redundancy strategy of the data hard disk pool; and merges the data in the cache hard disk pool and writes it to the data hard disk pool after merging. The above storage space allocation process does not need to rely on the length of the data written by the user. The data hard disk pool adopts a larger allocation unit, which reduces space fragmentation and metadata. At the same time, data merging is performed in combination with a merging termination condition based on the space waste rate. The data merging condition and the termination condition of the merged data are both the space waste rate. The length of the merged data is less than and close to an integer multiple of the allocation unit, which effectively reduces space waste.
[0111] When the length of the written data is L, the space waste rate W of the data written into the data hard disk pool is calculated and compared with the preset space waste rate threshold W t For comparison, when W≤Wt, data is written to the data hard disk pool; when W>Wt, data is written to the cache hard disk pool, and the calculation result based on the space waste rate is written to the corresponding hard disk pool. This eliminates the need to rely on the length of data written by the user. The data hard disk pool uses a larger allocation unit, which reduces space fragmentation and metadata. At the same time, data is merged based on the merging termination condition based on the space waste rate. The length of the merged data is less than and close to an integer multiple of the allocation unit, which reduces space waste and effectively improves space utilization. Different minimum allocation units are used for the cache hard disk pool and the data hard disk pool. The data hard disk pool uses a larger minimum allocation unit, combined with data merging based on the space waste rate, to reduce space fragmentation and metadata describing data storage information, and improve the access performance of the distributed storage system.
Claims
1. A distributed storage space allocation method, characterized in that: The following steps are involved: S1: Create a cache disk pool and a data disk pool; S2: Configure different minimum allocation units for the cache disk pool and data disk pool; S3: Select the hard disk pool to be written based on the data length and the redundancy policy of the data hard disk pool; S4: The data in the cache disk pool is merged and then written to the data disk pool.
2. A distributed storage space allocation method according to claim 1, characterized in that: The specific process of creating a cache hard disk pool and a data hard disk pool in step S1 is as follows: S11: creating a cache hard disk pool and setting a cache hard disk pool redundancy policy, wherein the cache hard disk pool is a solid state hard disk (SSD) used for temporarily storing data with a small data length; S12: Create a data hard disk pool and set a data hard disk pool redundancy policy. The data hard disk pool is a mechanical hard disk HHD, which is used to store data with a large length or data after merging multiple data.
3. A distributed storage space allocation method according to claim 2, characterized in that: When the cache hard disk pool is storing data, the data storage redundancy strategy of the cache hard disk pool is: n copies are set during storage. n copies means that the data to be stored is copied n times and stored in different storage nodes. The number of SSDs d1 in each storage node is determined based on business performance. When the business requires k IOPS of read and write operations per second, the number of storage nodes is N, and each SSD can provide m IOPS of read and write operations per second, each storage node selects d1 SSDs, d1 = (k*n) / m. If d1 is not an integer, it is rounded up.
4. A distributed storage space allocation method according to claim 2, characterized in that: When the data hard disk pool is storing data, a redundancy strategy is preset, and the data storage redundancy strategy of the data hard disk pool is: n copies or erasure code; Erasure coding divides data into K data blocks, encodes them, and generates M check blocks. There are any K data blocks or check blocks to reconstruct the complete data. The number of HDDs d2 in each storage node is determined according to the amount of stored data. When the estimated amount of data to be stored is total, the number of storage nodes is N, the capacity of each hard disk is S, and the data utilization rate of the redundancy strategy is util, each storage node selects d2 hard disks, d2 = total / (S*N*util)); if d2 is not an integer, it is rounded up.
5. A distributed storage space allocation method according to claim 1, characterized in that: The process of configuring different minimum allocation units for the cache hard disk pool in step S2 is: based on the fact that the data length of temporary data stored in the cache hard disk pool is small, a smaller minimum allocation unit is configured for the cache hard disk pool, wherein the smaller minimum allocation unit configured for the cache hard disk pool is allocate_1, and the size of allocate_1 is 4KB, 8KB, 16KB, or 32KB; The minimum allocation unit of the data disk pool is configured as allocate_2, and the size of allocate_2 is 64KB, 128KB, 256KB, or 512KB.
6. A distributed storage space allocation method according to claim 5, characterized in that: The specific process of selecting the hard disk pool to be written according to the data length and the redundancy policy of the data hard disk pool in step S3 is as follows: S31: when the length of the written data is L, the space waste rate W of the data written into the data hard disk pool is calculated; S32: Compare the calculated space waste rate W with the preset space waste rate threshold W t Make a comparison; S33: When W≤Wt, the data is written to the data hard disk pool; when W>Wt, the data is written to the cache hard disk pool.
7. A distributed storage space allocation method according to claim 6, characterized in that: In step S31, when the redundancy strategy is n copies, the specific process of calculating the space waste rate W of writing data into the data hard disk pool is as follows: S311: Calculate the actual occupied space R1. The specific calculation formula is as follows: R1=[(L+allocate_2-1) / allocate_2]*allocate_2*n; S312: Calculate the actual space utilization util_R1. The specific calculation formula is as follows: util_R1=(R1-L) / R1*100%; S313: Calculate theoretical space utilization util_M1. The specific calculation formula is as follows: util_M1=1 / n*100%; S314: Calculate the space waste rate W. The specific calculation formula is as follows: W=(util_M1-util_R1) / util_M1*100%; Among them, [x] represents the rounding operation, that is, taking the integer part of x.
8. A distributed storage space allocation method according to claim 6, characterized in that: In step S31, when the redundancy strategy is EC K+M, the specific process of calculating the space waste rate W of writing data into the data hard disk pool is as follows: Calculate the actual occupied space R2, R2=[(L+K*allocate_2-1) / (K*allocate_2)]*(K+M)*allocate_2; Calculate the actual space utilization util_R2. The specific calculation formula is as follows: util_R2=(R2-L) / R2*100%; Calculate the theoretical space utilization util_M2. The specific calculation formula is: util_M2=K / (K+M)*100% Calculate the space waste rate W. The specific calculation formula is: W=(util_M2-util_R2) / util_M2*100%.
9. A distributed storage space allocation method according to claim 6, characterized in that: In step S4, the data in the cache hard disk pool is merged and written into the data hard disk pool after merging. The specific process is as follows: S41: Read c pieces of data, which are a_1, a_2, a_3, ..., a_c in sequence, where c is a positive integer; S42: Traverse a_1, a_2, a_3, ..., a_c, the cumulative length of the first p data is Calculate the space waste rate W_p. If W_p≤Wt, end the traversal. S43: The data of a_1, a_2, ..., a_p are merged and written into the data hard disk pool, and the metadata describing the location information of a_1, a_2, ..., a_p is modified to reclaim the space of the cache hard disk pool.