Reconstruction method, system and equipment of distributed storage system and storage medium

By dynamically adjusting the maximum data capacity of the distributed storage system and deleting obsolete files, the problem of insufficient reconstruction reliability is solved, achieving high reconstruction efficiency and optimized utilization of storage space.

CN119002799APending Publication Date: 2024-11-22ZHENGZHOU INSPUR DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410774607.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-14
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing distributed storage systems suffer from insufficient reliability during the reconstruction process, especially due to reconstruction failures and wasted storage space caused by improper allocation of storage space.

Method used

Upon receiving a reconstruction request, the system determines the effective capacity of the storage pool based on the parameter information of the distributed storage system and the hard disk failure information. It then dynamically adjusts the maximum amount of data that can be accommodated by combining the preset storage space reservation ratio. Finally, it ensures that the actual usage capacity is lower than the maximum amount of data that can be accommodated by deleting files marked as obsolete, thereby initiating the reconstruction.

Benefits of technology

It improves the reliability of distributed storage system reconstruction, avoids waste caused by reserving storage space in advance, and ensures the success rate of the reconstruction process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002799A_ABST
    Figure CN119002799A_ABST
Patent Text Reader

Abstract

The invention discloses a distributed storage system reconstruction method, system and device and a storage medium, and is applied to the technical field of storage, and the method comprises the following steps: after receiving a reconstruction request, determining the effective capacity of a storage pool of a distributed storage system based on parameter information and hard disk fault information of the distributed storage system; determining the maximum receivable data volume in combination with the storage space reservation ratio; obtaining the actual use capacity of the distributed storage system; judging whether the actual use capacity is lower than the maximum receivable data volume or not; if yes, reconfiguration is started; and if not, deleting the file marked as an elimination state in the distributed storage system according to a preset deletion rule, so that reconstruction is started after the actual use capacity is lower than the maximum receivable data volume. By applying the scheme of the invention, the reliability of reconstruction of the distributed storage system can be guaranteed, and the storage space does not need to be reserved in advance, namely, the reliable execution of reconstruction is not guaranteed by sacrificing the storage space.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a distributed storage system reconstruction method, system, device and storage medium. Background Art

[0002] Distributed storage is a capacity-based storage system that can store a large number of files. For example, it can be used to store video files that require a large amount of storage space. In order to improve storage performance, overwriting is usually used to store files, which can reduce deletion operations and increase storage life. Figure 1 , is a schematic diagram of the current file storage structure, such as Figure 1 As shown in the figure, the maximum utilization space allocated to video storage will be divided into n blocks according to the capacity. The recorded video files can be written into the divided blocks. When the first block is full, the file is written into the second block, and so on. After all the block files are full, they are overwritten according to the rules. However, after running in this way for a period of time, the entire distributed storage system cluster has been in a high water level state, that is, a large number of files are stored. In order to ensure the reliability requirements of storage (reliability requirements in terms of accommodating disk and node failures), the write water level is usually set below the available capacity. For example, if there is a total of 100T storage space, 4T is reserved to accommodate disk failures, and the video file storage full water level is set to 96T. In addition, due to the distributed storage re-leveling mechanism, about 2% to 5% of temporary data will be generated during the reconstruction process. It is often necessary to reduce the write water level by more than 2%, otherwise the reconstruction task cannot be completed. In other words, in the current solution, the file storage based on distributed continuous overwrite writing achieves the purpose of high water level reconfigurability by lowering the threshold of file fullness, that is, a part of space is always reserved in advance for data reconstruction. For example, in one scenario, a 100T storage pool reserves 5T for failure response and 5T for reconstruction in failure scenarios. After the capacity is controlled to 90T, overwrite is enabled.

[0003] It can be seen that this solution based on distributed storage continuous overwriting achieves high-water mark reconstruction, but sacrifices a certain amount of storage space. In practical applications, it still cannot guarantee the thoroughness of reconstruction, that is, reconstruction failure is still prone to occur.

[0004] In summary, how to effectively ensure the reliability of the reconstruction of a distributed storage system is a technical problem that those skilled in the art urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed storage system reconstruction method, system, device and storage medium to effectively ensure the reliability of the distributed storage system reconstruction.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a distributed storage system reconstruction method, comprising:

[0008] After receiving the reconstruction request, determining the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information;

[0009] Based on the effective capacity of the storage pool and the preset storage space reservation ratio, determining the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction;

[0010] Obtaining the actual usage capacity of the distributed storage system;

[0011] Determining whether the actual used capacity is lower than the maximum accommodable data volume;

[0012] If yes, then start the reconstruction;

[0013] If not, the files marked as obsolete in the distributed storage system are deleted according to a preset deletion rule, so that the actual used capacity is lower than the maximum accommodable data volume, and then the reconstruction is started.

[0014] On the other hand, it also includes:

[0015] After the reconstruction is started, monitoring the remaining available capacity of the distributed storage system according to a preset period;

[0016] Whenever the proportion of the remaining available capacity in the effective capacity of the storage pool is lower than a preset first threshold, suspending reconstruction;

[0017] After pausing the reconstruction, the files marked as obsolete in the distributed storage system are deleted according to the deletion rule, so that the actual used capacity is lower than the maximum accommodable data volume, and then the suspension is ended and the reconstruction is continued.

[0018] On the other hand, based on the parameter information of the distributed storage system and the hard disk failure information, determining the effective capacity of the storage pool of the distributed storage system includes:

[0019] Based on the parameter information of the distributed storage system, determine the total number of hard disks, the capacity of a single hard disk, and the effective redundancy utilization rate of the distributed storage system;

[0020] Combined with the hard disk failure information, the effective capacity of the storage pool of the distributed storage system is determined in the manner of P=(N1-N2)×A×R;

[0021] Among them, P represents the effective capacity of the storage pool of the distributed storage system, N1 represents the total number of hard disks, and N2 represents the number of faulty hard disks obtained based on the hard disk failure information; A represents the capacity of a single hard disk, R represents the effective redundancy utilization rate, and in the copy redundancy mode, R=1 / B, B represents the number of copies preset in the copy redundancy mode, and in the erasure redundancy mode, R=k / (k+m), k represents the number of original data blocks preset in the erasure redundancy mode, and m represents the number of verification data blocks preset in the erasure redundancy mode.

[0022] On the other hand, according to the preset deletion rule, the files marked as obsolete in the distributed storage system are deleted so that the actual used capacity is lower than the maximum accommodable data amount, and then the reconstruction is started, including:

[0023] Calculate the difference between the actual usage capacity and the maximum accommodable data volume;

[0024] Determining the total size of files marked as obsolete in the distributed storage system;

[0025] Determine whether the total size is greater than the difference;

[0026] If not, the reconstruction process of the distributed storage system is suspended and an alarm is issued;

[0027] If yes, one or more files marked as obsolete in the distributed storage system are deleted, so that the actual used capacity is lower than the maximum accommodable data volume, and then reconstruction is started.

[0028] On the other hand, after deleting one or more files marked as obsolete in the distributed storage system so that the actual used capacity is lower than the maximum accommodable data volume, the reconstruction is started, including:

[0029] For the files marked as obsolete in the distributed storage system, selecting the files one by one;

[0030] Every time a file marked as eliminated is selected, the total capacity of all currently selected files is counted;

[0031] When the total capacity is greater than the maximum amount of data that can be accommodated, the selection is stopped, and the currently selected files marked as obsolete are deleted, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and after the deletion is completed, the reconstruction is started.

[0032] On the other hand, based on the effective capacity of the storage pool and the preset storage space reservation ratio, determining the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction includes:

[0033] Based on the effective capacity of the storage pool, the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction is determined in the manner of M=P×(1-a%-b%);

[0034] Among them, P represents the effective capacity of the storage pool of the distributed storage system, M represents the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction, a% is the proportion of temporary data in the reconstruction process in the effective capacity of the storage pool, the preset first reserved ratio, b% is the preset second reserved ratio, and both a and b are greater than 0.

[0035] On the other hand, it also includes:

[0036] After starting the reconstruction, the newly issued files are received;

[0037] According to the principle that the priority of creating a new space is the highest, the priority of overwriting is the second highest, and the priority of creating and writing a deleted space is the lowest, the newly issued file is written into the distributed storage system;

[0038] The new space creation and writing means writing the file into the free storage space in the distributed storage system;

[0039] For any storage space, when the storage space is created, the storage space is in an unoccupied idle state;

[0040] The overwrite write means writing the file into the storage space in the distributed storage system that is in an obsolete state;

[0041] For any storage space, when the files in the storage space are in the obsolete state, the storage space is in the obsolete state;

[0042] The deletion space creation write means writing the file into the storage space in the deletion state in the distributed storage system;

[0043] For any storage space, when a file in the storage space is in a deleted state, the storage space is in a deleted state;

[0044] Among them, for a file stored in the distributed storage system, when the existence time of the file reaches a preset time threshold, the state of the file is switched from an initial state to an eliminated state.

[0045] In a second aspect, the present invention provides a distributed storage system reconstruction system, comprising:

[0046] A storage pool effective capacity determination module, configured to determine the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information after receiving the reconstruction request;

[0047] A maximum accommodable data volume determination module, used to determine the maximum accommodative data volume of the distributed storage system for starting reconstruction based on the effective capacity of the storage pool and a preset storage space reservation ratio;

[0048] An actual used capacity acquisition module, used to acquire the actual used capacity of the distributed storage system;

[0049] Starting a determination module for determining whether the actual used capacity is lower than the maximum amount of data that can be accommodated;

[0050] If yes, triggering an execution module, wherein the execution module is used to start the reconstruction;

[0051] If not, the file deletion module is triggered, and the file deletion module is used to delete the files marked as obsolete in the distributed storage system according to the preset deletion rules, so that the actual usage capacity is lower than the maximum amount of data that can be accommodated, and then the execution module is triggered to start reconstruction.

[0052] In a third aspect, the present invention provides a distributed storage system reconstruction device, comprising:

[0053] Memory for storing computer programs;

[0054] A processor is used to execute the computer program to implement the steps of the distributed storage system reconstruction method as described above.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the distributed storage system reconstruction method as described above are implemented.

[0056] The technical solution provided by the embodiment of the present invention is applied, taking into account the situation that reconstruction failure may occur in traditional solutions. One of the reasons is that the storage space reserved for different occasions is different. When too much storage space is reserved, a lot of storage space is wasted, and when too little storage space is reserved, reconstruction failure is likely to occur.

[0057] In the solution of this application, there is no need to reserve storage space in advance. It is only necessary to determine whether the reconstruction can be started after receiving the reconstruction request. Specifically, after receiving the reconstruction request, it is necessary to determine the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information, and then determine the maximum amount of data that can be accommodated for the distributed storage system to start the reconstruction based on the effective capacity of the storage pool and the preset storage space reservation ratio. Since the maximum amount of data that can be accommodated is based on the effective capacity of the storage pool, the value of the reserved ratio is considered. Therefore, if the actual use capacity of the distributed storage system is lower than the maximum amount of data that can be accommodated, it means that the distributed storage system still has sufficient space to achieve reconstruction, and there is a high success rate after starting the reconstruction, so the reconstruction can be started. Correspondingly, if the actual use capacity of the distributed storage system is higher than the maximum amount of data that can be accommodated, it means that the distributed storage system still has sufficient space to achieve reconstruction. In this application solution, the files marked as obsolete in the distributed storage system will be deleted according to the preset deletion rules, so that the reconstruction is started after the actual use capacity is lower than the maximum amount of data that can be accommodated, so as to ensure the success rate of the reconstruction.

[0058] In summary, the solution of the present application is conducive to ensuring the reliability of the reconstruction of the distributed storage system, and the solution of the present application does not require the storage space to be reserved in advance, that is, the reliable execution of the reconstruction is not ensured by sacrificing the storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of a file storage structure;

[0061] Figure 2 A flowchart of a distributed storage system reconstruction method provided by a specific embodiment of the present invention;

[0062] Figure 3 A schematic diagram of three writing methods in a specific implementation manner of the present invention;

[0063] Figure 4 A schematic diagram of the structure of a distributed storage system provided by a specific implementation mode of the present invention;

[0064] Figure 5 A schematic diagram of the structure of a distributed storage device provided in a specific embodiment of the present invention;

[0065] Figure 6 The present invention is a schematic diagram of the structure of a computer-readable storage medium. DETAILED DESCRIPTION

[0066] The core of the present invention is to provide a distributed storage system reconstruction method, system, device and storage medium, which is conducive to ensuring the reliability of the reconstruction of the distributed storage system, and the present application scheme does not need to reserve storage space in advance, that is, the reliable execution of the reconstruction is not ensured by sacrificing storage space.

[0067] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0068] Please refer to Figure 2 , Figure 2 The following is a flowchart of a distributed storage system reconstruction method provided by a specific embodiment of the present invention. The distributed storage system reconstruction method may include the following steps:

[0069] Step S201: after receiving a reconstruction request, determining the effective capacity of a storage pool of the distributed storage system based on parameter information of the distributed storage system and hard disk failure information.

[0070] Specifically, each step of the solution of the present application can be implemented by a preset computer program. When a reconstruction request is received, it indicates that there is a faulty hard disk. Therefore, in the solution of the present application, it is necessary to determine the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk fault information.

[0071] The value of the effective capacity of the storage pool of the distributed storage system indicates how much storage space the distributed storage system can use to store data under the current circumstances. There are many specific implementation methods for determining the effective capacity of the storage pool. For example, in one case, this parameter exists in the parameter information of the distributed storage system, so it can be directly extracted from the parameter information of the distributed storage system. In some other cases, the effective capacity of the storage pool of the distributed storage system can be obtained by calculation based on the relevant data in the parameter information of the distributed storage system.

[0072] For example, in a specific implementation of the present invention, determining the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information described in step S201 may specifically include:

[0073] Based on the parameter information of the distributed storage system, determine the total number of hard disks, the capacity of a single hard disk, and the effective redundancy utilization rate of the distributed storage system;

[0074] Combined with the hard disk failure information, the effective capacity of the storage pool of the distributed storage system is determined in the manner of P = (N1-N2) × A × R;

[0075] Among them, P represents the effective capacity of the storage pool of the distributed storage system, N1 represents the total number of hard disks, N2 represents the number of faulty hard disks obtained based on hard disk fault information; A represents the capacity of a single hard disk, and R represents the effective redundancy utilization rate.

[0076] This implementation takes into account that when different storage redundancy strategies are adopted, the effective redundancy utilization rate of the distributed storage system is different. Specifically, in the replica redundancy mode, the effective redundancy utilization rate R=1 / B, where B represents the number of replicas preset in the replica redundancy mode, and in the erasure redundancy mode, the effective redundancy utilization rate R=k / (k+m), where k represents the number of original data blocks preset in the erasure redundancy mode, and m represents the number of check data blocks preset in the erasure redundancy mode.

[0077] Since the capacity of each hard disk in a distributed storage system is usually the same, after determining the total number of hard disks in the distributed storage system, the capacity of a single hard disk, and the effective redundancy utilization rate based on the parameter information of the distributed storage system, the effective capacity of the storage pool of the distributed storage system can be determined in the manner of P = (N1-N2) × A × R.

[0078] It can be seen that the effective capacity of the storage pool of the distributed storage system = (total number of hard disks - number of faulty hard disks) × capacity of a single hard disk × effective redundancy utilization rate. It can also be seen from the formula that when there are no faulty hard disks, the effective capacity of the storage pool is large. However, after a faulty hard disk appears, the effective capacity of the storage pool will decrease. The more faulty hard disks there are, the more the effective capacity of the storage pool will decrease.

[0079] For example, in one scenario, the distributed storage system stores video files, including 3 storage nodes, each storage node has 24 hard disks, each hard disk is 6T, and the replica redundancy mode is adopted, and 3 copies are adopted, that is, B=3. In this example, if no hard disk fails, the effective capacity of the storage pool at this time is P=72 disks×6T×1 / 3=144T. In other words, the capacity of 144T is the maximum write capacity of the video file in this example. For example, if one hard disk fails, the effective capacity of the storage pool at this time is P=(72-1)×6×1 / 3=142T.

[0080] Step S202: Based on the effective capacity of the storage pool and the preset storage space reservation ratio, determine the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction.

[0081] As described above, the effective capacity of the storage pool indicates how much storage space the distributed storage system can use to store data at most under the current circumstances.

[0082] During the reconstruction, some temporary data will be generated, and the distributed storage system may also receive new files, resulting in an increase in the actual usage capacity of the distributed storage system. Therefore, after obtaining the effective capacity of the storage pool, it is necessary to determine the maximum amount of data that can be accommodated when the distributed storage system starts the reconstruction according to the preset storage space reservation ratio. In other words, it is necessary to exclude a part of the effective capacity of the storage pool, and use the remaining as the maximum amount of data that can be accommodated when starting the reconstruction, to indicate that if the actual usage capacity of the distributed storage system exceeds this maximum amount of data that can be accommodated, there is a risk of reconstruction failure.

[0083] In a specific implementation of the present invention, step S202 may specifically include:

[0084] Based on the effective capacity of the storage pool, the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction is determined in the manner of M=P×(1-a%-b%);

[0085] Among them, P represents the effective capacity of the storage pool of the distributed storage system, M represents the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction, a% is the proportion of temporary data in the reconstruction process in the effective capacity of the storage pool, the preset first reserved ratio, b% is the preset second reserved ratio, and both a and b are greater than 0.

[0086] In this implementation, for the effective capacity P of the storage pool, two parts need to be reserved, one of which accounts for a%. The first reserved ratio is preset based on the proportion of temporary data in the reconstruction process in the effective capacity of the storage pool. For example, in actual applications, the proportion of temporary data in the reconstruction process in the effective capacity of the storage pool is approximately 2% to 5%. Therefore, in order to ensure the success rate of reconstruction, a% can be specifically set to 5%, for example.

[0087] In addition to reserving space for temporary data during the reconstruction process, in this implementation, a space of 1 b% is also reserved. This is because the distributed storage system may continue to receive new files during the reconstruction process. Therefore, after reserving such space, if the distributed storage system receives new files during the reconstruction process, but does not receive a particularly large number of new files, it will not affect the reconstruction process, which is conducive to ensuring the reconstruction success rate of the present application. For example, b% can be specifically set to 1%. Of course, it can be set and adjusted according to the specific business situation. For example, in a busy business situation, the value of b can be appropriately increased.

[0088] For example, in one of the above scenarios, the effective capacity of the storage pool P = 142T, and the maximum amount of data that can be accommodated M = P × (1-5% - 1%) = 142 × 94% = 133.48T.

[0089] Step S203: Acquire the actual usage capacity of the distributed storage system.

[0090] The actual usage capacity of the distributed storage system can be obtained through real-time monitoring or periodic monitoring. It can be understood that the maximum value of the actual usage capacity is the effective capacity P of the storage pool. At this time, the distributed storage system can no longer store any new files.

[0091] Step S204: Determine whether the actual used capacity is lower than the maximum accommodative data volume.

[0092] If yes, then execute step S205: start reconstruction;

[0093] If not, execute step S206: according to the preset deletion rule, delete the files marked as obsolete in the distributed storage system, so that the actual used capacity is lower than the maximum accommodative data amount, and then start the reconstruction.

[0094] From the above description, we can know that the maximum amount of data that can be accommodated is obtained by deducting the reserved storage space ratio from the effective capacity of the storage pool. Therefore, if the actual used capacity is lower than the maximum amount of data that can be accommodated, it means that if reconstruction is started currently, it will usually be successful, so reconstruction can be started directly.

[0095] When performing reconstruction, specifically, the data in each OSD of the failed hard disk is restored and saved in the OSD of the normal hard disk. There are many specific reconstruction algorithms, which can be set and adjusted according to actual needs. This application does not elaborate on this.

[0096] Correspondingly, if the actual used capacity is not less than the maximum amount of data that can be accommodated, it means that the actual used capacity is very high, that is, the actual used capacity is very close to the effective capacity of the storage pool. If reconstruction is started at this time, due to the temporary data generated during the reconstruction process and the distributed storage system may also receive new files during the reconstruction process, there is a high risk of reconstruction failure. In this regard, in the solution of the present application, the files marked as obsolete in the distributed storage system will be deleted according to the preset deletion rules.

[0097] It is understandable that, since the files marked as obsolete in the distributed storage system are deleted, the actual used capacity will be reduced. When the actual used capacity is lower than the maximum amount of data that can be accommodated, reconstruction can be started.

[0098] The specific content of the preset deletion rules can be set and adjusted according to actual needs, but only files marked as obsolete can be deleted. The specific conditions for marking files as obsolete can be set according to actual needs. For example, a simple implementation method is to determine based on the retention time of the file after it is placed in the distributed storage system, that is, for any file stored in the distributed storage system, when the existence time of the file does not reach the preset time threshold, the state of the file is the initial state, and when the existence time of the file reaches the preset time threshold, the state of the file is switched from the initial state to the obsolete state.

[0099] Of course, in other implementations, there may be more ways to mark files as obsolete, for example, directly changing one or more designated files to obsolete status according to the instructions of the staff.

[0100] In a specific embodiment of the present invention, it may also include:

[0101] After the reconstruction is started, the remaining available capacity of the distributed storage system is monitored according to a preset period;

[0102] Whenever the proportion of the remaining available capacity in the effective capacity of the storage pool is lower than a preset first threshold, suspending the reconstruction;

[0103] After pausing the reconstruction, the files marked as obsolete in the distributed storage system are deleted according to the deletion rules, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and then the suspension is ended and the reconstruction is continued.

[0104] This implementation method takes into account that although in the solution of the present application, reconstruction will only be initiated when the actual usage capacity is lower than the maximum amount of data that can be accommodated, which effectively guarantees the success rate of reconstruction, in a small number of cases, if no intervention is made in the reconstruction process, reconstruction failure may still occur. This is because in some cases, although storage space is reserved before the reconstruction begins, during the reconstruction process, the distributed storage system receives a particularly large number of new files, resulting in a lack of space to store the generated temporary data and the reconstruction fails.

[0105] In this regard, in this implementation, in order to further enhance the reliability of the reconstruction of the present application, after the reconstruction is started, the remaining available capacity of the distributed storage system will be monitored according to a preset period, for example, once every 5 minutes.

[0106] The remaining available capacity of the distributed storage system is the difference between the effective capacity of the storage pool and the actual used capacity. In addition, it can be understood that the temporary data generated during the reconstruction process also needs to be included in the actual used capacity.

[0107] Whenever the proportion of the remaining available capacity in the effective capacity of the storage pool is lower than the preset first threshold, it means that the remaining available capacity is insufficient. In order to ensure that the reconstruction process does not affect the writing of new files and to ensure the normal progress of the reconstruction process, in this implementation method, the reconstruction will be suspended at this time.

[0108] After pausing the reconstruction, the files marked as obsolete in the distributed storage system will be deleted according to the preset deletion rules, so that the actual used capacity is lower than the maximum amount of data that can be accommodated. Then the suspension can be ended and the reconstruction can be continued.

[0109] That is to say, in this implementation, after pausing the reconstruction, the operation of step S206 will be returned to be executed, so that the files marked as obsolete in the distributed storage system will be deleted according to the deletion rules, so that the actual used capacity can be lower than the maximum amount of data that can be accommodated before continuing the reconstruction.

[0110] In a specific implementation of the present invention, S206 may specifically include:

[0111] Calculate the difference between the actual used capacity and the maximum amount of data that can be accommodated;

[0112] Determine the total size of files marked as obsolete in the distributed storage system;

[0113] Determine whether the total size is greater than the difference;

[0114] If not, the reconstruction process of the distributed storage system is suspended and an alarm is issued;

[0115] If yes, one or more files marked as obsolete in the distributed storage system are deleted, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and then the reconstruction is started.

[0116] This implementation takes into account that in order to ensure that the actual usable capacity is lower than the maximum amount of data that can be accommodated after file deletion, the difference between the actual usable capacity and the maximum amount of data that can be accommodated can be calculated first, and the total size of the files marked as obsolete in the distributed storage system needs to be determined.

[0117] If the total size is greater than the difference, then in this specific implementation, one or more files marked as obsolete in the distributed storage system are selected for deletion, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and deletion is performed in units of files for ease of operation.

[0118] Correspondingly, if the total size is not greater than the difference, it means that even if all the files marked as obsolete are deleted, the actual use capacity cannot be lower than the maximum amount of data that can be accommodated, that is, the reliability of reconstruction cannot be guaranteed. Therefore, the reconstruction process of the distributed storage system can be suspended and an alarm can be issued at this time so that the staff can intervene to handle it. That is to say, in this implementation, when executing step S206, that is, according to the preset deletion rule, when deleting the files marked as obsolete in the distributed storage system, it is first necessary to determine whether the actual use capacity can be lower than the maximum amount of data that can be accommodated after the deletion action is performed. If not, the file can be not deleted first, and the reconstruction process can be suspended and an alarm can be issued.

[0119] Of course, in actual applications, if the reconstruction process of the distributed storage system is suspended and an alarm is sounded, if the staff does not intervene to handle it, the total size of the files marked as obsolete can also be updated according to the preset period. If it is found that the total size is already greater than the above difference in a certain period, one or more files marked as obsolete in the distributed storage system can be deleted, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and then the reconstruction can be restarted.

[0120] In a specific implementation of the present invention, deleting one or more files marked as obsolete in the distributed storage system so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and then starting the reconstruction may specifically include:

[0121] For files marked as obsolete in the distributed storage system, select the files one by one;

[0122] Every time a file marked as eliminated is selected, the total capacity of all currently selected files is counted;

[0123] When the total capacity is greater than the maximum amount of data that can be accommodated, the selection is stopped, and the currently selected files marked as obsolete are deleted, so that the actual used capacity is lower than the maximum amount of data that can be accommodated. After the deletion is completed, the reconstruction is started.

[0124] This implementation method takes into account that in order to improve storage performance and ensure storage life, deletion operations should be reduced. However, in the present application, in order to allow reconstruction to be initiated, it is indeed necessary to delete one or more files marked as obsolete in the distributed storage system. In this regard, this implementation method takes into account that for files marked as obsolete in the distributed storage system, files can be selected one by one. Whenever a file marked as obsolete is selected, the total capacity of all currently selected files is counted. Once the total capacity is greater than the maximum amount of data that can be accommodated, the selection can be stopped immediately, and then the currently selected files marked as obsolete can be deleted. It can be seen that this can avoid the need to delete too many files.

[0125] Furthermore, in one embodiment, for files marked as obsolete in a distributed storage system, when selecting files one by one, for example, the files marked as obsolete in the distributed storage system can be sorted in order from small to large, and after the sorting is completed, the selection starts from the smallest file, and each time a file marked as obsolete is selected, the total capacity of all currently selected files is counted, and the total capacity will gradually accumulate, and when the total capacity is greater than the maximum amount of data that can be accommodated, the currently selected files marked as obsolete can be deleted. And it can be seen that when the selection method of this embodiment is adopted, it is usually possible to stop the selection when the total capacity is slightly greater than the maximum amount of data that can be accommodated, which is conducive to minimizing the content that needs to be deleted to ensure the storage life.

[0126] In a specific embodiment of the present invention, it may also include:

[0127] After starting the reconstruction, the newly issued files are received;

[0128] Write newly issued files to the distributed storage system according to the principle that creating a new space has the highest priority, overwriting has the second highest priority, and deleting space has the lowest priority.

[0129] Among them, new space creation write means writing files to the free storage space in the distributed storage system;

[0130] For any storage space, when the storage space is created, the storage space is in an unoccupied idle state;

[0131] Overwrite means writing files to obsolete storage space in the distributed storage system.

[0132] For any storage space, when the files in the storage space are in the obsolete state, the storage space is in the obsolete state;

[0133] Deletion space creation write means writing files to the storage space in the deleted state in the distributed storage system;

[0134] For any storage space, when the files in the storage space are in the deleted state, the storage space is in the deleted state;

[0135] Among them, for a file stored in the distributed storage system, when the existence length of the file reaches a preset length threshold, the state of the file is switched from the initial state to the eliminated state.

[0136] This implementation takes into account that after the reconstruction is started, the distributed storage system may receive newly issued files, and when writing the newly issued files, this implementation gives priority to new space creation writes, followed by overwrite writes, and the lowest priority is deletion space creation writes.

[0137] For easier understanding, please refer to Figure 3 , which is a schematic diagram of three writing methods. For any storage space, when the storage space is created, the storage space is in an unoccupied idle state, and the idle state can be represented by Freestate=0. Therefore, it can be understood that when the distributed storage system is just created, all the storage spaces of the entire distributed storage system are in an unoccupied idle state, that is, Figure 3 As shown in , at this time, the status of all storage spaces is initialized to an idle state.

[0138] After the distributed storage system is created, files are gradually written. When a file is written into the distributed storage system, the file is in the initial state, and the storage space occupied by the file changes from the free state to the occupied state, which can be represented as Freestate=1.

[0139] For files stored in the distributed storage system, when the existence time of the file does not reach the preset time threshold, the state of the file is the initial state. Correspondingly, when the existence time of the file reaches the preset time threshold, it means that the file is expired, and the file is switched to the obsolete state. When a file is switched to the obsolete state, the storage space occupied by the file will change from the occupied state to the obsolete state, which can be expressed as Freestate=2.

[0140] Based on the above description, it can be known that files in the eliminated state will be deleted in some cases, and the state of the storage space released by the deleted files is the deleted state, which can be expressed as Freestate=3.

[0141] Therefore, if a file is written to a free storage space in a distributed storage system, it is called a new space creation write. If a file is written to an obsolete storage space in a distributed storage system, it is called an overwrite write. If a file is written to a deleted storage space in a distributed storage system, it is called a deleted space creation write.

[0142] Based on the above analysis, this implementation method takes into account that if there is idle storage space in the distributed storage system, in order to ensure the balance of the service life of different storage spaces, newly issued files should be written to the idle storage space first, that is, the creation of new space has the highest priority.

[0143] As can be seen from the above description, the storage space in the deleted state that appears in the present application scheme is the storage space reserved after deleting some files in order to achieve reconstruction. If the newly issued files are written to the storage space in the deleted state, it may cause the conditions for triggering the reconstruction to be unsatisfied, and it is necessary to continue to delete the files in the obsolete state, which increases the required deletion operations. Therefore, the priority of creating and writing the deletion space needs to be set to the lowest.

[0144] That is to say, if there is no free storage space, the newly issued file is preferably written into the obsolete storage space by overwriting.

[0145] Take the three storage nodes in the above implementation as an example, each storage node has 24 hard disks, each hard disk is 6T, and the replica redundancy mode is adopted with 3 copies, that is, B = 3. In this example, if there is no hard disk failure, the effective capacity of the storage pool at this time is P = 72 disks × 6T × 1 / 3 = 144T.

[0146] For example, the status of files and corresponding storage space in the distributed storage system is shown in Table 1.

[0147] Table 1:

[0148]

[0149] For example, if one hard disk fails, the effective capacity of the storage pool at this time is P = (72-1) × 6 × 1 / 3 = 142T. , and the maximum amount of data that can be accommodated is M = P × (1-5% - 1%) = 142 × 94% = 133.48T.

[0150] For example, if the actual used capacity is 132T, since 132T is less than 133.48T, it can be known from the above description that the operation of step S205 will be executed at this time, that is, the reconstruction will be started. And for example, in one embodiment, during the reconstruction process, the remaining available capacity of the distributed storage system will be monitored every 5 minutes. Whenever the remaining available capacity accounts for less than 1% of the effective capacity of the storage pool, the reconstruction can be suspended, and the files marked as obsolete in the distributed storage system are deleted according to the deletion rules. After the actual used capacity is less than the maximum amount of data that can be accommodated, the suspension is ended and the reconstruction is continued.

[0151] For example, if two hard disks fail, the effective capacity of the storage pool at this time is P = (72-2) × 6 × 1 / 3 = 140T. The maximum amount of data that can be accommodated is M = P × (1-5% - 1%) = 140 × 94% = 131.6T. The actual used capacity is 132T, which is greater than 133.48T, so it does not meet the reconstruction requirements. As can be seen from the previous description, the operation of step S206 will be executed at this time, so that the files marked as obsolete in the distributed storage system will be deleted according to the deletion rules. For example, when deleting, in one case, the difference between the actual used capacity 132T and the maximum amount of data that can be accommodated 131.6T is calculated to be 0.4T, that is, 409.6G. Taking Table 1 as an example, the total size of the files marked as obsolete in the distributed storage system is 600G+1024G+424G+1500G=3548G, and 3548G is greater than 409.6G. Therefore, one or more files marked as obsolete in the distributed storage system can be deleted, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and then the reconstruction can be started. For example, in this example, file 1 that occupies 600G can be deleted, so that the actual used capacity can be lower than the maximum amount of data that can be accommodated. Similarly, during the reconstruction process, the remaining available capacity of the distributed storage system can be monitored every 5 minutes, and the reconstruction can be suspended whenever the remaining available capacity accounts for less than 1% of the effective capacity of the storage pool.

[0152] For example, if three hard disks fail, the effective capacity of the storage pool at this time is P = (72-3) × 6 × 1 / 3 = 138T. The maximum amount of data that can be accommodated is M = P × (1-5% - 1%) = 138 × 94% = 129.72T. The actual used capacity is 132T, which is greater than 129.72T, so it does not meet the reconstruction requirements. As can be seen from the previous description, the operation of step S206 will be executed at this time, so that the files marked as obsolete in the distributed storage system will be deleted according to the deletion rules. For example, when deleting, the difference between the actual used capacity 132T and the maximum amount of data that can be accommodated 129.72T is calculated to be 2.28T, that is, 2334.72G. Taking Table 1 as an example, the total size of the files marked as obsolete in the distributed storage system is 600G+1024G+424G+1500G=3548G, and 3548G is greater than 2334.72G. Therefore, one or more files marked as obsolete in the distributed storage system can be deleted, so that the actual use capacity is lower than the maximum amount of data that can be accommodated, and then the reconstruction is started. For example, in this example, file 1 occupying 600G, file 2 occupying 1024G, and file 3 occupying 1924G (including file 3_1 in block 1 and file 3_2 in block 2) can all be deleted, so that the actual use capacity is lower than the maximum amount of data that can be accommodated.

[0153] For example, if 4 hard disks fail, the effective capacity of the storage pool at this time is P = (72-4) × 6 × 1 / 3 = 136T. The maximum amount of data that can be accommodated is M = P × (1-5% - 1%) = 136 × 94% = 127.84T. The actual used capacity is 132T, which is greater than 127.84T, so it does not meet the reconstruction requirements. As can be seen from the previous description, the operation of step S206 will be executed at this time, so that the files marked as obsolete in the distributed storage system will be deleted according to the deletion rules. For example, when deleting, the difference between the actual used capacity 132T and the maximum amount of data that can be accommodated 127.84T is calculated to be 4.16T, that is, 4259.84G. Taking Table 1 as an example, the total size of the files marked as obsolete in the distributed storage system is 600G+1024G+424G+1500G=3548G, and 3548G is less than 2334.72G. Therefore, even if all the files marked as obsolete in the current distributed storage system are deleted, the actual used capacity cannot be reduced to less than the maximum amount of data that can be accommodated. Therefore, in this example, reconstruction will not be started at this time, and the reconstruction process of the distributed storage system needs to be suspended and an alarm will be issued.

[0154] The technical solution provided by the embodiment of the present invention is applied, taking into account the situation that reconstruction failure may occur in traditional solutions. One of the reasons is that the storage space reserved for different occasions is different. When too much storage space is reserved, a lot of storage space is wasted, and when too little storage space is reserved, reconstruction failure is likely to occur.

[0155] In the solution of this application, there is no need to reserve storage space in advance. It is only necessary to determine whether the reconstruction can be started after receiving the reconstruction request. Specifically, after receiving the reconstruction request, it is necessary to determine the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information, and then determine the maximum amount of data that can be accommodated for the distributed storage system to start the reconstruction based on the effective capacity of the storage pool and the preset storage space reservation ratio. Since the maximum amount of data that can be accommodated is based on the effective capacity of the storage pool, the value of the reserved ratio is considered. Therefore, if the actual use capacity of the distributed storage system is lower than the maximum amount of data that can be accommodated, it means that the distributed storage system still has sufficient space to achieve reconstruction, and there is a high success rate after starting the reconstruction, so the reconstruction can be started. Correspondingly, if the actual use capacity of the distributed storage system is higher than the maximum amount of data that can be accommodated, it means that the distributed storage system still has sufficient space to achieve reconstruction. In this application solution, the files marked as obsolete in the distributed storage system will be deleted according to the preset deletion rules, so that the reconstruction is started after the actual use capacity is lower than the maximum amount of data that can be accommodated, so as to ensure the success rate of the reconstruction.

[0156] In summary, the solution of the present application is conducive to ensuring the reliability of the reconstruction of the distributed storage system, and the solution of the present application does not require the storage space to be reserved in advance, that is, the reliable execution of the reconstruction is not ensured by sacrificing the storage space.

[0157] Corresponding to the above method embodiment, an embodiment of the present invention further provides a distributed storage system reconstruction system, which can be referred to in correspondence with the above.

[0158] See also Figure 4 , is a schematic diagram of a structure of a distributed storage system reconstruction system in this aspect, including:

[0159] The storage pool effective capacity determination module 401 is used to determine the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information after receiving the reconstruction request;

[0160] A maximum accommodable data volume determination module 402, configured to determine the maximum accommodative data volume of the distributed storage system for starting reconstruction based on the effective capacity of the storage pool and a preset storage space reservation ratio;

[0161] The actual used capacity acquisition module 403 is used to acquire the actual used capacity of the distributed storage system;

[0162] Start the determination module 404, which is used to determine whether the actual used capacity is lower than the maximum amount of data that can be accommodated;

[0163] If yes, the execution module 405 is triggered, and the execution module 405 is used to start the reconstruction;

[0164] If not, the file deletion module 406 is triggered, and the file deletion module 406 is used to delete the files marked as obsolete in the distributed storage system according to the preset deletion rules, so that the actual usage capacity is lower than the maximum amount of data that can be accommodated, and then the execution module is triggered to start reconstruction.

[0165] In one embodiment of the present invention, a reconstruction process detection module is further included, which is used to:

[0166] After the reconstruction is started, monitoring the remaining available capacity of the distributed storage system according to a preset period;

[0167] Whenever the proportion of the remaining available capacity in the effective capacity of the storage pool is lower than a preset first threshold, suspending reconstruction;

[0168] After pausing the reconstruction, the file deletion module 406 is triggered to delete the files marked as obsolete in the distributed storage system according to the deletion rule, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and then the suspension is ended and the reconstruction is continued.

[0169] In one embodiment of the present invention, the storage pool effective capacity determination module 401 is specifically used to:

[0170] Based on the parameter information of the distributed storage system, determine the total number of hard disks, the capacity of a single hard disk, and the effective redundancy utilization rate of the distributed storage system;

[0171] Combined with the hard disk failure information, the effective capacity of the storage pool of the distributed storage system is determined in the manner of P=(N1-N2)×A×R;

[0172] Among them, P represents the effective capacity of the storage pool of the distributed storage system, N1 represents the total number of hard disks, and N2 represents the number of faulty hard disks obtained based on the hard disk failure information; A represents the capacity of a single hard disk, R represents the effective redundancy utilization rate, and in the copy redundancy mode, R=1 / B, B represents the number of copies preset in the copy redundancy mode, and in the erasure redundancy mode, R=k / (k+m), k represents the number of original data blocks preset in the erasure redundancy mode, and m represents the number of verification data blocks preset in the erasure redundancy mode.

[0173] In one embodiment of the present invention, the file deletion module 406 specifically includes:

[0174] A difference calculation unit, used to calculate the difference between the actual usage capacity and the maximum accommodable data volume;

[0175] A total size determination unit for obsolete files, used to determine the total size of files marked as obsolete in the distributed storage system;

[0176] A size judgment unit, used to judge whether the total size is greater than the difference;

[0177] If not, triggering an alarm unit to suspend the reconstruction process of the distributed storage system and issue an alarm;

[0178] If yes, a file deletion unit is triggered to delete one or more files marked as obsolete in the distributed storage system, so that the actual used capacity is lower than the maximum accommodable data volume, and then the reconstruction is started.

[0179] In one embodiment of the present invention, the file deletion unit is specifically used for:

[0180] For the files marked as obsolete in the distributed storage system, selecting the files one by one;

[0181] Every time a file marked as eliminated is selected, the total capacity of all currently selected files is counted;

[0182] When the total capacity is greater than the maximum amount of data that can be accommodated, the selection is stopped, and the currently selected files marked as obsolete are deleted, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and after the deletion is completed, the reconstruction is started.

[0183] In one embodiment of the present invention, the maximum accommodable data volume determination module 402 is specifically used to:

[0184] Based on the effective capacity of the storage pool, the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction is determined in the manner of M=P×(1-a%-b%);

[0185] Among them, P represents the effective capacity of the storage pool of the distributed storage system, M represents the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction, a% is the proportion of temporary data in the reconstruction process in the effective capacity of the storage pool, the preset first reserved ratio, b% is the preset second reserved ratio, and both a and b are greater than 0.

[0186] In one embodiment of the present invention, a newly issued file processing module is further included, which is used to:

[0187] After starting the reconstruction, the newly issued files are received;

[0188] According to the principle that the priority of creating a new space is the highest, the priority of overwriting is the second highest, and the priority of creating and writing a deleted space is the lowest, the newly issued file is written into the distributed storage system;

[0189] The new space creation and writing means writing the file into the free storage space in the distributed storage system;

[0190] For any storage space, when the storage space is created, the storage space is in an unoccupied idle state;

[0191] The overwrite write means writing the file into the storage space in the distributed storage system that is in an obsolete state;

[0192] For any storage space, when the files in the storage space are in the obsolete state, the storage space is in the obsolete state;

[0193] The deletion space creation write means writing the file into the storage space in the deletion state in the distributed storage system;

[0194] For any storage space, when a file in the storage space is in a deleted state, the storage space is in a deleted state;

[0195] Among them, for a file stored in the distributed storage system, when the existence time of the file reaches a preset time threshold, the state of the file is switched from an initial state to an eliminated state.

[0196] Corresponding to the above method and system embodiments, embodiments of the present invention further provide a distributed storage system reconstruction device, a computer-readable storage medium, and a computer program product, which can be referenced in correspondence with the above.

[0197] See also Figure 5 As shown, the device may include:

[0198] Memory 501, used for storing computer programs;

[0199] The processor 502 is configured to execute a computer program to implement the steps of the distributed storage system reconstruction method in any of the above embodiments.

[0200] The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the distributed storage system reconstruction method in any of the above embodiments are implemented.

[0201] See also Figure 6 The computer readable storage medium 60 stores a computer program 61, and when the computer program 61 is executed by the processor, the steps of the distributed storage system reconstruction method in any of the above embodiments are implemented. The computer readable storage medium 60 mentioned here includes RAM (Random Access Memory), memory, ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), register, hard disk, removable hard disk, or any other form of storage medium known in the technical field.

[0202] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0203] Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention, and the description of the above embodiments is only used to help understand the technical solution and its core idea of ​​the present invention. It should be pointed out that for ordinary technicians in the technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A distributed storage system reconstruction method, characterized in that: include: After receiving the reconstruction request, determining the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information; Based on the effective capacity of the storage pool and the preset storage space reservation ratio, determining the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction; Obtaining the actual usage capacity of the distributed storage system; Determining whether the actual used capacity is lower than the maximum accommodable data volume; If yes, then start the reconstruction; If not, the files marked as obsolete in the distributed storage system are deleted according to a preset deletion rule, so that the actual used capacity is lower than the maximum accommodable data volume, and then the reconstruction is started.

2. The distributed storage system reconstruction method according to claim 1, characterized in that: Also includes: After the reconstruction is started, monitoring the remaining available capacity of the distributed storage system according to a preset period; Whenever the proportion of the remaining available capacity in the effective capacity of the storage pool is lower than a preset first threshold, suspending reconstruction; After pausing the reconstruction, the files marked as obsolete in the distributed storage system are deleted according to the deletion rule, so that the actual used capacity is lower than the maximum accommodable data volume, and then the suspension is ended and the reconstruction is continued.

3. The distributed storage system reconstruction method according to claim 1, characterized in that: Determining the effective capacity of a storage pool of the distributed storage system based on parameter information of the distributed storage system and hard disk failure information includes: Based on the parameter information of the distributed storage system, determine the total number of hard disks, the capacity of a single hard disk, and the effective redundancy utilization rate of the distributed storage system; Combined with the hard disk failure information, the effective capacity of the storage pool of the distributed storage system is determined in the manner of P=(N1-N2)×A×R; Among them, P represents the effective capacity of the storage pool of the distributed storage system, N1 represents the total number of hard disks, and N2 represents the number of faulty hard disks obtained based on the hard disk failure information; A represents the capacity of a single hard disk, R represents the effective redundancy utilization rate, and in the copy redundancy mode, R=1 / B, B represents the number of copies preset in the copy redundancy mode, and in the erasure redundancy mode, R=k / (k+m), k represents the number of original data blocks preset in the erasure redundancy mode, and m represents the number of verification data blocks preset in the erasure redundancy mode.

4. The distributed storage system reconstruction method according to claim 1, characterized in that: According to a preset deletion rule, files marked as obsolete in the distributed storage system are deleted so that the actual used capacity is less than the maximum amount of data that can be accommodated, and then reconstruction is started, including: Calculate the difference between the actual usage capacity and the maximum accommodable data volume; Determining the total size of files marked as obsolete in the distributed storage system; Determine whether the total size is greater than the difference; If not, the reconstruction process of the distributed storage system is suspended and an alarm is issued; If yes, one or more files marked as obsolete in the distributed storage system are deleted, so that the actual used capacity is lower than the maximum accommodable data volume, and then reconstruction is started.

5. The distributed storage system reconstruction method according to claim 4, characterized in that: Deleting one or more files marked as obsolete in the distributed storage system so that the actual used capacity is less than the maximum amount of data that can be accommodated, and then starting reconstruction, including: For the files marked as obsolete in the distributed storage system, selecting the files one by one; Every time a file marked as eliminated is selected, the total capacity of all currently selected files is counted; When the total capacity is greater than the maximum amount of data that can be accommodated, the selection is stopped, and the currently selected files marked as obsolete are deleted, so that the actual used capacity is lower than the maximum amount of data that can be accommodated, and after the deletion is completed, the reconstruction is started.

6. The distributed storage system reconstruction method according to claim 1, characterized in that: Based on the effective capacity of the storage pool and the preset storage space reservation ratio, determining the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction includes: Based on the effective capacity of the storage pool, the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction is determined in the manner of M=P×(1-a%-b%); Among them, P represents the effective capacity of the storage pool of the distributed storage system, M represents the maximum amount of data that can be accommodated when the distributed storage system starts reconstruction, a% is the proportion of temporary data in the reconstruction process in the effective capacity of the storage pool, the preset first reserved ratio, b% is the preset second reserved ratio, and both a and b are greater than 0.

7. The distributed storage system reconstruction method according to any one of claims 1 to 6, characterized in that: Also includes: After starting the reconstruction, the newly issued files are received; According to the principle that the priority of creating a new space is the highest, the priority of overwriting is the second highest, and the priority of creating and writing a deleted space is the lowest, the newly issued file is written into the distributed storage system; The new space creation and writing means writing the file into the free storage space in the distributed storage system; For any storage space, when the storage space is created, the storage space is in an unoccupied idle state; The overwrite write means writing the file into the storage space in the distributed storage system that is in an obsolete state; For any storage space, when the files in the storage space are in the obsolete state, the storage space is in the obsolete state; The deletion space creation write means writing the file into the storage space in the deletion state in the distributed storage system; For any storage space, when a file in the storage space is in a deleted state, the storage space is in a deleted state; Among them, for a file stored in the distributed storage system, when the existence time of the file reaches a preset time threshold, the state of the file is switched from an initial state to an eliminated state.

8. A distributed storage system reconstruction system, characterized in that: include: A storage pool effective capacity determination module, configured to determine the effective capacity of the storage pool of the distributed storage system based on the parameter information of the distributed storage system and the hard disk failure information after receiving the reconstruction request; A maximum accommodable data volume determination module, used to determine the maximum accommodative data volume of the distributed storage system for starting reconstruction based on the effective capacity of the storage pool and a preset storage space reservation ratio; An actual used capacity acquisition module, used to acquire the actual used capacity of the distributed storage system; Starting a determination module for determining whether the actual used capacity is lower than the maximum amount of data that can be accommodated; If yes, triggering an execution module, wherein the execution module is used to start the reconstruction; If not, the file deletion module is triggered, and the file deletion module is used to delete the files marked as obsolete in the distributed storage system according to the preset deletion rules, so that the actual usage capacity is lower than the maximum amount of data that can be accommodated, and then the execution module is triggered to start reconstruction.

9. A distributed storage system reconstruction device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the distributed storage system reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the distributed storage system reconstruction method according to any one of claims 1 to 7 are implemented.