Method, device, equipment and storage medium for improving storage space utilization of erasure code

Abstract

A method, device, equipment and storage medium for improving the utilization of erasure code storage space, which relates to the field of data storage technology, wherein the method for improving the utilization of erasure code storage space includes: dividing the check shard into a replica data space and a check data space, wherein the replica data space only stores the replica data of a stripe that is not full, and the check data space only stores the check data of a full stripe; based on the size of the replica data or the existing data of the stripe, using the chunk size as a threshold, background aggregation or data recovery is performed on the check shard to convert the replica data and the check data. This application can reduce the waste of storage space and greatly improve the space utilization when the stripe is not full.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a method, device, equipment and storage medium for improving the utilization of erasure code storage space. Background Art

[0002] With the deepening of informatization, the amount of data generated by the whole society every day has shown explosive growth. Therefore, people's demand for the reliability and availability of data storage has become more and more urgent. Replicas and erasure codes are two common redundancy strategies in the distributed storage field. For some key business scenarios with high performance requirements, such as database scenarios, users often expect to adopt replica redundancy strategies to ensure data security, while ensuring read and write performance. For other business scenarios with low performance requirements and high storage capacity utilization, such as video storage scenarios, users often expect to adopt erasure code redundancy strategies, which ensure data security while improving storage capacity utilization.

[0003] Compared with the copy redundancy strategy, erasure coding can indeed greatly improve the space utilization of the storage system. However, because the corresponding verification data can only be calculated when the stripe is full of data, the space saving effect of erasure coding is greatly limited.

[0004] As we all know, for erasure code storage, the most ideal business scenario is that each write IO just fills an integer number of stripes, so that the verification data can be directly calculated. For the k+p (k original data blocks, p verification data blocks) erasure code model, the space utilization can reach the ideal k / (k+p). However, in actual business scenarios, it is impossible to write an integer number of stripes each time the write IO is completed. The common scenario is that the stripe is not fully written.

[0005] In view of the actual situation, there are currently two main response plans. The first is to adopt the zero-padding solution, and the second is to adopt the replica writing solution. Although these two solutions solve the problem that the stripe cannot be fully written in time, the zero-padding solution will cause the problem of waste of data and metadata space. Although the replica writing solution can solve the problems of space waste and write performance of the zero-padding solution, in the extreme scenario where the stripe has not been fully written, the space utilization rate of the replica writing solution is far from meeting the standard. Summary of the invention

[0006] The present application provides a method, device, equipment and storage medium for improving the utilization rate of erasure code storage space, which can improve writing performance and avoid wasting storage capacity space.

[0007] In a first aspect, an embodiment of the present application provides a method for improving the utilization of erasure code storage space, and the method for improving the utilization of erasure code storage space includes:

[0008] Divide the check shard into a replica data space and a check data space, wherein the replica data space stores only the replica data of the unfull stripe, and the check data space stores only the check data of the full stripe;

[0009] Based on the size of the replica data or existing data in the stripe, with the chunk size as the threshold, background aggregation or data recovery is performed on the verification shard to convert the replica data into the verification data.

[0010] In combination with the first aspect, in one implementation, based on the size of the replica data, with the chunk size as a threshold, background aggregation is performed on the verification shard to convert the replica data into verification data, including:

[0011] When the total size of the replica data in the stripe on the check shard is less than or equal to the chunk size, and there is no corresponding check data for the stripe, no operation is performed on the stripe;

[0012] When the total size of the replica data in the stripe on the check shard is greater than the chunk size and there is no corresponding check data for the stripe, the replica data is aggregated into check data by defaulting to 0 for the empty segment of the stripe, and the replica data of the stripe in the replica data space is deleted;

[0013] When there is new replica data in the stripe on the check shard and the stripe has corresponding check data, the stripe hole segment is treated as 0 by default, the check data is updated based on the new replica data, and all replica data of the stripe in the replica data space are deleted.

[0014] In combination with the first aspect, in one implementation, based on the size of existing data in the stripe, using the chunk size as a threshold, performing data recovery on the check shard, and converting the check data into replica data, includes:

[0015] When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is greater than the chunksize, and the stripe has corresponding check data, the check data is not restored to the replica data, but the check data is updated, and when calculating the new check data, the deleted replica data segment is processed as 0 by default;

[0016] When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is less than or equal to the chunk size, and the stripe has corresponding check data, the check data is restored to the replica data and the corresponding check data is deleted.

[0017] In combination with the first aspect, in one implementation, the check shard is divided into a replica data space and a check data space of equal size.

[0018] In a second aspect, an embodiment of the present application provides a device for improving the utilization of erasure code storage space, and the device for improving the utilization of erasure code storage space includes:

[0019] A partitioning module, which is used to divide the check slice into a replica data space and a check data space, wherein the replica data space only stores the replica data of the unfull stripe, and the check data space only stores the check data of the full stripe;

[0020] The data conversion module is used to perform background aggregation or data recovery on the verification shard based on the size of the replica data or the existing data in the stripe, with the chunk size as the threshold, so as to convert the replica data into the verification data.

[0021] In combination with the second aspect, in one implementation, the data conversion module performs background aggregation on the verification shard based on the size of the replica data and takes the chunk size as a threshold, and converts the replica data into verification data, including:

[0022] When the total size of the replica data in the stripe on the check shard is less than or equal to the chunk size, and there is no corresponding check data for the stripe, no operation is performed on the stripe;

[0023] When the total size of the replica data in the stripe on the check shard is greater than the chunk size and there is no corresponding check data for the stripe, the replica data is aggregated into check data by defaulting to 0 for the empty segment of the stripe, and the replica data of the stripe in the replica data space is deleted;

[0024] When there is new replica data in the stripe on the check shard and the stripe has corresponding check data, the stripe hole segment is treated as 0 by default, the check data is updated based on the new replica data, and all replica data of the stripe in the replica data space are deleted.

[0025] In combination with the second aspect, in one implementation, the data conversion module performs data recovery on the check shard based on the size of the existing data in the stripe and takes the chunk size as a threshold, and converts the check data into the copy data, including:

[0026] When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is greater than the chunksize, and the stripe has corresponding check data, the check data is not restored to the replica data, but the check data is updated, and when calculating the new check data, the deleted replica data segment is processed as 0 by default;

[0027] When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is less than or equal to the chunk size, and the stripe has corresponding check data, the check data is restored to the replica data and the corresponding check data is deleted.

[0028] In combination with the second aspect, in one implementation, the partitioning module is used to partition the check shard into a replica data space and a check data space of equal size.

[0029] In a third aspect, an embodiment of the present application provides a device for improving storage space utilization of erasure codes, wherein the device for improving storage space utilization of erasure codes comprises a processor, a memory, and a program for improving storage space utilization of erasure codes stored in the memory and executable by the processor, wherein when the program for improving storage space utilization of erasure codes is executed by the processor, the steps of any of the methods for improving storage space utilization of erasure codes described above are implemented.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which is stored a program for improving storage space utilization of erasure codes. When the program for improving storage space utilization of erasure codes is executed by a processor, the steps of any of the methods for improving storage space utilization of erasure codes described above are implemented.

[0031] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0032] The method for improving the utilization rate of erasure code storage space in the present application divides the check shard into a replica data space and a check data space, wherein the replica data space only stores replica data of a less-than-full stripe, and the check data space only stores check data of a full stripe; based on the size of the replica data or the existing data in the stripe, with the chunk size as a threshold, background aggregation or data recovery is performed on the check shard to convert the replica data and the check data into each other.

[0033] That is to say, the background aggregation process in the present application solution uses chunk size as a threshold. When the total size of the replica data in the stripe is greater than the chunk size, the background aggregation process will aggregate the stripe data into verification data (when calculating the verification data, the stripe hole segment is treated as 0 by default) and delete the replica data; in addition, for the stripe aggregated as verification data, when the total size of the original data it contains is less than or equal to the chunk size, the background aggregation process will restore the verification data to replica data and delete the verification data, so as to achieve the mutual conversion between replica data and verification data and achieve the purpose of optimizing space occupancy. The final space utilization of the present application will be as close to the theoretical extreme value k / (k+p) as possible, avoiding the waste of storage space as much as possible, and greatly improving the space utilization when the stripe is not fully written. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of an embodiment of a method for improving storage space utilization of erasure codes of the present application;

[0035] Figure 2 This is a processing strategy diagram for a full stripe in this application;

[0036] Figure 3 This is a processing strategy diagram for the incompletely written stripes of this application;

[0037] Figure 4 This is a schematic diagram of converting duplicate data into verification data in the first scenario of this application;

[0038] Figure 5 This is a schematic diagram of converting duplicate data into verification data in the second scenario of this application;

[0039] Figure 6 This is a schematic diagram of converting duplicate data into verification data in the third scenario of this application;

[0040] Figure 7 This is a schematic diagram of converting verification data into replica data in the first scenario of this application;

[0041] Figure 8 This is a schematic diagram of converting verification data into replica data in the second scenario of this application;

[0042] Fig. 9 This is a functional module diagram of an embodiment of a device for improving storage space utilization of erasure codes of the present application;

[0043] Fig.10 Schematic diagram of the hardware structure of the device for improving the storage space utilization of erasure codes involved in the embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0046] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0047] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0048] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0049] First, the relevant storage methods involved in this application are introduced:

[0050] Distributed storage system: a data storage system that stores data in multiple independent devices. Traditional network storage systems use centralized storage servers to store all data. Storage servers become the bottleneck of system performance and the focus of reliability and security, and cannot meet the needs of large-scale storage applications. Distributed network storage systems use an extensible system structure, use multiple storage servers to share storage loads, and use location servers to locate storage information. It not only improves system reliability, availability, and access efficiency, but is also easy to expand.

[0051] Erasure coded data storage: is a data protection method that divides the original data into k original data blocks according to the specified size, and then encodes it into p check data blocks according to the redundancy level, and finally stores the original data blocks and check data blocks in different locations to achieve the purpose of data protection.

[0052] Replica data storage: is a data protection method that treats the original data as a whole and stores n identical copies of the data in n different locations according to the redundancy level to achieve the purpose of data protection.

[0053] It is worth mentioning that the zero-filling scheme is mainly divided into direct zero-filling and zero-filling after waiting timeout:

[0054] Direct zero padding: Each time a write IO arrives, the partially filled stripes are directly processed by zero padding. This strategy is simple and crude, and the degree of space waste depends entirely on the foreground IO. In most cases, the space waste is very serious.

[0055] Fill with zeros after waiting timeout: Each time a write IO arrives, for the stripes that are not full, it is necessary to wait for a certain period of time, and then fill the strips with zeros after the timeout. This strategy has been improved on the original basis, but the degree of space waste still depends on the foreground IO. At the same time, the waiting operation reduces the IO write performance to a certain extent.

[0056] In addition, no matter which zero-padding scheme is used, metadata needs to be added to process the zero-padding segment, which indirectly increases the space usage. Therefore, it can be seen that the zero-padding scheme will lead to a waste of data and metadata space.

[0057] For the replica write scenario:

[0058] The zero-filling solution directly or indirectly causes space waste. The root cause is that the foreground IO in actual business scenarios cannot guarantee that each write IO will just fill an integer number of stripes. In view of this, the copy writing solution came into being. Its general process is as follows:

[0059] When a write IO arrives, the control process first divides the data into the corresponding stripes according to information such as IO offset, stripe size, chunk size, and redundancy level. For a full stripe, the check data corresponding to the stripe data is directly calculated, and then the original data block and the check data block are written to the disk together. For a stripe that is not full, it is directly written as a whole to the corresponding data shard and check shard in the form of a copy. A background task is started on the check shard to regularly detect the data written to the stripe. When it is detected that the stripe is full, the copy data on the stripe is aggregated into check data, and then the copy data is deleted.

[0060] Compared with the zero-padding solution, the replica write solution does not need to wait for the foreground IO to write the full stripe, so the IO write performance is not affected. Because there is no zero-padding logic, there is no space waste caused by the zero-padding solution.

[0061] However, the premise for stripe data to be aggregated as verification data by background tasks is that the stripe is full. Therefore, considering the extreme scenario, the foreground IO has not yet filled the stripe, and the data in the stripe has always existed in the form of copies. For the k+p erasure code model, the space utilization rate degenerates to 1 / (1+p).

[0062] Therefore, although the replica write solution solves the problems of space waste and write performance of the zero-fill solution, in extreme scenarios where the stripe has not been fully written, the space utilization of the replica write solution is far from meeting the standard.

[0063] For the k+p erasure code model, the ideal space utilization is k / (k+p). In order to approach this effect as closely as possible, this application solution improves the background aggregation process based on the replica write solution. The main ideas are:

[0064] The background aggregation task started on the verification shard periodically detects the replica data written to the stripe. The background aggregation process uses the chunk size as the threshold. When the total size of the replica data in the stripe is larger than the chunk size, the background aggregation process aggregates the stripe data into verification data (when calculating the verification data, the stripe hole segment is treated as 0 by default) and deletes the replica data. In addition, for the stripe aggregated as verification data, when the total size of the original data it contains is less than or equal to the chunk size, the background aggregation process restores the verification data to replica data and deletes the verification data, thereby realizing the mutual conversion between replica data and verification data and achieving the purpose of optimizing space occupancy.

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0066] In a first aspect, an embodiment of the present application provides a method for improving the storage space utilization of erasure codes.

[0067] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for improving the utilization rate of erasure code storage space in this application. Figure 1 As shown, the method for improving the utilization of erasure code storage space includes:

[0068] S1. Divide the check shard into a replica data space and a check data space, wherein the replica data space only stores replica data of a non-full stripe, and the check data space only stores check data of a full stripe.

[0069] S2. Based on the size of the replica data or the existing data in the stripe, with the chunk size as the threshold, background aggregation or data recovery is performed on the verification shard to convert the replica data into the verification data.

[0070] In this embodiment, the check shard is divided into two parts, which are specifically used to store corresponding data. In specific implementation, the check shard can be divided into a copy data space and a check data space of equal size.

[0071] It is worth noting that after the foreground IO arrives, the control process first divides the data into the corresponding stripes according to information such as IO offset, stripe size, chunksize, and redundancy level. Among them, IO offset refers to the IO offset, chunksize refers to the size of each chunk data block, that is, the stripe depth, and stripe size refers to the size of all chunk data blocks, that is, the sum of chunk blocks in all data shards.

[0072] To facilitate the explanation of the present application, the following description is based on erasure code 4+2 (4 data shards, 2 check shards) and a chunk size of 4K.

[0073] For the ideal case, see Figure 2 As shown in the figure, for a full stripe, the corresponding original data will be divided into four 4K chunk data blocks A, B, C, and D. At the same time, the control process will use the full stripe data to calculate two 4K check data blocks P and Q. Obviously, the original data blocks A, B, C, and D need to be stored in data shards 0, 1, 2, and 3 in sequence. The check shards are equally divided into replica data space and check data space. The replica data space only stores replica data of unfull stripes, and the check data space only stores check data of full stripes. Check data blocks P and Q are stored at corresponding offsets in the check data space of check shards P and Q, respectively.

[0074] Since the current business scenario is the most ideal, write IO just fills an integer number of stripes, so the control process can directly calculate the verification data.

[0075] However, in actual business scenarios, it is impossible to write an integer number of stripes every time you write IO. Figure 3 As shown, the common scenario is that the stripe is not fully written, and the original data is also split into chunk sizes and stored in the corresponding data shards. The stripe check data is not calculated, and all the data in the stripe is directly written as a whole to the corresponding offset in the replica data space in the form of a copy.

[0076] For this situation, this application performs background aggregation or data recovery on the verification shard based on the size of the replica data or existing data in the stripe, with chunk size as the threshold, to convert the replica data and verification data. Background aggregation is a background task started on the verification shard, which is used to regularly scan the replica data, process the logic of converting the replica data and verification data, and dynamically optimize the space occupancy rate.

[0077] It can be understood that the mutual conversion between the replica data and the verification data includes two situations, namely, the conversion of the replica data into the verification data and the conversion of the verification data into the replica data. These two situations are introduced below respectively.

[0078] When the stripe data meets the aggregation conditions, the replica data needs to be converted into verification data. The conversion of replica data into verification data is mainly divided into three scenarios:

[0079] (1) The total size of the replica data is less than or equal to the chunk size, and there is no verification data;

[0080] See also Figure 4 As shown in the figure, when the total size of the replica data in the stripe is less than or equal to the chunk size and the stripe has no corresponding check data, the background aggregation process will not perform any operation on the stripe.

[0081] The reason is that background aggregation aggregates the replica data in the stripe into fixed-size verification data, which is the chunk size. When the total size of the replica data in the stripe is smaller than the chunk size, the background aggregation not only adds processing operations but also adds space occupancy, which is contrary to the original intention of aggregation. Similarly, when the total size of the replica data in the stripe is equal to the chunk size, although the space occupancy does not change, new processing operations are added, which is also unnecessary.

[0082] (2) The total size of the replica data is larger than the chunk size, and there is no verification data;

[0083] See also Figure 5 As shown in the figure, when the total size of the replica data in the stripe is greater than the chunk size, and the stripe has no corresponding check data, the background aggregation process will aggregate the replica data into check data. Since the replica data that is not full of stripes cannot directly calculate the check data, in order to achieve the purpose of aggregation even for incomplete stripes, the stripe hole segment is treated as 0 by default when calculating the check data; it is worth noting that the default 0 treatment is not the zero padding operation in the zero padding scheme, it just treats the stripe hole segment as 0 by default when calculating. After the aggregation process calculates the check data and transfers it to the check data space, the data of the stripe in the replica data space will be deleted.

[0084] It can be seen that compared with the copy writing scheme before the improvement, the space is further saved, and the amount of space saved is p*(B+C).

[0085] (3) There is new copy data and verification data;

[0086] See also Figure 6 As shown in the figure, when there is new copy data in the stripe and the corresponding verification data exists in the stripe, the background aggregation process will use the new copy data to update the verification data; considering that there may be stripe holes when the verification data is calculated for the first time, when calculating the new verification data, it is necessary to combine the new copy data to find the existing empty segments in the stripe. If there are still empty segments in the stripe, when calculating the new verification data, the existing empty segments are still treated as 0 by default.

[0087] After calculating the new checksum data, the aggregation process updates it to the checksum data space and then deletes all duplicate data of the stripe in the duplicate data space.

[0088] When the stripe data does not meet the aggregation conditions, the verification data needs to be restored to the replica data. The conversion of verification data to replica data is mainly divided into two scenarios:

[0089] (1) When there is deleted replica data, the total size of the existing data in the stripe is greater than the chunk size, and the stripe has corresponding check data;

[0090] It is worth noting that the stripe existing data in this embodiment refers to Figure 2 For the data in the data shard, when converting the replica data into the verification data, the existing data of the stripe and the replica data are equivalent. However, when converting the verification data into the replica data, the replica data cannot fully represent the current valid data volume of the stripe, so the existing data of the stripe is used for analysis and explanation.

[0091] See also Figure 7As shown in the figure, when the background aggregation task scans and finds deleted replica data in the stripe, and the total size of the existing data in the stripe is greater than the chunk size, and the corresponding check data of the stripe also exists, the background aggregation process will update the check data; when calculating the new check data, the deleted replica data segment will be processed according to the default 0.

[0092] The reason why the check data is not converted to replica data is that when the total size of the existing data in the stripe is larger than the chunk size, if the check data is restored to replica data, it will cause a certain waste of storage space, which is contrary to the original intention of aggregation. If the check data is updated according to the default value of 0, the space gain will continue to be maintained.

[0093] (2) When there is deleted replica data, the total size of the existing data in the stripe is less than or equal to the chunk size, and the stripe has corresponding check data;

[0094] See also Figure 8 As shown in the figure, when the background aggregation task scans the stripe and finds that there is deleted replica data, and the total size of the existing data in the stripe is less than or equal to the chunk size, and the corresponding check data of the stripe also exists, the background aggregation process will restore the check data to replica data and then delete the corresponding check data.

[0095] The reason is that when the total size of the existing data in the stripe is less than or equal to the chunk size, if the check data is not restored to the replica data and the check data is forcibly maintained, a certain amount of storage space will be wasted, which is contrary to the original intention of aggregation.

[0096] After the above steps, when the total size of the replica data is less than or equal to the chunk size, the space utilization is a fixed value, namely 1 / (1+p); when the total size of the replica data is greater than the chunk size, the space utilization will continue to increase as the stripe data is written, namely from 1 / (1+p) to k / (k+p), thereby maximally avoiding the waste of storage space and greatly improving the space utilization when the stripe is not fully written.

[0097] To summarize, the method for improving the utilization of erasure code storage space in the present application divides the check shard into a replica data space and a check data space, wherein the replica data space only stores replica data of a less-than-full stripe, and the check data space only stores check data of a full stripe; based on the size of the replica data or the existing data in the stripe, with chunksize as the threshold, background aggregation or data recovery is performed on the check shard to convert the replica data and the check data into each other.

[0098] That is to say, the background aggregation process in the present application solution uses chunk size as a threshold. When the total size of the replica data in the stripe is greater than the chunk size, the background aggregation process will aggregate the stripe data into verification data (when calculating the verification data, the stripe hole segment is treated as 0 by default) and delete the replica data; in addition, for the stripe aggregated as verification data, when the total size of the original data it contains is less than or equal to the chunk size, the background aggregation process will restore the verification data to replica data and delete the verification data, so as to achieve the mutual conversion between replica data and verification data and achieve the purpose of optimizing space occupancy. The final space utilization of the present application will be as close to the theoretical extreme value k / (k+p) as possible, avoiding the waste of storage space as much as possible, and greatly improving the space utilization when the stripe is not fully written.

[0099] In a second aspect, an embodiment of the present application also provides a device for improving the utilization of erasure code storage space.

[0100] In one embodiment, referring to Fig. 9 , Fig. 9 This is a functional module diagram of an embodiment of the device for improving the utilization rate of erasure code storage space of the present application. Fig. 9 As shown, the device for improving the utilization of erasure code storage space includes:

[0101] A partitioning module, which is used to divide the check slice into a replica data space and a check data space, wherein the replica data space only stores the replica data of the unfull stripe, and the check data space only stores the check data of the full stripe;

[0102] The data conversion module is used to perform background aggregation or data recovery on the verification shard based on the size of the replica data or the existing data in the stripe, with the chunk size as the threshold, so as to convert the replica data into the verification data.

[0103] Furthermore, in one embodiment, the data conversion module performs background aggregation on the verification shard based on the size of the replica data and takes the chunk size as a threshold, and converts the replica data into verification data, including:

[0104] When the total size of the replica data in the stripe on the check shard is less than or equal to the chunk size, and there is no corresponding check data for the stripe, no operation is performed on the stripe;

[0105] When the total size of the replica data in the stripe on the check shard is greater than the chunk size and there is no corresponding check data for the stripe, the replica data is aggregated into check data by defaulting to 0 for the empty segment of the stripe, and the replica data of the stripe in the replica data space is deleted;

[0106] When there is new replica data in the stripe on the check shard and the stripe has corresponding check data, the stripe hole segment is treated as 0 by default, the check data is updated based on the new replica data, and all replica data of the stripe in the replica data space are deleted.

[0107] Furthermore, in one embodiment, the data conversion module performs data recovery on the check shard based on the size of the existing data in the stripe and takes the chunksize as a threshold, and converts the check data into the copy data, including:

[0108] When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is greater than the chunksize, and the stripe has corresponding check data, the check data is not restored to the replica data, but the check data is updated, and when calculating the new check data, the deleted replica data segment is processed as 0 by default;

[0109] When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is less than or equal to the chunk size, and the stripe has corresponding check data, the check data is restored to the replica data and the corresponding check data is deleted.

[0110] Furthermore, in one embodiment, the partitioning module is used to partition the check shard into a replica data space and a check data space of equal size.

[0111] Among them, the functional implementation of each module in the above-mentioned device for improving the utilization rate of erasure code storage space corresponds to the steps in the above-mentioned embodiment of the method for improving the utilization rate of erasure code storage space, and its functions and implementation processes will not be repeated here one by one.

[0112] In a third aspect, an embodiment of the present application provides a device for improving the utilization of erasure code storage space. The device for improving the utilization of erasure code storage space can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0113] Reference Fig.10 , Fig.10 Schematic diagram of the hardware structure of the device for improving the storage space utilization of erasure codes involved in the embodiment of the present application. In the embodiment of the present application, the device for improving the storage space utilization of erasure codes may include a processor, a memory, a communication interface, and a communication bus.

[0114] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0115] The communication interface includes an input / output (I / O) interface, a physical interface, and a logical interface, etc., which are used to interconnect the devices inside the erasure code storage space utilization improvement device, and an interface used to interconnect the erasure code storage space utilization improvement device with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0116] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0117] The processor may be a general-purpose processor, and the general-purpose processor may call the erasure code storage space utilization improvement program stored in the memory, and execute the erasure code storage space utilization improvement method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the erasure code storage space utilization improvement program is called may refer to the various embodiments of the erasure code storage space utilization improvement method of the present application, and will not be repeated here.

[0118] Those skilled in the art will understand that Fig.10 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0119] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0120] The computer-readable storage medium of the present application stores a program for improving storage space utilization of erasure codes, wherein when the program for improving storage space utilization of erasure codes is executed by a processor, the steps of the method for improving storage space utilization of erasure codes as described above are implemented.

[0121] Among them, the method implemented when the erasure code storage space utilization improvement program is executed can refer to the various embodiments of the erasure code storage space utilization improvement method of the present application, and will not be repeated here.

[0122] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0124] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for improving the utilization rate of erasure code storage space, characterized in that: The method for improving the utilization rate of erasure code storage space comprises: Divide the check shard into a replica data space and a check data space, wherein the replica data space stores only the replica data of the unfull stripe, and the check data space stores only the check data of the full stripe; Based on the size of the replica data or existing data in the stripe, with the chunk size as the threshold, background aggregation or data recovery is performed on the verification shard to convert the replica data to the verification data. Based on the size of the replica data, with the chunk size as the threshold, background aggregation is performed on the verification shard to convert the replica data into verification data, including: When the total size of the replica data in the stripe on the check shard is less than or equal to the chunk size, and there is no corresponding check data for the stripe, no operation is performed on the stripe; When the total size of the replica data in the stripe on the check shard is greater than the chunk size and there is no corresponding check data for the stripe, the replica data is aggregated into check data by defaulting to 0 for the empty segment of the stripe, and the replica data of the stripe in the replica data space is deleted; When there is new replica data in the stripe on the check shard, and the stripe has corresponding check data, the stripe hole segment is treated as 0 by default, the check data is updated based on the new replica data, and all replica data of the stripe in the replica data space are deleted; Based on the size of the existing data in the stripe, with the chunk size as the threshold, data recovery is performed on the verification shard, converting the verification data into replica data, including: When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is greater than the chunk size, and the stripe has corresponding check data, the check data is not restored to the replica data, but the check data is updated, and when calculating the new check data, the deleted replica data segment is processed as 0 by default; When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is less than or equal to chunksize, and the stripe has corresponding check data, the check data is restored to the replica data and the corresponding check data is deleted.

2. The method for improving the utilization rate of erasure code storage space according to claim 1, characterized in that: Divide the parity shard into replica data space and parity data space of equal size.

3. A device for improving the utilization rate of erasure code storage space, characterized in that: The device for improving the utilization rate of erasure code storage space comprises: A partitioning module, which is used to divide the check slice into a replica data space and a check data space, wherein the replica data space only stores the replica data of the unfull stripe, and the check data space only stores the check data of the full stripe; The data conversion module is used to perform background aggregation or data recovery on the verification shard based on the size of the replica data or the existing data of the stripe, taking the chunk size as the threshold, so as to convert the replica data into the verification data; The data conversion module performs background aggregation on the verification slice based on the size of the replica data and takes the chunk size as a threshold, and converts the replica data into verification data, including: When the total size of the replica data in the stripe on the check shard is less than or equal to the chunk size, and there is no corresponding check data for the stripe, no operation is performed on the stripe; When the total size of the replica data in the stripe on the check shard is greater than the chunk size and there is no corresponding check data for the stripe, the replica data is aggregated into check data by defaulting to 0 for the empty segment of the stripe, and the replica data of the stripe in the replica data space is deleted; When there is new replica data in the stripe on the check shard, and the stripe has corresponding check data, the stripe hole segment is treated as 0 by default, the check data is updated based on the new replica data, and all replica data of the stripe in the replica data space are deleted; The data conversion module performs data recovery on the check slice based on the size of the existing data in the stripe and takes the chunk size as a threshold, and converts the check data into the copy data, including: When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is greater than the chunk size, and the stripe has corresponding check data, the check data is not restored to the replica data, but the check data is updated, and when calculating the new check data, the deleted replica data segment is processed as 0 by default; When there is deleted replica data in the stripe on the check shard, the total size of the existing data in the stripe is less than or equal to chunksize, and the stripe has corresponding check data, the check data is restored to the replica data and the corresponding check data is deleted.

4. The device for improving storage space utilization of erasure codes according to claim 3, characterized in that: The partitioning module is used to partition the check slice into a replica data space and a check data space of equal size.

5. A device for improving the utilization of erasure code storage space, characterized in that: The device for improving the storage space utilization of erasure codes includes a processor, a memory, and a program for improving the storage space utilization of erasure codes stored in the memory and executable by the processor. When the program for improving the storage space utilization of erasure codes is executed by the processor, the steps of the method for improving the storage space utilization of erasure codes as described in claim 1 or 2 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for improving storage space utilization of erasure codes, wherein when the program for improving storage space utilization of erasure codes is executed by a processor, the steps of the method for improving storage space utilization of erasure codes as described in claim 1 or 2 are implemented.

Images