A storage redundancy conversion method and system

Abstract

The application discloses a storage redundancy conversion method and system, which comprises the following steps: S1, reading and selecting a strip to be subjected to redundancy conversion; S2, sorting the strips according to a deletion data ratio, and selecting the first X strips, and sorting the strips according to generation time; S3, selecting Y strips to be processed as strip conversion of a current batch; S4, deleting original check blocks; and S5, re-writing the remaining data blocks and the next batch of strips into data blocks according to N+M. The technical scheme adopted by the application can reduce disk copying, reduce cross-node bandwidth, and quickly recycle the deleted data, so as to reduce the influence on cluster services, reduce system pressure, and quickly complete the storage redundancy conversion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network storage, in particular to a storage redundancy conversion method and system. BACKGROUND

[0002] The most important technical direction in the information age is mainly based on the continuous development and progress of storage, computing and transmission. With the popularization of technology, a large amount of data files has been generated on the Internet. For a large amount of data files, in the field of storage, the current distributed storage scheme is basically adopted to meet the mass storage and access, and is also convenient for expansion.

[0003] In order to improve the utilization rate of storage space or increase the reliability level when the distributed storage is expanded or in other scenarios, the data redundancy ratio of the system needs to be changed, such as the redundancy ratio of the data and the check data of the error correction code. The data needs to be distributed as evenly as possible to each data node, and the more space needs to be released as soon as possible.

[0004] In the prior art, such as the invention name "Cross-rack-aware error correction code storage redundancy conversion method balanced redundancy conversion method and device" (application number 202310648438.X), the data layout between racks is balanced and the check block layout within the rack is aggregated, and the layout is maintained during the redundancy conversion process to reduce the cross-rack transmission flow caused by data block relocation and check block update. However, such a scheme has the following technical defects:

[0005] 1. All check blocks of the strip are aggregated in the same check rack, but in actual distributed storage, they are usually discretely distributed in different racks (nodes), and there is a problem of uneven flow (such as read request only needs to read data rack, while check rack is completely idle and cannot provide performance).

[0006] 2. It does not consider the case of deleting data in strip data. In practical storage scenarios, there is no deleted data in very ideal conditions, which is not consistent with the conventional scenario.

[0007] 3. Although the internal traffic of the rack has little effect, the disk copying still occupies a considerable amount of time. SUMMARY

[0008] To solve the technical problems existing in the prior art, the present application proposes a storage redundancy conversion method, which reduces disk copying, cross-node bandwidth and fast data recycling of deleted data as much as possible to minimize the impact on cluster business, reduce system pressure and quickly complete the storage redundancy conversion.

[0009] To achieve the above purpose, the technical scheme of the present application is: a storage redundancy conversion method, comprising the steps of:

[0010] S1: reading and selecting the stripe to be converted for redundancy;

[0011] S2: sorting the stripe according to the proportion of deleted data, and selecting the first X stripe, and sorting the stripe according to the generation time;

[0012] S3: selecting Y stripes for processing as the stripe conversion of the current batch;

[0013] S4: deleting the original check block;

[0014] S5: re-writing the remaining data block and the next batch of stripe according to N+M data block.

[0015] Further, in S3: selecting Y stripes for processing, the processing of the complete data block without deleted data includes:

[0016] selecting the complete data block without deletion of each host to make up N data blocks;

[0017] generating M check blocks to complete the conversion of a stripe.

[0018] Further, in S3: selecting Y stripes for processing, the processing of the data block with deleted data includes:

[0019] moving the data in the data block that is not deleted;

[0020] making up N data blocks for the data that is not deleted;

[0021] generating new M check blocks.

[0022] Further, in S5: re-writing the remaining data block and the next batch of stripe according to N+M data block, for the data block of the local machine, it is written into the local machine.

[0023] The application also provides a storage redundancy conversion system for implementing the storage redundancy conversion method, comprising a read-write unit, a sorting unit, and a conversion processing unit, wherein:

[0024] The read-write unit is used to read and select the stripe to be converted for redundancy, and re-write the remaining data block and the next batch of stripe according to N+M data block;

[0025] The sorting unit is used to sort the stripe according to the proportion of deleted data, and select the first X stripe, and sort the stripe according to the generation time;

[0026] The conversion processing unit is used to select Y stripes for processing as the stripe conversion of the current batch and delete the original check block after the conversion is completed.

[0027] Further, the conversion processing unit includes a complete data block group for processing complete data blocks without deleted data, including:

[0028] Selecting each host complete data block without deletion to make up N data blocks;

[0029] Generating M check blocks to complete the conversion of a stripe.

[0030] Further, the conversion processing unit includes a data block group to be recycled for processing data blocks with deleted data, including:

[0031] Moving the data in the data block that is not deleted;

[0032] Making up N data blocks for the data that is not deleted;

[0033] Generating new M check blocks.

[0034] Further, the read-write unit rewrites the remaining data blocks and the next batch of stripes in N+M data blocks, and for the local data blocks, it is written in the local machine.

[0035] The application also provides a storage medium, which stores a computer program, and the computer program is executed by a processor to perform the storage redundancy conversion method.

[0036] The application also provides an electronic device, which includes a processor, a storage medium, and a bus, the storage medium stores machine-readable instructions executable by the processor, the processor communicates with the storage medium through the bus when the electronic device is running, and the processor executes the machine-readable instructions to perform the storage redundancy conversion method.

[0037] The modules in the method and system disclosed in the application can be deployed on a target server, or each module can be independently deployed on a different target server. In particular, to provide more powerful computing processing capability, the modules can be deployed on a cluster target server as needed.

[0038] Therefore, the technical solution adopted by the application reduces disk copying, reduces cross-node bandwidth, and uses a fast data recycling method for deleted data to minimize the impact on cluster services, reduce system pressure, and quickly complete the storage redundancy conversion method, achieving the following technical effects:

[0039] 1. First, sort the stripes according to the proportion of deleted data, that is, sort the stripes according to the proportion of deleted data in each stripe, which can release the storage space with large deletion amount first.

[0040] 2. Selecting the proportion of deleted data to select the strip, so that the converted strip enters the data block proportion of the recycling process, and the data in the data block with deleted data and the data block with deleted data are more likely to complete a strip conversion as soon as possible.

[0041] 3. The application generates time-approximate strips, so that the data of the converted strip is close in position on the disk, which can greatly reduce the movement of the original data block on the disk (that is, it can reduce the overhead of reading and writing moving data to speed up the conversion process), and avoid the performance decline of the cluster due to the dispersion of the data block.

[0042] In order to have a more comprehensive understanding of the present application, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0044] Figure 1 A processing schematic diagram of a storage redundancy conversion method of an embodiment of the present application.

[0045] Figure 2 A schematic diagram of the strip obtained by sorting in an embodiment of the present application.

[0046] Figure 3 A schematic diagram obtained by time sorting in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In a storage system, a plurality of strips of different strip depths are included, each strip includes N data strip units (data blocks) and M check strip units (check blocks), the to-be-written data is divided into N data strip units, M check strip units are calculated according to the error correction code algorithm, and N data strip units and M check strip units are written into a determined strip.

[0048] For distributed storage, in order to improve storage space utilization or increase reliability level when expanding or in other scenarios, the data redundancy ratio of the system needs to be changed, for example, the redundancy ratio of the data and the check data of the error correction code, and the change process needs to distribute the data as evenly as possible to each data node, and needs to be completed as soon as possible to release more space faster.

[0049] The application provides an improved technical solution, which reduces the impact on cluster services, reduces system pressure and quickly completes storage redundancy conversion by reducing disk copying, reducing cross-node bandwidth and deleting data in a fast data recycling mode.

[0050] S1: reading and selecting a strip to be subjected to redundancy conversion;

[0051] S2: sorting the strips according to a data deletion ratio, and selecting the first X strips, and sorting the strips according to generation time;

[0052] S3: selecting Y strips for processing as the strip conversion of the current batch;

[0053] S4: deleting the original check block;

[0054] S5: re-writing the remaining data blocks and the next batch of strips according to N+M data blocks.

[0055] Please refer to Figure 1 , and the technical solution of the application will be described in detail below with reference to Figure 1 as an embodiment.

[0056] S1: reading and selecting a strip to be subjected to redundancy conversion;

[0057] In an actual storage system, according to the existing storage system, the storage resources are read and selected for redundancy conversion according to the storage resources or computing pressure of the existing system, so as to optimize the storage system, thereby improving the utilization rate of the storage space.

[0058] The application reads the original data and check data during strip conversion, so that the data recycling process can be integrated into the conversion process, reducing repeated data reading and writing, and accelerating the release of the deleted space during the strip conversion process.

[0059] S2: sorting the strips according to a data deletion ratio, and selecting the first X strips, and sorting the strips according to generation time;

[0060] Please refer to Figure 2 for a schematic diagram, taking an original storage node as 6 nodes, and n+m as the error correction code, when the node is expanded by two, it becomes 8 nodes, and the error correction code needs to be changed to N+M.

[0061] For different stripes in the storage system, in the process of frequent read and write access, because the data or information stored in each stripe is not completely the same, and the deletion time is also different, which brings great consumption to the access of data. In order to further optimize, the application selects the stripe to be converted first according to the proportion of deleted data, that is, according to the proportion of deleted data in each stripe, so that the storage space with large deletion amount can be released preferentially.

[0062] As can be seen, the proportion of deleted data is selected for stripe selection, so that the proportion of data blocks of the converted stripe entering the recycling process is approximately, and the data in the data blocks with deleted data and the data blocks without deleted data are more likely to complete a stripe conversion as soon as possible.

[0063] However, it is not enough to sort only according to the proportion of deleted data. Because the storage of data also has certain correlation in the time dimension, further optimization is needed in the time dimension on the basis of sorting according to the proportion of deleted data, that is, after sorting according to the proportion of deleted data, X stripes are selected for conversion, and then the stripes are sorted according to the generation time, that is, sorted according to the time sequence of stripe writing.

[0064] As shown in Figure 2 , first, the stripes are sorted according to the proportion of deleted data, and the stripe groups D1, D2...D6, P1, P2 in order are obtained.

[0065] X stripes are selected from the stripe groups D1, D2...D6, P1, P2, for example, n+m redundancy stripes with data deletion proportion of 0% to 5%, and then sorted according to the time sequence of stripe writing, because time similar stripes are combined together, which is more time local and friendly to data distribution, Y stripes are selected from X stripes (X≥Y), for example Figure 3 As shown in , three stripes d1 d2 d3 d4 p1 p2, d5 d6 d7 d8 p3 p4, and d9 d10 d11 d12 p5 p6 are selected.

[0066] By selecting stripes with similar generation time, the data of the converted stripes is close in position on the disk, which can greatly reduce the movement of the original data blocks on the disk (that is, the overhead of reading and writing moving data can be reduced to speed up the conversion process), and avoids the performance decline of the cluster due to the dispersion of data blocks.

[0067] S3: Select Y stripes for processing as the current batch of stripe conversion.

[0068] To further optimize, when the selected Y stripes are processed, the present application needs to consider two cases, one is the complete data block without deleted data, and the other is the data block with deleted data.

[0069] Processing the complete data block without deleted data includes:

[0070] Selecting the complete data block without deleted data of each host to make up N data blocks;

[0071] To change it to N+M stripes, because N+M is distributed on multiple nodes to ensure that data can be repaired through other node data and check data after node failure. At the same time, to speed up the redundancy conversion process, when selecting data blocks, the data blocks distributed in this node are preferentially selected. For example, D1 selects d1, D2 selects d2, D3 selects d6, D4 selects d4, D5 selects d11, D6 selects d12, D1D2 D3 D4 D5 D6 P1 P2 then form a new N+M stripe.

[0072] After making up N data blocks, M check blocks are generated to complete the conversion of a stripe.

[0073] Processing the data block with deleted data, for example Figure 2 d3, d8, d3, d8 in the above are not deleted data, which are moved to the to-be-recovered data block group, and wait for other to-be-recovered data block groups to enter the to-be-recovered data block process. After the to-be-recovered data blocks make up N data blocks, N+M stripes are calculated and generated to complete the conversion of a stripe.

[0074] As described above, the conversion of the current batch of Y stripes is completed.

[0075] S4: deleting the original check block to release the storage space.

[0076] S5: re-writing the remaining data blocks and the next batch of stripes according to N+M into data blocks. For the data blocks of the host, they are written into the host to reduce data migration and reduce frequent read and write of data.

[0077] Based on the above method embodiment, the present application further provides a storage redundancy conversion system for implementing the storage redundancy conversion method, including a read-write unit, a sorting unit, and a conversion processing unit, wherein: the read-write unit is used to read and select the stripes to be subjected to redundancy conversion and re-write the remaining data blocks and the next batch of stripes according to N+M into data blocks;

[0078] The sorting unit is used to sort the stripes according to the proportion of deleted data and select the first X stripes, and sort the stripes according to the generation time;

[0079] The conversion processing unit is configured to select Y pieces of data blocks for processing as a current batch of stripe conversion and delete the original parity blocks after the conversion is completed.

[0080] Further, the conversion processing unit includes a complete data block group configured to process complete data blocks without deleted data, including:

[0081] selecting complete data blocks without deletion for each host to make up N data blocks;

[0082] generating M parity blocks to complete the conversion of a stripe.

[0083] Further, the conversion processing unit includes a to-be-recycled data block group configured to process data blocks with deleted data, including:

[0084] moving the data in the data blocks that are not deleted;

[0085] making up N data blocks from the data that are not deleted;

[0086] generating new M parity blocks.

[0087] Further, the read-write unit rewrites the remaining data blocks and the next batch of stripes into N+M data blocks, and for the data blocks of the host, the data blocks are written into the host.

[0088] The embodiments of the present application also provide a storage medium, and the storage medium stores a computer program. When the computer program is run by a processor, the storage redundancy conversion method is executed.

[0089] The embodiments of the present application also provide an electronic device, which includes a processor, a storage medium and a bus. The storage medium stores machine readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus. The processor executes the machine readable instructions to execute the storage redundancy conversion method.

[0090] It should be noted that all or part of the steps of the various methods of the above embodiments can be split by a computer program to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium, which can include but is not limited to a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0091] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A storage redundancy conversion method, comprising the steps of: S1: reading and selecting a stripe to be subjected to redundancy conversion; S2: sorting the stripe according to a deletion data proportion, and selecting a first X stripe according to generation time; S3: selecting Y stripes for processing as a current batch of stripe conversion; S4: deleting original check blocks; S5: The remaining data blocks are re-written in the data blocks according to N+M with the next batch of stripes, wherein, N is the number of data blocks in a stripe, M is the number of check blocks in a stripe, and N+M is the total number of erasure code stripes.

2. The storage redundancy conversion method of claim 1, wherein S3 In the step of selecting Y stripes for processing, the processing of complete data blocks without deletion data includes: selecting complete data blocks without deletion of each host to make up N data blocks; generating M check blocks to complete the conversion of a stripe.

3. The storage redundancy conversion method of claim 1, wherein S3 In the step of selecting Y stripes for processing, the processing of data blocks with deletion data includes: moving data in the data block that is not deleted; making up N data blocks from the data that is not deleted; generating new M check blocks.

4. The storage redundancy conversion method of claim 1, wherein S5: re-writing the remaining data blocks and the next batch of stripes into data blocks according to N+M, and for the data blocks of the host, writing into the host.

5. A storage redundancy conversion system for implementing the storage redundancy conversion method according to any one of claims 1 to 4, characterized by It comprises a read-write unit, a sorting unit, and a conversion processing unit, wherein: the read-write unit is used to read and select a stripe to be subjected to redundancy conversion, and re-write the remaining data blocks and the next batch of stripes into data blocks according to N+M, wherein N is the number of data blocks in a stripe, M is the number of check blocks in a stripe, and N+M is the total number of erasure code stripes; the sorting unit is used to sort the stripe according to a deletion data proportion, and select a first X stripe according to generation time; the conversion processing unit is used to select Y stripes for processing as a current batch of stripe conversion and delete original check blocks after the conversion is completed.

6. The storage redundancy conversion system of claim 5, wherein, The conversion processing unit includes a complete data block group, which is used to process complete data blocks without deletion data, including: selecting complete data blocks without deletion of each host to make up N data blocks; generating M check blocks to complete the conversion of a stripe.

7. The storage redundancy conversion system of claim 5, wherein, The conversion processing unit includes a to-be-recycled data block group, which is used to process data blocks with deletion data, including: moving data in the data block that is not deleted; making up N data blocks from the data that is not deleted; generating new M check blocks.

8. The storage redundancy conversion system of claim 5, wherein, The read-write unit re-writes the remaining data blocks and the next batch of stripes into data blocks according to N+M, and for the data blocks of the host, writes into the host.

9. A storage medium, characterized by The storage medium stores a computer program, which is executed by the processor to perform the storage redundancy conversion method according to any one of claims 1 to 4.

10. An electronic device, comprising: It comprises: a processor, a storage medium, and a bus, the storage medium stores machine-readable instructions executable by the processor, the processor and the storage medium communicate through the bus when the electronic device is running, and the processor executes the machine-readable instructions to perform the storage redundancy conversion method according to any one of claims 1 to 4.

Images