Storage redundancy conversion method and system
By sorting and processing stripes in distributed storage, optimizing data block distribution and verification block generation, the problems of unconsidered traffic imbalance and data deletion are solved, fast redundant conversion and low system pressure are achieved, and storage is improved Efficiency and reliability.
Patent Information
- Application Number
- CN202411060888.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-05
AI Technical Summary
The prior art has problems such as unbalanced traffic, failure to consider data deletion and long disk copying time in distributed storage, resulting in low storage redundancy conversion efficiency and high system pressure.
By reading and sorting strips, selecting strips with high data ratio for processing, generating new verification blocks, reducing disk copy and cross-node bandwidth, using strips with similar generation times for conversion, optimizing data block distribution to reduce data blocks Mobile overhead.
It realizes the rapid completion of storage redundancy conversion, reduces the impact on cluster services, reduces system pressure, and improves storage space utilization and reliability.
Smart Images

Figure CN119025033A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network storage technology, and in particular to a storage redundancy conversion method and system. Background Art
[0002] The most important technical directions in the information age are mainly based on the continuous development and progress of storage, computing and transmission. With the popularization of technology, a huge amount of data files are generated on the Internet. For the massive data files, in the field of storage, the current distributed storage solution is basically adopted to meet the massive storage and access requirements, which is also easy to expand.
[0003] In order to improve storage space utilization or increase reliability levels during expansion or other scenarios, distributed storage needs to change the system's data redundancy ratio, such as the redundancy ratio of erasure coded data and checksum data. The data needs to be distributed as evenly as possible to each data node, and this needs to be done as quickly as possible to free up more space.
[0004] In the prior art, for example, the invention named "Cross-rack-aware erasure code storage redundancy conversion method and balanced redundancy conversion method and device" (application number 202310648438.X) formulates a balanced data layout between racks and an aggregated check block layout within a rack, and maintains the layout during the redundant conversion process to reduce the cross-rack transmission traffic caused by data block relocation and check block update. However, this solution has the following technical defects:
[0005] 1. The check blocks of all stripes are aggregated in the same check rack. However, in actual distributed storage, they are usually discretely distributed in different racks (nodes), resulting in traffic imbalance (for example, a read request only needs to read the data rack, while the check rack is completely idle and cannot provide performance).
[0006] 2. The situation where stripe data has deleted data is not considered. In practical storage scenarios, it is an ideal situation that data is not deleted, which is inconsistent with the normal scenario.
[0007] 3. Although the impact of intra-rack traffic is small, disk copying still takes a considerable amount of time. Summary of the invention
[0008] In order to solve the technical problems existing in the prior art, the present application proposes a storage redundancy conversion method, which minimizes the impact on cluster services, reduces system pressure, and quickly completes storage redundancy conversion by minimizing disk copies, cross-node bandwidth, and a fast data recovery method for deleted data.
[0009] To achieve the above purpose, the technical solution of the present application is: a storage redundancy conversion method, comprising the steps of:
[0010] S1: read and select the stripe to be converted into redundancy;
[0011] S2: Sort the stripes by the ratio of deleted data, select the first X stripes, and sort the stripes by generation time;
[0012] S3: select Y strips to be processed as the strip conversion of the current batch;
[0013] S4: Delete the original check block;
[0014] S5: re-write the remaining data blocks and the next batch of stripes into data blocks according to N+M.
[0015] Further, S3: selecting Y of the data blocks for processing, processing the complete data blocks without deleted data includes:
[0016] Select the complete data blocks that have not been deleted from each host to make up N data blocks;
[0017] Generate M check blocks to complete the conversion of one stripe.
[0018] Further, S3: selecting Y of the data blocks for processing, processing the data blocks with deleted data includes:
[0019] Move the undeleted data in the data block;
[0020] Gather N data blocks for the data that has not been deleted;
[0021] Generate M new check blocks.
[0022] Further, S5: the remaining data blocks and the stripes of the next batch are re-written into the data blocks according to N+M, and the data blocks of the local machine are written into the local machine.
[0023] The present invention also provides a storage redundancy conversion system for implementing the storage redundancy conversion method, comprising a read-write unit, a sorting unit, and a conversion processing unit, wherein:
[0024] The read / write unit is used to read and select the stripes to be converted to redundancy and rewrite the remaining data blocks and the next batch of stripes into data blocks according to N+M;
[0025] The sorting unit is used to sort the stripes according to the ratio of deleted data, select the first X stripes, and sort the stripes according to the generation time;
[0026] The conversion processing unit is used to select Y strips for processing as strip conversion of the current batch and delete the original check blocks after the conversion is completed.
[0027] Furthermore, the conversion processing unit includes a complete data block group, and the processing of the complete data block without deleted data includes:
[0028] Select the complete data blocks that have not been deleted from each host to make up N data blocks;
[0029] Generate M check blocks to complete the conversion of one stripe.
[0030] Furthermore, the conversion processing unit includes a to-be-reclaimed data block group, and the processing of the data blocks with deleted data includes:
[0031] Move the undeleted data in the data block;
[0032] Gather N data blocks for the data that has not been deleted;
[0033] Generate M new check blocks.
[0034] Furthermore, the read-write unit rewrites the remaining data blocks and the stripes of the next batch into the data blocks according to N+M, and writes the data blocks of the local machine into the local machine.
[0035] The present invention also provides a storage medium, on which a computer program is stored, and when the computer program is run by a processor, the storage redundancy conversion method is executed.
[0036] The present invention also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the storage redundancy conversion method.
[0037] In actual applications, the modules described in the method and system disclosed in the present application can be deployed in multiple modules on one target server, or each module can be independently deployed on different target servers. In particular, as needed, in order to provide more powerful computing and processing capabilities, the modules can also be deployed on cluster target servers as needed.
[0038] It can be seen that the technical solution adopted by this application achieves the following technical effects by minimizing the impact on cluster services, reducing system pressure, and quickly completing storage redundancy conversion by minimizing disk copies, reducing cross-node bandwidth, and quickly recovering deleted data:
[0039] 1. First, the stripes are sorted according to the proportion of deleted data, that is, they are sorted according to the redundancy ratio of the deleted data in each stripe, so that the storage space with a large amount of deleted data can be released first.
[0040] 2. Select the ratio of deleted data for stripe selection, so that the ratio of data blocks entering the recovery process of the converted stripe is similar, and the data blocks without deleted data and the undeleted data in the data blocks with deleted data are more likely to complete a stripe conversion as soon as possible.
[0041] 3. This application selects stripes with similar generation time so that the data of the converted stripes are close to each other on the disk, which can greatly reduce the location of the original data blocks on the disk (that is, it can reduce the overhead of reading and writing mobile data and speed up the conversion process), avoiding the excessive dispersion of data caused by moving data blocks, which may lead to a decrease in the performance of the cluster in the later stage.
[0042] In order to have a clearer and more comprehensive understanding of the present application, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.
[0044] Figure 1 A processing diagram of a storage redundancy conversion method according to an embodiment of the present application.
[0045] Figure 2 This is a schematic diagram of the strips obtained by sorting according to an embodiment of the present application.
[0046] Figure 3 This is a schematic diagram obtained by time sorting of the embodiments of the present application. DETAILED DESCRIPTION
[0047] In the storage system, there are multiple stripes with different stripe depths, each stripe contains N data stripe units (data blocks) and M check stripe units (check blocks). The data to be written is divided into N data stripe units, and M check stripe units are calculated according to the erasure code algorithm, and the N data stripe units and M check stripe units are written to the determined stripe.
[0048] For distributed storage, in order to improve storage space utilization or increase reliability levels during expansion or other scenarios, it is necessary to change the system's data redundancy ratio, such as the redundancy ratio of erasure coded data and checksum data. During the change process, the data needs to be distributed as evenly as possible to each data node, and it needs to be completed as quickly as possible to release more space faster.
[0049] The present application proposes an improved technical solution, which minimizes the impact on cluster services, reduces system pressure, and quickly completes storage redundancy conversion by minimizing disk copies, reducing cross-node bandwidth, and deleting data in a fast data recovery manner. The solution is: a storage redundancy conversion method, comprising the steps of:
[0050] S1: read and select the stripe to be converted into redundancy;
[0051] S2: Sort the stripes by the ratio of deleted data, select the first X stripes, and sort the stripes by generation time;
[0052] S3: select Y strips to be processed as the strip conversion of the current batch;
[0053] S4: Delete the original check block;
[0054] S5: Write the remaining data blocks and the next batch of stripes into the data blocks according to N+M.
[0055] See also Figure 1 , the following combination Figure 1 The following is an example to illustrate the technical solution of the present application in detail.
[0056] S1: Read and select the stripe to be converted for redundancy.
[0057] In an actual storage system, based on the existing storage system, the storage resources are judged according to the storage resources or computing pressure of the existing system, and the stripes to be converted for redundancy are read and selected to optimize the storage system, thereby improving the utilization of storage space.
[0058] This application will read the original data and verification data during stripe conversion, so that the data recovery process can be integrated into the conversion process, reducing repeated data reading and writing, and also speeding up the release of deleted space during the stripe conversion process.
[0059] S2: Sort the stripes by the ratio of deleted data, select the first X stripes, and sort the stripes by generation time.
[0060] See also Figure 2 In the schematic diagram shown, an original storage node is 6 nodes, and the erasure code is n+m. When the node is expanded by two, it becomes 8 nodes, and the erasure code needs to be changed to N+M.
[0061] For different stripes in the storage system, during frequent read and write accesses, since the data or information sources stored in each stripe are not exactly the same and the deletion time is also different, this brings a very large consumption to the data access. For further optimization, this application selects the stripes to be converted. First, the stripes are sorted according to the proportion of deleted data, that is, they are sorted according to the redundancy ratio of the deleted data in each stripe, so that the storage space with a large amount of deletions can be released first.
[0062] From this, we can see that by selecting the deleted data ratio for stripe selection, the ratio of data blocks entering the recovery process of the converted stripe is similar, and the data blocks without deleted data and the undeleted data in the data blocks with deleted data are more likely to complete a stripe conversion as soon as possible.
[0063] However, it is not enough to just sort according to the redundancy ratio of deleted data. Since the data storage also has a certain correlation in the time dimension, it is necessary to further optimize from the time dimension based on the sorting of the redundancy ratio of deleted data. That is, after sorting according to the redundancy ratio of deleted data, select X stripes that need to be converted, and then sort the stripes according to the generation time, that is, sort according to the time sequence of stripe writing.
[0064] like Figure 2 As shown, the stripes are first sorted according to the redundant ratio of deleted data to obtain a stripe group in the order of D1, D2...D6, P1, P2.
[0065] Select X stripes from the stripe group of D1, D2...D6, P1, P2, for example, delete n+m redundant stripes with data accounting for 0% to 5%, and then sort them according to the time sequence of stripe writing, because stripes with similar time are merged together to have more time locality and are more friendly to data distribution. Select Y stripes (X≥Y) from X stripes, for example Figure 3 As shown in the figure, three stripes are selected: stripe one (d1 d2 d3 d4 p1 p2), stripe two (d5 d6 d7 d8 p3 p4), and stripe three (d9 d10 d11 d12p5 p6).
[0066] The present application selects stripes with similar generation time, so that the data of the converted stripes are close to each other on the disk, which can greatly reduce the location of the original data blocks on the disk (that is, it can reduce the overhead of reading and writing mobile data and speed up the conversion process), avoiding the excessive dispersion of data caused by moving data blocks, which may lead to a decrease in the performance of the cluster in the later stage.
[0067] S3: Select Y strips for processing as the strip conversion of the current batch.
[0068] For further optimization, when selecting Y stripes for processing, the present application needs to consider two situations, one is a complete data block without deleted data, and the other is a data block with deleted data.
[0069] Processing of complete data blocks without deleted data includes:
[0070] Select the complete data blocks that have not been deleted from each host and collect N data blocks;
[0071] It needs to be converted into an N+M stripe, because N+M needs to be distributed on multiple nodes to ensure that data can be repaired through other node data and check data after a node fails. At the same time, to speed up the redundancy conversion process, when selecting data blocks, the data blocks to be distributed on this node should give priority to the data blocks of this host (node). For example, D1 selects d1, D2 selects d2, D3 selects d6, D4 selects d4, D5 selects d11, D6 selects d12, and D1D2 D3 D4 D5 D6 P1 P2 form a new N+M stripe.
[0072] After gathering N data blocks, M check blocks are generated to complete the conversion of one stripe.
[0073] Process the data blocks with deleted data, such as Figure 2 Among them, d3, d8, d3, and d8 are data that have not been deleted. Move d3 and d8 to the data block group to be recycled, and wait for other data block groups to be recycled to enter the data block process to be recycled. After the data blocks to be recycled have N data blocks, calculate and generate N+M stripes to complete a stripe conversion.
[0074] According to the above process, the conversion of the Y strips in the current batch is completed.
[0075] S4: Delete the original check block to release storage space.
[0076] S5: The remaining data blocks and the next batch of stripes are rewritten into data blocks according to N+M. For the local data blocks, they are written into the local machine, which reduces the migration of data and reduces the frequent reading and writing of data.
[0077] Based on the above method embodiment, the present application also provides a storage redundancy conversion system for implementing the storage redundancy conversion method, including a read-write unit, a sorting unit, and a conversion processing unit, wherein: the read-write unit is used to read and select the stripes to be converted into redundancy and re-write the remaining data blocks and the next batch of stripes into the data blocks according to N+M;
[0078] The sorting unit is used to sort the stripes according to the ratio of deleted data, select the first X stripes, and sort the stripes according to the generation time;
[0079] The conversion processing unit is used to select Y strips for processing as strip conversion of the current batch and delete the original check blocks after the conversion is completed.
[0080] Furthermore, the conversion processing unit includes a complete data block group, and the processing of the complete data block without deleted data includes:
[0081] Select the complete data blocks that have not been deleted from each host to make up N data blocks;
[0082] Generate M check blocks to complete the conversion of one stripe.
[0083] Furthermore, the conversion processing unit includes a to-be-reclaimed data block group, and the processing of the data blocks with deleted data includes:
[0084] Move the undeleted data in the data block;
[0085] Gather N data blocks for the data that has not been deleted;
[0086] Generate M new check blocks.
[0087] Furthermore, the read-write unit rewrites the remaining data blocks and the stripes of the next batch into the data blocks according to N+M, and writes the data blocks of the local machine into the local machine.
[0088] An embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the storage redundancy conversion method as described above is executed.
[0089] An embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform any of the storage redundancy conversion methods described above.
[0090] It should be noted that a person skilled in the art can understand that all or part of the split steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0091] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A storage redundancy conversion method, characterized in that it comprises the steps of: S1: read and select the stripe to be converted into redundancy; S2: Sort the stripes by the ratio of deleted data, select the first X stripes, and sort the stripes by generation time; S3: select Y strips to be processed as the strip conversion of the current batch; S4: Delete the original check block; S5: re-write the remaining data blocks and the next batch of stripes into data blocks according to N+M.
2. The storage redundancy conversion method according to claim 1, characterized in that S3 : Select Y of them for processing, and process the complete data blocks without deleted data including: Select the complete data blocks that have not been deleted from each host to make up N data blocks; Generate M check blocks to complete the conversion of one stripe.
3. The storage redundancy conversion method according to claim 1, characterized in that S3 : Select Y of them for processing, and process the data blocks with deleted data including: Move the undeleted data in the data block; Gather N data blocks for the data that has not been deleted; Generate M new check blocks.
4. The storage redundancy conversion method according to claim 1, characterized in that: S5: The remaining data blocks and the next batch of stripes are re-written into the data blocks according to N+M. For the data blocks of the local machine, they are written into the local machine.
5. A storage redundancy conversion system, used to implement the storage redundancy conversion method according to any one of claims 1 to 5, characterized in that: It includes a reading and writing unit, a sorting unit, and a conversion processing unit, among which: The read / write unit is used to read and select the stripes to be converted to redundancy and rewrite the remaining data blocks and the next batch of stripes into data blocks according to N+M; The sorting unit is used to sort the stripes according to the ratio of deleted data, select the first X stripes, and sort the stripes according to the generation time; The conversion processing unit is used to select Y strips for processing as strip conversion of the current batch and delete the original check blocks after the conversion is completed.
6. The storage redundancy conversion system according to claim 5, characterized in that: The conversion processing unit includes a complete data block group, and the processing of the complete data block without deleted data includes: Select the complete data blocks that have not been deleted from each host to make up N data blocks; Generate M check blocks to complete the conversion of one stripe.
7. The storage redundancy conversion system according to claim 5, characterized in that: The conversion processing unit includes a data block group to be recovered, and is used to process the data blocks with deleted data, including: Move the undeleted data in the data block; Gather N data blocks for the data that has not been deleted; Generate M new check blocks.
8. The storage redundancy conversion system according to claim 5, characterized in that: The read-write unit rewrites the remaining data blocks and the next batch of stripes into the data blocks according to N+M, and writes the local data blocks into the local computer.
9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the storage redundancy conversion method according to any one of claims 1 to 4 is executed.
10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the storage redundancy conversion method as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Cross-rack sensing erasure code storage system balance redundancy conversion method and device
CN116909475A
Storage system condition indicator and method
CN101095116A
Local repairable code redundancy conversion method based on stripe merging and storage medium
CN114564335A
Storage system garbage collection method and device
CN116880771A
Disk array device
JP2001051806A
