Efficient generation method for LRC (Local Repair Code) strip

Abstract

The invention discloses an efficient generation method for a local repair code strip, and mainly solves the technical problems that a network transmission and storage persistence bottleneck is formed due to global check block concentration and the network parallelism degree is limited by data block concentration in the prior art. The scheme comprises the following steps: 1) initializing a generation matrix and a logic layout; 2) dispersing global check blocks which are distributed in a centralized manner in an initial state to each local group in a stripe, and reconstructing the local groups to balance the distribution of the check blocks; 3) updating a generation matrix based on the reconstructed local group and calculating a coding coefficient; 4) uniformly distributing the data blocks to be written to different physical clusters by establishing a dynamic mapping mechanism between the local group and the physical cluster capacity and combining a block-by-block distribution strategy; according to the method, the parallel writing throughput of a large-scale distributed storage system can be remarkably improved while the efficient repairing performance is maintained, and the method is suitable for scenes such as cloud storage and distributed databases.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and further relates to data layout optimization technology in distributed storage systems, specifically a method for efficiently generating Local Reconstruction Code (LRC) stripes. It can be used for cold data storage on cloud storage platforms. Background Technology

[0002] Distributed storage systems are a data management technology that divides data into multiple blocks and stores them on different nodes to achieve load balancing and redundancy. Existing distributed storage systems typically employ a hierarchical architecture, organizing storage nodes into clusters.

[0003] Erasure coding (EC) is a data protection technology used to improve data reliability and storage efficiency in distributed storage systems. It uses mathematical algorithms to divide data into smaller blocks and generate a number of check blocks. These check blocks contain redundant information from the original data blocks, enabling the system to reconstruct lost data from the remaining data blocks and check blocks even if some data blocks are lost. This significantly reduces storage space requirements while ensuring data integrity. Locally Recoverable Codes (LRCs) are a special type of erasure coding that reduces data recovery costs by introducing local check blocks. Wide-striped LRCs ((n,k,r)-LRCs) further increase the code rate to reduce storage redundancy, where n is the total number of blocks in the stripe (including data blocks and check blocks), k is the number of original data blocks in the stripe, r is the local recovery parameter (meaning that any single block failure requires recovery from a maximum of r other blocks), and the code rate is the ratio of the number of original data blocks k to the total number of blocks n in the stripe. Wide-striped LRCs are widely used in commercial storage systems, such as Microsoft Azure Storage and Google File System.

[0004] Declustering refers to using algorithms to distribute data blocks or parity blocks evenly across more clusters, thereby making the load on each cluster more even and improving the system's throughput, load balancing, and reliability.

[0005] Currently, in existing block distribution strategies, global parity blocks are typically deployed centrally on a few clusters or nodes. This leads to frequent access to these clusters to update parity blocks during write operations, creating bottlenecks in network transmission and storage persistence. Data blocks, on the other hand, tend to be stored contiguously in a small number of clusters, limiting cluster-level parallel transmission capabilities during writes and exacerbating load imbalance. These issues result in decreased system write throughput and increased latency, severely impacting the overall performance of distributed storage systems. Existing technologies cannot simultaneously achieve a uniform distribution of parity blocks and data blocks, and lack a real-time scheduling mechanism to dynamically adapt to cluster load. Therefore, there is an urgent need to design a dedicated algorithm for balanced distribution of data blocks and parity blocks, optimizing the block layout strategy for wide-strip writes. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing an efficient method for generating local repair code stripes. This method solves the technical problems in existing technologies where the concentration of global check blocks leads to bottlenecks in network transmission and storage persistence, and the concentration of data blocks limits network parallelism. This invention first optimizes the distribution of check blocks by reconstructing the generation matrix, distributing the initially concentrated global check blocks into various local groups within the stripe. Then, based on the reconstructed stripe logical structure and combined with the real-time load status of the storage cluster, dynamic mapping is performed to evenly distribute the data blocks to be written to different physical clusters. This effectively eliminates network I / O bottlenecks during the writing process, significantly improving the parallel write throughput of large-scale distributed storage systems while maintaining high repair performance.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0008] (1) Initialize the generation matrix and logical layout:

[0009] enter Generating matrix and Initial local group All global check blocks are initially concentrated in the tail local group. Based on the total number of global check blocks and number of local groups Calculate the target global check block number that should be allocated to each local group. ;

[0010] (2) Reconstruct the local group to balance the distribution of check blocks:

[0011] Iterate through each local group, starting from the last local group. Random or sequential selection A subset of global check blocks is formed. and from the current local group Extract equal amounts of data blocks to form a subset of data blocks. ,in = ,1,..., ; Obtain the reconstructed local group This enables the uniform distribution of global check blocks across local groups.

[0012] (3) Update the generator matrix and calculate the coding coefficients:

[0013] Based on the reconstructed local group set In Galois The above is the original generating matrix The coefficient rows are summed, and the encoded coefficient rows of the local parity block are recalculated to obtain the updated generator matrix. and output the updated local group set;

[0014] (4) Initialize cluster capacity and mapping vector:

[0015] For each stripe to be written, count the local groups after each refactoring. Number of data blocks This initializes the capacity of the corresponding physical cluster. Simultaneously initialize dimensional mapping vector It is a zero vector used to record the mapping relationship between the cluster and the data block;

[0016] (5) Dynamically allocate data blocks and update parameters in real time:

[0017] (5.1) Receive contains Writing of data blocks to be allocated Request, scan the number of locations of all physical clusters. and the mapping vectors of each cluster ;

[0018] (5.2) Determine the current data to be allocated If the value is greater than or equal to 0, it indicates that there are unallocated data blocks, and the process continues to step (5.3); otherwise, the process proceeds directly to step (6).

[0019] (5.3) Prioritize selecting the physical cluster that is not selected and has the most available locations, and record the number of available locations as the available capacity. If there are no unselected available clusters, then select the cluster with the most available locations among all available clusters, and record the number of available locations as the available capacity. ;

[0020] (5.4) Allocate one data block to The corresponding cluster is identified, and after updating the relevant parameters in real time, the process returns to step (5.2).

[0021] (6) Generate complete bands:

[0022] Finish Each data block is distributed across various physical clusters, and the final mapping vector for each cluster is output. And based on this vector, call the updated generator matrix. The written data is encoded and calculated to generate the corresponding check block, thus completing the generation of the LRC stripe.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] First, by performing a shuffling and reconstruction on the generator matrix, the present invention distributes the originally centrally distributed global check blocks to various logical local groups, thereby solving the problem of excessive load on a single storage cluster caused by the over-concentration of global check blocks when performing stripe write tasks, thus improving the efficiency of the storage system.

[0025] Secondly, by establishing a dynamic mapping mechanism between "local group size" and "physical cluster capacity" and adopting a block-by-block allocation strategy with real-time parameter updates, the present invention enables the system to evenly distribute the data blocks to be written to multiple physical clusters according to the real-time load status of the cluster, thereby improving the parallelism of network transmission at the cluster level. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the overall implementation of the method of the present invention;

[0027] Figure 2 This is a flowchart of the check block shuffling algorithm provided in an embodiment of the present invention;

[0028] Figure 3 This is a flowchart of the online declustering algorithm for data blocks in this invention. Detailed Implementation

[0029] The present invention will now be further described with reference to the accompanying drawings.

[0030] Example 1: Refer to Appendix Figure 1 The present invention proposes an efficient method for generating Local Repair Code (LRC) stripes, which specifically includes the following steps:

[0031] Step 1) Initialize the generation matrix and logical layout:

[0032] enter Generating matrix and Initial local group All global check blocks are initially concentrated in the tail local group. Based on the total number of global check blocks and number of local groups Calculate the target global check block number that should be allocated to each local group. ;

[0033] Step 2) Reconstruct the local group to balance and distribute check blocks:

[0034] Iterate through each local group, starting from the last local group. Random or sequential selection A subset of global check blocks is formed. and from the current local group Extract equal amounts of data blocks to form a subset of data blocks. ,in = ,1,..., ; Obtain the reconstructed local group This enables the uniform distribution of global check blocks across local groups.

[0035] Step 3) Update the generator matrix and calculate the coding coefficients:

[0036] Based on the reconstructed local group set In Galois The above is the original generating matrix The coefficient rows are summed, and the encoded coefficient rows of the local parity block are recalculated to obtain the updated generator matrix. It then outputs the updated local group set. This embodiment describes the process in the Galois domain. The above is the original generating matrix The summation operation is performed on the coefficient row within the binary extended field to ensure that the computational efficiency of the encoded calculation is fully adapted to the binary characteristics of the data storage.

[0037] Step 4) Initialize cluster capacity and mapping vector:

[0038] For each stripe to be written, count the local groups after each refactoring. Number of data blocks This initializes the capacity of the corresponding physical cluster. Simultaneously initialize dimensional mapping vector This is a zero vector used to record the mapping relationship between clusters and data blocks. In this embodiment, the number of physical clusters mentioned above... Number of local groups Equal, each restructured local group corresponds to a unique physical cluster.

[0039] Step 5) Dynamically allocate data blocks and update parameters in real time:

[0040] Step 5.1) Receive the contents Writing of data blocks to be allocated Request, scan the number of locations of all physical clusters. and the mapping vectors of each cluster In this embodiment, the number of data blocks to be allocated in the initial state during this step is... Not less than 1, and The total number of available locations in each cluster is not less than .

[0041] Step 5.2) Determine the current data to be allocated. If the value is greater than or equal to 0, it indicates the existence of unallocated data blocks, and the process continues to step 5.3; otherwise, the process proceeds directly to step 6.

[0042] Step 5.3) Prioritize selecting the physical cluster that is not selected and has the most available locations, and record the number of available locations as the available capacity. If there are no unselected available clusters, then select the cluster with the most available locations among all available clusters, and record the number of available locations as the available capacity. In this embodiment, the specific implementation of finding the cluster with the largest number of unselected locations is as follows: from Select elements from the clusters that satisfy the mapping vector. And the number of positions Given a subset of clusters, calculate the number of locations for each cluster within that subset. Select The largest cluster is used as the available capacity. .

[0043] Step 5.4) Allocate one data block to After determining the corresponding cluster and updating the relevant parameters in real time, return to step 5.2). The real-time updating of relevant parameters in this step specifically includes the mapping vector value for the cluster. Add 1 to the available capacity of the cluster. Decrease by 1 to determine the number of data blocks to be allocated. Subtract 1.

[0044] Step 6) Generate complete bands:

[0045] Finish Each data block is distributed across various physical clusters, and the final mapping vector for each cluster is output. And based on this vector, call the updated generator matrix. Encoding calculations are performed on the written data to generate corresponding check blocks, thus completing the generation of the LRC stripe. In this embodiment, the above-mentioned encoding calculations on the written data include global check encoding and local check encoding on the written data blocks. The generated check blocks and data blocks together constitute a complete LRC stripe.

[0046] Example 2: The overall implementation steps of the efficient LRC strip generation method provided in this example are the same as in Example 1. Now, in conjunction with the appendix... Figure 2-3 ,by ,in , , ; 4 local groups data block 4 local check blocks (belonging to) ), 4 global check blocks Concentrated on , Initial composition: The implementation process of the method of the present invention will be further described in detail using specific examples (Uniform Cauchy LRCs).

[0047] Reference Figure 2 This invention relates to a verification block shuffling algorithm that initializes the generation matrix and logical layout, reconstructs local groups, and updates the generation matrix. Specifically, it involves shuffling and reconstructing the generation matrix to distribute the originally centrally distributed global verification blocks to various logical local groups. The algorithm includes the following steps:

[0048] Step a1. Calculate the number of target global check blocks that should be allocated to each local group. ,in It is the total number of global check blocks. This refers to the number of local groups; that is, each local group is allocated one global check block.

[0049] Step a2. Iterate through each local group, starting from... choose composition , From choice composition ,from choose composition ,from choose composition ,from choose composition ,from choose composition ,from choose composition ,from choose composition ;

[0050] Step a3. Reconstruct the local group according to the following formula:

[0051] ,

[0052] ;

[0053] Specifically, this includes: restructuring , Updated to Reconstruction , Updated to Reconstruction , Updated to Reconstruction ;

[0054] Step a4. Update the generated matrix based on the new local group. for ,calculate Coding coefficient behavior The coefficient row in Summation of a domain; computation Coding coefficient behavior The coefficient row in Summation of the domain, calculation Coding coefficient behavior The coefficient row in Summation of the domain, calculation Coding coefficient behavior The coefficient row in Summation of the domain;

[0055] Step a5. Output the updated generator matrix and local groups to achieve a uniform distribution of the global check block across local groups and the cluster:

[0056] , , , .

[0057] Reference Figure 3 This invention describes the process from initializing cluster capacity and mapping vectors, dynamically allocating data blocks and updating parameters in real time until generating a complete stripe, specifically for each write operation. The request involves using an online declustering algorithm to evenly distribute data blocks across multiple clusters, combined with the uniform distribution characteristics of check blocks, to avoid cluster-level transmission bottlenecks. Specifically, this includes the following steps:

[0058] Step b1. Input contains two data blocks Files , Location list of 8 clusters 8-dimensional mapping vector ;

[0059] Step b2. The first request contains two data blocks. For the first data block... Find Not Selected And the cluster with the largest number of available locations ,Will Assigned to ,renew , , ,Right now , , For the second data block Find Not Selected And the cluster with the largest number of available locations ,Will Assigned to ,renew , , ,Right now , , ;

[0060] Step b3. Based on the mapping vector finally generated in step b2 The generator matrix is ​​invoked to perform encoding calculations on the written data, generating the corresponding check block and completing the entire process of generating a stripe.

[0061] The LRC stripe generation method proposed in this invention can be widely applied to large-scale distributed storage system scenarios, with broad prospects for industrialization. In the cloud computing field, this method can adapt to the core needs of cloud storage platforms for high-concurrency writing and persistent storage of massive amounts of data. By optimizing the distribution of parity blocks and dynamic load mapping, it effectively eliminates network I / O bottlenecks, significantly improves write throughput, reduces storage costs for cloud service providers, and ensures high-reliability data repair capabilities. In big data storage scenarios, facing the high-frequency writing and elastic expansion requirements of petabyte-level data, this invention can break through the performance limitations of traditional centralized parity block layouts, improve the parallel processing efficiency of storage clusters, and help industries such as finance, government, and scientific research build a more efficient and stable big data storage foundation. In edge computing and IoT storage scenarios, this method can adapt to the characteristics of strong heterogeneity of edge node resources and large fluctuations in network bandwidth. Through dynamic load balancing strategies, it optimizes the cross-node distribution of data blocks, maintains efficient local repair capabilities, and improves the overall throughput performance of edge storage systems, providing more reliable data storage support for scenarios such as connected vehicles and industrial IoT; it has high commercial value and technology promotion potential.

[0062] The parts of this invention not described in detail are common knowledge to those skilled in the art.

[0063] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. An efficient method for generating locally repaired code (LRC) stripes, characterized in that, Includes the following steps: (1) Initialize the generation matrix and logical layout: enter Generating matrix and Initial local group All global check blocks are initially concentrated in the tail local group. Based on the total number of global check blocks and number of local groups Calculate the target global check block number that should be allocated to each local group. ; (2) Reconstruct the local group to balance the distribution of check blocks: Iterate through each local group, starting from the last local group. Random or sequential selection A subset of global check blocks is formed. and from the current local group Extract equal amounts of data blocks to form a subset of data blocks. ,in = ,1,..., ; Obtain the reconstructed local group This enables the uniform distribution of global check blocks across local groups. (3) Update the generator matrix and calculate the coding coefficients: Based on the reconstructed local group set In Galois The above is the original generating matrix The coefficient rows are summed, and the encoded coefficient rows of the local parity block are recalculated to obtain the updated generator matrix. and output the updated local group set; (4) Initialize cluster capacity and mapping vector: For each stripe to be written, count the local groups after each refactoring. Number of data blocks This initializes the capacity of the corresponding physical cluster. Simultaneously initialize dimensional mapping vector It is a zero vector used to record the mapping relationship between the cluster and the data block; (5) Dynamically allocate data blocks and update parameters in real time: (5.1) Receive contains Writing of data blocks to be allocated Request, scan the number of locations of all physical clusters. and the mapping vectors of each cluster ; (5.2) Determine the current data to be allocated If the value is greater than or equal to 0, it indicates that there are unallocated data blocks, and the process continues to step (5.3); otherwise, the process proceeds directly to step (6). (5.3) Prioritize selecting the physical cluster that is not selected and has the most available locations, and record the number of available locations as the available capacity. If there are no unselected available clusters, then select the cluster with the most available locations among all available clusters, and record the number of available locations as the available capacity. ; (5.4) Allocate one data block to The corresponding cluster is identified, and after updating the relevant parameters in real time, the process returns to step (5.2). (6) Generate complete bands: Finish Each data block is distributed across various physical clusters, and the final mapping vector for each cluster is output. And based on this vector, call the updated generator matrix. The written data is encoded and calculated to generate the corresponding check block, thus completing the generation of the LRC stripe.

2. The generation method according to claim 1, characterized in that: In step (3), the Galois domain is described. The above is the original generating matrix The summation operation is performed on the coefficient row within the binary extended field to ensure that the computational efficiency of the encoded calculation is fully adapted to the binary characteristics of the data storage.

3. The generation method according to claim 1, characterized in that: The number of physical clusters and the number of local groups mentioned in step (4) Equal, each restructured local group corresponds to a unique physical cluster.

4. The generation method according to claim 1, characterized in that: The receiving described in step (5.1) includes Writing of data blocks to be allocated Request, in which the number of data blocks to be allocated in the initial state. Not less than 1, and The total number of available locations in each cluster is not less than .

5. The generation method according to claim 1, characterized in that: The method for finding the cluster with the largest number of unselected locations in step (5.3) is as follows: from Select elements that satisfy the mapping vector from each cluster. And the number of positions Given a subset of clusters, calculate the number of locations for each cluster within that subset. Select The largest cluster is used as the available capacity. .

6. The generation method according to claim 1, characterized in that: The real-time updating of relevant parameters mentioned in step (5.4) includes the mapping vector value for the cluster. Add 1 to the available capacity of the cluster. Decrease by 1 to determine the number of data blocks to be allocated. Subtract 1.

7. The generation method according to claim 1, characterized in that: The encoding calculation performed on the written data in step (6) includes global verification encoding and local verification encoding of the written data block. The generated verification block and the data block together constitute a complete LRC stripe.

Images