An erasure code file processing method, device, equipment, medium and product

Abstract

The application discloses a kind of erasure code file processing method, device, equipment, medium and product, by real-time receiving the original data of user input;Using first proportion erasure code algorithm, the original data is handled, generates first erasure code file, and is stored to preset memory;Wherein, the first erasure code file includes several first data blocks and at least one first check block obtained by the original data division;According to second proportion erasure code algorithm, part of first data block and all first check block in N first erasure code file are read from the memory, at least one second check block is generated;The second check block and all first data blocks in the N first erasure code file are merged and stored to the memory as second erasure code file.Using the application, it can reduce data write delay, while avoiding the problem of flow amplification in data transfer process, effectively improve data storage efficiency.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and big data technology, and in particular to an erasure coding file processing method, apparatus, device, medium, and product. Background Technology

[0002] Erasure coding (EC) is a data redundancy protection technology in distributed storage. It divides the original data into multiple data blocks and generates corresponding check blocks. These data blocks and check blocks are distributed across different storage nodes. When some blocks are lost, the lost data can be recovered using the remaining data blocks and check blocks, thus ensuring data reliability and fault tolerance. Direct-Write EC generates erasure codes and writes them to disk directly during data writing, without going through an additional cache. It is suitable for scenarios with high data reliability requirements.

[0003] Erasure coding algorithms offer higher data storage efficiency and significantly save storage space compared to traditional multi-copy data storage methods. However, in practical applications, erasure coding algorithms require a certain number of data blocks to be accumulated before encoding operations can begin. While waiting for data, write operations cannot be completed in a timely manner, leaving system resources idle and significantly extending file write times. This not only affects the system's real-time performance, but for applications with stringent write latency requirements, such as high-performance block storage, real-time data processing, and high-speed trading systems, high latency can even cause applications to malfunction, severely limiting the application of erasure coding algorithms in these scenarios.

[0004] To reduce latency, existing technologies employ optimization strategies. One approach is to temporarily store data using small-part erasure coding, such as a 4+2 EC (split data into four data blocks, generating two parity blocks) file, requiring only 8KB of data. Later, to improve long-term storage efficiency, this is converted to large-part erasure coding, such as 8+3 EC, requiring 256KB of data. However, the traditional method requires reading all the original temporarily stored data, recalculating the large-part erasure coding, and then re-storing the new data blocks and parity blocks. This is equivalent to repeated reading, writing, and storage of data, generating significant additional network or storage traffic and increasing system resource consumption.

[0005] It is evident that in storage systems that primarily rely on small I / O (Input / Output) and have high I / O latency requirements, directly using large-scale erasure coding can lead to excessively high write latency. On the other hand, adopting a solution of small-scale temporary storage and asynchronous conversion to large-scale coding can result in severe traffic amplification and waste of system resources. Summary of the Invention

[0006] The purpose of this invention is to provide an erasure coding file processing method, apparatus, device, medium, and product that can reduce data write latency and avoid the problem of traffic amplification during data transfer, thereby effectively improving data storage efficiency.

[0007] To achieve the above objectives, embodiments of the present invention provide a method for processing erasure coding files, including: Receive raw data input by the user in real time; The original data is processed using a first proportional erasure coding algorithm to generate a first erasure coding file, which is then stored in a preset memory. The first erasure coding file includes several first data blocks and at least one first check block obtained by dividing the original data. According to the second proportional erasure coding algorithm, a portion of the first data blocks and all of the first check blocks from N first erasure coding files are read from the memory to generate at least one second check block; N > 1; The second verification block and all the first data blocks in the N first erasure coding files are merged and stored in the memory to form the second erasure coding file.

[0008] As an improvement to the above scheme, the step of reading a portion of the first data blocks and all of the first parity blocks from the memory of N first erasure coding files according to the second proportional erasure coding algorithm, and generating at least one second parity block, includes: According to the second proportional erasure coding algorithm, the number of second check blocks to be generated for the second erasure coding file is determined and denoted as the first target number. The number N of the first erasure coding files corresponding to the generation of one second erasure coding file is determined and each N first erasure coding files is combined into a file group. Based on the relationship between the N first erasure coding files and the second erasure coding files in the file combination, determine the number of the first data blocks to be read, denoted as the second target number; Read the first data blocks and all first check blocks that satisfy the second target number from the N first erasure coding files in the memory; Based on all the first verification blocks and the first data blocks of the second target number, a second verification block that satisfies the first target number is generated.

[0009] As an improvement to the above scheme, the step of determining the number of first data blocks to be read, denoted as the second target number, based on the relationship between the N first erasure coding files and the second erasure coding files in the file combination, includes: Construct a first generator matrix based on the first data block and the first check block of the N first erasure coding files; Based on the second data block of the second erasure coding file and the second verification block to be generated, construct the second generation matrix; The first generating matrix and the second generating matrix are decomposed into row vectors. Under the operation rules of the finite field, the coefficient matrix is ​​solved by Gaussian elimination to determine the linear correlation between the first check block of the N first erasure code files and the second check block of the second erasure code files. Based on the linear correlation, the number of the first data blocks to be read is determined and denoted as the second target number; As an improvement to the above scheme, the step of generating a second verification block that satisfies the first target number based on all the first verification blocks and the first data blocks of the second target number includes: Based on all the first verification blocks and the first data blocks of the second target quantity, the first second verification block is generated through coefficient matrix operations; Based on all the first check blocks and the generated second check blocks, the next second check block is generated through linear operation until the number of second check blocks that meet the first target number is generated.

[0010] As an improvement to the above solution, after receiving the raw data input by the user in real time, the method further includes: The original data is divided into several sector data blocks according to the preset cloud disk sector size; The sector data blocks are acquired in parallel using multiple data acquisition threads; The sector data blocks are aggregated to generate several aggregated data blocks; The process of using the first proportional erasure coding algorithm to process the original data and generate the first erasure coding file specifically involves: The aggregated data block is processed using a first proportional erasure coding algorithm to generate a first erasure coding file.

[0011] As an improvement to the above scheme, the aggregation process of the sector data blocks to generate several aggregated data blocks includes: The sector data blocks are aggregated using a preset aggregation strategy to generate several aggregated data blocks; The preset aggregation strategy is as follows: within a preset maximum waiting time, if the sector data blocks received in real time meet the preset aggregation quantity threshold, the sector data blocks that meet the preset aggregation quantity threshold will be aggregated; if the sector data blocks received in real time do not meet the preset aggregation quantity threshold, the existing sector data blocks will be padded with zeros until they meet the preset aggregation quantity threshold before aggregation.

[0012] As an improvement to the above solution, the aggregation process of the sector data blocks to generate several aggregated data blocks further includes: Based on the data characteristics of the aggregated data block, a preset data compression algorithm is used to compress the aggregated data block to obtain a compressed aggregated data block.

[0013] As an improvement to the above scheme, after processing the original data using the first proportional erasure coding algorithm to generate a first erasure coding file and storing it in a preset memory, the method further includes: A response message indicating successful data writing is returned to the user.

[0014] This invention also provides an erasure coding file processing apparatus, comprising: The raw data receiving module is used to receive raw data input by the user in real time. The first erasure coding generation module is used to process the original data using a first proportional erasure coding algorithm to generate a first erasure coding file and store it in a preset memory; wherein, the first erasure coding file includes a plurality of first data blocks and at least one first check block obtained by dividing the original data; The second erasure coding generation module is used to read a portion of the first data blocks and all the first check blocks from N first erasure coding files in the memory according to the second proportional erasure coding algorithm, and generate at least one second check block; N>1; and to merge the second check block and all the first data blocks in the N first erasure coding files into the memory as the second erasure coding file.

[0015] This invention also provides an erasure coding file processing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the erasure coding file processing method as described in any of the above embodiments.

[0016] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the erasure coding file processing method as described in any of the above embodiments.

[0017] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the erasure coding file processing method as described in any of the above embodiments.

[0018] Compared with existing technologies, the erasure coding file processing method, apparatus, device, medium, and product disclosed in this invention reduce write latency by selecting a small proportion of erasure coding as temporary storage for data to quickly write to disk and complete write I / O. Then, based on the large amount of data accumulated, the temporary storage data is asynchronously converted into a large proportion of erasure coding file to improve storage efficiency. The erasure coding conversion algorithm of this invention achieves efficient conversion of erasure coding by only updating the data checksum while retaining the original data without re-storing it. This solves the problems of increased write latency caused by the need to accumulate data to meet the requirements of the erasure coding algorithm in traditional direct write erasure coding file scenarios, or the traffic amplification and storage amplification caused by the need to read all the original data, regenerate all data blocks and checksum blocks and re-store them during the erasure coding transfer process. This effectively improves the efficiency of data storage. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating an erasure coding file processing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a traditional erasure coding file transfer scheme; Figure 3 This is a schematic diagram of the erasure coding file transfer scheme in an embodiment of the present invention; Figure 4 This is a flowchart illustrating a preferred erasure coding file processing method in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an erasure coding file processing device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0022] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0023] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0024] See Figure 1 This is a flowchart illustrating an erasure coding file processing method provided by an embodiment of the present invention. The embodiment of the present invention provides an erasure coding file processing method, the method comprising steps S11 to S14: S11. Receive raw data input by the user in real time; S12. The original data is processed using a first proportional erasure coding algorithm to generate a first erasure coding file and store it in a preset memory; wherein, the first erasure coding file includes a plurality of first data blocks and at least one first check block obtained by dividing the original data; S13. According to the second proportional erasure coding algorithm, read a portion of the first data blocks and all of the first check blocks from N first erasure coding files in the memory, and generate at least one second check block; N>1; S14. Merge the second verification block and all the first data blocks in the N first erasure coding files and store them in the memory as the second erasure coding file.

[0025] In this embodiment of the invention, in order to address the issues of reducing real-time I / O write latency and traffic amplification during data transfer in performance-sensitive storage systems (such as block storage) using erasure coding to store data, this embodiment proposes an erasure coding conversion algorithm that converts low-ratio erasure coding used for fast aggregated I / O to disk into high-ratio erasure coding while minimizing the reading and writing of original data. By optimizing the data writing process and data storage strategy, this algorithm reduces the write latency of direct-write erasure coding files while also reducing traffic amplification.

[0026] Specifically, the storage system receives the raw data written by the user in real time. First, it uses a first proportional erasure coding algorithm (small-proportion erasure coding) to divide the raw data into data blocks and generates check blocks, resulting in a first erasure coding file. Each first erasure coding file includes several first data blocks and at least one first check block. The first erasure coding file is then stored in a preset memory to complete persistent disk writing. After successful disk writing, the user's raw data writing is complete.

[0027] Preferably, after step S12, i.e., after processing the original data using the first proportional erasure coding algorithm to generate the first erasure coding file and storing it in a preset memory, the method further includes step S15: S15. Return a response message indicating successful data writing to the user.

[0028] In other words, after the data is successfully written to disk, the storage system immediately sends a notification to the user that the original data has been successfully written, and the user's write request is completed without any additional waiting.

[0029] Furthermore, the storage system performs asynchronous background operations to convert the temporary first erasure coding file into a final second erasure coding file using a preset erasure coding conversion algorithm before storage. This erasure coding conversion algorithm can recalculate the data checksum to generate the second checksum block while retaining the already stored first data block. This avoids the excessive bandwidth consumption problem caused by traditional erasure coding conversion algorithms requiring the regeneration of all data blocks and checksum blocks to replace the original erasure coding file.

[0030] During erasure coding conversion, firstly, based on the target's required second-ratio erasure coding algorithm (high-ratio erasure coding), a portion of the first data blocks from N first erasure coding files are read from the memory, along with all the first parity blocks from the N first erasure coding files; that is, it is not necessary to read all the data information from the first erasure coding files. Then, based on the portion of the first data blocks and all the first parity blocks, a second parity block of the second erasure coding file is generated. While retaining the original first data blocks of the first erasure coding files, the second parity block and all the first data blocks from the N first erasure coding files are merged and stored in the memory. There is no need to rewrite the original first data blocks; only the association between the first data blocks and the second parity blocks is established in the system index, forming the second erasure coding file. The entire process avoids redundant storage of original data, only updating the parity blocks, significantly reducing data read and write operations.

[0031] As an example, the traffic amplification of the traditional erasure coding transfer scheme and the erasure coding transfer scheme of the present invention are compared.

[0032] See Figure 2This diagram illustrates a traditional erasure coding file transfer scheme. This scheme uses a small 4+2 EC (Extended Encoding) erasure coding ratio for the Journal file (temporary log file). It only requires 8KB (2KB*4) of data to generate the erasure coding file and write it to disk, quickly fulfilling user write requests and significantly reducing real-time write latency. However, the actual replica count of 4+2 EC is 1.5, resulting in mediocre storage efficiency. To reduce the storage cost of long-term data storage, the system asynchronously performs a conversion operation in the background, converting the temporary 4+2 EC format data into a more efficient 8+3 EC format (1.375 replicas) as the final data file. This erasure coding ratio requires 256KB (32KB*8) of data to be accumulated.

[0033] like Figure 2 As shown, although this copy conversion process is asynchronous, the erasure coding file transfer process requires reading all the original data in the Journal to recalculate the erasure codes and re-storing the original data according to the 8+3 EC requirement. The Journal file storage is only used as temporary storage for data persistence. Specifically, 24 blocks of data are written initially, and 69 blocks of data are actually stored, including 36 data blocks and parity blocks written in 4+2 EC, 33 data blocks and parity blocks written in 8+3 EC, and the remaining 24 original data blocks read during the erasure coding file transfer process. The traffic amplification factor is 3.875 (i.e., (69+33) / 24).

[0034] See Figure 3 This diagram illustrates the erasure coding file transfer scheme in this embodiment of the invention. This scheme also uses 4+2 EC small-ratio erasure coding to process the original data, generating an erasure coding file (i.e., the first erasure coding file) and storing it on disk. Next, the system asynchronously performs a conversion operation in the background to convert it to an 8+3 EC large-ratio erasure coding file. An 8+3 EC file is constructed based on the data from the two 4+2 EC files. A polling method is used to continuously check whether the 4+2 EC format data in the write-ahead log has been converted to a certain group of 8+3 EC data. If so, the system continues to check subsequent data until unconverted 4+2 EC format data is found. If two 4+2 EC blocks are found to be unconverted, the erasure coding conversion algorithm is used to calculate and generate three new check blocks (i.e., the second check blocks) for the 8+3 EC file using all check blocks (i.e., the first check blocks) in the two 4+2 EC files and the minimized original data block (i.e., the first data block). The check blocks of the two 4+2 EC blocks that have been converted are updated and replaced. The original data block and the newly generated check blocks are logically merged into a new 8+3 EC file. There is no need to store the original data block repeatedly. Finally, garbage collection is triggered to reclaim storage space and reclaim the original check blocks.

[0035] and Figure 2 Compared with traditional transfer schemes, the main optimization of this scheme is that the number of direct data reads during the conversion of 4+2 EC erasure codes to 8+3 EC is reduced from 24 data blocks to 15, a decrease of 37.5%, and the number of data written is reduced from 33 blocks to 9 blocks, a decrease of 72.8%.

[0036] By employing the technical means of this invention, a small proportion of erasure coding is selected as temporary storage for data to quickly complete write I / O and reduce write latency. Then, based on the large amount of data accumulated, the temporary storage data is asynchronously converted into a large proportion of erasure coding files to improve storage efficiency. The erasure coding conversion algorithm of this invention achieves efficient conversion of erasure coding by only updating the data checksum while retaining the original data without re-storing it. This solves the problems of increased write latency caused by the need to accumulate data to meet the requirements of the erasure coding algorithm in traditional direct write erasure coding file scenarios, or the amplification of traffic and storage data caused by the need to read all the original data, regenerate all data blocks and checksum blocks and re-store them during the erasure coding conversion process. This effectively improves the efficiency of data storage.

[0037] As a preferred embodiment, the present invention further implements the above embodiments. Step S13, namely, reading a portion of the first data blocks and all of the first check blocks from the memory according to the second proportional erasure coding algorithm to generate at least one second check block, includes steps S131 to S134: S131. According to the second proportional erasure coding algorithm, determine the number of second check blocks to be generated for the second erasure coding file, denoted as the first target number, and determine the number N of first erasure coding files corresponding to the generation of one second erasure coding file, and combine N first erasure coding files into one file group. S132. Based on the relationship between the N first erasure coding files and the second erasure coding files in the file combination, determine the number of first data blocks to be read, denoted as the second target number; S133. Read from the memory the first data blocks and all first check blocks of the N first erasure coding files that satisfy the second target number; S134. Generate a second verification block that satisfies the first target number based on all the first verification blocks and the first data blocks of the second target number.

[0038] In this embodiment of the invention, the number of second parity blocks (i.e., the first target number) in the required second erasure coding file can be determined according to a preset second proportional erasure coding algorithm. Furthermore, based on the number of data blocks in the erasure coding files corresponding to the second and first proportional erasure coding algorithms, it can be determined that N data blocks from the first erasure coding files correspond to one data block from the second erasure coding file. For example, an 8+3 EC erasure coding file requires 3 parity blocks and is divided into 8 data blocks, corresponding to two sets of data blocks from a 4+2 EC erasure coding file.

[0039] After determining the above correspondence, calculate the minimum number of first data blocks (i.e., the second target number) required to calculate the second checksum in the N first erasure coding files, and then read the first data blocks and all first checksum blocks that satisfy the second target number from the N first erasure coding files in the memory for calculating the second checksum.

[0040] In a preferred embodiment, step S132, namely determining the number of first data blocks to be read based on the relationship between the N first erasure coding files and the second erasure coding files in the file combination, denoted as the second target number, includes: Construct a first generator matrix based on the first data block and the first check block of the N first erasure coding files; Based on the second data block of the second erasure coding file and the second verification block to be generated, construct the second generation matrix; The first generating matrix and the second generating matrix are decomposed into row vectors. Under the operation rules of the finite field, the coefficient matrix is ​​solved by Gaussian elimination to determine the linear correlation between the first check block of the N first erasure code files and the second check block of the second erasure code files. Based on the linear correlation, the number of the first data blocks to be read is determined and denoted as the second target number; In a preferred embodiment, step S134, namely, generating a second verification block that satisfies the first target number based on all the first verification blocks and the first data blocks of the second target number, includes: Based on all the first verification blocks and the first data blocks of the second target quantity, the first second verification block is generated through coefficient matrix operations; Based on all the first check blocks and the generated second check blocks, the next second check block is generated through linear operation until the number of second check blocks that meet the first target number is generated.

[0041] In this embodiment of the invention, the erasure coding conversion algorithm of this scheme is explained by taking the first proportional erasure coding algorithm as 4+2 EC format and the second proportional erasure coding algorithm as 8+3 EC format as examples.

[0042] First, clarify the data structures of the two sets of 4+2 EC and the target 8+3 EC, including the first data block, the second check block, and the first generator matrix of the 4+2 EC, as well as the first data block, the second check block to be generated, and the second generator matrix after the 8+3 EC is integrated.

[0043] The generator matrices of 4+2 EC and 8+3 EC are decomposed into row vectors. The coefficient matrix is ​​solved by Gaussian elimination in the finite field GF (2). The linear relationship between the first check block of 4+2 EC and the second check block of 8+3 EC is determined. The path to calculate the second check block is found using only a small amount of original data and the existing first check block, ensuring that only the minimum amount of original data needs to be read to complete the subsequent calculation. The three global check blocks of 8+3 EC are generated step by step. , , The calculation is performed sequentially and existing data is reused throughout the process, with no additional redundant reading.

[0044] calculate Read four local check blocks and one raw data block from two sets of 4+2 EC, and generate them by combining the coefficient matrix. This step alone requires reading a small amount of raw data. calculate Directly reuse the four read local check blocks and the generated ones. It is generated through linear operations without requiring any additional data reading. calculate Reuse read local check blocks and generated , It can be generated by solving the linear equation under GF (2) without the need for additional data reading.

[0045] Specifically, let the first data block of the first 4+2 EC group be... The first verification block is Its first generating matrix ,satisfy .

[0046] Let the first data block of the second 4+2 EC group be... The first verification block is Its first generating matrix ,satisfy .

[0047] The target 8+3 EC group of data blocks is i=1,2; j=1,2,3,4, that is , , ..., , ..., The second verification block is , , Its second generating matrix ,satisfy .

[0048] Analysis based on the generating matrix relationship: Will Represented as a row vector , , ,Will Represented as a row vector , ,Will Represented as a row vector , .

[0049] Under the operational rules of the Galois field GF(2) (elements are only 0 and 1, addition is XOR operation, and multiplication is AND operation), the coefficient matrix is ​​obtained by solving the following system of equations using methods such as Gaussian elimination. Used to determine the calculation of the 8+3EC group check block , , How to utilize the existing 4+2 EC block check? Information.

[0050]

[0051]

[0052]

[0053] because ,therefore It can be represented as ,in This is a data block within the first 4+2EC group, analyzed through the coefficient matrix. Determined, making The calculation only requires this one additional data block.

[0054] Calculate the check block : Read and Assuming each block size is s bytes, the amount of data read is 4s + s = 5s bytes. However, due to... These are existing parity blocks, which can be optimized in the storage system through caching mechanisms to achieve zero overhead for reading parity blocks (because they have already been read and stored in the cache during the previous 4+2 EC calculation). Therefore, the calculation... The actual number of bytes read was 5s bytes, and the net number of bytes read was s bytes.

[0055] Calculate the check block : analyze The second line and and Relationship, Discover It can be represented as .

[0056] The computation depends on and All of these are based on previous calculations. If the data has already been read or calculated, no additional data reading is required. Therefore, the calculation... The actual number of bytes read was 0, and the net number of bytes read was 0.

[0057] Calculate the check block : for The third line , It can be represented as .in It is the solution to the linear equation constructed on the Galois field GF(2), used to guarantee The calculation results meet the requirements of the 8+3 EC grouping check block and can make full use of the existing check blocks. and The information is used to optimize the calculation process.

[0058] The computation depends on and All of these are based on previous calculations. and If the data has already been read or calculated, no additional data reading is required. Therefore, the calculation... The actual number of bytes read was 0, and the net number of bytes read was 0.

[0059] As can be seen, the erasure coding conversion process in this algorithm requires reading all 4+2 EC erasure codes and part of the original data. and ) and update three new check codes , , Assuming a single data block size of 's' bytes, the actual user input data volume is 8s bytes, the data volume stored as two sets of 4+2 EC and checksum is 12s bytes, the erasure coding conversion data volume is 5s bytes, and updating the new erasure coding is 3s. The actual data volume read is 5s, and the actual data volume written is 15s (12s + 3s). The total data volume involved is 20s (12s + 5s + 3s), with a flow amplification factor of 2.5 (20s / 8s). The actual number of data replicas is 1.375 ((8s + 3s) / 8s).

[0060] The conversion from a traditional 4+2 EC to an 8+3 EC requires reading all the original data, recalculating the 8+3 EC, and then re-storing the data and checksum. Assuming a single data block size of 's' bytes, the actual user input data is 8s bytes, the data stored as two sets of 4+2 EC and checksum is 12s bytes, erasure coding conversion reads 8s bytes, and updating the 8+3 EC and erasure coding takes 11s. The actual data read is 8s, and the actual data written is 23s (12s + 11s). The actual data involved is 31s (12s + 8s + 11s), with a flow amplification factor of 3.875 (31s / 8s). The actual number of data replicas is 1.375 ((8s + 3s) / 8s).

[0061] By using the technical means of this invention, a quantitative comparison between this algorithm and traditional conversion algorithms shows that, while ensuring the same 1.375 replica count for 8+3 ECs, the algorithm significantly reduces the amount of data read and stored during the conversion process, reducing the traffic amplification factor from 3.875 to 2.5. Furthermore, this algorithm only requires updating 3 global check blocks and does not need to restore the original data, fundamentally reducing resource consumption.

[0062] Understandably, in practical storage system applications, a fixed set of N small-ratio erasure codes is selected to be converted into one large-ratio erasure code based on business I / O characteristics, erasure coding fault tolerance requirements, and storage efficiency. For example, two sets of 4+2 EC are converted to 8+3 EC, or three sets of 4+2 EC are converted to 12+4 EC. The core purpose is to balance data aggregation latency and traffic amplification optimization effects, avoiding excessively long aggregation times for small-ratio erasure codes due to an excessively large N, which would increase write latency. As long as the core conditions of matching the number of data blocks and linear correlation of check blocks are met, the erasure coding conversion algorithm of this solution can adapt to different conversions of N small-ratio erasure codes to one large-ratio erasure code without affecting the beneficial effects achieved by this invention.

[0063] As a preferred embodiment, the present invention further implements the above embodiments by adding the following method after receiving the raw data input by the user in real time: The original data is divided into several sector data blocks according to the preset cloud disk sector size; The sector data blocks are acquired in parallel using multiple data acquisition threads; The sector data blocks are aggregated to generate several aggregated data blocks; The process of using the first proportional erasure coding algorithm to process the original data and generate the first erasure coding file specifically involves: The aggregated data block is processed using a first proportional erasure coding algorithm to generate a first erasure coding file.

[0064] In this embodiment of the invention, when a user initiates any case-sensitive IO data (such as 10KB or 20KB) to the storage system, the storage system first splits the user data into multiple minimum IO blocks (i.e., sector data blocks) according to the cloud disk sector size (e.g., 4KB).

[0065] The storage system has a data processing service that uses multi-threading to improve the parallel processing capability of I / O. Each thread processes each user write I / O in the form of RTC (Run-to-Complete), which is the data acquisition thread, thereby quickly acquiring data.

[0066] All data acquisition threads share an IO aggregation engine, which receives sector data blocks from all threads, aggregates them, and outputs aggregated data blocks. Subsequently, erasure coding files are generated based on these aggregated data blocks.

[0067] In a preferred embodiment, the aggregation process of the sector data blocks to generate a plurality of aggregated data blocks includes: The sector data blocks are aggregated using a preset aggregation strategy to generate several aggregated data blocks. The preset aggregation strategy is as follows: if the sector data blocks received in real time meet the preset aggregation quantity threshold within a preset maximum waiting time, the sector data blocks that meet the preset aggregation quantity threshold are aggregated. If the sector data blocks received in real time do not meet the preset aggregation quantity threshold, the existing sector data blocks are padded with zeros until they meet the preset aggregation quantity threshold before being aggregated.

[0068] In a preferred embodiment, the step of aggregating the sector data blocks to generate a plurality of aggregated data blocks further includes: Based on the data characteristics of the aggregated data block, a preset data compression algorithm is used to compress the aggregated data block to obtain a compressed aggregated data block.

[0069] In this embodiment of the invention, the IO aggregation engine aggregates data according to a preset aggregation quantity threshold and a preset maximum waiting time. The aggregation quantity threshold is determined according to the first proportional erasure coding algorithm. For example, for 4+2EC erasure coding, the aggregation quantity threshold can be 8KB and the maximum waiting time can be 8µs. If 8KB of data is accumulated within 8µs, the process proceeds directly to the next step. If 8KB is not accumulated within 8µs, the existing data is padded with 0 to 8KB to ensure that the data volume requirement of 4+2EC is met.

[0070] Furthermore, the system determines whether to compress data based on its characteristics. For example, if the compression ratio of purely random data is low, it is skipped to avoid wasting CPU resources. If compression is suitable, the optimal compression algorithm is selected to further save on the amount of data transmitted and stored.

[0071] By employing the technical means of this invention, data is collected in parallel through multiple data acquisition threads, thereby improving data acquisition efficiency. The data is also aggregated and compressed according to the erasure coding algorithm requirements to ensure that data storage requirements are met and to save on the amount of data transmitted and stored.

[0072] See Figure 4 This is a flowchart illustrating a preferred erasure coding file processing method in an embodiment of the present invention. In this embodiment, the storage system includes an IO aggregation engine, a write-ahead log manager, and an erasure coding converter. The IO aggregation engine includes an IO aggregator and a data compression processor. It aggregates small, randomly written IOs and uses hierarchical data storage to achieve fast IO write-to-disk completion. Then, it triggers an asynchronous erasure coding transfer task to reduce data storage costs. The following explains the various components of the system architecture: IO Gather: A typical storage system usually has a data processing service that uses multi-threading to improve the parallel processing capability of IO. Each thread handles each user write IO using RTC (Run-to-Complete) (as shown in the IO Worker diagram). The IO Gather sits below the IO Workers and is shared by all IO Workers, thus allowing for the rapid accumulation of a certain amount of data. Taking a cloud disk with a sector size of 4KB as an example, the minimum IO size for each cloud disk is 4KB. Therefore, if the IOPS (IOs per second) of a single cloud disk is low, such as IOPS=16, it will take 4 seconds to accumulate 256KB of data. If all IO Workers (assuming there are 8 IO Workers) participate in data accumulation together, it only takes 500ms. Therefore, the write wait time for cloud disks in the system differs by a factor of 8. Furthermore, if a single IO Worker processes multiple cloud disks simultaneously, this wait time can be further reduced.

[0073] Furthermore, the design of an IO aggregator needs to include the ability to manage IO aggregation strategies. When the amount of data that does not reach a specified threshold within a certain time, the existing IO needs to be aggregated, and the remaining insufficient parts need to be padded with zeros. Taking a high-speed cloud disk as an example, the single-path latency is 200us, which is generally less than the network card polling interval, such as 8us.

[0074] Data compression processors primarily aim to reduce the amount of data transmitted and stored, thereby improving storage efficiency. This process requires CPU-intensive operations, typically requiring eight dedicated cores. Furthermore, hardware offloading solutions such as FPGAs are also common optimization methods to improve compression efficiency. In actual system implementations, whether to perform compression, or to use different compression methods based on data content, can be managed and optimized using adaptive algorithms.

[0075] The Write-Ahead Logging (WAL) manager is primarily responsible for persisting data collected (or already supplemented) by the IO aggregator and for responding to user write IO requests. This persistent storage is in the form of a Write-Ahead Logging (WAL), mainly including user IO data and transaction log information. The design requirements are fast disk writing and a certain level of data reliability (allowing for at least two data copies to be lost). In the actual system implementation, the entire IO process is completed after the WAL is written; whether the in-memory index is updated successfully does not affect the success of the IO. In the event of a failure to update the in-memory index, the index data can be recovered by scanning the WAL.

[0076] There are multiple strategies for persistent data storage on disk, such as the common 3-replica storage (fast IO writes, but low storage efficiency) and erasure coding files (high storage efficiency, but requires a certain amount of data to be accumulated). The specific solution selection needs to be determined based on the system's reliability, performance, and storage efficiency requirements. This embodiment of the invention focuses on data storage efficiency; therefore, the small proportion of erasure coding is a choice made after balancing storage efficiency and data accumulation requirements. For high-speed cloud disks with random access and small IO as typical scenarios, the commonly used erasure coding ratios are 4+2 EC (1.5 real data replicas, with a common data accumulation size of 8KB) and 8+3 EC (1.375 real data replicas, with a common data accumulation size of 256KB). In comparison, the 4+2 EC is the default storage strategy of the so-called write-ahead log processor, and conversely, the data accumulation threshold of the IO aggregator is 8KB.

[0077] Erasure coding transformer (EC Transformer): This invention utilizes a high-performance erasure coding transformation algorithm to achieve high-ratio erasure coding transformation of write-ahead log data. During the process, a high-ratio erasure coding algorithm checksum block is constructed using only the already generated checksum and a small amount of original data. Then, the original data from the write-ahead log and the newly generated checksum block are used to fully convert the data into high-ratio erasure coding persistent storage. Taking a high-speed cloud disk as an example, persistent storage typically uses 8+3 EC erasure coding storage, with an actual replica count of 1.375, a maximum tolerance for the corruption or loss of 3 data blocks, and a data consolidation threshold of 256KB.

[0078] Erasure coding converters need to be designed with capabilities such as data completion detection, data conversion task management, and erasure coding data read / write. They should periodically check the data write status in the write-ahead log to determine whether erasure coding conversion is necessary. Data processing is similar to that of an I / O aggregator, requiring the setting of data thresholds and completion wait time thresholds to determine whether data completion is required.

[0079] Furthermore, the erasure coding converter requires a new checksum data block generated by the erasure coding conversion algorithm proposed in this embodiment of the invention, and combines the original data in the write-ahead log with the newly generated checksum data block to form a complete 8+3 EC erasure coding storage. This process depends on the actual storage format of the erasure coding. One feasible storage format is to store the original erasure coding data block and the checksum block separately in partitions, and realize the logical association through the upper-level file index to facilitate subsequent erasure coding update operations. This part of the embodiments of the invention only describes one feasible implementation and is not limited to a certain specific implementation.

[0080] The complete IO processing and erasure coding conversion process is as follows: Figure 4 As shown in the diagram, ① to ⑥ represent the sequential relationship of user data receiving from the storage system to the final persistent storage. The IO processing is illustrated using a high-speed cloud disk with 4KB sectors as an example; the process is similar for other storage systems, differing only in IO size.

[0081] ① When user data is written to the storage system, since the sector size of the high-speed cloud disk is 4KB, user IO is split into multiple 4KB segments. Each independent cloud disk has an IO Worker thread for processing, and each IO Worker thread handles data from multiple cloud disks. Meanwhile, the IO Gather, shared by multiple IO Workers, is responsible for deciding whether to wait for data to accumulate based on the data accumulation threshold of 8KB and the maximum data accumulation time of 8µs. If the data is less than 8KB or the accumulation wait time is less than 8µs, other data continues to be processed, and polling is used for checking. If the data accumulates to 8KB, it is handed over to the data compression processor. If the data accumulation wait time exceeds 8µs but the data is not yet 8KB, zero-padding is performed to bring the data to 8KB before it is handed over to the data compression processor for further processing.

[0082] The data compression processor is responsible for determining whether data compression is necessary based on user data characteristics and size. For example, if certain data characteristics indicate a low compression ratio and data compression wastes CPU resources, then no compression is chosen. The processor also selects the appropriate compression algorithm. It should be noted that after data compression, the amount of data previously sufficient for the algorithm may become insufficient, requiring continued waiting, or the I / O aggregator's data accumulation threshold may be increased. Specific solutions are related to technical implementation and do not affect the main flow of this embodiment of the invention.

[0083] ② After the data is processed and aggregated by the IO aggregation engine, it meets the data volume requirement of 4+2EC and is then handed over to the write-ahead log processor for further processing.

[0084] ③ The write-ahead log processor generates 4 data blocks and 2 checksum blocks according to the 4+2 EC algorithm and writes them to disk along with the transaction log in the write-ahead log format. After successful disk write, the IO write is complete.

[0085] ④ Once the user's IO write request is completed, notify the user that the IO write was successful.

[0086] ⑤ The erasure coding converter uses a polling method to continuously check whether the 4+2 EC format data in the write-ahead log has been converted to a certain 8+3 EC block. If so, it continues to check subsequent data until it finds unconverted 4+2 EC format data. If it finds that two 4+2 EC blocks have not been converted, the erasure coding conversion algorithm proposed in this embodiment generates three new parity blocks based on the original data blocks of the two 4+2 EC blocks.

[0087] ⑥ The erasure coding converter is responsible for updating and replacing the check blocks of the two 4+2 EC groups that have been converted, logically merging the original data block and the newly generated check block into a new 8+3 EC group, marking the previous 4+2 EC group as converted, and triggering garbage collection to reclaim storage space.

[0088] The present invention employs a storage system design based on an efficient erasure coding conversion algorithm and a data aggregation write engine. This system achieves rapid data aggregation through a joint write engine that efficiently aggregates multiple random small I / O operations. It reduces write latency by selecting a small proportion of erasure coding as temporary data storage for rapid disk write I / O. Then, based on the large amount of aggregated data, the temporary storage data is asynchronously converted into a large proportion of erasure coding files to improve storage efficiency. This achieves improved write efficiency of direct-write erasure coding files by updating only the data checksum without re-storing the original data, thus avoiding the problem of re-storing the original data in traditional methods and improving data storage efficiency.

[0089] See Figure 5 This is a schematic diagram of the structure of an erasure coding file processing device provided in an embodiment of the present invention. The embodiment of the present invention provides an erasure coding file processing device 10, comprising: The raw data receiving module 11 is used to receive raw data input by the user in real time; The first erasure coding generation module 12 is used to process the original data using a first proportional erasure coding algorithm to generate a first erasure coding file and store it in a preset memory; wherein, the first erasure coding file includes a plurality of first data blocks and at least one first check block obtained by dividing the original data; The second erasure coding generation module 13 is used to read a portion of the first data blocks and all the first check blocks from N first erasure coding files in the memory according to the second proportional erasure coding algorithm, and generate at least one second check block; N>1; and to merge the second check block and all the first data blocks in the N first erasure coding files and store them in the memory as the second erasure coding file.

[0090] Preferably, the device further includes: The raw data splitting module is used to split the raw data into several sector data blocks according to the preset cloud disk sector size; The multi-threaded acquisition module is used to acquire the sector data block in parallel using multiple data acquisition threads; The data aggregation processing module is used to aggregate the sector data blocks to generate several aggregated data blocks.

[0091] The first erasure coding generation module 12 is specifically used to process the aggregated data block using a first proportional erasure coding algorithm to generate a first erasure coding file.

[0092] It should be noted that the erasure coding file processing device provided in this embodiment of the invention is used to execute all the process steps of the erasure coding file processing method in the above embodiment. The working principle and beneficial effects of the two are one-to-one, so they will not be described again.

[0093] This invention also provides an erasure coding file processing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the erasure coding file processing method as described in any of the above embodiments.

[0094] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the erasure coding file processing method as described in any of the above embodiments.

[0095] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the erasure coding file processing method as described in any of the above embodiments.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0097] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for processing erasure-coded files, characterized in that, include: Receive raw data input by the user in real time; The original data is processed using a first proportional erasure coding algorithm to generate a first erasure coding file, which is then stored in a preset memory. The first erasure coding file includes several first data blocks and at least one first check block obtained by dividing the original data. According to the second proportional erasure coding algorithm, a portion of the first data blocks and all of the first check blocks from N first erasure coding files are read from the memory to generate at least one second check block; N > 1; The second verification block and all the first data blocks in the N first erasure coding files are merged and stored in the memory to form the second erasure coding file.

2. The erasure coding file processing method as described in claim 1, characterized in that, The step of reading a portion of the first data blocks and all of the first check blocks from N first erasure coding files according to the second proportional erasure coding algorithm, and generating at least one second check block, includes: According to the second proportional erasure coding algorithm, the number of second check blocks to be generated for the second erasure coding file is determined and denoted as the first target number. The number N of the first erasure coding files corresponding to the generation of one second erasure coding file is determined and each N first erasure coding files is combined into a file group. Based on the relationship between the N first erasure coding files and the second erasure coding files in the file combination, determine the number of the first data blocks to be read, denoted as the second target number; Read the first data blocks and all first check blocks that satisfy the second target number from the N first erasure coding files in the memory; Based on all the first verification blocks and the first data blocks of the second target number, a second verification block that satisfies the first target number is generated.

3. The erasure coding file processing method as described in claim 2, characterized in that, The step of determining the number of first data blocks to be read, denoted as the second target number, based on the relationship between the N first erasure coding files and the second erasure coding files in the file combination, includes: Construct a first generator matrix based on the first data block and the first check block of the N first erasure coding files; Based on the second data block of the second erasure coding file and the second verification block to be generated, construct the second generation matrix; The first generating matrix and the second generating matrix are decomposed into row vectors. Under the operation rules of the finite field, the coefficient matrix is ​​solved by Gaussian elimination to determine the linear correlation between the first check block of the N first erasure code files and the second check block of the second erasure code files. Based on the linear correlation, the number of the first data blocks to be read is determined and denoted as the second target number.

4. The erasure coding file processing method as described in claim 3, characterized in that, The step of generating a second verification block that satisfies the first target number based on all the first verification blocks and the first data blocks of the second target number includes: Based on all the first verification blocks and the first data blocks of the second target quantity, the first second verification block is generated through coefficient matrix operations; Based on all the first check blocks and the generated second check blocks, the next second check block is generated through linear operation until the number of second check blocks that meet the first target number is generated.

5. The erasure coding file processing method as described in claim 1, characterized in that, After receiving the raw data input by the user in real time, the method further includes: The original data is divided into several sector data blocks according to the preset cloud disk sector size; The sector data blocks are acquired in parallel using multiple data acquisition threads; The sector data blocks are aggregated to generate several aggregated data blocks; The process of using the first proportional erasure coding algorithm to process the original data and generate the first erasure coding file specifically involves: The aggregated data block is processed using a first proportional erasure coding algorithm to generate a first erasure coding file.

6. The erasure coding file processing method as described in claim 5, characterized in that, The aggregation process of the sector data blocks to generate several aggregated data blocks includes: The sector data blocks are aggregated using a preset aggregation strategy to generate several aggregated data blocks; The preset aggregation strategy is as follows: within a preset maximum waiting time, if the sector data blocks received in real time meet the preset aggregation quantity threshold, the sector data blocks that meet the preset aggregation quantity threshold will be aggregated; if the sector data blocks received in real time do not meet the preset aggregation quantity threshold, the existing sector data blocks will be padded with zeros until they meet the preset aggregation quantity threshold before aggregation.

7. The erasure coding file processing method as described in claim 6, characterized in that, The aggregation process of the sector data blocks to generate several aggregated data blocks further includes: Based on the data characteristics of the aggregated data block, a preset data compression algorithm is used to compress the aggregated data block to obtain a compressed aggregated data block.

8. The erasure coding file processing method as described in claim 1, characterized in that, After processing the original data using a first proportional erasure coding algorithm to generate a first erasure coding file and storing it in a preset memory, the method further includes: A response message indicating successful data writing is returned to the user.

9. An erasure coding file processing device, characterized in that, include: The raw data receiving module is used to receive raw data input by the user in real time. The first erasure coding generation module is used to process the original data using a first proportional erasure coding algorithm to generate a first erasure coding file and store it in a preset memory; wherein, the first erasure coding file includes a plurality of first data blocks and at least one first check block obtained by dividing the original data; The second erasure coding generation module is used to read a portion of the first data blocks and all the first check blocks from N first erasure coding files in the memory according to the second proportional erasure coding algorithm, and generate at least one second check block; N>1; and to merge the second check block and all the first data blocks in the N first erasure coding files into the memory as the second erasure coding file.

10. An erasure coding file processing device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the erasure coding file processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the erasure coding file processing method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the erasure coding file processing method as described in any one of claims 1 to 8.

Images