Erasure code data storage method, device, equipment and medium

Abstract

The present application relates to the field of data storage technology, and specifically to a method, apparatus, device, and medium for erasure code data storage, which is used to improve the utilization of storage space. The method includes: if the first data is not a full stripe of data, padding the first data with padding data according to the length of the first stripe corresponding to the first data to obtain the padded first data of the same length as the first stripe, encoding the padded first data to obtain a first check block; writing the first data into the storage space corresponding to the first stripe, creating a hole in the storage space corresponding to the first stripe, and writing the first check block into other storage spaces; the first data and the hole fill the storage space corresponding to the first stripe; when reading the first data, if the first data is damaged, restoring the first data based on the other data and the first check block; the other data is obtained by padding the undamaged data in the first data with the padding data according to the position of the hole.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and specifically to a method, apparatus, device, and medium for erasure code data storage. Background Art

[0002] Erasure Coding (EC) is a coding error-tolerant technology first used in the communications industry to address data loss during transmission. Its basic principle is to segment the transmitted signal and add certain checks to correlate the segments. This allows the receiving end to still reconstruct the complete information even if some signal is lost during transmission.

[0003] In data storage, erasure coding divides data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. Conventional technology aligns data to full stripes when writing data that is not full, fills the leading and trailing holes with zeros, and then writes the padded data to disk. This results in a certain amount of write amplification, leading to low storage space utilization. Summary of the Invention

[0004] The embodiments of the present application provide a data storage method, apparatus, device, and medium based on erasure codes, for improving storage space utilization.

[0005] In a first aspect, the present application provides a data storage method based on erasure coding, comprising:

[0006] If the first address offset or data length of the first data to be stored is not an integer multiple of the stripe length, determining the first stripe corresponding to the first data according to the first address offset, data length and stripe length of the first data;

[0007] According to the length of the first stripe, padding the first data with padding data to obtain padded first data, and encoding the padded first data using a preset error correction code algorithm to obtain a first check block; wherein the length of the padded first data is equal to the length of the first stripe;

[0008] Writing the first data into the storage space corresponding to the first stripe, creating a hole in the storage space corresponding to the first stripe, and writing the first parity block into other storage spaces; wherein the first data and the hole fully occupy the storage space corresponding to the first stripe;

[0009] When reading the first data, if part of the first data is damaged, the first data is restored based on other data and the first check block; wherein the other data is obtained by filling the undamaged data in the first data with the filling data according to the position of the hole.

[0010] In this embodiment of the present application, if the first address offset or data length of the first data to be stored is not an integer multiple of the stripe length, that is, when writing the first data to a partially full stripe, after obtaining the first check block based on the padded first data, only the first data is written to the storage space, creating a hole. Compared to the prior art method of directly writing the padded first data to the storage space, this relatively reduces write operations, can reduce write amplification, and also reduces storage space waste, thereby improving storage space utilization.

[0011] In a possible embodiment, after writing the first check block into other storage space, the method further includes:

[0012] receiving a request to overwrite the first stripe with second data to be stored;

[0013] If the first address offset or data length of the second data is not an integer multiple of the stripe length, the target data is read from the storage space corresponding to the target stripe unit, and a second parity block is obtained based on the target data and the second data; wherein the target stripe unit is the smaller stripe unit of the first stripe unit and the second stripe unit required for writing the second data; and the second stripe unit is the remaining stripe unit in the first stripe except the first stripe unit.

[0014] The second data is written into the storage space corresponding to the first stripe unit, overwriting the original data in the storage space corresponding to the first stripe unit, and the second check block is written into the storage space corresponding to the first check block, overwriting the first check block.

[0015] In an embodiment of the present application, comparing the number of the first stripe units and the second stripe units is equivalent to comparing the number of stripe units required for writing the second data and the number of stripe units that will not be occupied, and reading the target data in the storage space corresponding to the smaller number of stripe units to calculate a new check block can minimize read request operations.

[0016] In a possible embodiment, reading target data from a storage space corresponding to a target stripe unit and obtaining a second check block according to the target data and the second data includes:

[0017] If the second stripe unit is the target stripe unit, reading the target data from the storage space corresponding to the second stripe unit;

[0018] The target data and the second data are encoded using the error correction code algorithm to obtain the second check block.

[0019] In an embodiment of the present application, the target data is read from the storage space corresponding to the second stripe unit that will not be occupied by the second data writing, which is equivalent to reading only the uncovered stripe units in a stripe without reading the entire stripe, thereby reducing the reading range.

[0020] In a possible embodiment, reading target data from a storage space corresponding to a target stripe unit and obtaining a second check block according to the target data and the second data includes:

[0021] If the first stripe unit is the target stripe unit, reading the target data from a storage space corresponding to the first stripe unit;

[0022] Performing an XOR operation on the target data and the second data, and encoding them using the error correction code algorithm to obtain an incremental check block;

[0023] An exclusive OR operation is performed on the incremental check block and the first check block to obtain the second check block.

[0024] In an embodiment of the present application, the target data is read from the storage space corresponding to the first stripe unit required for writing the second data, and a new check block is calculated locally by using the difference between the target data and the second data through an XOR operation, thereby eliminating the need for remote transmission of the check block.

[0025] In a possible embodiment, after writing the first check block into other storage space, the method further includes:

[0026] receiving a request to read third data included in the first data;

[0027] If the first address offset or data length of the third data is not an integer multiple of the stripe length, determining a third stripe unit corresponding to the third data according to the first address offset, data length, the stripe length, and stripe unit length of the third data;

[0028] The third data is read from the storage space corresponding to the third stripe unit.

[0029] In an embodiment of the present application, the reading granularity is reduced to the stripe unit level, and the third data is read by stripe unit, which can reduce unnecessary diffusion of read requests and reduce read request operations compared to the full stripe reading method.

[0030] In a second aspect, the present application provides an erasure coded data storage device, comprising:

[0031] a determination module configured to determine, if a first address offset or a data length of the first data to be stored is not an integer multiple of the stripe length, a first stripe corresponding to the first data according to the first address offset, the data length, and the stripe length of the first data;

[0032] an obtaining module, configured to pad the first data with padding data according to the length of the first stripe to obtain padded first data, and encode the padded first data using a preset error correction code algorithm to obtain a first check block; wherein the length of the padded first data is equal to the length of the first stripe;

[0033] a writing module, configured to write the first data into a storage space corresponding to the first stripe, create a hole in the storage space corresponding to the first stripe, and write the first parity block into another storage space; wherein the first data and the hole completely occupy the storage space corresponding to the first stripe;

[0034] A reading module is used to restore the first data based on other data and the first check block when reading the first data if part of the first data is damaged; wherein the other data is obtained by filling the undamaged data in the first data with the filling data according to the position of the hole.

[0035] In a possible embodiment, the apparatus further includes a receiving module, configured to: after writing the first check block into other storage space, receive a request to overwrite the first stripe with second data to be stored;

[0036] The obtaining module is further configured to: if the first address offset or data length of the second data is not an integer multiple of the stripe length, read the target data from the storage space corresponding to the target stripe unit, and obtain a second parity block based on the target data and the second data; wherein the target stripe unit is the smaller stripe unit of the first stripe unit and the second stripe unit required for writing the second data; and the second stripe unit is the remaining stripe unit in the first stripe except the first stripe unit;

[0037] The writing module is further used to: write the second data into the storage space corresponding to the first stripe unit, overwriting the original data in the storage space corresponding to the first stripe unit, and write the second check block into the storage space corresponding to the first check block, overwriting the first check block.

[0038] In a possible embodiment, the obtaining module is specifically configured to:

[0039] If the second stripe unit is the target stripe unit, reading the target data from the storage space corresponding to the second stripe unit;

[0040] The target data and the second data are encoded using the error correction code algorithm to obtain the second check block.

[0041] In a possible embodiment, the obtaining module is specifically configured to:

[0042] If the first stripe unit is the target stripe unit, reading the target data from a storage space corresponding to the first stripe unit;

[0043] Performing an XOR operation on the target data and the second data, and encoding them using the error correction code algorithm to obtain an incremental check block;

[0044] An exclusive OR operation is performed on the incremental check block and the first check block to obtain the second check block.

[0045] In a possible embodiment, the receiving module is further configured to: after writing the first check block into other storage space, receive a request to read third data included in the first data;

[0046] The determining module is further configured to: if the first address offset or the data length of the third data is not an integer multiple of the stripe length, determine a third stripe unit corresponding to the third data according to the first address offset, the data length, the stripe length, and the stripe unit length of the third data;

[0047] The reading module is further configured to read the third data from the storage space corresponding to the third stripe unit.

[0048] In a third aspect, the present application provides an electronic device, comprising:

[0049] a memory for storing program instructions;

[0050] The processor is configured to call the program instructions stored in the memory and execute the method according to any one of the first aspects according to the obtained program instructions.

[0051] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of erasure code data writing in the prior art;

[0053] Figure 2 Schematic diagram of erasure code data reading in the prior art;

[0054] Figure 3 A schematic diagram illustrating the relationship between stripes, blocks, and shards provided in an embodiment of the present application;

[0055] Figure 4 Creating an object diagram for erasure codes in the prior art;

[0056] Figure 5 A schematic diagram of a process flow of an erasure code data storage method provided in an embodiment of the present application;

[0057] Figure 6 Schematic diagram of erasure code creation object provided in an embodiment of the present application;

[0058] Figure 7 Schematic diagram of erasure code reading data in the prior art;

[0059] Figure 8 Schematic diagram of erasure code reading data provided in an embodiment of the present application;

[0060] Figure 9 Schematic diagram of erasure code overwriting in the prior art;

[0061] Figure 10 A schematic diagram of a partial write of the erasure code provided in an embodiment of the present application;

[0062] Figure 11 A schematic diagram of a partial update of an erasure code provided in an embodiment of the present application;

[0063] Figure 12 A schematic diagram of creating an erasure code column storage object provided in an embodiment of the present application;

[0064] Figure 13 Schematic diagram of overwriting erasure code column storage objects provided in an embodiment of the present application;

[0065] Figure 14 A schematic diagram of the basic flow of the erasure code data processing method according to an embodiment of the present application;

[0066] Figure 15 This is a schematic diagram of the processing flow of newly written data in an embodiment of the present application;

[0067] Figure 16 This is a schematic diagram of the data reading process according to an embodiment of the present application;

[0068] Figure 17 This is a schematic diagram of the processing flow of overwriting data in an embodiment of the present application;

[0069] Figure 18A schematic diagram comparing the steady-state performance before and after optimization of the 8k random write long stability test provided in an embodiment of the present application;

[0070] Figure 19 A structural diagram of an erasure code data storage device provided in an embodiment of the present application;

[0071] Figure 20 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to better understand the technical solutions provided by the embodiments of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0073] In the field of data storage, error control functions are categorized into three types: error detection codes, error correction codes, and erasure codes. Error detection codes only identify errors but do not correct them. Error correction codes not only identify but also correct them. Erasure correction codes not only identify and correct errors but also delete information that cannot be corrected if the error exceeds the correction range.

[0074] An erasure code consists of k data blocks and m parity blocks, which can be expressed as: n = k + m. The variable k represents the value of the original data or symbol, and the variable m represents the value of the additional or redundant symbols added to provide protection after a failure. The variable n represents the total value of the symbols created after the erasure coding process. Both data blocks and parity blocks can be referred to as storage blocks. If some storage blocks are damaged, the data of the damaged blocks can be calculated based on the data of the remaining storage blocks, thus ensuring that all data is not lost.

[0075] This involves the values of m and k. The m value controls how many failures can be tolerated before data is lost. The k value determines how many blocks each object's data will be striped into. A larger k value means lower storage overhead, but it also means that any write or read operation requires communication with more devices, increasing data read and write latency. The m value affects reliability and storage cost. A larger m value increases fault tolerance, while a smaller m value reduces data redundancy.

[0076] Taking the distributed storage system Ceph as an example, the following describes how erasure codes are applied to distributed storage systems.

[0077] Please refer to Figure 1, which is a schematic diagram of erasure code data writing in the prior art. Taking k=2 and m=1 as an example, after the client uploads an object named photo.jpg to the distributed storage system Ceph, it will call the corresponding erasure code algorithm in the primary object storage device (OSD) to encode the data of the object. Since k=2, the original content of the object ABCDEFGH is split into two shards, the content of stripe shard 1 is ABCD, and the content of stripe shard 2 is EFGH. The content of check stripe shard 3 is then calculated to be WXYZ. According to the rules specified by the data distribution map (crushmap) of the Ceph cluster, these three stripe shards are randomly distributed in three different object storage devices (OSD) to complete the storage operation of the photo.jpg object.

[0078] Please refer to Figure 2 , which is a schematic diagram of erasure code data reading in the prior art. After the client initiates a request to read photo.jpg, the primary OSD of this object (such as OSD1) will initiate a read request to other associated OSDs (such as OSD2 and OSD3). When the request is sent to OSD2 and OSD3, OSD2 fails and cannot respond to the request. In the end, it can only obtain the stripe shards of OSD1 (with the content of ABCD) and the stripe shards of OSD3 (with the content of WXYZ). At this time, OSD1, as the primary OSD, will perform erasure code decoding operations on the stripe shards of OSD1 and OSD3, calculate the content of OSD2's stripe shards (i.e., EFGH), reassemble the new content of photo.jpg (i.e., ABCDEFGH), and return the result to the client.

[0079] The above describes the writing and reading process of erasure coded data in the prior art, which involves basic concepts such as stripes and shards. These basic concepts are introduced below.

[0080] 1. Stripe: If the object to be encoded is too large to be encoded in one go, it can be divided into multiple passes. Each completed encoding part is called a stripe. The data in the stripes of the same object are continuous.

[0081] 2. Chunk, also known as stripe_unit: When an object is encoded based on erasure codes, multiple check blocks can be calculated through several data blocks each time the encoding is performed. Each block is the same size, and multiple data blocks form a stripe.

[0082] 3. Shard: Blocks with the same serial number in the same object are located on the same disk and constitute a shard of the object. The data between blocks within a shard is discontinuous.

[0083] Please refer to Figure 3 , which is a schematic diagram of the relationship between stripes, blocks, and shards provided in an embodiment of the present application. It can be seen that the erasure code data adopts a 4+2 structure, that is, k=4, m=2, and each stripe includes 4 data blocks, and these 4 data blocks are encoded to obtain 2 check blocks. That is, chunk1, chunk2, chunk3, and chunk4 in the first row constitute stripe1, and chunk1, chunk2, chunk3, and chunk4 of stripe1 are encoded to obtain chunk5 and chunk6 in the first row. chunk1, chunk2, chunk3, and chunk4 in the second row constitute stripe2, and chunk1, chunk2, chunk3, and chunk4 of stripe2 are encoded to obtain chunk5 and chunk6 in the second row. Each cylinder represents a disk, and the dotted box represents a shard, which includes multiple blocks with the same serial number, and one shard corresponds to one disk.

[0084] All writes to erasure-coded pool objects must update the entire stripe (all k+m data servers), which typically requires communication with more data servers than replicas, increasing latency. If a write only updates a portion of a stripe, the entire stripe must be read for the update, re-encoded, and then written to disk again. This results in lower I / O performance for small data in erasure-coded pools than in replicated pools.

[0085] Please refer to Figure 4, which is a schematic diagram of object creation for erasure codes in the prior art. In the scenario where k=4 and m=2, the length of each stripe unit is stripe_unit=4k, the length of each stripe is 16k, and each stripe includes 4 stripe units (blocks). When new data with a length of 17k is written, it is necessary to first create a data object (object) with a data length of 17k, in which chunk1 of stripe1 and chunk1 of stripe2 will be spliced into shard1 and written together to the first disk, and the remaining 15kb space in stripe2 will be filled with 0. The data actually written includes data of 8 stripe units and 4 check blocks, that is, the actual written data length is 48k, and the actual space utilization rate of 17 / 48=35.4% is much smaller than the theoretical value of 4 / 6=66.7%. When the length of the newly written data is much smaller than stripe_unit=4k, the space utilization rate will approach 0%. Moreover, in order to write 17KB of data to 6 disks, 6 data read and write (IO) requests need to be issued, which is much larger than the 3 IO requests for 3 replicas.

[0086] In order to improve the utilization of storage space, an embodiment of the present application provides a method for storing erasure coded data, which can be performed by an electronic device. The electronic device can be implemented through a terminal or a server, and the terminal can be a mobile terminal, a fixed terminal, or a portable terminal, such as a mobile phone, a multimedia computer, a multimedia tablet, a desktop computer, a notebook computer, a tablet computer, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, but is not limited to these.

[0087] The following describes the erasure code data storage method provided by the embodiment of the present application by taking the erasure code data storage method performed by the electronic device as an example. Figure 5 , which is a flow chart of a method for storing erasure coded data provided in an embodiment of the present application.

[0088] S501: If the first address offset or data length of first data to be stored is not an integer multiple of the stripe length, determine a first stripe corresponding to the first data according to the first address offset, data length and stripe length of the first data.

[0089] Specifically, when the electronic device receives a request to write first data to be stored, it determines whether the first data is data of a full stripe based on whether the first address offset and data length of the first data are integer multiples of the stripe length. If the first address offset or data length of the first data is not an integer multiple of the stripe length, indicating that the first data is data of a non-full stripe, the first stripe corresponding to the first data is determined based on the first address offset, data length, and stripe length of the first data.

[0090] S502 : Fill the first data with filling data according to the length of the first stripe to obtain the filled first data, and encode the filled first data using a preset error correction code algorithm to obtain a first check block.

[0091] Specifically, if the electronic device determines that the first data is data of a non-full stripe, it fills the leading and trailing holes of the first data with padding data such as 0s and 1s based on the length of the first stripe to obtain padded first data. The padded first data is then encoded using a preset error correction algorithm to obtain a first check block. The length of the padded first data is equal to the length of the first stripe, and the amount of padding data padded before the first data is equal to the starting address offset of the first data.

[0092] For example, the first address offset of the first data is offset=1k, the data length is length=7k, and the length of each stripe is 16k. Then 1 zero is padded before the first data, and 8 zeros are padded after the first data. Then, the data length of the first data after padded is 16k, which is the same as the stripe length.

[0093] S503: Write the first data into the storage space corresponding to the first stripe, create a hole in the storage space corresponding to the first stripe, and write the first check block into other storage spaces.

[0094] A hole is equivalent to an empty folder, storing no data. The electronic device uses the OP_ZERO mechanism of the underlying storage to create a hole, so that the first data and the hole occupy the storage space corresponding to the first stripe, reducing disk IO pressure and space waste.

[0095] S504: When reading the first data, if part of the first data is damaged, restore the first data based on other data and the first check block.

[0096] Specifically, the other data is obtained by filling undamaged data in the first data with padding data based on the location of the holes. When the electronic device reads the first data, if some of the first data is damaged, it can read the undamaged data in the first data from the storage space corresponding to the first stripe and fill the undamaged data in the first data with padding data based on the location of the holes to obtain the other data. The first check block is then read from the other storage space, and the other data and the first check block are decoded using a preset error correction code to recover the first data.

[0097] It should be noted that S503 and S504 are optional. In a possible embodiment, if the first address offset and data length of the first data are both integer multiples of the stripe length, indicating that the first data is full stripe data, the first data is directly encoded using an error correction code algorithm to obtain a third check block, the first data is written into the storage space corresponding to the first stripe, and the third check block is written into other storage spaces. This process can be referred to Figure 1 and Figure 2 The content of the discussion will not be repeated here.

[0098] Please refer to Figure 6 , which is a schematic diagram of the erasure code creation object provided in the embodiment of the present application. In the scenario where k=4 and m=2, the length of each stripe unit is stripe_unit=4k, the length of each stripe is 16k, and each stripe includes 4 stripe units (blocks). When the first data with a new data length of 7k is written, it is necessary to first create a data object (object) with a data length of 7k. Since the first data of 7k cannot occupy the entire 16k stripe, the OP_ZERO mechanism of the underlying storage is used to create a hole. When calculating the check blocks code1 and code2, the first data is still filled with 0, but when the transaction is sent to the data plane service, the part filled with 0 is turned into an OP_ZERO operation. For Figure 6 For the second stripe unit in the transaction, the transaction will carry two operations (OP): a write data operation (write data) and a create hole operation (do zero).

[0099] The above describes the process of how to newly write the first data. The following describes how to read the third data included in the first data.

[0100] The read algorithm for erasure codes typically reads the entire stripe. Even if only a small range within an object is read, the entire stripe of data is read. This leads to two problems: 1. Increased disk I / O waste on the data service. 2. The proliferation of read requests wastes network bandwidth and increases overall read latency.

[0101] Please refer to Figure 7, which is a schematic diagram of reading data by erasure code in the prior art. In the scenario where k = 4 and m = 2, the length of each stripe unit is stripe_unit = 4k, the length of each stripe is 16k, and each stripe includes 4 stripe units. For the first data of 34k, the third data to be read from it is offset = 17K and length = 18K. According to the stripe alignment rule, the actual stripe data to be read is new_offset = 16k and new_length = 32k, that is, read two stripes, stripe2 and stripe3. It can be seen that: new_offset < offset and new_length > length, and the actually read data is more than the data to be read.

[0102] Regarding the read amplification problem of non-full stripes, the embodiments of the present application provide an erasure code data reading method. After the electronic device receives a request to read the third data included in the first data, it determines to use different methods to read the third data according to whether the third data is full-stripe data. The following is an introduction by cases.

[0103] In the first case, the stripe alignment rule is used to read the third data.

[0104] If both the start address offset and the data length of the third data are not integer multiples of the stripe length, then according to the start address offset, data length, and stripe length of the third data, determine the target stripe corresponding to the third data, and read the third data from the storage space corresponding to the target stripe.

[0105] In the second case, the stripe unit alignment rule is used to read the third data.

[0106] If either the start address offset or the data length of the third data is not an integer multiple of the stripe length, then according to the start address offset, data length, stripe length, and stripe unit length of the third data, determine the third stripe unit corresponding to the third data, and read the third data from the storage space corresponding to the third stripe unit.

[0107] Please refer to Figure 8 , which is a schematic diagram of erasure code reading data provided by the embodiments of the present application. In the scenario where k = 4 and m = 2, the length of each stripe unit is stripe_unit = 4k, the length of each stripe is 16k, and each stripe includes 4 stripe units. For the first data of 7k, the third data to be read is offset = 1k and length = 3k. According to the block alignment, that is, the stripe unit alignment rule, the actual stripe unit data to be read is offset = 0 and length = 4k, that is, the data in the first stripe unit corresponding to the black rectangular box. The first stripe unit corresponds to one disk, so only this one disk needs to be read to obtain the third data.

[0108] The above describes the process of how to read the third data included in the first data. The following describes how to overwrite the second data based on the first data that has been written.

[0109] Overwriting is the most performance-intensive operation in erasure coding. However, because erasure codes store data in full stripes, existing technologies require stripe alignment when writing new data, and read all the blocks in the stripe that the new data will occupy. In reality, because the data in the written area will be overwritten and does not need to participate in the calculation of the check block, this data does not need to be read, resulting in a certain amount of read amplification. Furthermore, when writing data smaller than a single block, it also needs to be stripe-aligned before being written to the disk, resulting in write amplification and ultimately reduced performance.

[0110] Please refer to Figure 9 , which is a schematic diagram of erasure code overwrite in the prior art. For a first data piece with a length of 34KB, the second data piece to be overwritten has an offset of 30KB and a length of 20KB. According to the stripe alignment rules, the stripe data to be read is offset of 16KB and a length of 32KB, i.e., the data from stripe2 and stripe3. The data from stripe2 and stripe3 is then merged with the second data piece, and the parity block is recalculated. This effectively means that two stripes of data need to be read and three stripes of data need to be written.

[0111] To address the issue of penalty reading and writing due to overwriting, an embodiment of the present application provides a method for overwriting erasure coded data. The specific steps are as follows:

[0112] S1.1. Receive a request to overwrite the first stripe with second data to be stored.

[0113] After receiving a request to overwrite the first stripe with the second data to be stored, the electronic device may determine whether the second data is data that fills the stripe.

[0114] S1.2. If the first address offset or data length of the second data is not an integer multiple of the stripe length, read the target data from the storage space corresponding to the target stripe unit, and obtain a second check block based on the target data and the second data.

[0115] The target stripe unit is a smaller number of the first stripe unit and the second stripe unit required for writing the second data; the second stripe unit is the remaining stripe unit in the first stripe except the first stripe unit.

[0116] Specifically, if the first address offset or data length of the second data is not an integer multiple of the stripe length, indicating that the second data does not fill the stripe, the electronic device can determine the first stripe unit required for writing the second data based on the first address offset, data length, stripe length, and stripe unit length of the second data, and further determine the remaining second stripe units in the first stripe excluding the first stripe unit. The target stripe unit is then determined based on the number of the first stripe units and the second stripe units.

[0117] The electronic device obtains the second check block in different ways according to different target stripe units, which are described below in different cases.

[0118] Case 1: If the number of the second stripe units is smaller than the number of the first stripe units, the second stripe unit is determined as the target stripe unit, and a partial write method is used to obtain the second parity block.

[0119] If the second stripe unit is the target stripe unit, the target data is read from the storage space corresponding to the second stripe unit, and the target data and the second data are encoded using an error correction code algorithm to obtain a second check block.

[0120] Please refer to Figure 10 , which is a schematic diagram of partial writing of the erasure code provided in an embodiment of the present application. The second data to be overwritten is offset=21k, length=20k (i.e., the stripe unit data corresponding to the gray rectangular box). First, the uncovered stripe unit data (i.e., the stripe unit data corresponding to the black rectangular box) is read, and then the read stripe unit data is merged with the second data to be overwritten, and encoded through the erasure code algorithm (EC encode) to calculate the new check block.

[0121] Case 2: If the number of the first stripe units is smaller than the number of the second stripe units, the second stripe unit is determined as the target stripe unit, and a partial update method is used to obtain the second check block.

[0122] If the first stripe unit is the target stripe unit, the target data is read from the storage space corresponding to the first stripe unit, an XOR operation is performed on the target data and the second data, and the error correction code algorithm is used to encode the data to obtain an incremental check block, and an XOR operation is performed on the incremental check block and the first check block to obtain a second check block.

[0123] Please refer to Figure 11, which is a schematic diagram of the partial update of the erasure code provided in an embodiment of the present application. The second data to be overwritten is offset=30k, length=4k, and the first stripe unit required for writing the second data is the stripe unit corresponding to the black rectangular box. First, the stripe unit data to be overwritten is read (i.e., the stripe unit data corresponding to the gray rectangular box), and then the read stripe unit data (i.e., the stripe unit data corresponding to the gray rectangular box) and the second data to be overwritten are XORed with the data (i.e., the stripe unit data corresponding to the black rectangular box) to calculate the change before and after the modification, and then the change of the check block (i.e., the incremental check block) is calculated based on the change before and after the modification. Finally, the change of the check block (i.e., the incremental check block) is XORed with the original first check block to obtain a new check block (i.e., the second check block).

[0124] In case 3, if the number of the second stripe units is equal to the number of the first stripe units, the second stripe unit or the first stripe unit can be determined as the target stripe unit, and a partial write or partial update method can be used to obtain the second parity block. For details on partial writes or partial updates, please refer to the previous section and will not be repeated here.

[0125] S1.3. Write the second data into the storage space corresponding to the first stripe unit, overwriting the original data in the storage space corresponding to the first stripe unit, and write the second check block into the storage space corresponding to the first check block, overwriting the first check block.

[0126] Specifically, after obtaining the second check block, the electronic device overwrites the original data in the storage space corresponding to the first stripe unit with the second data, and overwrites the first check block with the second check block, thereby successfully writing the second data into the first stripe.

[0127] The above description applies to read and write scenarios involving non-full stripes. In full stripe read and write scenarios, if the read and write range exceeds a single stripe, read and write operations will inevitably be spread, requiring read and write operations from k data services to complete the read and write. The fundamental reason for this is that the data on a single data service is discontinuous. For example, a stripe consists of four stripe units, which are distributed across four different disks. Reading data from this stripe requires issuing read requests to all four disks.

[0128] In block storage scenarios, a block device is a continuous space that corresponds to multiple objects in a unified storage layer, each of which has a fixed size. Therefore, in one possible embodiment, one stripe corresponds to one storage space. When the first stripe consists of multiple stripes, the data corresponding to each first stripe is written to a different storage space.

[0129] In an embodiment of the present application, data row storage is converted into column storage, so that the data on a data service is continuous. During the reading and writing process, the read and write amplification is reduced, and the communication load with the data service is reduced. Especially in the overlay write scenario, the remote transmission of the check block is reduced, and the calculation of the check block is performed locally.

[0130] Please refer to Figure 12 , which is a schematic diagram of creating an erasure code column storage object provided in an embodiment of the present application. In the scenario where k=4 and m=2, the length of each stripe unit is stripe_unit=256k, and each stripe includes 4 stripe units, then the length of each stripe is 4*256=1m. The data to be written is offset=512k, length=768k. In the process of creating the object, the range of the data to be written in the entire object is calculated by offset and length. The data to be written occupies shard1 (i.e. stripe1) and shard2 (i.e. stripe2), and the rest of the space is filled with 0. Write the data of shard1 (i.e. stripe1) to one disk, and write the data of shard2 (i.e. stripe2) to another disk.

[0131] Please refer to Figure 13 , which is a schematic diagram of overwrite write of erasure code column storage objects provided in an embodiment of the present application. In the scenario where k=4 and m=2, the length of each stripe unit is stripe_unit=256k, each stripe includes 4 stripe units, and the length of each stripe is 4*256=1m. The newly written data is offset=3328k, length=5128k. First, the data to be overwritten (i.e., the stripe unit data corresponding to the black rectangular box) is read, and then the read data and the newly written data (i.e., the stripe unit data corresponding to the gray rectangular box) are XORed to calculate the incremental check block. The incremental check block is XORed with the original check block to obtain a new check block. This data arrangement method can basically achieve performance similar to that of three copies in a small range of read and write operations.

[0132] To more clearly illustrate the erasure code data storage method of the present invention, please refer to Figure 14 , which is a basic flow chart of a method for processing erasure code data according to an embodiment of the present application.

[0133] S1401. Create a data object.

[0134] Create data objects in a distributed storage system.

[0135] S1402: New data is written.

[0136] Upon receiving a request to write new data, the new data is written into the data object.

[0137] S1403: Read data.

[0138] Upon receiving a request to read data, the data is read from the data object.

[0139] S1404: Overwrite the written data.

[0140] Upon receiving a request to overwrite data, the data is overwritten into the data object.

[0141] Please refer to Figure 15 , is a schematic diagram of the processing flow of newly written data provided in an embodiment of the present application, and the specific steps are described as follows.

[0142] S1501. Calculate the first data to be written.

[0143] Calculate the offset and length of the first data after stripe alignment and the corresponding stripe.

[0144] S1502: Determine whether the first data is full-strip data.

[0145] If the first data is full stripe data, then S1503 is executed. If the first data is not full stripe data, then S1506 is executed. How to determine whether the first data is full stripe data can be referred to the above discussion and will not be repeated here.

[0146] S1503: Calculate the third check block.

[0147] The first data is encoded using an error correction code algorithm to obtain a third check block.

[0148] S1504: Generate a first transaction, including a data write operation and a hole creation operation.

[0149] S1505: Submit the first transaction to the data service that manages the disk.

[0150] S1506: Calculate the first check block.

[0151] According to the length of the first stripe, the first data is padded with padding data to obtain the padded first data, and the padded first data is encoded using a preset error correction code algorithm to obtain a first check block.

[0152] S1507: Generate a second transaction, which only includes a data write operation.

[0153] S1508: Submit the second transaction to the data service that manages the disk.

[0154] Please refer to Figure 16, is a schematic diagram of the processing flow of reading data provided in an embodiment of the present application, and the specific steps are described as follows.

[0155] S1601: Calculate the third data to be read.

[0156] Calculate the offset, length, and corresponding stripe of the third data after stripe alignment.

[0157] S1602: Determine whether the third data is full-strip data.

[0158] If the third data is full stripe data, then S1603 is executed. If the third data is not full stripe data, then S1605 is executed. How to determine whether the third data is full stripe data can be referred to the above discussion and will not be repeated here.

[0159] S1603: Generate a third transaction according to stripe alignment, including a data read operation.

[0160] S1604: Submit the third transaction to the data service that manages the disk.

[0161] S1605: Generate a fourth transaction according to the stripe unit alignment, including a data read operation.

[0162] S1606: Submit the fourth transaction to the data service that manages the disk.

[0163] Please refer to Figure 17 , is a schematic diagram of the processing flow of overwriting data provided in an embodiment of the present application, and the specific steps are described as follows.

[0164] S1701. Calculate the second data to be overwritten.

[0165] Calculate the offset, length, and corresponding stripe of the second data after stripe alignment.

[0166] S1702: Determine whether the second data is full-strip data.

[0167] If the second data is full stripe data, then S1703 is executed. If the second data is not full stripe data, then S1706 is executed. How to determine whether the second data is full stripe data can be referred to the above discussion and will not be repeated here.

[0168] S1703: Calculate the fourth check block.

[0169] The second data is encoded using an error correction code algorithm to obtain a fourth check block.

[0170] S1704: Generate a fifth transaction, including a data write operation and a hole creation operation.

[0171] S1705: Submit the fifth transaction to the data service that manages the disk.

[0172] S1706: Calculate the number of stripe units that need to be read for partial writing and partial updating.

[0173] The stripe unit that needs to be read for partial writes is the second stripe unit discussed above, and the stripe unit that needs to be read for partial updates is the first stripe unit discussed above. The calculation of the number of first and second stripe units is discussed above and will not be repeated here.

[0174] S1707: Determine whether the number of stripe units read by the partial write is less than the number of stripe units read by the partial update.

[0175] Determine whether the second stripe unit is smaller than the first stripe unit. If so, execute S1708; if not, execute S1712.

[0176] S1708: Read data of uncovered stripe units.

[0177] The uncovered stripe unit is the second stripe unit discussed above, and the target data is read from the storage space corresponding to the second stripe unit.

[0178] S1709. Calculate the check block for the read part and the newly written part.

[0179] The target data and the second data are encoded using an error correction code algorithm to obtain a second check block.

[0180] S1710: Generate a sixth transaction, which only includes a data write operation.

[0181] S1711. Submit the sixth transaction to the data service that manages the disk.

[0182] S1712: Read the data of the stripe unit to be overwritten.

[0183] The stripe unit to be overwritten is the first stripe unit discussed above, and the target data is read from the storage space corresponding to the first stripe unit.

[0184] S1713. Calculate the change in the data to be overwritten before and after modification.

[0185] Perform an XOR operation on the target data and the second data to obtain the change.

[0186] S1714: Calculate the change in the check block.

[0187] The change amount is encoded using an error correction code algorithm to obtain an incremental check block.

[0188] S1715: Calculate the check block.

[0189] An XOR operation is performed on the incremental check block and the first check block to obtain a second check block.

[0190] S1716: Generate a seventh transaction, which only includes a data write operation.

[0191] S1717. Submit the seventh transaction to the data service that manages the disk.

[0192] In order to simulate the current network business scenario, considering the MySQL database intensive small I / O write scenario, a long-term 8K random write test (EC4+2 model) was conducted to verify the steady-state performance before and after optimization. Figure 18 , which is a schematic diagram comparing the steady-state performance before and after the optimization of the 8k random write long stability test provided in the embodiment of the present application. The curve obtained by the error-correcting code data storage method provided in the embodiment of the present application is the performance curve after optimization, and the curve obtained by the prior art is the performance curve before optimization. The trend of the optimized performance curve is stable and much higher than the performance curve before optimization. This optimization is mainly aimed at the scenario of non-full stripe reading and writing, reducing the performance loss caused by read and write amplification, so the performance after optimization is greatly improved and the performance is stable.

[0193] Based on the same inventive concept, the present application also provides an erasure code data storage device, which can be specifically set in the electronic device discussed above, please refer to Figure 19 , the device comprises:

[0194] Determining module 1901, configured to determine a first stripe corresponding to the first data based on the first address offset, data length, and stripe length of the first data if the first address offset or data length of the first data to be stored is not an integer multiple of the stripe length;

[0195] An obtaining module 1902 is configured to pad the first data with padding data according to the length of the first stripe to obtain the padded first data, and to encode the padded first data using a preset error correction code algorithm to obtain a first check block; wherein the length of the padded first data is equal to the length of the first stripe;

[0196] The writing module 1903 is configured to write the first data into the storage space corresponding to the first stripe, create a hole in the storage space corresponding to the first stripe, and write the first parity block into other storage spaces; wherein the first data and the hole fully occupy the storage space corresponding to the first stripe;

[0197] The reading module 1904 is used to restore the first data based on other data and a first check block when reading the first data if part of the first data is damaged; wherein the other data is obtained by filling the undamaged data in the first data with filling data based on the position of the hole.

[0198] In a possible embodiment, the apparatus further includes a receiving module 1905, which is configured to: after writing the first check block into other storage spaces, receive a request to overwrite the first stripe with the second data to be stored;

[0199] The obtaining module 1902 is further configured to: if the first address offset or data length of the second data is not an integer multiple of the stripe length, read the target data from the storage space corresponding to the target stripe unit, and obtain a second parity block based on the target data and the second data; wherein the target stripe unit is the smaller stripe unit between the first stripe unit and the second stripe unit required for writing the second data; and the second stripe unit is the remaining stripe unit in the first stripe except the first stripe unit.

[0200] The writing module 1903 is further configured to write the second data into the storage space corresponding to the first stripe unit, overwriting the original data in the storage space corresponding to the first stripe unit, and write the second check block into the storage space corresponding to the first check block, overwriting the first check block.

[0201] In a possible embodiment, the obtaining module 1902 is specifically configured to:

[0202] If the second stripe unit is the target stripe unit, read the target data from the storage space corresponding to the second stripe unit;

[0203] The target data and the second data are encoded using an error correction code algorithm to obtain a second check block.

[0204] In a possible embodiment, the obtaining module 1902 is specifically configured to:

[0205] If the first stripe unit is the target stripe unit, read the target data from the storage space corresponding to the first stripe unit;

[0206] Performing an XOR operation on the target data and the second data, and encoding them using an error correction code algorithm to obtain an incremental check block;

[0207] An XOR operation is performed on the incremental check block and the first check block to obtain a second check block.

[0208] In a possible embodiment, the receiving module 1905 is further configured to: after writing the first check block into other storage space, receive a request to read the third data included in the first data;

[0209] The determination module 1901 is further configured to: if the first address offset or data length of the third data is not an integer multiple of the stripe length, determine a third stripe unit corresponding to the third data according to the first address offset, data length, stripe length, and stripe unit length of the third data;

[0210] The reading module 1904 is further configured to read the third data from the storage space corresponding to the third stripe unit.

[0211] It should be noted that Figure 19 The device can also be used to implement any of the erasure code data storage methods discussed above, which will not be repeated here.

[0212] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The device is equivalent to the electronic device discussed above. Please refer to Figure 20 , the device includes:

[0213] Memory 2002, used for storing program instructions;

[0214] The processor 2001 is configured to call the program instructions stored in the memory 2002 and execute the erasure code data storage method discussed above according to the obtained program instructions.

[0215] Processor 2001 may be a central processing unit (CPU), a digital processing unit, an image processor, or a combination thereof. Memory 2002 may be a volatile memory, such as random-access memory (RAM); or a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). Memory 2002 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 2002 may be a combination of the aforementioned memories.

[0216] As an embodiment, the processor 2001 can implement the erasure code data storage method discussed above. The processor 2001 can also implement the erasure code data storage method discussed above. Figure 19 The functions of the devices discussed above will not be described in detail here.

[0217] Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium storing a computer program. The computer program includes program instructions. When executed by a computer, the program instructions cause the computer to perform any of the erasure code data storage methods discussed above. Because the principles underlying the problems solved by the computer-readable storage medium are similar to those of the erasure code data storage method, the implementation of the computer-readable storage medium can be referred to as the implementation of the method, and any repetitive details will not be repeated.

[0218] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0219] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0220] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0222] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for storing erasure coded data, characterized in that: include: If the first address offset or data length of the first data to be stored is not an integer multiple of the stripe length, determining the first stripe corresponding to the first data according to the first address offset, data length and stripe length of the first data; According to the length of the first stripe, padding the first data with padding data to obtain padded first data, and encoding the padded first data using a preset error correction code algorithm to obtain a first check block; wherein the length of the padded first data is equal to the length of the first stripe; Writing the first data into the storage space corresponding to the first stripe, creating a hole in the storage space corresponding to the first stripe, and writing the first parity block into other storage spaces; wherein the first data and the hole fully occupy the storage space corresponding to the first stripe; When reading the first data, if part of the first data is damaged, the first data is restored based on other data and the first check block; wherein the other data is obtained by filling the undamaged data in the first data with the filling data according to the position of the hole.

2. The method according to claim 1, wherein After writing the first check block into other storage spaces, the method further includes: receiving a request to overwrite the first stripe with second data to be stored; If the first address offset or data length of the second data is not an integer multiple of the stripe length, the target data is read from the storage space corresponding to the target stripe unit, and a second parity block is obtained based on the target data and the second data; wherein the target stripe unit is the smaller stripe unit of the first stripe unit and the second stripe unit required for writing the second data; and the second stripe unit is the remaining stripe unit in the first stripe except the first stripe unit. The second data is written into the storage space corresponding to the first stripe unit, overwriting the original data in the storage space corresponding to the first stripe unit, and the second check block is written into the storage space corresponding to the first check block, overwriting the first check block.

3. The method according to claim 2, wherein Reading target data from a storage space corresponding to a target stripe unit, and obtaining a second check block according to the target data and the second data, including: If the second stripe unit is the target stripe unit, reading the target data from the storage space corresponding to the second stripe unit; The target data and the second data are encoded using the error correction code algorithm to obtain the second check block.

4. The method according to claim 2, wherein Reading target data from a storage space corresponding to a target stripe unit, and obtaining a second check block according to the target data and the second data, including: If the first stripe unit is the target stripe unit, reading the target data from a storage space corresponding to the first stripe unit; Performing an XOR operation on the target data and the second data, and encoding them using the error correction code algorithm to obtain an incremental check block; An exclusive OR operation is performed on the incremental check block and the first check block to obtain the second check block.

5. The method according to claim 1, wherein After writing the first check block into other storage spaces, the method further includes: receiving a request to read third data included in the first data; If the first address offset or data length of the third data is not an integer multiple of the stripe length, determining a third stripe unit corresponding to the third data according to the first address offset, data length, the stripe length, and stripe unit length of the third data; The third data is read from the storage space corresponding to the third stripe unit.

6. An erasure code data storage device, characterized in that: include: a determination module configured to determine, if a first address offset or a data length of the first data to be stored is not an integer multiple of the stripe length, a first stripe corresponding to the first data according to the first address offset, the data length, and the stripe length of the first data; an obtaining module, configured to pad the first data with padding data according to the length of the first stripe to obtain padded first data, and encode the padded first data using a preset error correction code algorithm to obtain a first check block; wherein the length of the padded first data is equal to the length of the first stripe; a writing module, configured to write the first data into a storage space corresponding to the first stripe, create a hole in the storage space corresponding to the first stripe, and write the first parity block into another storage space; wherein the first data and the hole completely occupy the storage space corresponding to the first stripe; A reading module is used to restore the first data based on other data and the first check block when reading the first data if part of the first data is damaged; wherein the other data is obtained by filling the undamaged data in the first data with the filling data according to the position of the hole.

7. The device according to claim 6, characterized in that The device further includes a receiving module, wherein the receiving module is configured to: After the first check block is written into other storage space, a request is received to overwrite the first stripe with second data to be stored; The acquisition module is further used for: If the first address offset or data length of the second data is not an integer multiple of the stripe length, the target data is read from the storage space corresponding to the target stripe unit, and a second parity block is obtained based on the target data and the second data; wherein the target stripe unit is the smaller stripe unit of the first stripe unit and the second stripe unit required for writing the second data; and the second stripe unit is the remaining stripe unit in the first stripe except the first stripe unit. The writing module is further used for: The second data is written into the storage space corresponding to the first stripe unit, overwriting the original data in the storage space corresponding to the first stripe unit, and the second check block is written into the storage space corresponding to the first check block, overwriting the first check block.

8. The device according to claim 7, wherein The acquisition module is specifically used for: If the second stripe unit is the target stripe unit, reading the target data from the storage space corresponding to the second stripe unit; The target data and the second data are encoded using the error correction code algorithm to obtain the second check block.

9. An electronic device, characterized in that: include: a memory for storing program instructions; A processor is configured to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 5 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 5.

Images