Data storage method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data storage method and device, equipment and medium, and relates to the technical field of distributed storage. The implementation scheme is: obtaining a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks; determining a target encoding strategy of the plurality of data blocks based on the storage requirement information; encoding the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and storing the plurality of data blocks and the plurality of check blocks in association.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of distributed storage, in particular to a data storage method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Distributed storage is a storage technology that stores data in multiple independent devices. Accordingly, a distributed storage system is a storage system that uses distributed storage technology, which includes multiple devices for storing data. In a distributed storage system, data is usually stored in a redundant manner to ensure the reliability and fault tolerance of the distributed storage system, avoiding data loss due to errors in part of the devices.

[0003] The methods described in this section can not be the methods that have been previously conceived or adopted. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, issues mentioned in this section should not be assumed to have been admitted to be prior art in any jurisdiction. SUMMARY

[0004] The present disclosure provides a data storage method and device, electronic equipment, computer readable storage medium and computer program product.

[0005] According to an aspect of the present disclosure, a data storage method is provided, comprising: obtaining a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks; determining a target encoding strategy of the plurality of data blocks based on the storage requirement information; encoding the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and storing the plurality of data blocks and the plurality of check blocks in association.

[0006] According to another aspect of the present disclosure, a data storage device is also provided, comprising: an obtaining unit configured to obtain a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks; a determining unit configured to determine a target encoding strategy of the plurality of data blocks based on the storage requirement information; an encoding unit configured to encode the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and a storage unit configured to store the plurality of data blocks and the plurality of check blocks in association.

[0007] According to another aspect of the present disclosure, an electronic equipment is also provided, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method according to the above.

[0008] According to another aspect of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method according to the above.

[0009] According to another aspect of the present disclosure, there is also provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to the above.

[0010] According to one or more embodiments of the present disclosure, a suitable encoding strategy can be dynamically selected based on the storage requirement information of the user, realizing flexible customization of the encoding strategy, and being able to meet the differentiated requirements of different users for data storage, thereby avoiding unnecessary waste of storage space, improving storage efficiency, and reducing storage cost.

[0011] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the embodiments. The illustrated embodiments are merely examples and do not limit the scope of the claims. In all the drawings, like reference numerals refer to like elements throughout the accompanying drawings.

[0013] Figure 1 A flowchart of a data storage method according to an embodiment of the present disclosure is shown;

[0014] Figure 2 A schematic diagram of strip storage according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A structural block diagram of a storage system according to some embodiments of the present disclosure is shown;

[0016] Figure 4 A structural block diagram of a storage system according to some other embodiments of the present disclosure is shown;

[0017] Figure 5 A structural block diagram of a storage system according to some other embodiments of the present disclosure is shown;

[0018] Figure 6 A structural block diagram of a data storage device according to an embodiment of the present disclosure is shown; and

[0019] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0022] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0023] In distributed storage systems (such as distributed object storage systems and distributed file storage systems), redundancy is typically used to store data to ensure the reliability and fault tolerance of the distributed storage system and to avoid data loss due to failure of some devices. When using erasure coding technology to encode the original data rows, it is necessary to specify the ratio of the original data to the check data.

[0024] Erasure coding (EC) is a common redundancy method in distributed storage systems. The basic idea of ​​EC is to encode the original data to obtain checksum data. When some of the original data is lost, the checksum data can be used to decode and retrieve the original data, thus achieving fault tolerance. When using erasure coding to encode the original data, the ratio of the original data to the checksum data needs to be specified.

[0025] The encoding and decoding mode of erasure code usually includes two kinds of encoding and decoding mode based on a generation matrix and based on a generation polynomial. The inventors have found through research that the encoding and decoding mode based on a generation matrix is simple to calculate and can ensure that a distributed storage system has a high throughput (i.e., the amount of data processed per unit of time), thereby improving read and write performance. However, this mode has poor sensing ability for silent errors, and when a silent error is encountered, if there is no system-level data integrity checking mechanism, data decoding errors will occur, causing data inconsistency. The encoding and decoding mode based on a generation polynomial has high computational complexity, resulting in low throughput of the system and limited read and write performance, but has strong sensing ability for silent errors (can detect silent errors) and good fault tolerance.

[0026] The inventors have noticed that a distributed storage system usually uses a fixed encoding scheme (including the ratio of the amount of original data and check data and the encoding and decoding mode) to implement erasure code storage, and the encoding scheme of all users is the same, and the encoding scheme cannot be flexibly customized for different users, and the differentiated needs of users cannot be met. If the encoding scheme provided by the system exceeds the needs of the user, unnecessary storage space waste, reduced storage efficiency, and increased service cost can be caused; if the encoding scheme provided by the system cannot meet the needs of the user, the user can be lost.

[0027] Therefore, embodiments of the present disclosure provide a data storage method, which can dynamically select a suitable encoding strategy based on storage requirement information of a user, so that the encoding strategy matches the needs of the user, and the flexible customization of the encoding strategy is realized, the differentiated needs of different users for data storage can be met, unnecessary storage space waste is avoided, storage efficiency is improved, and storage cost is reduced.

[0028] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A flowchart of a data storage method 100 according to an embodiment of the present disclosure is shown. The method 100 can be applied in a distributed storage system (hereinafter referred to as a "storage system") for example, and in particular, can be applied in any device (i.e., a server for storing data) of the storage system.

[0030] As shown in Figure 1 , the method 100 can include:

[0031] Step S110, obtaining a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks;

[0032] Step S120, determining a target encoding strategy of the plurality of data blocks based on the storage requirement information;

[0033] At step S130, encoding is performed on the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks.

[0034] At step S140, the plurality of data blocks and the plurality of check blocks are stored in association.

[0035] According to embodiments of the present disclosure, a suitable encoding strategy can be dynamically selected based on the storage requirement information of the user, realizing flexible customization of the encoding strategy, and being able to meet the differentiated requirements of different users for data storage, thereby avoiding unnecessary waste of storage space, improving storage efficiency, and reducing storage cost.

[0036] The following describes each step of the method 100 in detail.

[0037] At step S110, a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks are obtained.

[0038] In embodiments of the present disclosure, a data block refers to a group of data containing a certain amount of data. A check block refers to a group of data obtained after encoding a data block. According to some embodiments, the amount of data included in each data block can be pre-set. Typically, the amount of data included in each data block can be in the order of KB-MB, for example, 16 MB. According to some embodiments, the amount of data included in each check block can be the same as the amount of data included in each data block.

[0039] According to some embodiments, the plurality of data blocks to be stored can be obtained by dividing a data stream uploaded by a client device.

[0040] The client device may, for example, be an Internet of Things device (such as an industrial sensor, a road camera, a smart in-vehicle device, a smart home device, etc.), a mobile computing device (such as a mobile phone, a tablet computer, a smart wearable device, a notebook computer, etc.), a fixed computing device (such as a desktop computer, an industrial computer, a local area network gateway, a local area network server, etc.), etc., but is not limited thereto.

[0041] The client device can perform data collection or data processing, and upload the collected or processed data to the storage system in the form of a data stream, specifically, to one or more servers in the storage system. After receiving the data stream uploaded by the client device, the server divides the data stream according to the pre-set amount of data included in a single data block, and obtains a plurality of data blocks. For example, the client device can be a road camera. The road camera collects image data in real time, and uploads the collected image data to the server for storage. The server divides each 16 MB of image data into a data block. As the road camera continuously collects and uploads image data, the server will divide a plurality of data blocks with a size of 16 MB.

[0042] According to some embodiments, the plurality of data blocks to be stored can also be obtained by dividing data uploaded by the client device in bulk. For example, the client device can be a desktop computer, which uploads 128MB data to the server for storage at one time based on user's interaction. After receiving the 128MB data uploaded by the desktop computer, the server divides the 128MB data into 128 / 16 = 8 data blocks according to 16MB data per data block.

[0043] The storage requirement information is used to represent the user's requirement for data storage. According to some embodiments, the storage requirement information includes at least one of storage cost requirement information, storage reliability requirement information, and read-write performance requirement information.

[0044] The storage cost requirement information is used to represent the user's requirement for storage cost. According to some embodiments, a list of candidate values of the storage cost requirement information can be provided to the user so that the user selects one candidate value from the list as the storage cost requirement information. The candidate values of the storage cost requirement information can include, for example, standard, low cost, and extremely low cost. It should be understood that the present disclosure does not limit the number and content of the candidate values of the storage cost requirement information.

[0045] The storage reliability requirement information is used to represent the user's requirement for storage reliability. According to some embodiments, a list of candidate values of the storage reliability requirement information can be provided to the user so that the user selects one candidate value from the list as the storage reliability requirement information. The candidate values of the storage reliability requirement information can include, for example, 99.999% (i.e., 5 nines), 99.99999% (i.e., 7 nines), 99.9999999% (i.e., 9 nines), 99.999999999% (i.e., 11 nines), and the like. It should be understood that the present disclosure does not limit the number and content of the candidate values of the storage reliability requirement information.

[0046] The read-write performance requirement information is used to represent the user's requirement for read-write performance of the storage system. In some embodiments, the read-write performance of the storage system can also be referred to as Quality of Service (QoS). According to some embodiments, a list of candidate values of the read-write performance requirement information can be provided to the user so that the user selects one candidate value from the list as the read-write performance requirement information. The candidate values of the read-write performance requirement information can include, for example, read priority, write priority, and read-write balance. It should be understood that the present disclosure does not limit the number and content of the candidate values of the read-write performance requirement information.

[0047] According to some embodiments, the storage requirement information can be preset. For example, in a storage system, each user can correspond to one or more storage addresses, and the storage requirement information corresponding to each storage address can be preset. When the user stores a plurality of data blocks, the storage addresses of the plurality of data blocks need to be specified, and the storage requirement information corresponding to the storage addresses is the storage requirement information of the plurality of data blocks.

[0048] For another example, in an object storage system, a user can create one or more storage spaces (Bucket, also referred to as "Bucket") for storing data, and the storage requirement information can be set as an attribute of the storage space when the storage space is created or after the storage space has been created. When the user stores a plurality of data blocks, the storage space corresponding to the plurality of data blocks can be specified (i.e., the plurality of data blocks are stored in which storage space). After the server obtains the plurality of data blocks to be stored, the storage requirement information of the plurality of data blocks can be obtained by reading the attribute value of the storage space corresponding to the plurality of data blocks.

[0049] In step S120, based on the obtained storage requirement information, the target encoding strategy of the plurality of data blocks can be determined. The target encoding strategy is used to encode the plurality of data blocks.

[0050] According to some embodiments, the correspondence between the storage requirement information and the encoding strategy can be preset. Accordingly, in step S120, the target encoding strategy of the plurality of data blocks can be determined based on the preset correspondence between the storage requirement information and the encoding strategy. Specifically, the correspondence between the storage requirement information and the encoding strategy can be represented by a correspondence table. That is, the correspondence table between the storage requirement information and the encoding strategy can be preset, and the target encoding strategy can be determined by looking up the table.

[0051] For example, the exemplary correspondence between the storage requirement information and the encoding strategy can be as shown in Table 1:

[0052] Table 1

[0053]

[0054] Based on Table 1, for example, in the case where the storage cost requirement information is "standard", the storage reliability requirement information is "9 9", and the read-write performance requirement information is "read priority", the corresponding target encoding strategy is "strategy 1".

[0055] The target encoding strategy is an erasure code encoding strategy, which is used to encode the plurality of data blocks to be encoded. According to some embodiments, the target encoding strategy includes a first number of data blocks to be encoded each time and a number of check blocks generated each time. Accordingly, in step S130, the plurality of data blocks to be stored can be divided into at least one data group based on the first number and the second number included in the target encoding strategy, each data group including the first number of data blocks; and each data group is encoded respectively to generate the second number of check blocks corresponding to the data group. In step S140, each data group and the check blocks corresponding to the data group can be stored as a stripe, i.e., each stripe includes a data group (including the first number of data blocks) and the second number of check blocks corresponding to the data group. The first number and the second number corresponding to the stripe are referred to as the "parameter ratio" of the stripe. For example, a certain stripe includes 8 data blocks and 4 check blocks, and the parameter ratio of the stripe is 8+4.

[0056] For the above embodiments, the exemplary correspondence between the storage requirement information and the encoding strategy can be as shown in Table 2:

[0057] Table 2

[0058]

[0059] Based on Table 2, for example, in the case where the storage cost requirement information is "standard", the storage reliability requirement information is "9 9", and the read-write performance requirement information is "read priority", the corresponding target encoding strategy includes a first number y1 of data blocks to be encoded each time and a second number z1 of check blocks generated each time. The first number may, for example, be 8, 12, 14, 28, etc., and the second number may, for example, be 2, 4, 6, etc.

[0060] Generally, the lower the storage cost and the higher the write performance, the larger the corresponding first number, i.e., the more data blocks to be encoded each time. In some cases, extremely low cost can be achieved in combination with the hardware facilities of the storage system, such as using a lower-cost storage (e.g., a disk) to store data.

[0061] It can be understood that in some embodiments, the target encoding strategy corresponding to a set of storage requirement information can only include the first number and the second number (as shown in Table 2).

[0062] In other embodiments, in addition to including the first number and the second number, the target encoding strategy corresponding to a set of requirement information can also include a first value range to which the value of the first number belongs and a second value range to which the value of the second number belongs. In this case, the first number is an initial value within the first value range, and the second number is an initial value within the second value range.

[0063] The exemplary correspondence between the storage requirement information and the encoding strategy of the above embodiment can be shown in Table 3 as follows:

[0064] Table 3

[0065]

[0066] In Table 3, the encoding strategy corresponding to the storage requirement information of the ith group (i = 1, 2, 3,...) includes a first number y i , a second number z i , a first value range [p i , q i ] to which the value of y i belongs, and a second value range [s i , t i ] to which the value of z i belongs. It can be understood that the first number y i is an initial value within the first value range [p i , q i ], and the second number z i is an initial value within the second value range [s i , t i ].

[0067] Based on the above embodiment, in step S120, the first number of the data blocks to be encoded each time, the second number of the check blocks to be generated each time, the first value range to which the value of the first number belongs, and the second value range to which the value of the second number belongs can be determined based on the storage requirement information. Accordingly, in step S130, the multiple data blocks can be divided into at least one data group each including the first number of data blocks based on the determined first number and second number, and each data group can be encoded respectively to generate the second number of check blocks corresponding to the data group. After the storage of the multiple data blocks is completed, the access frequency of the multiple data blocks can be counted, and based on the access frequency, the values of the first number and the second number can be adjusted within the determined first value range and second value range respectively, so as to further improve the data access efficiency on the premise of meeting the user requirements. The specific steps of adjusting the first number and the second number based on the access frequency will be described below.

[0068] According to some embodiments, the target encoding strategy can only include the above first number and second number, or only include the above first number, second number, first value range and second value range, without including the target encoding mode. In this case, the storage system will adopt a default encoding mode (for example, the first encoding mode based on the generation polynomial or the second encoding mode based on the generation matrix) to encode each data group.

[0069] According to other embodiments, the target encoding strategy may also include a target encoding mode, which may include, for example, a first encoding mode based on a generator polynomial and a second encoding mode based on a generator matrix.

[0070] When the target encoding strategy includes a target encoding pattern, the correspondence between exemplary storage requirement information and encoding strategy can be shown in Table 4 below:

[0071] Table 4

[0072]

[0073] It should be noted that the correspondence between storage requirement information and encoding patterns in Table 4 is merely an example to facilitate understanding of the embodiments of this disclosure. The correspondence between storage requirement information and encoding patterns is not limited to that shown in Table 4.

[0074] Based on the pre-defined correspondence between storage requirement information and encoding strategy, the target encoding mode can be determined. Accordingly, in step S130, each data group can be encoded based on the determined target encoding mode (e.g., a first encoding mode based on a generator polynomial or a second encoding mode based on a generator matrix) to generate a second number of check blocks corresponding to that data group.

[0075] According to some embodiments, when the target encoding mode is a first encoding mode based on a generator polynomial, a generator polynomial can be used to encode the data group to generate a second number of check blocks, wherein the second number of check blocks includes a target check block, which is the XOR of the first number of data blocks included in the data group, for fault repair.

[0076] There can be various expressions for the generator polynomial. According to some embodiments, the generator polynomial can be, for example, the generator polynomial in the Reed-Solomon (RS) coding algorithm, with the following expression:

[0077] g(x)=(x-α 0 (x-α) 1 )…(x-α d-1 (1)

[0078] In the formula, g(x) is the generator polynomial of the variable x, d is the second scalar, and α is the generator of the Galois field (GF). 8 For example, α = 2.

[0079] The process of encoding the first number of data blocks within the data set using the generator polynomial shown in equation (1) can be as follows:

[0080] Let the dataset be denoted as M = [m0 m1…m k-1 ] T , where m i (i = 0, 2, ..., k-1) represents the i-th data block in the data set, k is the first quantity, and T represents transpose.

[0081] The data polynomial m(x) corresponding to the data set M is:

[0082]

[0083] The data set M is encoded using the generator polynomial g(x) in equation (1), and the parity check polynomial b(x) corresponding to the second number of parity blocks is:

[0084] b(x) = x d m(x) mod g(x) (3)

[0085] In equation (3), mod represents the remainder.

[0086] The verification polynomial b(x) can be expressed in the form of equation (4):

[0087]

[0088] b in equation (4) i This refers to the second number of parity blocks generated based on the RS coding algorithm. Based on RS coding, the following property exists: using the generator polynomial g(x) = (x - α) 0 (x-α) 1 )…(x-α d-1 Encoding the data set M, the XOR of all the parity blocks is equal to the XOR of all the data blocks (in the Galois field, the XOR operation is equivalent to the addition operation), that is, we have

[0089] In embodiments of this disclosure, the second number of check blocks b generated based on the RS encoding algorithm can be used. i Replace any parity block in the data block with the XOR of all data blocks. For example, replace the first parity block b0 with the XOR of all parity blocks, i.e., let Therefore, when only one block in a stripe (including a data group and a second number of check blocks corresponding to that data group) fails (the failed block is denoted as the "faulty block"), the faulty block can be quickly repaired, improving the repair efficiency of a single fault. The fault repair process will be detailed below.

[0090] According to other embodiments, when the target encoding mode is a second encoding mode based on a generator matrix, the generator matrix can be used to encode the data group to generate a second number of check blocks corresponding to the data group. Specifically, the generator matrix includes multiple encoding vectors, and a second number of encoding vectors can be obtained from the generator matrix; and the data group can be encoded using the second number of encoding vectors respectively (e.g., by multiplying the encoding vectors with the data group) to generate a second number of check blocks corresponding to the second number of encoding vectors respectively.

[0091] The generator matrix can be of various types. According to some embodiments, the generator matrix can be, for example, a Vandermonde matrix, a Cauchy matrix, etc., but is not limited thereto. The size of the generator matrix in this embodiment can be, for example, e*k, meaning the generator matrix includes e rows and k columns, where e is greater than or equal to a second quantity, and k is a first quantity. Preferably, the value of e is set to be greater than the second quantity, thereby facilitating the pruning or expansion of the check block in the second encoding mode. The specific steps for check block pruning and expansion will be detailed below.

[0092] As described above, in step S130, multiple data blocks can be divided into at least one data group based on the target encoding strategy (including a first number of data blocks encoded each time, a second number of check blocks generated each time, and a first numerical range to which the first number of values ​​belong, a second numerical range to which the second number of values ​​belong, and at least one of the target encoding strategy), each data group including a first number of data blocks; and each data group is encoded to generate a second number of check blocks corresponding to that data group, thereby obtaining multiple check blocks corresponding to the multiple data blocks to be stored.

[0093] After obtaining multiple check blocks corresponding to multiple data blocks through step S130, step S140 can be executed to associate and store the multiple data blocks with the multiple check blocks. Specifically, according to some embodiments, multiple data groups and multiple check blocks can be stored as at least one stripe, and each stripe includes a data group and a check block corresponding to that data group.

[0094] In embodiments of this disclosure, a data group (including a first number of data blocks) and its corresponding second number of parity blocks are denoted as a stripe. According to some embodiments, the individual blocks within a stripe can be stored on different servers to improve data storage reliability and fault tolerance. For example, a stripe may include 8 data blocks and 4 parity blocks, i.e., the stripe comprises a total of 12 blocks, which can be stored on different servers. Figure 2 A schematic diagram of block storage in this embodiment is shown. (As shown) Figure 2 As shown, storage system 200 includes, for example, 12 servers, servers 201-212. Stripe 220 (e.g.) Figure 2The eight data blocks 251-258 (shown in the shaded area) are stored in servers 201-208, and the four parity blocks 261-264 are stored in servers 209-212. It can be understood that servers 201-212, in addition to storing the blocks of stripe 220, can also store data blocks and / or parity blocks from other stripes. For example, server 205 also stores parity block 281, server 210 also stores data block 271, and server 211 also stores parity block 282.

[0095] According to some embodiments, when the target encoding strategy includes a first numerical range and a second numerical range, after storing multiple data blocks in association with multiple check blocks, method 100 may further include: counting the access frequency of multiple data blocks within a preset time period; adjusting the value of at least one of the first quantity and the second quantity based on the access frequency, wherein the adjusted value of the first quantity belongs to the first numerical range and the adjusted value of the second quantity belongs to the second numerical range; and encoding the multiple data blocks based on the adjusted first quantity and the second quantity.

[0096] Based on the above embodiments, the values ​​of the first quantity and the second quantity can be adjusted within the first and second numerical ranges respectively based on the access frequency, and multiple data blocks can be re-encoded based on the adjusted first and second quantities, thereby further improving data access efficiency while meeting user needs.

[0097] According to some embodiments, in an object storage system, a user's storage space (bucket) may have a dynamic adjustment flag attribute, which can be used to enable or disable the function of dynamically adjusting a first quantity and / or a second quantity based on access frequency. For example, if the dynamic adjustment flag is 1, the function of dynamically adjusting the first quantity and / or the second quantity based on access frequency is enabled; if the dynamic adjustment flag is 0, the function of dynamically adjusting the first quantity and / or the second quantity based on access frequency is disabled.

[0098] According to some embodiments, the ratio between the adjusted first quantity and the adjusted second quantity is negatively correlated with the access frequency. That is, the higher the access frequency, the smaller the value of the first quantity of data blocks encoded each time is relative to the second quantity of check blocks, thereby improving data recovery efficiency and data access efficiency.

[0099] Specifically, you can adjust only the first quantity, only the second quantity, or both. For example, if a bucket has a high access frequency in the last 7 days (e.g., exceeding a preset threshold), you can adjust the first and / or second values ​​to decrease the ratio of the first to the second quantity. For example, if the current first quantity is 8 and the second quantity is 4, the ratio is 8 / 4 = 2. If you only adjust the first quantity, you can decrease it, for example, by changing it to 6, resulting in a ratio of 6 / 4 = 1.5. If you only adjust the second quantity, you can increase it, for example, by changing it to 5, resulting in a ratio of 8 / 5 = 1.6. If you adjust both the first and second quantities simultaneously, they can either increase or decrease, as long as the adjusted ratio is less than the original ratio (i.e., 2).

[0100] As mentioned earlier, when encoding each data group using a first encoding mode based on a generator polynomial, the second number of check blocks corresponding to each data group includes a target check block. Based on the target check block, rapid fault repair can be achieved.

[0101] Accordingly, according to some embodiments, when each data group is encoded using a first encoding mode based on a generator polynomial, method 100 may further include: performing fault detection on at least one stripe, wherein the stripe includes any data group and a check block corresponding to the data group; and in response to determining that at least one of the at least one stripe includes at least one fault block, repairing at least one fault block based on the number and type of at least one fault block and a target check block.

[0102] According to some embodiments, stripe fault detection can be performed automatically by the storage system. For example, the storage system can periodically use algorithms such as CRC (Cyclic Redundancy Check) to check the data correctness of each block in the stripe. If the data in a block is incorrect, the block is marked as a faulty block. A faulty block can be either a data block or a parity block. That is, faulty blocks include both data blocks and parity blocks.

[0103] Repairing at least one faulty block, based on the number and type of at least one faulty block and the target verification block, can specifically include at least one of the following four cases.

[0104] According to some embodiments, in a first case, in response to determining that the number of at least one faulty block is one and that the faulty block is a data block, the faulty block is replaced by the XOR of the other data blocks in the strip and the target check block.

[0105] According to the above embodiments, if only one data block in a stripe fails, the XOR of the target parity block with other non-failed data blocks can be calculated. Then, the failed data block is replaced with the calculated XOR result, thus completing the repair of the failed data block. For example, if the data blocks in a stripe are m0, m1, ..., m7, and the parity blocks are b0, b1, b2, b3, where b0 is the target parity block, then... If the faulty block is data block m2, that is, data block m2 has failed, then the repaired data block can be calculated as m2' = b0 + m0 + m1 + m3 + m4 + m5 + m6 + m7.

[0106] According to some embodiments, in the second case, in response to determining that at least one faulty block is a verification block, the target verification block is marked as a faulty block, and the data groups in the strip are re-encoded to obtain the repaired verification blocks corresponding to each faulty block.

[0107] According to the above embodiment, if only the check block in the strip is faulty, then regardless of whether the target check block is faulty, the target check block is marked as a faulty block. Subsequently, the data groups in the strip can be re-encoded to obtain the repaired check block. For example, the data blocks in a certain strip are m0, m1, ..., m7, and the check blocks are b0, b1, b2, b3, where b0 is the target check block. If the faulty block is check block b1, then the target check block b0 is also marked as a faulty block. Then, the first encoding mode based on the generator polynomial is used to re-encode the data blocks m0, m1, ..., m7 to obtain the repaired target check block b0' and the repaired check block b1'.

[0108] According to some embodiments, in the third case, in response to determining that the number of at least one faulty block is greater than one and that at least one faulty block is a data block, the XOR of the target check block with other check blocks is determined as the repaired target check block, and at least based on the repaired target check block, a preset decoding algorithm is used to repair at least one faulty block.

[0109] The preset decoding algorithm can be, for example, a decoding algorithm corresponding to the RS encoding algorithm, i.e., the RS decoding algorithm. The RS decoding algorithm can, for example, include the following steps 1-4:

[0110] Step 1: Perform the standard RS decoding process based on the generator polynomial to calculate the adjoint polynomial.

[0111] Step 2: Create the error location polynomial.

[0112] Step 3: Mark the faulty block in the error location polynomial according to its location.

[0113] Step 4: Error correction based on the PGZ (Peterson-Gorenstein-Zierler) decoding algorithm (for specific steps, please refer to the paper Gorenstein, Daniel; Peterson, W. Wesley; Zierler, Neal (1960), "Two-Error Correcting Bose-Chaudhuri Codes are Quasi-Perfect", Information and Control, 3(3):291–294, doi:10.1016 / s0019-9958(60)90877-9.).

[0114] According to the above embodiments, by XORing the target check block with other check blocks to determine the repaired target check block, the second number of check blocks can be restored to the original state of the check blocks obtained based on RS encoding. Subsequently, using the RS decoding algorithm, the faulty data blocks can be repaired, thereby recovering the correct data blocks.

[0115] For example, in a certain strip, the data blocks are m0, m1, ..., m7, and the check blocks are b0, b1, b2, b3, where b0 is the target check block. If the faulty blocks are data blocks m2 and m3, then the target verification block after repair can be calculated. b0', b1, b2, and b3 are the original check blocks generated based on the RS encoding algorithm. Subsequently, based on b0', b1, b2, and b3, the RS decoding algorithm is used to decode the stripe to obtain the repaired and correct data blocks.

[0116] According to some embodiments, in the fourth case, in response to determining that the number of at least one faulty block is greater than one and that at least one faulty block includes a data block and a check block, a target check block is marked as a faulty block, and the above-described decoding algorithm is used to repair at least one faulty block.

[0117] According to the above embodiments, if multiple faulty blocks exist in the stripe, and the faulty blocks include both data blocks and check blocks, then the target check block is marked as a faulty block regardless of whether the target check block is faulty. Subsequently, the RS decoding algorithm is used to repair the faulty blocks, thereby recovering the correct data blocks and check blocks.

[0118] For example, in a certain strip, the data blocks are m0, m1, ..., m7, and the check blocks are b0, b1, b2, b3, where b0 is the target check block. If the faulty blocks are data block m2 and parity block b1, then the target parity block b0 is also marked as a faulty block. Subsequently, the RS decoding algorithm is used to decode the stripe to obtain the repaired correct data block and parity block.

[0119] As mentioned earlier, by using the second encoding mode based on the generator matrix to encode each data group, it is possible to quickly prune and expand the check block.

[0120] According to some embodiments, when each data group is encoded using a second encoding mode based on a generator matrix, the generated second number of check blocks are arranged in the order of their corresponding encoding vectors in the generator matrix. Method 100 may further include the following check block pruning step: in response to a check block pruning request, determining a third number of check blocks to be deleted; and deleting the third number of check blocks that appear later in the order of the second number of check blocks.

[0121] According to the above embodiments, it is possible to quickly prune the check blocks, reduce the number of check blocks while keeping the number of data blocks (i.e., the first number) unchanged, thereby improving storage efficiency and saving storage costs.

[0122] According to some embodiments, the check block pruning request may be issued by the user or automatically by the storage system (for example, as described above, the storage system may automatically reduce a second number of values ​​based on the access frequency, i.e., prune the check block).

[0123] For example, if the second quantity is 4, the data group is encoded using the second encoding mode based on the generator matrix, resulting in the second quantity of check blocks b0, b1, b2, and b3. Based on the check block pruning request, it is determined that 2 check blocks need to be deleted, i.e., the third quantity is 2. Therefore, 2 check blocks are deleted from check blocks b0, b1, b2, and b3 in reverse order, i.e., check blocks b2 and b3 are deleted.

[0124] According to some embodiments, when encoding the data group using a second encoding mode based on the generator matrix, method 100 may further include the following check block expansion step: in response to a check block expansion request, determining a fourth number of check blocks to be added; obtaining a fourth number of encoding vectors located after the second number of encoding vectors from the generator matrix; and encoding the data group using the fourth number of encoding vectors respectively to generate a fourth number of check blocks to be added corresponding to the fourth number of encoding vectors respectively.

[0125] According to the above embodiments, the rapid expansion of the parity blocks can be achieved. Under the premise that the number of data blocks (i.e., the first number) remains unchanged, the number of parity blocks can be increased, thereby improving the fault tolerance of the storage system.

[0126] According to some embodiments, the check block expansion request may be issued by the user or automatically by the storage system (for example, as described above, the storage system may automatically increase a second number of values ​​based on the access frequency, i.e., expand the check block).

[0127] For example, if the second quantity is 4, the data group is encoded using the second encoding mode based on the generator matrix, resulting in a second quantity of check blocks b0, b1, b2, and b3. These check blocks b0, b1, b2, and b3 are derived from the encoding vectors v0, v1, v2, and v3 in the generator matrix, respectively. Based on the check block expansion request, it is determined that two additional check blocks are needed, making the fourth quantity 2. Therefore, encoding vectors v4 and v5, located after encoding vector v3, are obtained from the generator matrix. These encoding vectors v4 and v5 are then used to encode the data group, resulting in the additional check blocks b4 and b5.

[0128] The above embodiments describe the process of pruning and expanding check blocks while keeping the number of data blocks (i.e., the first number) constant.

[0129] In other embodiments, check block pruning and expansion can also be performed when the first number of data blocks changes. In this case, check block pruning refers to decreasing the ratio of the second number of check blocks to the first number of data blocks, while check block expansion refers to increasing the ratio of the second number of check blocks to the first number of data blocks. For example, a stripe currently includes 8 data blocks and 4 check blocks, with a second number to first number ratio of 4 / 8 = 0.5. If the stripe is adjusted to include 16 data blocks and 4 check blocks, the second number to second number ratio is adjusted to 4 / 16 = 0.25, and the second number to first number ratio decreases; therefore, this adjustment is check block pruning.

[0130] According to some embodiments, method 100 may further include: in response to an encoding mode switching instruction, using a preset conversion matrix to realize the conversion between corresponding check blocks of different target encoding modes.

[0131] According to some embodiments, the encoding mode switching command can be issued by the user or automatically by the storage system. By performing the conversion between check blocks corresponding to different target encoding modes, user needs can be further met, storage efficiency can be improved, and storage costs can be saved.

[0132] According to some embodiments, generator polynomials and generator matrices are equivalent. Under the condition that d ≤ k-1 (where k is the first quantity and d is the second quantity), any generator polynomial-based encoding scheme (e.g., the generator polynomial-based RS encoding scheme) can be converted into a generator matrix-based encoding scheme with the same parameters. The proof is as follows:

[0133] In the Galois field GF(2^8), the generating polynomial is g(x)=(x-α). 0 (x-α) 1 )…(x-α d-1 ), that is, equation (1) above. α = 2. From the properties of generating polynomials, we can obtain:

[0134] m(1)=b(1) (5)

[0135] α d m(α)=b(α) (6)

[0136] …α d(d-1) m(α d-1 )=b(α d-1 (7)

[0137] In equations (5)-(7) above, m() is the data polynomial, b() is the check polynomial, and d is the second number of check blocks.

[0138] make:

[0139]

[0140]

[0141]

[0142] Then the above equations (5)-(7) can be expressed as:

[0143] Λ[A l |A r M = A l [b0…b d-1 ] T (11)

[0144] In equation (11) above, "|" represents the matrix concatenation operation, that is, [A l |A r ] indicates that matrix A l With matrix A r Concatenate along the row direction (i.e., concatenate horizontally left and right). M is the data group to be encoded. Equation (11) above can be transformed into:

[0145] [b0…b d-1 ] T =A l -1 Λ[A l |A r M (12)

[0146] That is, based on the generator polynomial g(x)=(x-α) 0 (x-α)1 )…(x-α d-1 The encoding scheme of ) is equivalent to that based on the generator matrix A l -1 Λ[A l |A r The encoding scheme of ].

[0147] According to some embodiments, if the check block of the current data set is generated based on a first coding pattern of a generator polynomial, it can be generated by using the corresponding coding vectors v0, ..., v in the generator matrix (e.g., a Vandermonde matrix). d-1 Multiplying with the data set M yields the parity blocks f0, ..., f0 in the second coding mode based on the generator matrix. d-1 This enables the conversion of the check block from the first encoding mode to the second encoding mode. In this embodiment, the preset conversion matrix is ​​the generator matrix.

[0148] Based on equation (12), if the check block of the current data group is generated based on the second encoding mode of the generator matrix, then matrix A can be used to... l -1 Λ[A l |A r Multiplying this by the data set M yields the check blocks b0, ..., b0 in the first coding mode based on the generator polynomial. d-1 This achieves the conversion of the check block from the second encoding mode to the first encoding mode. In this embodiment, the preset conversion matrix is ​​matrix A. l -1 Λ[A l |A r ].

[0149] The data storage method 100 according to the embodiments of this disclosure can flexibly encode data blocks based on user needs, and can expand and prune check blocks, as well as convert between check blocks of different encoding modes.

[0150] Figure 3 A structural block diagram of a storage system 300 according to an embodiment of the present disclosure is shown. Figure 3 As shown, the storage system 300 includes an encoding module 310, a decoding module 320, an expansion and trimming module 330, and a conversion module 340. It can be understood that modules 310-340 are all software modules.

[0151] The encoding module 310 can be configured to perform steps S110-S140 of the data storage method 100 of this disclosure, dynamically select a suitable target encoding strategy based on the user's storage requirements information, use the target encoding strategy to encode multiple data blocks to be stored to generate multiple check blocks, and associate the multiple data blocks with the multiple check blocks for storage.

[0152] The decoding module 320 can be configured to perform the data reading step, that is, to decode the stored data and return it to the user. It can be understood that the decoding strategy used by the decoding module 320 corresponds to the encoding strategy used by the encoding module 310.

[0153] The expansion and pruning module 330 can be configured to perform the expansion and pruning steps of the verification block in the data storage method 100 of the present disclosure embodiments.

[0154] The conversion module 340 can be configured to perform the conversion steps between check blocks of different encoding modes in the data storage method 100 of the present disclosure embodiments.

[0155] Figure 4 A structural block diagram of a storage system 400 according to other embodiments of the present disclosure is shown. For example... Figure 4 As shown, the storage system 400 includes a multi-user demand module 410, an access control module 420, a system control module 430, a decision control module 440, flexible encoding / decoding modules 450-1 to 450-3, and storage modules 460-1 to 460-3. Modules 410-440 can be implemented as software modules, for example, while modules 450 (including 450-1 to 450-3) and 460 (including 460-1 to 460-3) can be implemented as hardware modules, for example.

[0156] The multi-user requirement module 410 can be configured to set and store the correspondence between storage requirement information and encoding strategies.

[0157] The access control module 420 can be configured to perform the steps of the data storage method 100 to adjust the first quantity and the second quantity based on the access frequency.

[0158] The system control module 430 can be configured to provide commands and interfaces for adjusting system parameters, thereby adjusting the system parameters involved in the data storage method 100. System parameters may include, for example, the system's default encoding mode (based on a generator polynomial or a generator matrix), the expression of the generator polynomial, the expression of the generator matrix, the initial value of a first or second quantity, a first numerical range, and a second numerical range.

[0159] The decision control module 440 can be configured to execute step S120 of the data storage method 100, which determines the target encoding strategy based on storage requirement information.

[0160] The flexible encoding / decoding modules 450-1 to 450-3 can be implemented as the CPUs of each server in the storage system 400, used to perform encoding and decoding computation tasks.

[0161] Storage modules 460-1 to 460-3 can be implemented as storage devices in each server of the storage system 400, used for persistent storage of data blocks and parity blocks. Storage modules 460-1 to 460-3 can be, for example, HDD (Hard Disk Drive), SSD (Solid State Disk), SMR (Shingled Magnetic Recording), magnetic tape, etc.

[0162] The following combination Figure 5 This describes the encoding process and the check block trimming process according to embodiments of the present disclosure.

[0163] An exemplary encoding process for this embodiment of the disclosure is as follows:

[0164] Step 1: The storage system receives the data stream requesting to be written and divides the data stream into multiple data blocks of the same size (e.g., 16MB).

[0165] Step 2: The disk allocation module 510 sends a request to the decision control module 520 to obtain the target encoding strategy from the decision control module 520. The target encoding strategy includes a first number of data blocks encoded each time (e.g., 8), a second number of check blocks generated each time (e.g., 4), and a target encoding pattern (e.g., RS encoding pattern based on generator polynomial).

[0166] Step 3: Based on the target encoding strategy, each stripe includes 8 data blocks and 4 parity blocks, for a total of 12 blocks. The disk allocation module 510 allocates 12 disk blocks located on different storage nodes (servers) to these 12 blocks, with each block being 16MB, and creates an encoding calculation task.

[0167] Step 4: Bind the encoding computation task to a computing node (server) with available computing resources via pull or push. The elastic encoder / decoder module 530 in the computing node loads the task and determines that it is an 8+4 RS encoding task based on generator polynomials.

[0168] Step 5: The elastic encoder / decoder module 530 in the compute node encodes the 8 data blocks to obtain 4 check blocks.

[0169] Step 6: The elastic codec module 530 in the compute node concurrently writes these 12 blocks to the disk blocks of the pre-allocated storage modules 540-1 to 540-3 through the interface provided by the storage module.

[0170] Step 7: Record the metadata related to the stripe and complete the encoding task for a single stripe.

[0171] The exemplary verification block trimming process of this disclosure embodiment is as follows:

[0172] Suppose that a user's current stripe parameter configuration is 8+4, meaning that a stripe consists of 8 data blocks and 4 parity blocks.

[0173] Step 1: The storage system 500 obtains that the current operation is check block trimming, and the new stripe parameter ratio is 16 (first quantity) + 4 (second quantity).

[0174] Step 2: The storage system 500 selects two 8+4 stripes 550 under the same bucket through the data stream, and labels the blocks in the two stripes as n1-n12 and m1-m12 respectively. The storage location information of each block is obtained through the metadata management module.

[0175] Step 3: The storage system's disk allocation module 510 treats n1-n8 and m1-m8 as data blocks and allocates disk blocks for the newly generated parity blocks. n1-n8, m1-m8, and the newly allocated disk blocks are located on different storage nodes. A pruning task is then created.

[0176] Step 4: Use pull or push to make a compute node obtain the current pruning task.

[0177] Step 5: The elastic encoding / decoding module 530 in the compute node encodes n1-n8 and m1-m8 according to the new encoding algorithm assigned by the pruning task, generates 4 new check blocks, writes the 4 new check blocks into the disk blocks of the corresponding storage modules 540-1 to 540-3, and updates the metadata information.

[0178] Step 6: Using the GC (Garbage Collection) module of the storage system, mark the eight original parity blocks n9-n12 and m9-m12 as a collection task, and asynchronously mark the metadata of the corresponding disk blocks as available to reclaim disk space.

[0179] Step 7: Mark the access control of the new stripe as 0 through the decision control module 520 (if the stripe ID is reused), end this pruning task, and release the occupied memory resources.

[0180] According to another aspect of this disclosure, a data storage device is also provided. For example... Figure 6As shown, the device 600 may include: an acquisition unit 610 configured to acquire a plurality of data blocks to be stored and storage requirement information of the plurality of data blocks; a determination unit 620 configured to determine a target encoding strategy for the plurality of data blocks based on the storage requirement information; an encoding unit 630 configured to encode the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and a storage unit 640 configured to store the plurality of data blocks and the plurality of check blocks together.

[0181] According to embodiments of this disclosure, a suitable encoding strategy can be dynamically selected based on the user's storage requirements, realizing flexible customization of the encoding strategy. This can meet the differentiated data storage needs of different users, thereby avoiding unnecessary waste of storage space, improving storage efficiency, and reducing storage costs.

[0182] It should be understood that Figure 6 The individual units 610-640 of the device 600 shown can be used with reference to Figure 1 The steps S110-S140 in the described method 100 correspond to each other. Therefore, the operations, features, and advantages described above for method 100 also apply to device 600 and its constituent units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0183] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program modules. The above regarding... Figure 6 The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of the acquisition unit 610, determination unit 620, encoding unit 630, and storage unit 640 can be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0184] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein said memory stores a computer program that, when executed by said at least one processor, implements the method described above.

[0185] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the method described above when executed by a processor.

[0186] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described above.

[0187] See Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of hardware devices that can be applied to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0188] like Figure 7 As shown, the electronic device 700 may include at least one processor 701, working memory 702, input unit 704, display unit 705, speaker 706, storage unit 707, communication unit 708 and other output units 709 that are capable of communicating with each other via system bus 703.

[0189] Processor 701 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 701 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 701 may be configured to acquire and execute computer-readable instructions stored in working memory 702, storage unit 707, or other computer-readable media, such as program code of operating system 702a, program code of application program 702b, etc.

[0190] Working memory 702 and storage unit 707 are examples of computer-readable storage media for storing instructions that are executed by processor 701 to perform the various functions described above. Working memory 702 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage unit 707 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 702 and storage unit 707 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by processor 701 as a specific machine configured to perform the operations and functions described in the examples herein.

[0191] Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 706 can be any type of device capable of presenting information, and may include, but is not limited to, display unit 705, speaker 706, and other output units 709. Other output units 709 may include, but are not limited to, video / audio output terminals, vibrators, and / or printers. Communication unit 708 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth. TM Equipment, 802.11 equipment, Wi-Fi equipment, WiMAX equipment, cellular communication equipment and / or the like.

[0192] The application program 702b in working register 702 can be loaded to execute the various methods and processes described above, for example... Figure 1Steps S110-S140 are described above. For example, in some embodiments, the method 100 described above may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 707. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 700 via storage unit 707 and / or communication unit 708. When the computer program is loaded and executed by processor 701, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, processor 701 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0193] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0195] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0197] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0198] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0199] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0200] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A data storage method applied in a distributed storage system, wherein each user in the distributed storage system corresponds to one or more storage addresses, the method comprising: Obtain the user's multiple data blocks to be stored and the storage addresses of the multiple data blocks; Based on the preset correspondence between storage addresses and storage requirement information, the storage requirement information of the user for the multiple data blocks is obtained, wherein the storage requirement information includes at least one of the following: storage cost requirement information, storage reliability requirement information, and read / write performance requirement information; Based on the storage requirement information, the target encoding strategy for the multiple data blocks is determined; Based on the target encoding strategy, the plurality of data blocks are encoded to generate a plurality of check blocks corresponding to the plurality of data blocks; and The plurality of data blocks and the plurality of verification blocks are stored together.

2. The method as described in claim 1, wherein, The step of determining the target encoding strategy for the plurality of data blocks based on the storage requirement information includes: Based on the pre-defined correspondence between storage requirements and encoding strategies, the target encoding strategy for the multiple data blocks is determined.

3. The method as described in claim 1 or 2, wherein, The target encoding strategy includes a first number of data blocks encoded each time and a second number of check blocks generated each time, wherein, The step of encoding the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks includes: The plurality of data blocks are divided into at least one data group, each data group comprising the first number of data blocks; and Each data group is encoded separately to generate the second number of check blocks corresponding to that data group.

4. The method of claim 3, wherein, Associating and storing the plurality of data blocks and the plurality of check blocks includes: The plurality of data blocks and the plurality of check blocks are stored as at least one stripe, wherein each stripe includes a data group and a check block corresponding to the data group.

5. The method of claim 3, wherein, The target encoding strategy includes a first numerical range to which the first number of values ​​belong and a second numerical range to which the second number of values ​​belong. The method further includes: Count the access frequency of the multiple data blocks within a preset time period; Based on the access frequency, the value of at least one of the first quantity and the second quantity is adjusted, wherein the adjusted value of the first quantity belongs to the first value range, and the adjusted value of the second quantity belongs to the second value range; and The plurality of data blocks are encoded based on the adjusted first and second quantities.

6. The method of claim 5, wherein, The ratio between the adjusted first number and the adjusted second number is negatively correlated with the access frequency.

7. The method of claim 3, wherein, The target encoding strategy includes a target encoding pattern, which includes a first encoding pattern based on a generator polynomial, and wherein, The step of encoding each data group to generate the second number of check blocks corresponding to that data group includes: The data group is encoded using the generator polynomial to generate a second number of check blocks, wherein the second number of check blocks includes a target check block, which is the XOR of the first number of data blocks included in the data group, for fault repair.

8. The method of claim 7, further comprising: Fault detection is performed on at least one stripe; as well as In response to determining that at least one of the at least one stripes includes at least one faulty block, the at least one faulty block is repaired based on the number and type of the at least one faulty block and the target check block.

9. The method of claim 8, wherein, The repair of the at least one faulty block based on the number and type of the at least one faulty block and the target verification block includes at least one of the following: In response to determining that the number of the at least one faulty block is one and that the faulty block is a data block, the faulty block is replaced by the XOR of the other data blocks in the strip besides the faulty block and the target check block; In response to determining that the at least one faulty block is a verification block, the target verification block is marked as a faulty block, and the data group in the strip is re-encoded to obtain the repaired verification block corresponding to each faulty block; In response to determining that the number of the at least one faulty block is greater than one and that the at least one faulty block is a data block, the XOR of the target check block with other check blocks is determined as the repaired target check block, and at least based on the repaired target check block, a preset decoding algorithm is used to repair the at least one faulty block. as well as In response to determining that the number of the at least one faulty block is greater than one and that the at least one faulty block includes a data block and a check block, the target check block is marked as a faulty block, and the decoding algorithm is used to repair the at least one faulty block.

10. The method of claim 3, wherein, The target encoding strategy includes a target encoding pattern, which includes a second encoding pattern based on a generator matrix. The generator matrix includes multiple encoding vectors, and wherein... The step of encoding each data group to generate the second number of check blocks corresponding to that data group includes: Obtain the second number of encoded vectors from the generated matrix; and The data group is encoded using the second number of encoding vectors to generate a second number of check blocks corresponding to the second number of encoding vectors.

11. The method of claim 10, wherein, The second number of check blocks are arranged in the order of their corresponding encoding vectors in the generator matrix, and the method further includes: In response to the check block pruning request, determine the third number of check blocks to be deleted; and Delete the third check block that appears last in the second number of check blocks.

12. The method of claim 10, further comprising: In response to the check block expansion request, determine the fourth number of check blocks to be added; Obtain the fourth number of encoded vectors following the second number of encoded vectors; as well as The data group is encoded using the fourth number of encoding vectors to generate a fourth number of check blocks to be added, each corresponding to the fourth number of encoding vectors.

13. The method of claim 7, further comprising: In response to the encoding mode switching command, a preset conversion matrix is ​​used to realize the conversion between corresponding check blocks of different target encoding modes.

14. A data storage device applied in a distributed storage system, wherein each user in the distributed storage system corresponds to one or more storage addresses, the device comprising: The acquisition unit is configured as follows: Obtain the user's multiple data blocks to be stored and the storage addresses of the multiple data blocks; Based on the preset correspondence between storage addresses and storage requirement information, the storage requirement information of the user for the multiple data blocks is obtained, wherein the storage requirement information includes at least one of the following: storage cost requirement information, storage reliability requirement information, and read / write performance requirement information; The determining unit is configured to determine the target encoding strategy for the plurality of data blocks based on the storage requirement information; An encoding unit is configured to encode the plurality of data blocks based on the target encoding strategy to generate a plurality of check blocks corresponding to the plurality of data blocks; and The storage unit is configured to store the plurality of data blocks and the plurality of check blocks together.

15. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores a computer program that, when executed by the at least one processor, implements the method as described in any one of claims 1-13.

16. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program, when executed by a processor, implements the method as described in any one of claims 1-13.

17. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method as described in any one of claims 1-13.

Images