Data storage method, system, program product, electronic device and storage medium

By grouping data blocks and building initial stripes in a distributed storage system, the problems of high bandwidth and computing overhead in the data reconstruction process in erasure coding technology are solved, and more efficient data repair and storage system performance are achieved.

CN118467236BActive Publication Date: 2024-11-19JINAN INSPUR DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410910399.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-19
Estimated Expiration
2044-07-09

AI Technical Summary

Technical Problem

The existing erasure coding technology leads to high network bandwidth and computing overhead in the data reconstruction process in distributed storage systems, and the decoding and computing are complex, affecting system efficiency.

Method used

By grouping original data blocks, building initial stripes and local verification blocks, distributing storage to multiple storage nodes, and prioritizing data repair within the failed nodes, reducing read and computing resource consumption.

Benefits of technology

The number of original data blocks and verification blocks required during the data repair process is reduced, decoding calculation is simplified, and data repair efficiency and system performance are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118467236B_ABST
    Figure CN118467236B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a data storage method, system, program product, electronic device and storage medium, which relates to the field of computer technology. The method includes: obtaining original data blocks and global check blocks corresponding to original data; grouping the original data blocks in order to obtain groups corresponding to the original data; constructing a first number of initial stripes according to the original data blocks and global check blocks in each group, and determining the first number of initial stripes as the first stripe set corresponding to the original data; generating a first number of local check blocks corresponding to each group according to the initial stripes in the first stripe set; distributing and storing the first stripe set and the local check blocks in the storage node. The embodiment of the present invention simplifies the computational amount of the decoding calculation process, reduces the computational overhead of the distributed storage system, shortens the time for data repair, and improves the efficiency of data repair.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, system, program product, electronic device and storage medium. Background Art

[0002] In distributed storage systems, data redundancy is a common technology, which aims to restore the original data through redundant information when data errors or damage occur.

[0003] At present, Erasure Coding (EC) generally adopts a k+m structure, where k represents the number of original data blocks, m represents the number of check blocks linearly related to k original data blocks, and the total number of data blocks n is the sum of the number of original data blocks k and the number of check blocks m. When the redundancy level is n, n data blocks can be stored on n different storage nodes respectively, so that the failure of m nodes can be tolerated, and when no more than m storage nodes fail, only k data blocks need to be read from storage nodes that have not failed to reconstruct the data blocks in the failed nodes through calculation.

[0004] However, since the check block is linearly related to the k original data blocks, when a storage node fails or data is damaged and data reconstruction is required, all k data blocks need to be read and transmitted from the storage node that has not failed, resulting in a large amount of network bandwidth and input and output (I / O) resource consumption. In addition, during the data reconstruction process, complex decoding calculations are required based on the linear relationship between the check block and the k original data blocks, which increases the computational overhead of the storage system. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present invention provides a data storage method, system, program product, electronic device and storage medium.

[0006] In a first aspect, the present invention provides a data storage method, which is applied to a distributed storage system, wherein the distributed storage system includes a plurality of storage nodes; the method includes:

[0007] Obtain at least two original data blocks and a global check block corresponding to the original data;

[0008] Grouping the at least two original data blocks in order to obtain at least two groups corresponding to the original data;

[0009] According to the original data blocks and the global check blocks in each of the groups, a first number of initial stripes are constructed, and the first number of initial stripes are determined as a first stripe set corresponding to the original data; the initial stripes include a data segment in each original data block in the group and a global check segment in each global check block;

[0010] Generating a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set;

[0011] The first stripe set and the local check block are distributed and stored in the storage node.

[0012] Optionally, generating a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set includes:

[0013] Obtaining a second number of the global check blocks;

[0014] Classifying the original data blocks in the group into front-end data blocks and back-end data blocks according to positions of the original data blocks in the group;

[0015] The local check block corresponding to the group is calculated according to the data fragments corresponding to the front-end data blocks and the data fragments corresponding to the back-end data blocks in the initial stripe and the second number.

[0016] Optionally, classifying the original data blocks in the group into front-end data blocks and back-end data blocks according to positions of the original data blocks in the group includes:

[0017] obtaining a third number of original data blocks in the group;

[0018] Calculate the ratio of the third number to 2, and round down the ratio to obtain a fourth number;

[0019] According to the positions of the original data blocks in the group, a fourth number of original data blocks located at the front of the group are determined as front-end data blocks;

[0020] The original data blocks other than the front-end data blocks in the group are determined as the back-end data blocks.

[0021] Optionally, the first stripe set includes a first initial stripe and a second initial stripe; and calculating the local check block corresponding to the group according to the data fragment corresponding to the front data block and the data fragment corresponding to the rear data block in the initial stripe and the second quantity includes:

[0022] Calculate a first local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the first initial stripe;

[0023] Calculate a second local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the second initial stripe;

[0024] Calculate a third local check segment according to the data segment corresponding to the second initial data block in the second initial stripe and the data segment corresponding to the first data block in the second initial stripe;

[0025] Calculate a fourth local check segment according to the data segment corresponding to the second initial stripe data block and the data segment corresponding to the first initial stripe data block;

[0026] A local check block corresponding to the group is calculated according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second number.

[0027] Optionally, the calculating the local check block corresponding to the group according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second quantity includes:

[0028] Merging the first local check segment, the second local check segment, the third local check segment and the fourth local check segment to obtain an original local check block corresponding to the group;

[0029] The original local check block is modulo the second number to obtain a local check block corresponding to the group.

[0030] Optionally, distributing and storing the first stripe set and the local check block in the storage node includes:

[0031] Adding the local check block corresponding to each of the groups to the first stripe set to obtain a second stripe set; the second stripe set includes a first number of target stripes, the target stripes include the initial stripe, and the local check segments corresponding to the initial stripe in the local check block corresponding to each group;

[0032] The target stripes are distributed and stored in the storage nodes.

[0033] Optionally, constructing a first number of initial stripes according to the original data blocks in each of the groups and the global check blocks, and determining the first number of initial stripes as a first stripe set corresponding to the original data includes:

[0034] Determining a first number of local check blocks corresponding to the group;

[0035] According to the first number, each original data block in the group is divided into a first number of data segments, and each global check block is divided into a first number of global check segments;

[0036] constructing a first number of initial stripes according to the data segments of each original data block in each of the groups and the global check segments in each global check block;

[0037] The first number of initial stripes is determined as a first stripe set corresponding to the original data.

[0038] Optionally, the obtaining of at least two original data blocks and a global check block corresponding to the original data includes:

[0039] Get the preset group quantity;

[0040] Slicing the original data according to the preset number of groups to obtain at least two original data blocks corresponding to the original data; the number of original data blocks corresponding to the original data is an integer multiple of the preset number of groups;

[0041] A polynomial linearly related to the at least two original data blocks is determined to obtain a global check block.

[0042] Optionally, the method further comprises:

[0043] In the case where at least one of the storage nodes in the distributed storage system fails, obtaining a first failure number of the failed node; the failed node is a storage node that fails in the distributed storage system;

[0044] When the first failure number is less than or equal to the second number of the global check blocks, obtaining, from the storage nodes corresponding to the original data, a second failure number of target failure nodes in the target storage nodes corresponding to the group;

[0045] When the second failure number is greater than 0 and less than or equal to the first number, obtaining target data blocks and target local check blocks required for repairing lost data in the target failure node from target storage nodes other than the target failure node;

[0046] The lost data is calculated according to the target data block and the target local check block to obtain repair data, and the repair data is stored in the storage node again.

[0047] In a second aspect, the present invention provides a distributed storage system, the distributed storage system comprising a storage management component and a plurality of storage nodes; wherein:

[0048] The storage management component is used to obtain at least two original data blocks and a global check block corresponding to the original data; group the at least two original data blocks in order to obtain at least two groups corresponding to the original data; construct a first number of initial stripes according to the original data blocks and the global check blocks in each of the groups, and determine the first number of initial stripes as a first stripe set corresponding to the original data; the initial stripe includes a data fragment in each original data block in the group and a global check fragment in each global check block; and generate a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set, and distribute and store the first stripe set and the local check blocks in the storage node.

[0049] Optionally, the storage management component is further configured to perform the following steps:

[0050] Obtaining a second number of the global check blocks;

[0051] Classifying the original data blocks in the group into front-end data blocks and back-end data blocks according to positions of the original data blocks in the group;

[0052] The local check block corresponding to the group is calculated according to the data fragments corresponding to the front-end data blocks and the data fragments corresponding to the back-end data blocks in the initial stripe and the second number.

[0053] Optionally, the storage management component is further configured to perform the following steps:

[0054] obtaining a third number of original data blocks in the group;

[0055] Calculate the ratio of the third number to 2, and round down the ratio to obtain a fourth number;

[0056] According to the positions of the original data blocks in the group, a fourth number of original data blocks located at the front of the group are determined as front-end data blocks;

[0057] The original data blocks other than the front-end data blocks in the group are determined as the back-end data blocks.

[0058] Optionally, the first stripe set includes a first initial stripe and a second initial stripe; and the storage management component is further configured to perform the following steps:

[0059] Calculate a first local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the first initial stripe;

[0060] Calculate a second local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the second initial stripe;

[0061] Calculate a third local check segment according to the data segment corresponding to the second initial data block in the second initial stripe and the data segment corresponding to the first data block in the second initial stripe;

[0062] Calculate a fourth local check segment according to the data segment corresponding to the second initial stripe data block and the data segment corresponding to the first initial stripe data block;

[0063] A local check block corresponding to the group is calculated according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second number.

[0064] Optionally, the storage management component is further configured to perform the following steps:

[0065] Merging the first local check segment, the second local check segment, the third local check segment and the fourth local check segment to obtain an original local check block corresponding to the group;

[0066] The original local check block is modulo the second number to obtain a local check block corresponding to the group.

[0067] Optionally, the storage management component is further configured to perform the following steps:

[0068] Adding the local check block corresponding to each of the groups to the first stripe set to obtain a second stripe set; the second stripe set includes a first number of target stripes, the target stripes include the initial stripe, and the local check segments corresponding to the initial stripe in the local check block corresponding to each group;

[0069] The target stripes are distributed and stored in the storage nodes.

[0070] Optionally, the storage management component is further configured to perform the following steps:

[0071] Determining a first number of local check blocks corresponding to the group;

[0072] According to the first number, each original data block in the group is divided into a first number of data segments, and each global check block is divided into a first number of global check segments;

[0073] constructing a first number of initial stripes according to the data segments of each original data block in each of the groups and the global check segments in each global check block;

[0074] The first number of initial stripes is determined as a first stripe set corresponding to the original data.

[0075] Optionally, the storage management component is further configured to perform the following steps:

[0076] Get the preset group quantity;

[0077] Slicing the original data according to the preset number of groups to obtain at least two original data blocks corresponding to the original data; the number of original data blocks corresponding to the original data is an integer multiple of the preset number of groups;

[0078] A polynomial linearly related to the at least two original data blocks is determined to obtain a global check block.

[0079] Optionally, the storage management component is further configured to perform the following steps:

[0080] In the case where at least one of the storage nodes in the distributed storage system fails, obtaining a first failure number of the failed node; the failed node is a storage node that fails in the distributed storage system;

[0081] When the first failure number is less than or equal to the second number of the global check blocks, obtaining, from the storage nodes corresponding to the original data, a second failure number of target failure nodes in the target storage nodes corresponding to the group;

[0082] When the second failure number is greater than 0 and less than or equal to the first number, obtaining target data blocks and target local check blocks required for repairing lost data in the target failure node from target storage nodes other than the target failure node;

[0083] The lost data is calculated according to the target data block and the target local check block to obtain repair data, and the repair data is stored in the storage node again.

[0084] In a third aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the data storage method described in any one of the first aspects above.

[0085] In a fourth aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the data storage method described in any one of the first aspects above.

[0086] In a fifth aspect, the present invention provides a readable storage medium, which, when instructions in the readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the steps in the data storage method in any one of the embodiments of the first aspect above.

[0087] In an embodiment of the present invention, in the process of implementing data redundancy by using erasure code technology in a distributed storage system, the distributed storage system obtains at least two groups corresponding to the original data by grouping original data blocks corresponding to the original data, and constructs a first stripe set including a first number of initial stripes based on the original data blocks and global check blocks in each group, generates a first number of local check blocks corresponding to each group based on the initial stripes in the first stripe set, and distributes and stores the first stripe set and the local check blocks in storage nodes of the distributed storage system; in the case where at least one storage node in the distributed storage system fails and causes data stored in the failed node to be damaged or lost, the distributed storage system can perform data repair within the group based on the original data blocks and local check blocks in each group according to the number of failed nodes, thereby reducing the number of original data blocks and / or check blocks required in the data repair process, and reducing the consumption of network bandwidth and input and output resources caused by reading original data blocks and / or check blocks from storage nodes; and performing data repair based on a smaller number of original data blocks and / or local check blocks also simplifies the computational amount of the decoding calculation process, reduces the computational overhead of the distributed storage system, shortens the time for data repair, and improves the efficiency of data repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0089] Figure 1 is a schematic diagram of the architecture of a distributed storage system provided by an embodiment of the present invention;

[0090] Figure 2 is a flowchart of a data storage method provided by an embodiment of the present invention;

[0091] Figure 3 is a schematic diagram of an erasure code structure provided by an embodiment of the present invention;

[0092] Figure 4 is a schematic diagram of a first stripe set provided by an embodiment of the present invention;

[0093] Figure 5 is a schematic diagram of a second stripe set provided by an embodiment of the present invention;

[0094] Figure 6 is a flowchart of another data storage method provided by an embodiment of the present invention;

[0095] Figure 7 is a logical block diagram of a distributed storage system provided by an embodiment of the present invention;

[0096] Figure 8 It is a structural diagram of an electronic device. DETAILED DESCRIPTION

[0097] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0098] Method Embodiment

[0099] Reference Figure 1 , shows a schematic diagram of the architecture of a distributed storage system provided by an embodiment of the present invention, such as Figure 1 As shown, the distributed storage system includes a distributed storage client, a storage service component and a storage management component.

[0100] Among them, the distributed storage client is responsible for interacting with applications or users, sending read and write requests, and receiving responses from metadata servers and data servers as well as the original data that needs to be stored; the distributed storage client includes objects, blocks, virtual machines, containers and files.

[0101] The storage service component is used to provide access interface services for distributed storage clients; specifically, the storage service component includes object gateway service, block device service, and file system service; the object gateway service is used to provide distributed storage clients with a RESTful API (a Web service interface design specification that follows the REST principle) interface that is compatible with a specific object storage system; the block device service allows distributed storage clients to read and write data in blocks; the file system service is used to provide an interface between distributed storage clients and storage devices, so that data can be stored and accessed in the form of files.

[0102] As the core component of the distributed storage system, the storage management component provides distributed storage services in the form of objects. In the storage management component, data is stored as objects, each of which has a unique identifier and related data; the storage management component is responsible for distributing the objects on the various storage nodes of the storage pool, and provides functions such as data replication, recovery, and load balancing to ensure data reliability and high-performance access. The storage management component includes a storage pool, a metadata cluster, and a monitoring service cluster; among them, the storage pool consists of multiple storage nodes, each of which can contain multiple storage devices; the storage pool divides the storage resources to form different erasure storage pools to meet different storage requirements; specifically, the storage pool slices the original data of the distributed storage client (the default is fixed to 4MB), groups the objects, and finally distributes each group to the underlying storage device. In the process of distributing the storage of each group to the underlying storage device, the storage pool evenly distributes the data objects on the various storage nodes of the storage pool through a distributed data balanced placement algorithm to avoid data hotspots and improve the performance of the distributed storage system.

[0103] The metadata cluster is used to manage metadata in the distributed storage system. Metadata is data used to describe data, such as file attributes, access permissions, storage location, etc. The metadata cluster stores this information in memory or disk to quickly respond to client metadata requests.

[0104] The monitoring service cluster is used to monitor the status and performance of each component in the distributed storage system to ensure the stable operation of the system. The monitoring service cluster provides real-time information about the health status, performance bottlenecks, and potential problems of the distributed storage system by collecting and analyzing data.

[0105] Current correction coding technologies include correction coding based on Galois field operations. Galois field erasure codes are mainly Reed-Solomon (RS) codes, which are forward error-corrected channel codes. In distributed storage systems, RS codes have two parameters k and m, denoted as RS(k,m); where k represents the number of original data blocks and m represents the number of check blocks. For example, when the RS code is RS(8,2), it indicates that there are 8 original data blocks and 2 check blocks; the total number of data blocks n is the sum of the number of original data blocks k and the number of check blocks m, which is commonly expressed as n = k + m. When the redundancy level is n, n data blocks can be stored on object storage devices (Object StorageDaemon, OSD) of n different storage nodes respectively, so that m nodes can be tolerated to fail, and when no more than m storage nodes fail, only k data blocks need to be read from storage nodes that have not failed to reconstruct the data blocks in the failed node by calculation. The principle of the RS code is to obtain a data vector by multiplying k original data blocks into a vector D by a generator matrix B. The data vector is composed of k original data blocks and m check blocks. If a data block is lost, the lost data block can be restored through a series of calculations.

[0106] It is understandable that the k value affects the cost of data recovery. The smaller the k value, the smaller the data dispersion, the larger the impact of the failure, and the greater the cost of reconstructing the data. The larger the k value, the more data copies will increase the load on the distributed storage system network and disk, and the cost of reconstructing the data will also be greater. The m value affects reliability and storage cost. The larger the m value, the higher the fault tolerance; the smaller the m value, the lower the data redundancy. Therefore, it is necessary to balance the two and solve the cost of reconstructing data for the distributed storage system.

[0107] According to the principle of RS erasure code, since the check block is linearly related to the k original data blocks, when a storage node fails or data is damaged and data reconstruction is required, all k data blocks need to be read and transmitted from the storage node that has not failed, resulting in a large amount of network bandwidth and input and output resource consumption, resulting in excessively high data reconstruction costs; and during the data reconstruction process, complex decoding calculations are also required based on the linear relationship between the check block and the k original data blocks, increasing the computing overhead of the storage system. Excessively high data reconstruction costs have become one of the key issues that hinder its practical application in distributed storage systems.

[0108] The data storage method provided in the embodiment of the present invention can be applied to the distributed storage system as described above, and the distributed storage system includes multiple storage nodes; specifically, referring to Figure 2, shows a flowchart of a data storage method provided by an embodiment of the present invention, the method may include steps S101 to S105:

[0109] Step S101: Obtain at least two original data blocks and a global check block corresponding to the original data.

[0110] The original data is the data stored in the distributed storage system by the user or application through the distributed storage client. In the embodiment of the present invention, when the distributed storage system obtains the original data, it can slice the original data according to the preset data block size to obtain at least two original data blocks of fixed size; then, based on each original data block, it encodes and generates a global check block corresponding to the original data. It can be understood that the global check block is composed of a linear combination of the original data blocks corresponding to the original data.

[0111] Exemplarily, the distributed storage system may slice the original data according to a default data block size (4 megabytes, MB) to obtain at least two original data blocks of 4 MB in size.

[0112] Specifically, the original data block can be represented as D i , and i is an integer greater than 0 and less than or equal to k; k represents the total number of original data blocks, and k is an integer greater than or equal to 2.

[0113] The number of global check blocks corresponding to the original data is at least 1, and the total number of global check blocks can be expressed as t, where t is an integer greater than or equal to 1.

[0114] For example, refer to Figure 3 , which shows a schematic diagram of an erasure code structure provided by an embodiment of the present invention, such as Figure 3 As shown, the distributed storage system obtains the original data blocks (D1, D2, D3, D4, D5, D6, D7, D8, D9, D 10 ) is 10; the global check block obtained by the distributed storage system through step S101 is P4, and the number of global check blocks is 1.

[0115] Step S102: group the at least two original data blocks in order to obtain at least two groups corresponding to the original data.

[0116] In an embodiment of the present invention, when k original data blocks and t global check blocks corresponding to the original data are obtained through step S101, the distributed storage system can group the k original data blocks to obtain at least two groups corresponding to the original data, and the number of original data blocks in each group is aligned.

[0117] Specifically, the number of groups corresponding to the original data can be expressed as g, where g is an integer greater than or equal to 2; each group can be expressed as G l , where l is an integer greater than 0 and less than or equal to g.

[0118] The number of original data blocks contained in each group is the ratio of the total number of original data blocks to the number of groups. The number of original data blocks contained in each group can be expressed by the following formula:

[0119] (1)

[0120] in, Indicates the number of original data blocks contained in each group; Indicates the total number of original data blocks corresponding to the original data; Indicates the number of groups corresponding to the original data.

[0121] It should be noted that the data storage method provided in the embodiment of the present invention is implemented based on the Erasure Coding Stripe Layout technology. The Erasure Coding Stripe Layout is a method of combining original data blocks and erasure code blocks to provide data redundancy and fault tolerance. The Erasure Coding Stripe is the basic unit of the Erasure Coding Stripe Layout, which is used to split, encode and store data. In the embodiment of the present invention, the erasure code block includes a global check block and a local check block.

[0122] Specifically, when k original data blocks and t global check blocks are obtained through step S101, each erasure code stripe contains k+t data, and the k+t data include data fragments in each original data block and global check fragments in each global check block.

[0123] In an erasure code stripe, k+t data need to be arranged in a predetermined order to ensure the correlation between the original data blocks and the erasure code blocks in the erasure code stripe.

[0124] In an embodiment of the present invention, when the distributed storage system sequentially groups the original data blocks corresponding to the original data, the original data blocks corresponding to the original data may be grouped according to a predetermined order in the erasure code stripe to obtain groups corresponding to the original data.

[0125] For example, refer to Figure 3 , the 10 original data blocks obtained in step S101 can be divided into the first group (1 st group) and the second group (2 nd group), each group contains 5 original data blocks. Of course, for Figure 3 The 10 original data blocks in can also be divided into 5 groups in sequence, each group containing 2 original data blocks.

[0126] Step S103: construct a first number of initial stripes according to the original data blocks in each of the groups and the global check blocks, and determine the first number of initial stripes as a first stripe set corresponding to the original data.

[0127] The initial stripe includes one data fragment in each original data block in the group and one global check fragment in each global check block. The initial stripe is an erasure code stripe including one data fragment in each original data block in the group and one global check fragment in each global check block.

[0128] It can be understood that in the erasure code stripe layout, each original data block can be divided into data fragments with the same number as the initial stripes, and different data fragments of the same original data block are located in different initial stripes; accordingly, each global check block is divided into global check block fragments with the same number as the initial stripes, and different global check block fragments of the same global check block are located in different initial stripes; in the same initial stripe, each data block fragment and global check block fragment are arranged in a predetermined order.

[0129] In the embodiment of the present invention, the first stripe set can be represented as S, and S={S a |0<a≤s}, where S a represents any initial stripe in the first stripe set, and s represents the first number of initial stripes.

[0130] Reference Figure 4 , shows a schematic diagram of a first stripe set provided by an embodiment of the present invention, such as Figure 4 As shown, the first stripe set includes s initial stripes, namely (S1,......,S s ), each initial stripe includes each original data block in g groups (D1,......,D k ) and each global check block (P1, ..., P t ) is a global validation snippet in .

[0131] In the embodiment of the present invention, the number of initial stripes in the first stripe set is the same as the number of local check blocks corresponding to the group generated in step S104, both of which are the first number. Figure 3The first stripe set includes three initial stripes, each of which includes a data fragment from each original data block (D1, D2, D3, D4, D5) in the first group, and a data fragment from each original data block (D6, D7, D8, D9, D 10 ) and a global check fragment in the global check block (P4).

[0132] Step S104: Generate a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set.

[0133] Specifically, the distributed storage system can encode each data fragment of each group in the initial stripe to generate the first number of fifth local check fragments corresponding to the group; then splice the first number of fifth local check fragments in each initial stripe to obtain the first number of local check blocks corresponding to the group; in this way, the first number of local check blocks corresponding to each group can be generated.

[0134] It can be understood that, for any group in the initial stripe, the fifth local check segment generated by encoding the data segments of the group is composed of a linear combination of the data segments in the group.

[0135] In an embodiment of the present invention, the erasure code can be expressed as (k, g, s, t), where k represents the total number of original data blocks corresponding to the original data; g represents the number of groups corresponding to the original data, s represents the number of initial stripes in the first stripe set, and the number of local check blocks corresponding to each group; t represents the number of global check blocks corresponding to the original data.

[0136] Among them, both the local check block and the global check block can be parity check blocks.

[0137] Step S105: Distribute and store the first stripe set and the local check block in the storage node.

[0138] Specifically, when the first stripe set and the first number of local check blocks corresponding to each group are obtained through steps S101 to S104, the following can be done: Figure 1 The distributed data balanced placement algorithm in the storage pool shown distributes and stores the first stripe set and the local check blocks in the storage nodes.

[0139] It can be understood that after the first stripe set and the local check blocks are distributed and stored in the storage nodes, each data fragment in each initial stripe in the first stripe set, each global check fragment, and the local check block corresponding to each group are all stored in different storage nodes of the storage pool.

[0140] In the event that at least one storage node in the distributed storage system fails, the distributed storage system may prioritize data repair within the group based on the data repair method in the erasure code stripe layout.

[0141] Specifically, when the number of failures of storage nodes in a distributed storage system is less than or equal to the first number of local check blocks, the distributed storage system may decode and repair the lost data in the failed node based on the data block fragments in the group of the initial stripe where the failed node is located, and / or the fifth local check fragment corresponding to the initial stripe in the local check blocks corresponding to the group; when the number of failures of storage nodes in a distributed storage system is greater than the first number of local check blocks and less than or equal to the second number of global check blocks, the distributed storage system may decode and repair the lost data in the failed node based on the respective data fragments in the initial stripe where the failed node is located and the global check fragments; when the number of failures of storage nodes in a distributed storage system is greater than the second number of global check blocks, the distributed storage system abandons the operation of decoding and repairing the lost data in the failed node.

[0142] In summary, the data storage method provided by the embodiment of the present invention, in the process of using the erasure code technology to realize data redundancy in the distributed storage system, the distributed storage system obtains at least two groups corresponding to the original data by grouping the original data blocks corresponding to the original data, and constructs a first stripe set including a first number of initial stripes according to the original data blocks and the global check blocks in each group, generates a first number of local check blocks corresponding to each group based on the initial stripes in the first stripe set, and distributes and stores the first stripe set and the local check blocks in the storage nodes of the distributed storage system; at least one storage node in the distributed storage system fails, resulting in the failure of the storage nodes in the failed node When the stored data is damaged or lost, the distributed storage system can, according to the number of faulty nodes, prioritize data repair within the group based on the original data blocks and local check blocks in each group, thereby reducing the number of original data blocks and / or check blocks required in the data repair process, and reducing the consumption of network bandwidth and input and output resources caused by reading original data blocks and / or check blocks from storage nodes; and repairing data based on a smaller number of original data blocks and / or local check blocks also simplifies the computational complexity of the decoding calculation process, reduces the computational overhead of the distributed storage system, shortens the time for data repair, and improves the efficiency of data repair.

[0143] Based on the same hardware resources, the embodiment of the present invention not only maintains the high fault tolerance of the distributed storage system, but also significantly reduces the amount of data read and transmitted during the data reconstruction and repair process, saving the computational overhead of the erasure code decoding process, the system network bandwidth and I / O resources, and effectively shortening the time to reconstruct and repair data; and the data storage method provided by the embodiment of the present invention can be decoupled from the distributed storage module, transparent to the distributed storage client, does not affect the conventional erasure algorithm and space availability, and has high stability. In addition, the data storage method provided by the embodiment of the present invention can also be encapsulated, which improves the security of the data storage process, and the data storage method provided by the embodiment of the present invention is portable and universal in different hardware devices, expanding the scope of application of the embodiment of the present invention.

[0144] Optionally, the step S104 of generating a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set includes steps S1041 to S1043:

[0145] Step S1041: Obtain a second number of the global check blocks.

[0146] Step S1042: Classify the original data blocks in the group into front-end data blocks and back-end data blocks according to the positions of the original data blocks in the group.

[0147] Step S1043: Calculate a local check block corresponding to the group according to the data fragments corresponding to the front-end data blocks and the data fragments corresponding to the back-end data blocks in the initial stripe and the second number.

[0148] In an embodiment of the present invention, while the distributed storage system generates a local check block corresponding to each group based on the initial stripe in the first stripe set, it can also calculate the local check blocks corresponding to each group respectively according to the position of the original data block in the group and the second number of global check blocks.

[0149] Specifically, the distributed storage system may determine the second number of global check blocks based on the global check blocks obtained in step S101, and classify the original data blocks in the group into front-end data blocks and back-end data blocks according to the arrangement positions of the original data blocks in the group.

[0150] In an optional implementation, the distributed storage system may first round up the ratio of the third number of original data blocks in the group to 2 to obtain a fifth number, and then determine the fifth number of original data blocks located at the front of the group as front-end data blocks based on the fifth number, and determine the original data blocks other than the front-end data blocks in the group as back-end data blocks. Exemplarily, in the case where the group includes 4 original data blocks, the distributed storage system may determine the first two original data blocks in the group as front-end data blocks, and determine the last two original data blocks in the group as back-end data blocks; in the case where the group includes 5 original data blocks, the distributed storage system may determine the first three original data blocks in the group as front-end data blocks, and determine the last two original data blocks in the group as back-end data blocks.

[0151] After obtaining the second quantity through step S1041 and completing the division of the original data blocks in each group through step S1042, the distributed storage system can calculate the local check blocks corresponding to the group based on the data fragments corresponding to the front data blocks and the data fragments corresponding to the back data blocks in the initial stripe and the second quantity.

[0152] It should be noted that, in the embodiment of the present invention, the number of local check blocks corresponding to each group is the same as the number of initial stripes in the first stripe set, which is the first number.

[0153] Optionally, the step S1042 classifies the original data blocks in the group into front-end data blocks and back-end data blocks according to the positions of the original data blocks in the group, including steps A11 to A14:

[0154] Step A11: Obtain a third number of original data blocks in the group.

[0155] Step A12: Calculate the ratio of the third quantity to 2, and round down the ratio to obtain a fourth quantity.

[0156] Step A13: According to the positions of the original data blocks in the group, a fourth number of original data blocks located at the front of the group are determined as front-end data blocks.

[0157] Step A14: determine the original data blocks other than the front-end data blocks in the group as the back-end data blocks.

[0158] In an embodiment of the present invention, in the process of classifying the original data blocks in a group into front-end data blocks and back-end data blocks according to the positions of the original data blocks in the group, the distributed storage system may also first round down the ratio of the third number of original data blocks in the group to 2 to obtain a fourth number, and then, based on the fourth number and the positions of the original data blocks in the group, determine the fourth number of original data blocks located at the front of the group as front-end data blocks, and determine the original data blocks other than the front-end data blocks in the group as back-end data blocks.

[0159] Exemplarily, when a group includes five original data blocks, the distributed storage system may determine the first two original data blocks in the group as front-end data blocks, and determine the last three original data blocks in the group as back-end data blocks.

[0160] In the embodiment of the present invention, the number of rear-end data blocks in each group is also the same as the first number of initial stripes in the first stripe set.

[0161] Optionally, the first stripe set includes a first initial stripe and a second initial stripe; and step S1043, calculating the local check block corresponding to the group according to the data fragments corresponding to the front-end data blocks and the data fragments corresponding to the rear-end data blocks in the initial stripe and the second quantity, includes steps A21 to A25:

[0162] Step A21: Calculate a first local check fragment according to the data fragment corresponding to the latter data block in the first initial stripe and the data fragment corresponding to the former data block in the first initial stripe.

[0163] Step A22: Calculate a second local check fragment according to the data fragment corresponding to the latter data block in the first initial stripe and the data fragment corresponding to the former data block in the second initial stripe.

[0164] Step A23: Calculate a third local check fragment according to the data fragment corresponding to the rear data block in the second initial stripe and the data fragment corresponding to the front data block in the second initial stripe.

[0165] Step A24: Calculate a fourth local check fragment according to the data fragment corresponding to the latter data block in the second initial stripe and the data fragment corresponding to the former data block in the first initial stripe.

[0166] Step A25: Calculate the local check block corresponding to the group according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second number.

[0167] The first initial stripe is any initial stripe in the first stripe set, and the second initial stripe is any initial stripe in the first stripe set except the first initial stripe.

[0168] In an embodiment of the present invention, in the process of calculating the local check blocks corresponding to the group according to the data fragments corresponding to the front data blocks in the initial stripe, the data fragments corresponding to the rear data blocks, and the second number, the distributed storage system can calculate the local check fragments corresponding to the group according to the data fragments corresponding to the rear data blocks in the first initial stripe and the data fragments corresponding to the front data blocks in each initial stripe in the first stripe set, respectively, and calculate the local check fragments corresponding to the group according to the data fragments corresponding to the rear data blocks in the second initial stripe and the data fragments corresponding to the front data blocks in each initial stripe in the first stripe set, respectively, and finally calculate the local check blocks corresponding to the group according to each local check fragment and the second number of global check blocks. In this way, the association between the data fragments corresponding to the original data blocks in the group in the local check fragments can be improved, thereby improving the association between the local check blocks corresponding to the group calculated based on the local check fragments and each original data block in the group, thereby improving the reliability of the local check blocks.

[0169] In addition, when the distributed storage system calculates the first local check fragment through step A21 and calculates the second local check fragment through step A22, the first local check fragment and the second local check fragment can be determined as the local check fragments corresponding to each data fragment of the group in the first initial stripe; when the distributed storage system calculates the third local check fragment through step A23 and calculates the fourth local check fragment through step A24, the third local check fragment and the fourth local check fragment can be determined as the local check fragments corresponding to each data fragment of the group in the second initial stripe. It should be noted that in an embodiment of the present invention, the number of local check fragments corresponding to each data fragment of each group in any initial stripe is the same as the number of local check blocks corresponding to the group, and the number of local check fragments corresponding to each data fragment of each group in any initial stripe is the same as the number of rear-end data blocks in the group.

[0170] It can be understood that, for different groups corresponding to the original data blocks, the distributed storage system can respectively perform operations corresponding to step A21 to step A25 to generate local check blocks corresponding to the group.

[0171] The first local check fragment is a local check fragment that is linearly related to the data fragment corresponding to the rear data block in the first initial stripe and the data fragment corresponding to the front data block in the first initial stripe; the second local check fragment is a local check fragment that is linearly related to the data fragment corresponding to the rear data block in the first initial stripe and the data fragment corresponding to the front data block in the second initial stripe; the third local check fragment is a local check fragment that is linearly related to the data fragment corresponding to the rear data block in the second initial stripe and the data fragment corresponding to the front data block in the second initial stripe; the fourth local check fragment is a local check fragment that is linearly related to the data fragment corresponding to the rear data block in the second initial stripe and the data fragment corresponding to the front data block in the first initial stripe.

[0172] As an example, see Figure 3 , the initial stripe S1, initial stripe S2 and initial stripe S3 in the first stripe set, each initial stripe includes a data fragment of the original data blocks (D1, D2, D3, D4, D5) in the first group and the original data blocks (D6, D7, D8, D9, D 10 ), and here the generation of the first group of corresponding local check blocks is taken as an example for explanation.

[0173] like Figure 3 As shown, the data fragment in each stripe can be recorded as D x a , a represents the initial stripe identifier, a is an integer greater than 0 and less than or equal to the first number s, Figure 3 For example, a can be any integer from 1 to 3; x represents the identifier of the data segment in the initial stripe, x is greater than (i-1)e and less than or equal to ie, i represents the original data block identifier, Figure 3 For example, i can be any integer from 1 to 10, and e represents the number of original data blocks included in each group.

[0174] Before executing step A21, the initial stripe S1 may be determined as the first initial stripe, and the initial stripes S2 and S3 may be determined as the second initial stripes.

[0175] As an optional implementation, in step A21, the distributed storage system may store data fragments (D3, D4, D5) corresponding to the latter data blocks (D3, D4, D5) in the first initial stripe S1. 1 ,D4 1 ,D5 1 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the first initial stripe S1. 1 ,D2 1 ), calculate and obtain the first local verification fragment.

[0176] In step A22, the distributed storage system can store data fragments (D3, D4, D5) corresponding to the latter data blocks (D3, D4, D5) in the first initial stripe S1. 1 ,D4 1 ,D5 1 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the second initial stripe S2 2 ,D2 2 ), calculate the second local check fragment; according to the data fragment (D3) corresponding to the latter data block (D3, D4, D5) in the first initial stripe S1 1 ,D4 1 ,D5 1 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the second initial stripe S3. 3 ,D2 3 ), calculate and obtain the second local verification fragment.

[0177] In step A23, the distributed storage system can generate a data fragment (D3) corresponding to the latter data block (D3, D4, D5) in the second initial stripe S2. 2 ,D4 2 ,D5 2 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the second initial stripe S2 2 ,D2 2 ), calculate the third local check fragment; according to the data fragment (D3) corresponding to the latter data block (D3, D4, D5) in the second initial stripe S2 2 ,D4 2 ,D5 2 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the second initial stripe S3. 3 ,D2 3 ), calculate and obtain the third local check fragment. According to the data fragment (D3) corresponding to the latter data block (D3, D4, D5) in the second initial stripe S3 3 ,D4 3 ,D5 3 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the second initial stripe S3. 3 ,D2 3 ), calculate the third local check fragment; according to the data fragment (D3) corresponding to the latter data block (D3, D4, D5) in the second initial stripe S3 3 ,D4 3 ,D5 3 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the second initial stripe S2 2 ,D2 2), calculate and obtain the third local verification fragment.

[0178] In step A24, the distributed storage system can store data fragments (D3, D4, D5) corresponding to the latter data blocks (D3, D4, D5) in the second initial stripe S2. 2 ,D4 2 ,D5 2 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the first initial stripe S1. 1 ,D2 1 ), calculate the fourth local check fragment; according to the data fragment (D3) corresponding to the rear data block (D3, D4, D5) in the second initial stripe S3 3 ,D4 3 ,D5 3 ) and the data fragment (D1) corresponding to the first data block (D1, D2) in the first initial stripe S1. 1 ,D2 1 ), calculate and obtain the fourth local verification fragment.

[0179] In step A25, the distributed storage system can calculate the first group of corresponding local check blocks (P 1,1 ,P 1,2 ,P 1,3 ).

[0180] It should be noted that the distributed storage system can also calculate the second group of corresponding local check blocks (P 2,1 ,P 2,2 ,P 2,3 ), to avoid repetition, it will not be repeated here.

[0181] As another optional implementation, the data segments in each stripe are equally divided into three data sub-segments. x a The corresponding three data sub-fragments can be recorded as: D x a -1, D x a -2, D x a -3; For example, the data fragment of the original data block D1 in the initial stripe S1 is D1 1 , the data segment is D1 1 The corresponding three data sub-segments are D1 1 -1, D1 1 -2, D1 1-3. Based on the operations corresponding to steps A21 to A25, the distributed storage system calculates the first local check segment, the second local check segment, the third local check segment, and the fourth local check segment based on the data sub-segments corresponding to the data segments in the initial stripe, and calculates the local check block corresponding to the group according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second quantity.

[0182] Optionally, the step A25 of calculating the local check block corresponding to the group according to the first local check segment, the second local check segment, the third local check segment, and the second number of the fourth local check segment includes steps A251 to A252:

[0183] Step A251: merge the first local check segment, the second local check segment, the third local check segment and the fourth local check segment to obtain an original local check block corresponding to the group.

[0184] Among them, any original local check block corresponding to the group can be expressed by the following formula:

[0185] (2)

[0186] in, The original local check block corresponding to the group; Indicates the data block fragments corresponding to the first data block in the group in the first initial stripe and the second initial stripe; Indicates the data block fragments corresponding to the rear data blocks in the group in the first initial stripe and the second initial stripe; and Indicates the initial stripe identifier.

[0187] Step A252: take the original local check block modulo the second number to obtain a local check block corresponding to the group.

[0188] In the embodiment of the present invention, any local check block corresponding to a group is expressed by the following formula:

[0189] (3)

[0190] in, represents the modulo of the second number t of the global check block.

[0191] Optionally, the step S105 of distributing and storing the first stripe set and the local parity block in the storage node includes steps S1051 to S1052:

[0192] Step S1051: Add the local check block corresponding to each of the groups to the first stripe set to obtain a second stripe set.

[0193] The second stripe set includes a first number of target stripes, and the target stripes include the initial stripe and local check segments corresponding to the initial stripe in the local check blocks corresponding to each group.

[0194] Step S1052: Distribute and store the target stripes in the storage nodes.

[0195] In the embodiment of the present invention, in the process of distributing and storing the first stripe set and the local check blocks in the storage nodes, the distributed storage system can first add the local check blocks corresponding to each group to the first stripe set to obtain the second stripe set, and then Figure 1 The distributed data balanced placement algorithm in the storage pool shown distributes and stores each target stripe in the second stripe set to the storage nodes.

[0196] Reference Figure 5 , shows a schematic diagram of a second stripe set provided by an embodiment of the present invention, such as Figure 5 As shown, adding the local check block corresponding to the group to the first stripe set is specifically: adding the local check block corresponding to the group (P1*,......,P s *) and the local check fragments linearly related to the data fragments corresponding to the group in the initial stripe are added to the initial stripe (S1,...,S s ), the target stripe in the second stripe set (S1`,......,S s `); The number of target stripes in the second stripe set is the same as the number of initial stripes in the first stripe set, both of which are the first number, and each target stripe includes an initial stripe and a local check fragment corresponding to the initial stripe in the local check block corresponding to each group.

[0197] According to the data storage method provided by the embodiment of the present invention, in the process of distributing and storing the first stripe set and the local check blocks to the storage nodes, the distributed storage system first adds the local check blocks corresponding to each group to the first stripe set to obtain the second stripe set, and then distributes and stores the target stripes in the second stripe set to the storage nodes. Through each target stripe, the association between the data fragments corresponding to the group and the local check fragments corresponding to the group is strengthened, thereby improving the operability of the decoding process.

[0198] Optionally, step S103 constructs a first number of initial stripes according to the original data blocks in each group and the global check block, and determines the first number of initial stripes as a first stripe set corresponding to the original data, including steps S1031 to S1034:

[0199] Step S1031: Determine a first number of local check blocks corresponding to the group.

[0200] It can be understood that the larger the first number of local check blocks, the lower the redundancy of the original data blocks in the group and the higher the fault tolerance of the group; the smaller the first number, the lower the redundancy of the original data blocks in the group and the lower the fault tolerance of the group.

[0201] In the embodiment of the present invention, the number of local check blocks corresponding to each group is the same, which is the first number. The first number of local check blocks corresponding to the group can be determined according to the fault tolerance within the group in the actual application scenario, and the embodiment of the present invention does not specifically limit this.

[0202] Step S1032: According to the first number, divide each original data block in the group into a first number of data segments, and divide each global check block into a first number of global check segments.

[0203] Step S1033: construct a first number of initial stripes according to the data segments of each original data block in each of the groups and the global check segments in each global check block.

[0204] Step S1034: Determine the first number of initial stripes as a first stripe set corresponding to the original data.

[0205] Specifically, refer to Figure 4 , the distributed storage system can convert each global check block (P1, ..., P t ) is divided into s global check fragments, and each original data block (D1,......,D k ) is divided into s data segments; then, according to the data segments of each original data block in each group and the global check segments in each global check block, s initial stripes (S1, ..., S s ), and get Figure 4 The first strip collection shown.

[0206] Optionally, the step S101 of acquiring at least two original data blocks and a global check block corresponding to the original data includes steps S1011 to S1013:

[0207] Step S1011, obtaining the preset number of groups.

[0208] Step S1012: Slice the original data according to the preset number of groups to obtain at least two original data blocks corresponding to the original data; the number of original data blocks corresponding to the original data is an integer multiple of the preset number of groups.

[0209] Step S1013: determine a polynomial linearly related to the at least two original data blocks to obtain a global check block.

[0210] In an embodiment of the present invention, the distributed storage system may first obtain a preset number of groups corresponding to the original data during the process of obtaining at least two original data blocks and a global check block corresponding to the original data; it can be understood that the preset number of groups is the same as the number of groups corresponding to the original data obtained in step S102, and the preset number of groups is an integer greater than or equal to 2.

[0211] Considering that after the grouping in step S102, the number of original data blocks in each group needs to be aligned, it is beneficial to determine the first stripe set and generate the local check block in the subsequent steps. In an embodiment of the present invention, the distributed storage system can first determine the number of groups corresponding to the original data in step S102, that is, the preset number of groups, in the process of obtaining at least two original data blocks corresponding to the original data in step S101, and then slice the original data according to the preset number of groups to obtain the original data blocks corresponding to the original data, so that the number of original data blocks corresponding to the original data is an integer multiple of the preset number of groups, so that the number of original data blocks in each group obtained in step S102 is aligned.

[0212] In addition, in the process of obtaining the global check block corresponding to the original data, the distributed storage system can determine a polynomial linearly related to each original data block corresponding to the original data based on the original data blocks corresponding to the original data, and determine the polynomial as the global check block corresponding to the original data, so that the global check block is linearly related to each original data block corresponding to the original data.

[0213] Optionally, refer to Figure 6 , shows a flowchart of another data storage method provided by an embodiment of the present invention, the method comprising steps S106 to S115:

[0214] Step S106: At least one storage node in the distributed storage system fails.

[0215] The failure of a storage node refers to a failure of a storage device in the storage node, which causes damage or loss of data stored in the storage node.

[0216] Step S107: The distributed storage system obtains a first failure number of the failed node.

[0217] The faulty node is a storage node in the distributed storage system where a fault occurs.

[0218] Step S108: The distributed storage system determines whether the first number of failures is less than or equal to the second number of global check blocks.

[0219] Specifically, when the number of first faults is less than or equal to the second number of global check blocks, it indicates that the lost data in the faulty node can be repaired by erasure coding, and the distributed storage system executes step S109; when the number of first faults is greater than the second number of global check blocks, it indicates that the lost data in the faulty node cannot be repaired by erasure coding, and the distributed storage system executes step S115.

[0220] Step S109: The distributed storage system obtains, from the storage nodes corresponding to the original data, the second failure number of the target failure nodes in the target storage nodes corresponding to the group.

[0221] The storage node corresponding to the original data refers to the storage node where the distributed storage system stores the first stripe set and local check blocks corresponding to the original data in step S105. The target storage node is the storage node where the original data blocks and local check blocks in the group corresponding to the original data are located.

[0222] Specifically, the distributed storage system can determine the second fault number of the target fault node in the target storage node corresponding to each group according to the group corresponding to the original data. It can be understood that the second fault number is less than or equal to the first fault number, and the second fault number corresponds to the group one by one.

[0223] Step S110: The distributed storage system determines whether the second failure number is greater than 0 and less than or equal to the first number.

[0224] Specifically, when the second fault number is equal to 0, it indicates that there is no target faulty node in the target storage nodes corresponding to the group, and there is no need to decode and repair the lost data. The distributed storage system can execute step S115; when the second fault number is greater than 0 and less than or equal to the first number, it indicates that there is a target faulty node in the target storage nodes corresponding to the group, and the lost data can be decoded and repaired within the group. The distributed storage system executes steps S111 to S112; when the second fault number is greater than the first number, it indicates that there is a target faulty node in the target storage nodes corresponding to the group, but the number of target faulty nodes is too large to decode and repair the lost data within the group. The distributed storage system executes steps S113 to S114.

[0225] Step S111: The distributed storage system obtains target data blocks and target local check blocks required for repairing lost data in the target failed node from target storage nodes other than the target failed node.

[0226] The loss data may include at least one of a data segment of an original data block in the group and a local check block segment of a local check block corresponding to the group.

[0227] Specifically, the distributed storage system may first determine the initial stripe where the target faulty node is located, and then obtain the data segments of the target data blocks and the local check segments of the target local check blocks required to repair the lost data in the target faulty node from the target storage node corresponding to the initial stripe.

[0228] Step S112: The distributed storage system calculates the loss data according to the target data block and the target local check block, obtains the repair data, and stores the repair data in the storage node again.

[0229] Specifically, the distributed storage system may calculate the loss data according to the initial stripe corresponding to the failed node, according to the data fragments of the target data block in the initial stripe and the local check fragments of the target local check block, thereby obtaining the repair data.

[0230] Step S113: The distributed storage system obtains target data blocks and target global check blocks required to repair lost data in the failed node from storage nodes other than the failed node.

[0231] Specifically, the distributed storage system may first determine the initial stripe where the failed node is located, and then obtain the data segments of the target data blocks and the global check segments of the target global check blocks required to repair the lost data in the failed node from the storage nodes corresponding to the initial stripe.

[0232] Step S114: The distributed storage system calculates the loss data according to the target data block and the target global check block, obtains the repair data, and distributes and stores the repair data in the storage nodes.

[0233] Specifically, the distributed storage system can calculate the loss data according to the initial stripe corresponding to the failed node, according to the data fragments of the target data block in the initial stripe and the global check fragments of the target global check block, so as to obtain the repair data.

[0234] Step S115: The distributed storage system does not perform decoding and repair of lost data in the failed node.

[0235] In an optional implementation, after the distributed storage system stores the repair data back in the storage node, it can also check and verify the repair data to ensure that the content of the repair data is consistent with the original data, thereby improving the integrity and reliability of the data stored in the storage node.

[0236] The repair data may include at least one of the data fragments in the original data block, the local check fragments in the local check block, and the global check fragments in the global check block. The original data is at least one of the data fragments in the original data block, the local check fragments in the local check block, and the global check fragments in the global check block that the distributed storage system stores in the storage node through step S105.

[0237] It should be noted that when the erasure code is (k, g, s, t), the repair cost of data decoding and repair is , while the erasure code with (k,m) structure in the related art has a repair overhead of k for data decoding and repair. It can be seen that in the data storage method provided by the embodiment of the present invention, when at least one storage node in the distributed storage system fails and the data stored in the failed node is damaged or lost, the distributed storage system can perform data repair within the group based on the original data blocks and local check blocks in each group according to the number of failed nodes, thereby reducing the number of original data blocks and / or check blocks required in the data repair process, and reducing the consumption of network bandwidth and input and output resources caused by reading original data blocks and / or check blocks from storage nodes; and performing data repair based on a smaller number of original data blocks and / or local check blocks also simplifies the computational complexity of the decoding calculation process, reduces the computational overhead of the distributed storage system, shortens the time for data repair, and improves the efficiency of data repair.

[0238] Device Embodiment

[0239] Reference Figure 7 , shows a logical block diagram of a distributed storage system provided by an embodiment of the present invention, such as Figure 7 As shown, the distributed storage system includes a storage management component 701 and multiple storage nodes 702; wherein:

[0240] The storage management component is used to obtain at least two original data blocks and a global check block corresponding to the original data; group the at least two original data blocks in order to obtain at least two groups corresponding to the original data; construct a first number of initial stripes according to the original data blocks and the global check blocks in each of the groups, and determine the first number of initial stripes as a first stripe set corresponding to the original data; the initial stripe includes a data fragment in each original data block in the group and a global check fragment in each global check block; and generate a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set, and distribute and store the first stripe set and the local check blocks in the storage node.

[0241] Optionally, the storage management component is further configured to perform the following steps:

[0242] Obtaining a second number of the global check blocks;

[0243] Classifying the original data blocks in the group into front-end data blocks and back-end data blocks according to positions of the original data blocks in the group;

[0244] The local check block corresponding to the group is calculated according to the data fragments corresponding to the front-end data blocks and the data fragments corresponding to the back-end data blocks in the initial stripe and the second number.

[0245] Optionally, the storage management component is further configured to perform the following steps:

[0246] obtaining a third number of original data blocks in the group;

[0247] Calculate the ratio of the third number to 2, and round down the ratio to obtain a fourth number;

[0248] According to the positions of the original data blocks in the group, a fourth number of original data blocks located at the front of the group are determined as front-end data blocks;

[0249] The original data blocks other than the front-end data blocks in the group are determined as the back-end data blocks.

[0250] Optionally, the first stripe set includes a first initial stripe and a second initial stripe; and the storage management component is further configured to perform the following steps:

[0251] Calculate a first local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the first initial stripe;

[0252] Calculate a second local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the second initial stripe;

[0253] Calculate a third local check segment according to the data segment corresponding to the second initial data block in the second initial stripe and the data segment corresponding to the first data block in the second initial stripe;

[0254] Calculate a fourth local check segment according to the data segment corresponding to the second initial stripe data block and the data segment corresponding to the first initial stripe data block;

[0255] A local check block corresponding to the group is calculated according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second number.

[0256] Optionally, the storage management component is further configured to perform the following steps:

[0257] Merging the first local check segment, the second local check segment, the third local check segment and the fourth local check segment to obtain an original local check block corresponding to the group;

[0258] The original local check block is modulo the second number to obtain a local check block corresponding to the group.

[0259] Optionally, the storage management component is further configured to perform the following steps:

[0260] Adding the local check block corresponding to each of the groups to the first stripe set to obtain a second stripe set; the second stripe set includes a first number of target stripes, the target stripes include the initial stripe, and the local check segments corresponding to the initial stripe in the local check block corresponding to each group;

[0261] The target stripes are distributed and stored in the storage nodes.

[0262] Optionally, the storage management component is further configured to perform the following steps:

[0263] Determining a first number of local check blocks corresponding to the group;

[0264] According to the first number, each original data block in the group is divided into a first number of data segments, and each global check block is divided into a first number of global check segments;

[0265] constructing a first number of initial stripes according to the data segments of each original data block in each of the groups and the global check segments in each global check block;

[0266] The first number of initial stripes is determined as a first stripe set corresponding to the original data.

[0267] Optionally, the storage management component is further configured to perform the following steps:

[0268] Get the preset group quantity;

[0269] Slicing the original data according to the preset number of groups to obtain at least two original data blocks corresponding to the original data; the number of original data blocks corresponding to the original data is an integer multiple of the preset number of groups;

[0270] A polynomial linearly related to the at least two original data blocks is determined to obtain a global check block.

[0271] Optionally, the storage management component is further configured to perform the following steps:

[0272] In the case where at least one of the storage nodes in the distributed storage system fails, obtaining a first failure number of the failed node; the failed node is a storage node in the distributed storage system that fails;

[0273] When the first failure number is less than or equal to the second number of the global check blocks, obtaining, from the storage nodes corresponding to the original data, a second failure number of target failure nodes in the target storage nodes corresponding to the group;

[0274] When the second failure number is greater than 0 and less than or equal to the first number, obtaining target data blocks and target local check blocks required for repairing lost data in the target failure node from target storage nodes other than the target failure node;

[0275] The loss data is calculated according to the target data block and the target local check block to obtain repair data, and the repair data is stored in the storage node again.

[0276] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the data storage method of the above embodiment when the computer program / instruction is executed by a processor.

[0277] The present invention also provides an electronic device, see Figure 8, including: a processor 801, a memory 802, and a computer program 8021 stored in the memory and executable on the processor, wherein when the processor executes the program, the data storage method of the aforementioned embodiment is implemented.

[0278] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the data storage method of the above embodiment.

[0279] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0280] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

Claims

1. A data storage method, characterized in that: Applied to a distributed storage system, the distributed storage system includes a plurality of storage nodes; the method includes: Obtain at least two original data blocks and a global check block corresponding to the original data; Grouping the at least two original data blocks in order to obtain at least two groups corresponding to the original data; According to the original data blocks and the global check blocks in each of the groups, a first number of initial stripes are constructed, and the first number of initial stripes are determined as a first stripe set corresponding to the original data; the initial stripes include a data segment in each original data block in the group and a global check segment in each global check block; Generating a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set; Distribute and store the first stripe set and the local check block in the storage node; The step of generating a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set includes: Obtaining a second number of the global check blocks; Classifying the original data blocks in the group into front-end data blocks and back-end data blocks according to positions of the original data blocks in the group; Calculate a local check block corresponding to the group according to the data fragments corresponding to the front-end data blocks and the data fragments corresponding to the back-end data blocks in the initial stripe and the second number; The distributing and storing the first stripe set and the local check block in the storage node includes: Adding the local check block corresponding to each of the groups to the first stripe set to obtain a second stripe set; the second stripe set includes a first number of target stripes, the target stripes include the initial stripe, and the local check segments corresponding to the initial stripe in the local check block corresponding to each group; The target stripes are distributed and stored in the storage nodes.

2. The method according to claim 1, characterized in that The classifying the original data blocks in the group into front-end data blocks and rear-end data blocks according to the positions of the original data blocks in the group comprises: obtaining a third number of original data blocks in the group; Calculate the ratio of the third number to 2, and round down the ratio to obtain a fourth number; According to the positions of the original data blocks in the group, a fourth number of original data blocks located at the front of the group are determined as front-end data blocks; The original data blocks other than the front-end data blocks in the group are determined as the back-end data blocks.

3. The method according to claim 1, characterized in that The first stripe set includes a first initial stripe and a second initial stripe; and calculating the local check block corresponding to the group according to the data fragment corresponding to the front data block and the data fragment corresponding to the rear data block in the initial stripe and the second quantity, comprises: Calculate a first local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the first initial stripe; Calculate a second local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the second initial stripe; Calculate a third local check segment according to the data segment corresponding to the second initial data block in the second initial stripe and the data segment corresponding to the first data block in the second initial stripe; Calculate a fourth local check segment according to the data segment corresponding to the second initial stripe data block and the data segment corresponding to the first initial stripe data block; A local check block corresponding to the group is calculated according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second number.

4. The method according to claim 3, characterized in that The calculating the local check block corresponding to the group according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second quantity includes: Merging the first local check segment, the second local check segment, the third local check segment and the fourth local check segment to obtain an original local check block corresponding to the group; The original local check block is modulo the second number to obtain a local check block corresponding to the group.

5. The method according to claim 1, characterized in that The step of constructing a first number of initial stripes according to the original data blocks in each of the groups and the global check blocks, and determining the first number of initial stripes as a first stripe set corresponding to the original data includes: Determining a first number of local check blocks corresponding to the group; According to the first number, each original data block in the group is divided into a first number of data segments, and each global check block is divided into a first number of global check segments; constructing a first number of initial stripes according to the data segments of each original data block in each of the groups and the global check segments in each global check block; The first number of initial stripes is determined as a first stripe set corresponding to the original data.

6. The method according to claim 1, characterized in that The obtaining of at least two original data blocks and a global check block corresponding to the original data includes: Get the preset group quantity; Slicing the original data according to the preset number of groups to obtain at least two original data blocks corresponding to the original data; the number of original data blocks corresponding to the original data is an integer multiple of the preset number of groups; A polynomial linearly related to the at least two original data blocks is determined to obtain a global check block.

7. The method according to claim 1, characterized in that The method further comprises: In the case where at least one of the storage nodes in the distributed storage system fails, obtaining a first failure number of the failed node; the failed node is a storage node in the distributed storage system that fails; When the first failure number is less than or equal to the second number of the global check blocks, obtaining, from the storage nodes corresponding to the original data, a second failure number of target failure nodes in the target storage nodes corresponding to the group; When the second failure number is greater than 0 and less than or equal to the first number, obtaining target data blocks and target local check blocks required for repairing lost data in the target failure node from target storage nodes other than the target failure node; The lost data is calculated according to the target data block and the target local check block to obtain repair data, and the repair data is stored in the storage node again.

8. A distributed storage system, characterized in that: The distributed storage system includes a storage management component and a plurality of storage nodes; wherein: The storage management component is used to obtain at least two original data blocks and a global check block corresponding to the original data; group the at least two original data blocks in order to obtain at least two groups corresponding to the original data; construct a first number of initial stripes according to the original data blocks and the global check blocks in each of the groups, and determine the first number of initial stripes as a first stripe set corresponding to the original data; the initial stripe includes a data fragment in each original data block in the group and a global check fragment in each global check block; and generate a first number of local check blocks corresponding to each of the groups according to the initial stripes in the first stripe set, and distribute and store the first stripe set and the local check blocks in the storage node; The storage management component is further used to perform the following steps: Obtaining a second number of the global check blocks; Classifying the original data blocks in the group into front-end data blocks and back-end data blocks according to positions of the original data blocks in the group; Calculate a local check block corresponding to the group according to the data fragments corresponding to the front-end data blocks and the data fragments corresponding to the back-end data blocks in the initial stripe and the second number; Adding the local check block corresponding to each of the groups to the first stripe set to obtain a second stripe set; the second stripe set includes a first number of target stripes, the target stripes include the initial stripe, and the local check segments corresponding to the initial stripe in the local check block corresponding to each group; The target stripes are distributed and stored in the storage nodes.

9. The distributed storage system according to claim 8, characterized in that: The storage management component is further used to perform the following steps: obtaining a third number of original data blocks in the group; Calculate the ratio of the third number to 2, and round down the ratio to obtain a fourth number; According to the positions of the original data blocks in the group, a fourth number of original data blocks located at the front of the group are determined as front-end data blocks; The original data blocks other than the front-end data blocks in the group are determined as the back-end data blocks.

10. The distributed storage system according to claim 8, characterized in that: The first stripe set includes a first initial stripe and a second initial stripe; the storage management component is further configured to perform the following steps: Calculate a first local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the first initial stripe; Calculate a second local check segment according to the data segment corresponding to the second data block in the first initial stripe and the data segment corresponding to the first data block in the second initial stripe; Calculate a third local check segment according to the data segment corresponding to the second initial data block in the second initial stripe and the data segment corresponding to the first data block in the second initial stripe; Calculate a fourth local check segment according to the data segment corresponding to the second initial stripe data block and the data segment corresponding to the first initial stripe data block; A local check block corresponding to the group is calculated according to the first local check segment, the second local check segment, the third local check segment, the fourth local check segment and the second number.

11. The distributed storage system according to claim 10, characterized in that: The storage management component is further used to perform the following steps: Merging the first local check segment, the second local check segment, the third local check segment and the fourth local check segment to obtain an original local check block corresponding to the group; The original local check block is modulo the second number to obtain a local check block corresponding to the group.

12. The distributed storage system according to claim 8, characterized in that: The storage management component is further used to perform the following steps: Determining a first number of local check blocks corresponding to the group; According to the first number, each original data block in the group is divided into a first number of data segments, and each global check block is divided into a first number of global check segments; constructing a first number of initial stripes according to the data segments of each original data block in each of the groups and the global check segments in each global check block; The first number of initial stripes is determined as a first stripe set corresponding to the original data.

13. The distributed storage system according to claim 8, characterized in that: The storage management component is further used to perform the following steps: Get the preset group quantity; Slicing the original data according to the preset number of groups to obtain at least two original data blocks corresponding to the original data; the number of original data blocks corresponding to the original data is an integer multiple of the preset number of groups; A polynomial linearly related to the at least two original data blocks is determined to obtain a global check block.

14. The distributed storage system according to claim 8, characterized in that: The storage management component is further used to perform the following steps: When at least one of the storage nodes in the distributed storage system fails, obtaining a first failure number of the failed node; The faulty node is a storage node in the distributed storage system where a fault occurs; When the first failure number is less than or equal to the second number of the global check blocks, obtaining, from the storage nodes corresponding to the original data, a second failure number of target failure nodes in the target storage nodes corresponding to the group; When the second failure number is greater than 0 and less than or equal to the first number, obtaining target data blocks and target local check blocks required for repairing lost data in the target failure node from target storage nodes other than the target failure node; The lost data is calculated according to the target data block and the target local check block to obtain repair data, and the repair data is stored in the storage node again.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the data storage method according to any one of claims 1 to 7 is implemented.

16. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the data storage method according to any one of claims 1 to 7.

17. A readable storage medium, characterized in that: When the instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data storage method described in any one of claims 1 to 7.