Multi-block cooperative updating method and device based on erasure codes, electronic equipment and medium

Abstract

The invention belongs to the technical field of computer storage, and particularly discloses a multi-block cooperative updating method and device based on erasure codes, electronic equipment and a medium. The method comprises the following steps: determining storage nodes where each update data block and each check block in a stripe are located; determining an update tree constructed by each storage node; the updating tree is constructed by taking the lowest total transmission cost generated in the strip data updating process as a target, taking each storage node as a node and taking the transmission cost between the storage nodes as an edge; the transmission cost is determined based on the network bandwidth between the storage nodes; and according to a node data transmission link in the update tree, updating each verification block by using the update data of each update data block. Through the data updating method and device, the updating speed of the erasure code data can be effectively increased, the updating time is shortened, and therefore the data updating efficiency of the erasure code storage system is improved.

Description

Technical Field

[0001] This application belongs to the field of computer storage technology, and more specifically, relates to a multi-block collaborative update method, device, electronic device, and medium based on erasure code. Background Art

[0002] With the rapid development of information technology, a vast amount of new data is generated every moment and needs to be persistently stored. To avoid data loss caused by device failures, existing storage systems usually adopt erasure code technology to provide redundancy to maintain system reliability and reduce storage costs.

[0003] However, in the prior art, when an erasure code storage system updates erasure code data, to ensure the consistency of erasure code encoding, the update of any data block in the same stripe will trigger the update of all parity blocks, which will introduce a large amount of disk I / O overhead and network traffic, resulting in a slow update speed and a long update time during the system data update process. Therefore, the existing erasure code storage system has the problem of low data update efficiency. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the purpose of this application is to better implement the update of erasure code data, aiming to solve the problem of low data update efficiency existing in the existing erasure code storage system.

[0005] To achieve the above object, in a first aspect, this application provides a multi-block collaborative update method based on erasure code, including:

[0006] Determine the storage nodes where each updated data block and each parity block in the stripe are located;

[0007] Determine an update tree constructed by each of the storage nodes; the update tree is constructed with the lowest total transmission cost generated during the stripe data update process as the goal, using each of the storage nodes as nodes and the transmission cost between each of the storage nodes as edges; the transmission cost is determined based on the network bandwidth between each of the storage nodes;

[0008] Update each parity block using the update data of each updated data block according to the node data transmission link in the update tree.

[0009] Optionally, before determining the update tree constructed by each of the storage nodes, the method further includes:

[0010] Construct a data update tree based on the first storage nodes where each updated data block is located and the transmission cost between each of the first storage nodes;

[0011] Construct a check update tree based on the second storage nodes where each of the check blocks is located, the root node of the data update tree, the transmission costs between the storage node corresponding to the root node and each of the second storage nodes, and the transmission costs between each of the second storage nodes; the storage nodes include the first storage nodes and the second storage nodes;

[0012] Determine the update tree according to the data update tree and the check update tree.

[0013] Optionally, the constructing a data update tree based on the first storage nodes where each of the updated data blocks is located and the transmission costs between each of the first storage nodes includes:

[0014] Randomly select a storage node from each of the first storage nodes as the root node of the data update tree, and use the other first storage nodes as the first non-root nodes of the data update tree;

[0015] Based on the transmission costs between each of the first storage nodes, determine the transmission costs between the root node and each of the first non-root nodes;

[0016] According to the transmission costs between the root node and each of the first non-root nodes, starting from the root node, sort each of the first non-root nodes in ascending order;

[0017] Based on the sorting result and the first preset policy, insert each of the first non-root nodes in turn starting from the root node to obtain the data update tree; the first preset policy is used to determine that the total cost of node link data transmission is the smallest after each insertion of a first non-root node.

[0018] Optionally, the constructing a check update tree based on the second storage nodes where each of the check blocks is located, the root node of the data update tree, the transmission costs between the storage node corresponding to the root node and each of the second storage nodes, and the transmission costs between each of the second storage nodes includes:

[0019] Use the root node of the data update tree as the root node of the check update tree, and use each of the second storage nodes as the second non-root nodes of the check update tree;

[0020] Based on the transmission costs between the storage node corresponding to the root node and each of the second storage nodes, determine the transmission costs between the root node and each of the second non-root nodes;

[0021] According to the transmission costs between the root node and each of the second non-root nodes, starting from the root node, sort each of the second non-root nodes in ascending order;

[0022] Based on the sorting result and the second preset policy, insert each of the second non-root nodes successively starting from the root node to obtain the verification update tree; the second preset policy is used to determine that the total cost of node link data transmission is minimized each time a second non-root node is inserted.

[0023] Optionally, updating each of the check blocks by using the update data of each update data block according to the node data transmission link in the update tree includes:

[0024] Calculate a local data increment block of the node corresponding to each update data block based on the update data of each update data block;

[0025] According to the node data transmission link in the update tree, based on the local data increment block of the node corresponding to each update data block, calculate the calculation data block transmitted by each node in the update tree to its next-level node node by node until the target calculation data block received by the node corresponding to each check block is determined;

[0026] Update each of the check blocks based on the target calculation data block corresponding to each check block.

[0027] Optionally, the method further includes:

[0028] For any node in the node data transmission link of the data update tree, when the sum of the number of received calculation data blocks and the number of local data increment blocks is less than the number of check blocks in the stripe, transmit the local data increment block of the any node and the calculation data blocks it receives to its next-level node;

[0029] Otherwise, based on the local data increment block of the any node and the calculation data blocks it receives, determine check increment blocks equal to the number of check blocks in the stripe, and transmit each of the check increment blocks to its next-level node.

[0030] In a second aspect, the present application provides an erasure code-based multi-block collaborative update device, including:

[0031] A first processing module, configured to determine the storage nodes where each update data block and each check block in the stripe are located;

[0032] A second processing module, configured to determine an update tree constructed by each of the storage nodes; the update tree is constructed with each of the storage nodes as nodes and the transmission cost between each of the storage nodes as edges with the goal of minimizing the total transmission cost generated during the stripe data update process; the transmission cost is determined based on the network bandwidth between each of the storage nodes;

[0033] An update module, configured to update each check block by using the update data of each update data block according to the data transmission links of the nodes in the update tree.

[0034] In a third aspect, the present application provides an electronic device, including: at least one memory for storing a program; at least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, the processor is caused to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0036] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor is caused to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0037] It can be understood that the beneficial effects of the above second aspect to fifth aspect can refer to the relevant descriptions in the first aspect, and will not be elaborated here.

[0038] Generally speaking, compared with the prior art by the above technical solution conceived by the present application, the following beneficial effects are achieved:

[0039] A multi-block cooperative update method, apparatus, electronic device and medium based on erasure code provided by the present application, by considering the network bandwidth between each update data block and each check block in a stripe and the storage nodes where they are located, adopts a tree structure, with the goal of minimizing the total transmission cost generated during the stripe data update process, constructs an update tree with each storage node as a node and the transmission cost between storage nodes as an edge. In this way, after the system determines the storage nodes where each update data block and each check block in the stripe are located, the corresponding update tree can be retrieved, and according to the node data transmission links in the update tree, the bandwidth resources between storage nodes can be fully utilized to update each check block by collaborating multiple update data blocks at a lower transmission cost, which can effectively improve the erasure code data update speed, reduce the update time, and thus improve the data update efficiency of the erasure code storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is one of the flowchart diagrams of the multi-block cooperative update method based on erasure code provided by an embodiment of the present application;

[0041] Figure 2It is the second flowchart diagram of the multi-block collaborative update method based on erasure code provided by the embodiments of the present application;

[0042] Figure 3 It is the structural schematic diagram of the multi-block collaborative update device based on erasure code provided by the embodiments of the present application;

[0043] Figure 4 It is the structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0044] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The terms "first" and "second" in the description and claims of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first storage node and the second storage node are used to distinguish different types of storage nodes, rather than to describe the specific order of the storage nodes.

[0046] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0047] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of processing units refers to two or more processing units, etc.; a plurality of elements refers to two or more elements, etc.

[0048] First, the technical terms involved in the embodiments of the present application are introduced.

[0049] (1) Strip

[0050] Through erasure code technology, the original data can be divided into several data blocks, and then through specific matrix operations on these data blocks, several check blocks are generated. These data blocks and check blocks form a strip.

[0051] Exemplarily, in a storage system based on the Reed-Solomon code RS(n,k), a file is divided into k original data blocks, and the original data blocks are encoded into n total data blocks through an encoding matrix, where n = m + k, and m is the number of parity blocks. The set of n total data blocks is called a "strip". A strip is the smallest encoding unit in the Reed-Solomon code. Usually, each data block in a strip is stored in different storage nodes respectively. If no more than m nodes fail, the failed data can be recovered from the remaining k nodes.

[0052] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

[0053] Figure 1 It is one of the flow diagrams of a multi-block collaborative update method based on the Reed-Solomon code provided by the embodiments of the present application. As Figure 1 shown, it includes:

[0054] Step S1, determine the storage nodes where each updated data block and each parity block in the strip are located;

[0055] Step S2, determine the update tree constructed by each storage node; the update tree is constructed with each storage node as a node and the transmission cost between each storage node as an edge with the goal of minimizing the total transmission cost generated during the strip data update process; the transmission cost is determined based on the network bandwidth between each storage node;

[0056] Step S3, update each parity block using the update data of each updated data block according to the node data transmission link in the update tree.

[0057] Specifically, the updated data block described in the embodiments of the present application refers to the data block that needs to be updated in the strip, and it can be specifically determined according to the data update request sent by the client.

[0058] The transmission cost described in the embodiments of the present application is determined based on the network bandwidth between each storage node, and specifically, the reciprocal of the network bandwidth between the storage nodes can be taken. That is to say, the larger the network bandwidth between two storage nodes, the faster the speed of data block transmission between these two storage nodes, that is, the lower the transmission cost.

[0059] The node data transmission link in the update tree described in the embodiments of the present application can be represented as the data transmission link with the lowest transmission cost, and it can determine the lowest transmission cost required to update each parity block using each updated data block in the strip.

[0060] In an embodiment of the present application, in step S1, after obtaining the system stripe data update information, according to the storage node positions of the data blocks and parity blocks in the same stripe, the storage nodes where each updated data block and each parity block in the system stripe are located can be determined.

[0061] In an embodiment of the present application, the transmission cost between the storage nodes where each updated data block and each parity block in the stripe are located can be pre-calculated. Taking each storage node as a node and the transmission cost between each storage node as an edge, the total transmission cost generated during the stripe data update process under different tree structure combination methods is calculated, and then an update tree that can minimize the total transmission cost generated during the stripe data update process is determined.

[0062] Further, in step S2, according to the storage nodes where each updated data block and each parity block in the stripe are located, the update tree constructed by these storage nodes can be found from the pre-constructed update tree.

[0063] Furthermore, in an embodiment of the present application, in step S3, after determining the update tree constructed by the storage nodes where each updated data block and each parity block are located, the node data transmission link with low transmission cost established in the update tree can be followed, and the update data of each updated data block is used for node-by-node data encoding until the data update of the nodes corresponding to each parity block is completed, thereby realizing the rapid update of each data block and parity block in the stripe.

[0064] The multi-block cooperative update method based on erasure code in the embodiment of the present application, by considering the network bandwidth between the storage nodes where each updated data block and each parity block in the stripe are located, adopts a tree structure method, with the goal of minimizing the total transmission cost generated during the stripe data update process. Taking each storage node as a node and the transmission cost between each storage node as an edge to construct an update tree. In this way, after the system determines the storage nodes where each updated data block and each parity block in the stripe are located, the corresponding update tree can be retrieved, and according to the node data transmission link in the update tree, the bandwidth resources between the storage nodes are fully utilized, and multiple updated data blocks are used to update each parity block with a lower transmission cost, which can effectively improve the data update speed of the erasure code, reduce the update time, and thus improve the data update efficiency of the erasure code storage system.

[0065] Based on the content of the above embodiment, as an alternative embodiment, before determining the update tree constructed by each storage node, the method further includes:

[0066] Construct a data update tree based on the first storage nodes where each updated data block is located and the transmission cost between each first storage node.

[0067] Construct a checksum update tree based on the second storage nodes where each checksum block is located, the root node of the data update tree, the transmission costs between the storage node corresponding to the root node and each second storage node, and the transmission costs between the second storage nodes; the storage nodes include first storage nodes and second storage nodes.

[0068] Determine an update tree according to the data update tree and the checksum update tree.

[0069] Specifically, the first storage nodes described in the embodiments of the present application refer to the storage nodes where each updated data block in the stripe is located.

[0070] The second storage nodes described in the embodiments of the present application refer to the storage nodes where each checksum block in the stripe is located.

[0071] In the embodiments of the present application, before determining the update tree constructed by each storage node, it is first necessary to construct an update tree by using the storage nodes where each updated data block and checksum block are located and the transmission costs between the storage nodes.

[0072] More specifically, in the embodiments of the present application, the update tree can be constructed step by step by dividing it into a data update tree and a checksum update tree. First, a data update tree DT can be constructed by using the first storage nodes where each updated data block is located and the transmission costs between the first storage nodes through minimizing the analysis of the node data transmission costs.

[0073] Based on the content of the above embodiments, as an alternative embodiment, constructing a data update tree based on the first storage nodes where each updated data block is located and the transmission costs between the first storage nodes includes:

[0074] Randomly select a storage node from each of the first storage nodes as the root node of the data update tree, and use the other first storage nodes as the first non-root nodes of the data update tree;

[0075] Based on the transmission costs between the first storage nodes, determine the transmission costs between the root node and each first non-root node;

[0076] According to the transmission costs between the root node and each first non-root node, starting from the root node, sort each first non-root node in ascending order;

[0077] Based on the sorting result and the first preset policy, insert each first non-root node in turn starting from the root node to obtain the data update tree; the first preset policy is used to determine that after inserting a first non-root node each time, the total cost of the node link data transmission is minimized.

[0078] Specifically, the first non-root node described in the embodiment of the present application refers to other nodes in the data update tree DT except the root node, which may specifically include internal nodes and leaf nodes in the data update tree.

[0079] In the embodiment of the present application, it is assumed that the update data block set d includes r update data blocks, and each update data block d i The first storage node where the data block is located constitutes a node set N. Since each updated data block d i There is a one-to-one correspondence with the nodes in the set N. Each node D in the set N i Used to store updated data blocks d i .

[0080] For each node D in N i , first from each node D i A storage node is randomly selected as the root node of the data update tree, and the other nodes are used as non-root nodes of the data update tree, namely, the first non-root nodes. Further, the transmission cost between the root node and each first non-root node can be directly obtained according to the transmission cost between each first storage node, and then the transmission cost from the root node to the remaining first non-root nodes can be used to update each node D in N in ascending order, starting from the root node. i Sort and create a priority queue S i To record the sorting results.

[0081] Furthermore, based on the priority queue S i The sorting result and the first preset strategy of the records are inserted into each first non-root node in sequence starting from the root node. Specifically, by selecting the priority queue S i The first node in D is used to quickly find the inserted node. It can be understood that the first node selected is the same as D among all the uninserted nodes. i The transmission cost is the lowest. Then, it initializes the data update tree DT with the root node where the update data block is located as the initial tree, and removes the root from N. Then, according to the first preset strategy, each node is greedily inserted to construct the data update tree DT.

[0082] Among them, in the first preset strategy, scan each node D in DT x , from the priority queue S x Get the first node D y , insert D into DT y As D x and calculate the total transmission cost c. Once the candidate link D y →D x Introducing a smaller update cost, fnode and cnode are used to record Dx and D y After traversing each candidate link of all nodes in DT, cnode is used as the child node of fnode in DT, and cnode is removed from the set N and each priority queue S i Once all nodes in N have been inserted into DT, DT is returned, and finally the data update tree DT with the lowest total cost of link data transmission is obtained.

[0083] The method of the embodiment of the present application realizes the rapid construction of the data update tree by randomly selecting the root node of the data update tree, improving the update speed of the erasure code data; at the same time, by designing a fast greedy algorithm to determine the insertion nodes and positions during the construction of the update tree, it can adapt to the rapidly changing network bandwidth, avoid the update scheme from becoming obsolete and no longer optimal, and further improve the reliability of the erasure code data update.

[0084] Further, in the embodiment of the present application, the check update tree PT is started to be constructed.

[0085] Based on the content of the above embodiment, as an optional embodiment, based on the second storage node where each check block is located, the root node of the data update tree, the transmission cost between the storage node corresponding to the root node and each second storage node, and the transmission cost between each second storage node, a check update tree is constructed, including:

[0086] Taking the root node of the data update tree as the root node of the check update tree, and taking each second storage node as the second non-root node of the check update tree;

[0087] Based on the transmission cost between the storage node corresponding to the root node and each second storage node, determine the transmission cost between the root node and each second non-root node;

[0088] According to the transmission cost between the root node and each second non-root node, starting from the root node, sort each second non-root node in ascending order;

[0089] Based on the sorting result and the second preset strategy, insert each second non-root node in turn starting from the root node to obtain the check update tree; the second preset strategy is used to determine that after each insertion of a second non-root node, the total cost of node link data transmission is the smallest.

[0090] Specifically, the second non-root node described in the embodiment of the present application refers to other nodes in the check update tree PT except the root node, which may specifically include internal nodes and leaf nodes in the data update tree.

[0091] In the embodiment of the present application, similarly, it can be assumed that the check block set p contains m check blocks, and each check block p iThe second storage nodes where they are located form a node set N. For each node P in N i , use the root node of the above data update tree DT as the root node of the check update tree PT, and use each second storage node as the second non-root node of the check update tree PT.

[0092] Furthermore, similarly, according to the transmission costs between each second storage node, the transmission costs between the root node and each second non-root node can be directly obtained. Furthermore, according to the transmission costs from the root node to the other remaining second non-root nodes, starting from the root node, each node P in N i is sorted in ascending order, and a priority queue S i can be created to record the sorting result.

[0093] Furthermore, with reference to the construction method of the aforementioned data update tree DT, based on the sorting result recorded in the priority queue S i and the second preset policy, each second non-root node is inserted in turn starting from the root node. Specifically, by selecting the first node in the priority queue S i , the insertion node can be quickly found. Then, it initializes the check update tree PT with the node root where the updated data block is located as the initial tree, and removes root from N. Subsequently, according to the second preset policy, each node is greedily inserted to construct the check update tree PT.

[0094] Among them, in the second preset policy, scan each node P in PT x , obtain the first node P from the priority queue S x , insert P in PT y as the child node of P y , and calculate the total transmission cost c. Once the candidate link P x →P y introduces a smaller update cost, fnode and cnode can be used to record P x and P x . After traversing each candidate link of all nodes in PT, cnode is used as the child node of fnode in PT, and cnode is removed from the set N and each priority queue S y . Once all nodes in N are inserted into PT, PT is returned, and finally the check update tree PT with the lowest total transmission cost of link data is obtained. i

[0095] The method of the embodiment of the present application determines the insertion node and position during the construction of the check update tree through the designed above-mentioned fast greedy algorithm, can adapt to the rapidly changing network bandwidth, avoid the update scheme from becoming obsolete and no longer being optimal, and further improve the speed and reliability of erasure code data update.​

[0096] Further, in the embodiments of the present application, after separately constructing the verification update tree and the verification update tree, the overlapping root node positions in the two trees are covered, and the verification update tree and the verification update tree are spliced together, and finally an update tree with a dual-tree structure can be obtained.

[0097] The method of the embodiments of the present application constructs the data update tree and the verification update tree step by step by considering the differences in the update methods of the update data blocks and the verification data blocks to obtain the entire update tree, ensuring the stable and reliable structure of the constructed update tree, and further improving the reliability and accuracy of the system data update.

[0098] Based on the content of the above embodiments, as an alternative embodiment, according to the node data transmission link in the update tree, each verification block is updated by using the update data of each update data block, including:

[0099] Based on the update data of each update data block, calculate the local data increment block of the node corresponding to each update data block;

[0100] According to the node data transmission link in the update tree, based on the local data increment block of the node corresponding to each update data block, calculate the calculated data block transmitted by each node in the update tree to its next-level node node by node until determining the target calculated data block received by the node corresponding to each verification block;

[0101] Update each verification block based on the target calculated data block corresponding to each verification block.

[0102] Specifically, the local data increment block described in the embodiments of the present application refers to the data increment block locally generated by the storage node where the update data block is located, and it can be specifically calculated according to the original data of the update data block stored in the storage node and the update data of the update data block.

[0103] The calculated data block described in the embodiments of the present application refers to the data used by a node to calculate the data transmitted to its next-level node.

[0104] The target calculated data block described in the embodiments of the present application refers to the calculated data block received by the node corresponding to each verification block, and it contains the data for updating the verification block.

[0105] In the embodiments of the present application, in an erasure code RS(n,k) storage system, when the erasure code is updated, the RS code is usually configured by two parameters k and m, and it encodes k data blocks {d 1 ,…,d k} into m equally sized verification blocks {p 1 ,…,p m} through the following formula.

[0106]

[0107] wherein, X i,j (1 ≤ i ≤ k and 1 ≤ j ≤ m) represents the updated data block d i for calculating the parity block p j The encoding coefficients. Among them, once the parameters (k, m) are determined, these coefficients can be determined by the Vandermonde matrix. The set composed of these k + m blocks is called a stripe and is distributed on k + m nodes.

[0108] If an updated data block d i is updated to d i ′, then all m parity blocks in the stripe where d i is located need to be updated to ensure encoding consistency. Specifically, each parity block p j is updated to a new parity block p j ′ through the following update formula.

[0109] p j ′ = p j + X i,j (d i ′ - d i ) = p j + X i,j ΔD i = p j + ΔP j ;

[0110] Here, p j can be updated using the data increment d i ′ - d i (denoted by ΔD i ) or the parity increment X i,j (d i ′ - d i )(denoted by ΔP j ).

[0111] In addition, assume that r (1 ≤ r ≤ k) data blocks {d 1 , …, d r} are updated to {d 1 ′, …, d r ′}, and each new parity block p j ′ can be calculated and determined through the following formula, that is:

[0112]

[0113] In the embodiments of the present application, based on the update data of each updated data block d i , the update data of each updated data block d iLocal data increment block ΔD of the corresponding node i 。

[0114] Further, according to the node data transmission link in the update tree, based on the local data increment block ΔD of the node corresponding to each update data block i calculate, for each node in the update tree, the calculation data block transmitted by the node to its next-level node one by one until the target calculation data block received by the node corresponding to each check block is determined. Among them, the target calculation data block can use the above check block update data formula to calculate the target calculation data block ΔP corresponding to each check block j 。

[0115] Based on the content of the above embodiments, as an alternative embodiment, the method further includes:

[0116] For any node in the node data transmission link of the data update tree, when the sum of the number of received calculation data blocks and the number of local data increment blocks is less than the number of check blocks in the stripe, transmit the local data increment block of any node and the calculation data blocks it receives to its next-level node;

[0117] Otherwise, based on the local data increment block of any node and the calculation data blocks it receives, determine the check increment blocks with the same number as the number of check blocks in the stripe, and transmit each check increment block to its next-level node.

[0118] Specifically, in the embodiments of the present application, when each node D of the data update tree DT i performs data transmission, the calculation data blocks include data increment blocks ΔD i and check increment blocks ΔP j two categories.

[0119] Specifically, for the data update tree DT that transmits data from bottom to top, when it is determined that the number of data blocks to be pre-transmitted by any node in its node data transmission link is less than the number of check blocks m in the stripe, the data transmitted by the link is the data increment block ΔD i ; otherwise, based on the local data increment block of any node and the calculation data blocks it receives, determine the check increment blocks with the same number as the number of check blocks in the stripe, and transmit the check increment blocks ΔP j 。

[0120] Exemplarily, the number of check blocks m in the stripe is 2. One implementation is that in the data update tree DT, if the number of received calculation data blocks of the leaf node D i is 0 and 1 local data increment block ΔD i is generated, and the sum of the two is less than the number of check blocks in the stripe, then the leaf node D i only needs to transmit the local data increment block ΔD iIt can be transmitted to its next - level node.

[0121] Among them, the number of data blocks pre - transmitted by any node is equal to the sum of the number of computed data blocks received by the node and the number of local data increment blocks. Generally, the number of local data increment blocks is taken as 1.

[0122] Another implementation is that in the data update tree DT, for an internal node D i The number of computed data blocks received is 2, and 1 local data increment block ΔD is generated i , and the sum of the two is greater than the number of parity blocks in the stripe. Then this internal node D i will, according to the above - mentioned parity block update formula, calculate two temporary parity increment blocks ΔP i ' and ΔP 1 ' based on the number of computed data blocks received and the local data increment block ΔD 2 , and then transmit ΔP 1 ' and ΔP 2 ' to its next - level node.

[0123] Through node - by - node data block transmission, finally the root node will calculate m parity increment blocks from all the computed data blocks received and send them to m parity nodes P j .

[0124] In addition, it should be noted that for the top - down transmitted parity update tree PT, the node data transmission link always transmits parity increment blocks. Each non - leaf node in the parity update tree PT completes the update through the following three steps. First, it receives parity increment blocks from the parent node (data root node or other parity nodes), and the blocks it receives are the parity increment blocks of all parity blocks in the tree structure with itself as the root node. Then, it uses the above - mentioned parity block update formula to update itself. Finally, it sends the other received parity increment blocks to its next - level node. Here, the leaf nodes in the parity update tree PT only receive one parity increment block and update themselves.

[0125] The method of the embodiment of the present application adjusts the data calculation and transmission methods of each node in the data update tree by considering the relationship between the number of data blocks pre - transmitted by the nodes in the data update tree and the number of parity blocks in the stripe, ensuring the reliability of the data blocks transmitted to the corresponding nodes of each parity block and improving the fast effectiveness of the internal data update mechanism of the update tree.

[0126] Furthermore, in the embodiment of the present application, by using the target computed data block ΔP j corresponding to each parity block and the original data p j corresponding to each parity block, the updated parity blocks p j′, thus completing the data update for each check block.

[0127] The method of the embodiment of the present application uses the tree - shaped structure transmission link of the update tree, combines the local data increment blocks of the nodes corresponding to each update data block, updates each node in the update tree node by node. Through this update method, a more scientific and efficient data update method is realized, which can minimize the total transmission cost, improve the data update performance, and ensure the reliability of the system.

[0128] Figure 2 It is the second flow schematic diagram of the multi - block collaborative update method based on erasure code provided by the embodiment of the present application. As Figure 2 shown, in the embodiment of the present application, the process of constructing an update tree from r = 5 update data blocks {d 1 , d 2 , d 3 , d 4 , d 5} and m = 2 check blocks {p 1 , p 2} required for update in the set is shown. First, sort the storage nodes where each update data block and each check block are located to determine the node set N = {D 1 , D 2 , D 3 , D 4 , D 5}, and create a priority queue to record the sorting result. Then, construct the data update tree DT and the check update tree PT.

[0129] Among them, the transmission costs between the nodes in the node set N = {D 1 , D 2 , D 3 , D 4 , D 5} are shown in the upper table in the figure. The transmission costs between the storage nodes where the check blocks {P 1 , P 2} are located and the transmission costs between each storage node and the root node are shown in the lower table in the figure.

[0130] In this embodiment, an example is given by randomly selecting the storage node D 1 where the update data block d 1 is located as the root node. According to the transmission costs between the nodes, a priority queue is created, which can be represented as S 1 , S 2 , …, S 5 . Before inserting the first node, determine the data update tree DT = {D 1}, and sort according to the transmission costs between the root node and each first non - root node. The priority queue recording the sorting result can be represented as S1 = {D 2 , D 3 , D 4 , D 5}. The only node in DT is D 1 , obtaining the first node D 1 from S 2 and inserting it as a child node of D 1 . Before inserting the second node, DT = {D 1 , D 2}, and at this time the corresponding priority queue S 2 = {D 4 , D 5 , D 3}.

[0131] During the construction of the data update tree, according to the greedy algorithm designed in the first preset strategy, traverse each node in the data update tree DT and try each candidate link. For node D 2 , obtain the first node D 2 from S 4 , and calculate the transmission cost c = 6 for inserting the candidate link D 4 → D 2 ; while for node D 1 , obtain the current first node D 1 from S 3 , and calculate the transmission cost c = 5 for inserting the candidate link D 3 → D 1 . Finally, it inserts the D 3 node into node D 1 with the current minimum transmission cost c = 5. Then, in the above - mentioned manner of minimizing the transmission cost, insert the remaining two nodes D 4 and D 5 in sequence, retaining the tree - shaped structure of the "red arrow" transmission path, so that the total transmission cost of the data update tree DT's transmission link is minimized.

[0132] In this embodiment, the construction process of the verification update tree PT is similar to the above, determining the structure of the verification update tree PT with the minimum total transmission cost, which will not be elaborated here. Finally, the double - tree - structured update tree shown in the figure can be obtained.

[0133] Further, according to the node data transmission link of the update tree constructed above, use the update data of each update data block to update each verification block. Specifically, node D 2 receives two data increment blocks ΔD 4 and D 5 from D 4 and ΔD 5, and calculate the local data increment block ΔD 2 . It can be seen that node D 2 The sum W of the number of data increment blocks received by the node and the local data increment block 2,1 = 3 > 2 = m. Therefore, node D 2 Can follow the aforementioned check block update formula to combine these three data increment blocks ΔD 4 , ΔD 5 And ΔD 2 Aggregate into two temporary check increment blocks ΔP' 1 And ΔP' 2 , and send them to the root node D 1 . Finally, D 1 The two temporary check increment blocks collected and the data increment block ΔD transmitted by node D 3 Perform encoding calculations to obtain two check increment blocks ΔP 3 And ΔP 1 And ΔP 2 , and send them to the corresponding check nodes in the check update tree PT for data update of the check blocks.

[0134] Next, the multi-block collaborative update device based on erasure codes provided by the present application will be described. The multi-block collaborative update device based on erasure codes described below can be correspondingly referred to the multi-block collaborative update method described above.

[0135] Figure 3 Is the structural schematic diagram of the multi-block collaborative update device based on erasure codes provided by the embodiments of the present application. As Figure 3 Shown, the device includes:

[0136] The first processing module 10 is used to determine the storage nodes where each updated data block and each check block in the stripe are located;

[0137] The second processing module 20 is used to determine the update tree constructed by each storage node; the update tree is constructed with the goal of minimizing the total transmission cost generated during the stripe data update process, using each storage node as a node and the transmission cost between each storage node as an edge; the transmission cost is determined based on the network bandwidth between each storage node;

[0138] The update module 30 is used to update each check block using the update data of each update data block according to the data transmission link of each node in the update tree.

[0139] It can be understood that the detailed function implementation of the above-mentioned each unit / module can refer to the introduction in the foregoing method embodiments, and will not be elaborated here.

[0140] It should be understood that the above device is used to execute the method in the above embodiment. For the corresponding program modules in the device, their implementation principles and technical effects are similar to those described in the above method. The working process of the device can refer to the corresponding process in the above method, which will not be elaborated here.

[0141] The erasure code-based multi-block cooperative update device according to the embodiment of the present application, by considering the network bandwidth between each updated data block and each parity block in the stripe and adopting a tree structure, aims to minimize the total transmission cost generated during the stripe data update process. Taking each storage node as a node and the transmission cost between storage nodes as an edge, an update tree is constructed. In this way, after the system determines the storage nodes where each updated data block and each parity block in the stripe are located, the corresponding update tree can be retrieved, and according to the node data transmission link in the update tree, the bandwidth resources between storage nodes can be fully utilized to cooperate multiple updated data blocks to update each parity block at a lower transmission cost, which can effectively improve the erasure code data update speed, reduce the update time, and thus improve the data update efficiency of the erasure code storage system.

[0142] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, as Figure 4 shown. The electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute the method in the above embodiment.

[0143] In addition, when the logical instructions in the above memory 430 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0144] Based on the method in the above embodiment, the embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on the processor, it causes the processor to execute the method in the above embodiment.

[0145] Based on the method in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, it causes the processor to execute the method in the above embodiments.

[0146] It can be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0147] The method steps in the embodiment of the present application may be implemented in a hardware manner or by a processor executing software instructions. The software instructions may be composed of corresponding software modules. The software modules may be stored in a random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, hard disks, removable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may be located in an ASIC.

[0148] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0149] It can be understood that the various digital numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0150] It should be understood that expressions such as "including" and "may include" that can be used in the present application indicate the existence of the disclosed functions, operations, or constituent elements, and do not limit one or more additional functions, operations, and constituent elements. In the present application, terms such as "including" and / or "having" can be interpreted as indicating a specific characteristic, number, operation, constituent element, component, or a combination thereof, but cannot be interpreted as excluding the existence or possibility of addition of one or more other characteristics, numbers, operations, constituent elements, components, or a combination thereof.

[0151] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-block collaborative update method based on erasure coding, characterized in that: include: Determine the storage node where each updated data block and each check block in the stripe is located; Determining an update tree constructed by each of the storage nodes; The update tree is constructed with the goal of minimizing the total transmission cost generated during the stripe data update process, with each of the storage nodes as a node and the transmission cost between the storage nodes as an edge; The transmission cost is determined based on the network bandwidth between each of the storage nodes; According to the node data transmission link in the update tree, each of the check blocks is updated using the update data of each update data block.

2. The multi-block collaborative update method based on erasure coding according to claim 1 is characterized in that: Before determining the update tree constructed by each of the storage nodes, the method further includes: constructing a data update tree based on the transmission cost between the first storage nodes where each of the update data blocks is located and each of the first storage nodes; Based on the second storage nodes where each of the check blocks is located and the root node of the data update tree, the transmission cost between the storage node corresponding to the root node and each of the second storage nodes, and the transmission cost between each of the second storage nodes, a check update tree is constructed; the storage nodes include the first storage node and the second storage node; The update tree is determined according to the data update tree and the verification update tree.

3. The multi-block collaborative update method based on erasure coding according to claim 2 is characterized in that: The constructing of a data update tree based on the transmission cost between the first storage nodes where each of the updated data blocks is located and each of the first storage nodes comprises: Randomly select a storage node from the first storage nodes as the root node of the data update tree, and select the other first storage nodes as the first non-root nodes of the data update tree; Determine, based on the transmission costs between the first storage nodes, a transmission cost between the root node and each of the first non-root nodes; According to the transmission cost between the root node and each of the first non-root nodes, taking the root node as the starting point, sorting the first non-root nodes in ascending order; Based on the sorting result and the first preset strategy, each of the first non-root nodes is inserted in sequence starting from the root node to obtain the data update tree; the first preset strategy is used to determine that the total cost of node link data transmission is minimized after each insertion of a first non-root node.

4. The multi-block collaborative update method based on erasure coding according to claim 2 is characterized in that: The constructing of the check update tree based on the second storage nodes where each of the check blocks is located and the root node of the data update tree, the transmission cost between the storage node corresponding to the root node and each of the second storage nodes, and the transmission cost between each of the second storage nodes, comprises: Using the root node of the data update tree as the root node of the verification update tree, and using each of the second storage nodes as a second non-root node of the verification update tree; Determine a transmission cost between the root node and each of the second non-root nodes based on a transmission cost between the storage node corresponding to the root node and each of the second storage nodes; According to the transmission cost between the root node and each of the second non-root nodes, taking the root node as the starting point, sorting each of the second non-root nodes in ascending order; Based on the sorting result and the second preset strategy, each of the second non-root nodes is inserted in sequence starting from the root node to obtain the verification update tree; the second preset strategy is used to determine that the total cost of node link data transmission is minimized after each insertion of a second non-root node.

5. The method for multi-block collaborative updating based on erasure coding according to any one of claims 2 to 4, characterized in that: The updating of each of the check blocks using the update data of each of the update data blocks according to the node data transmission link in the update tree comprises: Based on the update data of each update data block, calculate the local data increment block of the node corresponding to each update data block; According to the node data transmission link in the update tree, based on the local data increment block of the node corresponding to each update data block, the calculation data block transmitted by each node in the update tree to its next-level node is calculated node by node, until the target calculation data block received by the node corresponding to each check block is determined; Based on the target calculation data block corresponding to each of the check blocks, each of the check blocks is updated.

6. The multi-block collaborative update method based on erasure coding according to claim 5 is characterized in that: The method further comprises: For any node in the node data transmission link of the data update tree, when the sum of the number of received calculation data blocks and the number of local data increment blocks is less than the number of check blocks in the stripe, the local data increment blocks of any node and the received calculation data blocks are transmitted to the next level node; Otherwise, based on the local data incremental blocks of any node and the calculation data blocks it receives, the check incremental blocks having the same number as the check blocks in the stripe are determined, and each of the check incremental blocks is transmitted to its next-level node.

7. A multi-block collaborative update device based on erasure code, characterized in that: include: A first processing module is used to determine the storage node where each updated data block and each check block in the stripe are located; A second processing module, used for determining an update tree constructed by each of the storage nodes; The update tree is constructed with the goal of minimizing the total transmission cost generated during the stripe data update process, with each of the storage nodes as a node and the transmission cost between the storage nodes as an edge; The transmission cost is determined based on the network bandwidth between each of the storage nodes; An update module is used to update each of the check blocks using the update data of each update data block according to the data transmission link of each node in the update tree.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images