Data processing method, electronic device, storage medium and program product

Abstract

The invention provides a data processing method, electronic equipment, a storage medium and a program product. A main storage node of the to-be-written data in the distributed storage cluster determines a standby storage node based on the master-slave backup matrix, sends a copy of the to-be-written data to the standby storage node, and sends a write completion response to the client after each standby storage node determines that the copy is received, therefore, the performance of the data write operation on a key path in the distributed storage cluster is equivalent to that of a copy mechanism, and high-performance data storage is realized. Meanwhile, under the condition that the standby storage node determines that the copies of the to-be-written data sent by the main storage node corresponding to the check line numbers are received, the corresponding check blocks are calculated and stored on the basis of the received copies, so that k data copies are converted into one check block, calculation and storage of the check blocks in an erasure code mechanism are achieved, and the reliability of the erasure code mechanism is improved. The storage overhead is equivalent to that of an erasure code mechanism and is far lower than that of a copy mechanism. In conclusion, a redundancy mechanism with high performance and low storage space overhead is realized.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of storage technology, and in particular, to a data processing method, electronic device, storage medium, and program product. Background Art

[0002] In a distributed storage cluster, a storage node failure may render the stored data inaccessible, impacting data availability. To ensure data availability, distributed storage clusters typically use data redundancy mechanisms. Common data redundancy mechanisms include replication and erasure coding.

[0003] In a replication mechanism, multiple copies of data are stored on different storage nodes. Even if some storage nodes fail, the remaining storage nodes still store copies of the data. However, replication consumes a lot of storage space. When the cost of storage media is high, the cost of a distributed storage cluster using replication will be very high.

[0004] Erasure coding ensures data availability with low storage space overhead by encoding original data as redundant data in the form of parity. Striping, a fundamental concept in erasure coding, refers to a set of original data and a set of parity blocks. Maintaining the striping concept often requires significant operational overhead in distributed storage clusters, resulting in performance that is typically far inferior to replication. Summary of the Invention

[0005] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of the present specification, a data processing method is proposed, which is applied to a storage node in a distributed storage cluster. A master-slave backup matrix is ​​deployed in the storage node, the number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster, the row numbers and column numbers both correspond to the node numbers of the storage nodes, the elements of the master-slave backup matrix have values ​​of 0 or 1, each row and each column has k elements with a value of 1, and for any two rows, there exists one column such that the element values ​​in this column of both rows are 1. When the storage node is a master storage node for data to be written, the method includes:

[0007] Receive a data write request sent by a client, and store the data to be written carried in the data write request;

[0008] Searching the master-slave backup matrix for a column number whose element value is 1 in the row corresponding to the storage node as a candidate column number;

[0009] Select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, where p is the fault tolerance threshold of the distributed storage cluster and p is less than k;

[0010] Sending a copy of the to-be-written data to the corresponding backup storage node according to the backup storage node number, and sending a write completion response to the client after each backup storage node determines that the copy has been received;

[0011] In which, the backup storage node is used to cache the copy of the data to be written after receiving the copy of the data to be written, and when it is determined that the copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it has been received, calculate the corresponding check block based on each received copy and store it, and the check row number is the row number with the element value of 1 in the column corresponding to the backup storage node in the master-slave backup matrix.

[0012] According to a second aspect of one or more embodiments of the present specification, a data processing method is proposed, which is applied to a storage node in a distributed storage cluster, wherein a master-slave backup matrix is ​​deployed in the storage node, the number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster, the row numbers and column numbers both correspond to the node numbers of the storage nodes, the elements of the master-slave backup matrix have values ​​of 0 or 1, each row and each column has k elements with a value of 1, and for any two rows, there exists one column such that the element values ​​in this column of both rows are 1, and when the storage node is a backup storage node for data to be written, the method includes:

[0013] receiving a copy of the data to be written sent by a primary storage node of the data to be written, and caching the copy;

[0014] Sending a receipt confirmation to the primary storage node, so that the primary storage node can send a write completion response to the client after receiving the receipt confirmation returned by each backup storage node of the data to be written;

[0015] Upon determining that a copy of the to-be-written data has been received from the primary storage node corresponding to each check row number maintained by the primary storage node, calculating and storing a corresponding check block based on each received copy, wherein the check row number is the row number of the column corresponding to the primary storage node in the primary-slave backup matrix having an element value of 1;

[0016] Among them, the primary storage node is used to search the column number with the element value of 1 in the row corresponding to the primary storage node in the master-slave backup matrix as a candidate column number after receiving the data write request sent by the client, select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, and send a copy of the data to be written according to the backup storage node number, where p is the fault tolerance threshold of the distributed storage cluster, and p is less than k.

[0017] Optionally, the method further includes:

[0018] receiving a data read request sent by a client, wherein the data read request is sent by the client after determining that a primary storage node of the data to be read is faulty;

[0019] If the node stores a check block corresponding to the to-be-read data specified by the data read request, determine the storage nodes corresponding to the check row numbers, excluding the failed node, as primary recovery storage nodes;

[0020] Obtaining a copy used for calculating the check block from the first-level recovery storage node as a recovery copy;

[0021] restoring the to-be-read data based on the check block and the recovery copy;

[0022] Return the data to be read to the client.

[0023] Optionally, when there is a faulty node among the first-level recovery storage nodes, the process of obtaining a recovery copy from the faulty node includes:

[0024] Determine, based on the column with the element value 1 in the row where the faulty node is located in the master-slave backup matrix, a storage node storing the check block of the recovery copy as a secondary recovery storage node;

[0025] Sending a recovery instruction for the recovery copy to the secondary recovery storage node, so that the secondary recovery storage node can recover the recovery copy based on the check block stored therein;

[0026] Receive the recovery copy returned by the secondary recovery storage node.

[0027] Optionally, the method further includes:

[0028] After storing the check block, the cached copies of each to-be-written data are deleted.

[0029] According to a third aspect of one or more embodiments of the present specification, a data processing method is proposed, which is applied to a client accessing a distributed storage cluster, wherein a master-slave backup matrix is ​​deployed in the storage nodes of the distributed storage cluster, the number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster, the row numbers and column numbers both correspond to the node numbers of the storage nodes, the elements of the master-slave backup matrix have values ​​of 0 or 1, each row and each column has k elements with a value of 1, and for any two rows, there exists a column such that the element values ​​of the column in both rows are 1, the method comprising:

[0030] Determine the primary storage node for the data to be written;

[0031] Send a data write request to the primary storage node so that the primary storage node stores the data to be written carried in the data write request, and search the master-slave backup matrix for the column number with the element value 1 in the row corresponding to the current storage node as a candidate column number; select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, where p is the fault tolerance threshold of the distributed storage cluster and p is less than k; and send a copy of the data to be written to the backup storage node according to the backup storage node number;

[0032] receiving a write completion response returned by the primary storage node, the response being sent by the primary storage node after each backup storage node determines that the copy has been received;

[0033] In which, the backup storage node is used to cache the copy of the data to be written after receiving the copy of the data to be written, and when it is determined that the copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it has been received, calculate the corresponding check block based on each received copy and store it, and the check row number is the row number with the element value of 1 in the column corresponding to the backup storage node in the master-slave backup matrix.

[0034] Optionally, the client is also deployed with the master-slave backup array, and the method further includes:

[0035] Determine the primary storage node for the data to be read;

[0036] In the event of a failure of the primary storage node, determining the backup storage node number of the to-be-read data based on the column number of the element value 1 in the row corresponding to the primary storage node in the master-slave backup matrix;

[0037] sending a data read request to the standby storage node according to the standby storage node number, so that the standby storage node determines the storage node corresponding to the check row number as a first-level recovery storage node if the check block corresponding to the data to be read is stored therein; and obtaining from the first-level recovery storage node a copy used for calculating the check block as a recovery copy, so as to recover the data to be read based on the check block and the recovery copy;

[0038] Receive the to-be-read data returned by the backup storage node.

[0039] According to a fourth aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the aforementioned method by running the executable instructions.

[0040] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the aforementioned method are implemented.

[0041] According to a sixth aspect of one or more embodiments of this specification, a computer program product is proposed, comprising a computer program / instruction, which implements the steps of the aforementioned method when executed by a processor.

[0042] By adopting the data processing scheme provided in this specification, the primary storage node of the data to be written can determine the backup storage node for the data to be written based on the master-slave backup matrix, so as to send a copy of the data to be written to the backup storage node, and after each backup storage node determines that it has received the copy, it sends a write completion response to the client, thereby making the performance of the data write operation on the critical path in the distributed storage cluster comparable to that of the copy mechanism, without the need to organize stripes, and achieving high-performance data storage. At the same time, when the backup storage node determines that it has received a copy of the data to be written sent by the primary storage node corresponding to each check row number it maintains, it calculates and stores the corresponding check blocks based on each received copy, and then converts k data copies into a check block, realizing the calculation and storage of the check blocks in the erasure code mechanism, and the storage overhead is comparable to that of the erasure code mechanism, which is much lower than that of the copy mechanism. In summary, by adopting the data processing scheme provided in this specification, a redundancy mechanism with both high performance and low storage space overhead is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of an access architecture of a distributed storage cluster provided by an exemplary embodiment.

[0044] Figure 2 It is a flowchart of a data processing method provided by an exemplary embodiment.

[0045] Figure 3 It is a flowchart of another data processing method provided by an exemplary embodiment.

[0046] Figure 4 It is a flowchart of another data processing method provided by an exemplary embodiment.

[0047] Figure 5 It is a schematic diagram of a master-slave backup array provided by an exemplary embodiment.

[0048] Figure 6 The figure is a flow chart of a data recovery method provided by an exemplary embodiment.

[0049] Figure 7 It is a structural diagram of a device provided by an exemplary embodiment.

[0050] Figure 8 It is a block diagram of a data processing device provided by an exemplary embodiment.

[0051] Figure 9 It is a block diagram of another data processing device provided by an exemplary embodiment.

[0052] Figure 10 It is a block diagram of another data processing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0053] In a distributed storage cluster, a storage node failure may render the stored data inaccessible, impacting data availability. To ensure data availability, distributed storage clusters typically use data redundancy mechanisms. Common data redundancy mechanisms include replication and erasure coding.

[0054] In a replication mechanism, multiple copies of data are stored on different storage nodes. Even if some storage nodes fail, the remaining storage nodes still store copies of the data. However, replication consumes a lot of storage space. When the cost of storage media is high, the cost of a distributed storage cluster using replication will be very high.

[0055] Erasure coding ensures data availability with low storage space overhead by encoding original data as redundant data in the form of parity. Striping, a fundamental concept in erasure coding, refers to a set of original data and a set of parity blocks. Maintaining the striping concept often requires significant operational overhead in distributed storage clusters, resulting in performance that is typically far inferior to replication.

[0056] Taking a distributed object storage cluster as an example, if each object is split into multiple raw data blocks and then encoded into stripes, since users generally access data at the object granularity, each read and write operation requires accessing multiple storage nodes, resulting in increased fanout for read and write operations. When the cluster is heavily loaded, the latency of read and write operations is limited by the slowest storage node accessed, making the tail latency problem more severe. In addition, in scenarios with fast storage media and small object sizes (such as distributed caches), the number of network requests will increase significantly, and frequent network requests will significantly reduce cluster performance.

[0057] If objects are not split into data blocks, multiple objects need to be organized into a stripe. This means that the storage cluster needs to incur some form of operational overhead to organize multiple objects stored on different storage nodes into a stripe. For example, some storage clusters organize stripes by proxying all read and write requests through a central node, but if this node fails, the entire storage cluster will be inaccessible. For another example, some storage clusters pre-allocate stripes according to the object key (such as the number) (such as objects 0-3 are allocated to stripe 0, objects 4-7 are allocated to stripe 1, etc.), but this strategy will result in wasted storage space when object writes are dispersed. Assume that only object 0 is written and objects 1-3 are not written. Stripe 0 only stores one valid object and two check blocks. The storage efficiency is comparable to the replication mechanism, and the advantages of the erasure code mechanism are lost.

[0058] This shows that distributed storage clusters, regardless of whether data is split or not, will incur significant operational overhead. This overhead stems from the cluster's need to organize data into stripes and distribute data across storage nodes in the cluster according to the stripe requirements.

[0059] To address the above issues, this specification provides a data processing solution that implements data reading and writing in an erasure coding mechanism without relying on the concept of stripes. Because it is stripe-free, it avoids the operational overhead associated with stripes, resulting in read and write performance comparable to that of a replication mechanism. Meanwhile, its storage space overhead is roughly equivalent to that of existing erasure coding mechanisms, thus achieving a redundancy mechanism that achieves both high performance and low storage space overhead.

[0060] Figure 1 FIG. 1 is a schematic diagram of an access architecture of a distributed storage cluster provided by an exemplary embodiment. Figure 1 As shown, a distributed storage cluster may include multiple storage nodes, such as storage node 11, storage node 12, storage node 13, ... storage node 1N, etc. The storage node may be a physical server including an independent host, or a virtual server hosted by a host cluster.

[0061] The access architecture of the distributed storage cluster also includes a client 2, which can access the distributed storage cluster to read and write data. The client can be a PC, mobile phone, tablet device, laptop computer, wearable device (such as smart glasses, smart watch, etc.), etc., and one or more embodiments of this specification are not limited to this.

[0062] As for the network for interaction between the client and the distributed storage cluster, a wired or wireless network can be selected to implement communication based on the communication method supported by the corresponding device, and this manual does not limit this.

[0063] Figure 2It is a flowchart of a data processing method provided by an exemplary embodiment.

[0064] In this embodiment, the data processing method can be applied to the main storage node in the distributed storage cluster. The main storage node can be specified by the client. For example, when the client performs data write access, it can determine a storage node as the main storage node in the storage nodes of the distributed storage cluster based on the key value of the data to be written.

[0065] In this embodiment, each storage node in the distributed storage cluster can be deployed with a master-slave backup matrix corresponding to the distributed storage cluster. The number of rows and columns of the master-slave backup matrix are the number of storage nodes in the distributed storage cluster, and the row number and column number correspond to the node number of the storage node, that is, each storage node can correspond to a row and a column in the master-slave backup matrix, and the corresponding row number and column number are the same. For example, storage node 1 corresponds to the first row and the first column in the master-slave backup matrix. The elements of the master-slave backup matrix have values ​​of 0 or 1, and each row and column has k elements with a value of 1, and for any two rows, there is one column such that the element values ​​of the two rows in this column are all 1. Wherein, k is the sum of a power of a prime number and 1.

[0066] Please refer to Figure 2 , the data processing method may include the following steps:

[0067] Step 202: Receive a data write request sent by a client, and store the data to be written carried in the data write request.

[0068] Step 204: Search the master-slave backup matrix for the column number whose element value is 1 in the row corresponding to the current storage node as a candidate column number.

[0069] Step 206 : Select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, where p is the fault tolerance threshold of the distributed storage cluster, and p is smaller than k.

[0070] In this embodiment, after receiving the data to be written from the client, the primary storage node may determine a backup storage node for the data to be written. The number of backup storage nodes may be p+1. p is the fault tolerance threshold of the distributed storage cluster, and p is less than k.

[0071] For example, the node number of the backup storage node can be determined based on the master-slave backup matrix, and thus the backup storage node can be determined. Specifically, the master-slave backup matrix can be used to search for candidate column numbers with an element value of 1 in the row corresponding to the current storage node (i.e., the primary storage node itself). The number of candidate column numbers is k. Then, p+1 column numbers are selected from these k candidate column numbers as the current storage node number.

[0072] For example, assume k = 4 and p = 2. A primary storage node corresponds to a row with four columns containing 1: columns 2, 4, 5, and 6. Three of the four columns, 2, 4, 5, and 6, can be selected as backup storage node numbers. Furthermore, assume that the primary storage node selects 2, 4, and 5 as backup storage node numbers. This means that the primary storage node has selected storage nodes 2, 4, and 5 as backup storage nodes for the data to be written.

[0073] This specification does not limit the method for selecting backup storage node numbers from candidate column numbers. For example, p+1 candidate column numbers may be randomly selected from the candidate column numbers as backup storage node numbers. For another example, p+1 candidate column numbers may be selected at intervals as backup storage node numbers. For another example, p+1 candidate column numbers may be selected sequentially as backup node numbers.

[0074] Step 208: Send the copy of the to-be-written data to the corresponding backup storage node according to the backup storage node number, and after each backup storage node determines that the copy has been received, send a write completion response to the client.

[0075] Based on the aforementioned step 206 , after determining the backup storage node numbers, the primary storage node may send copies of the to-be-written data to the corresponding backup storage nodes based on the backup storage node numbers.

[0076] Still taking 2, 4 and 5 as the backup storage node numbers as an example, the primary storage node can send copies of the to-be-written data to storage node 2, storage node 4 and storage node 5 respectively.

[0077] After receiving the copies of the data to be written, the three backup storage nodes can first cache the copies and return a receipt confirmation to the primary storage node. After receiving the receipt confirmation returned by each backup storage node for the data to be written, the primary storage node can determine that each backup storage node has received the copies and can then send a write completion response to the client.

[0078] For each backup storage node of the data to be written, when it is determined in the background that it has received a copy of the data to be written sent by the storage node (for the backup storage node, it is the main storage node) corresponding to each check row number it maintains, the corresponding check block is calculated and stored based on each received copy to realize the calculation and storage of the check block in the erasure code mechanism.

[0079] The check row number is the row number in the master-slave backup matrix where the value of the element in the column corresponding to the backup storage node is 1. For each backup storage node, there are k check row numbers. That is, each storage node maintains check blocks for the original data stored in the storage node corresponding to the k row numbers in its corresponding column where the value of the element is 1. In other words, k data replicas can be converted into one check block.

[0080] It can be seen from the above description that, by adopting the data processing scheme provided in this specification, the primary storage node of the data to be written can determine the backup storage node for the data to be written based on the master-slave backup matrix, so as to send a copy of the data to be written to the backup storage node, and after each backup storage node determines that it has received the copy, it sends a write completion response to the client, thereby making the performance of the data write operation on the critical path in the distributed storage cluster comparable to that of the copy mechanism, without the need to organize stripes, and achieving high-performance data storage. At the same time, when the backup storage node determines that it has received a copy of the data to be written sent by the primary storage node corresponding to each check row number it maintains, it calculates and stores the corresponding check blocks based on each received copy, and then converts k data copies into a check block, realizing the calculation and storage of the check blocks in the erasure code mechanism, and the storage overhead is comparable to that of the erasure code mechanism, which is much lower than that of the copy mechanism. In summary, by adopting the data processing scheme provided in this specification, a redundancy mechanism with both high performance and low storage space overhead is achieved.

[0081] Figure 3 It is a flowchart of another data processing method provided by an exemplary embodiment.

[0082] In this embodiment, the data processing method can be applied to a standby storage node for data to be written in a distributed storage cluster. The standby storage node can be determined by the primary storage node for the data to be written. For details, please refer to the aforementioned Figure 2 The embodiments shown in this specification will not be described in detail here.

[0083] Please refer to Figure 3 , the data processing method may include the following steps:

[0084] Step 302: Receive a copy of the data to be written sent by the primary storage node of the data to be written, and cache the copy.

[0085] Step 304: Send a reception confirmation to the primary storage node, so that the primary storage node can send a write completion response to the client after receiving the reception confirmation returned by each backup storage node of the data to be written.

[0086] In this embodiment, after receiving a copy of the data to be written, the backup storage node may first cache the copy and return a receipt confirmation to the primary storage node. After receiving the receipt confirmation returned by each backup storage node for the data to be written, the primary storage node may determine that each backup storage node has received the copy and may then send a write completion response to the client.

[0087] Step 306, when it is determined that a copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it has been received, the corresponding check block is calculated and stored based on the received copies, and the check row number is the row number with the element value of 1 in the column corresponding to this storage node in the master-slave backup matrix.

[0088] In this embodiment, the backup storage node can determine in the background that it has received a copy of the data to be written sent by the primary storage node corresponding to each check row number it maintains, and calculate and store the corresponding check blocks based on each received copy to realize the calculation and storage of the check blocks in the erasure code mechanism.

[0089] The check row number is the row number in the master-slave backup matrix where the value of the element in the column corresponding to the backup storage node is 1. For each backup storage node, there are k check row numbers. That is, each storage node maintains check blocks for the original data stored in the storage node corresponding to the k row numbers in its corresponding column where the value of the element is 1. In other words, k data replicas can be converted into one check block.

[0090] For example, assuming k = 4, backup storage node 2 maintains parity blocks for the original data in four storage nodes. Furthermore, assuming that backup storage node 2 maintains parity blocks for the original data in storage nodes 3, 7, 9, and 10, then after receiving copies of the data to be written from storage nodes 3, 7, 9, and 10, backup storage node 2 can calculate the XOR sum of these copies to obtain and store the parity block encoding the data to be written, thereby implementing the calculation and storage of parity blocks in the erasure coding mechanism.

[0091] Figure 4 It is a flowchart of another data processing method provided by an exemplary embodiment.

[0092] In this embodiment, the data processing method can be applied to a client accessing a distributed storage cluster, where a master-slave backup array corresponding to the distributed storage cluster is also deployed. Figure 4 , the data processing method may include the following steps:

[0093] Step 402: Determine the primary storage node for the data to be written.

[0094] In this embodiment, the client can obtain the key value of the data to be written, and then calculate the primary storage node number based on the key value, thereby determining the primary storage node of the data to be written. For example, the client can calculate the hash value of the key value of the data to be written, and then divide the hash value by the number of nodes in the distributed storage cluster to obtain the remainder, and then determine the primary storage node number based on the remainder. Of course, the above method is only an example. In other examples, the primary storage node number of the data to be written can also be determined by rounding down or other methods, and this specification does not impose any special restrictions on this.

[0095] Step 404: Send a data write request to the primary storage node so that the primary storage node can store the data to be written carried in the data write request, and search the column number with the element value of 1 in the row corresponding to this storage node in the master-slave backup matrix as a candidate column number; select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, where p is the fault tolerance threshold of the distributed storage cluster, and p is less than k; and send a copy of the data to be written to the backup storage node according to the backup storage node number.

[0096] Based on the above step 402, after determining the primary storage node of the data to be written, the client can send a data write request to the primary storage node. After receiving the data write request, the primary storage node can store the data to be written carried in the data write request, and can determine the backup storage node of the data to be written based on the master-slave backup matrix, and send a copy of the data to be written to each backup storage node. The processing and implementation of this part can refer to the above Figure 2 The embodiments shown are not described in detail here.

[0097] Among them, after receiving the copy of the data to be written, the backup storage node can cache the copy, and when it is determined that it has received the copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it, it calculates the corresponding check block based on each received copy and stores it. The processing and implementation of this part can refer to the aforementioned Figure 3 The embodiments shown are not described in detail here.

[0098] Step 406: Receive a write completion response returned by the primary storage node. The response is sent by the primary storage node after each backup storage node determines that the copy has been received.

[0099] It can be seen from the above description that, by adopting the data processing scheme provided in this specification, the client can determine the primary storage node of the data to be written, and send the data write request to the primary storage node. The primary storage node can determine the backup storage node for the data to be written based on the master-slave backup matrix, so as to send a copy of the data to be written to the backup storage node, and after each backup storage node determines that it has received the copy, it sends a write completion response to the client, thereby making the performance of the data write operation on the critical path in the distributed storage cluster comparable to that of the copy mechanism, without the need to organize stripes, and achieving high-performance data storage. At the same time, when the backup storage node determines that it has received a copy of the data to be written sent by the primary storage node corresponding to each check row number it maintains, it calculates and stores the corresponding check blocks based on each received copy, and then converts k data copies into a check block, realizing the calculation and storage of the check blocks in the erasure code mechanism, and the storage overhead is comparable to that of the erasure code mechanism, which is much lower than that of the copy mechanism. In summary, by adopting the data processing scheme provided in this specification, a redundancy mechanism with both high performance and low storage space overhead is achieved.

[0100] The following describes the specific implementation of this manual in detail from four aspects: master-slave backup array, data writing, data reading, and data recovery.

[0101] 1. Master-Slave Backup Array

[0102] In this embodiment, a master-slave backup matrix can be deployed in each storage node of the client and the distributed storage cluster. The master-slave backup matrix corresponds to the distributed storage cluster, and its number of rows and columns are the number of storage nodes in the distributed storage cluster. The row number and column number correspond to the node number of the storage node, that is, each storage node can correspond to a row and a column in the master-slave backup matrix, and the corresponding row number and column number are the same. The elements of the master-slave backup matrix have values ​​of 0 or 1, and each row and column has k elements with a value of 1, and for any two rows, there is a column such that the element values ​​of the two rows in this column are all 1.

[0103] For example, the master-slave backup matrix can be the correlation matrix of the Symmetric Balanced Incomplete Block Design (SBIBD). The design index of the symmetrical balanced incomplete block design is 1 and the order is k-1. Wherein, k=q+1, q is a power of a prime number, and the correlation matrix of the symmetrical balanced incomplete block design is k 2 -k+1 order square matrix.

[0104] Taking k=4 as an example, the order of the symmetrically balanced incomplete block design is 3, and the design index is 1. Then the correlation matrix of the symmetrically balanced incomplete block design is a square matrix of order 13. Please refer to Figure 5, a table showing the values ​​of each element in a master-slave backup matrix is ​​used as an example. The master-slave backup matrix is ​​a 13-row, 13-column matrix. The elements in the matrix are either 0 or 1. Each row and column has four elements with a value of 1. For any two rows, there is a column such that the value of the element in that column in both rows is 1. For example, the value of the element in the 13th column in the first and second rows is both 1.

[0105] Still Figure 5 For example, each storage node in the corresponding distributed storage cluster can correspond to a row and a column in the master-slave backup matrix. For example, storage node 1 corresponds to the first row and the first column in the master-slave backup matrix, storage node 2 corresponds to the second row and the second column in the master-slave backup matrix, and so on.

[0106] It is important to note that the order of a square matrix and the order of a symmetrically balanced incomplete block design are different technical concepts. The order of a square matrix refers to the number of rows or columns in the matrix, while the order of a symmetrically balanced incomplete block design refers to the number of treatments in the design.

[0107] In this embodiment, each storage node selects a backup storage node for the data to be written based on the behavior in the master-slave backup matrix, and performs encoding calculations on the copy check blocks of the data to be written based on the columns, which will be described in detail in subsequent embodiments.

[0108] 2. Data Writing

[0109] In this embodiment, when a client writes data to a distributed storage cluster, it can first determine the primary storage node corresponding to the data based on the hash value of the data key. For example, the hash value can be divided by the number of storage nodes in the distributed storage cluster, and the remainder can be determined as the primary storage node number. The number of storage nodes in the distributed storage cluster can be determined based on the master-slave backup array deployed on the client.

[0110] Still Figure 5 Taking the master-slave backup matrix shown as an example, assuming that the remainder of the quotient of the hash value of the key value of the data to be written and the number of storage nodes in the cluster is 1, it can be determined that the master storage node number is 1, that is, the master storage node of the data to be written is storage node 1, and the client can then send a data write request carrying the data to be written to the storage node 1 based on this number.

[0111] In this embodiment, storage node 1 acts as the primary storage node. After receiving a data write request sent by the client, it stores the data to be written carried in the data write request on the one hand, and on the other hand, it can determine the backup storage node for the data to be written based on the deployed master-slave backup array.

[0112] Specifically, storage node 1 may search for a column number with an element value of 1 in the row corresponding to the node in the master-slave backup matrix as a candidate column number, and then select p+1 column numbers from the candidate column numbers as backup storage node numbers for data to be written.

[0113] Still Figure 5 Taking the master-slave backup matrix shown as an example, the master storage node 1 corresponds to the first row of the matrix. The column numbers with element values ​​1 in the first row are 4, 10, 12 and 13, that is, the candidate column numbers are 4, 10, 12 and 13.

[0114] If the fault tolerance threshold p of the distributed storage cluster is 1, primary storage node 1 can select two candidate numbers as backup storage node numbers. For example, primary storage node 1 selects 12 and 13 as backup storage node numbers. That is, the primary storage node selects storage node 12 and storage node 13 as backup storage nodes for the data to be written.

[0115] If the fault tolerance threshold p of the distributed storage cluster is 2, primary storage node 1 can select three candidate nodes as backup storage node numbers. For example, primary storage node 1 selects 4, 12, and 13 as backup storage node numbers. That is, the primary storage node selects storage node 4, storage node 12, and storage node 13 as backup storage nodes for the data to be written.

[0116] If the fault tolerance threshold p of the distributed storage cluster is 3, primary storage node 1 can select four candidate column numbers as backup storage node numbers. That is, the primary storage node selects all candidate column numbers as backup storage node numbers. Then, storage nodes 4, 10, 12, and 13 are all backup storage nodes for the data to be written.

[0117] In this embodiment, after determining the backup storage node, the primary storage node can send a copy of the data to be written to each backup storage node. After receiving the copy of the data to be written from the primary storage node, each backup storage node can cache the copy and return a receipt confirmation to the primary storage node. After receiving the receipt confirmation from all selected backup storage nodes, the primary storage node can determine that all copies have been received and then return a write completion response to the client. At this point, the client knows that the data to be written has been successfully written to the distributed storage cluster, and the performance is comparable to that of a distributed storage cluster using a replication mechanism.

[0118] In this embodiment, for the backup storage node of the data to be written, after receiving the copy sent by the primary storage node corresponding to the check row number it maintains, it can encode each copy into a check block in the background, thereby realizing the calculation and storage of the check block in the erasure code mechanism.

[0119] Taking the backup storage node 12 for the data to be written as an example, the backup storage node 12 corresponds to the 12th column in the master-slave backup matrix. The row number with the element value 1 in the 12th column is the check row number maintained by the backup storage node 12. Please continue to refer to Figure 5 , the parity row numbers of backup storage node 12 are 1, 3, 9, and 13. After receiving the data copies sent by primary storage node 1, primary storage node 3, primary storage node 9, and primary storage node 13, backup storage node 12 can encode the corresponding parity blocks based on these four data copies. After encoding these four data copies into parity blocks, backup storage node 12 can delete these four cached data copies to free up storage space.

[0120] It should be noted that the data copies of primary storage node 1, primary storage node 3, primary storage node 9, and primary storage node 13 may not all be sent simultaneously. If backup storage node 12 has not received all the data copies sent by the primary storage nodes corresponding to the four check rows, it can first cache the received data copies and temporarily postpone encoding the check blocks. For example, if backup storage node 12 only receives the to-be-written data copies sent by primary storage node 1 and primary storage node 3, it can first cache these two data copies and then encode the check blocks after receiving the to-be-written data copies sent by primary storage node 9 and primary storage node 13.

[0121] In addition, the primary storage node and backup storage node in this specification are relative concepts. For data 1 to be written, storage node 1 may be the primary storage node and storage node 12 may be the backup storage node. For data 2 to be written, storage node 12 may be the primary storage node and storage node 8 may be the backup storage node.

[0122] 3. Data Reading

[0123] In this embodiment, when a client reads data from a distributed storage cluster, it first determines the primary storage node corresponding to the data to be read based on the hash value of the data key, and then sends a data read request to the primary storage node. If the primary node is normal, it returns the stored data to the client.

[0124] For example, still taking storage node 1 as the primary storage node, when the client determines that the primary storage node of the data to be read is storage node 1, it can send a data read request to the primary storage node 1. The primary storage node 1 can then search for the corresponding data to be read and return the found data to be read to the client.

[0125] This shows that since the data is not split, there is no need to access multiple storage nodes during data reading, which is more efficient.

[0126] In this embodiment, if the primary storage node of the data to be read fails, the client can read the data to be read from the backup storage node of the data to be read.

[0127] Specifically, the client can determine the backup storage node number of the data to be read from the master-slave backup matrix corresponding to the distributed storage cluster, and then send a data read request to the backup storage node. Still taking the master storage node as storage node 1 as an example, similar to the master storage node selecting the backup storage node, the client can search for the column number with the element value 1 in the first row corresponding to the master storage node 1 in the master-slave backup matrix. These column numbers may be the backup storage node numbers. Still taking the master storage node as storage node 1 as an example, similar to the master storage node selecting the backup storage node, the client can search for the column number with the element value 1 in the first row corresponding to the master storage node 1 in the master-slave backup matrix. Figure 5 For example, the client can find column numbers 4, 10, 11, and 12. However, if the fault tolerance threshold of the distributed storage cluster is not 3, which means the number of backup storage nodes is not 4, but 2 or 3, then the client cannot determine which of storage nodes 4, 10, 11, and 12 are backup storage nodes.

[0128] In one example, the client may send a query request to the storage nodes corresponding to all the column numbers found to query whether the storage nodes store the check blocks corresponding to the data to be read. For example, the client may send query requests to storage nodes 4, 10, 11 and 12 respectively, and may carry the key value of the data to be read in the query request. After receiving the query request, storage nodes 4, 10, 11 and 12 may search whether the corresponding check blocks are stored based on the key value, and return the query results to the client. Assuming that storage nodes 12 and 13 store the corresponding check blocks and return the stored responses to the client, the client may select one from storage nodes 12 and 13 to send a data read request, and the backup storage node that receives the data read request may then recover the data to be read based on the stored check blocks and other data copies, and return them to the client.

[0129] In another example, the client can directly send data read requests to the storage nodes corresponding to all the found column numbers. The storage node that stores the check blocks of the data to be read can then recover the data to be read based on the stored check blocks and other data replicas and return it to the client. Storage nodes that do not store the check blocks of the data to be read can return a response to the client stating that the data is not stored.

[0130] Of course, in other examples, the client may also use other methods to determine which of the found storage nodes store the check blocks of the data to be read, and this specification does not impose any special restrictions on this.

[0131] In this embodiment, the client may determine that the main storage node is faulty after not receiving the to-be-read data returned by the main storage node within a preset response time. The client may also determine that the main storage node is faulty after not receiving the to-be-read data returned by the main storage node for several consecutive times. The processing and implementation of this part can refer to relevant technologies, and this specification will not go into details here.

[0132] 4. Data Recovery

[0133] Please refer to Figure 6 The process of data recovery on the backup storage node may include the following steps:

[0134] Step 602: Receive a data read request sent by the client.

[0135] Step 604: When the current node stores a check block corresponding to the to-be-read data specified by the data read request, the storage nodes corresponding to the check row numbers, excluding the failed node, are determined as primary recovery storage nodes.

[0136] Step 606: Obtain the copy used for calculating the check block from the first-level recovery storage node as the recovery copy.

[0137] Step 608: Recover the data to be read based on the check block and the recovery copy.

[0138] In this embodiment, after receiving a data read request from a client, the backup storage node may determine whether a check block corresponding to the to-be-read data specified in the data read request is stored. For example, the backup storage node may determine whether a corresponding check block is stored based on the key value of the to-be-read data.

[0139] If a check block corresponding to the data to be read is stored, the recovery logic of the data to be read can be executed.

[0140] Specifically, the backup storage node can determine the storage nodes corresponding to the check row numbers maintained by itself, except for the faulty nodes (i.e., the faulty primary storage nodes), as first-level recovery storage nodes. These first-level recovery storage nodes are the primary storage nodes of the data copies used by the backup storage node to calculate the check blocks. The backup storage node can then read the copies used to calculate the check blocks from these first-level recovery storage nodes as recovery copies, and then recover the data to be read based on the check blocks and the read recovery copies.

[0141] Still Figure 5For example, the primary storage node is storage node 1, p is 1, and the primary storage node selects storage node 12 and storage node 13 as backup storage nodes. Assume that the primary storage node 1 fails, and the backup storage node 12 receives the data read request sent by the client, determines that the check block corresponding to the data to be read is stored, and then the storage nodes corresponding to the check row numbers (1, 3, 9 and 13) except the primary storage node 1 can be determined as the first-level recovery storage nodes, that is, storage node 3, storage node 9 and storage node 13 are determined as the first-level recovery storage nodes. Then, the backup storage node 12 can obtain the data copy of the calculated check block from storage node 3, storage node 9 and storage node 13 as the recovery copy, and then recover the data to be read based on the three recovery copies and the check block obtained.

[0142] If p is 2, it is assumed that the main storage node 1 selects storage node 4, storage node 12 and storage node 13 as backup storage nodes. In addition to the failure of the main storage node 1 in the distributed storage cluster, there may be another storage node failure. Assuming that storage node 4 fails, it means that the check block of the data to be read stored in storage node 4 has been lost. At the same time, two storage nodes corresponding to the check row numbers (1, 2, 4 and 10) maintained by storage node 13 have failed (storage node 1 and storage node 4 have failed), and the data to be read cannot be recovered without triggering recursive recovery. However, among the storage nodes corresponding to the check row numbers (1, 3, 9 and 13) maintained by storage node 12, only storage node 1 has failed. The data to be read can be recovered without triggering recursive recovery. The specific recovery process can be referred to the previous description, and this manual will not repeat it here. In this case, the client can determine the backup storage node 12 that does not need to trigger the recursive recovery operation by sending an inquiry message, and then send the data read request to the backup storage node 12.

[0143] If p is 3, primary storage node 1 selects storage node 4, storage node 10, storage node 12, and storage node 13 as backup storage nodes. In addition to the failure of primary storage node 1, two other storage nodes may fail in the distributed storage cluster. If the remaining two failed storage nodes are not both backup storage node 4 and backup storage node 12, the data to be read can be recovered without triggering recursive recovery. If backup storage node 4 and backup storage node 12 fail simultaneously, the remaining backup storage nodes 10 and 13 store the check blocks corresponding to the data to be read.

[0144] For example, backup storage node 10 maintains parity rows numbered 1, 7, 11, and 12. This means the parity blocks it stores are encoded from the data in storage nodes 1, 7, 11, and 12. In other words, storage nodes 7, 11, and 12 are its primary recovery storage nodes. However, storage node 12 also fails, preventing storage node 10 from directly recovering the data to be read.

[0145] For example, backup storage node 13 maintains parity row numbers 1, 2, 4, and 10. This means the parity blocks it stores are encoded from the data in storage nodes 1, 2, 4, and 10. In other words, storage nodes 2, 4, and 10 are its primary recovery storage nodes. However, storage node 4 also fails, preventing backup storage node 13 from directly recovering the data to be read.

[0146] In this case, recursive recovery is required to first obtain the recovery copy, and then restore the data to be read based on the recovery copy and the check block. That is, recursive recovery is performed when there is a faulty node in the first-level recovery storage node.

[0147] Still taking the backup storage node 10 as an example, the check block corresponding to the data to be read stored on it is recorded as x. If the storage node 12 in the primary storage node fails, the backup storage node 10 can first determine the storage node that stores the check block of the restored copy of the failed node as the secondary recovery storage node based on the column with the element value 1 in the row where the failed node (storage node 12) is located in the master-slave backup matrix. Please continue to refer to Figure 5 , the columns with element values ​​of 1 in the row of failed storage node 12 are 2, 8, 10, and 11, that is, the secondary recovery storage nodes are storage node 2, storage node 8, storage node 10, and storage node 11. If none of these four secondary recovery storage nodes are faulty, a recovery instruction for the recovery copy in storage node 12 can be sent to one of the secondary recovery storage nodes, and the secondary recovery storage node can then recover the recovery copy based on the parity block of the recovery copy stored on it. Taking secondary recovery storage node 8 as an example, secondary recovery storage node 8 can recover the recovery copy in storage node 12 based on the data in storage nodes 5, storage node 9, and storage node 10, as well as the parity block (denoted as y) of the recovery copy in storage node 12 stored thereon, and can return the recovered recovery copy to backup storage node 10. After obtaining the recovery copy in storage node 12, backup storage node 10 can recover the data to be read based on the parity block x of the data to be read stored thereon, the recovery copy in storage node 7, the recovery copy in storage node 11, and the recovery copy in storage node 12.

[0148] It can be seen from this that when using the data processing solution provided in this manual to recover data due to a storage node failure, if recursive recovery is required, there is at least one secondary recovery node that can restore the recovery copy lost in the failed primary recovery node, that is, the recursive depth is at most 1, and it will not bring a lot of additional performance consumption to the cluster.

[0149] It is worth noting that in this embodiment, if a copy of the data to be read is cached in the backup storage node, that is, the backup storage node has not yet encoded the check block, the copy of the data to be read can be directly returned to the client.

[0150] Step 610: Return the to-be-read data to the client.

[0151] In this embodiment, the backup storage node may return the recovered data to be read to the client to complete the data reading.

[0152] From the above description, it can be seen that by adopting the above scheme provided in this specification, every k data copies will be encoded and converted into 1 check block. For one piece of data, only (p+1) / k redundant data need to be stored additionally, and the storage space overhead occupied is much lower than the copy mechanism.

[0153] Figure 7 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 7 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710. Of course, it may also include hardware required for other functions. One or more embodiments of this specification can be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0154] Please refer to Figure 8 , the data processing device 800 can be applied to Figure 7In the device shown, the device can be a storage node in a distributed storage cluster to implement the technical solution of this specification. A master-slave backup matrix is ​​deployed in the storage node, and the number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster, and the row and column numbers correspond to the node numbers of the storage nodes. The elements of the master-slave backup matrix have values ​​of 0 or 1, and each row and column has k elements with a value of 1. For any two rows, there exists a column such that the element values ​​in this column of both rows are 1. When the storage node is the master storage node for data to be written, the data processing device 800 may include:

[0155] The write request receiving unit 801 receives a data write request sent by a client and stores the data to be written carried in the data write request;

[0156] A candidate search unit 802 searches the master-slave backup matrix for a column number whose element value is 1 in the row corresponding to the current storage node as a candidate column number;

[0157] The backup node determination unit 803 selects p+1 column numbers from the candidate column numbers as backup storage node numbers for the data to be written, where p is a fault tolerance threshold of the distributed storage cluster and p is less than k.

[0158] The replica sending unit 804 sends a replica of the to-be-written data to the corresponding backup storage node according to the backup storage node number, and after each backup storage node determines that the replica has been received, sends a write completion response to the client;

[0159] In which, the backup storage node is used to cache the copy of the data to be written after receiving the copy of the data to be written, and when it is determined that the copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it has been received, calculate the corresponding check block based on each received copy and store it, and the check row number is the row number with the element value of 1 in the column corresponding to the backup storage node in the master-slave backup matrix.

[0160] Please refer to Figure 9 The data processing device 900 can also be applied to Figure 7In the device shown, the device can be a storage node in a distributed storage cluster to implement the technical solution of this specification. A master-slave backup matrix is ​​deployed in the storage node, and the number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster, and the row and column numbers correspond to the node numbers of the storage nodes. The elements of the master-slave backup matrix have values ​​of 0 or 1, and each row and column has k elements with a value of 1. For any two rows, there exists a column such that the element values ​​in this column of both rows are 1. When the storage node is a backup storage node for data to be written, the data processing device 900 may include:

[0161] The copy cache unit 901 receives a copy of the data to be written sent by the primary storage node of the data to be written, and caches the copy;

[0162] A receiving confirmation unit 902 sends a receiving confirmation to the primary storage node, so that the primary storage node can send a write completion response to the client after receiving the receiving confirmation returned by each backup storage node of the data to be written;

[0163] The parity block calculation unit 903 calculates and stores corresponding parity blocks based on the received copies of the data to be written, upon determining that it has received copies of the data to be written from the primary storage node corresponding to each parity row number maintained by the primary storage node. The parity row number is the row number in the primary-slave backup matrix where the value of the element in the column corresponding to the primary storage node is 1.

[0164] Among them, the primary storage node is used to search the column number with the element value of 1 in the row corresponding to the primary storage node in the master-slave backup matrix as a candidate column number after receiving the data write request sent by the client, select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, and send a copy of the data to be written according to the backup storage node number, where p is the fault tolerance threshold of the distributed storage cluster, and p is less than k.

[0165] Optionally, the data processing device 900 further includes:

[0166] The data recovery unit 904 receives a data read request sent by a client, and the data read request is sent by the client after determining that the primary storage node of the data to be read has failed; when the local node stores a check block corresponding to the data to be read specified by the data read request, the storage node other than the failed node among the storage nodes corresponding to the check row number is determined as a first-level recovery storage node; the copy used to calculate the check block is obtained from the first-level recovery storage node as a recovery copy; the data to be read is recovered based on the check block and the recovery copy; and the data to be read is returned to the client.

[0167] Optionally, when there is a faulty node among the first-level recovery storage nodes, the process of obtaining a recovery copy from the faulty node includes:

[0168] Determine, based on the column with the element value 1 in the row where the faulty node is located in the master-slave backup matrix, a storage node storing the check block of the recovery copy as a secondary recovery storage node;

[0169] Sending a recovery instruction for the recovery copy to the secondary recovery storage node, so that the secondary recovery storage node can recover the recovery copy based on the check block stored therein;

[0170] Receive the recovery copy returned by the secondary recovery storage node.

[0171] Optionally, the check block calculation unit 903 is further configured to delete cached copies of each to-be-written data after storing the check block.

[0172] Please refer to Figure 10 The data processing device 1000 can also be applied to Figure 7 In the device shown, the device can be a client accessing a distributed storage cluster to implement the technical solution of this specification. A master-slave backup matrix is ​​deployed in the storage nodes of the distributed storage cluster. The number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster. The row numbers and column numbers correspond to the node numbers of the storage nodes. The elements of the master-slave backup matrix have values ​​of 0 or 1. Each row and column has k elements with a value of 1. For any two rows, there exists a column such that the element values ​​of both rows in this column are 1. The data processing device 1000 includes:

[0173] A master node determination unit 1001 determines a master storage node for data to be written;

[0174] A write request sending unit 1002 sends a data write request to the primary storage node so that the primary storage node stores the data to be written carried in the data write request, and searches the master-slave backup matrix for a column number with an element value of 1 in the row corresponding to the current storage node as a candidate column number; selects p+1 column numbers from the candidate column numbers as backup storage node numbers for the data to be written, where p is the fault tolerance threshold of the distributed storage cluster and p is less than k; and sends a copy of the data to be written to the backup storage node according to the backup storage node number;

[0175] A write completion response unit 1003 receives a write completion response returned by the primary storage node, the response being sent by the primary storage node after each backup storage node determines that the copy has been received;

[0176] In which, the backup storage node is used to cache the copy of the data to be written after receiving the copy of the data to be written, and when it is determined that the copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it has been received, calculate the corresponding check block based on each received copy and store it, and the check row number is the row number with the element value of 1 in the column corresponding to the backup storage node in the master-slave backup matrix.

[0177] Optionally, the client is also deployed with the master-slave backup matrix, and the master node determination unit 1001 is further used to determine the master storage node of the data to be read;

[0178] The write request sending unit 1002 is also used to determine the backup storage node number of the to-be-read data based on the column number of the element value 1 in the row corresponding to the primary storage node in the master-slave backup matrix when the primary storage node fails; send a data read request to the backup storage node according to the backup storage node number, so that the backup storage node determines the storage node corresponding to the check row number as a first-level recovery storage node when the check block corresponding to the to-be-read data is stored; and obtain a copy used to calculate the check block from the first-level recovery storage node as a recovery copy, so as to recover the to-be-read data based on the check block and the recovery copy; and receive the to-be-read data returned by the backup storage node.

[0179] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0180] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0181] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A data processing method, characterized in that: The invention is applied to a storage node in a distributed storage cluster, wherein a master-slave backup matrix is ​​deployed in the storage node, the number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster, the row number and column number both correspond to the node number of the storage node, the element value of the master-slave backup matrix is ​​0 or 1, each row and each column has k elements with the value of 1, and for any two rows, there is one column such that the element value of the two rows in this column is 1. When the storage node is a master storage node for data to be written, the method includes: Receive a data write request sent by a client, and store the data to be written carried in the data write request; Searching the master-slave backup matrix for a column number whose element value is 1 in the row corresponding to the storage node as a candidate column number; Select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, where p is the fault tolerance threshold of the distributed storage cluster and p is less than k; Sending a copy of the to-be-written data to the corresponding backup storage node according to the backup storage node number, and sending a write completion response to the client after each backup storage node determines that the copy has been received; In which, the backup storage node is used to cache the copy of the data to be written after receiving the copy of the data to be written, and when it is determined that the copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it has been received, calculate the corresponding check block based on each received copy and store it, and the check row number is the row number with the element value of 1 in the column corresponding to the backup storage node in the master-slave backup matrix.

2. A data processing method, characterized in that: The invention is applied to a storage node in a distributed storage cluster, wherein a master-slave backup matrix is ​​deployed in the storage node, the number of rows and columns of the master-slave backup matrix are both the number of storage nodes in the distributed storage cluster, the row number and column number both correspond to the node number of the storage node, the element value of the master-slave backup matrix is ​​0 or 1, each row and each column has k elements with the value of 1, and for any two rows, there is one column such that the element value of the two rows in this column is 1. When the storage node is a backup storage node for data to be written, the method includes: receiving a copy of the data to be written sent by a primary storage node of the data to be written, and caching the copy; Sending a receipt confirmation to the primary storage node, so that the primary storage node can send a write completion response to the client after receiving the receipt confirmation returned by each backup storage node of the data to be written; Upon determining that a copy of the to-be-written data has been received from the primary storage node corresponding to each check row number maintained by the primary storage node, calculating and storing a corresponding check block based on each received copy, wherein the check row number is the row number of the column corresponding to the primary storage node in the primary-slave backup matrix having an element value of 1; Among them, the primary storage node is used to search the column number with the element value of 1 in the row corresponding to the primary storage node in the master-slave backup matrix as a candidate column number after receiving the data write request sent by the client, select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, and send a copy of the data to be written according to the backup storage node number, where p is the fault tolerance threshold of the distributed storage cluster, and p is less than k.

3. The method according to claim 2, characterized in that The method further comprises: receiving a data read request sent by a client, wherein the data read request is sent by the client after determining that a primary storage node of the data to be read is faulty; If the node stores a check block corresponding to the to-be-read data specified by the data read request, determine the storage nodes corresponding to the check row numbers, excluding the failed node, as primary recovery storage nodes; Obtaining a copy used for calculating the check block from the first-level recovery storage node as a recovery copy; restoring the to-be-read data based on the check block and the recovery copy; Return the data to be read to the client.

4. The method according to claim 3, characterized in that In the case where there is a faulty node among the first-level recovery storage nodes, a process of obtaining a recovery copy from the faulty node includes: Determine, based on the column with the element value 1 in the row where the faulty node is located in the master-slave backup matrix, a storage node storing the check block of the recovery copy as a secondary recovery storage node; Sending a recovery instruction for the recovery copy to the secondary recovery storage node, so that the secondary recovery storage node can recover the recovery copy based on the check block stored therein; Receive the recovery copy returned by the secondary recovery storage node.

5. The method according to claim 2, characterized in that The method further comprises: After storing the check block, the cached copies of each to-be-written data are deleted.

6. A data processing method, characterized in that: A method for accessing a distributed storage cluster is provided. The method includes deploying a master-slave backup matrix in the storage nodes of the distributed storage cluster. The number of rows and columns of the master-slave backup matrix corresponds to the number of storage nodes in the distributed storage cluster. The row and column numbers correspond to the node numbers of the storage nodes. The elements of the master-slave backup matrix have values ​​of 0 or 1. Each row and column has k elements with a value of 1. For any two rows, there exists a column such that the values ​​of the elements in the column of the two rows are all 1. The method includes: Determine the primary storage node for the data to be written; Send a data write request to the primary storage node so that the primary storage node stores the data to be written carried in the data write request, and search the master-slave backup matrix for the column number with the element value 1 in the row corresponding to the current storage node as a candidate column number; select p+1 column numbers from the candidate column numbers as the backup storage node numbers of the data to be written, where p is the fault tolerance threshold of the distributed storage cluster and p is less than k; and send a copy of the data to be written to the backup storage node according to the backup storage node number; receiving a write completion response returned by the primary storage node, the response being sent by the primary storage node after each backup storage node determines that the copy has been received; In which, the backup storage node is used to cache the copy of the data to be written after receiving the copy of the data to be written, and when it is determined that the copy of the data to be written sent by the primary storage node corresponding to each check row number maintained by it has been received, calculate the corresponding check block based on each received copy and store it, and the check row number is the row number with the element value of 1 in the column corresponding to the backup storage node in the master-slave backup matrix.

7. The method according to claim 6, characterized in that The client also has the master-slave backup array deployed therein, and the method further includes: Determine the primary storage node for the data to be read; In the event of a failure of the primary storage node, determining the backup storage node number of the to-be-read data based on the column number of the element value 1 in the row corresponding to the primary storage node in the master-slave backup matrix; sending a data read request to the standby storage node according to the standby storage node number, so that the standby storage node determines the storage node corresponding to the check row number as a first-level recovery storage node if the check block corresponding to the data to be read is stored therein; and obtaining from the first-level recovery storage node a copy used for calculating the check block as a recovery copy, so as to recover the data to be read based on the check block and the recovery copy; Receive the to-be-read data returned by the backup storage node.

8. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium, characterized in that Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images