Data recovery method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data recovery method and device, computer equipment, a computer readable storage medium and a computer program product. The method is applied to a distributed system, the distributed system comprises a plurality of node groups, each node group comprises a storage data block and a verification data block of a storage node, and the method comprises the following steps: detecting each node group, and if a fault node exists in a target node group, judging whether the fault node exists in the target node group; obtaining a target verification data block and a target storage data block in a target node group where the fault node is located; the target verification data block is obtained by encoding the total storage data block in the target node group; and performing decoding processing according to the target verification data block and the target storage data block to obtain recovery data of the fault node. By adopting the method, the data recovery efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of distributed data storage, and in particular to a data recovery method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] As the scale of data storage continues to grow and become more widely distributed, the number of storage nodes in distributed file storage systems increases, and single point failures occur frequently. Therefore, data redundancy processing is required for distributed storage data.

[0003] In traditional technologies, distributed file storage systems first shard the stored data and store it in the distributed storage system's nodes. Simultaneously, a global parity block is generated in the global parity node using an erasure code (e.g., Reed-Solomon (RS) code) and the full stored data. In the event of a single node failure, the global parity node reads the normal data blocks of healthy nodes and recovers the data of the failed node based on the normal data blocks and the global parity block, generating the recovered data.

[0004] However, in conventional technologies, a global check node needs to read normal data blocks from all normal nodes. This occupies a large amount of network resources for data transmission of normal data blocks, resulting in poor data recovery efficiency. Summary of the Invention

[0005] Based on this, it is necessary to provide a data recovery method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0006] In a first aspect, the present application provides a data recovery method, which is applied to a distributed system, wherein the distributed system includes multiple node groups, each of which includes a storage data block and a check data block of a storage node, including:

[0007] Detecting each node group, and if a faulty node exists in the target node group, obtaining a target check data block and a target storage data block in the target node group where the faulty node exists; the target check data block is obtained by encoding all the storage data blocks in the target node group;

[0008] Decoding is performed on the target check data block and the target storage data block to obtain recovery data of the faulty node.

[0009] In one embodiment, during the detection of each node group, if a faulty node exists in the target node group, before obtaining the target check data block and the target storage data block in the target node group where the faulty node exists, the method further comprises:

[0010] Divide the target file into storage data blocks under multiple node groups according to a preset grouping strategy;

[0011] For each of the node groups, the stored data blocks under the node group are encoded according to a preset encoding strategy to obtain a verification data block corresponding to each of the node groups.

[0012] In one embodiment, dividing the target file into storage data blocks in a plurality of node groups according to a preset grouping strategy includes:

[0013] Determine the full data blocks and multiple node groups of the target file;

[0014] The full amount of data blocks are divided into the node groups according to a preset grouping strategy to obtain the node groups containing multiple storage data blocks.

[0015] In one embodiment, the detecting each node group and, if a faulty node exists in the target node group, obtaining a target check data block and a target storage data block in the target node group where the faulty node exists includes:

[0016] Detecting the node status of each node, and if there is a faulty node whose node status is faulty, determining the node group identifier corresponding to the faulty node, and determining the target node group where the faulty node is located based on the node group identifier;

[0017] A data recovery node is determined based on a load balancing strategy, and the target verification data block and the target storage data block in the target node group are obtained based on the data recovery node.

[0018] In one embodiment, the decoding process according to the target check data block and the target storage data block to obtain the recovery data of the faulty node includes:

[0019] The target check data block and the target storage data block are decoded according to a preset inverse matrix decoding algorithm to generate a target data block and obtain the recovery data of the fault node.

[0020] In one embodiment, the method further comprises:

[0021] Detect each node group. If the target check data block of the faulty node is unavailable, obtain the full storage data block and the global check data block; the global check data block is obtained by encoding the full storage data blocks of all node groups according to a preset encoding strategy;

[0022] Decoding is performed on the full storage data block and the global check data block to obtain recovery data of each of the faulty nodes.

[0023] In a second aspect, the present application further provides a data recovery device, comprising:

[0024] A first acquisition module is configured to detect each node group and, if a faulty node exists in a target node group, acquire a target check data block and a target storage data block in the target node group where the faulty node exists; the target check data block is obtained by encoding all the storage data blocks in the target node group;

[0025] The first decoding module is configured to perform decoding processing according to the target check data block and the target storage data block to obtain the recovery data of the faulty node.

[0026] In one embodiment, the apparatus further comprises:

[0027] A partitioning module is used to divide the target file into storage data blocks under multiple node groups according to a preset grouping strategy;

[0028] The encoding module is used to encode the storage data blocks under each node group according to a preset encoding strategy to obtain the verification data blocks corresponding to each node group.

[0029] In one embodiment, the partitioning module is specifically used to determine the full amount of data blocks and multiple node groups of the target file;

[0030] The full amount of data blocks are divided into the node groups according to a preset grouping strategy to obtain the node groups containing multiple storage data blocks.

[0031] In one embodiment, the first acquisition module is specifically configured to detect the node status of each node, and if there is a faulty node whose node status is faulty, determine the node group identifier corresponding to the faulty node, and determine the target node group where the faulty node is located based on the node group identifier;

[0032] A data recovery node is determined based on a load balancing strategy, and the target verification data block and the target storage data block in the target node group are obtained based on the data recovery node.

[0033] In one embodiment, the first decoding module is specifically configured to decode the target check data block and the target storage data block according to a preset inverse matrix decoding algorithm to generate a target data block and obtain the recovery data of the faulty node.

[0034] In one embodiment, the apparatus further comprises:

[0035] A second acquisition module is configured to detect each node group and, if the target check data block of the faulty node is unavailable, acquire the full storage data block and the global check data block; the global check data block is obtained by encoding the full storage data blocks of all node groups according to a preset encoding strategy;

[0036] The second decoding module is used to perform decoding processing based on the full storage data block and the global verification data block to obtain the recovery data of each of the fault nodes.

[0037] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0038] Detecting each node group, and if a faulty node exists in the target node group, obtaining a target check data block and a target storage data block in the target node group where the faulty node exists; the target check data block is obtained by encoding all the storage data blocks in the target node group;

[0039] Decoding is performed on the target check data block and the target storage data block to obtain recovery data of the faulty node.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0041] Detecting each node group, and if a faulty node exists in the target node group, obtaining a target check data block and a target storage data block in the target node group where the faulty node exists; the target check data block is obtained by encoding all the storage data blocks in the target node group;

[0042] Decoding is performed on the target check data block and the target storage data block to obtain recovery data of the faulty node.

[0043] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0044] Detecting each node group, and if a faulty node exists in the target node group, obtaining a target check data block and a target storage data block in the target node group where the faulty node exists; the target check data block is obtained by encoding all the storage data blocks in the target node group;

[0045] Decoding is performed on the target check data block and the target storage data block to obtain recovery data of the faulty node.

[0046] The above-mentioned data recovery method, apparatus, computer device, computer-readable storage medium, and computer program product detect each node group. If a faulty node exists within the target node group, the target group verification data and target group storage data within the target group containing the faulty node are obtained. The target group verification data is obtained by encoding the full amount of group storage data within the target group. The group storage data and the target group verification data are decoded to obtain the recovery data of the faulty node. Using this method, the target group verification data and group storage data within the node group are decoded to obtain the recovery data of the faulty node. Data within the faulty node is recovered by only reading the group storage data within the node group containing the faulty node. This avoids reading the full amount of stored data during the data recovery process, reduces network resource overhead and data transmission latency, and improves data recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A diagram of an application environment of a data recovery method in one embodiment;

[0049] Figure 2 1 is a flow chart of a data recovery method according to an embodiment;

[0050] Figure 3 A schematic diagram of storage node division in one embodiment;

[0051] Figure 4 A schematic diagram of a process for dividing nodes into groups according to a preset grouping strategy in one embodiment;

[0052] Figure 5 Schematic diagram of a process for dividing a full data block into node groups in one embodiment;

[0053] Figure 6 A schematic diagram of a process for determining a target node group and obtaining a target verification data block and a target storage data block in one embodiment;

[0054] Figure 7 A schematic diagram of detecting storage node status in one embodiment;

[0055] Figure 8 A schematic diagram of a process for generating recovery data for a failed node in one embodiment;

[0056] Figure 9 1 is a flow chart of data recovery for a faulty node when a target check data block of the faulty node is unavailable in one embodiment;

[0057] Figure 10 is a structural block diagram of a data recovery device in one embodiment;

[0058] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0060] In distributed file storage systems, data storage continues to grow in scale and become widely distributed, making single points of failure (e.g., node failures, network outages) a frequent occurrence. For example, in cloud storage, big data analytics, or edge computing scenarios, data loss can lead to business interruptions or data unavailability. Therefore, distributed file storage systems must ensure high availability while also enabling fast, low-cost failure recovery to meet real-time and cost-control requirements.

[0061] In traditional technology, a three-copy mechanism can be used to copy data in a distributed file storage system to three different nodes, allowing for rapid recovery of unavailable data in a failed node through redundant copies. However, the storage cost of a three-copy mechanism is three times that of the original data, resulting in a waste of storage resources. Based on this, traditional technology can also use traditional erasure coding to shard and store data across multiple nodes. The data blocks of each node are then mathematically encoded to produce a global checksum. Unavailable data blocks can then be recovered using the global checksum and the data blocks of all nodes, reducing storage resource consumption. However, traditional erasure coding requires reading data blocks from all nodes for data recovery, which consumes significant network resources for data block transmission, resulting in poor data recovery efficiency.

[0062] The data recovery method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the metadata management node 102 of the distributed storage system 100 communicates with the file storage node 104 through the network. The metadata management node 102 detects each node group in the file storage node 104. If there is a faulty node in the target node group, the target group verification data and target group storage data in the target group where the faulty node is located are obtained; the target group verification data is obtained by encoding the full amount of group storage data in the target group; the file storage node 104 decodes the group storage data and the target group verification data to obtain the recovery data of the faulty node. The distributed storage system 100 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0063] In an exemplary embodiment, Figure 2 As shown, a data recovery method is provided, which is applied to Figure 1 The distributed storage system in FIG. 1 is taken as an example to illustrate that the distributed system includes multiple node groups, each node group includes a storage data block and a check data block of a storage node, and includes the following steps 202 to 204. Among them:

[0064] Step 202: Detect each node group. If a faulty node exists in the target node group, obtain the target verification data block and the target storage data block in the target node group where the faulty node exists.

[0065] The target verification data block is obtained by encoding all the storage data blocks in the target node group.

[0066] In an embodiment of the present application, a distributed storage system serves as a service layer and includes metadata management nodes and storage nodes. The metadata management nodes are used to store metadata, load balancing, and grouping strategies; the storage nodes are used to store verification data blocks and storage data blocks after data sharding. The storage nodes of the distributed system are divided into multiple node groups, each of which contains multiple storage data blocks and verification data blocks.

[0067] In an exemplary embodiment, Figure 3As shown, the storage structure of the storage node is divided into multiple availability zones (AZs), namely AZ1 and AZ2. D1 to D10 in node group 1 are storage data blocks, and C1 and C2 in node group 1 are check data blocks; D11 to D20 in node group 2 are storage data blocks, and C3 and C4 in node group 2 are check data blocks. In addition, global check blocks are also stored in other availability zones of the distributed storage system. The global check blocks are used to recover data from the storage data blocks in a node group when both the storage data blocks and check data blocks in the node group are unavailable.

[0068] The distributed storage nodes perform real-time detection on the file storage nodes through the metadata management nodes. If the metadata management node detects that one or more storage data blocks in a storage node are unavailable, the storage node is determined to be a faulty node, and the node group where the faulty node is located is determined as the target node group. The check data blocks in the target node group are used as target check data blocks, and all normally available storage data blocks in the target node group are used as target storage data blocks. Then, the distributed storage nodes obtain the target check data blocks and target storage data blocks in the target node group.

[0069] Among them, the number of unavailable storage data blocks at the same time is less than the data recovery threshold of the target verification data block, and the target verification data blocks in the target node group where the faulty node is located are all available. At this time, since the target verification data block is obtained by encoding the target storage data block in the target node group, the distributed storage system can realize data recovery of the unavailable storage data block based on the target verification data block and the target storage data block in the target node group, thereby realizing data recovery of the unavailable storage data block in the faulty node based on the local data in the distributed system.

[0070] Step 204: Decode the target check data block and the target storage data block to obtain recovery data of the faulty node.

[0071] In the embodiment of the present application, the target check block is obtained by encoding all the target storage data blocks in the target node group where the faulty node is located. Therefore, when one or more storage data blocks in the target node group are unavailable, the distributed storage system decodes and processes the unavailable data of the faulty node based on the other normal target storage data blocks and target check data blocks in the target node group to obtain the recovery data of the faulty node. Then, the distributed storage node stores the reply data in the new node and uses the new node to replace the faulty node for business processing. The matrix operation complexity of data recovery based on the target check data block and the target storage data block is O( ), which is much lower than the O( ), and data recovery only requires the transmission of data blocks stored in the node group, reducing bandwidth usage by m / (k+m).

[0072] In the above data recovery method, the target group verification data and group storage data in the node group are decoded to obtain the recovery data of the faulty node. Only the group storage data in the node group where the faulty node is located is read to realize the recovery of the data in the faulty node, thereby avoiding the reading of the full amount of stored data during the data recovery process, reducing the overhead of network resources and the delay of data transmission, and improving the efficiency of data recovery.

[0073] In an exemplary embodiment, Figure 4 As shown, before step 202, the method further includes steps 402 to 404. Among them:

[0074] Step 402: Divide the target file into storage data blocks under a plurality of node groups according to a preset grouping strategy.

[0075] In the embodiment of the present application, first, the distributed storage system divides the target file into data blocks of fixed size. For example, the distributed storage system divides the target file into data blocks of 4MB in size, so that the size of each data block is uniform, and multiple storage data blocks are obtained. Specifically, the distributed storage system divides the divided storage data blocks into multiple node groups, each node group contains k storage data blocks and r check data blocks. For example, when k=10 and r=2, as Figure 3 As shown, a node group has 10 storage data blocks (D1 to D10). Optionally, in addition to node grouping, the distributed storage system also generates m global parity blocks, which cover the storage data blocks of all node groups. These global parity blocks are used to recover from cross-group failures. When storage nodes in different node groups fail simultaneously, these global parity blocks can be used for data recovery. For example, when the number of global parity blocks is m = 4, there are four global parity blocks to ensure data security across all storage nodes. The distributed storage system balances storage overhead and recovery cost by adjusting k (storage data blocks), r (parity blocks), and m (global parity blocks). For example, if the total storage space is N = k + r + m blocks and the storage overhead is N / k, then when k = 10, r = 2, and m = 4, the storage overhead of the distributed storage system is 16 / 10 = 1.6. Recovery of a failed node only requires accessing the storage cost of k + r blocks in the target node group.

[0076] Step 404 : For each node group, encode the stored data blocks under the node group according to a preset encoding strategy to obtain a verification data block corresponding to each node group.

[0077] In an embodiment of the present application, after completing the node grouping and the division of the storage data blocks of the target file, the distributed storage system performs encoding operations on the storage data blocks therein according to a preset encoding strategy for each node group, thereby generating a verification data block corresponding to each node group. The distributed storage system encodes the storage data blocks of each node group by LRC (Locally Repairable Codes) encoding to generate a verification data block corresponding to each node group. The verification data block of each node group is generated based on the storage data blocks within each node group. The verification data block contains redundant information of the storage data blocks within each node group. When a storage data block within the node group fails, the verification data block and other normal storage data blocks can be used for recovery.

[0078] In a specific embodiment, the generation of the check data block is to enable data recovery when a storage data block fails. When one or more storage data blocks in a certain node group are unavailable, since the check data block is obtained by encoding the full amount of storage data blocks in the node group, it can be decoded based on the target check data block and the target storage data block in the target node group to achieve data recovery of the unavailable storage data block. For example, when a storage data block (for example, D5) in node group 1 is unavailable, the check data blocks (C1 and C2) and other normal storage data blocks (for example, D1 to D4, D6 to D10) in the node group can be used for decoding to recover the data of the storage data block D5 in the failed node.

[0079] Optionally, after completing the division of the target file into node groups according to a preset grouping strategy and encoding the storage data blocks under the node groups to obtain the verification data blocks corresponding to each node group, the distributed storage system uses a preset encoding matrix and corresponding decoding algorithm to improve computing performance.

[0080] Specifically, during the encoding process, to achieve effective data encoding and verification, the distributed storage system pre-designs different types of encoding matrices. For each node group, the encoding matrix is ​​in the form of (k+r)×(k). This matrix consists of an identity matrix and a local check matrix, such as a Vandermonde matrix. The identity matrix ensures the integrity of the original data block, while the local check matrix is ​​used to generate local check blocks to achieve data redundancy and verification within the local group. For example, in a node group containing k storage data blocks and r check data blocks, this matrix can accurately encode the storage data blocks and generate corresponding check data blocks.

[0081] The encoding matrix of the global parity block is (m+d)×(d), and the global parity block covers all data blocks. This matrix is ​​used to generate a global parity block, which is used to handle node failures across node groups. The encoding matrix of the global parity block encodes all stored data blocks as a whole. The generated global parity block can recover data when failures occur simultaneously in different node groups. When the number of lost storage blocks dloss within a local group is less than or equal to the preset failure count, data recovery can be completed using only the data within the group and the local parity block. For example, in a local group with k=9 and r=2, if one data block is lost, the remaining k=8 blocks out of the k+r=11 blocks can be used to reconstruct the lost storage block. By enabling data recovery within the node group, the global parity block is not required, reducing the complexity and overhead of data recovery. If multiple groups of failures occur, meaning the number of lost storage blocks dloss exceeds the preset failure count, decoding requires the use of the global parity block. At this time, the distributed storage system needs to access all storage nodes of the node group to obtain the normal storage data blocks and the verification data blocks to achieve recovery of the lost data.

[0082] In this embodiment, the target file is divided into data blocks of fixed size and divided into multiple node groups, and a global check block is generated at the same time, thereby achieving reasonable grouping and distribution of data, avoiding large-scale data loss caused by single point failure, and using LRC encoding to encode the storage data blocks of each node group to generate check data blocks. The check data blocks contain redundant information of the storage data blocks. When a storage data block in a node group fails, it can be decoded and restored based on the check data blocks and normal storage data blocks, effectively improving the reliability and fault tolerance of the data, and ensuring the security and stability of data in the distributed storage system.

[0083] In an exemplary embodiment, Figure 5 As shown, step 402 includes steps 502 to 504. Among them:

[0084] Step 502: Determine the full data blocks and multiple node groups of the target file.

[0085] In an embodiment of the present application, the distributed storage system divides the target file into a preset number of storage data blocks according to the preset configuration parameters in the data sharding in the encoding process. For example, the preset number is d. When d=100, the target file is divided into 100 storage data blocks of appropriate size. The storage data blocks constitute the full data blocks of the target file and serve as the basis for subsequent data processing and storage. Furthermore, the distributed storage system calculates the number of node groups based on the preset node division rules to obtain multiple node groups. For example, the number of groups g=⌈d / k⌉, if the storage data block k=10 and the number of storage data blocks d=100, then the number of groups g=⌈100 / 10⌉=10, that is, 10 node groups are obtained, and each node group will be used to store a certain number of storage data blocks and verification data blocks.

[0086] Step 504 : Divide the full amount of data blocks into node groups according to a preset grouping strategy to obtain node groups containing multiple storage data blocks.

[0087] In an embodiment of the present application, the distributed storage system divides the full amount of storage data blocks into each node group to obtain a node group containing multiple storage data blocks. For example, when the number of storage data blocks d = 100, the number of groups g = 10, and the storage data blocks k = 10, the 100 data blocks will be evenly distributed to 10 node groups, and each node group contains 10 storage data blocks. This division method makes the data volume of each node group relatively balanced. Each node group contains a certain number of storage data blocks, which serve as specific data carriers of the target file and are used to be stored in the distributed storage system with the subsequently generated verification data blocks. At the same time, based on the grouping method of the node grouping, the generation of verification data blocks and global verification blocks provides a clear grouping basis.

[0088] In this embodiment, the full data blocks are divided into each node group according to a preset strategy, achieving balanced data distribution and making the data volume of each node group relatively balanced. The storage data blocks are used as data carriers of the target files and are stored in the node groups of the distributed storage system together with the subsequently generated verification data blocks. Data recovery can be performed through the verification data blocks and storage data blocks in the node groups, which can improve the efficiency of data recovery in the distributed storage system.

[0089] In an exemplary embodiment, Figure 6 As shown, step 202 includes steps 602 to 604. Among them:

[0090] Step 602 , detecting the node status of each node. If there is a faulty node, determining the node group identifier corresponding to the faulty node, and determining the target node group where the faulty node is located based on the node group identifier.

[0091] In the embodiment of the present application, the distributed storage system performs fault detection on the storage nodes through the metadata management cluster, and then detects the node status of each storage node to determine the faulty node, and determines the target node group where the faulty node is located. Figure 7 As shown, the metadata management cluster monitors the status of storage nodes through a node heartbeat mechanism, application-layer detection, or a distributed consistency protocol. When the distributed storage system detects a faulty node, it first determines the node group identifier corresponding to the faulty node. Each storage node is divided into a node group, each with a unique identifier. Using the node group identifiers, the distributed storage system can accurately determine the target node group where the faulty node resides, providing a clear target range for subsequent data recovery operations and enabling targeted data recovery.

[0092] For example, with the node heartbeat mechanism, storage nodes periodically send heartbeat signals to the metadata management cluster. If the metadata management cluster does not receive a heartbeat signal from a storage node within a preset time (for example, 30 seconds), the node status of the storage node will be marked as faulty. This allows real-time monitoring of the node status of storage nodes to promptly identify node offline conditions caused by hardware failures, network outages, and other factors. A storage node failure can be a failure of a specific storage data block within the storage node, or a failure of the entire storage node.

[0093] For application-layer detection, when a read or write operation fails on a storage node, the distributed storage system triggers a check. It then combines the responses of multiple other storage nodes to determine whether the storage node is faulty. For example, if multiple related storage nodes fail to respond when reading a data block, the storage node is considered faulty.

[0094] Distributed storage systems use protocols such as Raft (a consensus algorithm for distributed systems) or Paxos (a distributed consensus protocol) to reach consensus on the status of failed nodes. In a distributed system, multiple storage nodes may have different opinions on the status of a particular storage node. Consensus protocols ensure that all storage nodes agree on the status of a failed node, avoiding data confusion caused by inconsistent states and ultimately identifying the failed node.

[0095] Step 604: determine a data recovery node based on the load balancing strategy, and obtain a target verification data block and a target storage data block in the target node group based on the data recovery node.

[0096] In an embodiment of the present application, after determining the target node grouping, the distributed storage system determines the data recovery node based on the load balancing strategy. The load balancing strategy is to prevent certain storage nodes from suffering performance degradation or even new failures due to taking on too many recovery tasks. The distributed storage system monitors the load of each node in real time, and gives priority to storage nodes with low loads to participate in the recovery task. When data recovery is required, a suitable node is selected from the storage nodes with low loads as the data recovery node, and then the distributed storage system reads the target verification data block and the target storage data block in the target node group to the data recovery node, so as to make full use of the computing resources of the distributed storage system, improve the efficiency of data recovery, and avoid network congestion. Finally, the distributed storage system uses the target verification data block and the target storage data block, and the data recovery node can perform data recovery operations according to the corresponding decoding algorithm, thereby ensuring the integrity and availability of the data in the distributed storage system.

[0097] Optionally, in terms of load balancing, the distributed storage system dynamically adjusts the k, r, and m parameters based on data popularity in storage efficiency optimization. For example, cold data uses k=12 and r=3, and hot data uses small groups (k=4). A two-dimensional LRC design with horizontal grouping and vertical verification is used to reduce recovery costs. Parallel recovery supports parallel processing of multiple independent fault groups to improve overall throughput. At the same time, pipeline encoding is used to execute decoding and writing operations in stages to reduce latency.

[0098] In terms of data recovery, LRC reduces recovery time by 40% to 70% compared to the traditional RS algorithm under the failure type where the target parity data block within the target node group is available. The recovery time (T_RRS) of the traditional RS is composed of the data transmission time (transmitting 6 blocks × block size ÷ bandwidth) and the computation time of solving the 6×6 matrix (the complexity is O( ); the LRC recovery time (T_LRC) is calculated as the data transmission time of 3 blocks × block size ÷ bandwidth, and the computational time is the complexity of solving a 3×3 matrix, which is only 1 / 21.6 of the computational complexity of traditional RS. For example, when T_RRS is 100ms and T_LRC is 25ms, the recovery time is reduced by 75%. Regarding network bandwidth, LRC recovery bandwidth usage is reduced by 50%, which can reduce the impact on cluster read and write performance. Traditional RS recovery requires the transmission of a large number of data blocks (for example, RS (10, 6) requires the transmission of 6 blocks, consuming a total of 6MB of bandwidth), while LRC only requires the transmission of a small number of stored data blocks within the node group (for example, LRC (6, 2, 2) requires only 3 stored data blocks, consuming a total of 3MB of bandwidth).

[0099] In this embodiment, through the node heartbeat mechanism, application layer detection and distributed consistency protocol, the storage node status can be accurately detected and the faulty node can be determined. Then, the target node group can be clearly identified based on the node group identifier, and the target verification data block and target storage data block in the target node group can be read to the data recovery node, avoiding reading the full amount of storage data blocks, which can improve data recovery efficiency.

[0100] In an exemplary embodiment, step 204 includes step 2041. In which:

[0101] Step 2041 : Decode the target check data block and the target storage data block according to a preset inverse matrix decoding algorithm to generate a target data block and obtain the restored data of the faulty node.

[0102] In the embodiment of the present application, the distributed storage system decodes the target check data block and the target storage data block according to the preset inverse matrix decoding algorithm to generate the target data block and obtain the recovery data of the fault node. Figure 8 As shown, using the storage data blocks D1 to D10 and parity data blocks C1 and C2 in node group 1 as an example, if the unavailable storage data block in the failed node is D3, the distributed storage system uses the storage data blocks D1, D2, and D4 to D10 in node group 1 as the target storage data blocks, and the parity data blocks C1 and C2 in node group 1 as the target parity data blocks. It decodes the target storage data blocks and the target parity data blocks using a preset inverse matrix decoding algorithm (e.g., Gauss elimination algorithm) to generate a target data block, which it then uses as the recovery data for the failed node. Specifically, the distributed storage system multiplies the constructed inverse matrix with the matrix that maps the data blocks, leveraging the properties of the inverse matrix to eliminate the parity information introduced during the encoding process, thereby restoring the original data information and generating the target data block.

[0103] After obtaining the recovery data from the failed node, the distributed storage system stores the recovery data in the new node and updates the metadata in the metadata management cluster, for example, by updating the mapping table in the metadata management cluster. Furthermore, to improve the efficiency and performance of data recovery in the distributed storage system, the distributed storage system can adopt two strategies: parallel computing and locality optimization. For parallel computing, the distributed storage system parallelizes the reading and calculation operations of the storage data blocks and verification data blocks within the node group, avoiding the waiting time during sequential execution, increasing data processing speed, and improving overall recovery efficiency. Locality optimization focuses on reducing network latency. When selecting data recovery nodes, storage nodes within the same rack or availability zone (AZ) are prioritized as data recovery nodes, enabling data to be obtained nearby, reducing data loss during network transmission, and further ensuring the efficiency and stability of data recovery.

[0104] In this embodiment, the target group verification data and group storage data in the node group are decoded to obtain the recovery data of the faulty node. Only the group storage data in the node group where the faulty node is located is read to realize the recovery of the data in the faulty node, thereby avoiding the reading of the full amount of stored data during the data recovery process, reducing the overhead of network resources and the delay of data transmission, and improving the efficiency of data recovery.

[0105] In an exemplary embodiment, Figure 9 As shown, the method further includes steps 902 to 904. In which:

[0106] Step 902: Detect each node group. If the target check data block of the faulty node is unavailable, obtain the full storage data block and the global check data block.

[0107] The global check data block is obtained by encoding the full storage data blocks of all node groups according to a preset encoding strategy.

[0108] In an embodiment of the present application, when the distributed storage system detects the presence of a faulty node, it further checks whether the target verification data block corresponding to the faulty node is available. The target verification data block is verification information generated for the data blocks in each node group according to a preset coding strategy during the data storage process, and is used to recover data when the data is lost or damaged. Therefore, when the target verification data block in the target node group where the faulty node is located is unavailable, it means that the storage data blocks of the faulty node can no longer be recovered based on the target storage data blocks in the target node group. Therefore, the distributed storage system reads the full storage data into the node where the global verification data block is located, or reads the global verification data block and the available full storage data block into the data recovery node to achieve data recovery of the faulty node.

[0109] Step 904 : Decoding is performed based on the full storage data block and the global check data block to obtain the recovery data of each faulty node.

[0110] In the embodiment of the present application, since the global check data block is generated using a specific encoding strategy, the distributed storage system selects a corresponding decoding algorithm based on the preset encoding strategy and uses the decoding algorithm that matches the global check data block to perform data recovery and obtain the recovered data of each failed node. The decoding algorithm may include an inverse matrix decoding algorithm based on matrix operations, etc., which can utilize the relationship between the full storage data block and the global check data block to recover the failed storage data block.

[0111] In this embodiment, when the distributed storage system detects a faulty node and its target check data block is unavailable, it means that the faulty node data cannot be recovered based on the target storage data blocks within the target node group. At this point, the distributed storage system will read the full storage data to the node where the global check data block is located, or read the global check data block and the available full storage data blocks to the data recovery node. The global check data block is obtained by encoding the full storage data blocks of all node groups according to a preset encoding strategy. Finally, the distributed storage system selects a corresponding decoding algorithm based on the preset encoding strategy, and uses the relationship between the full storage data blocks and the global check data blocks to perform decoding processing to obtain the recovery data of each faulty node, thereby improving the stability of data recovery.

[0112] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0113] Based on the same inventive concept, embodiments of the present application also provide a data recovery device for implementing the aforementioned data recovery method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data recovery device embodiments provided below can be found in the limitations of the data recovery method above and will not be repeated here.

[0114] In an exemplary embodiment, Figure 10 As shown, a data recovery device 1000 and method are provided. The device is applied to a distributed system, wherein the distributed system includes multiple node groups, each node group includes a storage data block and a check data block of a storage node, including: a first acquisition module 1001 and a first decoding module 1002, wherein:

[0115] The first acquisition module 1001 is configured to detect each node group and, if a faulty node exists within the target node group, acquire the target check data block and target storage data block within the target node group where the faulty node exists. The target check data block is obtained by encoding all the storage data blocks within the target node group.

[0116] The first decoding module 1002 is configured to perform decoding processing based on the target verification data block and the target storage data block to obtain recovery data of the faulty node.

[0117] In one embodiment, the apparatus 1000 further includes:

[0118] A partitioning module is used to divide the target file into storage data blocks under multiple node groups according to a preset grouping strategy;

[0119] The encoding module is used to encode the storage data blocks under each node group according to a preset encoding strategy to obtain the verification data blocks corresponding to each node group.

[0120] In one embodiment, the partitioning module is specifically used to determine the full amount of data blocks and multiple node groups of the target file;

[0121] The full amount of data blocks are divided into node groups according to the preset grouping strategy to obtain node groups containing multiple storage data blocks.

[0122] In one embodiment, the first acquisition module 1001 is specifically configured to detect the node status of each node, and if there is a faulty node whose node status is faulty, determine the node group identifier corresponding to the faulty node, and determine the target node group to which the faulty node belongs based on the node group identifier;

[0123] A data recovery node is determined based on a load balancing strategy, and a target verification data block and a target storage data block in a target node group are obtained based on the data recovery node.

[0124] In one embodiment, the first decoding module 1002 is specifically configured to decode the target check data block and the target storage data block according to a preset inverse matrix decoding algorithm to generate a target data block and obtain the recovery data of the faulty node.

[0125] In one embodiment, the apparatus 1000 further includes:

[0126] The second acquisition module is used to detect each node group. If the target check data block of the faulty node is unavailable, the module obtains the full storage data block and the global check data block. The global check data block is obtained by encoding the full storage data blocks of all node groups according to a preset encoding strategy.

[0127] The second decoding module is used to perform decoding processing based on the full storage data block and the global verification data block to obtain the recovery data of each fault node.

[0128] Each module in the above-mentioned data recovery device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0129] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 11 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a data recovery method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0130] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0131] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0132] Detect each node group. If there is a faulty node in the target node group, obtain the target verification data block and target storage data block in the target node group where the faulty node is located. The target verification data block is obtained by encoding all the storage data blocks in the target node group.

[0133] Decoding is performed on the target check data block and the target storage data block to obtain the recovery data of the faulty node.

[0134] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0135] Divide the target file into storage data blocks under multiple node groups according to a preset grouping strategy;

[0136] For each node group, the stored data blocks under the node group are encoded according to a preset encoding strategy to obtain the verification data blocks corresponding to each node group.

[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0138] Determine the full data blocks and multiple node groups of the target file;

[0139] The full amount of data blocks are divided into node groups according to the preset grouping strategy to obtain node groups containing multiple storage data blocks.

[0140] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0141] Detect the node status of each node. If there is a faulty node, determine the node group identifier corresponding to the faulty node, and determine the target node group where the faulty node is located based on the node group identifier.

[0142] A data recovery node is determined based on a load balancing strategy, and a target verification data block and a target storage data block in a target node group are obtained based on the data recovery node.

[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0144] The target check data block and the target storage data block are decoded according to a preset inverse matrix decoding algorithm to generate a target data block and obtain the recovery data of the fault node.

[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0146] Check each node group. If the target check data block of the faulty node is unavailable, obtain the full storage data block and the global check data block; the global check data block is obtained by encoding the full storage data blocks of all node groups according to the preset encoding strategy;

[0147] Decoding is performed based on the full storage data block and the global check data block to obtain the recovery data of each fault node.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0149] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0151] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0152] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data recovery method, characterized in that: The method is applied to a distributed system, wherein the distributed system includes a plurality of node groups, each of the node groups includes a storage data block and a check data block of a storage node, and the method includes: Detecting each node group, and if a faulty node exists in the target node group, obtaining a target check data block and a target storage data block in the target node group where the faulty node exists; the target check data block is obtained by encoding all the storage data blocks in the target node group; Decoding is performed on the target check data block and the target storage data block to obtain recovery data of the faulty node.

2. The method according to claim 1, characterized in that Before detecting each node group and obtaining a target verification data block and a target storage data block in the target node group where the faulty node exists if the target node group contains the faulty node, the method further comprises: Divide the target file into storage data blocks under multiple node groups according to a preset grouping strategy; For each of the node groups, the stored data blocks under the node group are encoded according to a preset encoding strategy to obtain a verification data block corresponding to each of the node groups.

3. The method according to claim 2, characterized in that The step of dividing the target file into storage data blocks under a plurality of node groups according to a preset grouping strategy includes: Determine the full data blocks and multiple node groups of the target file; The full amount of data blocks are divided into the node groups according to a preset grouping strategy to obtain the node groups containing multiple storage data blocks.

4. The method according to claim 1, wherein The detecting each node group, and if a faulty node exists in the target node group, obtaining a target verification data block and a target storage data block in the target node group where the faulty node exists, includes: Detecting the node status of each node, and if there is a faulty node whose node status is faulty, determining the node group identifier corresponding to the faulty node, and determining the target node group where the faulty node is located based on the node group identifier; A data recovery node is determined based on a load balancing strategy, and the target verification data block and the target storage data block in the target node group are obtained based on the data recovery node.

5. The method according to claim 1, wherein The decoding process is performed according to the target check data block and the target storage data block to obtain the recovery data of the fault node, including: The target check data block and the target storage data block are decoded according to a preset inverse matrix decoding algorithm to generate a target data block and obtain the recovery data of the fault node.

6. The method according to claim 1, characterized in that The method further comprises: Detect each node group. If the target check data block of the faulty node is unavailable, obtain the full storage data block and the global check data block; the global check data block is obtained by encoding the full storage data blocks of all node groups according to a preset encoding strategy; Decoding is performed on the full storage data block and the global check data block to obtain recovery data of each of the faulty nodes.

7. A data recovery device, characterized in that: The device is applied to a distributed system, wherein the distributed system includes a plurality of node groups, each of the node groups includes a storage data block and a check data block of a storage node, and the device includes: A first acquisition module is configured to detect each node group and, if a faulty node exists in a target node group, acquire a target check data block and a target storage data block in the target node group where the faulty node exists; the target check data block is obtained by encoding all the storage data blocks in the target node group; The first decoding module is configured to perform decoding processing based on the target verification data block and the target storage data block to obtain the recovery data of the faulty node.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images