Data recovery method and device, storage medium and electronic equipment

The hash table is used to find the target node corresponding to the missing target encoding block in the failed node, and allocate the encoding block according to the bandwidth resource weight of the target node, which solves the problem of low data recovery efficiency in the cloud database, and realizes efficient resource utilization and network utilization Reduced load.

CN118964088APending Publication Date: 2024-11-15CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410990253.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In cloud databases, uneven allocation of node bandwidth resources leads to low data recovery efficiency and insufficient utilization of high bandwidth resources, resulting in excessive network load.

Method used

By obtaining the hash table between the node and the coded block, the target coded block missing from the faulty node is determined, and the encoding block request is sent to the target node according to the bandwidth resource weight allocation of the target node, and the target data block is reconstructed to repair the faulty node's data.

Benefits of technology

It improves the data repair efficiency in the unbalanced resource distribution scenario, reduces the network load during system data repair, and makes full use of high bandwidth resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964088A_ABST
    Figure CN118964088A_ABST
Patent Text Reader

Abstract

The invention discloses a data recovery method and device, a storage medium and electronic equipment, and relates to the technical field of data processing.The data recovery method comprises the steps that firstly, a hash table recording the corresponding relation between nodes and coding blocks is obtained, and the coding blocks are obtained by converting data blocks in a distributed storage system; determining a missing target coding block in the fault node, and obtaining a target node corresponding to the target coding block based on the hash table; then a coding block request is sent to a target node, and coding blocks stored in the target node are allocated according to the bandwidth resource weight of the target node; reconstructing a target data block according to the coding block returned by the target node and the coding matrix corresponding to the returned coding block; and repairing the data of the fault node by using the target data block. Compared with the prior art, the method has the advantages that the data recovery efficiency under the scene of unbalanced resource distribution is improved, and meanwhile, the network load during the system data recovery period is also reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data repair method, device, storage medium and electronic device. Background Art

[0002] Based on the highly decentralized nature of cloud databases, each node in the system needs to adopt a completely independent and peer-to-peer distributed storage model. Specifically, each node in the system needs to store a complete copy of the data to achieve the purpose of data consistency verification.

[0003] Currently, when a node in a cloud database crashes or loses data, it is necessary to send a synchronization request to other nodes in a broadcast manner to restore the data. The node that receives the recovery request will send data to the node to complete data synchronization.

[0004] However, a large number of redundant request messages and data information will cause a surge in data transmission in the cloud database network. In the scenario of unbalanced bandwidth resource distribution, data recovery tasks cannot be allocated according to node bandwidth resources, resulting in bandwidth-constrained nodes becoming the bottleneck of data recovery, and the bandwidth resources of high-bandwidth resource nodes cannot be effectively utilized, which seriously affects the data repair efficiency of the cloud database. Summary of the invention

[0005] In view of this, the present application provides a data repair method, device, storage medium and electronic device, the main purpose of which is to improve the current technical problem that data recovery tasks cannot be allocated according to node bandwidth resources, resulting in low data repair efficiency in cloud databases.

[0006] In a first aspect, the present application provides a data repair method, comprising:

[0007] Obtaining a hash table recording the correspondence between nodes and coding blocks, where the coding blocks are converted from data blocks in a distributed storage system;

[0008] Determine a target coding block missing in the faulty node, and obtain a target node corresponding to the target coding block based on the hash table;

[0009] Sending a coding block request to the target node, the coding blocks stored in the target node being allocated according to the bandwidth resource weight of the target node;

[0010] Reconstructing a target data block according to the coding block returned by the target node and a coding matrix corresponding to the returned coding block;

[0011] The target data block is used to repair the data of the failed node.

[0012] In a second aspect, the present application provides a data repair device, comprising:

[0013] An acquisition module is configured to acquire a hash table recording a correspondence between nodes and coding blocks, where the coding blocks are converted from data blocks in a distributed storage system;

[0014] A determination module is configured to determine a target coding block missing from a faulty node, and obtain a target node corresponding to the target coding block based on the hash table;

[0015] A request module, configured to send a coding block request to the target node, wherein the coding blocks stored in the target node are allocated according to the bandwidth resource weight of the target node;

[0016] A reconstruction module, configured to reconstruct a target data block according to the coding block returned by the target node and a coding matrix corresponding to the returned coding block;

[0017] The repair module is configured to repair the data of the failed node by using the target data block.

[0018] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0019] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0020] In a fifth aspect, the present application provides a computer program product having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the workload evaluation method described in the first aspect.

[0021] By means of the above technical scheme, the present application provides a data repair method, device, storage medium and electronic device, firstly obtains a hash table recording the correspondence between nodes and coding blocks, wherein the coding blocks are converted from data blocks in a distributed storage system; then determines the target coding blocks missing in the faulty node, and obtains the target node corresponding to the target coding blocks based on the hash table; then sends a coding block request to the target node, wherein the coding blocks stored in the target node are allocated according to the bandwidth resource weight of the target node; reconstructs the target data block according to the coding blocks returned by the target node and the coding matrix corresponding to the returned coding blocks; and uses the target data block to repair the data of the faulty node. Compared with the current prior art, the present application finds the target node corresponding to the target coding block missing in the faulty node through a hash table, obtains the coding block from the target node to reconstruct the target data block, and then uses the target data block to repair the data of the faulty node, thereby improving the data repair efficiency in the scenario of unbalanced resource distribution, and also reduces the network load during system data repair.

[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 A schematic diagram of a data repair method provided in an embodiment of the present application is shown;

[0026] Figure 2 A schematic diagram of a data repair method provided in an embodiment of the present application is shown;

[0027] Figure 3 A schematic diagram showing an example provided by an embodiment of the present application;

[0028] Figure 4 A schematic diagram showing an example provided by an embodiment of the present application;

[0029] Figure 5A schematic structural diagram of a data repair device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0031] In order to improve the current technical problem that the data recovery task cannot be allocated according to the node bandwidth resources, resulting in low data recovery efficiency of the cloud database. This embodiment provides a data recovery method, such as Figure 1 As shown, the method includes:

[0032] Step 101: Obtain a hash table that records the correspondence between nodes and coding blocks.

[0033] Among them, the coding block is obtained by converting the data block in the distributed storage system. As the number of data entries in the cloud database increases day by day, the data volume continues to expand. The ever-expanding block volume leads to a huge amount of network data required to be transmitted during the data recovery process. If the data block is used as the basic unit of data recovery, the basic unit volume required for data recovery is too large. In the scenario of unbalanced resource distribution, it will cause serious communication pressure on nodes with limited bandwidth resources, resulting in low data repair efficiency. Converting the basic unit stored in the cloud database distributed storage system from data blocks to coding blocks can reduce the amount of data transmission.

[0034] In some examples, a hash table is a data structure that maintains a mapping table or distributed database that records the correspondence between nodes and encoding blocks.

[0035] Step 102: Determine the target coding block missing in the faulty node, and obtain the target node corresponding to the target coding block based on the hash table.

[0036] Exemplarily, based on a weighted consistent hashing algorithm, coding blocks can be allocated to nodes according to the bandwidth resource weights of the nodes and a hash table can be established. When a node failure occurs, the target node corresponding to the target coding block missing in the failed node can be found in the hash table according to the hash algorithm.

[0037] Step 103: Send a coding block request to the target node.

[0038] The coding blocks stored in the target node are allocated according to the bandwidth resource weight of the target node, which can avoid the problem of a surge in system network load caused by sending data requests in a broadcast manner during traditional data recovery.

[0039] Step 104: reconstruct the target data block according to the coding block returned by the target node and the coding matrix corresponding to the returned coding block.

[0040] In some examples, a coding matrix corresponding to the coding block can be generated. There are multiple optional schemes for generating the coding matrix, which are not specifically limited in this embodiment. Then, the original data block data, i.e., the target data block, can be restored by multiplying the coding block set returned by the target node with the inverse matrix of the coding matrix corresponding to the coding block set.

[0041] Step 105: Use the target data block to repair the data of the failed node.

[0042] Exemplarily, the target data block contains all the data missing from the failed node, and the cloud database can be repaired by restoring the data of the failed node based on the data in the target data block.

[0043] Compared with the current existing technology, by applying the technical solution of this embodiment, a hash table recording the correspondence between nodes and coding blocks is first obtained, wherein the coding blocks are converted from data blocks in a distributed storage system; then the target coding blocks missing in the faulty node are determined, and the target node corresponding to the target coding blocks is obtained based on the hash table; then a coding block request is sent to the target node, wherein the coding blocks stored in the target node are weighted according to the bandwidth resources of the target node; the target data block is reconstructed according to the coding blocks returned by the target node and the coding matrix corresponding to the returned coding blocks; and the data of the faulty node is repaired using the target data block. The target node corresponding to the target coding block missing in the faulty node is found through the hash table, and the coding blocks are obtained from the target node to reconstruct the target data blocks, and then the data of the faulty node is repaired using the target data blocks, which improves the data repair efficiency in the scenario of unbalanced resource distribution, and also reduces the network load during system data repair.

[0044] In order to further illustrate the specific implementation process of the method of this embodiment, this embodiment provides the following Figure 2 The specific method shown includes:

[0045] Step 201: Obtain a hash table that records the correspondence between nodes and coding blocks.

[0046] The coding block is converted from the data block in the distributed storage system. The data repair process has a quadratic time complexity, that is, all nodes in the system send messages to other nodes except themselves, resulting in a quadratic increase in the number of messages as the number of nodes increases. Through the data block encoding and reconstruction mechanism, the amount of data that resource-constrained nodes need to transmit during the data recovery process can be reduced, thereby improving the operating efficiency of the cloud database.

[0047] Optionally, the method of this embodiment may include a process for establishing a hash table: assigning one or more virtual nodes to each node according to the bandwidth resource weight of each node; determining the virtual node corresponding to each coding block based on the hash value of each coding block and the hash value corresponding to each node; storing each coding block in the node corresponding to the virtual node, and updating the correspondence between the node and the coding block in the hash table.

[0048] For example, in a scenario where resources are unevenly distributed, if data recovery tasks cannot be allocated based on node bandwidth resources, nodes with limited bandwidth resources will become bottlenecks for data recovery. At the same time, the bandwidth resources of high-bandwidth nodes cannot be fully utilized, wasting bandwidth resources and seriously reducing the efficiency of cloud database data recovery. Therefore, in order to balance the system load, virtual nodes can be allocated to each node based on the bandwidth resource weight of each node to ensure efficient and fair resource utilization.

[0049] Exemplarily, the number of virtual nodes that each node should be allocated can be calculated based on the product of the ratio of its bandwidth to the total bandwidth and the total number of virtual nodes. If the bandwidth resources of node a are higher than those of node b, the virtual nodes allocated to node a will be more than the virtual nodes allocated to node b to ensure that nodes with high bandwidth resources will be allocated more coding blocks in the future. For example, if the bandwidth resource weight of node a is twice the bandwidth resource weight of node b, the virtual nodes allocated to node a are twice the virtual nodes allocated to node b. Then, after the node's coding block reallocation operation is completed, the correspondence between nodes and coding blocks in the hash table of each node is updated.

[0050] Optionally, the above-mentioned determining the virtual node corresponding to each coding block based on the hash value of each coding block and the hash value corresponding to each node includes: calculating the hash value of each virtual node, and obtaining the remainder corresponding to each virtual node based on the hash value of each virtual node; and, calculating the hash value of any coding block, and obtaining the remainder corresponding to any coding block based on the hash value of any coding block; according to the remainder corresponding to each virtual node, determining the first virtual node that is larger than the remainder corresponding to any coding block in a clockwise direction on the hash ring as the virtual node corresponding to any coding block.

[0051] For example, Figure 3 As shown in Figure 2, according to the bandwidth resource weight of each node, a corresponding number of virtual nodes are allocated to it. Calculate the hash values ​​of all nodes and compare the obtained hash values ​​to 2 32 Take the remainder of each node, and then combine each data block with each encoding block corresponding to the data block to get the hash value, and then add the hash value to 2 32 The remainder of each coding block is obtained by taking the modulus; after obtaining the coding node and the remainder of the coding block, the coding block is stored in the first node that is larger than the remainder in a clockwise direction.

[0052] Optionally, the method of this embodiment may also include a process of converting a data block into a coding block: dividing the data block into a first preset number of data slices; using erasure code (EC) technology, converting the data slices into a second preset number of coding blocks, the second preset number being greater than the first preset number.

[0053] In some examples, erasure coding is an error correction and data recovery technology designed primarily for distributed storage systems and network communications to improve data reliability and fault tolerance. Erasure coding technology can be used to convert the basic unit of cloud database storage from data blocks to coding blocks, which can reduce the amount of data transmission.

[0054] Exemplarily, the data block is first divided into m data slices, and the erasure code technology is used to convert the m data blocks of the original data into n (n>m) coding blocks of the same size, so that any m of the n coding blocks can reconstruct the original data block. In this way, even if any part of the data blocks (no more than nm blocks) are lost or unavailable, the lost data can be reconstructed according to the remaining data blocks and the verification blocks (Verification Block, VCB) generated based on the data blocks so that the original data is still available.

[0055] For example, Figure 4 As shown, the original data block B can be divided into m slices of equal size, and the missing part of the last slice is filled with 0. Before the encoding begins, the generator matrix G needs to be selected. If the generator matrix G is a Cauchy matrix, the formula is expressed as:

[0056]

[0057] Then, encoding is performed based on the generator matrix G to generate additional r check slices from the original data slices. The m original slices and r check slices form a coding block set E = {e1, e2, ..., em + r}. Among them, e1 to em are original data slices, and em + 1 to em + r are check slices calculated according to the generator matrix G. These slices can be stored or transmitted separately. When a part of the slices is lost, the lost data can be restored through the remaining slices and the properties of the generator matrix.

[0058] Step 202: Determine the target coding block missing in the faulty node, and obtain the target node corresponding to the target coding block based on the hash table.

[0059] For example, for each identified missing coding block, the hash value of each missing coding block can be used to query the hash table to determine another node or nodes storing the coding block. The method of this embodiment can not only locate the missing coding block in the faulty node, but also quickly find the target node through the hash table, thereby ensuring the continuity of the repair process and the integrity of the data.

[0060] Step 203: Send a coding block request to the target node.

[0061] The coding blocks stored in the target node are allocated according to the bandwidth resource weight of the target node.

[0062] For example, when a node crashes or data is lost, the node does not need to broadcast a coding block synchronization request, but only needs to request data repair data from the target node based on the correspondence between the node and the stored coding block stored in the hash table.

[0063] Optionally, after sending the coding block request to the target node, the method further includes: if no valid coding block is received from the target node before a preset time, re-broadcasting the coding block request to the target node and / or broadcasting the coding block request to other nodes.

[0064] For example, when sending a coding block request, a reasonable preset time, i.e., a timeout, needs to be set for the request. This time can be determined based on a comprehensive consideration of network conditions, expected processing time, and a certain buffer time. If no valid response is received before the preset timeout expires, you can try to resend the request to the same target node, or broadcast the coding block request to other nodes until a valid coding block is received.

[0065] Step 204: perform deduplication processing on the returned coding blocks, and perform an inverse operation on the coding matrix to obtain an inverse matrix of the coding matrix.

[0066] In some examples, deduplication of received coding blocks also needs to meet the requirement that the number of different coding blocks is greater than n, otherwise it is necessary to continue to send requests to the target node and / or other nodes.

[0067] Step 205: perform a product operation on the deduplicated coded block and the inverse matrix to obtain a target data block.

[0068] Exemplarily, the faulty node can restore the original data block data by multiplying the received coding block set with the inverse matrix of the coding matrix corresponding to the coding block set. The coding block set includes at least n different coding blocks and / or check blocks.

[0069] Step 206: Use the target data block to repair the data of the failed node.

[0070] Exemplarily, depending on whether the local data of the node is completely lost, the repair process can be divided into two categories: full repair and incremental repair.

[0071] In some examples, a hash value of the coding block can be calculated based on the missing coding block, and the obtained hash value can be used to calculate the hash value of the coding block. 32 The remainder is obtained by taking the remainder. According to the remainder obtained, the first node is found clockwise along the hash ring, and a missing code block request is sent to the node, and a timeout period is started. If a valid code block is not received from the node before the timeout period, the code block request is broadcast; if a code block is received from the node before the timeout period, the data block is reconstructed to complete the incremental repair of the faulty node.

[0072] In some examples, a request for a specified coding block can also be sent to nodes in the system based on a hash table that records the correspondence between nodes and coding blocks, and a timeout period is started. Then a waiting queue is constructed to receive coding blocks and delete the duplicated coding blocks. If a node does not reply with a valid coding block before the timeout period expires, a coding block request is resent to the node. When the length of the waiting queue is greater than n, the waiting queue is closed, and the data block reconstruction operation is performed to complete the full repair of the faulty node.

[0073] Compared with the current existing technology, by applying the technical solution of this embodiment, a more scientific and efficient data repair method is provided. The target node corresponding to the target coding block missing in the faulty node is found through a hash table, and the coding block is obtained from the target node to reconstruct the target data block, and then the target data block is used to repair the data of the faulty node, thereby improving the data repair efficiency in the scenario of unbalanced resource distribution, and also reducing the network load during the system data repair period.

[0074] Further, as Figure 1 and Figure 2 The specific implementation of the method shown in the embodiment provides a data repair device, such as Figure 5 As shown, the device includes: an acquisition module 31, a determination module 32, a request module 33, a reconstruction module 34, and a repair module 35.

[0075] An acquisition module 31 is configured to acquire a hash table recording a correspondence between nodes and coding blocks, where the coding blocks are converted from data blocks in a distributed storage system;

[0076] A determination module 32 is configured to determine a target coding block missing from a faulty node, and obtain a target node corresponding to the target coding block based on the hash table;

[0077] The request module 33 is configured to send a coding block request to the target node, wherein the coding blocks stored in the target node are allocated according to the bandwidth resource weight of the target node;

[0078] A reconstruction module 34 is configured to reconstruct a target data block according to the coding block returned by the target node and a coding matrix corresponding to the returned coding block;

[0079] The repair module 35 is configured to repair the data of the failed node using the target data block.

[0080] In some examples of this embodiment, the reconstruction module 34 is specifically configured to deduplicate the returned coding block and perform an inverse operation on the coding matrix to obtain an inverse matrix of the coding matrix; and multiply the deduplicated coding block and the inverse matrix to obtain the target data block.

[0081] In some examples of this embodiment, the acquisition module 31 is specifically configured to include the process of establishing the hash table: allocating one or more virtual nodes to each node according to the bandwidth resource weight of each node; determining the virtual node corresponding to each coding block based on the hash value of each coding block and the hash value corresponding to each node; storing each coding block in the node corresponding to the virtual node, and updating the correspondence between the node and the coding block in the hash table.

[0082] In some examples of this embodiment, the acquisition module 31 is further configured to calculate the hash value of each virtual node, and obtain the remainder corresponding to each virtual node based on the hash value of each virtual node; and calculate the hash value of any coding block, and obtain the remainder corresponding to any coding block based on the hash value of any coding block; according to the remainder corresponding to each virtual node, determine the first virtual node that is larger than the remainder corresponding to any coding block in a clockwise direction on the hash ring as the virtual node corresponding to any coding block.

[0083] In some examples of this embodiment, the acquisition module 31 is specifically configured to convert the data block into the coding block, including: dividing the data block into a first preset number of data slices; using erasure code technology, converting the data slices into a second preset number of coding blocks, the second preset number being greater than the first preset number.

[0084] In some examples of this embodiment, the request module 33 is specifically configured to re-broadcast the coding block request to the target node and / or broadcast the coding block request to other nodes if no valid coding block is received from the target node before the preset time arrives.

[0085] It should be noted that the data repair device provided in this embodiment and other corresponding descriptions of the functional units involved can be referred to in Figure 1 and Figure 2 The corresponding description in will not be repeated here.

[0086] Based on the above Figure 1 and Figure 2 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 The method shown.

[0087] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0088] Based on the above Figure 1 and Figure 2 The method shown, and Figure 5 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, such as a personal computer, a server, a laptop computer, an intelligent robot or other intelligent terminal, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The method shown.

[0089] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0090] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0091] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the solution of this embodiment, compared with the current prior art, this embodiment uses a hash table to find the target node corresponding to the target coding block missing in the faulty node, and obtains the coding block from the target node to reconstruct the target data block, and then uses the target data block to repair the data of the faulty node, thereby improving the data repair efficiency in the scenario of unbalanced resource distribution, and also reducing the network load during system data repair.

[0093] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0094] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A data repair method, characterized in that: include: Obtaining a hash table recording the correspondence between nodes and coding blocks, where the coding blocks are converted from data blocks in a distributed storage system; Determine a target coding block missing in the faulty node, and obtain a target node corresponding to the target coding block based on the hash table; Sending a coding block request to the target node, the coding blocks stored in the target node being allocated according to the bandwidth resource weight of the target node; Reconstructing a target data block according to the coding block returned by the target node and a coding matrix corresponding to the returned coding block; The target data block is used to repair the data of the failed node.

2. The method according to claim 1, characterized in that The reconstructing the target data block according to the coding block returned by the target node and the coding matrix corresponding to the returned coding block includes: Performing deduplication processing on the returned coding block, and performing an inverse operation on the coding matrix to obtain an inverse matrix of the coding matrix; The deduplicated coded block is multiplied by the inverse matrix to obtain the target data block.

3. The method according to claim 1, characterized in that: The process of establishing the hash table includes: According to the bandwidth resource weight of each node, one or more virtual nodes are allocated to each node; Determine a virtual node corresponding to each coding block based on the hash value of each coding block and the hash value corresponding to each node; Each coding block is stored in a node corresponding to the virtual node, and the corresponding relationship between the node and the coding block is updated in the hash table.

4. The method according to claim 3, characterized in that The determining, based on the hash value of each coding block and the hash value corresponding to each node, the virtual node corresponding to each coding block includes: Calculate the hash value of each virtual node, and calculate the remainder corresponding to each virtual node according to the hash value of each virtual node; and, Calculate the hash value of any coding block, and calculate the remainder corresponding to any coding block according to the hash value of any coding block; According to the remainder corresponding to each virtual node, the first virtual node whose remainder is larger than the remainder corresponding to any coding block is determined in a clockwise direction on the hash ring as the virtual node corresponding to any coding block.

5. The method according to claim 1, characterized in that The process of converting the data block into the coding block includes: Dividing the data block into a first preset number of data slices; The data slices are converted into a second preset number of coding blocks by using erasure coding technology, where the second preset number is greater than the first preset number.

6. The method according to claim 1, characterized in that After sending the encoding block request to the target node, comprising: If no valid coding block is received from the target node before the preset time arrives, the coding block request is broadcast to the target node again and / or the coding block request is broadcast to other nodes.

7. A data repair device, characterized in that: include: An acquisition module is configured to acquire a hash table recording a correspondence between nodes and coding blocks, where the coding blocks are converted from data blocks in a distributed storage system; A determination module is configured to determine a target coding block missing from a faulty node, and obtain a target node corresponding to the target coding block based on the hash table; A request module, configured to send a coding block request to the target node, wherein the coding blocks stored in the target node are allocated according to the bandwidth resource weight of the target node; A reconstruction module, configured to reconstruct a target data block according to the coding block returned by the target node and a coding matrix corresponding to the returned coding block; The repair module is configured to repair the data of the failed node by using the target data block.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product having a computer program stored thereon, characterized in that: When the computer program product is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Research and development tool chain-oriented cross-network software dependence self-consistent synchronization method and system

    CN122534083A