Data repair method and device

Abstract

The present disclosure proposes a data repair method and device, which relate to the field of computer technology. The data repair method includes: determining the coding blocks to be repaired in the current node; sorting the coding blocks to be repaired according to the distribution information of the coding blocks to be repaired in the disk; obtaining the available blocks required to repair the coding blocks to be repaired in sequence according to the sorting; repairing the coding blocks to be repaired according to the available blocks required to repair the coding blocks to be repaired. The present disclosure can sequentially pull and repair data, reduce the input and output (IO) times of reading and writing disks, reduce disk occupancy, and improve data repair efficiency.

Description

Technical Field

[0001] The present disclosure relates to the computer field, and in particular to a data repair method and device. Background Art

[0002] Distributed storage systems often use erasure code (EC) technology, which mainly uses the erasure code algorithm to encode the original data to obtain redundancy, and store the data and redundancy together to achieve fault tolerance. Data is stored in segments on each node. When a node is abnormal or the data is out of date, data can be pulled from other nodes and the data of the current node can be calculated according to the corresponding algorithm.

[0003] In the related art, a common implementation scheme for restoring data of a certain node is to query the key corresponding to the data block to be repaired from the metadata file, and pull the data corresponding to the key from other nodes for each data block to be repaired, until all the data blocks to be repaired are successfully repaired. In this way, the disk is directly read and written randomly according to the order of the keys in the meta file, which will lead to high disk usage and low data repair efficiency. In addition, the repair speed is limited because the content of the next data block is obtained for repair after one data block is repaired. Summary of the invention

[0004] A technical problem to be solved by the present disclosure is to provide a solution that can sequentially pull and repair data, reduce the number of input and output (IO) times of reading and writing disks, reduce disk occupancy, and improve data repair efficiency.

[0005] According to one aspect of the present disclosure, a data repair method is proposed, including: determining a coding block to be repaired in a current node; sorting the coding blocks to be repaired according to distribution information of the coding blocks to be repaired in a disk; sequentially acquiring available blocks required to repair the coding blocks to be repaired according to the sorting; and repairing the coding blocks to be repaired according to the available blocks required to repair the coding blocks to be repaired.

[0006] In some embodiments, the sorted coding blocks to be repaired are divided into at least one batch, and available blocks required to repair each batch of coding blocks to be repaired are obtained in batches; and based on the available blocks required to repair each batch of coding blocks to be repaired, the coding blocks to be repaired in the batch are batch repaired.

[0007] In some embodiments, the batch acquisition of free blocks is performed asynchronously with the batch repair.

[0008] In some embodiments, batch acquiring the available blocks required to repair each batch of coding blocks to be repaired includes: selecting multiple nodes from other nodes in the distributed storage system except the current node; concurrently initiating batch data acquisition requests to the multiple nodes; and receiving the available blocks required to repair the batch of coding blocks to be repaired returned by the multiple nodes.

[0009] In some embodiments, the method further includes: after determining the code block to be repaired in the current node, querying a storage information table according to a key of the code block to be repaired to obtain distribution information of the code block to be repaired in the disk.

[0010] In some embodiments, it also includes: reading the repaired coding blocks through an asynchronous reader / writer, and writing the repaired coding blocks into a pre-written log file; when a preset disk writing condition is met, writing the repaired coding blocks saved in the pre-written log file into the disk in batches.

[0011] In some embodiments, the preset disk writing condition includes: the number of data blocks written to the write-ahead log file is greater than a block number threshold; or the total number of bytes written to the write-ahead log file is greater than a byte number threshold.

[0012] According to another aspect of the present disclosure, a data repair device is also proposed, including: a determination module, configured to determine the coding blocks to be repaired in the current node; a sorting module, configured to sort the coding blocks to be repaired according to the distribution information of the coding blocks to be repaired in the disk; an acquisition module, configured to sequentially acquire the available blocks required to repair the coding blocks to be repaired according to the sorting; and a repair module, configured to repair the coding blocks to be repaired according to the available blocks required to repair the coding blocks to be repaired.

[0013] In some embodiments, the acquisition module is configured to divide the sorted coding blocks to be repaired into at least one batch, and batch acquire the available blocks required to repair each batch of coding blocks to be repaired; the repair module is configured to batch repair the coding blocks to be repaired in the batch according to the available blocks required to repair each batch of coding blocks to be repaired.

[0014] In some embodiments, the batch acquisition of available blocks by the acquisition module and the batch repair by the repair module are performed asynchronously.

[0015] In some embodiments, the acquisition module is configured to: select multiple nodes from other nodes in the distributed storage system except the current node; concurrently initiate batch data acquisition requests to the multiple nodes; and receive available blocks returned by the multiple nodes required to repair the batch of coding blocks to be repaired.

[0016] In some embodiments, the method further includes: a query module configured to query a storage information table according to a key of the coding block to be repaired, so as to obtain distribution information of the coding block to be repaired in the disk.

[0017] In some embodiments, a writing module is also included, and the writing module is configured to: read the repaired coding blocks through an asynchronous reader / writer, and write the repaired coding blocks into a pre-written log file; when a preset disk writing condition is met, write the repaired coding blocks saved in the pre-written log file into the disk in batches.

[0018] In some embodiments, the preset disk writing condition includes: the number of data blocks written to the write-ahead log file is greater than a block number threshold; or the total number of bytes written to the write-ahead log file is greater than a byte number threshold.

[0019] According to another aspect of the present disclosure, a data repair device is also proposed, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the above-mentioned data repair method based on instructions stored in the memory.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is also provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the above-mentioned data repair method is implemented.

[0021] Compared with the related art, in the embodiment of the present disclosure, by determining the coding blocks to be repaired in the current node, the coding blocks to be repaired are sorted according to the distribution information of the coding blocks to be repaired in the disk, and the available blocks required to repair the coding blocks to be repaired are obtained in sequence according to the sorting, and the coding blocks to be repaired are repaired based on the obtained available blocks. This can sequentially pull and repair data, reduce the input and output (IO) times of reading and writing the disk, reduce disk occupancy, and improve data repair efficiency.

[0022] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings.

[0025] in:

[0026] Figure 1 The figure is a flowchart of a data repair method according to some embodiments of the present disclosure.

[0027] Figure 2The present invention is a flowchart of obtaining available blocks required to repair a coding block to be repaired according to some embodiments of the present disclosure.

[0028] Figure 3 The present invention is a flowchart of repairing a coding block to be repaired according to available blocks according to some embodiments of the present disclosure.

[0029] Figure 4 Schematic diagram of a data repair method according to some other embodiments of the present disclosure.

[0030] Figure 5 It is a schematic diagram of the structure of a data repair device according to some embodiments of the present disclosure.

[0031] Figure 6 Schematic diagram of the structure of a data repair device according to some other embodiments of the present disclosure.

[0032] Figure 7 Schematic diagram of the structure of a data repair device according to some further embodiments of the present disclosure.

[0033] Figure 8 A schematic diagram of the structure of a computer system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0035] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0036] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0037] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0038] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0039] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0040] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0041] Figure 1 FIG. 1 is a flow chart of a data repair method according to some embodiments of the present disclosure. Figure 1 As shown, the data repair method includes:

[0042] Step S100: Determine the coding block to be repaired in the current node.

[0043] Among them, the coding block refers to all blocks obtained after the data block is operated by a coding algorithm such as an erasure code algorithm. The coding block includes a data block and a check block. A data block refers to a block generated after the original data object (such as a file) is segmented, and a check block is a redundant block obtained by encoding the original data block. The coding blocks corresponding to a file are stored in multiple nodes in a distributed storage system. Exemplarily, the node can be a computer device in a computer device cluster, or it can be a disk under the computer device. When a node is abnormal or data lags, the coding block stored in the node needs to be repaired.

[0044] In some embodiments, each node stores multiple directory files, such as a metadata directory file for storing information such as keys of coding blocks. In these embodiments, the keys of the coding blocks to be repaired in the current node are obtained from a local metadata directory file, such as a user metadata (userMeta) file. For example, for node 1, the userMeta file in node 1 is accessed to obtain the keys of all coding blocks to be repaired in node 1, thereby determining the coding blocks to be repaired in the current node.

[0045] In other embodiments, a data repair request is received, the data repair request including a key of a coding block to be repaired in a current node, and the data repair request is parsed to obtain the key of the coding block to be repaired in the current node.

[0046] Step S120: Sort the code blocks to be repaired according to the distribution information of the code blocks to be repaired in the disk.

[0047] In actual application scenarios, the keys of the coding blocks to be repaired in the current node obtained by step S100 are often sorted according to business needs, rather than sorted according to the distribution of the coding blocks on the disk. If data repair is performed directly on the keys of the coding blocks to be repaired obtained in step S100, random reading and writing will lead to high disk IO occupancy and low data repair efficiency. In view of this, the inventor of the present disclosure conceived that after determining the coding blocks to be repaired in the current node, the distribution information of the coding blocks to be repaired on the disk is obtained, and the coding blocks to be repaired are sorted according to the distribution information of the coding blocks to be repaired on the disk, so as to solve the problem of high disk IO occupancy and low data repair efficiency caused by random reading and writing during data repair.

[0048] In some embodiments, the distribution information of the code blocks to be repaired in the disk includes: the storage address of the code blocks to be repaired in the disk. In these embodiments, the keys of the code blocks to be repaired are sorted according to the order of the storage addresses of the code blocks to be repaired in the disk. For example, assuming that the keys of the code blocks to be repaired obtained by step S100 are key1, key2, key3, and key4, the keys of the code blocks to be repaired after sorting are key2, key4, key1, and key3 according to the order of the storage addresses of the code blocks to be repaired in the disk.

[0049] Step S140: according to the order, the available blocks required to repair the coding block to be repaired are obtained in sequence.

[0050] According to the principle of erasure code algorithm, encoding n data blocks can obtain n+m coding blocks, where these n+m coding blocks include n data blocks and m check blocks. When less than or equal to m coding blocks among the n+m coding blocks are lost, the lost coding blocks can be restored by any n available coding blocks (i.e., usable blocks).

[0051] In a distributed storage system, the same file is encoded into multiple encoding blocks, which are stored in multiple nodes in a dispersed manner. Each node can store encoding blocks of multiple files. In each node, the storage order of encoding blocks of multiple files on the disk is consistent. For example, node 1 stores encoding block a1 of file a, encoding block b1 of file b, and encoding block c1 of file c in sequence, node 2 stores encoding block a2 of file a, encoding block b2 of file b, and encoding block c2 of file c in sequence, and node 3 stores encoding block a3 of file a, encoding block b3 of file b, and encoding block c3 of file c in sequence.

[0052] In view of this, after the coding blocks to be repaired are sorted according to step S120, available blocks required to repair the coding blocks to be repaired are sequentially obtained from other nodes according to the sorting. In this way, compared with the out-of-order reading of available blocks in the related art, the sequential reading of available blocks reduces disk IO, improves the efficiency of obtaining available blocks, and thus improves the efficiency of data repair.

[0053] Step S160: repairing the coding block to be repaired according to the available blocks required to repair the coding block to be repaired.

[0054] In the embodiment of the present disclosure, the data repair of the current node is realized through the above steps. Compared with the related art, the present disclosure can sequentially pull and repair data through the above steps, reduce the IO times of reading and writing disks during data repair, reduce disk occupancy, and improve data repair efficiency.

[0055] Figure 2 The present invention is a flowchart of obtaining available blocks required to repair a coding block to be repaired according to some embodiments of the present disclosure. Figure 2 This is a detailed description of step S140. Figure 2 As shown, step S140 includes step S141 and step S142.

[0056] In step S141, the sorted coding blocks to be repaired are divided into at least one batch.

[0057] In some embodiments, the number of batches into which the sorted coding blocks to be repaired are divided is determined based on factors such as the amount of data of the coding blocks to be repaired, their distribution on the disk, network performance, and memory usage.

[0058] In step S142, available blocks required for repairing each batch of coding blocks to be repaired are obtained in batches.

[0059] In some embodiments, step S142 specifically includes: selecting multiple nodes from other nodes in the distributed storage system except the current node; concurrently initiating batch data acquisition requests to the selected multiple nodes for each batch of coding blocks to be repaired; and receiving available blocks required to repair the batch of coding blocks to be repaired returned by the selected multiple nodes.

[0060] During specific implementation, multiple nodes can be arbitrarily selected from other nodes except the current node in the distributed storage system, or multiple nodes can be selected according to the set selection strategy. After selecting multiple nodes, determine whether to establish a connection with each node; if a connection has been established with the node, send a batch data acquisition request to the node through the connection; if a connection has not been established with the node, create a connection with the node according to the IP address and port of the node, and send a batch data acquisition request to the node through the connection. Among them, the batch data acquisition request declares a list of data to be pulled. The selected multiple nodes respond to the batch data acquisition request and return the available blocks required to repair the batch of coding blocks to be repaired in sequence. Exemplarily, the data repair device can communicate with other nodes through a remote procedure call (RPC) protocol such as gRPC.

[0061] In some embodiments, considering the retry and fuse problems in abnormal situations such as node network jitter, when the data repair device batch obtains the available blocks required to repair each batch of the coding blocks to be repaired, the following steps are also included: determine whether the available blocks required to repair the current batch of coding blocks to be repaired returned by the selected node are received within the preset time; if yes, continue to send the batch data acquisition request for the next batch to the selected node; if not, determine whether the retry number threshold is reached; if the retry number threshold is not reached, re-initiate the batch data acquisition request for the current batch to the selected node, and increase the retry number by 1; if the retry number threshold is reached, output an error prompt message, or select another node from other nodes except the current node and the selected multiple nodes, and send a batch data acquisition request to the newly selected node. Through the above processing steps, the applicability and flexibility of the data repair method are improved.

[0062] In the disclosed embodiment, the above steps are used to sequentially and batch-acquire available blocks required for repairing the data to be repaired, which reduces the number of requests sent to other nodes and the number of disk IOs, thereby improving the efficiency of obtaining available blocks from the selected multiple nodes. Furthermore, by concurrently initiating batch data acquisition requests to the selected multiple nodes, the efficiency of obtaining available blocks can be further improved, thereby improving the efficiency of data repair.

[0063] Figure 3 The present invention is a flowchart of repairing a coding block to be repaired according to available blocks according to some embodiments of the present disclosure. Figure 3 This is a detailed description of step S160. Figure 3 As shown, step S160 includes step S161 and step S162.

[0064] In step S161, available blocks required to repair each batch of coded blocks to be repaired are read asynchronously.

[0065] In some embodiments, in order to further improve the efficiency of data repair, step S140 and step S160 are performed asynchronously. For example, the main thread performs the batch data repair work shown in step S160, and the created coroutine performs the batch data acquisition work shown in step S140, and after obtaining the available blocks from the selected nodes, the obtained available blocks are written to the channel.

[0066] In some embodiments, step S161 specifically includes: determining whether there is data in the channel; if there is data in the channel, obtaining data from the channel, and determining whether the data obtained from the channel is erroneous; if the data obtained from the channel is not erroneous, recording the data, that is, the available block required to repair the coding block to be repaired, so that the coding block to be repaired can be repaired based on the available block in the future; if the data obtained from the channel is erroneous, adding 1 to the error count value, and determining whether the error count value is greater than the redundancy number; if the error count value is greater than the redundancy number, outputting an error prompt message; if the error count value is less than or equal to the redundancy number, terminating the process of reading data from the channel this time; if there is no data in the channel, terminating the process of reading data from the channel this time.

[0067] In the above embodiment, the following optional implementation method can be used to determine whether the data obtained from the channel is erroneous: check the key and data cyclic redundancy check code (CRC) of the available block obtained from the channel; if the check of the key of the available block or the check of the data CRC fails, it is determined that the data obtained from the channel is erroneous; if the check of the key of the available block and the check of the data CRC both pass, it is determined that the data obtained from the channel is not erroneous.

[0068] In step S162, batch repair is performed on the coding blocks to be repaired in each batch according to the available blocks required to repair the coding blocks to be repaired.

[0069] In some embodiments, step S161 specifically includes: determining whether all available blocks required to repair the batch of coding blocks to be repaired have been successfully read, or whether all nodes have been visited; if so, repairing the batch of coding blocks to be repaired; if not, continuing to wait until all available blocks required to repair the batch of coding blocks to be repaired have been successfully read, or all nodes have been visited.

[0070] In the disclosed embodiment, the above steps are used to asynchronously and batch-wise repair the coding blocks to be repaired, which can further improve the efficiency of data repair.

[0071] Figure 4 FIG. 1 is a flow chart of a data repair method according to some other embodiments of the present disclosure. Figure 4 As shown, the data repair method includes:

[0072] Step S100: Determine the coding block to be repaired in the current node.

[0073] In some embodiments, each node stores multiple directory files, such as a metadata directory file for storing information such as keys of coding blocks. In these embodiments, the keys of the coding blocks to be repaired in the current node are obtained from a local metadata directory file, such as a user metadata (userMeta) file. For example, for node 1, the userMeta file in node 1 is accessed to obtain the keys of all coding blocks to be repaired in node 1, thereby determining the coding blocks to be repaired in the current node.

[0074] In other embodiments, a data repair request is received, the data repair request including a key of a coding block to be repaired in a current node, and the data repair request is parsed to obtain the key of the coding block to be repaired in the current node.

[0075] Step S110: query the storage information table according to the key of the code block to be repaired to obtain the distribution information of the code block to be repaired in the disk.

[0076] In some embodiments, a storage information table is pre-set to store the distribution information of the coding block in the disk. In these embodiments, the storage information table can be queried according to the key of the coding block to be repaired in the current node to conveniently and quickly obtain the distribution information of the coding block to be repaired in the disk.

[0077] Step S120: Sort the code blocks to be repaired according to the distribution information of the code blocks to be repaired in the disk.

[0078] In some embodiments, the distribution information of the code blocks to be repaired in the disk includes: the storage address of the code blocks to be repaired in the disk. In these embodiments, the keys of the code blocks to be repaired are sorted according to the order of the storage addresses of the code blocks to be repaired in the disk. For example, assuming that the keys of the code blocks to be repaired obtained by step S100 are key1, key2, key3, and key4, the keys of the code blocks to be repaired after sorting are key2, key4, key1, and key3 according to the order of the storage addresses of the code blocks to be repaired in the disk.

[0079] Step S140: according to the order, the available blocks required to repair the coding block to be repaired are obtained in sequence.

[0080] In some embodiments, step S140 specifically includes: dividing the sorted coding blocks to be repaired into at least one batch, and batch-acquiring available blocks required to repair the coding blocks to be repaired in each batch.

[0081] In specific implementation, the number of batches into which the sorted coding blocks to be repaired are divided can be determined based on factors such as the amount of data of the coding blocks to be repaired, their distribution on the disk, network performance, and memory usage.

[0082] In an optional implementation, batch acquiring the available blocks required to repair each batch of coding blocks to be repaired specifically includes: selecting multiple nodes from nodes other than the current node in the distributed storage system; concurrently initiating batch data acquisition requests to the selected multiple nodes for each batch of coding blocks to be repaired; and receiving the available blocks required to repair the batch of coding blocks to be repaired returned by the selected multiple nodes.

[0083] In the embodiment of the present disclosure, through steps S100 to S140, the available blocks required for repairing the data to be repaired are sequentially and batched, the number of requests sent to other nodes is reduced, and the number of disk IOs is reduced, and the efficiency of obtaining available blocks from the selected multiple nodes is improved. Furthermore, by concurrently initiating batch data acquisition requests to the selected multiple nodes, the efficiency of obtaining available blocks can be further improved, thereby improving the efficiency of data repair.

[0084] Step S160: repairing the coding block to be repaired according to the available blocks required to repair the coding block to be repaired.

[0085] In some embodiments, step S160 specifically includes: performing batch repair on the batch of coding blocks to be repaired according to available blocks required to repair each batch of coding blocks to be repaired.

[0086] In some embodiments, in order to further improve data repair efficiency, step S140 and step S160 are performed asynchronously.

[0087] Step S170: read the repaired coding block through the asynchronous reader / writer, and write the repaired coding block into the pre-written log file.

[0088] Considering serial reading and writing, the reading and writing speeds will affect each other. Therefore, the inventor of the present disclosure thinks of creating an asynchronous reader / writer to asynchronously read and write the repaired coding block.

[0089] In some embodiments, step S170 specifically includes: reading the repaired coding block from the channel through an asynchronous reader / writer; determining whether the repaired coding block read is erroneous; if the repaired coding block read is not erroneous, writing the repaired coding block to a write-ahead log file (wal file); if the repaired coding block read is erroneous, determining whether the end identifier (EOF) of the data block is read; if the end identifier of the data block is read, ending; if the end identifier of the data block is not read, outputting an error prompt message.

[0090] In some embodiments, after writing the repaired coding block to the wal file, it also includes: determining whether writing the wal file fails; if so, ending; if not, updating the number of data blocks written to the wal file and updating the total number of bytes written to the write-ahead log file.

[0091] Step S180: When a preset disk writing condition is met, the repaired coding blocks saved in the pre-written log file are written to the disk in batches.

[0092] In some embodiments, the preset disk writing condition includes: the number of data blocks written to the write-ahead log file is greater than a block number threshold; or the total number of bytes written to the write-ahead log file is greater than a byte number threshold.

[0093] In specific implementation, when the preset disk writing conditions are met, the sync interface can be called to write the repaired code blocks saved in the pre-written log file to the disk in batches, so as to achieve the purpose of reducing disk IO.

[0094] In the embodiment of the present disclosure, the above steps are used to efficiently repair the data of the current node. Compared with the related art, the embodiment of the present disclosure can sequentially and batch pull data, batch repair data, concurrently read and write data, and batch write repaired data through the above steps, which reduces the number of IO times of reading and writing disks during data repair, reduces disk occupancy, and improves data repair efficiency.

[0095] Figure 5 FIG. 1 is a schematic diagram of the structure of a data repair device according to some embodiments of the present disclosure. Figure 5 As shown, the data repair device includes: a determination module 510, a sorting module 520, an acquisition module 530, and a repair module 540.

[0096] The determination module 510 is configured to determine the coding block to be repaired in the current node.

[0097] In some embodiments, each node stores multiple directory files, such as metadata directory files for storing information such as keys of coding blocks. In these embodiments, the determination module 510 obtains the keys of the coding blocks to be repaired in the current node from the local metadata directory files, such as the user metadata (userMeta) file. For example, for node 1, the userMeta file in node 1 is accessed to obtain the keys of all coding blocks to be repaired in node 1, thereby determining the coding blocks to be repaired in the current node.

[0098] In some other embodiments, the determination module 510 receives a data repair request including a key of a coding block to be repaired in a current node, and parses the data repair request to obtain the key of the coding block to be repaired in the current node.

[0099] The sorting module 520 is configured to sort the code blocks to be repaired according to the distribution information of the code blocks to be repaired in the disk.

[0100] In some embodiments, the distribution information of the coding blocks to be repaired in the disk includes: the storage address of the coding blocks to be repaired in the disk. In these embodiments, the sorting module 520 sorts the keys of the coding blocks to be repaired according to the order of the storage addresses of the coding blocks to be repaired in the disk. For example, assuming that the keys of the coding blocks to be repaired obtained by the determination module 510 are key1, key2, key3, and key4, the keys of the coding blocks to be repaired after sorting are key2, key4, key1, and key3 according to the order of the storage addresses of the coding blocks to be repaired in the disk.

[0101] The acquisition module 530 is configured to sequentially acquire available blocks required to repair the coding block to be repaired according to the order.

[0102] In some embodiments, the acquisition module 530 is configured to divide the sorted coding blocks to be repaired into at least one batch, and batch-acquire available blocks required to repair the coding blocks to be repaired in each batch.

[0103] In specific implementation, the acquisition module 530 may determine the number of batches into which the sorted coding blocks to be repaired are divided according to factors such as the data volume of the coding blocks to be repaired, the distribution on the disk, the network performance, and the memory usage.

[0104] In an optional implementation, the acquisition module 530 batch-acquires the available blocks required to repair each batch of coding blocks to be repaired, specifically including: the acquisition module 530 selects multiple nodes from other nodes except the current node in the distributed storage system; for each batch of coding blocks to be repaired, the acquisition module 530 concurrently initiates batch data acquisition requests to the selected multiple nodes; the acquisition module 530 receives the available blocks required to repair the batch of coding blocks to be repaired returned by the selected multiple nodes.

[0105] In the disclosed embodiment, the ordering module and the acquisition module are used to sequentially and batch-acquire the available blocks required for repairing the data to be repaired, which reduces the number of requests sent to other nodes and the number of disk IOs, thereby improving the efficiency of obtaining available blocks from the selected multiple nodes. Furthermore, by initiating batch data acquisition requests to the selected multiple nodes concurrently through the acquisition module, the efficiency of obtaining available blocks can be further improved, thereby improving the efficiency of data repair.

[0106] The repair module 540 is configured to repair the coding block to be repaired according to the available blocks required to repair the coding block to be repaired.

[0107] In some embodiments, the repair module 540 is configured to perform batch repair on the batch of coding blocks to be repaired according to available blocks required to repair each batch of coding blocks to be repaired.

[0108] In some embodiments, in order to further improve data repair efficiency, the acquisition module 530 acquires available blocks in batches and the repair module 540 repairs blocks in batches asynchronously.

[0109] In the embodiment of the present disclosure, the data repair of the current node is realized through the above device. Compared with the related art, the present disclosure can sequentially pull and repair data through the above device, reduce the IO times of reading and writing disks during data repair, reduce disk occupancy, and improve data repair efficiency.

[0110] Figure 6 FIG. 1 is a schematic diagram of the structure of a data repair device according to some other embodiments of the present disclosure. Figure 6 As shown, the data repair device includes: a determination module 510 , a query module 511 , a sorting module 520 , an acquisition module 530 , a repair module 540 , and a writing module 550 .

[0111] The determination module 510 is configured to determine the coding block to be repaired in the current node.

[0112] In some embodiments, each node stores multiple directory files, such as metadata directory files for storing information such as keys of coding blocks. In these embodiments, the determination module 510 obtains the keys of the coding blocks to be repaired in the current node from the local metadata directory files, such as the user metadata (userMeta) file. For example, for node 1, the userMeta file in node 1 is accessed to obtain the keys of all coding blocks to be repaired in node 1, thereby determining the coding blocks to be repaired in the current node.

[0113] The query module 511 is configured to query the storage information table according to the key of the code block to be repaired, so as to obtain the distribution information of the code block to be repaired in the disk.

[0114] In some embodiments, a storage information table is pre-set to store the distribution information of the coding block in the disk. In these embodiments, the query module 511 can query the storage information table according to the key of the coding block to be repaired in the current node to conveniently and quickly obtain the distribution information of the coding block to be repaired in the disk.

[0115] The sorting module 520 is configured to sort the code blocks to be repaired according to the distribution information of the code blocks to be repaired in the disk.

[0116] In some embodiments, the distribution information of the coding blocks to be repaired in the disk includes: the storage address of the coding blocks to be repaired in the disk. In these embodiments, the sorting module 520 sorts the keys of the coding blocks to be repaired according to the order of the storage addresses of the coding blocks to be repaired in the disk. For example, assuming that the keys of the coding blocks to be repaired obtained by the determination module 510 are key1, key2, key3, and key4, the keys of the coding blocks to be repaired after sorting are key2, key4, key1, and key3 according to the order of the storage addresses of the coding blocks to be repaired in the disk.

[0117] The acquisition module 530 is configured to sequentially acquire available blocks required to repair the coding block to be repaired according to the order.

[0118] In some embodiments, the acquisition module 530 is configured to divide the sorted coding blocks to be repaired into at least one batch, and batch-acquire available blocks required to repair the coding blocks to be repaired in each batch.

[0119] The repair module 540 is configured to repair the coding block to be repaired according to the available blocks required to repair the coding block to be repaired.

[0120] In some embodiments, the repair module 540 is configured to perform batch repair on the batch of coding blocks to be repaired according to available blocks required to repair each batch of coding blocks to be repaired.

[0121] In some embodiments, in order to further improve data repair efficiency, the acquisition module 530 acquires available blocks in batches and the repair module 540 repairs blocks in batches asynchronously.

[0122] The writing module 550 is configured to read the repaired coding blocks through an asynchronous reader and writer, and write the repaired coding blocks into a pre-written log file; when a preset disk writing condition is met, the repaired coding blocks saved in the pre-written log file are batch-written into the disk.

[0123] In some embodiments, the preset disk writing condition includes: the number of data blocks written to the write-ahead log file is greater than a block number threshold; or the total number of bytes written to the write-ahead log file is greater than a byte number threshold.

[0124] In specific implementation, when the preset disk writing conditions are met, the writing module 550 can call the sync interface to write the repaired coding blocks saved in the pre-written log file to the disk in batches, so as to reduce the disk IO.

[0125] In some embodiments, the writing module 550 is further configured to: after writing the repaired coding block to the wal file, determine whether writing the wal file fails; if so, end; if not, update the number of data blocks written to the wal file and update the total number of bytes written to the write-ahead log file.

[0126] In the embodiment of the present disclosure, the above device is used to efficiently repair the data of the current node. Compared with the related art, the embodiment of the present disclosure can pull data sequentially and in batches, repair data in batches, read and write data concurrently, and write repaired data in batches through the above device, which reduces the number of IO times of reading and writing disks during data repair, reduces disk occupancy, and improves data repair efficiency.

[0127] Figure 7 is a block diagram showing a data repair device according to some other embodiments of the present disclosure.

[0128] like Figure 7 As shown, the data repair device 700 includes a memory 710; and a processor 720 coupled to the memory 710. The memory 710 is used to store instructions for executing the corresponding embodiments of the data repair method. The processor 720 is configured to execute the data repair method in any of the embodiments of the present disclosure based on the instructions stored in the memory 710.

[0129] Figure 8 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0130] like Figure 8 As shown, the computer system 800 may be embodied in the form of a general-purpose computing device. The computer system 800 includes a memory 810, a processor 820, and a bus 830 that connects various system components.

[0131] The memory 810 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing at least one of the corresponding embodiments of the data repair method. The non-volatile storage medium may include, but is not limited to, a disk memory, an optical memory, a flash memory, etc.

[0132] The processor 820 can be implemented by a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the judgment module and the determination module can be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or can be implemented by a dedicated circuit that executes corresponding steps.

[0133] The bus 830 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0134] The computer system 800 may also include an input / output interface 840, a network interface 850, a storage interface 860, etc. These interfaces 840, 850, 860, the memory 810, and the processor 820 may be connected via a bus 830. The input / output interface 840 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 850 may provide a connection interface for various networked devices. The storage interface 860 may provide a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0135] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0137] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0138] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0139] The data repair method and device in the above-mentioned embodiments can reduce the number of IO times of reading and writing the disk, reduce the disk occupancy rate, and improve the data repair efficiency.

[0140] So far, the data repair method and device according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

Claims

1. A data repair method, comprising: Determine the coding block to be repaired in the current node; Sorting the code blocks to be repaired according to the distribution information of the code blocks to be repaired in the disk, including: sorting the code blocks to be repaired according to the order of storage addresses of the code blocks to be repaired in the disk; Divide the sorted coding blocks to be repaired into at least one batch, and batch-acquire available blocks required to repair the coding blocks to be repaired in each batch; The batch of coding blocks to be repaired is repaired in batches according to the available blocks required for repairing each batch of coding blocks to be repaired, wherein batch acquisition of available blocks is performed asynchronously with batch repairing.

2. The data repair method according to claim 1, wherein: The available blocks required to batch-acquire the repaired coded blocks for each batch include: Selecting multiple nodes from other nodes in the distributed storage system except the current node; Concurrently initiating batch data acquisition requests to the multiple nodes; Receive the available blocks returned by the multiple nodes and required to repair the batch of coding blocks to be repaired.

3. The data repair method according to claim 1, further comprising: After the code block to be repaired in the current node is determined, the storage information table is queried according to the key of the code block to be repaired to obtain the distribution information of the code block to be repaired in the disk.

4. The data repair method according to claim 1, further comprising: Read the repaired coding block through an asynchronous reader / writer, and write the repaired coding block into a write-ahead log file; When the preset disk writing condition is met, the repaired coding blocks saved in the pre-written log file are written to the disk in batches.

5. The data repair method according to claim 4, wherein the preset disk writing condition comprises: The number of data blocks written to the write-ahead log file is greater than the block number threshold; or The total number of bytes written to the write-ahead log file is greater than the bytes threshold.

6. A data repair device, comprising: A determination module, configured to determine a coding block to be repaired in a current node; The sorting module is configured to sort the code blocks to be repaired according to the distribution information of the code blocks to be repaired in the disk, including: sorting the code blocks to be repaired according to the order of storage addresses of the code blocks to be repaired in the disk; An acquisition module is configured to divide the sorted coding blocks to be repaired into at least one batch, and batch-acquire available blocks required to repair the coding blocks to be repaired in each batch; The repair module is configured to perform batch repair on the batch of coding blocks to be repaired according to the available blocks required for repairing each batch of coding blocks to be repaired, wherein the batch acquisition of available blocks by the acquisition module is performed asynchronously with the batch repair by the repair module.

7. The data repair device according to claim 6, wherein: The acquisition module is configured as follows: Selecting multiple nodes from other nodes in the distributed storage system except the current node; Concurrently initiating batch data acquisition requests to the multiple nodes; Receive the available blocks returned by the multiple nodes and required to repair the batch of coding blocks to be repaired.

8. The data repair device according to claim 6, further comprising: The query module is configured to query the storage information table according to the key of the code block to be repaired, so as to obtain the distribution information of the code block to be repaired in the disk.

9. The data repair device according to claim 6, further comprising a writing module, wherein the writing module is configured to: Read the repaired coding block through an asynchronous reader / writer, and write the repaired coding block into a write-ahead log file; When the preset disk writing condition is met, the repaired coding blocks saved in the pre-written log file are written to the disk in batches.

10. The data repair device according to claim 9, wherein: The preset disk writing conditions include: The number of data blocks written to the write-ahead log file is greater than the block count threshold; or The total number of bytes written to the write-ahead log file is greater than the bytes threshold.

11. A data repair device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the data repair method according to any one of claims 1 to 5 based on instructions stored in the memory.

12. A computer-readable storage medium having computer program instructions stored thereon, wherein the instructions, when executed by a processor, implement the data repair method according to any one of claims 1 to 5.

Images