A method and apparatus for repairing bad blocks of disks in a stream replication cluster
By selecting a target node in the streaming replication cluster to obtain and verify data blocks, the problem of unrepairable bad blocks on database disks is solved, achieving efficient bad block repair and ensuring high availability and data integrity of the database.
Patent Information
- Application Number
- CN202210129182.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-11
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2042-02-11
AI Technical Summary
Existing technologies cannot effectively repair bad blocks on database disks, leading to data read errors and affecting database access. Furthermore, traditional high-availability cluster management may have improperly set switching thresholds when disk read errors occur, potentially resulting in frequent switching or performance loss.
In the streaming replication cluster, a target node is selected, and the corresponding data block is obtained from the target node by calling a preset function. After verification, the data block is overwritten to repair bad blocks.
It enables efficient repair of bad blocks in streaming replication clusters, ensuring high disk availability and data accuracy, while reducing system overhead and the risk of frequent switching.
Smart Images
Figure CN114461455B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to database technology, and in particular to a method and device for repairing disk bad blocks in a stream replication cluster. BACKGROUND
[0002] Database files may encounter various abnormalities (system I / O errors, virus damage to files, human errors, cosmic ray radiation, etc.) when writing to disks, resulting in errors in the file content on the disks.
[0003] Databases provide checksums mechanisms to check when reading disk files, and if a file checksum fails, an error is reported and the current transaction is terminated or a zero write operation is performed on the bad block. However, checksums can only detect file bad blocks and cannot repair them.
[0004] When users use traditional large commercial databases, storage is usually selected for professional storage arrays. Storage arrays generally have high availability capabilities, typically including RAID1, RAID5, etc. When a disk block or even a disk group fails, data access is not affected. However, in the current popular domestic systems, the storage system used is much simpler, and many scenarios do not use storage with high availability functions. At this time, if the database managed data encounters a disk failure, the data block cannot be read or the read data is incorrect, and the corresponding database access statement will report an error. In this case, relying solely on high-availability cluster management is not enough. Because ordinary disk read errors affect specific statements, it is not necessary to switch servers. If the switching condition in the high-availability cluster management is set to include disk read errors, the server switching threshold is too low, which may cause switching to be too frequent.
[0005] From the perspective of the database, high availability of storage traditionally adopts a disk mirroring mode. However, ordinary disk mirroring has certain burdens on performance and management complexity. Therefore, a disk bad block repair method is needed to repair bad blocks. SUMMARY
[0006] The embodiments of the present application provide a method and device for repairing disk bad blocks in a stream replication cluster, which selects a target node in the stream replication cluster to overwrite the bad blocks, thereby repairing the bad blocks.
[0007] The embodiments of the present application provide a method for repairing disk bad blocks in a stream replication cluster, the stream replication cluster including at least one master node and a plurality of backup nodes, wherein each backup node is used to backup at least part of the data of the at least one master node, and the method for repairing disk bad blocks includes:
[0008] determining a target node from the rest of the nodes, the target node containing a data block corresponding to the bad block;
[0009] calling a preset function according to the bad block parameter of the bad block to obtain data of the corresponding data block from the target node;
[0010] checking the obtained data, and repairing the bad block with data of the data block if the checking is passed.
[0011] In some embodiments, the determination of the existence of the bad block in the current node is achieved by the following steps:
[0012] reading data in each data block of the disk of the current node and calculating a checking parameter;
[0013] comparing the calculated checking parameter with a preset checking parameter;
[0014] determining the data block as a bad block if the comparison is inconsistent;
[0015] checking a header field of the data block if the comparison is consistent;
[0016] determining the data block as a bad block if there is a logical error in the header field of the data block.
[0017] In some embodiments, the determination of the target node from the rest of the nodes includes:
[0018] selecting a target node from the standby nodes by a preset algorithm so that a data block corresponding to the bad block in the target node has the closest data version if the master node is determined to have a bad block;
[0019] determining the master node as the target node if the standby node is determined to have a bad block.
[0020] In some embodiments, the disk bad block repairing method further includes:
[0021] reading bad block chain table information and determining whether there is data of the bad block in the local cache based on the bad block chain table information if the current node is determined to have a bad block;
[0022] repairing the bad block with the local cache if the data of the bad block is determined to exist in the local cache;
[0023] determining a target node from the rest of the nodes if the data of the bad block is determined to not exist in the local cache.
[0024] In some embodiments, the checking of the obtained data and the repairing of the bad block with data of the data block if the checking is passed includes:
[0025] In the case that the primary node has a bad block, the database transaction log sequence number LSN of the corresponding data block obtained from the target node is determined;
[0026] The LSN of the corresponding data block obtained from the target node is compared with the LSN of the current database in the primary node;
[0027] In the case that the LSN of the corresponding data block obtained from the target node is equal to or exceeds the LSN of the current database, the bad block is repaired by using the corresponding data block obtained from the target node;
[0028] In the case that the LSN of the corresponding data block obtained from the target node lags behind the LSN of the current database, no repair is performed, and the next round of detection is waited.
[0029] In some embodiments, the obtained data is checked, and in the case that the check passes, the bad block is repaired by using the data of the data block, which includes:
[0030] In the case that the backup node has a bad block, the LSN of the current database in the backup node is compared with the LSN of the corresponding data block obtained from the target node;
[0031] If the LSN of the corresponding data block obtained from the target node is less than the LSN of the current database, repair is performed;
[0032] If the LSN of the corresponding data block obtained from the target node is greater than or equal to the LSN of the current database, no repair is performed.
[0033] In some embodiments, the disk bad block repair method further includes:
[0034] After the bad block is repaired by using the data of the data block, the repaired bad block information is updated to a specified file.
[0035] In some embodiments, the disk bad block repair method further includes:
[0036] After reading the disk file and checking out the bad block, a specified dynamic library is loaded, and a preset repair function is called to perform repair.
[0037] Embodiments of the present application also propose a disk bad block repair device of a stream replication cluster, the stream replication cluster including at least one primary node and a plurality of backup nodes, wherein each backup node is used to backup at least part of data of the at least one primary node, and the disk bad block repair device includes a processor configured to:
[0038] In the case that the current node has a bad block, a target node is determined from the remaining nodes, and the target node contains a data block corresponding to the bad block;
[0039] According to the bad block parameter of the bad block, a preset function is called to obtain data of a corresponding data block from the target node;
[0040] The obtained data is checked, and in a case that the checking is passed, the data of the data block is used to repair the bad block.
[0041] The embodiment of the application further provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement steps of the disk bad block repairing method of the stream replication cluster.
[0042] The embodiment of the application implements the overwriting of the bad block by selecting a target node in the stream replication cluster, obtaining data of a corresponding data block from the target node, and repairing the bad block by using the data of the data block after checking the obtained data.
[0043] The above description is only a summary of the technical scheme of the application, in order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS
[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals are intended to identify the same components throughout the accompanying drawings. In the drawings:
[0045] Figure 1 The basic architecture of the stream replication cluster of the embodiment of the application is shown in the figure;
[0046] Figure 2 The basic flow of the disk bad block repairing method of the embodiment of the application is shown in the figure. DETAILED DESCRIPTION
[0047] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While example embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.
[0048] This invention provides a method for repairing bad blocks on a streaming replication cluster, wherein the streaming replication cluster includes at least one master node and multiple backup nodes, wherein each backup node is used to back up at least a portion of the data of the at least one master node. Figure 1 As shown, the difference between the standby node and the primary node in this example is that the standby node has data query capabilities but lacks other data operation capabilities. All data on the standby node originates solely from the primary node. Specifically, during the initial backup, the standby node can copy at least a portion of the required data from the primary node. Subsequently, it only needs to receive the primary node's operation logs and execute the data operations recorded in the logs for the corresponding data blocks to maintain data consistency between the standby node and the primary node, allowing users to query data on any standby node. Figure 2 As shown, the disk bad block repair method includes:
[0049] In step S201, if it is determined that the current node has bad blocks, a target node is determined from the remaining nodes. The target node contains the data block corresponding to the bad block. In this example, the current node can be either a primary node or a backup node. Specifically, it can be detected whether the current node has bad blocks by setting up, for example, a scanning program, or by using scripts, etc., without being limited to any particular method. In this example, a bad block in a node refers to a data block whose data is unusable. In this case, in this example, a target node is determined from the remaining nodes, and the target node contains a complete data block corresponding to the bad block.
[0050] In step S202, based on the bad block parameters, a preset function is invoked to retrieve the corresponding data block from the target node. For example, an SQL statement calling a custom function `fetch_data` can be constructed and sent to the target node database for execution to obtain the corresponding data block.
[0051] In step S203, the acquired data is verified. If the verification passes, the bad block is repaired using the data from that data block. Specifically, the acquired data block can be verified, for example, by checking the data version. Since there may be data latency between the primary and backup nodes, the data versions of the primary and backup nodes may be inconsistent. If the data version of the bad block is higher than the data version in the target node, data repair cannot be performed. Therefore, in this application, data verification is used to overwrite the data once it is determined that the correct data has been acquired, thus completing the bad block repair.
[0052] In this embodiment of the invention, a target node is selected in the streaming replication cluster, the number of corresponding data blocks is obtained from the target node, and the obtained data is verified and then used to repair the bad block, thereby overwriting the bad block and completing the repair of the bad block.
[0053] In some embodiments, the determination that the current node has a bad block is achieved by the following steps:
[0054] Reading data in each data block of the disk of the current node and calculating a check parameter;
[0055] Comparing the calculated check parameter with a preset check parameter;
[0056] In the case of inconsistency, determining that the data block is a bad block;
[0057] In the case of consistency, checking the header field of the data block;
[0058] In the case that the header field of the data block has a logical error, determining that the data block is a bad block.
[0059] In this example, the active discovery of bad blocks in the node is taken as an example. The bad block scanning is initiated actively when the system is idle or according to the user's operation instruction. The scanning is performed in units of blocks, and the default block size can be set to 8kbytes. After the file block is read into the memory, the checksum is calculated according to the content of the data block, and compared with the checksum stored in the block header when the previous data block is written to the disk. If the values of the two times are inconsistent, it indicates that the data block has been damaged. If the checksums are matched, the header field of the block is further checked, and if a logical error occurs, it indicates that the data block has been damaged. The information of the bad block is recorded in a bad block linked list. At the same time, the bad block number information is also updated to the specified statistical data file db_blockguard.dat.
[0060] In some embodiments, the determination of a target node from the remaining nodes includes:
[0061] In the case that the primary node has a bad block, a target node is selected from the backup nodes by a preset algorithm, so that the data block corresponding to the bad block in the target node has the closest data version. That is, in this example, the bad block scanning in the primary node is taken as an example. When a bad block is found, an optimal database data block is selected from a plurality of backup nodes for overwriting according to a preset block selection algorithm. Specifically, a UDF fetch_data() custom function can be used to read the file block installed on the remote backup node, and a UDF db_blockguard core program module is responsible for receiving the start or stop scanning instruction of the command line tool.
[0062] In the case that the backup node has a bad block, the primary node is taken as the target node.
[0063] In some embodiments, the disk bad block repair method further includes:
[0064] In the case where it is determined that the current node has a bad block, the bad block chain table information is read, and it is determined whether the data of the bad block exists in the local cache based on the bad block chain table information. In the case where it is determined that the data of the bad block exists in the local cache, the bad block is repaired by using the local cache. In the case where it is determined that the current node has a bad block, the information in the bad block chain table is read in sequence, and it is preferentially determined whether the data block exists in the local share buffer. If the data block exists, it is not necessary to obtain the block information from the slave node to perform the repair, and the local buffer can be directly overwritten.
[0065] In the case where it is determined that the data of the bad block does not exist in the local cache, a target node is determined from the remaining nodes.
[0066] In some embodiments, the obtained data is checked, and in the case where the check passes, the bad block is repaired by using the data of the data block.
[0067] In the case where it is determined that the master node has a bad block, the database transaction log sequence number LSN of the corresponding data block obtained from the target node is determined.
[0068] The LSN of the corresponding data block obtained from the target node is compared with the LSN of the current database in the master node.
[0069] In the case where the LSN of the corresponding data block obtained from the target node is equal to or exceeds the LSN of the current database, the bad block is repaired by using the corresponding data block obtained from the target node.
[0070] In the case where the LSN of the corresponding data block obtained from the target node lags behind the LSN of the current database, the repair is not performed, and the next round of detection is waited.
[0071] Specifically, in a stream replication environment, the master node and the slave node can have a delay, and the data block obtained from the remote node database can not be the same version as the data block to be repaired. Therefore, the version of the bad block and the data of the obtained data block needs to be verified before the block is overwritten and replaced. For example, when a bad block is found on the master node, when a perfect data block is obtained from the slave node, if the LSN of the data block of the slave node is later than the LSN of the bad block of the master node, it indicates that the data of the slave node has not been updated according to the latest write-ahead log (WAL), and if such a data block is used to perform the repair, an error will be caused.
[0072] In the case where the check and verification fail, all information of the current damaged data block is unusable, including its LSN, and therefore other information needs to be used for verification, such as the last LSN number of the current database-currentlatest LSN.
[0073] In the case where it is determined that the master node has a bad block, the repair can be implemented in the following manner:
[0074] Record the latest database transaction log sequence number (LSN) of the master node when a bad block is discovered (e.g., marked as "latestlsn master").
[0075] By using a preset block selection algorithm, an optimal backup node is selected as the target node for block repair.
[0076] Determine the latest database transaction log sequence number (LSN) of the target node (e.g., mark it as "latest lsn backup").
[0077] If it is determined that "latest lsn backup" is equal to or greater than "latest lsn primary", the bad block is repaired using the database in the target node.
[0078] When the "latest lsn backup" lags behind the "latest lsn primary", it is not repaired temporarily and is stored in the list to be repaired, waiting for the next cycle check to attempt repair. During this process, the backup node's latest LSN is constantly catching up with the primary node's latest LSN based on WAL. Once it catches up with the primary node's latest LSN, block repair can be performed.
[0079] In some embodiments, verifying the acquired data, and repairing the bad block with the data block if the verification passes, includes:
[0080] If it is determined that there are bad blocks in the standby node, compare the LSN of the current database in the standby node with the LSN of the corresponding data block obtained from the target node;
[0081] If the LSN of the corresponding data block obtained from the target node is less than the LSN of the current database, then perform the repair.
[0082] If the LSN of the corresponding data block obtained from the target node is greater than or equal to the LSN of the current database, no repair is performed.
[0083] Specifically, this example further illustrates the situation when a bad block occurs on the standby node:
[0084] Record the latest database transaction log sequence number (LSN) of the standby node when a bad block is discovered (marked as "latestlsn standby").
[0085] The LSN of the block obtained from the master node (marked as "LSN master").
[0086] If the "LSN master" is less than the "latest lsn standby" of the standby node, it can be determined that the obtained block is the correct version. And if the "LSN master" of the obtained block is greater than the "latest lsn standby" of the standby node, no repair is performed. In this case, the bad block will be repaired in the subsequent WAL replay.
[0087] And the method of the present application, regardless of whether the bad block is in the master node or the standby node, if the write fails when the repair process occurs, the bad block will still be a bad block even if the write fails, and will be found and repaired in the next round of scanning.
[0088] In some embodiments, the disk bad block repair method further comprises: after repairing the bad block with the data of the data block, updating the repaired bad block information to a specified file. Specifically, the repaired bad block information will be updated to the statistical data file db_blockguard.dat, so that statistics and queries can be facilitated.
[0089] In some embodiments, the disk bad block repair method further comprises:
[0090] After reading the disk file and checking out the bad block, a specified dynamic library is loaded, and a preset repair function is called to perform repair. Specifically, in the present example, corresponding to the application scenario of real-time repair, after reading the disk file and discovering the bad block, the user in the real-time use process, the scanning program will immediately capture the bad block information and repair it in real time. Specifically, the repair can be performed by loading the libpqblockguard dynamic library and calling the repair function repair_block(). The repair function will perform the connection master node or standby node and the repair bad block operation, and the process is the same as the foregoing active repair, which will not be described here.
[0091] The disk bad block repair method of the present embodiment can guarantee the high availability of the disk in the production environment. In the use process, the bad block is found and can be repaired immediately, and the user request has almost zero delay. In the idle business state (for example, in the early morning), the bad block detection and repair can be actively initiated, and the disk data security and accuracy are fully guaranteed. And the disk bad block repair method of the present embodiment can guarantee the high availability of the disk in a low-cost and low-system-cost manner.
[0092] The embodiment of the present application also proposes a disk bad block repair device for a stream replication cluster, the stream replication cluster comprising at least one master node and a plurality of standby nodes, wherein each standby node is used to backup at least part of the data of the at least one master node, and the disk bad block repair device comprises a processor configured to:
[0093] In the case where it is determined that the current node has a bad block, a target node is determined from the remaining nodes, and the target node contains a data block corresponding to the bad block.
[0094] According to the bad block parameter of the bad block, a preset function is called to obtain data of the corresponding data block from the target node;
[0095] The obtained data is checked, and in a case where the checking is passed, the bad block is repaired by using the data of the data block.
[0096] The embodiment of the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the disk bad block repairing method of the stream replication cluster.
[0097] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles, or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles, or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article, or device including the element.
[0098] The above-mentioned embodiment numbers of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.
[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server or network device, etc.) execute the methods described in various embodiments of the application.
[0100] The embodiments of the application are described above in combination with the drawings, but the application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many forms under the inspiration of the application without departing from the purpose of the application and the scope protected by the claims. These all belong to the protection of the application.
Claims
1. A method for repairing a bad block of a disk of a flow replication cluster, characterized by, The stream replication cluster comprises at least one master node and a plurality of backup nodes, wherein each backup node is used to backup at least part of data of the at least one master node, and the disk bad block repairing method comprises: In a case where it is determined that the current node has a bad block, a target node is determined from the remaining nodes, and the target node contains a data block corresponding to the bad block; According to the bad block parameter of the bad block, a preset function is called to obtain data of the corresponding data block from the target node; The obtained data is checked, and in a case where the checking is passed, the bad block is repaired by using the data of the data block; Determining a target node from the remaining nodes comprises: In a case where it is determined that the master node has a bad block, a target node is selected from the backup nodes by using a preset algorithm, so that the data block corresponding to the bad block in the target node has the closest data version; In a case where it is determined that the backup node has a bad block, the master node is taken as the target node; The disk bad block repairing method further comprises: In a case where it is determined that the current node has a bad block, bad block chain table information is read, and it is determined whether the data of the bad block exists in the local cache based on the bad block chain table information; In a case where it is determined that the data of the bad block exists in the local cache, the bad block is repaired by using the local cache; In a case where it is determined that the data of the bad block does not exist in the local cache, a target node is determined from the remaining nodes.
2. The method of claim 1, wherein the method further comprises: Determining that the current node has a bad block is achieved by the following steps: Data in each data block of the disk of the current node is read, and a check parameter is calculated; The calculated check parameter is compared with a preset check parameter; In a case where the comparison is inconsistent, the data block is determined to be a bad block; In a case where the comparison is consistent, a header field of the data block is checked; In a case where there is a logical error in the header field of the data block, the data block is determined to be a bad block.
3. The method of claim 1, wherein the method further comprises: The obtained data is checked, and in a case where the checking is passed, the bad block is repaired by using the data of the data block, which comprises: In a case where it is determined that the master node has a bad block, a database transaction log sequence number LSN of the corresponding data block obtained from the target node is determined; The LSN of the corresponding data block obtained from the target node is compared with an LSN of a current database in the master node; In a case where the LSN of the corresponding data block obtained from the target node is equal to or exceeds the LSN of the current database, the bad block is repaired by using the corresponding data block obtained from the target node; In a case where the LSN of the corresponding data block obtained from the target node lags behind the LSN of the current database, the repairing is not performed, and the next round of detection is waited.
4. The method of claim 1, wherein the method further comprises: The obtained data is checked, and in a case where the checking is passed, the bad block is repaired by using the data of the data block, which comprises: In a case where it is determined that the backup node has a bad block, an LSN of a current database in the backup node is compared with an LSN of the corresponding data block obtained from the target node; If the LSN of the corresponding data block obtained from the target node is less than the LSN of the current database, the repairing is performed; If the LSN of the corresponding data block obtained from the target node is greater than or equal to the LSN of the current database, the repairing is not performed.
5. The method of claim 1, wherein the method further comprises: The disk bad block repair method further comprises: after the bad block is repaired with the data of the data block, updating the repaired bad block information to a specified file.
6. The method of claim 1, wherein the method further comprises: The disk bad block repair method further comprises: After reading the disk file and checking out the bad block, a specified dynamic library is loaded, and a preset repair function is called to perform repair.
7. A disk bad block repair apparatus of a flow replication cluster, characterized by, The stream replication cluster comprises at least one master node and a plurality of backup nodes, wherein each backup node is used to backup at least part of data of the at least one master node, and the disk bad block repair device comprises a processor configured to: In a case where it is determined that the current node has a bad block, a target node is determined from the remaining nodes, and the target node contains a data block corresponding to the bad block; According to the bad block parameter of the bad block, a preset function is called to obtain the data of the corresponding data block from the target node; The obtained data is checked, and in a case where the checking is passed, the bad block is repaired with the data of the data block; Determining a target node from the remaining nodes comprises: In a case where it is determined that the master node has a bad block, a target node is selected from the backup nodes by a preset algorithm, so that the data block corresponding to the bad block in the target node has the closest data version; In a case where it is determined that the backup node has a bad block, the master node is taken as the target node; In a case where it is determined that the current node has a bad block, the bad block chain table information is read, and it is determined whether the data of the bad block exists in the local cache based on the bad block chain table information; In a case where it is determined that the data of the bad block exists in the local cache, the bad block is repaired by using the local cache; In a case where it is determined that the data of the bad block does not exist in the local cache, a target node is determined from the remaining nodes.
8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the disk bad block repair method of the stream replication cluster according to any one of claims 1 to 6.
Citation Information
Patent Citations
Block storage node data restoration method and storage medium
CN111488238A
Automated backup and restore of a disk group
US20190377643A1